Method for training neural network with elastic feature compression capability, and neural network system with elastic feature compression capability

Abstract

A neural network is provided that includes a layer having a set of weights. The neural network is trained based on a first compression quality level, where the set of weights and a first set of batch normalization coefficients are used for the layer, so that the set of weights and the first set of batch normalization coefficients are trained for the first compression quality level. The neural network is then trained based on a second compression quality level, where the set of weights that have been trained for the first compression quality level and a second set of batch normalization coefficients are used for the layer, so that the set of weights are trained for the first and second compression quality levels, and the second set of batch normalization coefficients are trained for the second compression quality level.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of U.S. Provisional Patent Application No. 63 / 345,918, filed on May 26, 2022, the entire contents of which are incorporated herein by reference. Technical Field

[0003] The present disclosure relates to a neural network, and more particularly to a neural network with elastic feature compression capability. Background Art

[0004] Artificial neural networks (or simply "neural networks") are usually composed of multiple layers of artificial neurons. Each layer can perform a transformation on its input and produce an output, which serves as the input of the next layer. For example, a convolutional neural network includes multiple convolutional layers, each of which can include multiple convolutional kernel maps and a set of batch normalization coefficients to perform convolution and batch normalization on an input feature map and produce an output feature map for use by the next layer.

[0005] However, the memory capacity of the neural network accelerator is usually limited and insufficient to store all the convolution kernel graphs, batch normalization coefficient groups and feature maps generated during the neural network operation, so external memory is usually required to store these data. Therefore, the operation of the neural network involves the transmission of a large amount of data between the neural network accelerator and the external memory, which will cause power consumption and delay. Summary of the invention

[0006] Therefore, an object of the present disclosure is to provide a method for training a neural network so that the neural network has elastic feature compression capability.

[0007] According to the present disclosure, the neural network includes a plurality of neuron layers, one of which has a weight set and contains a data compression program using a data compression-decompression algorithm. The method includes the following steps: (A) training the neural network according to a first compression setting corresponding to a first compression quality level by a neural network accelerator, wherein during the training of the neural network in step (A), a first set of batch normalization coefficients corresponding to the first compression quality level is used for one of the neuron layers; (B) outputting the weight set (optional) and the first set of batch normalization coefficients trained in step (A) by the neural network accelerator for use by the neural network in the following situations: the neural network is executed to decompress and multiply-accumulate a feature map to be processed in one of the neuron layers substantially according to the first compression quality level; (C) training the neural network according to a second compression setting corresponding to a second compression quality level different from the first compression quality level by the neural network accelerator, wherein during the training of the neural network in step (C), the weight set (optional) and the first set of batch normalization coefficients trained in step (A) are outputted for use by the neural network in the following situations: the neural network is executed to decompress and multiply-accumulate a feature map to be processed in the one of the neuron layers substantially according to the first compression quality level; (C) training the neural network according to a second compression setting corresponding to a second compression quality level different from the first compression quality level by the neural network accelerator, wherein during the training of the neural network in step (C), the weight set (optional) and the first set of batch normalization coefficients trained in step (A) are outputted. The weight set trained in step (A) and step (C) and a second set of batch normalization coefficients corresponding to the second compression quality level are used in one of the neural network layers; and (D) outputting the weight set trained in both step (A) and step (C) and the second set of batch normalization coefficients trained in step (C) by the neural network accelerator, wherein the weight set trained in both step (A) and step (C) is used by the neural network when the neural network is executed to decompress and multiply-accumulate the compressed feature map to be processed in one of the neural network layers substantially according to either the first compression quality level or the second compression quality level, and the second set of batch normalization coefficients trained in step (C) is used by the neural network when the neural network is executed to decompress and multiply-accumulate the compressed feature map to be processed in one of the neural network layers substantially according to either the first compression quality level or the second compression quality level. At least one of the first compression quality level and the second compression quality level is a lossy compression level.

[0008] Another object of the present disclosure is to provide a neural network system with flexible feature compression capabilities. The neural network system includes a neural network accelerator and a memory device. In some embodiments, the neural network accelerator is configured to execute the neural network that has been trained using the method disclosed herein. The memory device is accessible to the neural network accelerator and stores the weight set that has been trained in the method, the first set of batch normalization coefficients that have been trained in the method, and the second set of batch normalization coefficients that have been trained in the method. The neural network accelerator is configured to perform the following operations: (a) select one of the first compression quality level and the second compression quality level for one of the neural layers; (b) store a compressed input feature map corresponding to one of the neural layers and compressed at the selected one of the first compression quality level and the second compression quality level in the memory device; (c) load the compressed input feature map from the memory device to one of the neural layers; (d) decompress the compressed input feature map for the selected one of the first compression quality level and the second compression quality level to obtain a decompressed (e) loading the weight set from the memory device; (f) performing a multiply-accumulate operation on the decompressed input feature map using the weight set to generate an operational feature map; (g) loading one of the first set of batch normalization coefficients and the second set of batch normalization coefficients corresponding to the selected one of the first compression quality level and the second compression quality level from the memory device; and (h) performing batch normalization on the operational feature map using the loaded one of the first set of batch normalization coefficients and the second set of batch normalization coefficients to generate a normalized feature map for use by the next neuron layer.

[0009] In some embodiments, the neural network accelerator is configured to enable a neural network including multiple neural layers to perform corresponding operations. The memory device is accessible to the neural network accelerator and stores a weight set corresponding to one of the neural layers and multiple groups of batch normalization coefficients corresponding to one of the neural layers. The weight set is applicable to multiple compression quality levels, and each of the multiple groups of batch normalization coefficients is applicable to a respective one of the compression quality levels. The neural network accelerator is configured to perform the following operations: (a) select one of the compression quality levels for one of the neural layers; (b) store a compressed input feature map corresponding to one of the neural layers and compressed at one of the selected compression quality levels into the memory device; (c) load the compressed input feature map from the memory device to one of the neural layers; (d) decompress the compressed input feature map for one of the selected compression quality levels to obtain a decompressed input feature map; (e) load the compressed input feature map from the memory device to the one of the neural layers. (f) using the weight set to perform a multiply-accumulate operation on the decompressed input feature map to generate an operational feature map; (g) loading from the memory device one of the multiple sets of batch normalization coefficients applicable to the selected one of the compression quality levels; and (h) using the loaded one of the multiple sets of batch normalization coefficients to perform batch normalization on the operational feature map to generate a normalized feature map for use by a next neuron layer, the next neuron layer being a neuron layer immediately following the one of the neuron layers in the neuron layers. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Other features and advantages of the present disclosure will become apparent in the following detailed description of the embodiments and with reference to the accompanying drawings. Please note that various features may not be drawn to scale.

[0011] Figure 1 is a schematic diagram illustrating a convolutional neural network.

[0012] Figure 2 is a block diagram illustrating an embodiment of a neural network system according to the present disclosure.

[0013] Figure 3 is a block diagram illustrating an embodiment including multiple sets of batch normalization coefficients corresponding to multiple compression quality levels of a single layer.

[0014] Figure 4 is a flow chart illustrating the operation of the embodiment.

[0015] Figure 5is a flow chart illustrating an embodiment of a method for training a neural network according to the present disclosure.

[0016] Figure 6 is a block diagram illustrating an embodiment of the method for training the neural network in more detail.

[0017] Figure 7 is a block diagram illustrating a bottleneck residual block of a MobileNet architecture.

[0018] Figure 8 is a block diagram illustrating a scenario in which the embodiment of the neural network system is implemented in the bottleneck residual block of the MobileNet architecture.

[0019] Fig. 9 is a block diagram illustrating a ResNet architecture.

[0020] Fig.10 is a block diagram illustrating a scenario in which the embodiment of the neural network system is implemented in a portion of a ResNet architecture. DETAILED DESCRIPTION

[0021] Before the present disclosure is described in more detail, it should be noted that in the following description, reference numerals in the figures or terminal portions of reference numerals have been repeated between the figures to indicate corresponding or similar elements, which may optionally have similar features.

[0022] See also Figure 1, a neural network is depicted including multiple neuron layers, each neuron layer performs a transformation on its input to produce an output, wherein the neural network can be configured for, for example, artificial intelligence (AI) denoising, AI style transfer, AI temporal super-resolution, AI spatial super-resolution, or AI image generation, but the disclosure is not limited thereto. The neuron layer may include multiple computational layers. Each computational layer outputs a feature map (also called an "activation map") as input to the next layer. In some embodiments, each computational layer may perform a multiply-accumulate (e.g., convolution) operation, pooling (optional), batch normalization (BN) operation, and activation operation on the input feature map. The pooling operation can be omitted, and a computing layer uses one or more weights (hereinafter referred to as "weight sets", note that, for example, in convolutional neural networks, the term "weights" can sometimes be interchanged with "cores") to perform a multiplication-accumulation operation on the input feature map to generate an operational feature map (wherein the number of weights in the weight set corresponds to the number of channels of the operational feature map), uses a set of BN coefficients to perform batch normalization on the operational feature map to generate a normalized feature map, and then uses an activation function to process the normalized feature map to generate an output feature map, which serves as an input feature map of the next layer. In this embodiment, the neural network is taken as but not limited to a convolutional neural network (CNN) as an example, and the neuron layer of the CNN may include multiple convolutional layers (i.e., the computing layer mentioned above), optionally including one or more fully connected (FC) layers connected to each other. Each of the convolutional layer and the FC layer outputs a feature map (also called an "activation map"), which serves as an input of the next layer. In an exemplary embodiment, each convolution layer performs convolution (corresponding to the aforementioned multiplication and accumulation operation), pooling (optional), batch normalization (BN), and activation operations on an input feature map. In this embodiment, the pooling operation is omitted, and a convolution layer uses one or more core maps (referred to as "core map sets" in the exemplary embodiments below) to perform convolution on the input feature map to generate a convolution feature map (i.e., the above-mentioned operation feature map) (wherein the number of the (multiple) core maps in the core map set corresponds to the number of channels of the convolution feature map), batch normalization is performed on the convolution feature map using a set of BN coefficients to generate a normalized feature map, and then the normalized feature map is processed using an activation function to generate an output feature map, which serves as an input feature map of the next layer. The set of BN coefficients may include a set of scaling coefficients and a set of offset coefficients. During the batch normalization, in a first step, the convolution feature map may be normalized using its mean and standard deviation pair to obtain a preliminary normalized feature map. Subsequently, the elements of the preliminary normalized feature map may be multiplied by the scaling factor and then added to the offset factor to obtain the normalized feature map. In other words, the batch normalization may include the steps of normalization, scaling and offset.

[0023] See also Figure 2 , shows an embodiment of a neural network system with flexible feature compression capability according to the present disclosure, which includes a neural network accelerator 1 (hereinafter referred to as accelerator 1), and a memory device 2 physically separated but electrically connected to the accelerator 1. The accelerator 1 can be implemented using, for example, a graphics processing unit (GPU), a field programmable gate array (FPGA), an application-specific integrated circuit (ASIC), etc., and the present disclosure is not limited thereto. The accelerator 1 includes a computing unit 11 to perform the above-mentioned convolution, batch normalization and activation function. The computing unit 11 may include, for example, a processor core, a convolution circuit, a register, etc., but the present disclosure is not limited thereto. The memory device 2 may use, for example, a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a synchronous graphic random access memory (SGRAM), a high bandwidth memory (HBM), a flash memory, a solid state drive, a hard disk, other appropriate memory devices, or any combination thereof, but the present disclosure is not limited thereto. The term "memory device" is a general concept and does not necessarily mean homogeneous and monolithic memory. In some embodiments, the memory device 2 may include one or more on-chip memory arrays. In some embodiments, the memory device 2 may include one or more external memory chips. In some examples, the memory device 2 may be distributed in a neural network system. In an exemplary embodiment, the memory device 2 is an external memory device including one or more external memory chips, but the disclosure is not limited thereto. Due to the limited memory capacity of the accelerator 1, the core atlas of each convolutional layer ( Figure 2 The accelerator 1 loads the corresponding core atlas, BN coefficients and feature maps from the external memory device 2.

[0024] In this embodiment, the computing unit 11 compresses the output feature map of one or more neural layers to reduce the data transmission between the accelerator 1 and the external memory device 2, and thereby reduce the power consumption and delay of the neural network. In addition, the computing unit 11 is configured to selectively use one of a plurality of predetermined compression quality levels to perform data compression for each neural layer used to compress the output feature data, and the BN coefficients corresponding to the neural layer include a plurality of sets of BN coefficients trained for the predetermined compression quality levels, such as Figure 3As shown. Different compression quality levels correspond to different compression ratios. Generally, a higher compression quality level corresponds to a smaller compression ratio. The computing unit 11 can select a predetermined compression quality level according to the compression quality setting determined by the user, or according to various operation conditions of the neural network system, wherein the various operation conditions include, for example, the workload of the accelerator 1 (such as selecting a lower compression quality when the workload is large), the temperature of the accelerator 1 (which can be obtained through a temperature sensor) (such as selecting a lower compression quality when the temperature is high), the battery power (when the power of the neural network system is supplied by a battery device) (such as when the battery power is low, select a lower compression quality), the available storage space of the memory device 2 (such as when the available storage space is insufficient, select a lower compression quality), the available bandwidth of the memory device 2 (such as when the available bandwidth is narrow, select a lower compression quality), the time length set for completing the task to be completed by the neural network (such as when the time length set in this way is short, select a lower compression quality), the type of task to be completed by the neural network (such as when the task is to preview an image, select a lower compression quality), etc., but the present disclosure is not limited to this.

[0025] See also Figure 4 For the sake of brevity, the operation of the computing unit 11 to implement elastic feature compression will be described for a single neuron layer. In fact, the described operation can be implemented in multiple neuron layers.

[0026] In step S1, the computing unit 11 selects one of the predetermined compression quality levels for the neuron layer, and loads a compressed input feature map corresponding to the neuron layer from the external memory device 2. The compressed input feature map is an output of the last neuron layer (i.e., a neuron layer immediately before the neuron layer), and has been compressed using a predetermined compression quality level that is the same as the predetermined compression quality level selected for the neuron layer. In the present embodiment, this compression is performed using a JPEG or JPEG-like (e.g., some operations of JPEG compression, such as header encoding, can be omitted) compression method, which is a lossy compression. It should be noted that the compressed input feature map can be composed of multiple compressed parts, and due to the limited memory capacity of the accelerator 1, the computing unit 11 can load one of the compressed parts at a time for subsequent steps.

[0027] In step S2, the computing unit 11 decompresses the compressed input feature map with respect to the selected predetermined compression quality level to obtain a decompressed input feature map.

[0028] In step S3, in step S3, the computing unit 11 loads a core atlas corresponding to the neuron layer and trained for each predetermined compression quality level from the external memory device 2, and uses the core atlas to convolve the decompressed input feature map to generate a convolution feature map.

[0029] In step S4, the computing unit 11 loads a set of batch normalization coefficients that have been trained for the selected predetermined compression quality level from the external memory device 2, and uses the loaded set of batch normalization coefficients to perform batch normalization on the convolutional feature map to generate a normalized feature map for use by the next neuron layer. The next neuron layer is a neuron layer immediately following the neuron layer.

[0030] In step S5, the computing unit 11 processes the normalized feature map using an activation function to generate an output feature map. The activation function may be, for example, a rectified linear unit (ReLU), a leaky ReLU, a sigmoid linear unit (SiLU), a Gaussian error linear unit (GELU), other appropriate functions or any combination thereof.

[0031] In step S6, the computing unit 11 selects a predetermined compression quality level for the next neuron layer, compresses the output feature map using the predetermined compression quality level selected for the next neuron layer, and stores the compressed output feature map in the external memory device 2. The compressed output feature map will serve as the compressed input feature map of the next neuron layer. Step S6 is a data compression process using a JPEG or JPEG-like compression method in this embodiment, but the present disclosure is not limited to any specific compression method.

[0032] Figure 5 is a flow chart illustrating the steps of an embodiment of a method for training a neural network for the above-mentioned neural network system and having elastic feature compression capability. For the sake of brevity, these steps may be described for a single neuron layer (hereinafter referred to as a "specific neuron layer") of the neural network, but these steps may also be applied to other neuron layers.

[0033] Through steps S11 to S16, the accelerator 1 trains the neural network according to a first compression quality setting indicating or corresponding to a first compression quality level (which is one of the predetermined compression quality levels), wherein a first set of batch normalization coefficients corresponding to the first compression quality level is a core atlas and the first set of batch normalization coefficients used for the specific neuron layer to train the specific neuron layer. Subsequently, the accelerator 1 outputs the core atlas and the first set of batch normalization coefficients trained through steps S11 to S16 for use by the neural network when performing decompression and multiplication (convolution) of a feature map to be processed compressed in the specific neuron layer substantially according to the first compression quality level. The term "substantially" as used herein may generally mean that the error of a given value or range is within 20%, preferably within 10%. For example, in practice, the core atlas trained for a compression quality level of 80 and the first set of batch normalization coefficients can be used to perform decompression and multiplication accumulation (e.g., convolution) on the compressed feature map to be processed according to a compression quality level of 75 (since the error will be (80-75) / 80=6.25%, 75 is consistent with the above interpretation of "substantially").

[0034] In step S11 , the accelerator 1 performs a first compression-related data processing on a first input feature map to obtain a first processed feature map, wherein the first compression-related data processing is associated with data compression having the first compression quality level.

[0035] In step S12, the accelerator 1 performs a first decompression-related data processing on the first processed feature map to obtain a second processed feature map, wherein the first decompression-related data processing is related to data decompression and corresponds to the first compression quality level.

[0036] See also Figure 6In this embodiment, the accelerator 1 uses the JPEG algorithm for paired compression and decompression as the first compression-related data processing and the first decompression-related data processing, respectively, but the present disclosure is not limited to the use of the JPEG algorithm. First, the accelerator 1 generates a quantization table (Q-table) according to the first compression quality level (i.e., a predetermined compression quality level indicated by the first compression quality setting), and uses the generated Q-table to perform the first compression-related data processing and the first decompression-related data processing. Optionally, the accelerator 1 may round the elements of the Q-table to the nearest power of 2 to simplify the subsequent quantization process in the first compression-related data processing and the subsequent inverse quantization process in the first decompression-related data processing. JPEG compression (i.e., compression of the JPEG algorithm) is a lossy compression, which can be divided into a first part and a second part following the first part. The first part is a lossy part, which includes discrete cosine transform (DCT) and quantization, wherein quantization is a lossy operation. The second part is a lossless part, which includes differential pulse code modulation (DPCM) encoding of DC coefficients, zigzag scanning and run-length encoding of AC coefficients, Huffman encoding and header encoding, each of which is a lossless operation (i.e., the second part includes only lossless operations). The paired decompression of the JPEG algorithm includes the inverse operations of the above compression operations, such as header parsing, Huffman decoding, run-length decoding and inverse zigzag scanning of AC coefficients, DPCM decoding, inverse quantization and inverse DCT of DC coefficients. Since one of the purposes of training the neural network is to correctly train the core atlas and the set of batch normalization coefficients and the compression of the lossless second part and the decompression of the corresponding part have no effect on the training results, the compression of the second part and the decompression of the corresponding part can be omitted during training, thereby reducing the overall time required to train the neural network. In other words, in this embodiment, the first compression-related data processing may include only the first part of JPEG compression (e.g., only DCT and quantization), and the first decompression-related data processing may include only the inverse operation of the first part of JPEG compression (e.g., only inverse quantization and inverse DCT). However, when the trained neural network is used in actual application, partial compression of the first part and the second part is performed to achieve the purpose of reducing the amount of data, and the same is true for the decompression of the corresponding parts.

[0037] In step S13, the accelerator 1 uses the core atlas to perform convolution on the second processed feature map to generate a first convolution feature map.

[0038] In step S14, the accelerator 1 uses the first set of batch normalization coefficients to perform batch normalization on the first convolutional feature map to obtain a first normalized feature map for use by a next neuron layer, the next neuron layer being a neuron layer immediately following the specific neuron layer. The first set of batch normalization coefficients may include a set of scaling coefficients and a set of offset coefficients for performing scaling and offsetting in the batch normalization performed on the first convolutional feature map.

[0039] In step S15, the accelerator 1 uses an activation function to process the first normalized feature map, and uses the processed first normalized feature map as an input feature map of the next neuron layer.

[0040] In step S16, after the neural network generates a final output, the accelerator 1 performs back propagation on the neural network used in steps S11 to S15 to modify the corresponding core atlas and the corresponding group of batch normalization coefficients for each neuron layer (for example, the core atlas used in step S13 and the first group of batch normalization coefficients used for the specific neuron layer in step S14).

[0041] Thus, each core graph in the core graph set and the first set of batch normalization coefficients for the specific neuron layer have been trained for the first compression quality level.

[0042] After training the neural network using a batch of training data at the first compression quality level, the accelerator 1 outputs the core atlas (optional) and a first set of batch normalization coefficients of the specific neuron layer adapted to the first compression quality level (step S17). Figure 5 and Figure 6 , and then applying the second compression quality setting to select another predetermined compression quality level (hereinafter referred to as "second compression quality level") different from the first compression quality level, wherein one of the first compression quality level and the second compression quality level is a lossy compression level, or both the first compression quality level and the second compression quality level are lossy compression levels. Through steps S21 to S26, the accelerator 1 (see Figure 2) trains the neural network according to the second compression quality setting corresponding to the second compression quality level, wherein the core atlas that has been trained for the first compression quality level through steps S11 to S16 and a second set of batch normalization coefficients corresponding to the second compression quality level are used for the specific neuron layer, so that the core atlas and the second set of batch normalization coefficients that have been trained for the first compression quality level are trained for the second compression quality level. Subsequently, the accelerator 1 outputs the core graph set and the second set of batch normalization coefficients, wherein the core graph has been trained for the first compression quality level through steps S11 to S16 and has been trained for the second compression quality level through steps S21 to 26, for use by the neural network when executing in the specific neuron layer, substantially decompressing and multiplying and accumulating the compressed feature graph to be processed according to either the first compression quality level or the second compression quality level, and the second set of batch normalization coefficients has been trained for the second compression quality level through steps S21 to S26, for use by the neural network when executing in the specific neuron layer, substantially decompressing and multiplying and accumulating the compressed feature graph to be processed according to the second compression quality level.

[0043] In step S21 , the accelerator 1 performs a second compression-related data processing on a second input feature map to obtain a third processed feature map, wherein the second compression-related data processing is related to data compression with the second compression quality level.

[0044] In step S22, the accelerator 1 performs a second decompression-related data processing on the second processed feature map to obtain a fourth processed feature map, wherein the second decompression-related data processing is related to data decompression and the second compression quality level.

[0045] The accelerator 1 generates a Q-table according to the second compression quality level, and uses the generated Q-table to perform the second compression-related data processing and the second decompression-related data processing. The details of the second compression-related data processing and the second decompression-related data processing are similar to those of the first compression-related data processing and the first decompression-related data processing, and are not repeated here for the sake of brevity.

[0046] In step S23, the accelerator 1 performs convolution on the fourth processed feature map using the core atlas modified in step S16 to generate a second convolution feature map.

[0047] In step S24, the accelerator 1 performs batch normalization on the second convolution feature map using the second set of batch normalization coefficients to obtain a second normalized feature map for use by the next neuron layer. The second set of batch normalization coefficients may include a set of scaling coefficients and a set of offset coefficients for performing scaling and offsetting in the batch normalization performed on the second convolution feature map.

[0048] In step S25, the accelerator 1 uses the activation function to process the second normalized feature map, and uses the processed second normalized feature map as an input feature map of the next neuron layer.

[0049] In step S26, after the neural network generates a final output, the accelerator 1 performs backpropagation on the neural network used in steps S21 to S25 to modify the corresponding core atlas and the corresponding set of batch normalization coefficients for each neuron layer (for example, the core atlas modified in step S16 and used in step S23, and the second set of batch normalization coefficients used for the specific neuron layer in step S24).

[0050] Therefore, each core graph in the core atlas and the second set of batch normalization coefficients of the specific neuron layer have been trained for the second compression quality level. In step S27, the accelerator 1 outputs the core atlas applicable to the first compression quality level and the specific neuron layer of the second compression quality level, and the second set of batch normalization coefficients applicable to the specific neuron layer of the first compression quality level.

[0051] In some embodiments, steps S11 to S16 may be iteratively performed using multiple mini-batches of training data sets, and / or steps S21 to S26 may be iteratively performed using multiple mini-batches of training data sets. A mini-batch is a subset of a training data set. In some embodiments, a mini-batch may include 256, 512, 1024, 2048, 4096, or 8192 training samples, but the disclosure is not limited to these specific numbers. Batch gradient descent training is a special case in which the mini-batch size is set to the total number of examples in the training data set. Stochastic gradient descent (SGD) training is another special case in which the mini-batch size is set to 1. In some embodiments, the iterations of steps S11 to S16 and the iterations of steps S21 to S26 do not need to be performed in any particular order. In other words, the iterations of steps S11 to S16 and the iterations of steps S21 to S26 may be performed alternately (e.g., in the order of S11 to S16, S21 to S26, S11 to S16, S21 to S26, ..., and finally S17 and S27). It should be noted that step S17 is not necessarily performed before steps S21 to S26. In other embodiments, step S17 may be performed together with step S27. The present disclosure does not limit the specific order of step S17 and steps S21-S26.

[0052] As a result, for the particular neuron layer, the core atlas has been trained for both the first compression quality level and the second compression quality level, the first set of batch normalization coefficients has been trained for the first compression quality level, and the second set of batch normalization coefficients has been trained for the second compression quality level. If desired, the particular neuron layer can be trained for other compression quality levels in a similar manner, so that the core atlas of the particular neuron layer is trained for multiple additional compression quality levels, and the particular neuron layer includes additional multiple sets of batch normalization coefficients trained for the no additional compression quality levels, respectively, and the present disclosure is not limited to only two compression quality levels. In addition, each neuron layer of the neural network can be trained in the same manner as the particular neuron layer, so that the neural network is suitable for multiple compression quality levels and has flexible feature compression capabilities.

[0053] Figure 7 exemplarily shows a bottleneck residual block of a MobileNet architecture, and Figure 8 It is illustrated how to implement a bottleneck residual block using an embodiment of the neural network system, wherein Figure 8 Blocks A, B, and C in Figure 7 The accelerator 1 converts an uncompressed feature map M ALoad from the external memory device 2 into its on-chip buffer and load a core atlas K A The uncompressed feature map M A Perform a 1×1 convolution (see Figure 7 ), followed by batch normalization and ReLU6 functions (see Figure 7 The “batch normalization” and “ReLU6” of block A in , thereby generating a feature map M B The accelerator 1 loads the BN coefficient group BN A Then, the accelerator 1 selects a Q-table corresponding to a predetermined compression quality level indicated by the compression quality setting S_B to compress the feature map M B , and the compressed feature map cM B When the process proceeds to block B, the accelerator 1 loads the compressed feature map cM from the external memory device 2. B and decompress the compressed feature map cM using the Q-table selected according to the compression quality setting S_B B The operations of step B and step C are similar to those of step A and will not be repeated here. After the batch normalization of block C, the accelerator 1 loads the uncompressed feature map M A And the uncompressed feature map M A The outputs of block C are aggregated (e.g., summed or concatenated) together to produce an uncompressed feature map M D The compression quality settings S_B and S_C may indicate the same compression quality level or different compression quality levels, and the present disclosure is not limited thereto.

[0054] Fig. 9 A ResNet architecture is shown exemplarily, and Fig.10 A portion of the ResNet architecture implemented using an embodiment of the neural network system is shown ( Fig. 9 The part surrounded by the dotted line), where Fig.10 Blocks D and E in Fig. 9 The accelerator 1 loads a compressed feature map cM compressed at a compression quality level indicated by the compression quality setting S_D from the external memory device 2. D , decompressing the compressed feature map cM using a Q-table selected according to the compression quality setting S_D D , and the decompressed feature map dM D Then, the accelerator 1 loads a core atlas KD , to decompress the feature map dM D Perform a 3×3 convolution (see Fig. 9 ), followed by batch normalization and ReLU functions (see Fig. 9 The “batch normalization” and “ReLU” of block D in the above example generate a feature map M. E The accelerator 1 loads the BN coefficient group (BN D1 , BN D2 ...) one of the BN coefficient groups corresponding to a predetermined compression quality level indicated by the compression quality setting S_D, thereby performing the batch normalization. Then, the accelerator 1 selects a Q-table corresponding to a predetermined compression quality level indicated by the compression quality setting S_E to compress the feature map M E , and the compressed feature map cM E When the process proceeds to block E, the accelerator 1 loads the compressed feature map cM from the external memory device 2. E and decompress the compressed feature map cM using the Q-table selected according to the compression quality setting S_E E The operation of block E is similar to that of block D, so for the sake of brevity, it is not repeated here. After the batch normalization of block E, the accelerator 1 loads the decompressed feature map dM from the on-chip buffer. D , and the decompressed feature map dM D The outputs of block E are aggregated (eg, summed or connected) together to obtain a result feature map. Then, the accelerator 1 performs a ReLU function on the result feature map to generate a feature map M F , compressing the feature map M using a Q-table selected according to the compression quality setting S_F F , and the compressed feature map cM F The compression quality settings S_D, S_E, and S_F may indicate the same compression quality level or different compression quality levels, but the present disclosure is not limited thereto.

[0055] Table 1 compares the present embodiment with the prior art using two ResNet neural networks represented by ResNet-A and ResNet-B, wherein the prior art uses only one set of batch normalization coefficients for different compression quality levels in a single neuron layer, while the embodiment disclosed herein uses multiple different sets of batch normalization coefficients for different compression quality levels in a single neuron layer. Four compression levels corresponding to four quality levels were tested. Taking ResNet-A as an example, the prior art achieved accuracies of 69.7%, 66.8%, 42.6% and 14.9% according to the four compression levels, respectively. In contrast, the present embodiment achieved 69.8%, 69.1%, 66.6% and 64%, which is 49.1% better than the baseline (64%-14.9%=49.1% at quality level 50). Experiments on ResNet-B also show that the present embodiment enables a neural network to adapt to multiple (four in the example) compression quality levels better than the prior art.

[0056] Table 1

[0057]

[0058] In summary, according to the embodiments of the neural network system disclosed herein, for a single neuron layer, a core graph set that has been trained for multiple predetermined compression quality levels, and multiple groups of batch normalization coefficients that have been trained for the multiple predetermined compression quality levels, respectively, are included, and thus the neural network system has flexible feature compression capabilities. In some embodiments, during the training of the neural network, compression-related training only includes the lossy part of the full compression process (i.e., the lossless part is omitted), and decompression-related training only includes the reverse operation of the lossy part of the full compression process, thereby reducing the total time required for training.

[0059] In the above description, for the purpose of explanation, many specific details are provided in order to fully understand the (multiple) embodiments. However, it is obvious to a skilled person that one or more other embodiments can be implemented without involving some of these specific details. It should also be noted that the references to "one embodiment", "one implementation method", and embodiments indicated by serial numbers in this specification mean that a specific feature, structure or characteristic can be included in the practice of the present disclosure. It should be further noted that in the description, various features are sometimes intentionally combined in a single embodiment, figure or description to simplify the disclosure and illustrate the understanding of various innovative aspects; this does not mean that every feature needs to be implemented in the presence of all other features. In other words, in any described embodiment, when the implementation of one or more features or specific details does not affect the implementation of another one or more features or specific details, the group or multiple features can be selected to be implemented separately without the other one or more features or specific details. It should also be further noted that when implementing the present disclosure, one or more features or specific details from one embodiment can be implemented together with one or more features or specific details from another embodiment as needed.

[0060] However, the above is only an embodiment of the present invention and should not be used to limit the scope of the present invention. All simple equivalent changes and modifications made according to the claims and the description of the present invention are still within the scope of the present invention.

Claims

1. A method for training a neural network, the neural network comprising a plurality of neuron layers, wherein one neuron layer has a weight set and contains a data compression program using a data compression-decompression algorithm, the method comprising the following steps: (A) training the neural network according to a first compression setting corresponding to a first compression quality level by a neural network accelerator, wherein during the training of the neural network in step (A), a first set of batch normalization coefficients corresponding to the first compression quality level is used for the one of the neural layers; (B) outputting the first set of batch normalization coefficients trained in step (A) for use by the neural network when: the neural network is executed to decompress and perform multiplication and accumulation on a feature map to be processed compressed according to the first compression quality level in one of the neural layers; (C) training the neural network by the neural network accelerator according to a second compression setting corresponding to a second compression quality level different from the first compression quality level, wherein during the training of the neural network in step (C), the set of weights that have been trained in step (A) and a second set of batch normalization coefficients corresponding to the second compression quality level are used in one of the neuron layers; and (D) outputting the weight set trained in both step (A) and step (C) and the second set of batch normalization coefficients trained in step (C), wherein the weight set trained in both step (A) and step (C) is used by the neural network when the neural network is executed to decompress and perform multiplication accumulation on the compressed feature map to be processed in one of the neural layers substantially according to either the first compression quality level or the second compression quality level, and the second set of batch normalization coefficients trained in step (C) is used by the neural network when the neural network is executed to decompress and perform multiplication accumulation on the compressed feature map to be processed in one of the neural layers substantially according to the second compression quality level; At least one of the first compression quality level and the second compression quality level is a lossy compression level.

2. The method according to claim 1, wherein: Step (A) comprises the following sub-steps: (A-1) performing a first compression-related data processing on a first input feature map to obtain a first processed feature map, wherein the first compression-related data processing is associated with the data compression-decompression algorithm using the first compression quality level; (A-2) performing a first decompression-related data processing on the first processed feature map to obtain a second processed feature map, wherein the first decompression-related data processing is related to data decompression and the first compression quality level; (A-3) using the weight set to perform a multiply-accumulate operation on the second processed feature map to generate a first operated feature map; (A-4) performing batch normalization on the first operational feature map using the first set of batch normalization coefficients to obtain a first normalized feature map for use by a next neuron layer, wherein the next neuron layer is a neuron layer immediately following the one of the neuron layers among the neuron layers; and (A-5) performing back propagation on the neural network used in sub-steps (A-1) to (A-4) to modify the set of weights and the first set of batch normalization coefficients; and Step (C) comprises the following sub-steps: (C-1) performing a second compression-related data processing on a second input feature map to obtain a third processed feature map, wherein the second compression-related data processing is associated with the data compression-decompression algorithm using the second compression quality level; (C-2) performing a second decompression-related data processing on the third processed feature map to obtain a fourth processed feature map, wherein the second decompression-related data processing is related to data decompression and the second compression quality level; (C-3) performing a multiply-accumulate operation on the fourth processed feature map using the weight set modified in sub-step (A-5) to generate a second operated feature map; (C-4) performing batch normalization on the second operational feature map using the second set of batch normalization coefficients to obtain a second normalized feature map for use by the next neuron layer; and (C-5) performing back propagation on the neural network used in sub-steps (C-1) to (C-4) to modify the weight set and the second set of batch normalization coefficients that have been modified in sub-step (A-5).

3. The method according to claim 2, wherein: The data compression-decompression algorithm includes a lossy portion having a lossy operation, and a lossless portion following the lossy operation of the lossy portion, and the first compression-related data processing and the second compression-related data processing each include only the lossy portion of the lossy compression; and Each of the first decompression-related data processing and the second decompression-related data processing only includes an inverse operation of the lossy portion of the lossy compression.

4. The method according to claim 3, wherein: Each of the first compression-related data processing and the second compression-related data processing consists of only a discrete cosine transform (DCT) and a quantization operation.

5. The method according to claim 2, wherein: The first set of batch normalization coefficients includes a first set of scaling coefficients and a first set of offset coefficients, the first set of scaling coefficients being used to perform a scaling operation in the batch normalization performed on the first operating feature map, and the first set of offset coefficients being used to perform an offset operation in the batch normalization performed on the first operating feature map; and The second set of batch normalization coefficients includes a second set of scaling coefficients and a second set of offset coefficients, the second set of scaling coefficients being used to perform a scaling operation in the batch normalization performed on the second operating characteristic map, and the second set of offset coefficients being used to perform an offset operation in the batch normalization performed on the second operating characteristic map.

6. The method according to claim 1, wherein: Step (A) and step (C) are performed iteratively and interleaved using multiple small batches of training data sets.

7. The method according to claim 1, wherein: The neural network is one of those used for artificial intelligence (AI) denoising, AI style transfer, AI temporal super-resolution, AI spatial super-resolution and AI image generation.

8. A neural network system, comprising: a neural network accelerator configured to execute the neural network trained using the method of claim 1; and a memory device accessible to the neural network accelerator and storing the weight set trained in the method, the first set of batch normalization coefficients trained in the method, and the second set of batch normalization coefficients trained in the method; in, The neural network accelerator assembly is configured to perform the following operations: selecting one of the first compression quality level and the second compression quality level for the one of the neuron layers; storing a compressed input feature map corresponding to the one of the neural layers and compressed at the selected one of the first compression quality level and the second compression quality level in the memory device; Loading the compressed input feature map from the memory device to one of the neuron layers; decompressing the compressed input feature map for the selected one of the first compression quality level and the second compression quality level to obtain a decompressed input feature map; loading the weight set from the memory device; Using the weight set to perform a multiply-accumulate operation on the decompressed input feature map to generate an operational feature map; loading from the memory device one of the first set of batch normalization coefficients and the second set of batch normalization coefficients corresponding to the selected one of the first compression quality level and the second compression quality level; and Batch normalization is performed on the computational feature map using the one of the first set of batch normalization coefficients and the second set of batch normalization coefficients loaded therein to generate a normalized feature map for use by the next neuron layer.

9. The neural network system according to claim 8, wherein: The neural network accelerator is also configured to perform the following operations: Processing the normalized feature map using an activation function to generate an output feature map; selecting one of the first compression quality level and the second compression quality level for the next neuron layer; compressing the output feature map at one of the first compression quality level and the second compression quality level selected for the next neuron layer; and The compressed output feature map is stored in the memory device.

10. The neural network system according to claim 9, wherein: The memory device includes an external memory chip storing the compressed input feature map and the compressed output feature map.

11. The neural network system according to claim 8, wherein: The first set of batch normalization coefficients includes a first set of scaling coefficients and a first set of offset coefficients for performing a scaling operation and an offset operation in the batch normalization when the first set of batch normalization coefficients is the one of the first set of batch normalization coefficients and the second set of batch normalization coefficients loaded therein; and The second set of batch normalization coefficients includes a second set of scaling coefficients and a second set of offset coefficients, which are used to perform scaling operations and offset operations in the batch normalization when the second set of batch normalization coefficients is the one of the first set of batch normalization coefficients and the second set of batch normalization coefficients loaded therein.

12. The neural network system according to claim 8, wherein: The neural network accelerator is configured to select one of the first compression quality level and the second compression quality level for the one of the neural layers according to at least one selected from the first factor to the seventh factor; and The first factor is a workload of the neural network accelerator, the second factor is a temperature of the neural network accelerator, the third factor is a battery charge of the battery device when the power of the neural network system is provided by a battery device, the fourth factor is an available storage space of the memory device, the fifth factor is an available bandwidth of the memory device, the sixth factor is a length of time set to complete a task to be completed by the neural network, and the seventh factor is a type of the task to be completed by the neural network.

13. The neural network system according to claim 8, wherein: The neural network is one of those used for artificial intelligence (AI) denoising, AI style transfer, AI temporal super-resolution, AI spatial super-resolution and AI image generation.

14. A neural network system comprising: a neural network accelerator configured to enable a neural network including a plurality of neural layers to perform corresponding operations; and a memory device accessible to the neural network accelerator and storing a weight set corresponding to one of the neuron layers and a plurality of batch normalization coefficients corresponding to the one of the neuron layers; in, The set of weights is applicable to a plurality of compression quality levels, and each of the plurality of sets of batch normalization coefficients is applicable to a respective one of the compression quality levels; and in, The neural network accelerator assembly is configured to perform the following operations: selecting one of the compression quality levels for the one of the neuron layers; storing a compressed input feature map corresponding to the one of the neural layers and compressed at the selected one of the compression quality levels in the memory device; Loading the compressed input feature map from the memory device to one of the neuron layers; decompressing the compressed input feature map with respect to the selected one of the compression quality levels to obtain a decompressed input feature map; loading the weight set from the memory device; Using the weight set to perform a multiply-accumulate operation on the decompressed input feature map to generate an operational feature map; loading from the memory device one of the plurality of sets of batch normalization coefficients applicable to the selected one of the compression quality levels; and Batch normalization is performed on the computational feature map using one of the multiple sets of batch normalization coefficients loaded therein to generate a normalized feature map for use by a next neuron layer, the next neuron layer being a neuron layer immediately following the one of the neuron layers in the neuron layers.

15. The neural network system according to claim 14, wherein: The neural network accelerator is also configured to perform the following operations: Processing the normalized feature map using an activation function to generate an output feature map; selecting one of the compression quality levels for the next neuron layer; compressing the output feature map at one of the compression quality levels selected for the next neuronal layer; and The compressed output feature map is stored in the memory device.

16. The neural network system according to claim 15, wherein: The memory device includes an external memory chip storing the compressed input feature map and the compressed output feature map.

17. The neural network system according to claim 14, wherein: Each of the plurality of sets of batch normalization coefficients includes a set of scaling coefficients and a set of offset coefficients for performing a scaling operation and an offset operation in the batch normalization when the set of batch normalization coefficients is the one of the plurality of sets of batch normalization coefficients loaded therein.

18. The neural network system of claim 14, wherein: The neural network accelerator is configured to select one of the compression quality levels for the one of the neural layers based on at least one selected from the first factor to the seventh factor; and The first factor is a workload of the neural network accelerator, the second factor is a temperature of the neural network accelerator, the third factor is a battery charge of the battery device when the power of the neural network system is provided by a battery device, the fourth factor is an available storage space of the memory device, the fifth factor is an available bandwidth of the memory device, the sixth factor is a length of time set to complete a task to be completed by the neural network, and the seventh factor is a type of the task to be completed by the neural network.

19. The neural network system according to claim 14, wherein: The neural network is one of those used for artificial intelligence (AI) denoising, AI style transfer, AI temporal super-resolution, AI spatial super-resolution and AI image generation.

20. The neural network system according to claim 14, wherein: One of the neuron layers is a convolutional layer.

Images