Methods for quantizing, training, and using deep separable convolutional neural networks

Abstract

Methods for quantizing, training, and using deep separable convolutional neural networks are provided. These network layers include one or more pointwise convolution layers adapted to perform pointwise convolutions, the pointwise convolution layers each including a plurality of pointwise convolution layer weights, and one or more depth convolution layers adapted to perform depth convolutions, the depth convolution layers each including a plurality of pointwise convolution layer weights. The quantization methods include quantizing a plurality of pointwise convolution layer weights to a plurality of quantized pointwise convolution layer weights within a first discrete range, and quantizing the plurality of pointwise convolution layer weights to a plurality of quantized pointwise convolution layer weights within a second discrete range, where a cardinality of the first discrete range is strictly less than a cardinality of the second discrete range.

Description

Technical Field

[0001] The subject matter disclosed herein relates to a quantization method for a computer implementation of a deep separable convolutional neural network. The subject matter also relates to a method for training a computer implementation of a deep separable convolutional neural network.

[0002] The subject matter disclosed herein also relates to a computer-implemented method for using a deep separable convolutional neural network on a device. The subject matter disclosed herein also relates to a volatile or non-volatile computer-readable medium comprising data representing a computer program, wherein the computer program includes instructions for causing a processor system to perform one of the methods, and the subject matter disclosed herein further relates to a processor system comprising memory and one or more processors, wherein the memory includes instructions for causing the one or more processors to perform one of the methods. Background Technology

[0003] Convolutional Neural Networks (CNNs) form a key component in many practical applications, such as computer vision, object recognition, image processing, image recognition, image classification, medical imaging, image generation, and other image analysis applications. For example, a CNN can be trained to recognize traffic participants in camera images. Once trained, the CNN can then be used in autonomous vehicles for object recognition tasks, such as identifying traffic participants, like pedestrians, near the vehicle, and enabling the vehicle to react to these other traffic participants when needed, such as by steering, braking, or triggering warnings.

[0004] CNNs typically consist of one or more convolutional layers adapted to perform convolutions on feature inputs, such as image inputs. Feature inputs can have... The size, where M represents the feature dimension, and M represents the number of input channels. For example, in the case of a general RGB image input, M=3. Typically, a convolutional layer consists of a kernel composed of several, for example, N filters. These filters typically have... The size, where Let M represent the kernel dimension, and M correspond to the number of input channels for the feature input. In a general CNN setup, the number of parameters in the convolutional kernel can then be equated to the number of filters multiplied by the size of those filters, i.e. And the computational cost can be .

[0005] Depth-Separable CNNs (DSCNNs) are a typical CNN architecture. While general CNNs consist of one or more convolutional layers for performing convolutions, DSCNNs typically include one or more pointwise convolutional layers (PWCs) and one or more depthwise convolutional layers (DWCs). The execution of general convolutions is decomposed into pointwise convolutions performed through PWC layers and depthwise convolutions performed through DWC layers. This architecture typically reduces the computational demands on the network, making DSCNNs particularly suitable for deployment on resource-constrained and / or mobile devices, such as edge devices.

[0006] In Deep Convolutional (DWC), the convolution kernel can be divided into single-channel forms. A separate filter can be created for each channel of the input data. Given the feature input of a general CNN, which is repeated as described above, the input data can include M channels. Using the same notation as described above, each of the M individual filters can have... The size of the input data. Then, a separate convolution operation can be performed on each channel, using a separate channel, and the output includes a dimension corresponding to the number of channels in the input data. In the DWC setup, the number of parameters in the convolution kernel can then be equivalent to the number of filters multiplied by the size of those filters, i.e. And the computational cost can be .

[0007] Typically, pointwise convolution (PWC) can be used after DWC to combine the DWC outputs in a new feature map, thereby reducing the output dimension of the DWC. PWC usually consists of 1x1 convolutions. Using the same notation as described above, PWC can use N convolutions of size... The filters, where M can correspond to the number of channels in DWC, and N can correspond to the number of filters that a general CNN would use. In PWC settings, the number of parameters in the convolution kernel can then be equated to the number of filters multiplied by the filter size, i.e. And the computational cost can be .

[0008] Since PWC can combine feature maps from DWC and generate new feature maps based on the number of convolutional kernels, the outputs of the combined DWC and PWC can correspond to the outputs of traditional convolutional layers in a general CNN with the same parameters. Therefore, the number of parameters in the combined DWC and PWC layers can be... And the computational cost can be The ratio of the number of parameters to the ratio of computational complexity can therefore be: .

[0009] Due to its structure, DSCNN is generally parameter-efficient and computationally inexpensive, and is an alternative to the kernels of larger CNNs. Examples of DSCNNs include: MobileNets (Howard et al. (2017) MobileNets: EfficientConvolutional Neural Networks for Mobile Vision Applications, Sandler et al. (2018) MobileNetV2: Inverted Residuals and Linear Bottlenecks, and Howard et al. (2019) Searching for MobileNetV3), EfficientNet (Tan and Le (2019) EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks and (2021) EfficientNetV2: Smaller Models and Faster Training), and ConvNeXt (Liu et al. (2022) A ConvNet for the 2020s and Woo et al. (2023) ConvNeXt V2: Co-Designing and Scaling ConvNets with Masked Autoencoders). DSCNNs typically perform well in the aforementioned application tasks, such as computer vision, and due to their architecture, they generally do not require as many computational resources as general-purpose CNNs. Therefore, a trade-off between task performance and resource requirements is usually desirable. However, models developed using DSCNNs often become increasingly complex, thereby increasing the computational cost of DSCNNs, leading to high energy consumption and environmental burden. This is particularly problematic when, as described above, complex models of DSCNNs are used on resource-constrained and / or mobile devices, such as edge devices, to perform application tasks. For example, the device or apparatus may have only a limited computational budget to perform the application task, there may be limitations on the number of computations that can be performed per unit time, limited memory available for temporarily storing data during the application task, and limited and / or constrained computational performance and / or energy resources, such as battery capacity, etc. This budget may also be dynamic, i.e., it may change over time, for example, due to other processes executing sequentially with interference. In some examples, it may not be possible to know in advance how much computational budget is available at a particular point in time. This can be problematic because evaluating complex DSCNNs can be computationally expensive.Therefore, minimizing the memory size and / or computational cost of convolutional neural networks while maintaining their desired performance in application tasks may be a valuable objective.

[0010] Quantization methods are known for compressing CNNs, thereby reducing the memory size and energy cost of models using CNNs while maintaining the desired accuracy. Computation in CNNs is typically reduced from floating-point format to integer operations. However, there are limitations to the compression capabilities of quantization methods while maintaining model accuracy. Quantization to bit widths below 8 bits often impacts the accuracy of typical CNN models, thus reducing task performance. To account for this performance degradation, lower bit-width quantization methods typically require extensive training methods involving multiple stages in which the model's bit width is progressively reduced. However, these training methods are extensive and cost-intensive in terms of time and / or computation, often based on knowledge distillation from a larger teacher model, and / or require specific, custom model architectures. Furthermore, in such inference stages, quantized DSCNN models can be provided on resource-constrained devices without local hardware support. For example, this device could be an edge device. This device could include hardware, such as edge hardware for general purposes, which may not support the user-defined operations typical of the inference stage. For these reasons, training and inference methods in general CNN and / or DWSCNN applications are often not suitable for general-purpose, resource-constrained devices.

[0011] The drawback of existing methods for quantizing DWSCNN is that they are either based on widely used training methods (which are unsuitable for many DSCNN applications) or cannot achieve lower bit widths of weights below 8 bits while maintaining the desired accuracy. Summary of the Invention

[0012] It would be advantageous to improve the quantization methods used to compress DSCNNs, while maintaining the accuracy suitable for deployment on resource-constrained devices in an appropriate manner.

[0013] According to a first aspect of the present invention, a computer-implemented quantization method for a deep separable convolutional neural network is provided, wherein the deep separable convolutional neural network includes a plurality of network layers, each of the plurality of network layers includes a plurality of network weights, wherein the plurality of network layers includes: - Suitable for performing one or more pointwise convolutional layers, each of which includes a plurality of pointwise convolutional layer weights, and - Suitable for performing one or more deep convolutional layers, each of said one or more deep convolutional layers including a plurality of deep convolutional layer weights, wherein said quantization method includes quantizing said plurality of network weights, including: - Quantize the plurality of pointwise convolutional layer weights into a plurality of quantized pointwise convolutional layer weights within a first discrete range, and - Quantize the plurality of deep convolutional layer weights into a plurality of quantized deep convolutional layer weights within a second discrete range, wherein the first discrete range has a cardinality (Kardinalität) that is strictly lower than that of the second discrete range.

[0014] According to another aspect of the invention, a volatile or non-volatile computer-readable medium includes data representing a computer program, wherein the computer program includes instructions that cause a processor system to perform one of the methods described herein.

[0015] According to another aspect of the present invention, a processor system is provided, wherein the processor system includes a memory and one or more processors, wherein the memory includes instructions that cause the one or more processors to perform one of the methods described herein.

[0016] The above measures include: quantization of pointwise convolutional layer weights (PWC) and depthwise convolutional layer weights (DWC), wherein these quantizations are different from each other. These quantizations are different from each other in the following sense: PWC layer weights are quantized to weights within a first discrete range, and DWC layer weights are quantized to weights within a second discrete range, wherein the first discrete range has a cardinality that is strictly lower than the second discrete range.

[0017] The inventors have determined that DSCNN, due to its structure and architectural design, is suitable for applying different quantizations to different types of convolutional layers within DSCNN. Furthermore, the different operations performed by different types of convolutional layers, namely DWC and PWC, contribute differently to the total cost in terms of time and computation, resulting in an uneven distribution of computational cost. For example, in the MobileNetV2 model, DWC can account for 1.9% of the model parameters and 33.6% of the model energy cost, while PWC can account for 61.2% of the model parameters and 66.1% of the model energy cost. This makes it worthwhile to quantize the weights in PWC and DWC layers to different bit widths. A first discrete range with a lower cardinality corresponds to a lower bit width than a second discrete range. If the weights of the PWC layer, corresponding to the higher-cost PWC, are more strongly quantized—that is, quantized to a lower bit width than the weights of the DWC layer, corresponding to DWC—the weights of the DWC layer can remain at a higher bit width, thus allowing the corresponding part of the DSCNN to still operate with the desired accuracy.

[0018] The above measures achieve compression of DSCNN, thereby reducing its memory size and energy cost. This is achieved through a quantization method that performs stronger quantization on the typically more expensive PWC layers than on the DWC layers, which are crucial for maintaining their accuracy and performance. This quantization method is particularly suitable for and can be deployed on resource-constrained devices because it improves and optimizes the use of computational resources without requiring extensive training methods. The lower memory requirements and applicability to general-purpose hardware on such devices also ensure the suitability of the quantization method for resource-constrained equipment.

[0019] Optionally, the first discrete range includes a base of 2 or 3. Such a first discrete range can correspond to the weights of a PWC layer that include binary or ternary weights. Optionally, the second discrete range includes a base in the range of 3 to 256, for example, 16. Such a second discrete range can correspond to the weights of a DWC layer that include weights of 2 to 8 bits, for example, 4 bits. Optionally, the first discrete range includes a first base, and the second discrete range includes a second base, wherein the first base multiplied by fifty is less than the second base. By maintaining, for example, 8-bit DWC and reducing PWC to ternary weights, all computations can be kept in int8 format. Int8 addition generally enjoys broad support across hardware platforms and eliminates expensive multiplication, unlike int4 or int2 operations that existing methods may be based on. By using an 8-bit bit width, the accuracy of task performance is generally not compromised, which is important for DWC operations. For models based on 8-bit integer operations, the Pareto boundary of energy consumption and memory size can be improved.

[0020] Optionally, the plurality of network layers further includes one or more activation layers, each of which includes an activation function, each of which is adapted to perform activation on the input of the corresponding activation layer, wherein the activation includes applying the corresponding activation function to the input, and wherein the method further includes adapting the one or more activation functions of the one or more activation layers to one or more adapted activation functions, wherein each of the one or more adapted activation functions is different from the corresponding activation function in the one or more activation functions from the one or more activation layers. Optionally, the one or more activation functions include non-parametric activation functions, such as ReLU activation functions, ReLU activation functions with upper limits, Hardswish activation functions, Vorzeichen activation functions, or LeakyReLU activation functions. Furthermore, the one or more adapted activation functions may include parametric activation functions, such as PReLU activation functions. Replacing ReLU-like and / or non-parametric activations with parametric activation functions, such as PReLU, can be a parametrically efficient way to improve model performance. Parameters used for parametric PReLU activation functions may improve model performance more strongly than using these parameters for PWC layer weights or DWC layer weights.

[0021] In another aspect of the invention, a computer-implemented method for training a deep separable convolutional neural network can be provided, wherein the deep separable convolutional neural network includes a plurality of network layers, each of the plurality of network layers includes a plurality of network weights, wherein the plurality of network layers includes: - Suitable for performing one or more pointwise convolutional layers, each of which includes a plurality of pointwise convolutional layer weights; - Suitable for performing one or more depthwise convolutional layers, each of which includes a plurality of depthwise convolutional layer weights, and - One or more activation layers, wherein each of the one or more activation layers is adapted to perform activation on the input of the corresponding activation layer. - The method described therein includes: - Iteratively train the multiple network weights on a training dataset, wherein the training dataset includes multiple input samples, and training for the input samples includes: - During forward propagation (Vorwärtsdurchlauf), - Simulate the quantization steps of a network layer using the quantization method according to the invention, wherein the plurality of network weights are quantized into a plurality of quantized network weights. - Quantizes the activations of the input sample (Eingabeprobe) performed by one or more activation layers into quantized activations within a discrete range. - By rescaling the quantized network weights and rescaling the quantized activations, the quantized network weights and activations are dequantized, resulting in dequantized network weights and activations. - During backpropagation, gradient estimation is performed using the dequantized network weights and the dequantized activations. - Based on the gradient estimation, adapt the dequantized network weights and the dequantized activations. - Provides trained deep separable convolutional neural networks for inference (Inferenz).

[0022] The above measures enable a quantization-aware training method based on the quantization method according to the invention. This quantization-aware training allows for the achievement of a model with desired accuracy while maintaining a low overall computational cost. When using the quantization method according to the invention, network weights can be quantized and / or dequantized accordingly. The network weights can be updated using gradient descent, which can be standard gradient descent.

[0023] Optionally, the training method further includes quantizing the plurality of network weights using the quantization method according to the invention before providing the trained deep separable convolutional neural network for inference. This results in a plurality of quantized network weights, which are maintained during inference. Converting the network weights to fixed, quantized weights after the training portion of the training method enables efficient inference.

[0024] In another aspect of the invention, a computer-implemented method is provided for using a deep separable convolutional neural network on a computing-restricted device, mobile device, and / or autonomous device, such as an autonomous robot and / or autonomous vehicle, to perform one or more of computer vision, object recognition, image processing, image recognition, image classification, medical imaging, and / or image generation, wherein the deep separable convolutional neural network is trained using the training method according to the invention.

[0025] The above measures enable an inference method based on the training method according to the invention. Since the training method according to the invention is based on a quantization method, such an inference method ensures that the trained DSCNN is optimized not only for the application task but also for its applicability on devices on which the DSCNN is deployed.

[0026] It will be apparent to those skilled in the art that two or more of the above embodiments, implementations, and / or optional aspects of the invention can be combined in any manner deemed useful.

[0027] Based on this specification, modifications and variations corresponding to the modifications and variations of one of these units can be performed by those skilled in the art, in any device, system, network, computer-implemented method, and / or computer-readable medium. Attached Figure Description

[0028] Further details, aspects, and implementations are described by way of example only with reference to the accompanying drawings. The elements in the drawings are shown for simplicity and clarity only and are not drawn to scale. In the drawings, elements corresponding to those already described may have the same reference numerals. In the drawings: Figure 1a schematically illustrates an implementation of a device including a depthwise separable convolutional neural network; Figure 1b schematically illustrates an example of a system in which a deep separable convolutional neural network is applied within an autonomous vehicle; Figure 2a schematically illustrates an implementation of a depth-separable convolutional neural network; Figure 2b schematically illustrates an implementation of the quantized depthwise separable convolutional neural network; Figure 3 schematically illustrates a portion of the training steps in an implementation of a method for training a depthwise separable convolutional neural network; Figure 4 schematically illustrates an implementation of the quantization method for depthwise separable convolutional neural networks; Figure 5 schematically illustrates an implementation of a method for training a depthwise separable convolutional neural network; Figure 6 schematically illustrates an implementation of a method for using a depthwise separable convolutional neural network; Figure 7a schematically illustrates a computer-readable medium, the writable portion of which includes a computer program according to one embodiment; and Figure 7b schematically illustrates a processor system according to one embodiment.

[0029] List of reference numerals The following list of reference numerals and abbreviations is intended to facilitate understanding of the drawings and should not be construed as limiting the claims. 10 Input Images 20 Quantification Steps 21, 21' Pointwise Convolutional Layer Weights 21.1, 21.1' Quantized pointwise convolutional layer weights 21.2, 21.2' Dequantized pointwise convolutional layer weights 22. Weights of deep convolutional layers 22.1 Quantized weights of depthwise convolutional layers 22.2 Dequantized weights of deep convolutional layers 23,23' Activation function 23.1,23.1' Adapted activation function 24,24' Activation 24.1,24.1' Quantified activation Activation of dequantization at 24.2, 24.2' 30. Inverse Quantization Steps 31,31' Input Sample 40 Gradient Estimation 100 System 110 equipment 111 Processor System 112 Memory 113 Communication Interface 114 Autonomous Vehicles 115 Image Sensor 116 pedestrians 200, 201, 202 Depthwise Separable Convolutional Neural Networks 210,210',211,211' Pointwise convolutional layers 220, 221 depth convolutional layers Activation layer 230,230',231,231' Network layers 240 and 241 Part of the 300 training steps 301 Forward Propagation 302 Backpropagation 310 Training Dataset 400 Quantization methods for depthwise separable convolutional neural networks 410 Quantizing Network Weights 411 Quantizing the weights of pointwise convolutional layers 412 Quantizing the weights of deep convolutional layers 420 Adapted Activation Function 500 Methods for Training Deeply Separable Convolutional Neural Networks 501 Training Network Weights 502 provides a trained network for inference. 503 Quantizing Network Weights 504 Maintain quantified network weights 510 Forward Propagation 511 Simulation Quantization Steps 512 Quantitative Activation 520 pairs of quantized network weights and activation dequantization 530 Backward Propagation 531 Perform gradient estimation 540 Adapted inverse quantization network weights and activations 600 Methods for Using Deeply Separable Convolutional Neural Networks 601 Using a trained deep separable convolutional neural network 1000 Optical Storage Device 1001 memory card The data stored in 1020 and 1021 1110 Subsystem or component 1120 Subprocessing System 1122 Memory 1124 Application-Specific Integrated Circuit 1126 Communication Interface 1130 connection 1140 processor system. Detailed Implementation

[0030] While the subject matter disclosed herein may be embodied in a variety of different forms, one or more specific embodiments are illustrated in the accompanying drawings and described in detail herein, and it should be understood that this disclosure is exemplary with respect to the principles of the subject matter disclosed herein and is not intended to limit it to the specific embodiments shown and described.

[0031] To better understand, the elements of the implementation method in operation will be described below. However, it is obvious that the arrangement of the corresponding elements enables them to perform the described functions.

[0032] Furthermore, the subject matter disclosed herein is not limited to these embodiments, but also includes any other combination of features described herein or listed in the various dependent claims.

[0033] Figure 1a schematically illustrates an implementation of system 100, which includes device 110, comprising a Deeply Separable Convolutional Neural Network (DSCNN) 200. DSCNN 200 can use image 10 as input. DSCNN 200 can be a trained neural network. DSCNN 200 can be adapted to perform one or more application tasks. The one or more application tasks can perform one or more of computer vision, object recognition, image processing, image recognition, image classification, medical imaging, and image generation. DSCNN 200 can contain one or more models for computer vision, object recognition, image processing, image recognition, image classification, medical imaging, image generation, and / or other image analysis applications, and / or be included in one or more models among these models.

[0034] System 110 may include a subprocessing system 111, a memory 112, and a communication interface 113. System 110 can access input data, such as sensor data obtained from one or more sensors, such as radar data, lidar data, ultrasonic data, or image sensor data. For example, input data can be retrieved from data memory 112 via communication interface 113. Data memory 112 may be local memory of system 110, such as a local hard drive or local storage. However, memory 112 may also be non-local memory, such as network storage, like cloud storage. In other examples, system 110 may access input data directly from one or more sensors, for example, without needing to store the input data at least temporarily on data memory 112. In such examples, communication interface 113 may be a sensor interface connected to one or more sensors.

[0035] The subprocessing system 130 can be adapted to perform the aforementioned application task when using DSCNN 200. DSCNN 200 can include one or more input layers, multiple intermediate layers, and one or more output layers to generate the output of DSCNN 200.

[0036] When performing application tasks using DSCNNs, whose networks can be quite complex, systems like System 100 typically have only a limited computational budget to execute those tasks. For example, Device 110 could be a resource-constrained device with limitations in memory and / or computational resources, or a mobile and / or autonomous device, such as an autonomous robot and / or autonomous vehicle. In some examples, it may not be possible to know in advance how much computational budget will be available at a particular point in time.

[0037] Typically, system 110 can communicate with external memory, input devices, output devices, and / or one or more sensors, for example, via a computer network. The computer network can be the Internet, an intranet, a LAN, a WLAN, etc. System 110 may include a communication interface 113, which is configured to communicate internally or externally as needed. For example, communication interface 113 can be, for example, a wired interface, such as an Ethernet interface, an optical interface, etc., or a wireless interface, such as a radio interface, such as a Wi-Fi, 4G, or 5G radio interface.

[0038] Typically, system 110 can be implemented in or as a processor system, for example, by using one or more processor circuits, such as a microprocessor. The processor system may include subprocessing systems that can be implemented wholly or partially as computer instructions stored on system 110, for example in electronic memory of system 110, and can be executed by the microprocessor of system 110. In a hybrid implementation, the subprocessing system may be implemented partly in hardware, for example as a coprocessor, such as a coprocessor for machine learning, and partly in software stored and executed on system 110. Parameters and / or input data for the machine learning model may be stored locally on system 110 or stored in cloud storage. Typically, the memory may be distributed across multiple distributed sub-memories. The memory may be wholly or partially electronic memory, magnetic memory, etc. For example, the memory may have volatile and non-volatile portions. A portion of the memory may be write-protected. System 110 may have a user interface, which may include known elements such as one or more buttons, a keyboard, a display, a touchscreen, etc. User interfaces can be configured to enable user interaction to configure the system, apply trained machine learning models to input data, etc.

[0039] Typically, system 110 can be implemented in a single device. This system typically includes a microprocessor that executes corresponding software stored within the system; for example, this software can be downloaded and / or stored in a suitable memory, such as volatile memory like RAM or non-volatile memory like flash memory. Alternatively, the system can be implemented wholly or partially as programmable logic, such as a field-programmable gate array (FPGA). The system can also be implemented wholly or partially as a so-called application-specific integrated circuit (ASIC), such as an integrated circuit (IC) adapted for its specific purpose. For example, these circuits can be implemented in CMOS form, for example, by using hardware description languages ​​such as Verilog, VHDL, etc. In particular, system 110 may include circuitry for evaluating machine learning models, such as neural networks.

[0040] Figure 1b An example of the application of device 110, which includes DSCNN 200, in an autonomous vehicle 114 is illustrated schematically. Device 110 can be adapted to perform object recognition in autonomous vehicle 114 using DSCNN 200. Here, the application task of object recognition may be to generate labels, such as bounding boxes. Such labels can label the location and size of objects in planar images, such as image input obtained from an image sensor, and classify their contents. In this implementation, DSCNN 200 may have been trained to recognize traffic participants, such as pedestrians 116, in camera images obtained from an image sensor, such as camera 115. After training, the trained DSCNN 200 can be used by device 100 in vehicle 114 for object recognition. Camera images detected by camera 115 can be temporarily stored in data memory 112 connected to communication interface 113. Processing subsystem 111 can then perform classification using the stored camera images as input, thereby enabling vehicle 114 to identify traffic participants 116 near vehicle 114. In some examples, device 110 may be adapted to control one or more actuators in vehicle 114 or another computer-controlled machine, for example, via an actuator interface that may be part of or outside device 110. By controlling one or more actuators, vehicle 114 can be controlled to react appropriately to these other traffic participants 116 when needed, such as by steering, braking, or triggering warnings. While the foregoing specifically pertains to automobiles, it is evident that device 110 can control any other computer-controlled machine via the actuator interface. Figure 2a An implementation of a Deep Separable Convolutional Neural Network 200 is illustrated schematically. The Deep Separable Convolutional Neural Network (DSCNN) 200 is a convolutional neural network (CNN) equipped with a typical structure. DSCNNs typically form a parameter-efficient and computationally inexpensive alternative to conventional CNNs using dense convolutions with large kernel sizes. The DSCNN 200 may include multiple network layers 210, 210', 220, 230, 230', 240. Each of the multiple network layers 210, 210', 220, 230, 230', 240 may include multiple network weights 21, 21', 22.

[0041] The multiple network layers 210, 210', 220, 230, 230', 240 may include one or more pointwise convolutional layers 210, 210'. The one or more pointwise convolutional layers 210, 210' may be adapted to perform pointwise convolution (PWC). For example, the one or more pointwise convolutional layers 210, 210' may be adapted to perform pointwise 1x1 convolution. Each of the one or more pointwise convolutional layers 210, 210' may include multiple pointwise convolutional layer weights 21, 21'.

[0042] The plurality of network layers 210, 210', 220, 230, 230', 240 may further include one or more depthwise convolutional layers 220. The one or more pointwise convolutional layers 220 may be adapted to perform depthwise convolution (DWC). For example, the one or more depthwise convolutional layers 220 may be adapted to perform depthwise 3x3 convolution. Each of the one or more depthwise convolutional layers 220 may each include multiple input channels. Each of the one or more depthwise convolutional layers 220 may be adapted to independently extract information from the corresponding multiple input channels using a 3x3 kernel, such as a convolutional kernel. Each of the one or more depthwise convolutional layers 220 may include multiple depthwise convolutional layer weights 22.

[0043] In one implementation, the neural network 200 may include at least two pointwise convolutional layers 210 and 210'. A first pointwise convolutional layer 210 from the at least two pointwise convolutional layers 210 and 210' may be adapted to project the input to a higher-dimensional latent space. This projection may function as an upward projection to a higher-dimensional latent space. The input may be the input of a depthwise separable convolutional neural network 200. A second pointwise convolutional layer 210' from the at least two pointwise convolutional layers 210 and 210' may be adapted to project a second input to a lower-dimensional latent space. The second input may have the same dimension as the higher-dimensional latent space. For example, the second input may contain the output of a depthwise convolutional layer 220 from the one or more depthwise convolutional layers 220. In this way, the projection may function as a downward projection of the latent dimension to a lower-dimensional latent space, for example, a space with a lower initial dimension. The second pointwise convolutional layer 210' from the at least two pointwise convolutional layers 210, 210' can be used to perform pointwise convolution to generate a linear combination of the outputs of the depthwise convolutional layer 220.

[0044] In one implementation, the plurality of network layers 210, 210', 220, 230, 230', 240 may further include one or more activation layers 230, 230'. Each of the one or more activation layers 230 and 230' may include activation functions 23, 23'. The one or more activation functions 23 and 23' may include non-parametric activation functions. Non-parametric activation functions may include one or more ReLU-like activation functions, such as ReLU activation functions, ReLU activation functions with upper limits, hard-slip activation functions, sign activation functions, or leaky ReLU activation functions. An example of a ReLU activation function with upper limits is ReLU6. Each of the one or more activation layers 230, 230' may be adapted to perform activation on the input of the corresponding activation layer 230, 230'. The activation may include applying the corresponding activation function 23, 23' to the input. The inputs may include the outputs of convolutional layers 210, 210', 220, such as the outputs of the DWC layers from the one or more DWC layers 220. The outputs of activation layers 230, 230', typically referred to as activations 24, 24', may include the outputs of activation functions 23, 23'. The outputs of activation layers 230, 230' may include inputs for subsequent convolutional layers 210, 210', 220, such as the PWC layer 210'. The one or more activation layers 230, 230' may also perform batch normalization. The plurality of network layers 210, 210', 220, 230, 230', 240 may also include one or more other network layers 240. The one or more other network layers 240 may include one or more linear network layers 240. For example, the one or more other network layers 240 may include one or more multilayer perceptrons (MLPs) containing multiple linear layers. The linear layers and / or MLPs may be positioned at the ends of the DSCNN and / or receive the outputs of convolutional layers 210, 210', 220 and / or activation layers 230, 230', such as flattened outputs. For example, one or more inputs to the one or more other network layers 240 may include the outputs of the second PWC layer 210' and / or activation layer 230', such as flattened outputs. The one or more other network layers 240 may be adapted to apply operations, such as batch normalization, to the one or more inputs. The one or more other network layers 240, such as MLPs, may be adapted to convert the encoded features generated by the convolutional layers 210, 210', 220 into outputs, wherein the outputs characterize the application tasks to which the DSCNN can be applied.For example, in the case of an application task of image classification, the output may include a category prediction. The one or more other network layers 240 may include dropout layers and / or pooling layers.

[0045] Figure 2bAn embodiment of a quantized depthlessly separable convolutional neural network 201 is schematically illustrated. The quantized DSCNN 201 may include multiple network layers 211, 211', 221, 231, 231', 241. Each of the multiple network layers 210, 210', 220, 230, 230', 240 may include multiple quantized network weights 21.1, 21.1', 22.1, 24.1, 24.1'. The multiple network layers 211, 211', 221, 231, 231', 241 may include one or more pointwise convolutional layers 211, 211'. The one or more pointwise convolutional layers 211, 211' may correspond to one or more pointwise convolutional layers 210, 210' following the quantization step 20. Each of the one or more pointwise convolutional layers 211, 211' may include a plurality of quantized pointwise convolutional layer weights 21.1, 21.1'. The plurality of network layers 211, 211', 221, 231, 231', 241 may also include one or more depthwise convolutional layers 221. The one or more pointwise convolutional layers 221 may correspond to the one or more pointwise convolutional layers 220 after quantization step 20. Each of the one or more depthwise convolutional layers 221 may include a plurality of quantized depthwise convolutional layer weights 22.1. In one embodiment, the plurality of network layers 211, 211', 221, 231, 231', 241 may also include one or more activation layers 231, 231'. The one or more activation layers 231, 231' may correspond to the one or more activation layers 230, 230' after quantization step 20. Each of the one or more activation layers 231, 231' may include one or more quantized activations 24.1, 24.1' and / or adapted activation functions 23.1, 23.1'. Quantizing activations 24, 24' may include quantizing one or more outputs of activation functions 23, 23' of the corresponding activation layers 230, 230'. Activating quantized 24, 24' may result in quantized activations 24.1, 24.1'. Each adapted activation function 23.1, 23.1' may differ from the corresponding activation function 23, 23' in the one or more activation functions 23, 23' of the one or more activation layers 230, 230'. The one or more adapted activation functions 23.1, 23.1' may include parameterized activation functions. Parameterized activation functions may include parameterized ReLU activation functions, such as PReLU activation functions. The plurality of network layers 211, 211', 221, 231, 231', 241 may also include one or more other network layers 241.The one or more other network layers 241 may correspond to one or more of the one or more other network layers 240 after quantization step 20. In quantization step 20, the plurality of network weights 21, 21', 22 may be quantized. Quantization of the plurality of network weights 21, 21', 22 may be performed during quantization step 20. Quantization of the plurality of network weights 21, 21', 22 may include quantization of the plurality of pointwise convolutional layer weights 21, 21'. The pointwise convolutional layer weights 21, 21' may be quantized into a plurality of quantized pointwise convolutional layer weights 21.1, 21.1'. The quantized pointwise convolutional layer weights 21.1, 21.1' may lie within a first discrete range. The first discrete range may have a first cardinality. The first discrete range and / or the first cardinality may correspond to a first bit width.

[0046] Quantization of the plurality of network weights 21, 21', 22 may further include quantization of the plurality of deep convolutional layer weights 22. The deep convolutional layer weights 22 may be quantized into a plurality of quantized deep convolutional layer weights 22.1. The quantized deep convolutional layer weights 22.1 may lie within a second discrete range. The second discrete range may have a second cardinality. The second discrete range and / or the second cardinality may correspond to a second bit width. The first cardinality corresponding to the first discrete range may be strictly lower than the second cardinality corresponding to the second discrete range.

[0047] Quantization step 20 may further include quantization of activations 24, 24'. Activations 24, 24' may be quantized into quantized activations 24.1, 24.1'. The quantized activations 24.1, 24.1' may lie within a third discrete range. The third discrete range may have a third cardinality. The third discrete range and / or the third cardinality may correspond to a third bit width. The third discrete range, the third cardinality, and / or the third bit width may correspond to and / or be equivalent to the second discrete range, the second cardinality, and / or the second bit width.

[0048] In one embodiment, the first discrete range may include a first cardinality of 2 or 3. Such a first discrete range may correspond to quantized PWC layer weights 21.1, 21.1' including binary or ternary weights. The second discrete range may include a second cardinality of 3 to 256, such as 16. Such a second discrete range may correspond to quantized DWC layer weights 22.1 including 2 to 8 bits, such as 4 bits. In one embodiment, the first discrete range may include a first cardinality, and the second discrete range may include a second cardinality such that the first cardinality multiplied by fifty is less than the second cardinality. In one embodiment, the third discrete range may include a third cardinality of 3 to 256, such as 256. Such a third discrete range may correspond to quantized activations 24.1, 24.1' including 2 to 8 bits, such as 8 bits of activation.

[0049] By maintaining, for example, an 8-bit DWC and reducing the PWC to ternary weights, all computations can be kept in int8 format. The activations can also be quantized, for example, to 8-bit operations, by quantizing activations 24, 24' to obtain quantized activations 24.1, 24.1'. In this case, all computations in DSCNN can be reduced to int8 additions without multiplication. This elimination of expensive multiplications may not be achievable using int4 or int2 operations employed by existing quantization methods. Furthermore, int8 additions are generally widely supported across hardware platforms in modern computer architectures. Since much of the cost of DSCNN, such as energy cost and / or parameter size, is provided by the PWC layers 210, 210', the lower-cost DWC can maintain a higher bit width, such as 8 bits. This also allows DSCNN to additionally regain accuracy by recovering any expressive power that might have been lost by limiting the representational power of the PWC through the use of lower-bit-width weights, such as ternary weights.

[0050] In one implementation, one or more of network layers 210, 210', 220, 230, 230', and 240 may include one or more input channels. One or more of network layers 210, 210', 220, 230, 230', and 240 may also include one or more output channels. In one implementation, quantizing the plurality of network weights 21, 21', and 22 may include determining a scaling factor for each of the one or more output channels. Quantizing the plurality of network weights 21, 21', and 22 for each of the one or more output channels may further include scaling the network weights 21, 21', and 22 using the determined scaling factor, thereby quantizing the network weights 21, 21', and 22 from the plurality of network weights 21, 21', and 22. Quantizing the network weights 21, 21', and 22 from the plurality of network weights 21, 21', and 22 may further include rounding the scaled network weights to a rounding value within a discrete range. The discrete range can be either a first discrete range or a second discrete range. If the network weights 21, 21', and 22 are PWC layer weights 21 and 21', then the discrete range can be the first discrete range. If the network weights 21, 21', and 22 are DWC layer weights 22, then the discrete range can be the second discrete range.

[0051] In one implementation, a scaling factor can be determined for an output channel in one or more output channels of the corresponding network layers 210, 210', 220, 230, 230', 240. In one implementation, the scaling factor for an output channel in one or more output channels of the corresponding network layers 210, 210', 220, 230, 230', 240 can be determined by the average absolute value of the network weights 21, 21', 22 of the output channel on one or more input channels of the corresponding network layers 210, 210', 220, 230, 230', 240. This determination by the average absolute value of the network weights 21, 21', 22 of the output channel on one or more input channels of the corresponding network layers 210, 210', 220, 230, 230', 240 can be referred to as average absolute value quantization. In one implementation, the determination of the scaling factor for the output channel in one or more output channels of the corresponding network layers 210, 210', 220, 230, 230', 240 using the maximum absolute value of the network weights 21, 21', 22 of the output channel can be referred to as AbsMax quantization. In one implementation, the scaling factor for the output channel in one or more output channels of the corresponding network layers 210, 210', 220, 230, 230', 240 can be determined using the minimum absolute value of the network weights 21, 21', 22 of the output channel. This determination using the minimum absolute value of the network weights 21, 21', 22 of the output channel can be referred to as AbsMin quantization. In one implementation, the scaling factor for the output channel in one or more output channels of the corresponding network layers 210, 210', 220, 230, 230', 240 can be determined using uniform non-negative real values. The unified non-negative real value can be independent of the network weights 21, 21', 22 of the output channel.

[0052] The following describes an example for quantization step 20. Here, PWC layer weights 21, 21' can be quantized into ternary weights 21.1, 21.1'. PWC layer weights 21, 21' can be quantized into ternary weights 21.1, 21.1' using channel-wise average absolute quantization. DWC layer weight 22 can be quantized into 8-bit integer weights 22.1. DWC layer weight 22 can be quantized into 8-bit integer weights 22.1 using channel-wise average absolute quantization. Such quantization step 20 enables accurate computation performed by DWC layer 221 between efficient ternary projections performed by PWC layers 211, 211' using a higher bit width.

[0053] set up This is the weight matrix of PWC layer 210, 210', where It can represent the output channel dimension. Indicates the input channel dimension, and This represents the kernel size. For a 1x1 PWC, the kernel size K=1 can be omitted, and is... The PWC layer weights 21, 21' can be quantized into a discrete range. The ternary weights in the formula are 21.1, 21.1', which corresponds to a bit width of 1.58 bits. First, the average absolute value of each output channel can be calculated as a scaling factor: .

[0054] Using scaling factor In this case, a quantized PWC layer weight matrix can be generated through rounding and clamping. This includes the quantized PwC layer weights 21.1 and 21.1': .

[0055] This can be introduced ε To avoid division by zero. For example In addition, it can also be done by using functions. To perform rounding and clamping, where round( x () can be rounding x to the nearest integer.

[0056] set up This is the weight matrix of layer 220 in the DWC layer, where It can represent the output channel dimension. It can represent the input channel dimension, and K K can represent the kernel size. Typically, K can be an integer. For example, for a 3×3 DWC core, the core size... The DWC layer weights 22 can be quantized as 8-bit precision weights 22.1 in the discrete range {−128,…,127}. First, the maximum absolute value of each output channel can be calculated as a scaling factor: .

[0057] When using the scaling factor β, the quantized DWC layer weight matrix, consisting of the quantized DWC layer weights 22.1, can be generated by rounding and clamping, similar to the quantized PWC layer weights 21.1, 21.1'. : .

[0058] For example, each additional DSCNN layer 240, such as an MLP, or each other network layer 240, which may include the one or more linear network layers, may be quantized in a manner similar to the quantization scheme used for the DWC layer weights 22. For example, if the one or more DWC layers 220 can be quantized into one or more quantized DWC layers 221 including quantized DWC layer weights 22.1, wherein the quantized DWC layer weights 22.1 can be quantized to 8-bit precision, then the one or more other network layers 240 including one or more linear layers and / or MLPs may also be quantized to 8-bit precision in quantization step 20.

[0059] One or more activation layers 230, 230' in DSCNN 200 can also be quantized. This can result in additional computational cost reduction during the inference phase of DSCNN 200. Tensor-level quantization can be performed on the activations in the one or more activation layers 230, 230'. An AbsMax quantization scheme similar to that used for the weights 22 in the DWC layer can be selected. Let... This is the input used for the activation layer. Here, B can represent the batch size. and The height and width of the input X can be represented. The corresponding activation layers 230, 230' can perform activation on the input X. This can be done by applying activation functions 23, 23' to the input X. Activation functions 23, 23' can include, for example, ReLU6 activation functions, which are typically general ReLU activation functions that include a cut-off at a value of 6. The activation of activation layers 230, 230' can be quantized as 8-bit precision activation in the discrete range {-128, ..., 127}. First, the maximum absolute value of the activation for each batch element can be calculated as a scaling factor: .

[0060] Using scaling factor In this case, similar to the quantized PWC layer weights 21.1, 21.1', quantized activations can be generated through rounding and clamping. : .

[0061] Furthermore, the quantization scheme for activation layers 230 and 230' can include replacing the ReLU6 activation functions 23 and 23' with PRELU activation functions 23.1 and 23.1'. This replacement could be a parametrically efficient way to improve the performance of the final application task of the quantized DSCNN version 201. Alternative quantization schemes also exist. These schemes can include min-max quantization, where weight values ​​can be rescaled by a factor derived from the minimum and maximum weight values ​​of the output channels and / or batch elements. In a uniform quantization scheme, the entire range of weight values ​​can be divided into multiple equally sized intervals.

[0062] Figure 3 A portion of training step 300 in an embodiment of a method for training a depthlessly separable convolutional neural network 200 is illustrated schematically. The DSCNN 200 may comprise a plurality of network layers 210, 210', 220, 230, 230', 240. Each of the plurality of network layers 210, 210', 220, 230, 230', 240 may include a plurality of network weights 21, 21', 22. The plurality of network layers 210, 210', 220, 230, 230', 240 may include one or more pointwise convolutional layers 210, 210'. The one or more pointwise convolutional layers 210, 210' may be adapted to perform pointwise convolutions, wherein each of the one or more pointwise convolutional layers 210, 210' may include a plurality of pointwise convolutional layer weights 21, 21'. The plurality of network layers 210, 210', 220, 230, 230', 240 may further include one or more deep convolutional layers 220. The one or more deep convolutional layers 220 may be adapted to perform deep convolution. Each of the one or more deep convolutional layers 220 may include a plurality of deep convolutional layer weights 22. The plurality of network layers 210, 210', 220, 230, 230', 240 may further include one or more activation layers 230, 230'. The one or more activation layers 230, 230' may be adapted to perform activation on the input of the corresponding activation layer 230, 230'. One or more outputs of the activation functions 23, 23' may include one or more activations 24, 24'.

[0063] The training method for DSCNN 200 can be a quantization-aware training method. In the training method of DSCNN 200, the plurality of network weights 21, 21', 22 can be trained iteratively. The plurality of network weights 21, 21', 22 can be trained iteratively on a training dataset 310. The training dataset 310 may include a plurality of input samples 31, 31'. During the training step 300 of the training method, a forward propagation 301 can be performed for the input samples 31, 31'. During the forward propagation 301, a quantization step 20 can be simulated. The quantization step can be performed on one or more of the network layers 210, 210', 220, 230, 230', 240. According to one embodiment, the quantization step 20 can be performed using quantization method 400. During quantization step 20, the plurality of network weights 21, 21', 22 can be quantized into a plurality of quantized network weights 21.1, 21.1', 22.1, 24.1, 24.1'. During quantization step 20, the activations 24, 24' of the input samples 31, 31' that can be performed by the one or more activation layers 230, 230' can also be quantized into quantized activations 24.1, 24.1' within a discrete range.

[0064] In the dequantization step 30, the plurality of quantized network weights 21.1, 21.1', 22.1 can be dequantized. According to one embodiment, the plurality of quantized network weights 21.1, 21.1', 22.1 can be dequantized in a manner similar to the quantization method. The quantized activations 24.1, 24.1' can also be dequantized. The dequantization of the plurality of quantized network weights and / or quantized activations can be performed by rescaling the plurality of quantized network weights 21.1, 21.1', 22.1, 24.1, 24.1' and rescaling the quantized activations, resulting in a plurality of dequantized network weights 21.2, 21.2', 22.2, 24.2, 24.2' and dequantized activations.

[0065] During training step 300, backpropagation 302 can also be performed. During backpropagation 302, gradient estimation 40 can be performed. Gradient estimation 40 can be performed with dequantized network weights 21.2, 21.2', 22.2, 24.2, 24.2' and / or dequantized activations. Based on gradient estimation 40, dequantized network weights 21.2, 21.2', 22.2, 24.2, 24.2' and / or dequantized activations can be adapted.

[0066] Figure 4 An embodiment of a quantization method 400 for a Deeply Separable Convolutional Neural Network (DSCNN) 200 is schematically illustrated. The DSCNN can use an image as input. The DSCNN can be included in and / or comprised of one or more models for computer vision, object recognition, image processing, image recognition, image classification, medical imaging, image generation, and / or other image analysis applications. The DSCNN 200 may include multiple network layers 210, 210', 220, 230, 230', 240. Each of the multiple network layers 210, 210', 220, 230, 230', 240 may include multiple network weights 21, 21', 22. The multiple network layers 210, 210', 220, 230, 230', 240 may include one or more pointwise convolutional layers (PWC) 210, 210'. The one or more PWC layers 210, 210' are suitable for performing pointwise convolutions. Each of the one or more PWC layers 210, 210' may include multiple PWC layer weights 21, 21'. The multiple network layers 210, 210', 220, 230, 230', 240 may include one or more depthwise convolutional (DWC) layers 220. The one or more DWC layers 220 are suitable for performing depthwise convolutions. Each of the one or more DWC layers 220 may include multiple DWC layer weights 22.

[0067] Quantization method 400 may include step 410 for quantizing the plurality of network weights 21, 21', 22. Step 410 for quantization may include a sub-step 411 for quantizing the plurality of PWC layer weights 21, 21' into a plurality of quantized PWC layer weights 21.1, 21.1' within a first discrete range. Step 410 for quantization may include a sub-step 412 for quantizing the plurality of DWC layer weights 22 into a plurality of quantized DWC layer weights 22.1 within a second discrete range. The first discrete range may have a cardinality significantly lower than the second discrete range.

[0068] In one embodiment, the plurality of network layers 210, 210', 220, 230, 230', 240 may further include one or more activation layers 230, 230'. Each of the one or more activation layers 230, 230' may include an activation function 23, 23'. Each of the one or more activation layers 230, 230' may be adapted to perform activation on the input of the corresponding activation layer 230, 230'. The activation may include applying the corresponding activation function 23, 23' to the input.

[0069] In one embodiment, the quantization method 400 may further include an optional step 420 for adapting one or more activation functions 23, 23' of the one or more activation layers 230, 230' to one or more adapted activation functions 23.1, 23.1'. Each of the one or more adapted activation functions 23.1, 23.1' may be different from the corresponding activation function 23, 23' in the one or more activation functions 23, 23' from the one or more activation layers 230, 230'.

[0070] Figure 5 An implementation of a method 500 for training a Deeply Separable Convolutional Neural Network (DSCNN) 200 is illustrated schematically. The DSCNN can use images as input. The DSCNN can be included in and / or comprised of one or more models for computer vision, object recognition, image processing, image recognition, image classification, medical imaging, image generation, and / or other image analysis applications. The DSCNN 200 may include multiple network layers 210, 210', 220, 230, 230', 240. Each of the multiple network layers 210, 210', 220, 230, 230', 240 may include multiple network weights 21, 21', 22. The multiple network layers 210, 210', 220, 230, 230', 240 may include one or more pointwise convolutional layers (PWC) 210, 210'. The one or more PWC layers 210, 210' are suitable for performing pointwise convolutions. Each of the one or more PWC layers 210, 210' may include multiple PWC layer weights 21, 21'. The multiple network layers 210, 210', 220, 230, 230', and 240 may include one or more depthwise convolutional (DWC) layers 220. The one or more DWC layers 220 are suitable for performing depthwise separable convolutions. Each of the one or more DWC layers 220 may include multiple DWC layer weights 22. The multiple network layers 210, 210', 220, 230, 230', and 240 may also include one or more activation layers 230 and 230'. Each of the one or more activation layers 230, 230' may include activation functions 23 and 23'. Each of the one or more activation layers 230, 230' can be applied to activate the input of the corresponding activation layer 230, 230'. The activation may include applying the corresponding activation function 23, 23' to the input.

[0071] The training method 500 of DSCNN 200 can be a quantization-aware training method 500. The training method 500 of DSCNN 200 may include a training phase 501, in which the plurality of network weights 21, 21', 22 are iteratively trained on a training dataset 310 during training step 300. The training dataset 310 may include a plurality of input samples 31, 31'.

[0072] Training phase 501 may include multiple training steps 300. During training step 300 of training 501 in training method 500, forward propagation 510 may be performed on input samples 31, 31'. During forward propagation 510, quantization step 20 may be simulated in simulation step 511 of forward propagation 510. The quantization step may be performed on one or more of the network layers 210, 210', 220, 230, 230', 240. Quantization step 20 may be performed according to one embodiment when using quantization method 400. During quantization step 20, the plurality of network weights 21, 21', 22 may be quantized into a plurality of quantized network weights 21.1, 21.1', 22.1, 24.1, 24.1'. Forward propagation 510 may also include an additional quantization step 512. During the additional quantization step 512 of the forward propagation, the activations of the input samples 31, 31' that can be performed by the one or more activation layers 230, 230' can also be quantized into quantized activations within a discrete range.

[0073] The training phase 501 may also include a dequantization step 520. The dequantization step 520 may be included in the forward propagation 510. During the dequantization step 520 of the training phase 501, the plurality of quantized network weights 21.1, 21.1', 22.1 may be dequantized. The plurality of quantized network weights 21.1, 21.1', 22.1 may be dequantized in a manner similar to the quantization method, according to one embodiment. Quantized activations may also be dequantized. Dequantization of the plurality of quantized network weights and / or quantized activations may be performed by rescaling the plurality of quantized network weights 21.1, 21.1', 22.1, 24.1, 24.1' and rescaling the quantized activations, resulting in a plurality of dequantized network weights 21.2, 21.2', 22.2, 24.2, 24.2' and dequantized activations. For example, when using the AbsMax quantization scheme for ternary PWC weighted quantization, by using each output channel With scaling factor Multiplying and dequantizing the output channel, and then convolving it with the input, allows for the computation of the layer's forward propagation. This ensures that the gradient is properly computed in gradient estimation step 40.

[0074] Training step 300 of training phase 501 may further include backpropagation 530. This backpropagation may include a gradient estimation step 531. During the gradient estimation step 531 of backpropagation 530, gradient estimation 40 may be performed. Gradient estimation 40 may be performed using dequantized network weights 21.2, 21.2', 22.2, 24.2, 24.2' and / or dequantized activations.

[0075] The training step 300 of the training phase 501 may further include an adaptation step 540. During the adaptation step 540 of the training phase 501, the dequantized network weights 21.2, 21.2', 22.2, 24.2, 24.2' and / or the dequantized activations may be adapted based on gradient estimation 40.

[0076] In one implementation, the training method 500 may further include an optional quantization step 502 provided prior to the trained DSCNN 200 for inference. During the optional quantization step 502, a plurality of network weights 21, 21', 22 are quantized. According to one implementation, the plurality of network weights 21, 21', 22 may be quantized using quantization method 400. The optional quantization step 502 may result in a plurality of quantized network weights 21.1, 21.1', 22.1.

[0077] The training method 500 may further include an inference phase 503. In the inference phase 503, the trained DSCNN 200 can be provided for inference. For example, at the inference time point, the input can be directly fed with ternary weights, such as those of a PwC layer, using an Absmax quantization scheme. Perform a convolution, which reduces the convolution to the sum of the input values, and then a scaling factor can be applied. Scale it.

[0078] In one implementation, the training method 500 may further include an optional step 504 in which a plurality of quantized network weights 21.1, 21.1', 22.1 generated by the optional quantization step 502 are kept constant during inference.

[0079] During training method 500, the plurality of network weights 21, 21', 22 may, for example, be maintained with 32-bit floating-point precision. For example, the plurality of network weights 21, 21', 22 may be updated using gradient descent, such as standard gradient descent. The plurality of network weights 21, 21', 22 may be quantized and / or dequantized during the ongoing runtime. The plurality of network weights 21, 21', 22 may be converted to fixed quantized network weights 21.1, 21.1', 22.1, 24.1, 24.1' after training phase 501 of the quantization-aware training method 500. Converting to fixed quantized network weights 21.1, 21.1', 22.1, 24.1, 24.1' after training phase 501 enables an efficient inference phase. To efficiently propagate gradients, for example, through a rounding function, a straight-through gradient estimator (STE) can be used in gradient estimation step 40. Using STE enables the use of standard optimization algorithms such as stochastic gradient descent (SGD). Training method 500 can be applied to a regular DSCNN model, such as MobileNetV2. For example, a learning rate scheduler can be applied using a cosine decay strategy, which can use a target learning rate. Data augmentation can be applied, such as random pruning and random horizontal flipping. These strategies enable improvements in training and validation accuracy. Convergence can be improved by setting weight decay during training, for example, at points in the middle of the training process, such as setting it to 0.

[0080] During training phase 500, another modification could be to replace non-parametric activation functions, such as ReLU6 activation, with parametric activation functions, such as PReLU. This can be a computationally inexpensive way to recover some of the model's expressive power lost during quantization and does not allow the model to alter the data flow. After training phase 501, other network layers, such as batch normalization layers and activation layers containing PReLU activation functions, can be fused. For example, such layers can be fused with the preceding quantized convolutional layers. This can reduce the computational cost during the inference phase.

[0081] The following example illustrates the distribution of the ternary PWC weights before and after training.

[0082] During initialization, the ternary weights in PWC may have a nearly uniform distribution between values ​​-1, 0, and 1, with no significant differences between these layers. This could be due to the initialization scheme, such as the He normal initialization scheme used in the MobileNetV2 model. For a given weight matrix... The weights can be initialized by sampling from a normal distribution: .

[0083] To quantize PWC weights into ternary weights, absolute mean quantization can be applied. A channel-by-channel scaling factor can be calculated according to one implementation method. This calculation may include approximating the expected value of the absolute value of the weights: .

[0084] Since each weight can be drawn from a normal distribution, the expected value can be calculated as follows: Here, the value of σ is independent of the selected output channel i. If before quantization... By rescaling the weights, the variance can be changed: .

[0085] Since the variance of the rescaled weight matrix is ​​now known, the distribution of the rounded and clamped ternary weights can be derived by observing the number of weights between the rounding thresholds of -0.5 and 0.5. The integral of the probability density function of the corresponding normal distribution is: .

[0086] This means that, during initialization, approximately 31.0% of the weights can be rounded to 0. Due to the symmetry of the normal distribution, the remaining weights can be rounded and clamped approximately equally to -1 and 1, assigning approximately 34.5% weight to each value respectively.

[0087] While the distribution of ternary weights in pointwise convolutions may be roughly uniform at initialization, it can shift towards a more uneven distribution after training, with an increase in the number of zero values ​​in specific layers. During training, the model appears to automatically learn to prune unimportant input connections by setting the corresponding weights to zero. Although the relative number of zero weights may vary, non-zero weight values ​​may be relatively uniformly distributed between -1 and 1. This balance between positive and negative weights may result in stable activations with lower variability in their own size. This behavior can be attributed to the direct use of batch normalization (which promotes input centering) and / or uniform initialization of weights after pointwise convolutions.

[0088] Below is some PyTorch-style pseudocode.

[0089] The following is pseudocode illustrating the quantization process for pointwise convolution and depthwise convolution weights.

[0090] def quantisiere_falt(gewicht, eps = 1e-5): """ Parameters: Weights (tensors): The weights of the convolutional modules. The expected weight shape is [c_out, c_in, k, k]. eps (float, optional): Prevents small epsilon values ​​from being divided by zero. """ ifgewicht.form[2:] == (1,1): # Pointwise convolution """ Pointwise convolution is quantized into ternary weights using channel-wise absolute mean quantization. """ #Calculate the channel-by-channel scaling factor scale = 1.0 / gewichte.abs().glaetten(start_dim=1). mittelwert(dim=-1, keepdim=True).clamp_(min=eps) #Re-form the scaling factor scale = scale.entquetschen(-1).entquetschen(-1) # [c_out, 1, 1, 1] # Quantitative Weights quant_gewicht = (gewicht * scale).runden().clamp_(-1, 1) return quant_gewicht, scale else: #Depthwise Convolution """ The depthwise convolution is quantized into 8-bit weights using channel-wise Absmax quantization. """ #Calculate the channel-by-channel scaling factor scale = 127,0 / gewichte.abs().glaetten(start_dim=1). max(dim=-1, keepdim=True).werte().clamp_(min=eps) #Re-form the scaling factor scale = scale. entquetschen (-1). entquetschen(-1) # [c_out, 1, 1, 1] # Quantization weights quant_gewicht = (gewicht * scale).runden() .clamp_(-128, 127) return quant_gewicht, scale The following is pseudocode for quantization activation. """ The activation product is quantized to 8 bits using tensor-by-tensor Absmax quantization. """ defquantisiere_aktivierung(x, eps = 1e-5): """ Parameters: x (Tensor): The input to be quantized. The desired shape is [stapel_groesse, c_in, höhe, breite]. eps (float, optional): Prevents small epsilon values ​​from being divided by zero. """ #Calculate the scaling factor for each tensor scale = 127,0 / x.abs().glaetten(start_dim=1).max(dim=-1, keepdim=True).werte().clamp_(min=eps) #Re-form the scaling factor scale = scale.entquetschen(-1).entquetschen(-1) # [stapel_groesse, 1, 1, 1] #Quantification Input quant_x = (x * scale).runden().clamp_(-128, 127) return quant_x, scale The following explains the pseudocode for the quantized convolution module. class quantisierteFalt(): def __init__(self, float_gewicht): """ Parameters: float_gewicht (Tensor): The underlying (initial) element applied during training. (Floating-point weights) """ self.float_gewicht = float_gewicht def vorwaerts(self, x): if self.training: #Training Forward Propagation # Quantize weights during ongoing operation quant_gewicht, scale_gewicht = quantisiere_falt(self.float_gewicht) #Quantitative Activation quant_x, scale_x = quantisiere_aktivierung(x) # Dequantize both before convolution. quant_gewicht / = scale_gewicht quant_x / = scale_x #Pass-through gradient estimator quant_gewicht = self.float_gewicht + (quant_gewicht - self.float_gewicht).detach() quant_x = x + (quant_x - x).detach() ausgabe = falten(quant_x, quant_gewicht) return ausgabe else: #Forward propagation of reasoning #Weights can be pre-quantized and set. quant_gewicht, scale_gewicht = quantisiere_falt(self.float_gewicht) #Quantitative Activation quant_x, scale_x = quantisiere_aktivierung(x) # Perform convolution with low bit width ausgabe = falten(quant_x, quant_gewicht) #Dequantization after convolution ausgabe / = scale_gewicht ausgabe / = scale_x return

[0091] Figure 6 An embodiment of method 600 using Deeply Separable Convolutional Neural Networks (DSCNNs) 200, 201 is schematically illustrated. DSCNNs 200, 201 can be used in computationally resource-constrained devices, mobile devices, and / or autonomous devices, such as autonomous robots and / or autonomous vehicles 114. DSCNNs 200, 201 can be used to perform one or more application tasks. Application tasks may include one or more of computer vision, object recognition, image processing, image recognition, image classification, medical imaging, and / or image generation. According to one embodiment, DSCNNs 200, 201 may be trained using method 500.

[0092] Method 600 may include a training step. This training step may include steps for performing method 500. Training method 500 may be performed on a graphics processing unit (GPU). Method 600 may further include step 601: for example, during the inference phase, using DSCNN 200, 201 on a resource-constrained device. The resource-constrained device may have computational resource limitations, such as a consumer CPU and / or an edge device, such as in an autonomous vehicle. The device may include mobile devices and / or autonomous devices, such as autonomous robots and / or autonomous vehicles. Step 601 using DSCNN may include using DSCNN to perform one or more of computer vision, object recognition, image processing, image recognition, image classification, medical imaging, and / or image generation, wherein DSCNN 200, 201 may be trained according to one embodiment using method 500. The trained, quantized DSCNN 200, 201 may be used during the inference phase to perform such application tasks, such as deriving predictions for a given input using trained weights of convolutional layers. The derivation of predictions can be performed without backpropagating errors through DSCNN and adapting convolutional layer weights. DSCNN 200 and 201 can use images as input. DSCNN 200 and 201 can be included in one or more models for computer vision, object recognition, image processing, image recognition, image classification, medical imaging, image generation, and / or other image analysis applications, and / or contain one or more of these models.

[0093] It will be apparent to those skilled in the art that there are numerous possibilities for performing one or more methods 400, 500, 600 according to one embodiment. For example, the steps may be performed in the order shown, but the order of the steps may also vary, or some steps may be performed in parallel. Furthermore, other method steps may be inserted between the steps. The inserted steps may represent a refinement of the methods 400, 500, 600 described herein, or may not be associated with methods 400, 500, 600. For example, some steps may be performed at least partially in parallel. Furthermore, a particular step may not have been fully completed before the next step begins.

[0094] Implementations of methods 400, 500, and 600 can be executed by means of software containing instructions for execution of methods 400, 500, and 600 by a processor system. This software may contain only steps executed by a system-specific subunit. The software may be stored on a suitable storage medium, such as a hard disk, memory, optical disk, etc. The software may be transmitted via cable, wirelessly, or via a data network, such as the Internet. The software may be provided on a server for download and / or remote use. Implementations of methods 400, 500, and 600 can also be executed using bitstreams arranged to configure programmable logic, such as a field-programmable gate array (FPGA), to execute methods 400, 500, and 600.

[0095] Obviously, the subject matter disclosed herein also extends to computer programs, particularly to computer programs on a carrier, suitable for implementing the subject matter disclosed herein in practice. The program can be in the form of source code, object code, intermediate source code, and object code, for example, it can be in a partially compiled form, or in any other form suitable for prior implementation of the method. One implementation related to a computer program product includes computer-executable instructions corresponding to each processing step of at least one of the illustrated methods. These instructions can be subdivided into subroutines and / or stored in one or more files, which can be statically or dynamically linked. Another implementation related to a computer program product includes computer-executable instructions corresponding to each device, unit, and / or portion of at least one illustrated system and / or product.

[0096] Methods 400, 500, and 600 can be computer-implemented methods. For example, access to and transmission of training data and / or reception of other input data can be performed through communication interfaces, such as electronic interfaces, network interfaces, storage interfaces, etc. For example, storage or retrieval of training parameters can be performed through electronic storage devices, such as working memory, hard disks, etc. For example, adaptation of stored parameters can be performed through electronic computing devices, such as computers. Each of the methods 400, 500, or 600 described in this specification can be implemented on a computer as a computer-implemented method 400, 500, or 600, implemented as dedicated hardware or a combination of both.

[0097] Figure 7aComputer-readable media 1000 having a writable portion 1010 and computer-readable media 1001 also having a writable portion are schematically shown. Computer-readable media 1000 is shown in the form of an optically readable medium. Computer-readable media 1001 is shown in the form of an electronic memory, in this case, a memory card. Computer-readable media 1000 and 1001 can store data 1020, wherein the data can specify instructions that, when executed by a processor system, cause the processor system to perform embodiments of methods 400, 500, and 600. Data 1020 may include a computer program 1020 according to one embodiment. The computer program 1020 may be embodied on the computer-readable media 1000 as a physical marker or by magnetization of the computer-readable media 1000. However, any other suitable embodiments are also contemplated. Furthermore, it should be noted that although computer-readable media 1000 is shown herein as an optical disc, computer-readable media 1000 can be any suitable computer-readable medium, such as a hard disk, solid-state storage, flash memory, etc., and can be non-writable or writable. Computer program 1020 may include instructions that cause the processor system to perform implementations of methods 400, 500, and 600.

[0098] Figure 7bA processor system 1140 is shown, which may include or represent a system suitable for performing quantization, training, and / or inference methods 400, 500, 600, as described elsewhere in this specification. The processor system 1140 may include a device 110 according to one embodiment, wherein the device 110 is adapted to perform the quantization, training, and / or inference methods 400, 500, 600 as described elsewhere in this specification. The processor system 1140 may include one or more subsystems or components 1110. For example, a processing subsystem 1120 may be provided for executing computer program components to perform the methods 400, 500, 600 as described elsewhere in this specification. A memory 1122 may be provided for storing program code, data, etc. A communication subsystem 1126, such as a network interface, may enable communication with other entities. In some examples, an application-specific integrated circuit 1124 may be provided to perform all or part of the processing associated with the methods described elsewhere in this specification. Processing subsystem 1120, memory 1122, dedicated IC 1124, and communication subsystem 1126 can be interconnected via connection 1130, such as a bus. While system 1140 is shown as having one of these components, various components may be replicated in different embodiments. For example, processing subsystem 1120 may include multiple microprocessors adapted to independently perform the methods described herein, or adapted to perform steps or subroutines of methods 400, 500, 600 described herein, such that multiple processors cooperate to achieve the functions described herein. If system 1140 is implemented in a cloud computing system, cloud server, and / or computing field, the various hardware components may further belong to separate physical systems. For example, processing subsystem 1120 may include a first processor in a first server and a second processor in a second server.

[0099] Processor system 1140 is suitable for training DSCNN 200, quantizing, validating, and / or validating other DSCNNs, and / or acquiring, receiving, and / or generating training data for training dataset 310 used for such training. DSCNN 200 can be trained on processor system 1140 to perform application tasks, as mentioned elsewhere. Processor system 1140 can receive training data 310 as input from another device. Processor system 1140 can be part of system 100. Processor system 1140 is suitable for receiving, sending, transmitting, forwarding, processing, monitoring, filtering, and / or storing data streams. Processor system 1140 may include one or more sensors that can determine environmental measurements in the form of sensor signals, which may be given by, for example, digital images, such as medical images, video, radar, lidar, ultrasound, motion thermal imaging, or audio signals. Image data may be obtained from sensor data. Processor system 1140 can produce a classification of the data as output. This output can be used to control, for example, actuators. For example, processor system 1140 may include resource-constrained and / or mobile device 110, which may include, for example, autonomous vehicle 140, including sensors that identify the presence of objects in the vehicle environment. Classification tasks may include: classifying data from sensors, identifying the presence of objects in sensor data, and / or performing semantic segmentation of the data, such as regarding traffic signs, road surfaces, pedestrians, and vehicles. Classification may include assigning labels from a given set of labels to the entire image. From a set of labels, for example, containing types of traffic participants, image classification may decide whether an image displays labels from that set. In autonomous vehicles, image classification may be applied, for example, to label images from image sensors on and / or within the vehicle, such as forward-facing cameras. In applications for object recognition, the location of objects, such as labeled objects, may be further determined. This can be particularly useful when multiple traffic participants are present in an image. Based on classification and / or object recognition, a decision-making process may be performed. This classification may be the classification of data that processor system 1140 may have transmitted over a communication network, and / or the classification of data that processor system 1140 may have forwarded over a communication network. Other categorized tasks may include: identifying anomalies in technical systems, calculating control signals for controlling technical systems, such as computer-controlled machines like robotic systems, vehicles, home appliances like washing machines, power tools, manufacturing machines, personal assistants, or access control systems, or for controlling control signals for systems used for information transmission, such as monitoring systems or medical systems like medical imaging systems.In applications of the trained DSCNN 200, such as in robotics and automated and / or autonomous driving, as described above, the training method 500 and the subsequent inference method 600, during the training and / or inference phases, are able to achieve optimal performance of the trained DSCNN 200 on device 110 by using data including images, radar data, etc., on device 110, where the DSCNN can be trained and / or used. Optimal performance of the neural network 200 can be determined with reference to the technical limitations considered for the edge device 110, such as computational performance and / or energy resources.

[0100] Processor system 1140 may be adapted to generate test data, validation data, and / or validation data to check whether the trained DSCNN 200 can be safely trained, used, and / or run on processor system 1140. Processor system 1140 may also be adapted to generate test data, validation data, and / or validation data to check whether the trained DSCNN 200 can be safely trained on a device, which may be an internal or external device relative to processor system 1140. The device may be a resource-constrained and / or mobile device. Processor system 1140 may be adapted to determine whether there are sufficient storage, computing, and / or energy resources on the device to train and / or use DSCNN 200. After training, such as according to method 500 as described elsewhere in this specification, DSCNN 200 may be deployed according to the implementation described in this specification, in accordance with inference or usage method 600. Processor system 1140 may be adapted for both training DSCNN 200 and using the trained DSCNN 200. The processor system 1140 can also generate a training dataset 310 and / or input samples 31, 31' for training another DSCNN 200. The system 1140 can also train the other DSCNN 200.

[0101] Please note that the method 500 disclosed herein for training DSCNN and the method 600 disclosed herein for using DSCNN may be part of the same computer implementation of method 500, 600.

[0102] Examples, implementation methods, or optional features, whether or not described as non-limiting, should not be construed as limiting the claimed invention.

[0103] Note that the above embodiments are illustrative and not limiting of the invention, and those skilled in the art will be able to devise many alternative embodiments without departing from the scope of the appended claims. Any reference numerals in parentheses in the claims should not be construed as limiting the claims. The use of the verb "comprising" and its variations does not exclude the presence of elements or steps other than those described in the claims. The article "a" or "an" preceding an element does not exclude the presence of one of a plurality of such elements. The expression "at least one" preceding a group or enumeration of elements indicates the selection of all or any subset of the elements from that enumeration or group. For example, the expression "at least one A, B, and C" should be understood as only A, only B, only C, A and B, A and C, B and C, or all of A, B, and C. The invention can be implemented by hardware comprising a plurality of different elements and by a computer programmed accordingly. In device claims enumerating a plurality of means, a plurality of these means can be embodied by the same hardware element. The fact that specific measures are described only in dependent claims that differ from one another does not mean that combinations of these measures cannot be used advantageously.

[0104] In the claims, references enclosed in parentheses refer to reference numerals in the drawings of exemplary embodiments or formulas of embodiments, thereby improving the comprehensibility of the claims. These references should not be construed as limiting the claims.

Claims

1. A computer-implemented quantization method (400) for a deep separable convolutional neural network (DSCNN) (200), wherein the deep separable convolutional neural network (200) includes multiple network layers (210, 210', 220, 230, 230', 240), each of the multiple network layers (210, 210', 220, 230, 230', 240) includes multiple network weights (21, 21', 22), and the multiple network layers (210, 210', 220, 230, 230', 240) include: - Suitable for performing one or more pointwise convolutional layers (210, 210') to perform pointwise convolutions, wherein each of the one or more pointwise convolutional layers (210, 210') includes a plurality of pointwise convolutional layer weights (21, 21'), and - One or more depthwise convolutional layers (220) suitable for performing depthwise convolutions, each of the one or more depthwise convolutional layers (220) including a plurality of depthwise convolutional layer weights (22). - The quantization method (400) wherein the quantization (410) includes quantizing (410) the plurality of network weights (21, 21', 22), including: - The plurality of pointwise convolutional layer weights (21, 21') are quantized (411) into a plurality of quantized pointwise convolutional layer weights (21.1, 21.1') within a first discrete range, and - The plurality of deep convolutional layer weights (22) are quantized (412) into a plurality of quantized deep convolutional layer weights (22.1) within a second discrete range, wherein the first discrete range has a cardinality that is strictly lower than that of the second discrete range.

2. The method (400) according to any one of the preceding claims, wherein - The first discrete range includes bases of 2 or 3. - The second discrete range includes bases in the range of 3 to 256, such as 16, and / or - The first discrete range includes a first base, and the second discrete range includes a second base, wherein the first base multiplied by fifty is less than the second base.

3. The method (400) according to any one of the preceding claims, wherein - The one or more pointwise convolutional layers (210, 210') are suitable for performing pointwise 1x1 convolutions, and / or - Each of the one or more deep convolutional layers (220) includes multiple input channels, and each of the one or more deep convolutional layers (220) is adapted to independently extract information from the corresponding multiple input channels using a 3x3 kernel.

4. The method (400) according to any one of the preceding claims, wherein the depthwise separable convolutional neural network (200) comprises at least two pointwise convolutional layers (210, 210'), wherein - The first pointwise convolutional layer (210) is suitable for projecting the input of the depthwise separable convolutional neural network (200) into a higher-dimensional latent space, and - The second pointwise convolutional layer (210') is suitable for projecting the second input onto a lower-dimensional latent space, wherein the second input has the dimension of the higher-dimensional latent space.

5. The method (400) of claim 4, wherein the second input comprises the output of a deep convolutional layer (220) from the one or more deep convolutional layers (220), and the second pointwise convolutional layer (210') is adapted to perform pointwise convolution to generate a linear combination of the outputs of the deep convolutional layer (220).

6. The method (400) according to any one of the preceding claims, wherein the plurality of network layers (210, 210', 220, 230, 230', 240) further comprises one or more activation layers (230, 230'), wherein each of the one or more activation layers (230, 230') includes an activation function (23, 23'), wherein each of the one or more activation layers (230, 230') is adapted to perform activation on an input to the corresponding activation layer (230, 230'), wherein the activation includes applying the corresponding activation function (23, 23') to the input, the method (400) further comprising adapting one or more activation functions (23, 23') of the one or more activation layers (230, 230') to one or more adapted activation functions (23.1, 23.1'), wherein the one or more adapted activation functions (23.1, 23.1') Each of 23.1' is different from the corresponding activation function (23, 23') in the one or more activation functions (23, 23') from the one or more activation layers (230, 230').

7. The method (400) according to claim 6, wherein - The one or more activation functions (23, 23') include non-parametric activation functions, such as ReLU activation functions, ReLU activation functions with upper limits, Hardswish activation functions, sign activation functions, or LeakyReLU activation functions, and - The one or more adapted activation functions (23.1, 23.1') include parameterized activation functions, such as the PReLU activation function.

8. The method (400) according to any one of the preceding claims, wherein, Each of the multiple network layers (210, 210', 220, 230, 230', 240) includes one or more input channels and one or more output channels. The quantization (410) of the multiple network weights (21, 21', 22) includes: for each of the one or more output channels, - Determine the scaling factor. - The network weights (21, 21', 22) are scaled using a determined scaling factor to quantize the network weights (21, 21', 22) across multiple network weights (21, 21', 22), and the scaled network weights are rounded to a rounded value within a discrete range, wherein the discrete range is a first discrete range or a second discrete range.

9. The method (400) according to claim 8, wherein, The scaling factor is determined by any of the following for one or more output channels of the corresponding network layer (210, 210', 220, 230, 230', 240): - The average absolute value of the network weights (21, 21', 22) of the output channel on one or more input channels of the corresponding network layer (210, 210', 220, 230, 230', 240); - The maximum absolute value of the network weights (21, 21', 22) of the output channel. - The minimum absolute value of the network weights (21, 21', 22) of the output channel. - A uniform non-negative real value that is independent of the network weights (21, 21', 22) of the output channel.

10. A computer implementation method (500) for training a deep separable convolutional neural network (DSCNN) (200), wherein the deep separable convolutional neural network (200) includes a plurality of network layers (210, 210', 220, 230, 230', 240), each of the plurality of network layers (210, 210', 220, 230, 230', 240) including a plurality of network weights (21, 21', 22), the plurality of network layers (21, 21', 22) including: - One or more pointwise convolutional layers (210, 210') suitable for performing pointwise convolution, wherein each of the one or more pointwise convolutional layers (210, 210') includes a plurality of pointwise convolutional layer weights (21, 21'). - It also includes depthwise convolutional layers (220) suitable for performing depthwise convolutions, wherein each of the one or more depthwise convolutional layers (220) includes a plurality of depthwise convolutional layer weights (22), and - One or more activation layers (230, 230'), wherein each of the one or more activation layers (230, 230') is adapted to perform activation on the input of the corresponding activation layer (230, 230'). The method (500) includes: - Iteratively train (501) the multiple network weights (21, 21', 22) on the training dataset (310), wherein the training dataset (310) includes multiple input samples (31, 31'), wherein training (501) for the input samples (31, 31') includes: - During forward propagation (301, 510), - Simulate (511) the quantization step (20) of the network layers (210, 210', 220, 230, 230', 240) using the quantization method (400) according to any one of the preceding claims, wherein the plurality of network weights (21, 21', 22) are quantized into a plurality of quantized network weights ( 21.1、21.1'、22.1）， - The activation (24, 24') of the input sample (31, 31') performed by the one or more activation layers (230, 230') is quantized (512) into quantized activations (24.1, 24.1') in a discrete range. - By rescaling the multiple quantized network weights ( 21.1, 21.1', 22.1) and rescaling the quantized activations (24.1, 24.1') to apply the multiple quantized network weights ( 21.1, 21.1', 22.1) and the quantized activations (24.1, 24.1') are dequantized (520), which results in multiple dequantized network weights (21.2, 21.2', 22.2) and dequantized activations (24.2, 24.2'). - During backpropagation (302, 530), gradient estimation (531) is performed with the dequantized network weights (21.2, 21.2', 22.2) and the dequantized activations (24.2, 24.2'). - Based on the gradient estimation (40), adapt (540) the dequantized network weights (21.2, 21.2', 22.2) and the dequantized activations (24.2, 24.2'). - Provides (503) trained deep separable convolutional neural networks (200) for inference.

11. The method (500) according to claim 10, further comprising: - Before providing (503) the trained deep separable convolutional neural network (200) for inference, the plurality of network weights (21, 21', 22) are quantized (502) using the quantization method (400) according to any one of claims 1 to 9, resulting in a plurality of quantized network weights (21.1, 21.1', 22.1), and - During the inference, the multiple quantized network weights (21.1, 21.1', 22.1) are maintained (504).

12. A computer-implemented method (600) for using (601) a deep separable convolutional neural network (200, 201) on a computationally limited device, mobile device and / or autonomous device, such as an autonomous robot and / or autonomous vehicle, to perform one or more of computer vision, object recognition, image processing, image recognition, image classification, medical imaging and / or image generation, wherein said deep separable convolutional neural network (200, 201) is trained using the method (500) according to claim 10 or claim 11.

13. The method (400, 500, 600) according to any one of the preceding claims, wherein the deep separable convolutional neural network (200, 201) uses an image as input and / or includes one or more models for computer vision, object recognition, image processing, image recognition, image classification, medical imaging, image generation and / or other image analysis applications.

14. A volatile or non-volatile computer-readable medium (1000) comprising data (1020, 1021) representing a computer program, wherein the computer program comprises instructions that cause a processor system (1140) to perform the method (400, 500, 600) according to any one of claims 1 to 13.

15. A processor system (1140) comprising a memory (1122) and one or more processors (1120), wherein the memory (1122) includes instructions that cause the one or more processors (1120) to perform the method (400, 500, 600) according to any one of claims 1 to 13.

Images