Instruction set architecture for neural network quantization and packaging

By using a single-instruction mechanism on mobile devices to process the feature vectors of neural network layers in parallel, the efficiency problem of neural network deployment on resource-constrained devices is solved, and efficient neural network inference is achieved.

CN119013656APending Publication Date: 2024-11-22QUALCOMM INC
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202380033259.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2022-04-28
Filing Date
2023-03-20
Publication Date
2024-11-22

AI Technical Summary

Technical Problem

Existing general-purpose processors require significant computational, storage, and power resources to perform neural network operations, making it difficult to efficiently deploy neural networks on resource-constrained mobile devices.

Method used

It adopts a single instruction mechanism and utilizes a vector processor to process the feature vector elements of the neural network layer in parallel. Through quantization and activation function operations, it reduces computation, storage and power consumption.

Benefits of technology

Enables efficient neural network inference on resource-constrained platforms, saving computing, storage, and power resources and extending device uptime.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119013656A_ABST
    Figure CN119013656A_ABST
Patent Text Reader

Abstract

The invention relates to initiating a sequence of computing operations related to a neural network using a single instruction. An electronic device receives a single instruction to apply neural network operations to a set of M-bit elements stored in one or more input vector registers. In response to the single instruction, the electronic device implements the neural network operation on the set of M-bit elements to generate a set of P-bit elements by: obtaining the set of M-bit elements from the one or more input vector registers; quantizing each M-bit element in the set of M-bit elements from M-bits to P-bits; and packing the set of P-bit elements into an output vector register. P is smaller than M. In some embodiments, the neural network operation is a quantization operation including at least a multiplication with respect to a quantization factor and an addition with respect to a zero.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] Priority application

[0002] This application claims priority to U.S. patent application serial number 17 / 732,361, filed on April 28, 2022, entitled “INSTRUCTION SET ARCHITECTURE FOR NEURAL NETWORK QUANTIZATION AND PACKING,” which is incorporated herein by reference in its entirety. Technical Field

[0003] The present application generally relates to data processing, including but not limited to methods, systems, and apparatus for using a single instruction to initiate a sequence of computing operations corresponding to a neural network function or operation. Background Art

[0004] Neural networks (NNs) have been widely used in deep learning techniques to simulate the operation of the human brain and solve complex data-driven problems in various applications. Examples of these applications include, but are not limited to, facial recognition, stock market prediction, handwriting recognition, image processing, and speech synthesis. Each neural network includes one or more layers of artificial neurons configured to convert input data into output data. For each neural network layer, the artificial neurons combine the incoming data in a weighted and linear manner, and apply a nonlinear activation function to the combined data to generate intermediate data to be processed by the next neural network layer of the output data of the neural network. These neural network operations involve computational operations that can be implemented on existing general-purpose processors (e.g., central processing units (CPUs)). However, general-purpose processors typically repeat similar or identical computational operations on data associated with individual artificial neurons, regardless of any neural network architecture, thereby requiring a large amount of computational, storage, and power resources. This makes the deployment of relatively complex neural networks almost untrustworthy on mobile devices with limited resources. It is highly desirable to provide an efficient hardware and / or software mechanism that implements computational operations associated with neural network functions in electronic devices (particularly mobile devices). Summary of the invention

[0005] Various specific implementations of the systems, methods, and devices within the scope of the appended claims each have several aspects, no single aspect of which is solely responsible for the properties described herein. Without limiting the scope of the appended claims, after considering this disclosure and in particular after considering the section entitled "Detailed Description of the Invention", it will be understood how to use various aspects of some specific implementations to initiate a sequence of computational operations corresponding to a neural network function or operation (e.g., neuron-level activation and quantization) using a single instruction. After issuing the single instruction, the sequence of operations is automatically implemented in a predefined order to process multiple feature vector elements of a neural network layer in parallel using processor and register resources arranged based on a single instruction. This single instruction-initiated arrangement helps save computational, storage, and power resources utilized during a neural network inference process implemented on a resource-constrained platform (e.g., wearable devices, smart home appliances, modem devices, Internet of Things (IOT) devices, and mobile phones).

[0006] In one aspect of the present application, a method for quantizing neural network data is employed at an electronic device (e.g., a mobile device). The electronic device has one or more processors and a memory storing one or more programs. The one or more processors include a plurality of vector registers. The method includes: receiving a single instruction for applying a neural network operation to a set of M-bit elements stored in one or more input vector registers; and in response to the single instruction, implementing the neural network operation on the set of M-bit elements to generate a set of P-bit elements. P is less than M. Implementing the neural network operation also includes: obtaining the set of M-bit elements from the one or more input vector registers; quantizing each M-bit element in the set of M-bit elements from M bits to P bits of a corresponding element in the set of P-bit elements; and packing the set of P-bit elements into an output vector register.

[0007] In some specific implementations, the single instruction includes a quantization factor and a zero point, and the neural network operation is a quantization operation, and the quantization operation includes at least a multiplication with respect to the quantization factor and an addition with respect to the zero point. The addition is after the multiplication.

[0008] In some specific implementations, quantizing each M-bit element in the set of M-bit elements further includes: quantizing the set of M-bit elements from M bits to P bits of the corresponding element in the set of P-bit elements in parallel using single instruction multiple data (SIMD).

[0009] In some embodiments, the set of M-bit elements includes a first set of M-bit elements, and the set of P-bit elements includes a first set of P-bit elements. The method also includes: based on the size of the one or more input vector registers, dividing the input feature vector into multiple sets of M-bit elements, the multiple sets of M-bit elements including the first set of M-bit elements and one or more remaining sets of M-bit elements. The method also includes: in response to one or more of the aforementioned instructions, loading each set of M-bit elements from the memory of the electronic device to the corresponding input vector register or storing each set of M-bit elements to the corresponding input vector register. The method also includes: repeating the single instruction to implement the neural network operation for each M-bit element in the one or more remaining sets of M-bit elements to generate a corresponding remaining set of P-bit elements.

[0010] In another aspect of the present application, a method for applying an activation function to neural network data is employed at an electronic device (e.g., a mobile device). The electronic device has one or more processors and a memory storing one or more programs. The one or more processors include one or more input vector registers. The method includes: receiving a single instruction for applying a linear activation operation to a set of first elements stored in one or more input vector registers, and the single instruction includes one or more scalars and a bias value. The method also includes: in response to the single instruction, implementing the linear activation operation on the set of first elements to generate an output element set. Implementing the linear activation operation also includes: for each element in the set of first elements, detecting a sign value of a corresponding element in the set of first elements; selecting a corresponding scalar from the one or more scalars based on the sign value; and applying the linear activation operation to the corresponding element in the set of first elements based on the selected corresponding scalar and the bias value to generate a corresponding element in the set of output elements. The method also includes: quantizing the set of output elements.

[0011] In some specific implementations, the set of first elements includes a first set of first elements. The method also includes: based on the size of the one or more input vector registers, splitting the first feature vector into multiple sets of first elements, the multiple sets of first elements including the first set of first elements and one or more remaining sets of first elements; in response to one or more of the aforementioned instructions, loading each set of first elements from the memory of the electronic device to the corresponding input vector register or storing each set of first elements to the corresponding input vector register; and repeating the single instruction to implement the linear activation operation for each first element in the one or more remaining sets of first elements. In addition, in some specific implementations, the electronic device also includes an output vector register, and the set of output elements includes a first set of output elements, and implementing the linear activation operation also includes: packing the first set of output elements in the output vector register.

[0012] In another aspect, an electronic device includes one or more processors and a memory storing one or more programs. The one or more processors include at least one or more input vector registers. The electronic device is configured to implement any of the above methods.

[0013] In another aspect, a non-transitory computer-readable storage medium stores one or more programs and at least one or more input vector registers. The one or more programs include instructions that, when executed by one or more processors, cause the one or more processors to perform any of the above methods.

[0014] In yet another aspect, an apparatus comprises means for performing any of the above methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] For a better understanding of the various described implementations, reference should be made to the following detailed description in conjunction with the following drawings, in which like reference numerals refer to corresponding parts throughout the drawings.

[0016] Figure 1 An example data processing environment having one or more servers communicatively coupled to a plurality of electronic devices according to some implementations.

[0017] Figure 2A is a block diagram illustrating an electronic device configured to process data using a neural network according to some implementations.

[0018] Figure 2B is a block diagram of a processor of an example electronic device 104 according to some specific implementations.

[0019] Figure 3Ais an example neural network (NN) applied to process data according to some specific implementations, and Figure 3B is an example artificial neuron in a neural network according to some specific implementations.

[0020] Figure 4A and Figure 4B An example linear activation function configured to be implemented by a linear activation operation at an artificial neuron of a neural network is shown according to some implementations.

[0021] Figure 5 is a flow chart of a process for implementing a linear activation operation in response to a single instruction in a vector processor according to some specific implementations.

[0022] Figure 6 is a block diagram of an example vector processor configured to implement a single instruction for a linear activation function associated with a neural network layer, according to some implementations.

[0023] Fig. 7A is an example artificial neuron involving a sequence of neural network operations (e.g., quantization, dequantization) according to some specific implementations, and Figure 7B are two example artificial neurons coupled in series and involved in a sequence of neural network operations according to some specific implementations.

[0024] Figure 8 is a flow chart of a process for implementing a quantization operation on data elements of a neural network in response to a single instruction according to some specific implementations.

[0025] Fig. 9 is a flow chart of a process for implementing a quantization operation in response to a single instruction in a vector processor according to some specific implementations.

[0026] Fig.10 is a block diagram of an example vector processor configured to implement a single instruction for quantization associated with a neural network layer, according to some implementations.

[0027] Fig.11 is a flow chart of a method for quantizing features of a neural network layer according to some specific implementations.

[0028] Fig.12 is a flow chart of a method for applying a linear activation function to a neural network layer according to some specific implementations.

[0029] For a better understanding of the various described implementations, reference should be made to the following detailed description in conjunction with the following drawings, in which like reference numerals refer to corresponding parts throughout the drawings. Like reference numerals refer to corresponding parts throughout the drawings. DETAILED DESCRIPTION

[0030] Reference will now be made in detail to specific implementations, examples of which are illustrated in the accompanying drawings. In the following detailed description, numerous non-limiting specific details are set forth to aid in understanding the subject matter presented herein. However, it will be apparent to one of ordinary skill in the art that various alternatives may be used without departing from the scope of the claims, and that the subject matter may be practiced without these specific details.

[0031] Figure 1 1 is an example data processing environment 100 with one or more servers 102 communicatively coupled to multiple electronic devices 104 according to some specific implementations. Multiple electronic devices 104 may include, for example, a network interface device 104A, a tablet computer 104B, a mobile phone 104C, a wearable device 104D, or a smart, multi-sensing, network-connected home device 104E (e.g., a surveillance camera, a smart TV device, a drone), or other Internet of Things (IOT) devices 104F. Each electronic device 104 can collect data or user input, execute a user application and present output on its user interface. The collected data or user input can be processed locally at the electronic device 104 and / or remotely by the server 102. One or more servers 102 provide system data (e.g., boot files, operating system images, and user applications) to the electronic device 104, and process the data or user input received from the electronic device 104. In some specific implementations, different functional modules are executed simultaneously at the server 102 and the electronic device 104 to enable the user application for the user of the electronic device 104. In some implementations, data processing environment 100 also includes one or more storage devices 106 for storing data related to server 102 , electronic device 104 , and user applications executed on server 102 and electronic device 104 .

[0032] The server 102, the electronic device 104, and the storage device 106 are communicatively coupled to each other and are configured to exchange data via one or more communication networks 108, which are media for providing communication links between devices and computers connected together within the data processing environment 100. The one or more communication networks 108 may include connections, such as wired communication links, wireless communication links, or fiber optic cables. Examples of one or more communication networks 108 include a local area network (LAN), a wide area network (WAN) (such as the Internet), or a combination thereof. The one or more communication networks 108 are optionally implemented using any known network protocol, including various wired or wireless protocols, such as Ethernet, Universal Serial Bus (USB), FIREWIRE, Long Term Evolution (LTE), Global System for Mobile Communications (GSM), Enhanced Data GSM Environment (EDGE), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Bluetooth, Wi-Fi, Voice over Internet Protocol (VoIP), Wi-MAX, or any other suitable communication protocol.

[0033] Examples of network interface devices 104A include routers, modems, switches, gateways, hubs, or smart dedicated whole-house control nodes. For each electronic device 104 different from the network interface device 104A, a connection to one or more communication networks 108 can be established directly (e.g., using 3G / 4G / 5G connectivity to a wireless operator) or through the network interface device 104A or through any combination thereof. For example, an LTE modem provides a multi-mode single-chipset connectivity solution that is built to support reliable, optimized cellular connectivity for IOT devices 104F (such as asset trackers, health monitors, security systems, smart city sensors, and smart meters) and a range of wearable devices 104D. This LTE modem helps battery-powered IoT devices 104F operate in the field for extended durations (e.g., close to 10 years or more) and efficiently interacts with small-form-factor, low-power, wide-area connectivity-required IoT devices 104F.

[0034] The one or more communication networks 108 may represent the Internet, a global collection of networks and gateways that use the Transmission Control Protocol / Internet Protocol (TCP / IP) protocol suite to communicate with each other. At the heart of the Internet is a backbone of high-speed data communication lines between major nodes or host computers, consisting of thousands of commercial, government, educational and other computer systems that route data and messages.

[0035] In some embodiments, deep learning techniques are applied to the data processing environment 100 to process data (e.g., visual data, audio data) via a user application executed at the electronic device 104. These deep learning techniques can identify information contained in the data, match the data with other data, classify the data, or synthesize related data using a data processing model created based on one or more neural networks to process the data. These data processing models are trained using training data before being applied to process the data. In some specific implementations, the server 102 controls the data processing process, that is, trains the data processing model, obtains data provided by the electronic device via the communication network 108, processes the data using the data processing model, and returns the processed data to the electronic device. Alternatively, in some specific implementations, the server 102 and the electronic device 104 collaborate to complete the data processing process. The data processing model is trained at the server 102 and provided to a separate electronic device 104. Each electronic device 104 obtains data (e.g., captures video data via an internal camera, receives data via the communication network 108) and processes the data locally using the data processing model. In addition, in some embodiments, the electronic device 104 has a large amount of computing, storage, and power resources, and independently completes the training of the data processing model and the reasoning of the data with little or no participation of the server 102.

[0036] In some specific implementations of the present application, the electronic device 104 (e.g., the network interface device 104A) has limited computing, storage, and power resources. Such electronic devices 104 do not train and directly utilize a data processing model created based on one or more neural networks and trained by the server 102. In addition, the data processing model is modified before being deployed to the electronic device 104. For example, one or more neural networks are pruned or quantized by the server 102 so that the corresponding data processing model can be simplified in view of the resources of the electronic device 104. When the modified data processing model is executed at the electronic device 104, the electronic device 104 applies a predefined single instruction to implement the corresponding neural network operation associated with one or more neural networks (e.g., quantization, application of activation functions). The predefined single instruction initiates a predefined sequence of computing operations within the vector processor of the electronic device to implement the corresponding neural network operation. Through these components, the electronic device 104 reserves its own computing, storage, and power resources during neural network reasoning.

[0037] Figure 2Ais a block diagram showing an electronic device 104 configured to process data using a neural network according to some specific implementations. The electronic device 104 includes one or more processors 202 (e.g., CPU 202A, vector processor 202B), one or more network interfaces 204, memory 206, and one or more communication buses 208 (sometimes referred to as chipsets or systems on a chip (SoC)) for interconnecting these components. The vector processor 202B is different from the CPU 202A and is located outside it. In some specific implementations, the electronic device 104 includes one or more input devices 210 for facilitating user input, such as a keyboard, a mouse, a voice command input unit or microphone, a touch screen display, a touch-sensitive input pad, a gesture capture camera, or other input buttons or controls. In addition, in the example, the electronic device 104 uses a microphone for voice recognition or a camera for gesture recognition to supplement or replace the keyboard. In some embodiments, the electronic device 104 includes one or more optical cameras (e.g., RGB cameras), scanners, or light sensor units for capturing images of, for example, a graphic serial code printed on the electronic device. In some implementations, the electronic device 104 also includes one or more output devices 212 that enable presentation of a user interface and display of content, including one or more speakers and / or one or more visual displays. Optionally, the electronic device 104 includes a location detection device for determining the location of the electronic device 104, such as a GPS (Global Positioning System) or other geolocation receiver.

[0038] Memory 206 includes high-speed random access memory, such as DRAM, SRAM, DDR RAM or other random access solid-state memory devices; and optionally, includes non-volatile memory, such as one or more magnetic disk storage devices, one or more optical disk storage devices, one or more flash memory devices, or one or more other non-volatile solid-state storage devices. Memory 206 optionally includes one or more storage devices located away from one or more processors 202. Memory 206 or alternatively non-volatile memory within memory 206 includes non-transitory computer-readable storage media. In some embodiments, memory 206 or non-transitory computer-readable storage media of memory 206 stores the following programs, modules, and data structures or subsets or supersets thereof:

[0039] An operating system 214, which includes procedures for handling various basic system services and for performing hardware-related tasks;

[0040] A network communication module 216 for connecting each electronic device 104 to other devices (e.g., a server 102, an electronic device 104, or a storage device 106) via one or more network interfaces 204 (wired or wireless) and one or more communication networks 108 (such as the Internet, other wide area networks, local area networks, metropolitan area networks, etc.);

[0041] A user interface module 218 for enabling presentation of information (e.g., graphical user interfaces for applications 224, widgets, their websites and web pages and / or games, audio and / or video content, text, etc.) at each electronic device 104 via one or more output devices 212 (e.g., displays, speakers, etc.);

[0042] An input processing module 220 for detecting one or more user inputs or interactions from one of the one or more input devices 210 and interpreting the detected inputs or interactions;

[0043] A web browser module 222 for navigating, requesting (e.g., via HTTP), and displaying its websites and web pages, including a web interface for logging into a user account associated with the electronic device 104 or another electronic device, controlling the electronic device 104 (if associated with a user account), and editing and reviewing settings and data associated with the user account;

[0044] One or more user applications 224 for execution by an electronic device 104 (e.g., a game, a social networking application, a smart home application, and / or other web-based or non-web applications for controlling another electronic device 104 and reviewing data captured by such devices);

[0045] a data processing module 226 for processing the data using the data processing model 236 to identify information contained in the data, match the data with other data, classify the data, or synthesize related content data, wherein in some embodiments, the data processing module 226 is associated with one of the user applications 224 to process the data in response to a single instruction received from the user application 224; and

[0046] One or more databases 228, the one or more databases being used to store data including at least one or more of the following:

[0047] o Device settings 230, which include common device settings of the electronic device 104

[0048] (e.g., service levels, device models, storage capacity, processing power, communication capabilities, etc.);

[0049] o User account information 232 for one or more user applications 224, such as user name, security questions, account history data, user preferences, and predefined account settings;

[0050] o Network parameters 234 of one or more communication networks 108, such as IP addresses, subnet masks, default gateways, DNS servers, and host names;

[0051] o a data processing model 236 for processing data (e.g., video, images, audio, or text) using a neural network; and

[0052] o Data and results 238 , which are respectively obtained by and output to the electronic device 104 , wherein the data is processed by a data processing model 236 located locally at the electronic device 104 to provide associated results to be presented on the electronic device 104 .

[0053] In some implementations, the data processing module 226 includes a linear activation module 240 configured to be implemented by the vector processor 202B. The linear activation module 240 is configured to apply a linear activation function in parallel to a set of feature vector elements associated with a neural network layer in response to a single instruction. The linear activation function is implemented for each feature vector element via a sequence of computational operations (e.g., multiplication, addition). The single instruction initiates the implementation of the sequence of computational operations corresponding to the linear activation function.

[0054] In some implementations, the data processing module 226 includes a quantization module 242 configured to be implemented by the vector processor 202B. The quantization module 242 is configured to quantize the set of feature vector elements associated with the neural network layer in parallel in response to a single instruction. The quantization function is implemented for each feature vector element via a sequence of computational operations (e.g., multiplication, addition). The single instruction initiates the implementation of the sequence of computational operations corresponding to the quantization function.

[0055] Each element in the element identified above can be stored in one or more memory devices in the memory device mentioned previously, and corresponds to the instruction set for performing the above functions. The modules or programs (that is, instruction sets) identified above do not need to be implemented as independent software programs, processes, modules or data structures, and therefore the various subsets of these modules can be combined or otherwise rearranged in various embodiments. In some embodiments, memory 206 optionally stores the subset of the modules and data structures identified above. In addition, memory 206 optionally stores additional modules and data structures not described above.

[0056] Figure 2B 2 is a block diagram of a processor 202 of an example electronic device 104 according to some specific implementations. The processor 202 includes an instruction cache 243, a data cache 244, a register 246, and an arithmetic unit 248. These general components of the processor 202 are configured to implement neural network operations stored in the instruction cache 243. Specifically, the instruction cache 243 is a first-in, first-out (FIFO) memory, and multiple instructions are stored in the instruction cache 243. The retrieval of the multiple instructions follows the same order and sequence as the storage of the multiple instructions. The processor 202 also includes a sequencer 250 and an instruction sort multiplexer 252. The sequencer 250 is configured to control the instruction sort multiplexer 252 to select each individual instruction arranged in the instruction cache 243, and the valid instruction 254 is currently selected and implemented by the processor 202.

[0057] The processor 202 also includes a load / store unit 256 configured to execute load / store instructions associated with the effective instructions 254, generate virtual addresses for load and store operations, and load data from the memory 258 or store data from the register 246 back to the memory 258 in response to the effective instructions 254. The memory 258 includes the data cache 244 (e.g., L1 cache) of the processor 202. In some specific implementations, the memory 258 also includes a higher-level cache (e.g., L2 cache and L3 cache) and a dynamic random access memory (DRAM) located outside the processor 202. In some cases, the load / store unit 256 directly extracts data to be used to implement the effective instructions 254 from the data cache 244. Conversely, in some cases, the data to be used does not exist in the data cache 244, and the load / store unit 256 accesses a higher-level cache or DRAM to extract data to be used to implement the effective instructions 254. The implementation result of the effective instructions 254 is optionally stored in the register 246 and used directly by subsequent instructions. Alternatively, the load / store unit 256 stores the result in the register 246 and moves it further to the memory 258 (eg, the data cache 244 ).

[0058] According to the effective instruction 254, the processor 202 is configured to allocate multiple registers 246 and arithmetic units 248, which are organized to complete the effective instruction 254 end-to-end without being interrupted by additional communication with the instruction cache 243. The register 246 optionally includes a vector register file 246A and a vector predicate file 246B. The vector register file 246A corresponds to one or more input vector registers 260, an output vector register 262, one or more scalar registers 264 and an intermediate result register 266. The data in the intermediate result register 266 is cleared after the single instruction is completed. The vector predicate file 246B stores the predicates associated with each computing operation used to implement the effective instruction 254.

[0059] In some implementations, the processor 202 is a vector processor 202B, and the vector processor 202B is part of a larger heterogeneous modem computer system 104A (e.g., an LTE modem). The input data or output data of the neural network layer corresponds to a feature vector including a plurality of elements. The effective instruction 254 includes an activation instruction or a quantization instruction regarding the feature vector of the neural network layer. In some cases, the plurality of elements are processed in parallel by the vector processor 202B in response to the effective instruction 254. Alternatively, in some cases, the plurality of elements are grouped into a plurality of element subsets. The vector processor 202B processes each element subset in parallel in response to the effective instruction 254, and repeats the effective instruction 254 until all element subsets are processed. With respect to each individual effective instruction 254, a sequence of basic computational operations is implemented to process the data loaded in the input vector register 260 in parallel without being interrupted by loading instructions or data from the memory 258.

[0060] Figure 3A is an example neural network (NN) 300 applied to process data according to some specific implementations, and Figure 3B is an example artificial neuron 320 (also referred to as a node or filter) in a neural network 400 according to some implementations. The neural network 300 includes a set of artificial neurons 320 connected by links 312. Each artificial neuron 320 receives one or more node inputs x 1 、x 2 、x 3 , ..., and x n and apply a propagation function to input x from one or more nodes 1 、x 2 、x 3 , ..., and x n Generate node output y i . One or more node inputs x 1 、x 2 、x3 , ..., and x n According to the propagation function, based on the corresponding weight w 1 、w 2 、w 3 , ..., and w n Similarly, when the node output y is transmitted via one or more links 312 i When provided to one or more other artificial neurons 320, the weight w associated with each link 312 is applied to the node output y i In the example, the propagation function of the artificial neuron 320 includes: 1 、x 2 、x 3 , ..., and x n The activation function 324 is applied to the linear weighted combination 322 of x. In some cases, one or more node inputs x 1 、x 2 、x 3 , ..., and x n The linear weighted combination 322 includes the node bias b i . Activation function 324 is optionally linear or non-linear.

[0061] The collection of artificial neurons 320 is organized into one or more layers in the neural network 300. Optionally, the one or more layers include a single layer that serves as both an input layer 302 and an output layer 306. Optionally, the one or more layers include an input layer 302 for receiving input, an output layer 306 for providing output, and zero or more hidden layers 304 (e.g., 304A and 304B) between the input layer 302 and the output layer 306. A deep neural network has more than one hidden layer 304 between the input layer 302 and the output layer 306. In the neural network 300, each layer is connected only to its immediately preceding layer and / or its immediately succeeding layer. In some embodiments, a neural network layer (e.g., input layer 302, hidden layer 304B) is a fully connected layer because each artificial neuron 320 in the layer is connected to each artificial neuron 320 in its immediately succeeding layer.

[0062] Figure 4A and Figure 4B An example linear activation function 324 is shown configured to be implemented by a linear activation operation at an artificial neuron 320 of a neural network 300 according to some implementations. In the neural network 300, each artificial neuron 320 has a corresponding activation function 324 that activates the corresponding activation function at a given node input x. 1 、x 2 , ..., and x nThe node output y of the corresponding neuron 320 is defined by the weighted combination 322 of . If the artificial neuron 320 is part of the input layer 302, then one or more node inputs x 1 、x 2 , ..., and x n Corresponds to the elements of the input feature vector. If the artificial neuron 320 is part of the output layer 306 or the hidden layer 304, then one or more node inputs x 1 、x 2 , ..., and x n corresponds to the node output y of the artificial neuron 320 of one or more other layers in the neural network 300. In the example, the neural network layer (e.g., Figure 3B The output layer 306 and hidden layer 304A in the neural network layer are after the fully connected layer, and each artificial neuron 320 in this layer is connected to each artificial neuron 320 in its immediately previous layer. The node output y of the artificial neuron 320 of the neural network layer i Together they form the output feature vector and are determined as follows:

[0063] Y=f(W·X+b) (1)

[0064] Where f is the activation function, W∈R^(m×n) is the weight matrix, and X∈R^(n×1) is the input x of the corresponding node. 1 、x 2 , ..., and x n The input feature vector is formed, b∈R^(m×1) is the bias vector, and Y∈R^(m×1) is the output feature vector of the neural network layer. If the neural network layer is after the fully connected layer, no element in the weight matrix is ​​equal to 0.

[0065] The node input x of each artificial node 320 of the layer of the neural network 300 in the linear activation unit 400 is 1 、x 2 , ..., and x n The weighted combination 322 of implements a linear activation function 324. The linear activation function 324 includes one or more scalars, for example, a first scalar α and a second scalar β. The linear activation unit 400 is represented as follows:

[0066]

[0067] Where x is the node input x corresponding to artificial neuron 320 1 、x 2 , ..., and x n A weighted combination of 322. Reference Figure 4A, for each artificial node 320 of the neural network layer, if the weighted combination 322 of the node inputs is equal to or greater than 0, the weighted combination 322 of the node inputs is multiplied by the first scalar α to generate the corresponding node output y i Alternatively, for each artificial node 320, if the weighted combination 322 of the node inputs is less than 0, the weighted combination 322 of the node inputs is multiplied by the second scalar β to generate the corresponding node output y i .

[0068] refer to Figure 4B In some implementations, the linear activation unit includes a rectified linear unit (ReLU) 410, and the first scaler α and the second scaler β of the linear activation function 324 of the neural network layer are equal to 1 and 0, respectively. For each artificial node 320 of the neural network layer, if the weighted combination 322 of the node input is equal to or greater than 0, the corresponding node output y i is equal to the weighted combination 322 of the node inputs, and if the weighted combination 322 of the node inputs is less than 0, the corresponding node output y i =0. Alternatively, in some implementations, the linear activation unit includes a leaky ReLU 420, and the first scalar α of the linear activation function 324 of the neural network layer is equal to 1, and the second scalar β is less than 1 (e.g., equal to 0.1). For each artificial node 320 of the neural network layer, if the weighted combination 322 of the node inputs is equal to or greater than 0, the corresponding node output y i is equal to the weighted combination 322 of the node inputs, and if the weighted combination 322 of the node inputs is less than 0, the corresponding node output y i is equal to the weighted combination 322 of the node inputs multiplied by a second scalar β that is less than 1.

[0069] Alternatively, in some implementations, the linear activation unit includes a parameter ReLU 430, and the first scaler α of the linear activation function 324 of the neural network layer is equal to 1, and the second scaler β is equal to, for example, β i For each artificial node 320 of the neural network layer, if the weighted combination 322 of the node inputs is equal to or greater than 0, the corresponding node output y i is equal to the weighted combination 322 of the node inputs, and if the weighted combination 322 of the node inputs is less than 0, the corresponding node output y i = equal to the weighted combination of the node inputs 322 multiplied by a second scalar β i The second scaler β of each neural network layer i A calibrator vector is formed, which is trained using the weights W and biases b of the neural network 300.

[0070] Figure 5 2 is a flow chart of a process 500 for implementing a linear activation operation in response to a single instruction in a vector processor 202B according to some specific implementations. As described above, each artificial node 320 of a neural network layer corresponds to a weighted combination 502 of node inputs. For the entire neural network layer, a plurality of weighted combinations 502 of node inputs corresponding to the artificial nodes 320 form a first feature vector 504 to be provided to a linear activation unit 400, 410, 420, or 430. In the linear activation units 400 to 430, each weighted combination 502 of node inputs requires a scaling operation conditioned on the sign of the corresponding weighted combination 502. When the weighted combinations 502 of node inputs for the artificial neurons 320 of the neural network layer are stored in one or more input vector registers 260, the vector processor 202B extracts each weighted combination 502, determines the sign of the weighted combination 502, selects a scalar based on the sign using a comparison followed by a multiplexer, and multiplies the weighted combination 502 by the selected scalar. Specifically, these computation operations are initiated by a single instruction for activation and are completed as part of the corresponding linear activation operation.

[0071] The single instruction corresponds to one or more input vector registers 260, output vector registers 262, and scalar registers 264. The one or more input vector registers 260 store a set 512 of first elements, for example, a subset of a weighted combination 502 of node inputs of a neural network layer. The single instruction includes one or more scalars, and the scalar register 264 stores one or more scalars. In response to the single instruction, a linear activation operation is implemented on the set 512 of first elements stored in and extracted from the one or more input vector registers 260 to generate an output element set 514. For each element in the set 512 of first elements, the vector processor 202B detects a sign value 516 of a corresponding element in the set 512 of first elements, and selects a corresponding scalar 518 from one or more scalars stored in the scalar register 264 based on the sign value 516. The vector processor 202B then applies a linear activation operation 520 to each element in the set 512 of first elements based on the selected corresponding scalar 518 and a bias value 522 to generate a corresponding element in the output element set 514. The bias value 522 is set with a single instruction. The vector processor 202B temporarily stores the output element set 514 in the output vector register 262. In some implementations, the vector processor 202B extracts the output element set 514 from the output vector register 262 and continues to quantize the output element set 514 for further processing in the next neural network layer or for output to and storage in the memory 258.

[0072] In some embodiments, a linear activation operation 520 is applied to a corresponding element in a first set of elements 512 by multiplying the corresponding element 524 by a corresponding scaler 518 to produce a scaled corresponding element 524 and adding a bias value 522 to the scaled corresponding element 512 to produce a biased corresponding element 526. For example, a k-bit shift is applied and the corresponding bias value 522 is 1<<k-1. Additionally, in some implementations, the corresponding elements in the first set of elements 512 are 32 bits and the scaled corresponding elements 524 are 64 bits. The shift can be from 0 bits to 63 bits. In an example, a 15-bit shift is applied (i.e., k = 15) and the corresponding bias value 522 is a constant value 1<<14 or 0x4000. The vector processor 202B adds the bias value 522 to the biased corresponding element 524 to obtain a biased corresponding element 526, and right-shifts the biased corresponding element 526 by 15 bits to obtain a shifted corresponding element 528, which is further saturated to 32 bits to provide the corresponding element in the output set of elements 514. In other words, the scaler 518 for the linear activation operation 520 is a fixed-point number, and the bias value 522 is a rounding value added to the scaled corresponding element 524 as part of fixed-point arithmetic. In the example, the first element Vuu.V32s[i] (512) is a 32-bit signed number. Each scalar α or β is a 16-bit scalar value stored in the half of a 32-bit scalar register 264. The scaled corresponding element 524 is equal to α×Vuu.V32s[i] and has 48 bits. The scaled corresponding element 524 is combined with the rounded bias of the bias value 522 and right-shifted by 15 bits. The semantics of performing these operations are as follows:

[0073]

[0074] where Vuu.V32s[i]*scale_a is the scaled corresponding element 524 with 48 bits. The scaled corresponding element 524 is maintained in a 64-bit container.

[0075] In other words, the one or more scalars include a first scalar α and a second scalar β, which are stored in two parts (e.g., Rt.h[1] and Rt.h[0]) of the scalar register 264. For each first element 512, a corresponding scalar is selected based on the sign value 516. Based on determining that the sign value 516 corresponds to a positive sign, the vector processor 202B selects the first scalar α. Based on determining that the sign value 516 corresponds to a negative sign, the vector processor 202B selects the second scalar β. More specifically, in some implementations, for the ReLU 410, the one or more scalars include 1 and 0. Based on determining that the sign value 516 corresponds to a positive sign, the vector processor 202B selects the corresponding scalar 1, and based on determining that the sign value 516 corresponds to a negative sign, the vector processor 202B selects the corresponding scalar 0. In some implementations, for the vulnerable ReLU 420, the one or more scalars include 1 and a predefined vulnerability parameter β. Based on determining that the sign value 516 corresponds to a positive sign, the vector processor 202B selects the corresponding scalar 1, and based on determining that the sign value 516 corresponds to a negative sign, the vector processor 202B selects the corresponding scalar predefined vulnerability parameter. The predefined vulnerability parameter β is less than 1. In some specific implementations, for the parameter ReLU 430, the one or more scalars include 1 and a predefined filter-based scalar. Based on determining that the sign value corresponds to a positive sign, the vector processor 202B selects the corresponding scalar 1, and based on determining that the sign value corresponds to a negative sign, the vector processor 202B selects the predefined filter-based scalar β. i The weights and biases of the neural network are used to train a predefined filter-based scaler β i .

[0076] refer to Figure 5 In some implementations, one or more input vector registers 260 are configured to store a set 512 of first elements having a total of a first number of M-bit elements. The output vector register 262 is configured to store a set 514 of output elements having a second number of M-bit elements. The second number is equal to the first number. The scalar register 510 has a single M-bit element and is configured to store two N-bit scalars from which the corresponding scalar is selected. M is equal to 2N. For example, a pair of input vector registers 260 has a total of 64 32-bit elements, as does the output vector register 262. The scalar register 510 has 32 bits for storing a first 16-bit scalar and a second 16-bit scalar corresponding to the positive sign and negative sign of each first element 512, respectively.

[0077] In some specific implementations, a linear activation operation is implemented in parallel on the set 512 of first elements using single instruction multiple data (SIMD). The set 512 of first elements optionally fills a subset or all of one or more input vector registers 260. In some embodiments, the set 512 of first elements forms a first feature vector 504, which includes a weighted combination 502 of node inputs for all artificial nodes 320 of the neural network layer. Optionally, the first feature vector 504 is loaded from the memory 258 of the electronic device 104 to one or more input vector registers 260. Optionally, the first feature vector 504 is stored in one or more input vector registers 260 in response to one or more of the aforementioned instructions (e.g., related to the weighted combination 322) and is used directly in response to a single instruction. In response to a single instruction, the set 512 of first elements is extracted from one or more input vector registers 260.

[0078] Alternatively, the set 512 of first elements includes a first set 512A of first elements. The first feature vector 504 includes a weighted combination 502 of node inputs for all artificial nodes 320 of the neural network layer and has more elements than the size of one or more input vector registers 260. In other words, the first feature vector 504 has so many elements 502 that they do not fit into the one or more input vector registers 260 and must be processed by more than one instruction. Based on the size of the one or more input vector registers 260, the first feature vector 504 is divided into multiple sets 512 of first elements, the multiple sets of first elements including the first set 512A of first elements and one or more remaining sets 512R of first elements. In response to one or more of the aforementioned instructions (e.g., an addition operation implemented according to formula (1)), each set 512 of first elements is loaded from the memory 258 of the electronic device 104 to the corresponding input vector register 260 or previously stored to the corresponding input vector register 260. The single instruction is repeated to implement a linear activation operation 520 for each first element in the one or more remaining sets 512R of first elements. Specifically, each single instruction corresponds to applying the linear activation function 324 to a single set 512 of first elements, and the first feature vector 504 requires more than one instruction to be fully processed.

[0079] In addition, in some implementations, the output elements 514 of two different sets 512 of first elements are packed into the same output vector register 262. For example, in response to a first single instruction, a first set 514A of output elements generated from a set 512A of first elements is stored in a first portion of the output vector register 262. In response to a second single instruction, one of the remaining sets 512R of first elements is processed to generate a remaining set 514R of a corresponding output element, which is then stored in a second portion of the output vector register 262. The second portion is completely different from the first portion without any overlap. Alternatively, in some implementations, the output elements 514 of each set 512 of first elements are packed into different output vector registers 262 without sharing an output vector register 262 with any other set 512 of first elements.

[0080] The artificial node 320 corresponding to the neural network layer receives an input feature vector 530 (X), which includes multiple node inputs x 1 、x 2 , ..., and x n The first feature vector 504 is generated from the input feature vector 530 according to formula (1) and includes the node input x 1 、x 2 , ..., and x n . Specifically, according to formula (1), the vector processor 202B multiplies the input feature vector 530 with the weight matrix W to obtain an intermediate feature vector WX, and adds the bias vector b to the intermediate feature vector WX to generate a first feature vector 504 including a set 512 of first elements. In addition, in some specific implementations, each element of the input feature vector 530 (X), the weight matrix W, and the bias vector b includes a fixed-point P-bit number. Before multiplication and addition, the vector processor 202B dequantizes each element of the input feature vector 530 (X), the weight matrix W, and the bias vector b to a fixed-point M-bit number. M is greater than P. In the example, M and P are equal to 32 and 8, respectively. In the example, the electronic device 104 includes a mobile device, and the weight matrix W and the bias vector b have quantized P-bit elements and are stored in the memory 258 in association with the neural network layer to which the linear activation function 324 is applied. Reference is made below Fig. 7A and Figure 7B Explains more details about quantization and dequantization of neural networks.

[0081] Figure 6 3 is a block diagram of an example vector processor 202B configured to implement a single instruction for a linear activation function 324 associated with a neural network layer according to some specific implementations. For example, the single instruction is composed of vscale_relu (Vuu ,R t ):rnd:sat indicates. According to the single instruction, the vector processor 202B organizes multiple registers 246 and arithmetic units 248 to implement the single instruction without communicating with the instruction cache 243. The register 246 includes a vector register file 246A and a vector predicate file 246B. The vector register file 246A corresponds to one or more input vector registers 260, an output vector register 262, one or more scalar registers 264, and multiple intermediate result registers 266. The vector predicate file 246B stores the predicate associated with each calculation operation for implementing the single instruction, and translates the semantics of the single instruction into the vector predicate file 246B. The intermediate result register 266 and the arithmetic unit 248 are organized according to the vector predicate file 246B, and cooperate to process the input provided by the input vector register 260 and the scalar register 264 and generate the output stored in the output vector register 262 according to the single instruction for activation.

[0082] In the example, the intermediate result registers 266 include a vector register read flip-flop 602, a scalar read register 604, pipeline registers 606, 608, and 610, a multiplier result register 612, and a round result register 614. The plurality of arithmetic units 248 include a scalar multiplexer 616, a multiplier 618, a rounding logic unit 620, and a data write multiplexer 622. Each arithmetic unit 248 is optionally a basic arithmetic logic unit (ALU) in a combination of two or more basic ALUs, and each basic ALU is a combinational digital circuit that performs arithmetic and bitwise operations on integer binary numbers.

[0083] Specifically, each element in the set 512 of first elements is extracted from one or more input vector registers 260. The sign value 516 is extracted and stored in the vector register read flip-flop 602, and the corresponding element 512 is stored in the pipeline register 606. The scalar read register 604 stores one or more scalar values ​​extracted from the scalar register 264. The multiplexer 616 is controlled by the sign value 516 stored in the read flip-flop 602 to select the corresponding scalar 518. The corresponding selected scalar 518 is stored in the pipeline register 608. The corresponding element 512 and the selected scalar 518 stored in the pipeline registers 606 and 608 are multiplied by the multiplier 618 to generate the scaled corresponding element 524 stored in the multiplier result register 612. The scaled corresponding element 524 is processed by the rounding logic unit 620 to generate the shifted corresponding element 528 stored in the rounding result register 614. The shifted corresponding element 528 is saturated to 32 bits to provide the corresponding output element 514 stored in the pipeline register 610. The data write multiplexer 622 writes the corresponding output element 514 stored in the pipeline register 610 to the corresponding element of the output vector register 262.

[0084] The vector processor 202B implements a single instruction using conditional scaling based on the sign value 516 of each first element 512 and the corresponding scalar 518. This conditional scaling includes at least three computational operations (compare, multiplexer, and scale), and is implemented in response to a single instruction rather than separate sequential instructions arranged in an instruction queue of the instruction cache 243. In addition, the conditional scaling is configured to be based on linear units using different scaling values ​​(e.g., based on Figure 4A or Figure 4B In the example, the conditional scaling is implemented using the ReLU 410, 420, 430 or 440 in 410. In the quantized inference calculation, the dequantization scaling can be performed together with the activation function scaling using a single instruction. In the example, the processor resources are reused in each calculation stage to achieve a three-fold speedup, thereby managing the workload of neural network inference in an efficient manner. In addition to implementing linear activation functions for neural networks, conditional scaling can also be used in non-machine learning applications that need to scale vector elements by a scalar value based on the sign of each vector element.

[0085] Fig. 7A is an example artificial neuron 320 involving a sequence of neural network operations (e.g., quantization, dequantization) according to some specific implementations, and Figure 7B300 are two example artificial neurons 320 coupled in series and involved in a sequence of neural network operations according to some implementations. As described above, the neural network 300 includes a set of artificial neurons 320 connected by links 312 and organized into one or more layers (e.g., input layer 302, hidden layer 304, output layer 306). Each artificial neuron 320 receives one or more node inputs x 1 、x 2 、x 3 , ..., and x n And generate node output y from one or more node inputs based on the propagation function i The propagation function consists of applying an input x to one or more nodes. 1 、x 2 、x 3 , ..., and x n The activation function 324 is applied to the linear weighted combination 322 of x. Specifically, one or more node inputs x of each artificial node 320 1 、x 2 、x 3 , ..., and x n Based on the corresponding weight w 1 、w 2 、w 3 , ..., and w n and node deviation b i to combine.

[0086] In some implementations, the electronic device 104 implements the neural network 300 at a low level of precision (e.g., using 8-bit fixed-point representation) to preserve limited computational, storage, or power resources. i , node deviation b i , node input x i and node output y i is quantized and stored at a low level of precision, while the neural network computation is implemented at a high level of precision (e.g., using 32-bit fixed-point representation). That is, when the propagation function is implemented at each artificial neuron 320, the weight w 1 、w 2 、w 3 , ..., and w n , node deviation b i and node input x 1 、x 2 、x 3 , ..., and x n Each of them is dequantized at a high level of precision into a corresponding weight DQw i , node deviation DQb i Or node input DQx iThe dequantized weight DQw 1 ,DQ 2 ,DQ 3 , ..., and DQw n , node deviation DQb i and node input DQx 1 、DQx 2 、DQx 3 , ..., and DQx n Combined with a linear weighted combination 322 and processed using an activation function 324, the node output y is generated with a high level of accuracy. i The node output y of the corresponding artificial neuron 320 is i Quantized to Qy at a low level of precision i Quantization node output Qy i The elements of the input feature vector X are provided to the next layer 704 of the neural network 300 or output from the neural network 300.

[0087] For the layer immediately following the fully connected layer, the weight matrix W, the input feature vector X, and the bias vector b are quantized to W using affine integer quantization. q , X q , b q The propagation function of the layer immediately following the fully connected layer is modified as follows:

[0088] Y=f(dequant(W q )·dequant(X q )+dequant(b q )) (3)

[0089] where dequant(X q ) is defined as:

[0090] DQ i =dequant(X q )=S x (X q -ZP x ) (4)

[0091] Where S x is the quantization scale, and ZP x is the quantized zero point of X. After the elements of the output feature vector Y have been calculated at the high precision level, the output feature vector Y is quantized to the low precision level as follows:

[0092] Y q =quantize(Y) (5)

[0093] where quantize(Y) is defined as:

[0094]

[0095] Where S y is the quantization scale, and ZP y It is the quantization zero point of Y. The quantization node outputs Qy i are the elements of the quantized output feature vector Y.

[0096] For each neural network layer, the quantized output feature vector Y is optionally stored in the output vector register 262 of the last instruction that generated the quantized output feature vector Y, and is directly applied to the next instruction that uses the quantized output feature vector Y. Alternatively, in some implementations, the quantized output feature vector Y is moved from the output vector register 262 associated with the last instruction to the memory 258. When the next instruction (e.g., a dequantization instruction) that uses the quantized output feature vector Y is initiated, the quantized output feature vector Y is loaded from the memory 258 to one or more input registers 260 associated with the next instruction.

[0097] refer to Figure 7B In some implementations, the artificial neurons 320 belong to the first layer 702 and the next layer 704 immediately following the first layer 702. The quantized node output Qy of the artificial neurons 320 of the first layer 702 i The elements of the input feature vector X are provided to the next layer 704 of the neural network 300. In the next layer 704, the second propagation function includes one or more node inputs x 1 、x 2 、x 3 , ..., and x n The second activation function 324' and the second linear weighted combination 322' are the node output y of the first layer 702. 1 ,y 2 ,y 3 , ..., and y n In the next layer 704, the nodes in the first layer 702 output y 1 ,y 2 ,y 3 , ..., and y n Based on the corresponding weight w of the next layer 704 1 、w 2 、w 3 ,……,w n and node deviation b i And further combination.

[0098] Likewise, when the second propagation function is implemented at each artificial neuron 320 of the next layer 704, the weight w of the next layer 704 1 、w 2 、w3 , ..., and w n and node deviation b i Each of the two is dequantized into a corresponding weight DQw i Or node deviation DQb i The node output Qy of the first layer 702 is i Each node output in is dequantized to the corresponding node output DQy at a high precision level. i The dequantized weight DQw of the next layer 704 is 1 ,DQ 2 ,DQ 3 , ..., and DQw n and node deviation DQb i And the node output DQy of the first layer 702 1 ,DQ 2 ,DQ 3 , ..., and DQy n are combined into a linear weighted combination 322' and processed using a second activation function 324' to generate the node output y of the next layer 704 at a high level of accuracy i The node output y in the next layer 704 i Quantified as Qy i .

[0099] In some implementations, the result of the linear weighted combination 322 or 322' is stored in the output vector register 262 in response to the instruction associated with the linear weighted combination 322. After receiving a single instruction for implementing the following linear activation function 324 or 324', the output vector register 262 is reconfigured as one or more input vector registers 260 of the single instruction, and the result of the linear weighted combination 322 or 322' is directly extracted for processing by the corresponding linear activation operation (e.g., in process 500). Alternatively, in some implementations, the result of the linear weighted combination 322 or 322' is stored in the output vector register 262 and moved to the memory 258 of the corresponding processor 202 (e.g., vector processor 202B). After receiving a single instruction for implementing the linear activation function 324 or 324', the result of the linear weighted combination 322 or 322' is loaded from the memory 258 to the one or more input vector registers 260 of the single instruction for the corresponding linear activation operation.

[0100] Figure 8 is a flow chart of a process 800 for implementing a quantization operation on a data element of a neural network in response to a single instruction according to some specific implementations. i The activation output vector Y (802) is formed, and based on equations (5) and (6) (i.e., based on the quantization factor and zero point ZP y (806)) to activate each individual element y of the output vector Y (802) i A quantization operation is applied. In other words, for each neural network layer, the activation output vector Y (802) serves as the input feature vector 802 for the quantization operation. A subset of the elements of the input feature vector 802 is stored in one or more input vector registers 260. The vector processor 202B is configured to quantize the subset of elements of the input feature vector 802 based on the quantization factor 804 and the zero point 806. The corresponding set of quantized elements forms a subset of the output feature vector 808 and is stored in the output vector register 262. Specifically, the sequence of computational operations is initiated by a corresponding single instruction for quantization, and the sequence of computational operations is performed in parallel on the subset of elements of the input feature vector 802 using the processor 202 (e.g., the vector processor 202B).

[0101] During the quantization operation, the quantization factor 804 is converted into a mantissa 804A and an integer exponent 804B. The mantissa 804A is a fixed point number. In the example, the mantissa 804A is represented as 16S15, which indicates that the mantissa 804A is a 16-bit fixed point number with a sign bit and 15 fraction bits, and has a value between [-1, 1-2 -15 ] range. Each element y of the input feature vector 802 i 804, which is implemented by a sequence of computational operations 812 to 814 of processor 202. Specifically, the corresponding element y is multiplied (810) by the quantization factor 804. i The mantissa 804A is multiplied to obtain a corresponding scaled element 812A, and the scaled element 812A is shifted (812B) based on the integer exponent 804B, for example, right by 16 bits. i is represented as 16Sx, where x is a positive integer. Mantissa 804A is represented as 16S15, and therefore, the shifted element 812C is represented as 32Sx. Based on integer exponent 804B (i.e., 2 k ) to apply another shift (e.g., saturated left shift, rounded right shift) to the shifted element 812C, and discard one or more fraction bits (814A) to obtain a corresponding M-bit integer 814B, e.g., a 32-bit integer. Zeros 806 are added (816) to the corresponding M-bit integer 814B to obtain a corresponding offset integer 818. The corresponding offset integer 818 is saturated (820) to a quantized P-bit element 908 (e.g., a quantized 8-bit element) for storage in an element of the output vector register 262.

[0102] In some implementations, a single instruction associated with the quantization operation is represented by vmap_wb. The vmap_wb instruction is issued from the instruction cache 243. In response to the vmap_wb instruction, each element y in the subset of the input feature vector 802 is i Processed by an ordered sequence of computational operations 812, 814, 816, and 820. Each computational operation (eg, 812A) is optionally implemented by an elementary ALU or a combination of two or more elementary ALUs.

[0103] Fig. 9 is a flow chart of a process 900 for implementing a quantization operation in response to a single instruction in the vector processor 202B according to some specific implementations. As described above, the node output y at each artificial neuron 320 is calculated according to equations (5) and (6). i For each neural network layer, the node output y corresponding to the artificial node 320 i The input feature vector 802 is formed, which is obtained and quantized by the vector processor 202B. Each node outputs y i The rounded shift operation 904, the addition operation 816 and the saturation operation 820 are processed in sequence. The rounded shift operation 904 outputs y at each node. i Specifically, these computational operations are initiated by a single instruction for quantizing a subset of elements of the input feature vector 802 corresponding to the output nodes of the neural network layer.

[0104] Specifically, a single instruction corresponds to one or more input vector registers 260, output vector registers 262, and scalar registers 264. One or more input vector registers 260 store a set 902 of M-bit elements, for example, a node output y of a neural network layer. i A subset of . The single instruction includes a quantization factor 804 and a zero point 806, and the scalar register 264 stores the quantization factor 804 and the zero point 806. Based on formula (6), the quantization operation includes at least a multiplication about the quantization factor 804 and an addition about the zero point 806. In response to the single instruction, a quantization operation is implemented on a set 902 of M-bit elements stored in one or more input vector registers 260 and extracted from the one or more input vector registers to generate a set 908 of P-bit elements. Each M-bit element 902 is quantized from M bits to the P bits of the corresponding P-bit element 908. For example, a 32-bit element 902 is quantized to an 8-bit element 908. In the vector processor 202B, the quantization operation is implemented by a combination of a rounded shift operation 904, an addition operation 816, and a saturation operation 820.

[0105] The quantization factor 804 includes a first exponent shift value 804C and a second exponent shift value 804D. The vector processor 202B detects a sign value 906 for each element in the set 902 of M-bit elements and selects a corresponding exponent shift value 804C or 804D stored in the scalar register 264 based on the sign value 906. The vector processor 202B then applies a rounded shift operation 904 to a corresponding scaling element 812A for each element in the set 512 of first elements based on the selected corresponding exponent shift value 804C or 804D. An example rounded shift operation 904 is described as follows:

[0106] rshift(a,n):

[0107] If n>0, then:

[0108] round_bias=(1<<(n-1))

[0109] out=(a+round_bias)>>n

[0110] otherwise:

[0111] out=sat(a<<(-n))

[0112] Wherein a is the scaled element 812A scaled from the element 902, and n is the selected exponent shift value 912 (i.e., 804C or 804D). More specifically, in accordance with determining that the selected one of the first exponent shift value and the second exponent shift value 912 is positive, the vector processor 202B applies a rounded right shift to the corresponding scaled element 812A based on the selected one of the first exponent shift value and the second exponent shift value 912. Conversely, in accordance with determining that the selected one of the first exponent shift value and the second exponent shift value is negative, the vector processor 202B applies a saturated left shift to the corresponding scaled element 812A based on the selected one of the first exponent shift value and the second exponent shift value 912.

[0113] In response to a single instruction associated with quantization, a set 902 of M-bit elements loaded into one or more input registers 260 is processed in parallel in the vector processor 202B, for example using SIMD. In some embodiments, the set 902 of M-bit elements optionally fills a subset or all of the one or more input vector registers 260. The set 902 of M-bit elements forms an input feature vector 802 that includes the node output y for all artificial nodes 320 of the neural network layer. i Optionally, the set 902 of M-bit elements of the input feature vector 802 is loaded from the memory 258 of the electronic device 104 to one or more input vector registers 260. Optionally, in response to one or more of the foregoing instructions (e.g., Figure 5 ), stores a set 902 of M-bit elements of the input feature vector 802 to one or more input vector registers 260. After one or more of the preceding instructions, the set 902 of M-bit elements is extracted from the input vector register 260 and used directly in response to a single instruction associated with quantization.

[0114] Alternatively, in some implementations, the set 902 of M-bit elements includes a first set 902A of M-bit elements, and the set of P-bit elements includes a first set 908A of P-bit elements. The input feature vector 802 includes the node outputs of all artificial nodes 320 of the neural network layer to be quantized, and has more elements than the size of the one or more input vector registers 260. In other words, the input feature vector 802 has so many elements that they do not fit into the one or more input vector registers 260 and must be processed using more than one instruction. Based on the size of the one or more input vector registers 260, the input feature vector 802 is split into multiple sets of M-bit elements, the multiple sets of M-bit elements including the first set 902A of M-bit elements and one or more remaining sets 902R of M-bit elements. Optionally, each set 902 of M-bit elements is loaded from the memory 258 of the electronic device 104 to the corresponding input vector register 260. Optionally, in response to one or more of the foregoing instructions (e.g., with Figure 5 ), each set of M-bit elements is stored to a corresponding input vector register 260. The single instruction is repeated to quantize each M-bit element in one or more remaining sets 902R of M-bit elements to generate corresponding remaining sets 908R of P-bit elements.

[0115] In addition, if the input feature vector 802 is partitioned into multiple sets 512 of M-bit elements, the single instruction is repeated the same number of times to quantize the input feature vector 802. In other words, an instruction sequence including the same number of instructions is implemented continuously to fully quantize the input feature vector 802. In response to each single remainder instruction associated with each remainder set 902R of M-bit elements, the vector processor 202B obtains the corresponding remainder set 902R of M-bit elements of the input feature vector 802 from the corresponding input vector register 260. Each M-bit element in the corresponding remainder set of M-bit elements is quantized from M bits to the P bits of the corresponding element in the corresponding remainder set 908R of P-bit elements. The corresponding remainder set 908R of P-bit elements is packed into the corresponding output vector register 262. In addition, in some specific implementations, the single instruction associated with the first set 902A of M-bit elements corresponds to the first instruction signature, and at least one single remainder instruction corresponds to the second instruction signature. The corresponding remainder set 908R of P-bit elements is generated in response to the at least one single remainder instruction. Reference Fig. 9 , a first set 908A of P-bit elements and a corresponding remaining set 908R of P-bit elements are packed into the first and second halves of the output vector register 262, respectively.

[0116] One or more input vector registers 260 are configured to store a first number of M-bit elements 902. The output vector register 262 is configured to store a second number of N-bit elements. The second number is equal to or less than the first number. The output vector register 262 is partially or completely filled with a set 908 of P-bit elements. The quantization ratio is M / X. The spatial ratio of the input register 260 and the output register 262 is equal to or greater than M / X. In an example, the first number and the second number are equal to 64 and 32, respectively. M and N are both equal to 32, and P is equal to 4. Each N-bit element in the output vector register 262 stores up to 4 quantized P-bit elements 908. One or more input vector registers 260 include a pair of 32 register elements each having 32 bits. The output vector register 262 includes 32 register elements each having 32 bits. In response to a single instruction having one of two predefined characteristics, the corresponding half of the output vector register 262 is filled. In another example, the first number and the second number are equal to 64 and 16, respectively. M and N are both equal to 32, and P is equal to 4. Each N-bit element in the output vector register 262 stores up to 4 quantized P-bit elements 908, which are packed into a single register element of 32 bits. One or more input vector registers 260 include a pair of 32 register elements, each with 32 bits. The output vector register 262 includes 16 register elements, each with 32 bits. In response to a single instruction, the output vector register 262 is completely filled.

[0117] In the example, a pair of input vector registers 260 and corresponding output vector registers 262 have 32-bit elements. The two sets of 32-bit elements stored in registers 260 are quantized to 8-bit elements in registers 262 by two variants of the vmap_wb instruction as follows:

[0118]

[0119]

[0120] The single instruction is implemented based on an input vector register pair 260 (Vuu) having 64 32-bit elements, a scalar register pair 264 (Rtt) having 4 16-bit elements, and an output vector register 262 (Vx) having 32 32-bit elements. One half of the output vector register 262 is filled due to the execution of the single instruction. The scalar register 264 includes Rtt.h[2] and Rtt.h[0] storing exponent shift values ​​804C and 804D and Rtt.h[1] storing zero 806. The exponent shift values ​​804C and 804D are applied to the shift operation 904 (which is rounded) with rounding. Figure 8 813 in operation 814. A set 902 (Vuu) of M-bit elements extracted from the input vector register 260 (corresponding to the shift in the operation 813 in operation 814) is shifted. Each M-bit element (Vuu) is shifted based on an exponent shift value 804C or 804D (i.e., Rtt.h[2] or Rtt.h[0]). If the sign value 906 of the corresponding M-bit element 902 (Vuu) is positive, a scaled element 812A of the corresponding M-bit element 902 (Vuu) is processed using a shift using the first exponent shift value 804C. If the first exponent shift value 804C is positive, the shift is a rounding right shift, and if the first exponent shift value 804C is negative, the shift is a saturating left shift. Conversely, if the sign value 906 of the corresponding M-bit element (Vuu) is negative, the scaled element 812A of the corresponding M-bit element 902 (Vuu) is processed using a shift using the second exponent shift value 804D. If the second exponent shift value 804D is positive, the shift is a rounding right shift, and if the second exponent shift value 804D is negative, the shift is a saturating left shift.

[0121] In some implementations, the set 902 of M-bit elements corresponds to the node output of the output layer 306 of the neural network 300. The output layer 306 generates the input feature vector 802, which is quantized in the process 900 into an output feature vector 808 including a set 908 of P bits. The set 908 of P-bit elements of the output feature vector 808 is stored in the memory 258 of the electronic device 104. Such quantized elements 908 help reserve storage resources of the electronic device 104, especially when the electronic device 104 is a mobile device. Therefore, this single instruction requires a minimum temporary memory footprint for holding the layer output.

[0122] In some specific implementations, reference Figure 7B , a set 902 of M-bit elements corresponds to the node output of the first layer 702, which is the input layer 302 or hidden layer 304 of the neural network 300. The second layer 704 is coupled to the first layer 702 and is immediately after the first layer. An input feature vector 802 is generated from the first layer 702 and quantized into an output feature vector 808 including a set 908 of P-bit elements. After the output feature vector 808 is quantized from the input feature vector, the second layer 704 obtains the output feature vector 808, for example, directly from the output vector register 262 to generate a weighted combination of the set 908 of P-bit elements. The second layer 704 includes a plurality of weights and biases, and each of the plurality of weights, biases, and elements 908 of the output feature vector 808 includes a P-bit fixed-point number. Each P-bit fixed-point number is dequantized so that the output feature vector 808 is applied to the second layer 704 using the dequantized weights, dequantized biases, and dequantized elements of the output feature vector 808 to generate a second input vector. In response to the second single instruction, a subset of the second input vector is quantized using process 900. Therefore, the quantized P-bit elements 908 need to be stored in the output vector register 262 or memory 258 in the appropriate order for immediate loading and dequantization associated with the second layer 704.

[0123] In some implementations, a single activate-quantize instruction is applied to perform the weighted combination 322 (i.e., Figure 5 The first element 512 in implements the activation function 324 and outputs y to the node i Quantization operations are implemented. Specifically, when SIMD is applied, in response to a single activation-quantization instruction, a subset of the first feature vector 504 is processed by the activation function 324, quantized, and packed in the output vector register 262. When the first feature vector 504 is divided into multiple subsets of elements, the single activation-quantization instruction is repeated for each subset 512 of the first elements until all elements of the first feature vector 512 are processed.

[0124] Fig.10 2 is a block diagram of an example vector processor 202B configured to implement a single instruction for quantization associated with a neural network layer according to some specific implementations. According to the single instruction for quantization, the vector processor 202B organizes multiple registers 246 and an arithmetic unit 248 to implement the single instruction without being interrupted by communication with the instruction cache 243 during quantization. The registers 246 include a vector register file 246A and a vector predicate file 246B. The vector register file 246A corresponds to one or more input vector registers 260, an output vector register 262, one or more scalar registers 264, and a plurality of intermediate result registers 266. The vector predicate file 246B stores predicates associated with computational operations for implementing the single instruction, and translates the semantics of the single instruction into the vector predicate file 246B. Intermediate result registers 266 and arithmetic unit 248 are organized according to vector predicate file 246B and cooperate to process inputs provided by input vector registers 260 and scalar registers 264 and generate quantized outputs stored in output vector registers 262 according to a single instruction for quantization.

[0125] In the example, the intermediate result registers 266 include a vector register read flip-flop 1002, a scalar read register 1004, pipeline registers 1006 to 1012, a rounding adjustment register 1014, an adder result register 1016, a shifter intermediate result register 1018, a shift result register 1020, an adder operation register 1022, and a calculation result register 1024. The plurality of arithmetic units 248 include a rounding adjuster 1026, a vector adder 1028, a vector variable shifter 1030, a shifter post-processor 1032, an output zero selector 1034, a vector adder 1036, a saturator 1038, a byte packing multiplexer 1040, and a write data multiplexer 1042. Each arithmetic unit 248 is optionally a basic ALU or a combination of two or more basic ALUs, and each basic ALU is a combinational digital circuit that performs arithmetic and bitwise operations on integer binary numbers.

[0126] Specifically, each element in the set 902 of M-bit elements is extracted from one or more input vector registers 260. The sign value 906 is extracted and stored in the vector register read flip-flop 1002, and the corresponding M-bit element 902 is stored in the pipeline register 1006. The scalar read register 1004 stores information associated with the quantization factor 804 and the zero point 806 extracted from the scalar register 264. The quantization factor 804 is processed by the rounding adjuster 1026, the shifter 1030 and the post-processor 1032 to obtain the M-bit integer 814B stored in the shift result register 1020. The zero point 806 is passed by the pipeline registers 1008 to 1012, selected by the output zero point selector 1034 and stored in the adder operation register 1022. The adder 1036 adds the zero point 806 stored in the adder operation register 1022 to the corresponding M-bit integer 814B stored in the shift result register 1020 to obtain the corresponding offset integer 818, which is saturated to a quantized P-bit element 908 (e.g., a quantized 8-bit element) by the saturator 1038. The quantized P-bit element 908 is further packed into the calculation result register 1024 by the packing multiplexer 1040 for storage in the output vector register 262.

[0127] In some implementations, the quantized P-bit elements 908 are packed in the output vector register 262 to enable consistent operations for each neural network layer. In the example, these P-bit elements 908 are packed in an order consistent with the order of the set 902 of M-bit elements in the input feature vector 802, so these P-bit elements 908 are tightly packed without zero padding. Instructions are executed using resources orthogonal to the multiplier to achieve pipeline optimization in the quantization-based reasoning path. In other words, these instructions do not use multiplier resources, while some related calculations (e.g., steps in the same quantization-based reasoning path for the next input vector) must be implemented using multipliers. The instruction saves multiplier resources to implement related calculations simultaneously with these instructions, thereby facilitating a quantization-based reasoning path or facilitating the pipeline of a quantization-based reasoning path. Quantization based on a single instruction implementation includes scaling the accumulator result by fixed-point multiplication, exponent shift, bias addition, and saturation. Thus, a single instruction for quantization fuses multiple computational operations, enabling efficient pipelining and packing, and reducing cycle costs for computing quantized inference models at electronic devices 104 (eg, mobile devices) with limited resources.

[0128] Fig.111 is a flow chart of a method 1100 for quantizing features of a neural network layer according to some specific implementations. For convenience, each of the methods 1100 is described as being implemented by a processor of the electronic device 104. Each of the methods 1100 is optionally implemented by a non-transitory computer-readable storage medium (e.g., Figure 2A 242 in the quantization module 242) and by one or more processors of the electronic device 104 (e.g., Figure 2B 2B) executed by the vector processor 202B in FIG. Each of the operations shown in FIG. 1100 may correspond to instructions stored in a computer memory or a non-transitory computer-readable storage medium (e.g., the memory 206 in FIG. 2 ). The computer-readable storage medium may include a magnetic or optical disk storage device, a solid-state storage device (such as a flash memory), or one or more other non-volatile memory devices. The instructions stored on the computer-readable storage medium may include one or more of the following: source code, assembly language code, object code, or other instruction formats interpreted by one or more processors. Some operations in method 1100 may be combined and / or the order of some operations may be changed.

[0129] The electronic device 104 receives (1102) a single instruction for applying a neural network operation to a set 902 of M-bit elements stored in one or more input vector registers 260. In response to the single instruction, the electronic device 104 implements (1104) the neural network operation on the set 902 of M-bit elements to generate a set 908 of P-bit elements. P is less than (1106) M. Specifically, in response to the single instruction, the electronic device 104 obtains (1108) the set 902 of M-bit elements from the one or more input vector registers 260, quantizes (1110) each M-bit element in the set 902 of M-bit elements from the M-bits of the corresponding element in the set 908 of P-bit elements to P-bits, and packs (1112) the set 908 of P-bit elements into the output vector register 262.

[0130] In some specific implementations, the set 902 of M-bit elements forms (1114) an input feature vector 802. The electronic device 104 loads (1116) the input feature vector 802 from the memory 258 of the electronic device to one or more input vector registers 260 or stores (1118) the input feature vector 802 to one or more input vector registers 260 in response to one or more of the aforementioned instructions. For example, the one or more aforementioned instructions include one or more linear activation instructions applied to the elements of the feature vector to generate the first set 902 of M-bit elements. The elements of the feature vector processed by the one or more linear activation functions are stored in one or more output vector registers associated with the one or more linear activation instructions. The one or more output vector registers associated with the one or more linear activation instructions are directly applied as one or more input vector registers 260 of a single instruction associated with quantization, and the activation elements of the feature vector stored in the output vector registers are extracted and quantized in response to the single instruction.

[0131] Alternatively, in some implementations, the set 902 of M-bit elements includes (1120) a first set 902A of M-bit elements, and the set 908 of P-bit elements includes a first set 908A of P-bit elements. Based on the size of the one or more input vector registers 260, the electronic device 104 divides (1122) the input feature vector 802 into a plurality of sets 902 of M-bit elements, the plurality of sets of M-bit elements including the first set 902A of M-bit elements and one or more remaining sets 902R of M-bit elements. The electronic device loads (1124) each set 902 of M-bit elements from a memory of the electronic device to a corresponding input vector register 260 or stores (1126) each set 902 of M-bit elements to a corresponding input vector register 260 in response to one or more of the foregoing instructions, and repeats (1128) a single instruction to implement a neural network operation on each M-bit element in the one or more remaining sets 902R of M-bit elements to generate a corresponding remaining set 908R of P-bit elements. In some implementations, one or more of the foregoing instructions include one or more linear activation instructions applied to the elements of the feature vector to generate the first set 902 of M-bit elements.

[0132] In addition, in some specific implementations, in response to each single remainder instruction, the electronic device 104 obtains the corresponding remainder set 902R of the M-bit elements of the input feature vector 802 from the corresponding input vector register 260, quantizes each M-bit element in the corresponding remainder set 902R of the M-bit elements from the M-bit to the P-bit of the corresponding element in the corresponding remainder set 908R of the P-bit elements, and packs the corresponding remainder set 908R of the P-bit elements into the corresponding output vector register 262. In addition, in some specific implementations, the single instruction associated with the set 902 of the M-bit elements corresponds to the first instruction signature, and at least one single remainder instruction corresponds to the second instruction signature. The corresponding remainder set 908R of the P-bit elements is generated in response to the at least one single remainder instruction, and the first set 908A of the P-bit elements and the corresponding remainder set 908R of the P-bit elements are packed in the first half and the second half of the output vector register 262, respectively.

[0133] In some specific implementations, reference Figure 8 , the single instruction includes a quantization factor 804 and a zero point 806, and the neural network operation is a quantization operation, the quantization operation including at least a multiplication with respect to the quantization factor 804 and an addition with respect to the zero point 806, the addition being subsequent to the multiplication. In addition, in some implementations, during quantization, the electronic device 104 converts the quantization factor 804 into a mantissa 840A and an integer exponent 804B. For each M-bit element in the set 902 of M-bit elements, the electronic device 104 then sequentially multiplies the corresponding M-bit element with the mantissa 804A to obtain a corresponding scaling element, shifts the corresponding scaling element based on the integer exponent 804B, discards one or more fraction bits to obtain a corresponding M-bit integer, adds the zero point 806 to the corresponding M-bit integer to obtain a corresponding offset integer, and saturates the corresponding offset integer to a quantized P-bit element 908 to be stored in the output vector register 262.

[0134] In addition, in some specific implementations, reference Fig. 9, the integer exponent 804B includes a first exponent shift value 804C and a second exponent shift value 804D. During quantization, the electronic device 104 loads the first exponent shift value 804C, the second exponent shift value 804D, and the zero point 806 in the scalar register 264, and selects one of the first exponent shift value 804C and the second exponent shift value 804D based on the sign of the corresponding M-bit element 902. Based on determining that the selected one of the first exponent shift value 804C and the second exponent shift value 804D is positive, the electronic device 104 applies a rounded right shift to the scaling element 812A of the corresponding M-bit element 902 based on the selected one of the first exponent shift value 804C and the second exponent shift value 804D. In response to determining that the selected one of the first exponent shift value 804C and the second exponent shift value 804D is negative, the electronic device 104 applies a saturated left shift to the scaled element 812A of the corresponding M-bit element 902 based on the selected one of the first exponent shift value 804C and the second exponent shift value 804D.

[0135] In some implementations, one or more input vector registers 260 are configured to store a first number of M-bit elements 902. Output vector register 262 is configured to store a second number of N-bit elements, and the second number is less than the first number. Output vector register 262 is partially or completely filled with a set 908 of P-bit elements. In some embodiments, a first ratio of the first number to the second number is equal to a second ratio of M and P. The first ratio and the second ratio are equal to a third number. Every third number of quantized elements of the input feature vector 802 is packed into the corresponding element of the output vector register 262. For example, the first number and the second number are equal to 64 and 16, respectively, and M and N are both equal to 32.

[0136] In the example, M is 32, and P is 8. One or more input vector registers 260 include a pair of 32 register elements each having 32 bits. The output vector register includes 16 register elements each having 32 bits. In response to a single instruction, the output vector register is fully filled. Every four elements in the set 902 of M-bit elements are quantized into four P-bit elements 908, and the four P-bit elements 908 are packed into a single register element of 32 bits. On the contrary, in another example, M is 32, and P is 8. One or more input vector registers 260 include a pair of 32 register elements each having 32 bits. The output vector register includes 32 register elements each having 32 bits. In response to a single instruction having one of two predefined features, the corresponding half of the output vector register is filled.

[0137] In some implementations, the electronic device 104 quantizes the set of M-bit elements 902 from M bits to P bits of corresponding elements in the set of P-bit elements 908 in parallel using (1128) single instruction multiple data (SIMD).

[0138] In some implementations, the electronic device 104 is a mobile device and is configured to process the neural network 300 having the output layer 306. The output layer 306 is configured to generate the input feature vector 802 including the set 902 of M-bit elements. The output feature vector 808 includes the set of P bits and is quantized from the input feature vector 802 and output by the neural network 300.

[0139] In some implementations, the electronic device 104 is a mobile device and is configured to process the neural network 300. The neural network includes a plurality of layers including at least a first layer 702 and a second layer 704, the second layer 704 being coupled to the first layer 702 and immediately following the first layer. The first layer 702 is configured to generate an input feature vector 802 including a set 902 of M-bit elements. The output feature vector 808 includes a set 908 of P-bit elements. The second layer 704 is configured to receive and process the output feature vector 808 after quantizing the output feature vector 808 from the input feature vector 802.

[0140] Furthermore, in some specific implementations, where the second layer 704 includes a plurality of weights and biases, and each of the plurality of weights, biases, and elements of the output feature vector 808 includes a P-bit fixed-point number, the electronic device 104 dequantizes each of the plurality of weights, biases, and elements of the output feature vector 808, applies the second layer 704 to the output feature vector 808 using the dequantized weights, dequantized biases, and dequantized elements of the output feature vector 808 to generate a second input vector, and quantizes a subset of the second input vector in response to a second single instruction.

[0141] It should be understood that describing Fig.11 The specific order in which the operations are performed is exemplary only and is not intended to indicate that the described order is the only order in which the operations may be performed. A person of ordinary skill in the art will recognize various ways to reorder the operations described herein. In addition, it should be noted that the description herein of Figures 1 to 10 and Fig.12 The details of the other processes described may also be applied in an interchangeable manner in method 1100. For the sake of brevity, these details are not repeated here.

[0142] Fig.121 is a flowchart of a method 1200 for applying a linear activation function to a neural network layer according to some specific implementations. For convenience, each of the methods 1200 is described as being implemented by a processor of an electronic device 104. Each of the methods 1200 is optionally governed by instructions stored in a non-transitory computer-readable storage medium and executed by one or more processors (e.g., a vector processor) of the electronic device 104. Each of the operations shown in Figure 1200 may correspond to instructions stored in a computer memory or a non-transitory computer-readable storage medium (e.g., the memory 206 in Figure 2). The computer-readable storage medium may include a magnetic or optical disk storage device, a solid-state storage device (such as a flash memory), or one or more other non-volatile memory devices. The instructions stored on the computer-readable storage medium may include one or more of the following: source code, assembly language code, object code, or other instruction formats interpreted by one or more processors. Some operations in the method 1200 may be combined and / or the order of some operations may be changed.

[0143] The electronic device 104 receives (1202) a single instruction for applying a linear activation operation to a set 512 of first elements stored in one or more input vector registers 260. The single instruction includes one or more scalars 518 and a bias value 522. In response to the single instruction, the electronic device 104 implements (1204) the linear activation operation on the set 512 of first elements to generate a set of output elements 514. For each element in the set 512 of first elements, the electronic device 104 detects (1206) a sign value 516 of a corresponding element in the set 512 of first elements, selects (1208) a corresponding scalar 518 from one or more scalars 518 based on the sign value 516, and applies (1210) the linear activation operation to the corresponding element in the set 512 of first elements based on the selected corresponding scalar 518 and the bias value 522 to generate a corresponding element in the set 514 of output elements. The electronic device 104 quantizes (1212) the set 514 of output elements.

[0144] In some implementations, the set 512 of first elements forms (1214) the first feature vector 504. The electronic device 104 loads (1216) the first feature vector 504 from the memory of the electronic device to the one or more input vector registers 260 or stores (1218) the first feature vector 504 to the one or more input vector registers 260 in response to one or more of the foregoing instructions. In response to the single instruction, the electronic device 104 obtains the set 512 of first elements from the one or more input vector registers 260. In contrast, in some implementations, the set 512 of first elements includes (1220) the set 512A of first elements. Based on the size of the one or more input vector registers 260, the electronic device 104 divides (1222) the first feature vector 504 into a plurality of sets 512 of first elements, the plurality of sets of first elements including the first set 512A of first elements and one or more remaining sets 512R of first elements. In response to one or more of the foregoing instructions, the electronic device 104 loads (1224) each set 512 of first elements from a memory of the electronic device to a corresponding input vector register 260 or stores (1226) each set 512 of first elements to a corresponding input vector register 260. The electronic device 104 repeats (1228) the single instruction to implement a linear activation operation on each first element in one or more remaining sets 512R of first elements.

[0145] Additionally, in some implementations, the electronic device further includes an output vector register 262, and the set of output elements 514 includes a first set 514A of output elements. After applying the linear activation operation, the electronic device 104 packs the first set 514A of output elements in the output vector register 262. Additionally, in some implementations, the first set 514A of output elements is packed in a first portion of the output vector register 262, and the repeating single instruction includes packing a remaining set 514R of at least one output element in a second portion of the output vector register 262. The second portion is different from the first portion.

[0146] In some implementations, the electronic device 104 obtains the input feature vector 530 of the neural network 300 layer by multiplying the input feature vector 530 with the weight matrix W to obtain an intermediate feature vector and adding the bias vector b to the intermediate feature vector to generate a first feature vector 504 including a set 512 of first elements. In addition, in some implementations, the electronic device 104 obtains the weight matrix W and the bias vector b. Each element of the input feature vector 530, the weight matrix, and the bias vector includes a fixed-point P-bit number. Before multiplication and addition, the electronic device 104 dequantizes each element of the input feature vector 530, the weight matrix, and the bias vector to a fixed-point M-bit number. M is greater than P. In the example, M and P are equal to 32 and 8, respectively. In addition, in some implementations, the electronic device 104 includes a mobile device, and the weight matrix and the bias vector have quantized P-bit elements and are stored in the memory 258 in association with the neural network 300 layer.

[0147] In some implementations, the linear activation operation is configured to enable a linear activation unit 400 of a filter layer of the neural network, and the one or more scalars 518 include a first scalar and a second scalar. The electronic device 104 selects the corresponding scalar based on the sign value 516 by selecting the first scalar based on determining that the sign value 516 corresponds to a positive sign and selecting the second scalar based on determining that the sign value 516 corresponds to a negative sign.

[0148] In some implementations, the linear activation operation is configured to enable a rectified linear unit (ReLU) 410 of a filter layer of the neural network, and the one or more scalars include 1 and 0. The electronic device 104 selects the corresponding scalar based on the sign value 516 by selecting the corresponding scalar 1 based on determining that the sign value 516 corresponds to a positive sign and selecting the corresponding scalar 0 based on determining that the sign value 516 corresponds to a negative sign.

[0149] In some implementations, the linear activation operation is configured to enable a leaky ReLU 420 of a filter layer of the neural network, and the one or more scalars include 1 and a predefined leak parameter. The electronic device 104 selects the corresponding scalar based on the sign value 516 by selecting the corresponding scalar 1 based on determining that the sign value 516 corresponds to a positive sign and selecting the corresponding scalar predefined leak parameter based on determining that the sign value 516 corresponds to a negative sign. The predefined leak parameter is less than 1.

[0150] In some implementations, the linear activation operation is configured to enable a parameter ReLU 430 of a filter layer of the neural network, and the one or more scalars include 1 and a predefined filter-based scalar. The electronic device 104 selects the corresponding scalar based on the sign value 516 by selecting the corresponding scalar 1 based on determining that the sign value 516 corresponds to a positive sign and selecting the predefined filter-based scalar based on determining that the sign value 516 corresponds to a negative sign. The predefined filter-based scalar is trained using the weights and biases of the neural network.

[0151] In some implementations, the electronic device further includes an output vector register 262 and a scaler register. The one or more input vector registers 260 are configured to store a set 512 of first elements having a total of a first number of M-bit elements. The output vector register 262 is configured to store an output element set 514 having a second number of M-bit elements, the second number being equal to the first number. The scaler register has a single M-bit element and is configured to store two N-bit scalars, from which the corresponding scalar is selected, M being equal to 2N.

[0152] In some implementations, the electronic device 104 applies a linear activation operation to the corresponding elements in the set 512 of first elements by multiplying the corresponding elements with the corresponding scalars to generate the scaled corresponding elements and adding the bias value 522 to the scaled corresponding elements to generate the biased corresponding elements. In addition, in some implementations, the corresponding elements in the set 512 of first elements have 32 bits and the scaled corresponding elements have 64 bits. The electronic device 104 adds the bias value 522 by shifting the biased corresponding elements right by 15 bits to the shifted corresponding elements and saturating the shifted corresponding elements to 32 bits to provide the corresponding elements in the set 514 of output elements.

[0153] In some implementations, the linear activation operation is implemented (1230) on the set 512 of first elements using single instruction multiple data (SIMD).

[0154] It should be understood that describing Fig.12 The specific order in which the operations are performed is exemplary only and is not intended to indicate that the described order is the only order in which the operations may be performed. A person of ordinary skill in the art will recognize various ways to reorder the operations described herein. In addition, it should be noted that the description herein of Figures 1 to 11 The details of the other processes described may also be interchangeably applied to method 1200. For the sake of brevity, these details are not repeated here.

[0155] In an example, a single instruction for quantization and a single instruction for activation are applied in a modem device to implement a neural network. The communication between the modem device and the IoT device 104F is made efficient so that a battery-powered IoT device 104F can operate in the field for an extended duration (e.g., approaching 10 years or more) and maintain low-power, wide-area connectivity in a small form factor.

[0156] Specific implementation examples are described in at least the following numbered clauses:

[0157] Clause 1. A method, the method comprising at an electronic device having one or more processors and a memory storing one or more programs, the one or more processors including a plurality of vector registers, the method comprising: receiving a single instruction for applying a neural network operation to a set of M-bit elements stored in one or more input vector registers; and in response to the single instruction, implementing the neural network operation on the set of M-bit elements to generate a set of P-bit elements, where P is less than M, the method comprising: obtaining the set of M-bit elements from the one or more input vector registers; quantizing each M-bit element in the set of M-bit elements from M bits to P bits of a corresponding element in the set of P-bit elements; and packing the set of P-bit elements into an output vector register. M and P are different positive integers.

[0158] Clause 2. A method according to clause 1, wherein the set of M-bit elements forms an input feature vector, the method further comprising: loading the input feature vector from the memory of the electronic device to the one or more input vector registers; or storing the input feature vector to the one or more input vector registers in response to one or more of the aforementioned instructions.

[0159] Clause 3. The method according to clause 1, wherein the set of M-bit elements includes a first set of M-bit elements, and the set of P-bit elements includes a first set of P-bit elements, and the method further includes: based on the size of the one or more input vector registers, dividing the input feature vector into multiple sets of M-bit elements, the multiple sets of M-bit elements including the first set of M-bit elements and one or more remaining sets of M-bit elements; in response to one or more of the aforementioned instructions, loading each set of M-bit elements from the memory of the electronic device to the corresponding input vector register or storing each set of M-bit elements to the corresponding input vector register; and repeating the single instruction to implement the neural network operation on each M-bit element in the one or more remaining sets of M-bit elements to generate a corresponding remaining set of P-bit elements.

[0160] Clause 4. According to the method described in Clause 3, repeating the single instruction to implement the neural network operation for each M-bit element in the one or more remaining sets of M-bit elements also includes, in response to each single remaining instruction: obtaining the corresponding remaining set of M-bit elements of the input feature vector from the corresponding input vector register; quantizing each M-bit element in the corresponding remaining set of M-bit elements from M bits to P bits of the corresponding element in the corresponding remaining set of P-bit elements; and packing the corresponding remaining set of P-bit elements into the corresponding output vector register.

[0161] Clause 5. According to the method described in Clause 4, the single instruction associated with the set of M-bit elements corresponds to a first instruction characteristic, and at least one single remainder instruction corresponds to a second instruction characteristic; a corresponding remainder set of P-bit elements is generated in response to the at least one single remainder instruction; and the first set of P-bit elements is packed in a first half of the output vector register, and the corresponding remainder set of P-bit elements is packed in a second half of the output vector register.

[0162] Clause 6 A method according to any one of the preceding clauses, wherein the single instruction includes a quantization factor and a zero point, and the neural network operation is a quantization operation, the quantization operation at least including a multiplication with respect to the quantization factor and an addition with respect to the zero point, the addition being after the multiplication.

[0163] Clause 7. A method according to clause 6, wherein quantizing each M-bit element in the set of M-bit elements further comprises, in sequence: converting the quantization factor into a mantissa and an integer exponent; and for each M-bit element in the set of M-bit elements: multiplying the corresponding M-bit element by the mantissa to obtain a corresponding scaling element; shifting the corresponding scaling element based on the integer exponent; discarding one or more fraction bits to obtain a corresponding M-bit integer; adding the zero point to the corresponding M-bit integer to obtain a corresponding offset integer; and saturating the corresponding offset integer into a quantized P-bit element to be stored in the output vector register.

[0164] Clause 8. A method according to clause 7, wherein the integer exponent includes a first exponent shift value and a second exponent shift value, and quantizing each M-bit element in the set of M-bit elements further includes: loading the first exponent shift value, the second exponent shift value and the zero point in a scalar register; wherein shifting the corresponding scaling element based on the integer exponent also includes, for each M-bit element in the set of M-bit elements: selecting one of the first exponent shift value and the second exponent shift value based on a sign of the corresponding M-bit element; based on determining that the selected one of the first exponent shift value and the second exponent shift value is positive, applying a rounded right shift to the corresponding scaling element based on the selected one of the first exponent shift value and the second exponent shift value; and based on determining that the selected one of the first exponent shift value and the second exponent shift value is negative, applying a saturated left shift to the corresponding scaling element based on the selected one of the first exponent shift value and the second exponent shift value.

[0165] Clause 9. A method according to any of the preceding clauses, wherein: the one or more input vector registers are configured to store a first number of M-bit elements; the output vector register is configured to store a second number of N-bit elements, the second number being less than the first number; and the output vector register is partially or completely filled with the set of P-bit elements.

[0166] Clause 10. A method according to clause 9, wherein: M is 32 and P is 8; the one or more input vector registers include a pair of 32 register elements, each having 32 bits; the output vector register includes 16 register elements, each having 32 bits; and in response to the single instruction, the output vector register is fully filled, and every four elements in the set of M-bit elements are quantized into four P-bit elements, and the four P-bit elements are packed into a single register element of 32 bits.

[0167] Clause 11. A method according to clause 9, wherein: M is 32 and P is 8; the one or more input vector registers include a pair of 32 register elements, each having 32 bits; the output vector register includes 32 register elements, each having 32 bits; and in response to the single instruction having one of two predefined characteristics, the corresponding half of the output vector register is filled.

[0168] Clause 12. A method according to any of the preceding clauses, wherein quantizing each M-bit element in the set of M-bit elements further comprises: using single instruction multiple data (SIMD) in parallel to quantize the set of M-bit elements from M bits to P bits of the corresponding elements in the set of P-bit elements.

[0169] Clause 13. A method according to any of the preceding clauses, wherein: the electronic device is a mobile device and is configured to process a neural network having an output layer; the output layer is configured to generate an input feature vector comprising a set of M-bit elements; and the output feature vector comprises the P-bit set and is quantized from the input feature vector and output by the neural network.

[0170] Clause 14. A method according to any one of clauses 1 to 12, wherein: the electronic device is a mobile device and is configured to process a neural network; the neural network includes multiple layers including at least a first layer and a second layer, the second layer is coupled to the first layer and immediately follows the first layer; the first layer is configured to generate an input feature vector including a set of M-bit elements; the output feature vector includes a set of P-bit elements; and the second layer is configured to receive and process the output feature vector after quantizing the output feature vector from the input feature vector.

[0171] Clause 15. A method according to clause 14, wherein the second layer includes multiple weights and biases, and each of the multiple weights, biases and elements of the output feature vector includes a P-bit fixed-point number, the method further comprising: dequantizing each of the multiple weights, biases and elements of the output feature vector; applying the second layer to the output feature vector using the dequantized weights, dequantized biases and dequantized elements of the output feature vector to generate a second input vector; and quantizing a subset of the second input vector in response to a second single instruction.

[0172] Clause 16. An electronic device comprising one or more processors and a memory storing one or more programs, the one or more processors including at least one or more input vector registers, wherein the electronic device is configured to implement the method according to any one of clauses 1 to 15.

[0173] Clause 17. A non-transitory computer-readable storage medium storing one or more programs, the one or more programs comprising instructions that, when executed by one or more processors including at least one or more input vector registers, cause the one or more processors to perform a method as described in any one of clauses 1 to 15.

[0174] Clause 18. An apparatus comprising means for implementing the method of any one of clauses 1 to 15. The apparatus has one or more processors and a memory storing one or more programs, and the one or more processors include a plurality of vector registers.

[0175] The above description has been provided with reference to specific implementations. However, the above illustrative discussion is not intended to be exhaustive or limited to the precise forms disclosed. In view of the above teachings, many modifications and variations are possible. Specific implementations are selected and described in order to best explain the disclosed principles and their practical applications, thereby enabling others to best utilize the present disclosure and various implementations with various modifications suitable for the specific use contemplated.

[0176] The terms used in the description of the various specific implementations described herein are only for the purpose of describing a specific specific implementation, and are not intended to be limited. As used in the description of the various specific implementations described and the attached claims, the singular forms of "one", "a kind of" and "the" are intended to also include plural forms, unless the context clearly indicates otherwise. It will also be understood that the term "and / or" as used herein refers to and covers any and all possible combinations of one or more of the associated listed items. It will be further understood that the terms "including", "comprising" specify the existence of stated features, wholes, steps, operations, elements and / or parts when used in this specification, but do not exclude the existence or addition of one or more other features, wholes, steps, operations, elements, parts and / or their groups. In addition, it will be understood that although the terms "first", "second" etc. can be used to describe various elements in this article, these elements should not be limited by these terms. These terms are only used to distinguish one element from another element.

[0177] As used herein, the term "if" is optionally interpreted to mean "when" or "at" or "in response to determining that" or "in response to detecting that" or "in accordance with determining that" depending on the context. Similarly, the phrase "if it is determined that" or "if [a stated condition or event] is detected" is optionally interpreted to mean "upon determining that" or "in response to determining that" or "upon detecting [a stated condition or event]" or "in response to detecting [a stated condition or event]" or "in accordance with determining that [a stated condition or event] is detected," depending on the context.

[0178] For the purpose of explanation, the foregoing description has been described with reference to specific implementations. However, the above illustrative discussion is not intended to be exhaustive or to limit the claims to the precise forms disclosed. In view of the above teachings, many modifications and variations are possible. Specific implementations are selected and described in order to best explain the operating principles and practical applications, thereby enabling others skilled in the art to implement.

[0179] Although the various figures illustrate multiple logical stages in a particular order, stages that are not dependent on order may be reordered and other stages may be combined or split. Although some reorderings or other groupings are specifically mentioned, other reorderings or other groupings will be apparent to one of ordinary skill in the art, and thus the orderings and groupings presented herein are not an exhaustive list of alternatives. Furthermore, it should be appreciated that the stages may be implemented in hardware, firmware, software, or any combination thereof.

Claims

1. A method implemented at an electronic device, the electronic device having one or more processors and a memory storing one or more programs, the one or more processors including a plurality of vector registers, the method comprising: receiving a single instruction for applying a neural network operation to a set of M-bit elements stored in one or more input vector registers, where M is a positive integer; as well as In response to the single instruction, implementing the neural network operation on the set of M-bit elements to generate a set of P-bit elements, where P is less than M, the method comprising: obtaining the set of M-bit elements from the one or more input vector registers; quantizing each M-bit element in the set of M-bit elements from M bits to P bits of a corresponding element in the set of P-bit elements, where P is a positive integer; and The set of P-bit elements is packed into an output vector register.

2. The method according to claim 1, wherein the set of M-bit elements forms an input feature vector, the method further comprising: loading the input feature vector from the memory of the electronic device to the one or more input vector registers; or In response to one or more of the foregoing instructions, the input feature vector is stored to the one or more input vector registers.

3. The method of claim 1 , wherein the set of M-bit elements comprises a first set of M-bit elements, and the set of P-bit elements comprises a first set of P-bit elements, the method further comprising: partitioning the input feature vector into a plurality of sets of M-bit elements based on a size of the one or more input vector registers, the plurality of sets of M-bit elements comprising the first set of M-bit elements and one or more remaining sets of M-bit elements; In response to one or more of the foregoing instructions, load each set of M-bit elements from the memory of the electronic device to a corresponding input vector register or store each set of M-bit elements to the corresponding input vector register; as well as The single instruction is repeated to implement the neural network operation on each M-bit element of the one or more remaining sets of M-bit elements to generate a corresponding remaining set of P-bit elements.

4. The method of claim 3, repeating the single instruction to implement the neural network operation for each M-bit element in the one or more remaining sets of M-bit elements further comprising, in response to each single remaining instruction: Obtaining from the corresponding input vector register a corresponding remaining set of M-bit elements of the input feature vector; quantizing each M-bit element in the corresponding remaining set of M-bit elements from M bits to P bits of a corresponding element in the corresponding remaining set of P-bit elements; and The respective remaining sets of P-bit elements are packed into respective output vector registers.

5. The method according to claim 4, wherein: said single instruction associated with said first set of M-bit elements corresponds to a first instruction signature, and at least one single remaining instruction corresponds to a second instruction signature; generating said corresponding remainder set of P-bit elements in response to said at least one single remainder instruction; and The first set of P-bit elements is packed in a first half of the output vector register, and the corresponding remaining sets of P-bit elements are packed in a second half of the output vector register.

6. The method according to claim 1, wherein the single instruction includes a quantization factor and a zero point, and the neural network operation is a quantization operation, the quantization operation at least includes a multiplication with respect to the quantization factor and an addition with respect to the zero point, the addition being after the multiplication.

7. The method of claim 6, wherein quantizing each M-bit element in the set of M-bit elements further comprises, in order: converting the quantization factor into a mantissa and an integer exponent; as well as For each M-bit element in the set of M-bit elements: Multiplying the corresponding M-bit element by the mantissa to obtain a corresponding scaling element; shifting the corresponding scaled element based on the integer exponent; discarding one or more fractional bits to obtain a corresponding M-bit integer; adding the zero point to the corresponding M-bit integer to obtain a corresponding offset integer; and The corresponding offset integer is saturated into a quantized P-bit element to be stored in the output vector register.

8. The method of claim 7, wherein the integer exponent comprises a first exponent shift value and a second exponent shift value, and quantizing each M-bit element in the set of M-bit elements further comprises: loading the first exponent shift value, the second exponent shift value, and the zero point in a scalar register; Wherein shifting the corresponding scaled element based on the integer exponent further comprises, for each M-bit element in the set of M-bit elements: selecting one of the first exponent shift value and the second exponent shift value based on the sign of the corresponding M-bit element; applying a rounded right shift to the corresponding scaled element based on a selected one of the first exponent shift value and the second exponent shift value in accordance with determining that the selected one of the first exponent value and the second exponent value is positive; as well as Based on determining that the selected one of the first exponent value and the second exponent value is negative, applying a saturating left shift to the corresponding scaled element based on the selected one of the first exponent shift value and the second exponent shift value.

9. The method according to claim 1, wherein: The one or more input vector registers are configured to store a first number of M-bit elements; The output vector register is configured to store a second number of N-bit elements, the second number being less than the first number; and The output vector register is partially or completely filled with the set of P-bit elements.

10. The method according to claim 9, wherein: M is 32, and P is 8; The one or more input vector registers include a pair of 32 register elements each having 32 bits; The output vector register comprises 16 register elements each having 32 bits; and In response to the single instruction, the output vector register is completely filled, and every four elements in the set of M-bit elements are quantized into four P-bit elements, and the four P-bit elements are packed into a single register element of 32 bits.

11. The method according to claim 9, wherein: M is 32, and P is 8; The one or more input vector registers include a pair of 32 register elements each having 32 bits; The output vector register comprises 32 register elements each having 32 bits; and In response to the single instruction having one of two predefined characteristics, a corresponding half of the output vector register is filled.

12. The method of claim 1 , wherein quantizing each M-bit element in the set of M-bit elements further comprises: The set of M-bit elements is quantized in parallel from M bits to P bits of the corresponding elements in the set of P-bit elements using single instruction multiple data (SIMD).

13. The method of claim 1, wherein: The electronic device is a mobile device and is configured to process a neural network having an output layer; The output layer is configured to generate an input feature vector comprising the set of M-bit elements; and The output feature vector includes the P bit set and is quantized from the input feature vector and output by the neural network.

14. The method of claim 1, wherein: The electronic device is a mobile device and is configured to process a neural network; The neural network includes a plurality of layers including at least a first layer and a second layer, the second layer being coupled to the first layer and immediately following the first layer; The first layer is configured to generate an input feature vector comprising the set of M-bit elements; The output feature vector comprises the set of P-bit elements; and The second layer is configured to receive and process the output feature vector after quantizing the output feature vector from the input feature vector.

15. The method of claim 14, wherein the second layer comprises a plurality of weights and biases, and each of the plurality of weights, biases, and elements of the output feature vector comprises a P-bit fixed point number, the method further comprising: dequantizing each of the plurality of weights, biases, and elements of the output feature vector; applying the second layer to the output feature vector using the dequantized weights, dequantized biases, and dequantized elements of the output feature vector to generate a second input vector; as well as In response to a second single instruction, a subset of the second input vector is quantized.

16. An electronic device, comprising one or more processors and a memory storing one or more programs, wherein the one or more processors include a plurality of vector registers, wherein the electronic device is configured to perform: receiving a single instruction for applying a neural network operation to a set of M-bit elements stored in one or more input vector registers; and In response to the single instruction, the neural network operation is implemented on the set of M-bit elements to generate a set of P-bit elements, M and P are different positive integers, where P is less than M, the method comprising: obtaining the set of M-bit elements from the one or more input vector registers; quantizing each M-bit element in the set of M-bit elements from M bits to P bits of a corresponding element in the set of P-bit elements; and The set of P-bit elements is packed into an output vector register.

17. The electronic device of claim 16, wherein the set of M-bit elements forms an input feature vector, and the electronic device is configured to load the input feature vector from the memory of the electronic device to the one or more input vector registers.

18. The electronic device of claim 16, wherein the set of M-bit elements forms an input feature vector, and the electronic device is configured to store the input feature vector to the one or more input vector registers in response to one or more of the foregoing instructions.

19. A non-transitory computer readable storage medium storing one or more programs, the one or more programs comprising instructions that, when executed by one or more processors including at least one or more input vector registers, cause the one or more processors to perform: receiving a single instruction for applying a neural network operation to a set of M-bit elements stored in one or more input vector registers; and In response to the single instruction, the neural network operation is implemented on the set of M-bit elements to generate a set of P-bit elements, where P is less than M, and M and P are different positive integers, the method comprising: obtaining the set of M-bit elements from the one or more input vector registers; quantizing each M-bit element in the set of M-bit elements from M bits to P bits of a corresponding element in the set of P-bit elements; and The set of P-bit elements is packed into an output vector register.

20. A device having one or more processors and a memory storing one or more programs, wherein the one or more processors include a plurality of vector registers, the device comprising: means for receiving a single instruction for applying a neural network operation to a set of M-bit elements stored in one or more input vector registers; as well as means for implementing the neural network operation on the set of M-bit elements to generate a set of P-bit elements in response to the single instruction, wherein P is less than M, and M and P are different positive integers, the apparatus comprising: means for obtaining said set of M-bit elements from said one or more input vector registers; means for quantizing each M-bit element in the set of M-bit elements from M bits to P bits of a corresponding element in the set of P-bit elements; and Means for packing said set of P-bit elements into an output vector register.