Systems and methods for sparse vector quantization of DEPP neural networks

Abstract

The invention relates to a sparse vector quantization (DEPP) neural network system and method. A method for generating a vector quantized deep neural network includes receiving a training dataset including one or more images, text information, sound information, training a neural model with the training dataset to adjust one or more parameters associated with one or more deep neural network layers, segmenting one or more parameters associated with each of the one or more deep neural network layers into one or more segments based on a type and a size of each of the one or more deep neural network layers, generating one or more fixed codebooks, where the fixed codebooks include a predetermined number of codewords, each of the one or more segments is replaced with one of the codewords, and in response to replacing the one or more segments with one of the codewords, a trained neural model with the one or more fixed codebooks is output.

Description

Technical Field

[0001] This disclosure relates to training deep neural networks to create vector-quantized deep neural networks. Background Technology

[0002] Deep neural networks have proven to be very powerful tools for a wide range of tasks, from image and speech recognition to natural language processing. However, these networks are often accompanied by a major challenge: a large number of parameters. Deep neural networks consist of millions or even billions of learnable parameters, which makes them require a great deal of computation and resources to train and deploy. Summary of the Invention

[0003] The first embodiment discloses a method for generating a vector quantization deep neural network, comprising: receiving a training dataset including one or more image, text, and sound information; training a neural model with the training dataset to adjust one or more parameters associated with one or more deep neural network layers; dividing one or more parameters associated with each of the one or more deep neural network layers into one or more segments based on the type and size of each of the one or more deep neural network layers; generating one or more fixed codebooks, wherein the fixed codebooks include a predetermined number of codewords; replacing each of the one or more segments with one of the codewords; and outputting a trained neural model utilizing the one or more fixed codebooks in response to replacing the one or more segments with one of the codewords.

[0004] A second embodiment discloses an apparatus for generating a vector quantized deep neural network, comprising one or more processors configured to receive a training dataset comprising one or more images, train a neural model with the training dataset to generate a first layer having weighted parameters, segment one or more parameters associated with each of the one or more deep neural network layers into one or more segments based on the type and size of each of the one or more deep neural network layers, generate one or more fixed codebooks, wherein the fixed codebooks include a predetermined number of codewords, replace each of the one or more segments with one of the codewords, and in response to replacing the one or more segments with one of the codewords, output a trained neural model utilizing the one or more fixed codebooks.

[0005] A third embodiment discloses a system for generating a vector quantized deep neural network, comprising one or more processors configured to receive a training dataset comprising one or more images, train a neural model with the training dataset to generate a first layer having weighted parameters, segment one or more parameters associated with each of the one or more deep neural network layers into one or more segments based on the type and size of each of the one or more deep neural network layers, generate one or more fixed codebooks, wherein the fixed codebooks include a predetermined number of codewords, replace each of the one or more segments with one of the codewords, and in response to replacing the one or more segments with one of the codewords, output a trained neural model utilizing the one or more fixed codebooks. Attached Figure Description

[0006] Figure 1 A system for training a neural network according to one embodiment is shown.

[0007] Figure 2 A computer-implemented method for training and utilizing a neural network, according to one embodiment, is shown.

[0008] Figure 3 The figure shows a flowchart of an example process 300 for generating a vector quantization machine learning model according to one embodiment.

[0009] Figure 4 The figure illustrates a flowchart of an example process 400 for generating a vector quantization machine learning model according to one embodiment, in which multiple layers are replaced with a global codebook.

[0010] Figure 5 A schematic diagram depicts the interaction between a computer-controlled machine and a control system according to one embodiment.

[0011] Figure 6 A description is provided of a device configured to control a vehicle according to one embodiment. Figure 5 A schematic diagram of the control system, wherein the vehicle can be a partially autonomous vehicle, a fully autonomous vehicle, a partially autonomous robot, or a fully autonomous robot.

[0012] Figure 7 Depicting Figure 5 A schematic diagram of a control system configured to control manufacturing machines, such as stamping presses, cutting machines, or gun drills, of a manufacturing system (e.g., a part of a production line).

[0013] Figure 8 Depicting Figure 5 A schematic diagram of a control system configured to control power tools, such as a drill or drive with at least a partially autonomous mode.

[0014] Figure 9 Depicting a person configured to control an automated personal assistant Figure 5 A schematic diagram of the control system.

[0015] Figure 10 Depicting Figure 5 A schematic diagram of a control system configured to control a monitoring system, such as a control access system or a monitoring system.

[0016] Figure 11 Depicting Figure 5 A schematic diagram of a control system configured to control an imaging system, such as an MRI device, an X-ray imaging device, or an ultrasound device.

[0017] Figure 12 The figure illustrates an example vectorization of a quantized linear layer according to one embodiment. Detailed Implementation

[0018] This document describes embodiments of the present disclosure. However, it should be understood that the disclosed embodiments are merely examples, and other embodiments may take various alternative forms. The drawings are not necessarily drawn to scale; some features may be enlarged or reduced to show details of specific components. Therefore, the specific structural and functional details disclosed herein should not be construed as limiting, but merely as a representative basis for teaching those skilled in the art to use the embodiments in various ways. As will be understood by those skilled in the art, various features illustrated and described with reference to any of the drawings may be combined with features illustrated in one or more other drawings to produce embodiments that are not explicitly illustrated or described. The combinations of illustrated features provide representative embodiments of typical applications. However, for a particular application or implementation, various combinations and modifications of features consistent with the teachings of this disclosure may be required.

[0019] The terms “a,” “one,” and “the” used here refer to the singular and plural, respectively, unless the context clearly indicates otherwise. For example, a “processor” programmed to perform various functions refers to one processor programmed to perform each of those functions, or more than one processor programmed together to perform each of those functions.

[0020] In one embodiment, a system and method introduces an innovative approach to address the challenge of high parameter counts in deep neural networks. In one embodiment, the method may employ the concept of vector quantization, a data compression technique to reduce the number of parameters used in deep neural networks. By implementing vector quantization, the system can aim to retain key information from the original parameters while significantly reducing their total number.

[0021] This system and method can operate similarly to vector-quantized deep neural networks, with one difference being the use of a fixed codebook during training. This codebook contains only the values ​​0, 1, and -1, which effectively eliminates floating-point operations.

[0022] Referring now to the embodiments illustrated in the accompanying drawings, these embodiments can be applied to machine learning models or neural networks. Figure 1 A system 100 for training a neural network (e.g., a deep neural network) is illustrated. System 100 may include an input interface for accessing training data 102 of the neural network, wherein the training data may include text, audio, bitmap images, vector images, video, or any suitable data object. For example, as... Figure 1 As illustrated, the input interface can be comprised of a data storage interface 104, which can access the training data 102 from the data storage device 106. For example, the data storage interface 104 can be a memory interface or a permanent storage interface, such as a hard disk or SSD interface, and can also be a personal, local area network (LAN), or wide area network (WAN) interface, such as a Bluetooth, Zigbee, or Wi-Fi interface, or an Ethernet or fiber optic interface. The data storage device 106 can be the internal data storage of the system 100, such as a hard disk drive or SSD, and can also be external data storage, such as a network-accessible data storage device.

[0023] In some embodiments, data memory 106 may further include a data representation 108 of an untrained version of the neural network, which can be accessed by system 100 from data memory 106. However, it should be understood that the training data 102 and data representation 108 of the untrained neural network may also be accessed from different data memories, for example, via different subsystems of data storage interface 104. Each subsystem may be of the type described above for data storage interface 104. In other embodiments, the data representation 108 of the untrained neural network may be generated internally by system 100 based on the design parameters of the neural network, and therefore may not be explicitly stored on data memory 106. System 100 may also include a processor subsystem 110, which may be configured to provide an iterative function as a replacement for the layers of the neural network to be trained during operation of system 100. Here, the layers of the replaced layers may have mutually shared weights and may receive initial activations and a portion of the layer's input as the output of the previous layer or as the input of the first layer of the layer. Processor subsystem 110 may be further configured to iteratively train the neural network using training data 102. Here, the training iterations of processor subsystem 110 may include a forward propagation portion and a backward propagation portion. Processor subsystem 110 may be configured to, in addition to other operations defining the executable forward propagation portion, determine an equilibrium point of the iterative function at which the iterative function converges to a fixed point, wherein determining the equilibrium point includes using a numerical root-finding algorithm to find the root solution of the iterative function minus its input, and to perform the forward propagation portion by providing the equilibrium point as an alternative to the output of the layers in the neural network. System 100 may also include an output interface for outputting a data representation 112 of the trained neural network, which may also be referred to as trained model data 112. For example, as... Figure 1 As illustrated, the output interface can be comprised of a data storage interface 104, which in these embodiments is an input / output (“IO”) interface through which trained model data 112 can be stored in data memory 106. For example, the data representation 108 defining an “untrained” neural network can be at least partially replaced by the data representation 112 of a trained neural network during or after training, because the parameters of the neural network, such as the network weights, hyperparameters, and other types of parameters, can be adapted to reflect training on the training data 102. This also... Figure 1 The figures are illustrated by reference numerals 108 and 112, which refer to the same data records on data storage 106. In other embodiments, data representation 112 may be stored separately from data representation 108 defining the "untrained" neural network. In some embodiments, the output interface may be separate from data storage interface 104, but it can generally be of the type described above for data storage interface 104.

[0024] The structure of System 100 is an example of a system that can be used to train the deep neural model described herein. Figure 2 Additional structures for manipulating and training deep neural models are shown.

[0025] Figure 2 A system 200 is depicted that implements the machine learning models described herein, such as the deep neural machine learning models described herein. System 200 can be implemented to perform the image quantization process described herein. System 200 may include at least one computing system 202. Computing system 202 may include at least one processor 204 operatively connected to memory unit 208. Processor 204 may include one or more integrated circuits implementing the functions of a central processing unit (CPU) 206. CPU 206 may be a commercially available processing unit implementing an instruction set such as x86, ARM, Power, or MIPS instruction set families. During operation, CPU 206 may execute stored program instructions retrieved from memory unit 208. The stored program instructions may include software controlling the operation of CPU 206 to perform the operations described herein. In some examples, processor 204 may be a system-on-a-chip (SoC) that integrates the functions of CPU 206, memory unit 208, network interface, and input / output interface into a single integrated device. Computing system 202 may implement an operating system for managing various aspects of operation. Figure 2 The diagram shows a processor 204, a CPU 206, and a memory 208, but of course, more than one of each can be used in the whole system.

[0026] Memory cell 208 may include volatile and non-volatile memory for storing instructions and data. Non-volatile memory may include solid-state memory, such as NAND flash memory, magnetic and optical storage media, or any other suitable data storage device that retains data when the computing system 202 is disabled or loses power. Volatile memory may include static and dynamic random access memory (RAM) for storing program instructions and data. For example, memory cell 208 may store a machine learning model 210 or algorithm, a training dataset 212 for the machine learning model 210, and the original source dataset 216.

[0027] The computing system 202 may include a network interface device 222 configured to provide communication with external systems and devices. For example, the network interface device 222 may include wired and / or wireless Ethernet interfaces defined by the Institute of Electrical and Electronics Engineers (IEEE) 802.11 standards family. The network interface device 222 may include a cellular communication interface for communicating with cellular networks (e.g., 3G, 4G, 5G). The network interface device 222 may also be configured to provide a communication interface to an external network 224 or the cloud.

[0028] External network 224 may be referred to as the World Wide Web or the Internet. External network 224 can establish standard communication protocols between computing devices. External network 224 can allow information and data to be easily exchanged between computing devices and the network. One or more servers 230 can communicate with external network 224.

[0029] The computing system 202 may include an input / output (I / O) interface 220, which may be configured to provide digital and / or analog inputs and outputs. The I / O interface 220 is used to transfer information between internal memory and external input and / or output devices (e.g., HMI devices). The I / O interface 220 may include associated circuitry or bus networks for transferring information to or between one or more processors and memory. For example, the I / O interface 220 may include digital I / O logic lines that can be read or set by one or more processors, handshake lines for monitoring data transfers via the I / O lines, timing and counting devices, and other known structures that provide these functions. Examples of input devices include keyboards, mice, sensors, etc. Examples of output devices include monitors, printers, speakers, etc. The I / O interface 220 may include additional serial interfaces (e.g., Universal Serial Bus (USB) interfaces) for communicating with external devices.

[0030] The computing system 202 may include a human-machine interface (HMI) device 218, which may include any device that enables the system 200 to receive control input. Examples of input devices may include HMI inputs such as a keyboard, mouse, touchscreen, voice input device, and other similar devices. The computing system 202 may include a display device 232. The computing system 202 may include hardware and software for outputting graphical and textual information to the display device 232. The display device 232 may include an electronic display screen, projector, printer, or other suitable device for displaying information to a user or operator. The computing system 202 may also be configured to allow interaction with remote HMIs and remote display devices via a network interface device 222.

[0031] System 200 can be implemented using one or more computing systems. While this example depicts a single computing system 202 implementing all the described features, the intention is that various features and functions can be decoupled and implemented by multiple computing units that communicate with each other. The specific system architecture chosen may depend on a variety of factors.

[0032] System 200 can implement machine learning algorithm 210, which is configured to analyze raw source dataset 216. Raw source dataset 216 may include raw or unprocessed sensor data, which may represent the input dataset of the machine learning system. Raw source dataset 216 may include video, video clips, images, text-based information, audio or human speech, time-series data (e.g., pressure sensor signals varying over time), and raw or partially processed sensor data (e.g., radar images of objects). (See reference...) Figure 5-11 Several different examples of inputs are shown and described. In some examples, machine learning algorithm 210 may be a neural network algorithm (e.g., a deep neural network) designed to perform a predetermined function. For example, a neural network algorithm may be configured in an automotive application to identify street signs or pedestrians in an image. One or more machine learning algorithms 210 may include algorithms configured to operate the deep neural network machine learning model described herein.

[0033] Computer system 200 may store a training dataset 212 for machine learning algorithm 210. Training dataset 212 may represent a set of previously constructed data used to train machine learning algorithm 210. Machine learning algorithm 210 can use training dataset 212 to learn weighting factors associated with a neural network algorithm. Training dataset 212 may include a set of source data that has the corresponding results or outcomes that machine learning algorithm 210 attempts to replicate through the learning process. In this example, training dataset 212 may include input images containing objects (e.g., street signs). Input images may include various scenes for object identification.

[0034] Machine learning algorithm 210 can operate in learning mode using training dataset 212 as input. Machine learning algorithm 210 can perform multiple iterations using data from training dataset 212. With each iteration, machine learning algorithm 210 can update its internal weighting factors based on the obtained results. For example, machine learning algorithm 210 can compare its output (e.g., a reconstructed or supplemented image in the case of image data as input) with those results included in training dataset 212. Since training dataset 212 includes expected results, machine learning algorithm 210 can determine when performance is acceptable. After machine learning algorithm 210 reaches a predetermined performance level (e.g., 100% consistency with results associated with training dataset 212) or converges, machine learning algorithm 210 can be performed using data not in training dataset 212. It should be understood that in this disclosure, "convergence" can mean that a set (e.g., predetermined) number of iterations has occurred, or the residual is sufficiently small (e.g., the change in approximate probability during iteration is less than a threshold), or other convergence conditions. The trained machine learning algorithm 210 can be applied to new datasets to generate annotated data.

[0035] Machine learning algorithm 210 can be configured to identify specific features in raw source data 216. Raw source data 216 may include multiple instances or input datasets that require supplementary results. For example, machine learning algorithm 210 can be configured to identify the presence of road signs in video images and annotate their occurrence. Machine learning algorithm 210 can be programmed to process raw source data 216 to identify the presence of specific features. Machine learning algorithm 210 can be configured to identify features in raw source data 216 as predetermined features (e.g., road signs). Raw source data 216 can be obtained from various sources. For example, raw source data 216 can be actual input data collected by a machine learning system. Raw source data 216 can be machine-generated for testing the system. As an example, raw source data 216 may include raw video images from a camera.

[0036] In one example, the original source data 216 may include image data representing an image. Applying the machine learning algorithms described herein (e.g., deep neural models), the output may be a quantized version of the input image.

[0037] Given the above description of the machine learning model, and the model configured to implement it... Figure 1-2 Structural examples, Figure 3 The figure illustrates a flowchart of an example process 300 for generating a machine learning model using vector quantization, as described herein. In some embodiments, processor 204 may execute one or more process blocks of process 300.

[0038] Given the above description of the machine learning model, and the model configured to implement it... Figure 1-2 Structural examples, Figure 3 The figure illustrates a flowchart of an example process 300 for generating a vector quantization machine learning model with a fixed codebook, as described herein. In some implementations, processor 204 may execute one or more process blocks of process 300.

[0039] At 302, process 300 may receive a training dataset comprising one or more images. In some embodiments, processor 204 may receive training data as audio files, video files, text files, digital files, bitmap image files, vector image files, sensor output signals, and / or any suitable data object. In some embodiments, the training data may be annotated with one or more labels describing the training data or a portion thereof. For example, the training data may be vector images of animals, wherein each image includes a label describing the animal in the image. In some embodiments, the images may contain two or more animals in the training images and may have two or more labels proportionally, wherein each label will have a description of the animal and the location on the image associated with that animal.

[0040] At 304, process 300 can train a neural model using a training dataset to generate a first layer with weighted parameters. In some embodiments, processor 204 can train the neural model using the training dataset to identify inputs based on the training. For example, processor 204 can provide input of unlabeled images for the neural model to analyze. In some embodiments, processor 204 analyzes the input image based on the weighted parameters generated during the training of the neural model. In some embodiments, processor 204 can output result labels for images with confidence scores higher than a predetermined threshold.

[0041] At 306, process 300 may, during the training phase but after the training data has been analyzed, divide the first layer into a first predetermined number of segments based on the fact that the first layer is a first type of layer. For example, based on the fact that the first layer is a linear layer, processor 204 may divide the first layer into 8 segments. In some embodiments, the first layer is divided into 8 non-overlapping segments, wherein the number of weighting parameters is equally distributed in these segments.

[0042] Process 300 can be configured to create a codebook, which is a collection of codewords representing clusters of similar data points. In such embodiments, the codebook may be a fixed codebook. Process 300 may partition the data space into non-overlapping regions, each associated with a specific codeword from the codebook. In some embodiments, during encoding, each data vector (e.g., model parameters) may be mapped to the nearest codeword, which can effectively quantize the data. In some embodiments, during decoding, the original data can be reconstructed using the corresponding codeword from the codebook.

[0043] The codebook construction may include, assuming, M codewords of size B, where M is twice B. The first B codewords may consist of a single 1 in one bit position, with the remaining bits set to zero. The second set of B codewords mirrors the first set, but replaces the 1s with -1s. For example, for M=8 and B=4, the codebook would look like this: |1|0|0|0| |0|1|0|0| |0|0|1|0| |0|0|0|1| |-1|0|0|0| |0|-1|0|0| |0|0|-1|0| |0|0|0|-1| In linear and convolutional layers, applying codewords to an input block requires B multiplications and B-1 additions. However, in the embodiments disclosed below, multiplication and addition are unnecessary because there is only one non-zero term in the codeword. Instead, the system and method can directly pass the corresponding input value of 1 and negate and pass the input value of -1.

[0044] At 308, process 300 can generate a fixed codebook by replacing the weighting parameters in each segment of the first layer with codewords based on finding a representative vector in a vectorized dictionary that is most closely related to the weighting parameters of each segment. The codebook includes multiple codewords equal to a first predetermined number of segments, and each codeword includes a representative vector. For example, processor 204 can generate the codebook by replacing the weighting parameters in each segment of the first layer with codewords based on finding a representative vector in a vectorized dictionary that is most closely related to the weighting parameters of each segment. The vectorized dictionary is a pre-generated codeword table corresponding to one or more parameters. In some embodiments, the layer can be recreated by reverse-searching for codewords in the vectorized dictionary and replacing the codewords with weighting parameters.

[0045] Procedure 300 can be configured to introduce vector quantization into deep neural networks by partitioning the parameters of each layer into non-overlapping regions tailored to the layer type. For 2D convolutional layers, procedure 300 can partition the weight matrix along the channel input and channel output dimensions. For example, the weight matrix may have the shape (Cout, Cin, K, K). Procedure 300 can partition the weight matrix into Cout * Cin / L regions of size K * K * L.

[0046] For example, process 300 can employ linear layers, dividing the weights into groups of L parameters, where L is a hyperparameter. To maintain compatibility with the single codebook of all linear layers, a value of L is chosen such that all linear layer sizes are divisible by L. For example, L=8 works well for most linear layer sizes. When the weight matrix of a linear layer has the shape (Nin, Nout), the systems and methods described herein divide the weight matrix into Nout * Nin / L partitions of size L.

[0047] Process 300 can determine the size of each codebook. For example, E can be used as a hyperparameter to define the number of codewords in each codebook. Hyperparameters can be tuned to optimize model performance. Furthermore, E can vary for each type of layer, but the system and method described in this paper can be configured to utilize E=16, which provides excellent results in a wide range of use cases.

[0048] In another example of the exemplary embodiment, the first layer may have 512 weighted parameters. In some embodiments, the first layer may be divided into a predetermined number, such as 8, based on the type of the layer. In some embodiments, dividing the first layer with 512 weighted parameters by 8 generates 64 segments, each segment having 8 weighted parameters. In some embodiments, the 8 weighted parameters for each segment can be looked up in a vectorized dictionary. The weighted parameters in that segment are replaced with the codewords closest to these 8 weighted parameters. In some embodiments, the first layer with 512 weighted parameters is replaced by a codebook with 64 codewords.

[0049] At 310, process 300 can respond to updating the neural model with the codebook, outputting a trained neural model that replaces the codebook of the first layer. For example, processor 204 can output a trained neural model that is smaller in memory and can be operated with fewer resources in terms of processing.

[0050] although Figure 3 An example block diagram of process 300 is shown, but in some implementations, process 300 may include more than Figure 3The boxes depicted may include additional boxes, fewer boxes, different boxes, or boxes arranged differently. Additionally or alternatively, two or more boxes of process 300 may be executed in parallel.

[0051] Figure 4 This is a flowchart of an example process 400 for generating a machine learning model with vector quantization, in which multiple layers are replaced with a fixed codebook, as described herein. In some embodiments, processor 204 may execute one or more process blocks of process 400. In step 401, the system may receive a training dataset comprising data. The data can be any type of data, such as image data, text information, sound information, radar, lidar, etc. In some embodiments, the training dataset may be referred to as input data.

[0052] In step 403, the system can train a neural network model using the training dataset. The system can adjust the parameters associated with each layer of the network based on the classification of the training dataset. These parameters can be updated using the training dataset until a certain number of iterations or a certain convergence threshold is reached.

[0053] At 405, process 400 can segment one or more parameters associated with each of the one or more deep neural network layers into one or more segments based on the type and size of each of the one or more deep neural network layers. For example, for a 2D convolutional layer, the processor can partition the weight matrix along the channel input and channel output dimensions. In some embodiments, the weight matrix has a shape of (Cout, Cin, K, K). In some embodiments, the processor can partition the weight matrix into Cout * Cin / L regions of size K * K * L. This partitioning strategy may be effective for modern deep neural networks, as they often utilize 1×1 and 3×3 2D convolutional layers.

[0054] In step 407, the system and process may generate one or more fixed codebooks. The fixed codebook may include a predetermined number of codewords. The codebook may contain only 0, 1, and -1, effectively eliminating floating-point operations. Codebook construction involves assuming M codewords of size B, where M is equal to twice B.

[0055] In step 409, the system and process can replace each of one or more segments with one of the codewords. The system can update the fixed codebook by replacing the weighted parameters in each segment of the second layer with codewords based on finding a representative vector in the vectorized dictionary that is most closely related to the weighted parameters of each segment.

[0056] In step 411, a trained neural model utilizing one or more fixed codebooks is output. The trained model can be output in response to replacing one or more segments with codewords. In one example, the codebook can be used to replace the first and second layers. For example, processor 204 can remove the second layer from the neural model because the weight parameters of the second layer have been incorporated into the codebook.

[0057] In another example, process 400 can employ linear layers, dividing the weights into groups of L parameters, where L is a hyperparameter. To maintain compatibility with a single codebook used for all linear layers, a value of L can be chosen such that all linear layer sizes are divisible by L. For example, L=8 works well for most linear layer sizes. When the weight matrix of a linear layer has a shape of (Nin, Nout), processor 204 can divide the weight matrix into Nout * Nin / L partitions of size L.

[0058] although Figure 4 An example block diagram of process 400 is shown, but in some implementations, process 400 may include more than Figure 4 The additional boxes depicted, fewer boxes, different boxes, or boxes arranged differently. Additionally or alternatively, two or more boxes of process 400 can be executed in parallel.

[0059] In a linear layer, the input can be an image or any other type of input described. The next layer can be the activation of the previous layer. The input can be multiplied by the entire codebook (fixed codebook). The result is constructed from a multiplication lookup table. For an index lookup table, the index matrix will show the indexes of the blocks of the weight matrix that will come from each entry in the codebook. Thus, the parameters can be derived from the entries in the fixed codebook. Based on the indexes and the multiplication table, the system can construct the output. The multiplication table will include the result of multiplying the entire codebook by the inputs. For each neuron in the network (e.g., 128 neurons) and the layer input is 64, the first neuron will have 64 inputs, and these 64 inputs will be divided into 8 parameters or 8 indices. The system can multiply the fixed codebook by each input of each layer. The final output can then be sent to the next layer until the last layer is satisfied. When sent to the last layer, the output of the last layer can indicate the final output. In one embodiment, the network may have already been trained when the fixed codebook is applied. Once it has been trained, the system can efficiently proliferate.

[0060] In training models using vector quantization, the output of the final layer can vary depending on the specific application and the model architecture. In one example, the final output could be a compressed representation. In vector quantization used for data compression, the final layer might output a sequence of indices corresponding to codewords in a codebook. Each index can represent the codeword closest to the input vector, efficiently encoding the input data in a compressed form.

[0061] In another example, the reconstructed data can be output. In applications where the goal is to reconstruct the input data from its quantized representation (e.g., in an autoencoder with vector quantization), the final layer might output the reconstructed data. This reconstructed data is obtained by mapping the indices back to their corresponding codewords in the codebook and combining them to form the output.

[0062] In yet another example, the final output may include a feature representation. In feature learning or embedding tasks, the final layer may output a feature vector representing the input data in a new space, which can be defined or utilized using codewords. This can be useful for image recognition, where the feature representation can be used for classification or other tasks.

[0063] In one example, the final output could be a cluster assignment. In a clustering context (such as k-means clustering), the final layer might output a cluster assignment for each data point. This means that each input vector is assigned to the nearest cluster centroid (codeword).

[0064] The following is the pseudocode for a sparse vector quantization linear layer.

[0065] SparseVQLinearLayer(input, index_matrix, N in , N out B): # Input: # Input: A sequence of dimensions N in Input data # Index matrix: has dimension N in / B * N out The index matrix (integers) # B: Integer = Codebook size / 2 # Output output = zeros(N out ) multiplication_lookup_table = BuildMultiplicationLookupTable (input,B) for i = 0 to N out : forj = 0 to (N in / B): output[i] += multiplication_lookup_table [j, index_matrix[j,i]] returnoutput FunctionBuildMultiplicationLookupTable (input, B): MLT = zeros(N in / B, 2*B) for i = 0 to (N in / B): forj = 0 to B: MLT[i, j] = input[i*B+j] MLT[i, B+j] = - input[i*B+j] returnMLT.

[0066] The machine learning model described in this paper can be used in many different applications, not just in the context of road sign image processing. Figure 6-11 This illustrates another application where image quantization can be used. Figure 5 The diagram illustrates the structure of machine learning models used for training and using these applications (and other applications). Figure 5 A schematic diagram depicting the interaction between a computer-controlled machine 500 and a control system 502 is shown. The computer-controlled machine 500 includes actuators 504 and sensors 506. Actuators 504 may include one or more actuators, and sensors 506 may include one or more sensors. Sensors 506 are configured to sense the condition of the computer-controlled machine 500. Sensors 506 may be configured to encode the sensed condition into a sensor signal 508 and transmit the sensor signal 508 to the control system 502. Non-limiting examples of sensors 506 include video, radar, lidar, ultrasonic, and motion sensors. In one embodiment, sensor 506 is an optical sensor configured to sense an optical image of the environment approaching the computer-controlled machine 500.

[0067] The control system 502 is configured to receive sensor signals 508 from the computer-controlled machine 500. As described below, the control system 502 can also be configured to calculate actuator control commands 510 based on the sensor signals and transmit the actuator control commands 510 to the actuator 504 of the computer-controlled machine 500.

[0068] like Figure 5 As shown, the control system 502 includes a receiving unit 512. The receiving unit 512 can be configured to receive sensor signals 508 from sensor 506 and convert the sensor signals 508 into input signals x. In an alternative embodiment, the sensor signals 508 are received directly as input signals x, without a receiving unit 512. Each input signal x can be a part of each sensor signal 508. The receiving unit 512 can be configured to process each sensor signal 508 to generate each input signal x. The input signals x can include data corresponding to the image recorded by sensor 506.

[0069] The control system 502 includes a classifier 514. The classifier 514 can be configured to classify the input signal x into one or more labels using a machine learning (ML) algorithm, such as the neural network described above. The classifier 514 is configured to be parameterized by parameters, such as those described above (e.g., parameter θ). The parameter θ can be stored in and provided by non-volatile memory 516. The classifier 514 is configured to determine an output signal y from the input signal x. Each output signal y includes information assigning one or more labels to each input signal x. The classifier 514 can transmit the output signal y to a conversion unit 518. The conversion unit 518 is configured to convert the output signal y into an actuator control command 510. The control system 502 is configured to transmit the actuator control command 510 to an actuator 504, which is configured to actuate the computer-controlled machine 500 in response to the actuator control command 510. In another embodiment, actuator 504 is configured to actuate computer-controlled machine 500 directly based on output signal y.

[0070] Once actuator 504 receives actuator control command 510, actuator 504 is configured to perform an action corresponding to the relevant actuator control command 510. Actuator 504 may include control logic configured to translate actuator control command 510 into a second actuator control command for controlling actuator 504. In one or more embodiments, actuator control command 510 may be used to control a display instead of an actuator or in addition to an actuator.

[0071] In another embodiment, instead of the computer-controlled machine 500, the control system 502 includes the sensor 506, or in addition to the computer-controlled machine 500 including the sensor 506. Instead of the computer-controlled machine 500, the control system 502 may include the actuator 504, or in addition to the computer-controlled machine 500 including the actuator 504.

[0072] like Figure 5As shown, the control system 502 also includes a processor 520 and a memory 522, wherein the processor 204 is interchangeable with the processor 520. The processor 520 may include one or more processors. The memory 522 may include one or more memory devices. A classifier 514 of one or more embodiments (e.g., machine learning algorithms, such as those described above with respect to pre-trained classifier 306) may be implemented by the control system 502, which includes non-volatile memory 516, processor 520, and memory 522.

[0073] Non-volatile memory 516 may include one or more permanent data storage devices, such as hard disk drives, optical disk drives, magnetic tape drives, non-volatile solid-state devices, cloud storage, or any other device capable of permanently storing information. Processor 520 may include one or more devices selected from a high-performance computing (HPC) system, including high-performance cores, microprocessors, microcontrollers, digital signal processors, microcomputers, central processing units, field-programmable gate arrays, programmable logic devices, state machines, logic circuits, analog circuits, digital circuits, or any other device that manipulates signals (analog or digital) based on computer-executable instructions residing in memory 522. Memory 522 may include a single memory device or multiple memory devices, including but not limited to random access memory (RAM), volatile memory, non-volatile memory, static random access memory (SRAM), dynamic random access memory (DRAM), flash memory, cache memory, or any other device capable of storing information.

[0074] Processor 520 can be configured to read computer-executable instructions into memory 522 and execute the computer-executable instructions, which reside in non-volatile memory 516 and embody one or more ML algorithms and / or methods of one or more embodiments. Non-volatile memory 516 may include one or more operating systems and applications. Non-volatile memory 516 may store programs compiled and / or interpreted from computer programs created using various programming languages ​​and / or technologies, including but not limited to Java, C, C++, C#, Objective C, Fortran, Pascal, JavaScript, Python, Perl, and PL / SQL, individually or in combination.

[0075] When executed by processor 520, the computer-executable instructions in non-volatile memory 516 can cause control system 502 to implement one or more ML algorithms and / or methods disclosed herein. Non-volatile memory 516 may also include ML data (including data parameters) supporting the functionality, features, and processes of one or more embodiments described herein.

[0076] Program code embodying the algorithms and / or methods described herein can be distributed individually or collectively as a program product in a variety of different forms. The program code can be distributed using a computer-readable storage medium having computer-readable program instructions thereon for causing a processor to execute aspects of one or more embodiments. Essentially non-transitory computer-readable storage media can include tangible media, implemented in any method or technology, that are volatile and non-volatile, as well as removable and non-removable, for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer-readable storage media can also include RAM, ROM, erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other solid-state memory technologies, portable optical disc read-only memory (CD-ROM) or other optical storage, magnetic tape cassettes, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and can be read by a computer. Computer-readable program instructions can be downloaded from the computer-readable storage medium to a computer, another type of programmable data processing apparatus, or another device, or downloaded via a network to an external computer or external storage device.

[0077] Computer-readable program instructions stored in a computer-readable medium can be used to direct a computer, other type of programmable data processing apparatus, or other device to operate in a particular manner, causing the instructions stored in the computer-readable medium to produce an article of art including instructions that implement the functions, actions, and / or operations specified in a flowchart or diagram. In some alternative embodiments, according to one or more embodiments, the functions, actions, and / or operations specified in the flowchart and diagram can be reordered, processed sequentially, and / or processed concurrently. Furthermore, any flowchart and / or diagram may include more or fewer nodes or blocks than illustrated according to one or more embodiments.

[0078] These processes, methods, or algorithms can be implemented, in whole or in part, using suitable hardware components such as application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), state machines, controllers, or other hardware components or devices, or a combination of hardware, software, and firmware components.

[0079] Figure 6A schematic diagram of a control system 502 configured to control a vehicle 600, which may be at least partially autonomous or at least partially autonomous, or a robot, is depicted. The vehicle 600 includes actuators 504 and sensors 506. Sensors 506 may include one or more video sensors, cameras, radar sensors, ultrasonic sensors, lidar sensors, and / or position sensors (e.g., GPS). One or more of these specific sensors may be integrated into the vehicle 600. In the context of sign recognition and processing as described herein, sensor 506 is a camera mounted to or integrated into the vehicle 600. Alternatively, or in addition to the one or more specific sensors identified above, sensor 506 may include a software module configured to determine the state of actuator 504 upon execution. A non-limiting example of the software module includes a weather information software module configured to determine the current or future state of weather near the vehicle 600 or other locations.

[0080] The classifier 514 of the control system 502 of the vehicle 600 can be configured to detect objects near the vehicle 600 based on the input signal x. In such an embodiment, the output signal y may include information characterizing objects near the vehicle 600. An actuator control command 510 can be determined based on this information. The actuator control command 510 can be used to avoid collisions with the detected objects.

[0081] In embodiments where vehicle 600 is at least partially automated, actuator 504 may be embodied in the brakes, propulsion system, engine, transmission system, or steering mechanism of vehicle 600. Actuator control command 510 can be determined to control actuator 504, causing vehicle 600 to avoid collisions with detected objects. Detected objects can also be classified according to what classifier 514 deems them most likely to be, such as pedestrians or trees. Actuator control command 510 can be determined based on the classification. In the event of a potential hostile attack, the system described above can be further trained to better detect objects or recognize changes in lighting conditions or angles of sensors or cameras on vehicle 600.

[0082] In other embodiments where vehicle 600 is at least partially autonomous, vehicle 600 may be a mobile robot configured to perform one or more functions, such as flying, swimming, diving, and walking. The mobile robot may be at least partially autonomous lawnmower or at least partially autonomous cleaning robot. In such embodiments, actuator control commands 510 may be determined such that the propulsion unit, steering unit, and / or braking unit of the mobile robot can be controlled, enabling the mobile robot to avoid collisions with identified objects.

[0083] In another embodiment, vehicle 600 is at least partially autonomous in the form of a gardening robot. In such an embodiment, vehicle 600 may use an optical sensor, sensor 506, to determine the state of plants in the environment near vehicle 600. Actuator 504 may be a nozzle configured to spray chemicals. Based on the species and / or state of the identified plants, actuator control command 510 may be determined to cause actuator 504 to spray an appropriate amount of appropriate chemical onto the plants.

[0084] Vehicle 600 may be a partially autonomous robot in the form of a household appliance. Non-limiting examples of household appliances include washing machines, stoves, ovens, microwave ovens, or dishwashers. In such a vehicle 600, sensor 506 may be an optical sensor configured to detect the state of an object to be processed by the household appliance. For example, in the case of a washing machine, sensor 506 may detect the state of the clothes inside the washing machine. Actuator control command 510 may be determined based on the detected state of the clothes.

[0085] Figure 7 A schematic diagram of a control system 502 is depicted, which is configured to control a system 700 (e.g., a manufacturing machine), such as a punch press, cutting machine, or gun drill, which is part of a manufacturing system 702, such as a production line. The control system 502 may be configured to control an actuator 504, which is configured to control the control system 700 (e.g., the manufacturing machine).

[0086] Sensor 506 of system 700 (e.g., manufacturing machine) may be an optical sensor configured to capture one or more characteristics of manufactured product 704. Classifier 514 may be configured to determine the state of manufactured product 704 based on one or more captured characteristics. Actuator 504 may be configured to control system 700 (e.g., manufacturing machine) based on the determined state of manufactured product 704 for subsequent manufacturing steps of manufactured product 704. Actuator 504 may be configured to control system 700 (e.g., manufacturing machine) on subsequent manufactured product 706 of system 700 (e.g., manufacturing machine) based on the determined state of manufactured product 704.

[0087] Figure 8 A schematic diagram of a control system 502 is depicted, which is configured to control a power tool 800, such as a drill or actuator, having at least a partially autonomous mode. The control system 502 may be configured to control an actuator 504, which is configured to control the power tool 800.

[0088] The sensor 506 of the power tool 800 may be an optical sensor configured to capture one or more characteristics of the working surface 802 and / or the fastener 804 driven into the working surface 802. The classifier 514 may be configured to determine the state of the working surface 802 and / or the fastener 804 relative to the working surface 802 based on one or more captured characteristics. This state may be that the fastener 804 is flush with the working surface 802. Alternatively, this state may be the hardness of the working surface 802. The actuator 504 may be configured to control the power tool 800 such that the drive function of the power tool 800 is adjusted based on the determined state of the fastener 804 relative to the working surface 802 or one or more captured characteristics of the working surface 802. For example, if the state of the fastener 804 is flush with the working surface 802, the actuator 504 may interrupt the drive function. As another non-limiting example, the actuator 504 may apply additional or less torque depending on the hardness of the working surface 802.

[0089] Figure 9 A schematic diagram is depicted of a control system 502 configured to control an automated personal assistant 900. The control system 502 can be configured to control an actuator 504, which is also configured to control the automated personal assistant 900. The automated personal assistant 900 can be configured to control household appliances such as a washing machine, stove, oven, microwave oven, or dishwasher.

[0090] Sensor 506 may be an optical sensor and / or an audio sensor. The optical sensor may be configured to receive video images of the user 902's posture 904. The audio sensor may be configured to receive voice commands from the user 902.

[0091] The control system 502 of the automated personal assistant 900 can be configured to determine an actuator control command 510 configured to be controlled by the control system 502. The control system 502 can be configured to determine the actuator control command 510 based on a sensor signal 508 from a sensor 506. The automated personal assistant 900 is configured to transmit the sensor signal 508 to the control system 502. The classifier 514 of the control system 502 can be configured to execute a gesture recognition algorithm to recognize a gesture 904 made by the user 902, determine the actuator control command 510, and transmit the actuator control command 510 to the actuator 504. The classifier 514 can be configured to retrieve information from non-volatile memory in response to the gesture 904 and output the retrieved information in a form suitable for the user 902 to receive.

[0092] Figure 10A schematic diagram of a control system 502 configured to control a monitoring system 1000 is depicted. The monitoring system 1000 can be configured to physically control access through a door 1002. A sensor 506 can be configured to detect scenarios related to determining whether access is permitted. The sensor 506 can be an optical sensor configured to generate and transmit image and / or video data. The control system 502 can use such data to detect faces.

[0093] The classifier 514 of the control system 502 of the monitoring system 1000 can be configured to determine a person's identity by interpreting image and / or video data by matching the identities of known persons stored in non-volatile memory 516. The classifier 514 can be configured to generate an actuator control command 510 in response to the interpretation of the image and / or video data. The control system 502 is configured to transmit the actuator control command 510 to the actuator 504. In this embodiment, the actuator 504 can be configured to lock or unlock door 1002 in response to the actuator control command 510. In other embodiments, non-physical logical access control is also possible.

[0094] The monitoring system 1000 can also be a surveillance system. In such an embodiment, sensor 506 may be an optical sensor configured to detect the scene under surveillance, and control system 502 is configured to control display 1004. Classifier 514 is configured to determine the classification of the scene, such as whether the scene detected by sensor 506 is suspicious. Control system 502 is configured to send actuator control command 510 to display 1004 in response to the classification. Display 1004 may be configured to adjust the displayed content in response to actuator control command 510. For example, display 1004 may highlight objects deemed suspicious by classifier 514.

[0095] Figure 11 A schematic diagram of a control system 502 is depicted, configured to control an imaging system 1100, such as an MRI device, an X-ray imaging device, or an ultrasound device. A sensor 506 may be, for example, an imaging sensor. A classifier 514 may be configured to determine a classification of all or part of the sensed image. The classifier 514 may be configured to determine or select an actuator control command 510 in response to a classification obtained by a trained neural network. For example, the classifier 514 may interpret a region of the sensed image as a potential anomaly. In this case, an actuator control command 510 may be determined or selected to cause the display 1102 to display the image and highlight the potentially anomalous region.

[0096] Figure 12Figure 1200 depicts a vector-quantized linear system according to this disclosure. In some embodiments, Figure 1200 includes an index matrix 1202, which may include model parameters of a neural model. The index matrix 1202 may work in conjunction with a codebook 1204, wherein the codebook has E entries, each entry being of size 1. The index matrix 1202 works with the codebook 1204 to produce intermediate results in a weight matrix 1206. Each parameter in the weight matrix 1206 is an entry in the codebook 1204. Once the weight matrix 1206 is generated based on the index matrix 1202 and the codebook 1204, the weight matrix 1206 can function as a normal linear layer in the neural model.

[0097] While exemplary embodiments have been described above, this does not mean that these embodiments describe all possible forms contained in the claims. The language used in this specification is descriptive and not restrictive, and it should be understood that various changes may be made without departing from the spirit and scope of this disclosure. As previously stated, features of various embodiments may be combined to form further embodiments of the invention that may not be explicitly described or illustrated. While various embodiments may have been described as providing an advantage or superiority over other embodiments or prior art implementations with respect to one or more desired features, those skilled in the art will recognize that one or more features or characteristics may be compromised to achieve desired overall system properties depending on the particular application and implementation. These properties may include, but are not limited to, cost, strength, durability, lifecycle cost, merchantability, appearance, packaging, size, suitability, weight, manufacturability, ease of assembly, etc. Therefore, to the extent that any embodiment is described as less desirable than other embodiments or prior art in achieving the desired features, these embodiments are not outside the scope of this disclosure and may be desirable for a particular application.

Claims

1. A method for generating vector-quantized deep neural networks, the method comprising: Receive a training dataset that includes one or more images, text information, and sound information; The neural model is trained using the training dataset to adjust one or more parameters associated with one or more deep neural network layers; Based on the type and size of each of the one or more deep neural network layers, divide one or more parameters associated with each of the one or more deep neural network layers into one or more segments; Generate one or more fixed codebooks, wherein the fixed codebooks include a predetermined number of codewords; Replace each of one or more segments with one of the codewords; and In response to replacing one or more segments with one of the codewords, the output is a trained neural model using one or more fixed codebooks.

2. The method according to claim 1, wherein the predetermined number of codewords is equal to twice the codeword size.

3. The method according to claim 1, wherein the predetermined number of codewords is always an even number.

4. The method of claim 1, wherein the one or more deep neural network layers are linear layers or convolutional layers.

5. The method of claim 1, wherein the method comprises multiplying a fixed codebook by the input for each neuron of the first type layer.

6. The method of claim 1, wherein the method includes eliminating floating-point multiplication and floating-point addition.

7. The method of claim 1, wherein the one or more codewords are equal to three values, namely 0, 1 or -1.

8. The method according to claim 1, wherein, The one or more images are at least one of digital, text, audio, vector image, bitmap image, and sensor signal.

9. An apparatus for generating vector-quantized deep neural networks, comprising one or more processors, said processors being configured to: Receive a training dataset that includes one or more images; The neural model is trained using the training dataset to generate a first layer with weighted parameters; Based on the type and size of each of the one or more deep neural network layers, divide one or more parameters associated with each of the one or more deep neural network layers into one or more segments; Generate one or more fixed codebooks, wherein the fixed codebooks include a predetermined number of codewords; Replace each of one or more segments with one of the codewords; and In response to replacing one or more segments with one of the codewords, the output is a trained neural model using one or more fixed codebooks.

10. The device of claim 9, wherein the fixed codebook is applied to the input of each layer of the neural model.

11. The device according to claim 9, wherein, The one or more processors are also configured to, at the trained neural model, in response to applying the product of the fixed codebook and the input to one or more layers of the trained neural model, output a final value at the final layer.

12. The device according to claim 9, wherein, The number of code words is equal to twice the size of the code words.

13. The device according to claim 9, wherein the number of codewords is always even.

14. The device of claim 9, wherein the first layer is a linear layer or a convolutional layer.

15. The device of claim 9, wherein the fixed codebook includes only one non-zero entry in the codeword.

16. The apparatus of claim 9, wherein the method includes utilizing a multiplication lookup table applied to the input data.

17. The device of claim 9, wherein the first layer of the first type can be at least one of a linear layer, a 1×1 convolutional layer, and a 3×3 convolutional layer.

18. A system for generating vector-quantized deep neural networks, comprising one or more processors configured to: Receive a training dataset that includes one or more images; The neural model is trained using the training dataset to generate a first layer with weighted parameters; Based on the type and size of each of the one or more deep neural network layers, divide one or more parameters associated with each of the one or more deep neural network layers into one or more segments; Generate one or more fixed codebooks, wherein the fixed codebooks include a predetermined number of codewords; Replace each of one or more segments with one of the codewords; and In response to replacing one or more segments with one of the codewords, the output is a trained neural model using one or more fixed codebooks.

19. The system of claim 18, wherein the codebook comprises a first set of codewords and a second set of codewords, wherein the second set of codewords mirrors the first set of codewords except that -1 values ​​are used instead of 1 values.

20. The system according to claim 18, wherein, After training and vectorization, the neural model is used by at least one of mobile devices, IoT devices, and wearable devices.

Images