Compression of neural networks with orthogonal matrices

Abstract

Embodiments herein relate to a neural network compression technique in which a weight matrix within a neural network is transformed via matrix multiplication with an orthogonal matrix. The orthogonal matrix is derived from a calibration data set, which is generally selected to widely represent expected runtime input data, and the transformation causes the resulting modified weight matrix to have components ordered by relative importance. The modified weight matrix is incorporated into a compressed neural network with fewer weights. By removing one or more components of lower importance, the size of the neural network (and thus its storage and execution overhead) is reduced while still maintaining acceptable performance levels.

Description

Technical Field

[0001] This disclosure relates to methods, systems, and computer programs for compressing neural networks, as well as compressed neural networks obtained using such techniques and their applications. Background Technology

[0002] Neural networks can be used to output information based on input data. Neural networks have been applied in many technological fields, such as image processing, video processing, audio processing and other forms of signal processing, cybersecurity, and natural language processing. Furthermore, generative neural networks have been used to generate synthetic image or video sequences, synthetic audio (e.g., music), text, and more. As neural networks become more complex, the amount of processing required to compute the output from the neural network's inputs also increases. In addition, the amount of memory required to store neural networks increases. A trained neural network comprises weights learned during a structured training process on (multiple) training sets. Neural networks always require significant computational resources to train, especially on large training sets. However, after training, significant storage resources are needed to store the trained neural network, and significant computational resources are required each time such a network is executed on its inputs. This consideration becomes increasingly important with the emergence of 'large' neural networks (such as transformers with billions of weights) and with the increasing range of scenarios where trained neural networks are deployed in systems with limited processing and storage resources (such as wearable devices, mobile devices, Internet of Things (IoT) devices, autonomous vehicles, drones, and other mobile robots). Summary of the Invention

[0003] The present invention is provided to present, in a simplified form, the selection of concepts further described below in the detailed embodiments. The present invention is not intended to identify key or essential features of the claimed solutions, nor is it intended to limit the scope of the claimed solutions. The claimed solutions are also not limited to embodiments that address any or all of the disadvantages pointed out herein.

[0004] The embodiments described herein relate to a neural network compression technique in which the weight matrix within the neural network is transformed via matrix multiplication with an orthogonal matrix. The orthogonal matrix is ​​derived from a calibration dataset (which is typically chosen to broadly represent the expected runtime input data), and the transformation results in a modified weight matrix with components ordered by relative importance. The modified weight matrix is ​​incorporated into a compressed neural network with fewer weights. By removing one or more less important components, the size of the compressed neural network (and therefore its storage and execution overhead) is reduced while still maintaining an acceptable level of performance. Attached Figure Description

[0005] Specific embodiments will now be described by way of example only, referring to the following schematic diagrams, wherein: Figure 1 A flowchart of the neural network method is shown.

[0006] Figure 2 The flowchart of the LayerNorm normalization method is shown.

[0007] Figure 3 A flowchart of a neural network method with self-attention and a multilayer perceptron (MLP) structure is shown.

[0008] Figure 4 Part of the neural network method with RMSnorm normalization is shown.

[0009] Figure 5 A flowchart of a neural network method with self-attention and MLP structure is shown.

[0010] Figure 6 A block diagram is shown illustrating the steps in a method for converting a LayerNorm normalized block into a StandardNorm normalized block.

[0011] Figure 7 A block diagram illustrating the steps in a method for applying orthogonal matrices to a neural network is shown.

[0012] Figure 8 A flowchart of a portion of a neural network-based method with orthogonal linear layers is shown.

[0013] Figure 9 A flowchart of a neural network-based method with self-attention and MLP structure is shown.

[0014] Figure 10 A flowchart of a neural network approach with a Generative Pretrained Transformer (GPT) block is shown.

[0015] Figure 11 A flowchart of a neural network method with GPT blocks is shown.

[0016] Figure 12 A non-limiting example of a computing system is illustrated schematically. Detailed Implementation

[0017] As mentioned above, so-called 'large' neural networks (such as transformers with billions or more learned weights) not only require substantial resources for training, but also significant computational resources for storage once trained (because each weight needs to be electronically stored) and processing resources for runtime execution (because each additional weight increases the number of computations that need to be performed). Even smaller neural networks with fewer weights have storage and runtime execution overheads that can be significant in many contexts. For example, challenges arise when storing and executing neural networks of any size in systems with limited computational resources, such as mobile devices, wearables, IoT, or 'edge' devices with limited storage and processing resources. For battery-powered devices, reduced power consumption is an important goal, and one way to reduce power consumption is to reduce the computational overhead of neural networks stored and executed on such devices.

[0018] This paper describes a network compression technique that can significantly improve the efficiency of neural network computation, with minimal performance degradation in the worst case. Given a trained neural network to be deployed, a system that "shrinks" the neural network is provided by slicing off a portion of the network's weight matrix. This "shrinking" significantly reduces the computational requirements for deploying the neural network.

[0019] In addition to reducing the amount of processing required for a neural network to produce its output from its input, this neural network compression technique also reduces the amount of storage required to store the compressed neural network in memory. The examples described herein illustrate a method for neural network compression that minimizes the performance loss caused by compression (which can be measured in terms of perplexity, accuracy, or precision).

[0020] The motivation behind the aforementioned compression techniques is to reduce the amount of storage and computational resources consumed by a trained neural network while maintaining an acceptable level of performance. In an example embodiment, this is achieved by modifying the weight matrix within the neural network based on a calibration dataset, resulting in a modified weight matrix that includes multiple components ordered by relative importance. This, in turn, allows one or more components of lower importance to be removed, leading to a reduction in network size and a less significant impact on performance (compared to simply reducing the size of the original weight matrix). In this way, the resulting compressed neural network produces improvements in computer systems configured to store and execute the compressed neural network, as the system is able to achieve a given performance level while consuming fewer storage and processing resources at runtime than the original (uncompressed) neural network.

[0021] The examples described in this article provide a method for determining the form of processing blocks for neural networks. This form allows blocks to be truncated with minimal performance loss.

[0022] Neural networks can interface with the real world in both their inputs and their outputs / effects. For example, multiple candidate actions can be evaluated via causal reasoning to select the action (or subset of actions) with the highest estimation validity, and the selected action can be executed on (multiple) physical systems to produce tangible real-world results. Inputs can take the form of measurable physical quantities, such as energy, material properties, processing, memory / storage resource usage in a computer system, therapeutic effects, etc. Such quantities can be measured directly using sensor systems, or estimated from measurements of another physical quantity or quantities.

[0023] The examples described in this article can achieve a given level of perplexity while reducing memory and processing requirements. Some examples also reduce the amount of data passed between blocks of a neural network.

[0024] Some examples described in this article use calibration datasets to determine orthogonal matrices at the output of processing blocks of a neural network. A calibration dataset can represent a dataset intended for use with a neural network.

[0025] A neural network may include one or more processing blocks. Each processing block may include one or more weight matrices. Normalization blocks may be located between each processing block. Unlike normalization blocks that include LayerNorm operations, some LayerNorm operations can be absorbed into previous processing blocks, and some LayerNorm operations can be absorbed into subsequent processing blocks. This allows the use of StandardNorm operations instead of LayerNorm between each processing block. In other examples, neural networks using root mean square (RMS) normalization can be similarly transformed.

[0026] When the neural network is in the normalized form described above, an orthogonal matrix can be determined for each normalized block. For each normalized block, the orthogonal matrix can be applied to subsequent processing blocks, and the transpose of the orthogonal block can be applied to previous processing blocks. This results in a modified weight matrix in each processing block comprising multiple components ordered by relative importance. At least one component with relatively low importance can then be removed from at least one modified weight matrix, resulting in at least one truncated weight matrix. This can include removing the least important X% of components from the modified weight matrix based on their order of relative importance. This provides compression of the neural network. By removing relatively low-importance components, the performance degradation of the neural network is minimized. In some examples, this method is applied to compress neural networks that have already been trained.

[0027] This method has many practical applications, including processing and generating images, videos, text, and audio from one or more physical devices (such as cameras, microphones, sensors, etc.). Technical applications can include attention-based neural network technologies, such as image generation, audio signal processing, and audio or music generation. Another application is cybersecurity, where cybersecurity knowledge can be obtained from structured models and used, for example, to achieve cyber threat detection and / or remediation by inducing or instructing devices to take remedial or mitigation actions.

[0028] Figure 1 A neural network is shown, which is generated from some signals 102 Calculate the prediction result 114 The algorithm. The network consists of a series of weight matrices. and nonlinear Definition. In the context of language modeling, It is a matrix where rows correspond to elements of a sequence of words, characters, or lexical units, and columns are called channels. In other applications, Elements can represent, for example, image pixels or image regions, segments of audio data, sensor data points, etc. The dimension is represented as (Sequence length) and (Channels). The weight matrices and nonlinearities are arranged into blocks, each containing two weight matrices and a single nonlinearity. The dimensions of the weight matrices within a block are arranged such that the input and output channels of the block are equal to... ,Notice' 'Used to represent the original input to the network and the transformed version of the original input computed within the network.

[0029] Figure 1 The matrix operations performed at processing block 104, as defined by Algorithm 1, are also shown. The processing block consists of a weight matrix. and Bias and and nonlinear Definition. Processing block received signal matrix 102 As input. Matrix Multiply and bias Added to results Then the results Multiply by the second weight matrix and bias Add to results, give Then the nonlinear operation Applying this result, we give Nonlinear operations, such as sigma, can be performed element-wise or across elements: this includes attention mechanisms and multilayer perceptron algorithms. Computational blocks can operate on multiple signals simultaneously (so-called batch operations).

[0030]

[0031] Figure 1 It also shows an arrangement in sequence and a distribution of execution. Normalization The neural network consists of a series of processing blocks 104, 108, and 112, following the normalization blocks 106 and 110. The normalization operation can be... LayerNorm , RMSNorm , L1Norm One of the others.

[0032] Figure 2 An example normalization block 202 is shown, which performs LayerNorm normalization on the matrix input 204 from the previous processing block. LayerNorm normalization includes applying a normalization method to the input signal matrix. The three operations are: 1. Perform operation 206 on matrix input 204. 2. Calculate the matrix. In the row mean of values and standard deviation And normalized The value is given by the following formula: (1) in and These are the scaling factor and the normalization bias, respectively. The LayerNorm operation is shown in Algorithm 2 below.

[0033]

[0034] The neural network computes predictions by processing the signal sequentially via block and normalization operations. This is illustrated in Algorithm 2 below. The number of blocks is determined by... It indicates, and the first Each block is composed of This indicates that the number of normalization operations also increases from... It indicates, and the first The block normalization operation is performed by express.

[0035]

[0036] The neural network forms considered in this example encompass a wide range of neural network architectures, including transformer architectures. For example, blocks can alternate between attention and MLP (Multilayer Perceptron) structures, with LayerNorm normalization in between.

[0037] Figure 3 An example architecture with self-attention structure 306 and MLP structure 314 is shown, with LayerNorm normalization 304 and 312 in between. Input signal 302 is received and processed by LayerNorm normalization 304 and self-attention block structure 306. At step 308, the operation from Algorithm 2 is performed. + The processed signal 310 is obtained. The processed signal 310 is further processed by LayerNorm normalization 312 and MLP block structure 314. At step 316, the operation from Algorithm 2 is performed. + To obtain the processed signal 318. Note that the neural network described herein can be used as a subroutine within a larger network, such as an encoder-decoder architecture and / or with language modeling or other application-specific heads.

[0038] Normalization operations can be LayerNorm , RMSNorm , L1Norm One of the others. Figure 4 An example neural network is shown, where an RMSnorm is combined with a linear layer for normalization operations. Input signal 402 is fed into a linear layer 404, which processes each element of the input matrix 402. Execute operation The output 406 of the linear layer 404 is fed into the normalization block 408. The normalization block 408 performs an RMSNorm operation, thereby dividing each element of the input matrix 406 by... The output 410 from the normalization block 408 is fed into the linear layer 412, which performs a linear operation on each element of the input matrix 410. ,in It's scaling, and It is the offset, and the output is 414.

[0039] Figure 5 This illustrates how the RMSOrm normalization block can alternate between attention and MLP (Multilayer Perceptron) structures. Input signal 502 is fed to RMSOrm normalization block 504. The output of normalization block 504 is fed to the first linear (Lin) layer 506 of self-attention block 508. In step 512, the operations from Algorithm 2 are performed on the original signal 502 and the block normalized signal from the last layer 510 of the self-attention structure 508. + The output of step 512 is matrix 514, which is fed into the RMSnorm normalization block 516. The output of normalization block 516 is fed into the first linear layer 518 of MLP block 520. In step 524, the operation from Algorithm 2 is performed on signal 514 and another block of normalized signals from the last layer 522 of MLP structure 520. + .

[0040] Standard form Before manipulating the neural network, the normalization operation can be configured in a standard form. This form does not change the computational complexity of Algorithm 3, nor does it change the neural network output. Its purpose is to achieve the rotation of the weight matrix as described below. As shown in Algorithm 4, the StandardNorm block is formed by rotating the signal matrix... Each line consists of a single operation: (2)

[0041] To convert a neural network with LayerNorms into its standard form, the preceding and subsequent linear operations on each LayerNorm are modified.

[0042] Figure 6 This illustrates the reordering of operations for a neural network with LayerNorm into standard form. In step 608, performed at normalization block 610, mean subtraction is absorbed into the second weight matrix 604 from the previous block 602. and first bias 606 In step 616, the first weight matrix 612 is used. and Subsequent The first bias of block 616 is 614 To apply rescaling. These operations are shown in Algorithm 5, where... Operation connects rows along the column axis , Operation to calculate scaling matrix The diagonal.

[0043]

[0044] Once each pair of blocks has been updated, the network can be reproduced by replacing each block with the modified block and each instance of LayerNorm with StandardNorm. This has no effect on the output of the neural network and the impact on the required computation is negligible. Note that a similar transformation can be performed on neural networks using RMS normalization.

[0045] Rotation of the weight matrix Because the network is in a standard form, a simplification process can be applied. First, a calibration dataset is selected, which is used to compute some orthogonal matrices that are then used to modify the neural network. The calibration set can represent the task the network is designed for. For example, in this work, a dataset used for sparsity and quantization has been used.

[0046] Algorithm 6 illustrates the process for the weight matrix used to rotate the processing block. The input signal matrix of the neural network associated with the calibration dataset... The rows in are represented as ,in As a shorthand, symbols can be used. Calculate the expected orthogonal matrix at each normalized block. In Algorithm 6, The operation is used to concatenate the input matrix in the column direction. The rows in the matrix give the matrix .matrix The transpose of express. The operation is used to find the eigenvectors of the matrix, and The operation represents matrix multiplication. The processing block set of a neural network consists of... The number of blocks is indicated by... It indicates, and the first Block by This indicates that the algorithm's output is a set of orthogonal matrices { }, Each of them Associated with standard normalized blocks.

[0047]

[0048] Applying orthogonal matrices Using the computed set { as in Algorithm 6} }, each { Then it is applied to each corresponding block in the network. Figure 7 This illustrates how the orthogonal matrix calculated as in Algorithm 6 is applied to processing block 706 (block l), which is preceded by processing block 700 (block l-1). In step 710, the first weight matrix 708 of the block is used... Multiply by the previously normalized 702 The transpose of the associated orthogonal matrix 704 is used to modify processing block 706. In step 720, the second matrix 712 is... Second bias Each is multiplied by the orthogonal matrix 718 associated with the subsequent normalization 716. Then, in subsequent processing block 726 (block l+1), orthogonal matrix 718 is used, where the first weight matrix of processing block 726 is modified by multiplying by the transpose of orthogonal matrix 718. Algorithm 7 shows how the orthogonal matrix computed in Algorithm 6 is applied to processing blocks in a neural network. Algorithm 7 also defines matrices used in the modified forward propagation process, which will be described further below. In step 722, the matrix of processing block 706 is... Defined as In step 746, the matrix of processing block 726 is... Defined as .

[0049] In Algorithm 6, the input to block 0... This is the untransformed calibration dataset. The nth output of block 0 is the nth input of the normalized block 0. At normalized block 0, calculate It becomes the nth input to block 1 (updated) ), and so on. Regarding blocks ll, l, and l+1 (700, 706, 726) intertwined with normalized blocks l-1 and l (702, 716), in Figure 6 The bottom of the diagram depicts this input / output relationship.

[0050] Figure 8 A neural network with alternating linear layers 804, 812 and an RMSnorm normalization block 808 is shown; the matrices used in the linear layers (as shown in...) Figure 4 (The middle) can be obtained from orthogonal matrices as in Algorithm 7. and replace.

[0051] Figure 9 This demonstrates how the orthogonal matrix can be modified by applying it, as calculated in Algorithm 7. Figure 5 The neural network in the algorithm. The matrices in linear layers 506, 510, 518, and 522 are orthogonal matrices as shown in Algorithm 7. and Instead, we give orthogonal linear layers 906, 910, 918, and 922. Now, by introducing an orthogonal matrix... and The original linear operation Change to

[0052]

[0053] Algorithm 8 illustrates the steps for performing a modified forward propagation on a neural network. Algorithm 8 computes the exact same forward propagation as the original network, where all... The calculations cancel each other out. This is because... The input signal is given at the beginning of Algorithm 8 by... Modify; the first block 'undo's the operation because it has already been pre-multiplied in Algorithm 7. The output of the first block is entirely from... The modification is then passed to the next processing block via standard normalization, which has similarly used... Please make the changes. Note that standard normalization does not affect... and The elimination of this is because the standard normalization block does not affect the orientation of the input matrix in the following sense: .

[0054] It is not necessary to apply orthogonal matrices in all blocks of the neural network. Figure 10 A neural network with two Generative Pretrained Transformer (GPT) blocks 1002 and 1012 is shown. GPT block 1002 consists of linear layers 1004 and 1005, a self-attention (SA) layer 1006, and an MLP layer 1008. The output of GPT block 1002 is fed into linear layer 1010. The output of linear layer 1010 is fed into a second GPT block 1012. The second GPT block 1012 consists of linear layer 1014, an SA layer 1018 with linear layers 1016 and 1020, a linear layer 1022, and an MLP layer 1024 with linear layers 1026 and 1028. The output of GPT block 1030 is fed into linear layer 1030. As indicated by the arrows, orthogonal matrices are selectively applied to linear layers 1010, 1016, 1020, 1026, 1028, and 1030, thus giving orthogonal linear layers.

[0055] Orthogonal matrices can also be applied to the entire block or a set of blocks. Figure 11 A neural network with four GPT blocks 1102, 1104, 1106, and 1108 is shown. Orthogonal matrices are selectively applied to the linear layers in the set of blocks 1104, 1106, and 1108, but not to the linear layers in GPT block 1102.

[0056]

[0057] reduce To date, novel ways of computing the forward propagation of a given neural network have been defined, including constructing orthogonal matrices that project the signals between each block onto its principal components. Smaller principal components can be removed by shaving the weight matrix of a portion within each block, thereby reducing the number of channels required and lowering the overall computational complexity. In one embodiment, the amount of shaving to be applied is controlled by a configurable parameter. Furthermore, since any orthogonal matrix can be freely chosen, the matrix projected onto their largest principal components can be selected.

[0058] For example, when extracting entities from average text document data using a traditional large language model, the computation required takes approximately one minute to complete. This processing time can be reduced by shaving the weight matrix in the neural network used in the large language model, thereby reducing computational and memory requirements. Further shaving is expected to result in a worse replication of the original neural network while reducing the computation required for forward propagation. However, experiments show that this technique can be used to shave up to approximately 30% of the lowest relative importance weights in a large language model with only a small loss in performance (measured in perplexity). In practice, the minimum relative percentage (%) that can be removed will depend on the given performance requirements. For example, shaving more than 30% may be possible where a higher loss in performance is acceptable. It should be understood that it is impossible to impose an absolute limit on the percentage reduction in all cases. However, given defined performance requirements (such as the maximum permissible decrease in performance relative to the uncompressed neural network measured on predefined performance metrics such as perplexity, accuracy, precision, F1 score, etc.), it will be possible to determine the number of weights that can be removed from the modified weight matrix or multiple weight matrices through conventional experimentation. The tolerable amount of reduction can vary depending on the type of model (e.g., text, image, audio, multimodal, etc.), which can also be verified through conventional experiments on the modified weight matrix or multiple weight matrices. In this context, 'relative importance' refers to the importance between different weights in the modified weight matrix, evaluated in terms of their impact on performance or their estimated impact on performance. Relative importance can be quantified using techniques such as Singular Value Decomposition (SVD) or Principal Component Analysis (PCA), where SVD or PCA scores are generally expected to at least approximately correlate with performance (in the sense that removing components with higher PCA or SVD scores will have a greater performance impact on average than removing components with lower PCA or SVD scores). Note that removing relatively low-importance components does not necessarily imply a 'thresholding' approach (e.g., removing a fixed percentage or number of weights with the lowest importance score). For example, in another embodiment, probabilistic methods can be used to remove weights with a probability determined by their importance score (making lower-score weights more likely to be removed).

[0059] Algorithm 9 illustrates the steps followed when shaving the weight matrix of a neural network. The original channel width is determined by... This indicates that the new expected channel width is In Algorithm 9, brackets [] are used to indicate selecting elements from a matrix. For example, the operation... Select matrix Select all rows in the matrix, but only select the matrix. The front of the middle Column. Operations Select matrix The front of the middle Rows and matrices All columns in the table. Operations. Select Matrix All rows and matrices The front of the middle Column. Operations = Select bias The front of the middle Okay. Operation Select matrix The front of the middle line and front List.

[0060]

[0061] To compute the forward propagation of the truncated neural network, Algorithm 8 is applied. Note that each matrix multiplication reduces the factor. / This improves efficiency and reduces processing. It also reduces the amount of memory required to store the neural network.

[0062] The number of neurons in an MLP layer can also be reduced by projecting the weight matrix onto the nearest set of orthogonal polynomials. In addition to the reductions mentioned above, a second-order acceleration can be achieved in one of the main computations in the SA layer.

[0063] The projection of the signal from each block onto its principal component can be accomplished using singular value decomposition (SVD). Because the reconstructed angle is a 'sufficient statistic' of the error, L1 norm SVD works better than naive SVD. This approximates the spherical reduction algorithm (in... (In the small error mechanism). Within an MLP, the angle from the orthogonal matrix to the weight vector is all that's needed. Therefore, a rotation that makes the data orthogonal to the 'north pole' is chosen. It is advantageous to remove the last element: .

[0064] The ML architecture described in this paper, and especially the neural network compression mechanism, has many practical applications in various technical fields. In a broad sense, neural networks can be configured, for example, as declarative networks for tasks such as classification or regression (declarative networks, broadly speaking, learn to generate predictions about previously unseen data) or as generative networks (broadly speaking, have the ability to generate new data points). Applications of neural networks include image classification or extracting information from images (e.g., classifying images, image regions, or image pixels; locating objects in an image, such as by predicting object bounding boxes, etc.), text classification, extracting structured or semi-structured information from text, audio signal classification (e.g., classifying different parts of an audio signal, such as in the context of speech recognition, to separate speech from non-speech, or to convert speech into text), extracting information from sensor signals (e.g., using classification or regression networks that operate on signals from one or more sensors to perform measurements, such as in machine control applications (e.g., such measurements can be used to measure physical characteristics of machines or systems (such as vehicles, robots, manufacturing systems, energy generation systems, etc.) or physical characteristics associated with machines or systems), or in medical sensing applications (such as patient monitoring or diagnosis) (e.g., monitoring and classifying a patient's vital signs)). Other applications include generating images (e.g., based on text or non-text input), text (e.g., translating text from one language to another, or generating responses to user text input), audio data (e.g., synthesized speech, music, or other sounds) or music (e.g., in the form of numerical or symbolic musical notation), computer code executable on a processor (e.g., computer code for controlling or implementing technical processes on a computer or machine, such as expressing code (e.g., in response to user instructions in natural language) or converting or compiling code (e.g., source code, object code, or machine code) from one programming language to another), modeling or simulating physical, chemical, and other technical systems, or discovering new chemical components or new uses for them (including applications of 'drug discovery,' discovering new therapeutic compounds or drugs, or new therapeutic uses). Furthermore, when using neural network compression methods (as described above, which can be learned and shared across multiple applications / modalities), any of the aforementioned applications can be improved in terms of performance (e.g., accuracy, precision, robustness / reliability). Additionally, less memory and / or processing resources are required when performing any of the aforementioned applications using neural network compression methods. The system also has applications in cybersecurity.For example, the described methods can be used to construct a dedicated cybersecurity knowledge base to support neural networks performing cybersecurity functions, such as identifying anomalous or potentially suspicious data points or signals in cybersecurity data (which may, for example, embody cybersecurity telemetry collected using endpoint software and / or multiple network monitoring components) or patterns indicating potentially suspicious activities or behaviors. This allows appropriate reporting, remediation, or other cybersecurity actions (e.g., generating alerts, terminating or isolating applications, services, or processes, revoking user or application privileges, etc.) to be taken based on the output of the knowledge base-supported neural network (e.g., detection output indicating potentially suspicious activities / behaviors that have been detected, or another form of cybersecurity detection results). Generative cybersecurity models supported by the knowledge base can, for example, be configured to generate 'synthetic' cybersecurity data, for example, for the purpose of training, testing, or validating other cybersecurity components and models.

[0065] Figure 12 A non-limiting example of a computing system 1200 is schematically shown, such as a connected computing device or system capable of performing one or more of the methods or processes described above, including filtering data and implementations of the structured knowledge base described above. The computing system 1200 is shown in a simplified form. The computing system 1200 includes a logic processor 1202, volatile memory 1204, and non-volatile storage device 1206. The computing system 1200 may optionally include a display subsystem 1208, an input subsystem 1210, a communication subsystem 1212, and / or... Figure 12Other components not shown. The logic processor 1202 includes one or more physical (hardware) processors configured to perform processing operations. For example, the logic processor 1202 may be configured to execute instructions as part of one or more applications, programs, routines, libraries, objects, components, data structures, or other logical constructs. The logic processor 1202 may include one or more hardware processors configured to execute software instructions based on an instruction set architecture such as a central processing unit (CPU), a graphics processing unit (GPU), or other forms of accelerator processor. Additionally or alternatively, the logic processor 1202 may include (multiple) hardware processors in the form of logic circuits or firmware devices configured to execute hardware-implemented logic (programmable or non-programmable) or firmware instructions. The (multiple) processors of the logic processor 1202 may be single-core or multi-core, and the instructions executed thereon may be configured for sequential, parallel, and / or distributed processing. The various components of the logic processor may optionally be distributed among two or more separate devices that may be remotely located and / or configured for collaborative processing. Various aspects of the logic processor 1202 can be virtualized and executed by remotely accessible networked computing devices configured in a cloud computing setup. In this case, these virtualized aspects run on different physical logic processors on various different machines. The non-volatile storage device 1206 includes one or more physical devices configured to store instructions executable by the logic processor 1202 to implement the methods and processes described herein. When implementing such methods and processes, the state of the non-volatile storage device 1206 can be transformed—for example, to store different data. The non-volatile storage device 1206 may include removable and / or built-in physical devices. The non-volatile storage device 1206 may include optical memory (e.g., CD, DVD, HD-DVD, Blu-ray Disc, etc.), semiconductor memory (e.g., ROM, EPROM, EEPROM, FLASH memory, etc.), and / or magnetic memory (e.g., hard disk drive) or other mass storage technologies. Non-volatile storage device 1206 may include non-volatile, dynamic, static, read / write, read-only, sequential access, location-addressable, file-addressable, and / or content-addressable devices. Volatile memory 1204 may include one or more physical devices, including random access memory. Volatile memory 1204 is typically used by logic processor 1202 to temporarily store information during the processing of software instructions. Various aspects of logic processor 1202, volatile memory 1204, and non-volatile storage device 1206 may be integrated together into one or more hardware logic components.For example, such hardware logic components may include field-programmable gate arrays (FPGAs), application-specific integrated circuits (PASICs / ASICs), application-specific standard products (PSSPs / ASSPs), systems-on-a-chip (SOCs), and complex programmable logic devices (CPLDs). The terms "module," "program," and "engine" can be used to describe aspects of a computing system 1200 typically implemented in software by a processor to perform specific functions using portions of volatile memory, involving transformative processing of functions specifically configured for the processor. Thus, a module, program, or engine can be instantiated using portions of volatile memory 1204 via a logic processor 1202 that executes instructions held by non-volatile memory device 1206. Different modules, programs, and / or engines can be instantiated from the same applications, services, code blocks, objects, libraries, routines, APIs, functions, etc. Similarly, the same module, program, and / or engine can be instantiated from different applications, services, code blocks, objects, routines, APIs, functions, etc. The terms “module,” “program,” and “engine” can encompass single or grouped executable files, data files, libraries, drivers, scripts, database records, etc. When included, the display subsystem 1208 can be used to present a visual representation of data held by the non-volatile storage device 1206. The visual representation can take the form of a graphical user interface (GUI). Since the methods and processes described herein alter the data held by the non-volatile storage device and thus transform the state of the non-volatile storage device, the state of the display subsystem 1208 can also be transformed to visually represent changes in the underlying data. The display subsystem 1208 may include one or more display devices utilizing virtually any type of technology. Such display devices may be combined with the logic processor 1202, the volatile memory 1204, and / or the non-volatile storage device 1206 in a shared housing, or such display devices may be peripheral display devices. When included, the input subsystem 1210 may include or interface with one or more user input devices, such as a keyboard, mouse, touchscreen, or game controller. In some embodiments, the input subsystem may include or interface with a selected Natural User Input (NUI) component. Such components may be integrated or peripheral, and the transduction and / or processing of input actions may be handled on-board or off-board. Example NUI components may include microphones for speech and / or speech recognition; infrared, color, stereo, and / or depth cameras for machine vision and / or gesture recognition; head trackers, eye trackers, accelerometers, and / or gyroscopes for motion detection and / or intent recognition; and electric field sensing components for assessing brain activity; and / or any other suitable sensors. When included, the communication subsystem 1212 may be configured to communicatively couple the various computing devices described herein to each other and to other devices.The communication subsystem 1212 may include wired and / or wireless communication devices compatible with one or more different communication protocols. As a non-limiting example, the communication subsystem may be configured to communicate via a wireless telephone network, or a wired or wireless local area network or wide area network. In some embodiments, the communication subsystem may allow the computing system 1200 to send messages to and / or receive messages from other devices via a network such as the Internet. The term computer-readable medium, as used herein, may include computer storage media. Computer storage media may include volatile and non-volatile, removable and non-removable media (e.g., volatile memory 1204 or non-volatile storage device 1206) implemented in any method or technology for storing information such as computer-readable instructions, data structures, or program modules. Computer storage media may include RAM, ROM, electrically erasable read-only memory (EEPROM), flash memory or other storage technologies, CD-ROM, digital versatile disc (DVD) or other optical storage, magnetic tape cassettes, magnetic tape, disk storage or other magnetic storage devices, or any other article of manufacture that can be used to store information and can be accessed by a computing device (e.g., computing system 1200 or its component devices). Computer storage media do not include carrier waves or other propagated or modulated data signals. Communication media can be embodied in computer-readable instructions, data structures, program modules, or other data in modulated data signals (such as carrier waves or other transmission mechanisms), and include any information transmission medium. The term "modulated data signal" can describe a signal having one or more characteristics set or altered in a manner that encodes information in the signal. By way of example and not limitation, communication media can include wired media such as wired networks or direct wired connections, and wireless media such as acoustic, radio frequency (RF), infrared, and other wireless media.

[0066] A first aspect of this paper provides a computer-implemented method comprising: determining an orthogonal matrix using a neural network applied to a calibration dataset, the neural network including: a first processing block and a normalization block having inputs connected to the output of the first processing block; multiplying a first weight matrix of the first processing block with the orthogonal matrix to obtain: a modified first weight matrix including multiple components ordered by relative importance; removing at least one component of relatively low importance from the modified first weight matrix to obtain a truncated first weight matrix; and generating, in computer storage, a compressed neural network including a compressed first processing block, the compressed first processing block being configured to apply the truncated first weight matrix to inputs received at the compressed first processing block.

[0067] In one embodiment, the neural network may include a second processing block having an input connected to the output of a normalization block. The method includes: multiplying a second weight matrix of the second processing block by the transpose of an orthogonal matrix to obtain: a modified second weight matrix including a plurality of components ordered by relative importance; removing at least one component of relatively low importance from the modified second weight matrix to obtain a truncated second weight matrix; and generating, in computer storage, a compressed neural network including a compressed second processing block, based on the neural network, the compressed second processing block being configured to apply the truncated second weight matrix to an input received at the compressed second processing block.

[0068] In the above example embodiment, the first processing block may correspond to Figure 7 Block 706 (block l) is used in the first processing block, and the second processing block can correspond to block 726 (l+1). In this example, the first weight matrix corresponds to the first weight matrix in block l. The weight matrix, and the second weight matrix corresponds to the weight matrix in block l+1. Weight matrix, where the orthogonal matrix corresponds to .

[0069] The method may include determining the second weight matrix of the second processing block by performing a scaling operation on the third weight matrix.

[0070] The method may include: generating an input to the first processing block using a third processing block of a neural network preceding the first processing block and a second normalized block of a neural network connected between the third processing block and the first processing block; and determining a calibration result matrix based on the input, wherein the orthogonal matrix may be computed by multiplying the calibration result matrix by the eigenvectors of the calibration result matrix and the transpose of the calibration result matrix.

[0071] exist Figure 7 In the example, the third processing block can correspond to processing block 700 (l-1), and the second normalization block can correspond to normalization block 702 (l-1).

[0072] The method may include: when the neural network includes a third weight matrix, determining the first weight matrix of the first processing block by performing a subtraction operation based on the fourth weight matrix and the average value at the normalized output.

[0073] The method may include: providing input to a compressed first processing block.

[0074] Normalization blocks can execute standard normalization functions.

[0075] The method may include: using a compressed neural network applied to an input to generate an output, the input including at least one of the following: image data, video data, audio data, text data, cybersecurity data, sensor data, and medical data.

[0076] The input can be measured by a sensor.

[0077] Output can enable physical devices to perform actions based on the output.

[0078] The method may include: using a compressed neural network applied to the input to generate an output, the output including at least one of the following: image data, video data, audio data, text data, cybersecurity data, sensor data, and medical data.

[0079] A second aspect of this document provides a computer system comprising: at least one processor coupled to at least one memory, and the at least one memory configured to execute executable instructions that, when executed, cause the at least one processor to: determine an orthogonal matrix using a neural network applied to a calibration dataset, the neural network including: a first processing block and a normalization block having inputs connected to the output of the first processing block; multiplying a first weight matrix of the first processing block with the orthogonal matrix to obtain: a modified first weight matrix including a plurality of components ordered by relative importance; removing at least one component of relatively low importance from the modified first weight matrix to obtain a truncated first weight matrix; and generating, based on the neural network, a compressed neural network in computer storage including a compressed first processing block, the compressed first processing block being configured to apply the truncated first weight matrix to inputs received at the compressed first processing block.

[0080] In an embodiment, the neural network may include a second processing block having an input connected to the output of a normalization block, and at least one processor may be configured to: multiply a second weight matrix of the second processing block by the transpose of an orthogonal matrix to obtain: a modified second weight matrix including a plurality of components ordered by relative importance; remove at least one component of relatively low importance from the modified second weight matrix to obtain a truncated second weight matrix; and generate, in computer storage, a compressed neural network including a compressed second processing block, the compressed second processing block being configured to apply the truncated second weight matrix to the input received at the compressed second processing block.

[0081] The at least one processor is configured to determine the second weight matrix of the second processing block by performing a scaling operation on the third weight matrix.

[0082] At least one processor can be configured to determine the first weight matrix of the first processing block by performing a subtraction operation based on the fourth weight matrix and the average value at the normalized output when the neural network includes a third weight matrix.

[0083] At least one processor can be configured to provide input to a compressed first processing block.

[0084] Normalization blocks can execute standard normalization functions.

[0085] At least one processor can be configured to use a compressed neural network applied to an input to generate an output, the input including at least one of the following: image data, video data, audio data, text data, cybersecurity data, sensor data, and medical data.

[0086] Output can enable physical devices to perform actions based on the output.

[0087] A third aspect of this document provides a computer-readable storage medium storing computer-readable instructions configured to, when executed by at least one processor: receive input; process the input using a compressed neural network, the compressed neural network including a truncated first weight matrix obtained by: determining an orthogonal matrix using a neural network applied to a calibration dataset, the neural network including: a first processing block and a normalization block, the normalization block having an input connected to the output of the first processing block; multiplying the first weight matrix of the first processing block with the orthogonal matrix to obtain: a modified first weight matrix including multiple components ordered by relative importance; removing at least one component of relatively low importance from the modified first weight matrix to obtain the truncated first weight matrix.

[0088] It should be understood that the above embodiments are disclosed as examples only. Other variations or use cases will become apparent to those skilled in the art once the disclosure herein has been given. The scope of this disclosure is not limited to the above embodiments, but only to the appended claims.

Claims

1. A computer-implemented method, comprising: An orthogonal matrix (718) is determined using a neural network (100) applied to a calibration dataset. The neural network (100) includes a first processing block (706) and a normalization block (716), the normalization block (716) having an input connected to the output of the first processing block (706). Multiplying the first weight matrix (712) of the first processing block (706) with the orthogonal matrix (718) yields a modified first weight matrix comprising multiple components ordered by relative importance. Remove at least one component with relatively low importance from the modified first weight matrix to obtain a truncated first weight matrix; Based on the neural network (100), a compressed neural network including a compressed first processing block is generated in computer storage, the compressed first processing block being configured to apply the truncated first weight matrix to the input received at the compressed first processing block.

2. The computer-implemented method of claim 1, wherein the neural network (100) includes a second processing block (726) having an input connected to the output of the normalization block (716), the method comprising: Multiplying the second weight matrix (736) of the second processing block with the transpose of the orthogonal matrix (718) yields a modified second weight matrix comprising multiple components ordered by relative importance. Remove at least one relatively less important component from the modified second weight matrix to obtain a truncated second weight matrix; Based on the neural network (100), a compressed neural network including a compressed second processing block is generated in computer storage, the compressed second processing block being configured to apply the truncated second weight matrix to the input received at the compressed second processing block.

3. The computer-implemented method according to claim 2, the method comprising: The second weight matrix (736) of the second processing block (726) is determined by performing a scaling operation on the third weight matrix.

4. The computer-implemented method according to any of the preceding claims, comprising: The input to the first processing block (706) is generated using a third processing block of the neural network (100) preceding the first processing block (706) and a second normalization block (702) of the neural network connected between the third processing block and the first processing block (706); as well as The calibration result matrix is ​​determined based on the input, wherein the orthogonal matrix is ​​calculated by multiplying the calibration result matrix by its transpose and the eigenvectors.

5. The computer-implemented method according to claim 3, the method comprising: When the neural network includes the third weight matrix, the first weight matrix of the first processing block (706) is determined by performing a subtraction operation based on the fourth weight matrix and the average value at the normalized output.

6. The computer-implemented method according to any preceding claim, the method comprising: The input is provided to the compressed first processing block.

7. The computer-implemented method according to any of the preceding claims, wherein the normalization block performs a standard normalization function (206).

8. The computer-implemented method according to any of the preceding claims, the method comprising: The compressed neural network is used to generate an output from an input, the input of which includes at least one of the following: image data, video data, audio data, text data, cybersecurity data, sensor data, and medical data.

9. The computer-implemented method of claim 8, wherein the input is measured by a sensor.

10. The computer-implemented method of claim 8 or 9, wherein the output causes a physical device to perform an action based on the output.

11. The computer-implemented method according to any preceding claim, the method comprising: The compressed neural network is applied to the input to generate the output, which includes at least one of the following: image data, video data, audio data, text data, cybersecurity data, sensor data, and medical data.

12. A computer system, comprising: At least one memory is configured to store executable instructions; as well as At least one processor is coupled to the at least one memory, and the at least one processor is configured to execute the executable instructions, which, upon execution, cause the at least one processor to: An orthogonal matrix (718) is determined using a neural network (100) applied to a calibration dataset. The neural network (100) includes a first processing block (706) and a normalization block (716), the normalization block (716) having an input connected to the output of the first processing block (706). Multiplying the first weight matrix (712) of the first processing block (706) with the orthogonal matrix (718) yields a modified first weight matrix comprising multiple components ordered by relative importance. Remove at least one component with relatively low importance from the modified first weight matrix to obtain a truncated first weight matrix; Based on the neural network (100), a compressed neural network including a compressed first processing block is generated in computer storage, the compressed first processing block being configured to apply the truncated first weight matrix to the input received at the compressed first processing block.

13. The computer system of claim 12, wherein the neural network (100) includes a second processing block (726) having an input connected to the output of the normalization block (716), and the at least one processor is configured to: Multiplying the second weight matrix (736) of the second processing block (726) with the transpose of the orthogonal matrix (718) yields a modified second weight matrix comprising multiple components ordered by relative importance. Remove at least one component with relatively low importance from the modified second weight matrix to obtain a truncated second weight matrix; Based on the neural network (100), a compressed neural network including a compressed second processing block is generated in computer storage, the compressed second processing block being configured to apply the truncated second weight matrix to the input received at the compressed second processing block.

14. The computer system of claim 13, wherein the at least one processor is configured to: The second weight matrix of the second processing block is determined by performing a scaling operation on the third weight matrix.

15. The computer system of claim 14, wherein the at least one processor is configured to: When the neural network includes the third weight matrix, the first weight matrix of the first processing block (706) is determined by performing a subtraction operation based on the fourth weight matrix and the average value at the output of the normalization (702).

16. The computer system according to any one of claims 12 to 15, wherein the at least one processor is configured to provide input to the compressed first processing block.

17. The computer system according to any one of claims 12 to 16, wherein the normalization block performs a standard normalization function (206).

18. The computer system according to any one of claims 12 to 17, wherein the at least one processor is configured to: The compressed neural network is used to generate an output from an input, the input of which includes at least one of the following: image data, video data, audio data, text data, cybersecurity data, sensor data, and medical data.

19. The computer system of claim 18, wherein the output causes a physical device to perform an action based on the output.

20. A computer-readable storage medium storing computer-readable instructions configured to, when executed by at least one processor: Receive input; The input is processed using a compressed neural network, the compressed neural network including a truncated first weight matrix obtained through the following operation: The orthogonal matrix (718) is determined using a neural network applied to the calibration dataset, the neural network comprising: A first processing block (706) and a normalization block (716), the normalization block (716) having an input connected to the output of the first processing block (706). Multiplying the first weight matrix (712) of the first processing block (706) with the orthogonal matrix (718) yields a modified first weight matrix comprising multiple components ordered by relative importance. The truncated first weight matrix is ​​obtained by removing at least one component with relatively low importance from the modified first weight matrix.

Images