Interlayer feature similarity network-based sparsification method, device, medium and equipment

Abstract

The present disclosure relates to a network sparsification method and device based on inter-layer feature similarity, medium and equipment, the method comprising: collecting and storing image data sets, extracting samples in the image data sets; inputting the samples in the image data sets into a neural network for forward propagation, and extracting and storing features of the samples in the image data sets by each layer of the neural network during the forward propagation; calculating the similarity between the features of different layers of the neural network; and reducing the inter-layer similarity of the neural network by an inter-layer similarity reduction method based on center kernel alignment. The present disclosure is the first method to indirectly improve network sparseness by reducing inter-layer similarity of the network. The method is applied in the fields of neural network pre-training, neural network pruning, neural network sparse training, etc., and all have achieved performance improvement. Since the method improves the sparseness of the network, it promotes the acceleration and compression of the network.

Description

Technical Field

[0001] The present disclosure relates to the technical field of supervised training of deep learning models. More specifically, the present disclosure relates to a network sparsification method, system, medium, and device based on inter-layer feature similarity. Background Art

[0002] Deep neural networks have developed rapidly in recent years, achieving significant breakthroughs in numerous fields, including but not limited to computer vision and natural language processing, and have become a key technology in machine learning. However, deep neural networks rely heavily on over-parameterized model structures, which leads to significant waste of resources. Technical experts have proposed various approaches to address these issues, including model pruning, sparse training, and manual or automated model architecture design. Sparse training, among other approaches, aims to train sparse networks from scratch, thereby reducing training and inference overhead. It is an emerging class of network sparsification methods that reduces resource waste.

[0003] A recent study has shown a close relationship between overparameterization and inter-layer feature similarity (i.e., the similarity between features at different layers). Specifically, features at different layers in overparameterized models have significantly greater similarity. Inspired by this, this paper proposes to increase network sparsity by reducing inter-layer feature similarity, i.e., using a similarity approach to address the sparsity problem. Summary of the Invention

[0004] The purpose of the present invention is to provide a network sparsification method based on inter-layer feature similarity. Specifically, the present invention proposes a sparse regularization term (CKA-SR) based on center kernel alignment (CKA), which introduces the center kernel alignment (CKA) metric as a regularization term into the training loss. This sparse regularization term is a plug-and-play term that forces a reduction in inter-layer feature similarity and indirectly increases network sparsity. Based on this regularization term, the present invention proposes a complete network sparsification method based on inter-layer feature similarity.

[0005] To achieve the above technical objectives, the present disclosure provides a network sparsification method based on inter-layer feature similarity, comprising:

[0006] Collecting and storing image data sets, and extracting samples from the image data sets;

[0007] Inputting samples in the image dataset into a neural network for forward propagation, and extracting and storing features of the samples in the image dataset through each layer of the neural network during the forward propagation process;

[0008] Calculate the similarity between features at different layers of a neural network;

[0009] The inter-layer similarity of the neural network is reduced by an inter-layer similarity reduction method based on center kernel alignment.

[0010] Furthermore, the similarity between features of different layers of the neural network is calculated as follows:

[0011] The feature similarity between different layers of the neural network is calculated using a method based on center kernel alignment.

[0012] Furthermore, the method based on center kernel alignment for calculating feature similarity between different layers of a neural network specifically includes:

[0013] For calculating the feature maps X and Y of different layers in the neural network, the feature similarity calculator based on center kernel alignment gives the similarity calculation formula as follows:

[0014]

[0015] Among them, CKA L (X,Y) is the feature similarity based on the center kernel alignment, ||C|| F represents the Frobenius norm of C, where C is Y T X, X T X or Y T Y, X T is the transpose of feature map X, Y T is the transpose of the feature map Y.

[0016] Furthermore, reducing the inter-layer similarity of the neural network by the inter-layer similarity reduction method based on center kernel alignment specifically includes:

[0017] Backpropagation is performed using a loss function that introduces a sparse regularization term based on center kernel alignment to adjust the parameters of the neural network;

[0018] The calculated inter-layer feature similarity is introduced into the calculation of the loss function, and the calculation formula is:

[0019]

[0020] in, is the regularization term, is the sparse regularization term, and β is The weight of , S is the total number of stages in the network, s represents the current stage number, s = 1, 2, ..., S;

[0021] When S=1, N s is the total number of layers;

[0022] When S>1, N s is the number of layers in each stage s, wij is the weight of the similarity measure between the layer features of the i-th layer and the j-th layer;

[0023] i, j represents the number of layers as natural numbers;

[0024] X i and X j Represent the feature maps of the i-th layer and the j-th layer respectively.

[0025] Furthermore, the neural network includes: ResNet or Vision Transformer;

[0026] The image dataset includes: CIFAR10, CIFAR100 or ImageNet.

[0027] To achieve the above technical objectives, the present disclosure can also provide a network thinning device based on inter-layer feature similarity, including:

[0028] A data acquisition module, configured to acquire and store image data sets and extract samples from the image data sets;

[0029] A feature extraction module is used to input samples in the image data set into a neural network for forward propagation, and during the forward propagation process, features of the samples in the image data set are extracted and stored through each layer of the neural network;

[0030] A calculation module, used to calculate the similarity between features of different layers of the neural network;

[0031] An adjustment module is used to reduce the inter-layer similarity of the neural network by an inter-layer similarity reduction method based on center kernel alignment.

[0032] Furthermore, the calculation module is specifically used for:

[0033] For calculating the feature maps X and Y of different layers in the neural network, the feature similarity calculator based on center kernel alignment gives the similarity calculation formula as follows:

[0034]

[0035] Among them, CKA L (X,Y) is the feature similarity based on the center kernel alignment, ||C|| F represents the Frobenius norm of C, where C is Y T X, X T X or Y T Y, X T is the transpose of feature map X, Y T is the transpose of the feature map Y.

[0036] Furthermore, the adjustment module is specifically used to:

[0037] Backpropagation is performed using a loss function that introduces a sparse regularization term based on center kernel alignment to adjust the parameters of the neural network;

[0038] The calculated inter-layer feature similarity is introduced into the calculation of the loss function, and the calculation formula is:

[0039]

[0040] in, is the regularization term, is the sparse regularization term, and β is The weight of , S is the total number of stages in the network, s represents the current stage number, s = 1, 2, ..., S;

[0041] When S=1, N s is the total number of layers;

[0042] When S>1, N s is the number of layers in each stage s, w ij is the weight of the similarity measure between the layer features of the i-th layer and the j-th layer;

[0043] i, j represent the number of layers as natural numbers;

[0044] X i and X j Represent the feature maps of the i-th layer and the j-th layer respectively.

[0045] To achieve the above technical objectives, the present disclosure can also provide a computer storage medium on which a computer program is stored. When the computer program is executed by a processor, it is used to implement the steps of the above-mentioned network sparsification method based on inter-layer feature similarity.

[0046] To achieve the above technical objectives, the present disclosure also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and runnable on the processor. When the processor executes the computer program, the steps of the above-mentioned network sparsification method based on inter-layer feature similarity are implemented.

[0047] The beneficial effects of the present disclosure are:

[0048] The method disclosed in this paper explicitly reduces the similarity between network layers and indirectly increases the sparsity of the network. This paper provides a detailed proof that this method can indirectly increase the sparsity of the network. Specifically, we first use the information bottleneck theory to prove from the perspective of information theory that by minimizing the above regularization term Can reduce the intermediate feature representation and the mutual information between the input feature representation X Furthermore, this disclosure proves that: reducing mutual information Equivalent to reducing the Frobenius norm of the model weight matrix W This is equivalent to the network parameters being more concentrated around 0, which increases the network sparsity. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 A schematic flow chart of the method of embodiment 1 of the present disclosure is shown;

[0050] Figure 2 A schematic flow chart of the method of embodiment 1 of the present disclosure is shown;

[0051] Figure 3 A schematic diagram showing the effect of the method of Example 1 of the present disclosure is shown;

[0052] Figure 4 A schematic structural diagram of a device according to embodiment 2 of the present disclosure is shown;

[0053] Figure 5 A structural diagram of Example 4 of the present disclosure is shown. DETAILED DESCRIPTION

[0054] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are merely illustrative and are not intended to limit the scope of the present disclosure. In addition, in the following description, descriptions of well-known structures and technologies are omitted to avoid unnecessary confusion of the concepts of the present disclosure.

[0055] The accompanying drawings illustrate various schematic diagrams of structures according to embodiments of the present disclosure. These figures are not drawn to scale, and certain details are exaggerated and may be omitted for clarity. The shapes of the various regions and layers shown in the figures, as well as their relative sizes and positional relationships, are merely exemplary and may deviate in practice due to manufacturing tolerances or technical limitations. Those skilled in the art may design regions / layers with different shapes, sizes, and relative positions as needed.

[0056] Example 1:

[0057] like Figure 1 As shown:

[0058] A network sparsification method based on inter-layer feature similarity, comprising:

[0059] S101: Collect and store an image dataset, and extract samples from the image dataset;

[0060] S102: Inputting samples in the image dataset into a neural network for forward propagation, and extracting and storing features of the samples in the image dataset through each layer of the neural network during the forward propagation process;

[0061] S103: Calculate the similarity between features of different layers of the neural network;

[0062] S104: reducing the inter-layer similarity of the neural network by an inter-layer similarity reduction method based on center kernel alignment.

[0063] Furthermore, the S103 specifically includes:

[0064] The feature similarity between different layers of the neural network is calculated using a method based on center kernel alignment.

[0065] Furthermore, the method based on center kernel alignment for calculating feature similarity between different layers of a neural network specifically includes:

[0066] For calculating the feature maps X and Y of different layers in the neural network, the feature similarity calculator based on center kernel alignment gives the similarity calculation formula as follows:

[0067]

[0068] Among them, CKA L (X,Y) is the feature similarity based on the center kernel alignment, ||C|| F represents the Frobenius norm of C, where C is Y T X, X T X or Y T Y, X T is the transpose of feature map X, Y T is the transpose of the feature map Y.

[0069] Furthermore, the S104 specifically includes:

[0070] Backpropagation is performed using a loss function that introduces a sparse regularization term based on center kernel alignment to adjust the parameters of the neural network;

[0071] The calculated inter-layer feature similarity is introduced into the calculation of the loss function, and the calculation formula is:

[0072]

[0073] in, is the regularization term, is the sparse regularization term, and β is The weight of , S is the total number of stages in the network, s represents the current stage number, s = 1, 2, ..., S;

[0074] When S=1, N s is the total number of layers;

[0075] When S>1, N s is the number of layers in each stage s, w ij is the weight of the similarity measure between the layer features of the i-th layer and the j-th layer;

[0076] i, j represent the number of layers as natural numbers;

[0077] X i and X j Represent the feature maps of the i-th layer and the j-th layer respectively.

[0078] Furthermore, the neural network includes: ResNet or Vision Transformer;

[0079] The image dataset includes: CIFAR10, CIFAR100 or ImageNet.

[0080] The following combination Figure 2 Detailed explanation of the technical solution of this disclosure:

[0081] like Figure 2 As shown, the technical solution of the present disclosure includes three stages:

[0082] Phase 1:

[0083] Perform forward propagation on the network and extract features of each layer of the sample.

[0084] like Figure 2 As shown, samples from image datasets including but not limited to CIFAR10, CIFAR100, ImageNet, etc. are passed into neural network structures including but not limited to ResNet, Vision Transformer, etc. for forward propagation.

[0085] In this method, the samples are passed into the neural network for forward propagation in strict accordance with the training settings and network structure of methods such as image classification, network pruning, and network sparse training.

[0086] During each forward propagation iteration, the method disclosed herein retains and stores the sample features extracted from each layer of the neural network during the forward propagation process. Thus, the method disclosed herein extracts all inter-layer features of the neural network. These features are the intermediate features of the image samples during the forward propagation process.

[0087] Specifically,

[0088] For residual networks (such as ResNet18, ResNet34, ResNet50, etc.), the feature maps of several layers in the same stage have the same size, while the feature maps of layers in different stages have different sizes.

[0089] For Vision Transformer (such as ViT_huge, ViT_large, ViT_base, DeiT_tiny, DeiT_small, DeiT_base, etc.), the feature map sizes of all its layers are the same.

[0090] Phase 2:

[0091] Calculate the similarity between sample features extracted by each layer of the neural network.

[0092] For the inter-layer features of the above image samples, the feature similarity between them is calculated using the center kernel alignment method. The calculation rules of the feature similarity calculator based on center kernel alignment are as follows:

[0093] For feature maps X and Y, the feature similarity calculator based on center kernel alignment gives the similarity calculation formula as follows:

[0094]

[0095] Among them, CKA L (X,Y) is the feature similarity based on the center kernel alignment, ||C|| F represents the Frobenius norm of C, where C is Y T X, X T X or Y T Y, X T is the transpose of feature map X, Y T is the transpose of the feature map Y.

[0096] Phase 3:

[0097] Through the inter-layer similarity reduction method based on center kernel alignment, the similarity between neural network layers is reduced and the sparsity of the neural network is improved.

[0098] The inter-layer feature similarity calculated by the above formula is introduced into the loss calculation and used as a regularization term.

[0099] Specifically, the calculation formula of the loss function of the sparse regularization term based on center kernel alignment is:

[0100]

[0101] in, is the sparse regularization term CKA-SR, and β is The weight of . S is the number of stages in the network. For a network with only one stage, such as the DeiT network, N s is the total number of layers. For networks with multiple stages, such as ResNet, N s is the number of layers in each stage s. ij is the weight of the similarity measure between the layer features of layer i and layer j, which is optional. X0 is the input feature, and X i is the output feature of the i-th layer. It is added to the loss function as a regularization term, which forces the reduction of the sum of the similarities between all layers in the network, that is, it forces the reduction of the similarity between the layers of the network.

[0102] For the loss function that introduces the above-mentioned center kernel alignment-based sparsity regularization term, this method performs backpropagation to adjust the parameters of the neural network, which explicitly reduces the similarity between layers of the network and indirectly increases the sparsity of the network.

[0103] like Figure 3 As shown:

[0104] The method disclosed in this paper explicitly reduces the similarity between network layers and indirectly increases the sparsity of the network. This paper provides a detailed proof that this method can indirectly increase the sparsity of the network. Specifically, we first use the information bottleneck theory to prove from the perspective of information theory that by minimizing the above regularization term Can reduce the intermediate feature representation and the mutual information between the input feature representation X Furthermore, this disclosure proves that: reducing mutual information Equivalent to reducing the Frobenius norm of the model weight matrix W This is equivalent to the network parameters being more concentrated around 0, which increases the network sparsity.

[0105] Example 2:

[0106] like Figure 4 As shown, in order to achieve the above technical objectives, the present disclosure can also provide a network thinning device based on inter-layer feature similarity, including:

[0107] The data acquisition module 201 is used to acquire and store image data sets and extract samples from the image data sets;

[0108] A feature extraction module 202 is configured to input samples from the image dataset into a neural network for forward propagation, and extract and store features of the samples from the image dataset through each layer of the neural network during the forward propagation process;

[0109] A calculation module 203 is used to calculate the similarity between features of different layers of the neural network;

[0110] The adjustment module 204 is configured to reduce the inter-layer similarity of the neural network by using an inter-layer similarity reduction method based on center kernel alignment.

[0111] Among them, the data acquisition module 201 in the network thinning device based on inter-layer feature similarity described in the present disclosure is connected to the feature extraction module 202, the calculation module 203 and the adjustment module 204 in sequence.

[0112] Furthermore, the calculation module 203 is specifically configured to:

[0113] For calculating the feature maps X and Y of different layers in the neural network, the feature similarity calculator based on center kernel alignment gives the similarity calculation formula as follows:

[0114]

[0115] Among them, CKA L (X,Y) is the feature similarity based on the center kernel alignment, ||C|| F represents the Frobenius norm of C, where C is Y T X, X T X or Y T Y, X T is the transpose of feature map X, Y T is the transpose of the feature map Y.

[0116] Furthermore, the adjustment module 204 is specifically configured to:

[0117] Backpropagation is performed using a loss function that introduces a sparse regularization term based on center kernel alignment to adjust the parameters of the neural network;

[0118] The calculated inter-layer feature similarity is introduced into the calculation of the loss function, and the calculation formula is:

[0119]

[0120] in, is the regularization term, is the sparse regularization term, and β is The weight of , S is the total number of stages in the network, s represents the current stage number, s = 1, 2, ..., S;

[0121] When S=1, N s is the total number of layers;

[0122] When S>1, N s is the number of layers in each stage s, wij is the weight of the similarity measure between the layer features of the i-th layer and the j-th layer;

[0123] i, j represents the number of layers as natural numbers;

[0124] X i and X j Represent the feature maps of the i-th layer and the j-th layer respectively.

[0125] The method disclosed in this paper explicitly reduces the similarity between network layers and indirectly increases the sparsity of the network. This paper provides a detailed proof that this method can indirectly increase the sparsity of the network. Specifically, we first use the information bottleneck theory to prove from the perspective of information theory that by minimizing the above regularization term Can reduce the intermediate feature representation and the mutual information between the input feature representation X Furthermore, this disclosure proves that: reducing mutual information Equivalent to reducing the Frobenius norm of the model weight matrix W This is equivalent to the network parameters being more concentrated around 0, which increases the network sparsity.

[0126] Example 3:

[0127] The present disclosure can also provide a computer storage medium having a computer program stored thereon, which is used to implement the steps of the above-mentioned network sparsification method based on inter-layer feature similarity when the computer program is executed by a processor.

[0128] The computer storage medium of the present disclosure may be implemented using semiconductor memory, magnetic core memory, magnetic drum memory, or magnetic disk memory.

[0129] Semiconductor memory, primarily used in computers, consists of two types of semiconductor memory components: MOSFET and bipolar. MOSFET components offer high integration and simple manufacturing processes, but are relatively slow. Bipolar components, however, are complex to manufacture, consume high power, and have low integration, but are fast. The advent of NMOSFET and CMOS has made MOSFET memory a dominant component of semiconductor memory. NMOSFET is fast; for example, Intel's 1K-bit static random access memory has an access time of 45ns. CMOS, on the other hand, is more power-efficient, with a 4K-bit CMOS static memory access time of 300ns. These semiconductor memories are all random access memories (RAM), meaning they can be read and written randomly during operation. Semiconductor read-only memories (ROMs), on the other hand, can be read randomly but not written to during operation. They are used to store pre-programmed programs and data. ROMs are further divided into non-rewritable fuse-type read-only memories (PROMs) and rewritable EPROMs.

[0130] Magnetic core memory offers low cost and high reliability, with over 20 years of practical experience. Magnetic core memory was widely used as main memory before the mid-1970s. Its storage capacity can reach over 10 bits, with access times as fast as 300ns. Typical international magnetic core memory capacities range from 4MS to 8MB, with access cycles of 1.0 to 1.5μs. Even after the rapid development of semiconductor memory replaced magnetic core memory as main memory, magnetic core memory continued to be used as high-capacity expansion memory.

[0131] Magnetic drum memory is a type of external storage device that uses magnetic recording. Due to its fast information access speed and stable and reliable operation, it is gradually being replaced by disk storage due to its smaller capacity. However, it is still used as external storage for real-time process control computers and medium and large computers. To meet the needs of small and microcomputers, ultra-small magnetic drums have emerged, featuring small size, light weight, high reliability, and ease of use.

[0132] Disk storage is a type of magnetically recorded external storage. It combines the advantages of both magnetic drum and tape storage: its storage capacity is larger than that of a magnetic drum, its access speed is faster than that of tape storage, and it can be stored offline. Therefore, disks are widely used as large-capacity external storage in various computer systems. Disks are generally divided into two categories: hard disk and floppy disk.

[0133] There are many types of hard disk storage. Structurally, they are categorized as removable and fixed. Removable disks have interchangeable platters, while fixed disks have fixed platters. Both removable and fixed disks come in either multi-disc or single-disc configurations, and can be categorized as fixed-head or removable-head types. Fixed-head disks have smaller capacities, lower recording densities, and higher access speeds, but are also more expensive. Removable-head disks have higher recording densities (up to 1000-6250 bits per inch), resulting in larger capacities, but slower access speeds compared to fixed-head disks. Disk storage can reach hundreds of megabytes, with a bit density of 6250 bits per inch and a track density of 475 tracks per inch. Multi-disc removable disk storage, due to its interchangeable disk packs, offers significant off-chip capacity, high capacity, and high speed, enabling the storage of large amounts of intelligence data. These storage capacities are widely used in online information retrieval systems and database management systems.

[0134] Example 4:

[0135] The present disclosure also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the above-mentioned network sparsification method based on inter-layer feature similarity are implemented.

[0136] Figure 5 FIG. 1 is a schematic diagram of the internal structure of an electronic device in one embodiment. Figure 5As shown, the electronic device includes a processor, a storage medium, a memory and a network interface connected via a system bus. Among them, the storage medium of the computer device stores an operating system, a database and computer-readable instructions, and the database may store a control information sequence. When the computer-readable instructions are executed by the processor, the processor can implement a network sparsification method based on the similarity of inter-layer features. The processor of the electrical device is used to provide computing and control capabilities to support the operation of the entire computer device. The memory of the computer device may store computer-readable instructions. When the computer-readable instructions are executed by the processor, the processor can execute a network sparsification method based on the similarity of inter-layer features. The network interface of the computer device is used to connect and communicate with the terminal. Those skilled in the art can understand that Figure 5 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0137] The electronic device includes but is not limited to smart phones, computers, tablets, wearable smart devices, artificial intelligence devices, mobile power supplies, etc.

[0138] In some embodiments, the processor may be composed of an integrated circuit, such as a single packaged integrated circuit, or a plurality of packaged integrated circuits with the same or different functions, including one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and a combination of various control chips. The processor is the control core (Control Unit) of the electronic device, connecting the various components of the entire electronic device using various interfaces and circuits, and executing programs or modules stored in the memory (such as executing remote data reading and writing programs, etc.), as well as calling data stored in the memory, to perform various functions of the electronic device and process data.

[0139] The bus may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. The bus is configured to enable connection and communication between the memory and at least one processor, etc.

[0140] Figure 5Only the electronic device with components is shown, and it can be understood by those skilled in the art that Figure 5 The structure shown does not limit the electronic device, and may include fewer or more components than shown in the figure, or combine certain components, or arrange the components differently.

[0141] For example, although not shown, the electronic device may further include a power source (such as a battery) for powering the various components. Preferably, the power source may be logically connected to the at least one processor via a power management device, thereby implementing functions such as charge management, discharge management, and power consumption management through the power management device. The power source may further include any components such as one or more DC or AC power sources, recharging devices, power failure detection circuits, power converters or inverters, and power status indicators. The electronic device may further include various sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be described in detail here.

[0142] Furthermore, the electronic device may also include a network interface. Optionally, the network interface may include a wired interface and / or a wireless interface (such as a WI-FI interface, a Bluetooth interface, etc.), which is generally used to establish a communication connection between the electronic device and other electronic devices.

[0143] Optionally, the electronic device may further include a user interface, which may be a display or an input unit (such as a keyboard). Optionally, the user interface may also be a standard wired interface or a wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touch device. The display may also be appropriately referred to as a display screen or a display unit, which is used to display information processed in the electronic device and to display a visual user interface.

[0144] Furthermore, the computer-usable storage medium may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function, etc.; the data storage area may store data created according to the use of the blockchain node, etc.

[0145] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the module division is merely a logical function division, and other division methods may be used in actual implementation.

[0146] The modules described as separate components may or may not be physically separate, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed across multiple network elements. Some or all of the modules may be selected to achieve the purpose of the solution of this embodiment according to actual needs.

[0147] In addition, the functional modules in various embodiments of the present invention may be integrated into a single processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or hardware plus software functional modules.

[0148] The above describes the embodiments of the present disclosure. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of the present disclosure. The scope of the present disclosure is defined by the appended claims and their equivalents. Without departing from the scope of the present disclosure, those skilled in the art may make various substitutions and modifications, which are intended to fall within the scope of the present disclosure.

Claims

1. A network sparsification method based on inter-layer feature similarity, characterized in that: include: Collecting and storing image data sets, and extracting samples from the image data sets; Inputting samples in the image dataset into a neural network for forward propagation, and extracting and storing features of the samples in the image dataset through each layer of the neural network during the forward propagation process; Calculate the similarity between features at different layers of a neural network; reducing the inter-layer similarity of the neural network by an inter-layer similarity reduction method based on center kernel alignment; The method of reducing the inter-layer similarity of the neural network by using the inter-layer similarity reduction method based on center kernel alignment specifically includes: Backpropagation is performed using a loss function that introduces a sparse regularization term based on center kernel alignment to adjust the parameters of the neural network; The calculated inter-layer feature similarity is introduced into the calculation of the loss function, and the calculation formula is: in, is the regularization term, is the sparse regularization term, and β is The weight of , S is the total number of stages in the network, s represents the current stage number, s = 1, 2, ..., S; When S=1, N s is the total number of layers; When S>1, N s is the number of layers in each stage s, w ij is the weight of the similarity measure between the layer features of the i-th layer and the j-th layer; i, j represent the number of layers as natural numbers; X i and X j Represent the feature maps of the i-th layer and the j-th layer respectively.

2. The method according to claim 1, characterized in that The calculation of the similarity between features of different layers of the neural network specifically includes: The feature similarity between different layers of the neural network is calculated using a method based on center kernel alignment.

3. The method according to claim 2, characterized in that The method based on center kernel alignment is used to calculate the feature similarity between different layers of the neural network, specifically including: For calculating the feature maps X and Y of different layers in the neural network, the feature similarity calculator based on center kernel alignment gives the similarity calculation formula as follows: Among them, CKA L (X,Y) is the feature similarity based on the center kernel alignment, ||C|| F represents the Frobenius norm of C, where C is Y T X, X T X or Y T Y, X T is the transpose of feature map X, Y T is the transpose of the feature map Y.

4. The method according to any one of claims 1 to 3, characterized in that The neural network includes: ResNet or Vision Transformer; The image dataset includes: CIFAR10, CIFAR100 or ImageNet.

5. A network sparsification device based on inter-layer feature similarity, characterized in that: include: A data acquisition module, configured to acquire and store image data sets and extract samples from the image data sets; A feature extraction module is used to input samples in the image data set into a neural network for forward propagation, and during the forward propagation process, features of the samples in the image data set are extracted and stored through each layer of the neural network; A calculation module, used to calculate the similarity between features of different layers of the neural network; An adjustment module, configured to reduce the inter-layer similarity of the neural network by using an inter-layer similarity reduction method based on center kernel alignment; The adjustment module is specifically used for: Backpropagation is performed using a loss function that introduces a sparse regularization term based on center kernel alignment to adjust the parameters of the neural network; The calculated inter-layer feature similarity is introduced into the calculation of the loss function, and the calculation formula is: in, is the regularization term, is the sparse regularization term, and β is The weight of , S is the total number of stages in the network, s represents the current stage number, s = 1, 2, ..., S; When S=1, N s is the total number of layers; When S>1, N s is the number of layers in each stage s, w ij is the weight of the similarity measure between the layer features of the i-th layer and the j-th layer; i, j represent the number of layers as natural numbers; X i and X j Represent the feature maps of the i-th layer and the j-th layer respectively.

6. The device according to claim 5, characterized in that The calculation module is specifically used for: For calculating the feature maps X and Y of different layers in the neural network, the feature similarity calculator based on center kernel alignment gives the similarity calculation formula as follows: Among them, CKA L (X,Y) is the feature similarity based on the center kernel alignment, ||C|| F represents the Frobenius norm of C, where C is Y T X, X T X or Y T Y, X T is the transpose of feature map X, Y T is the transpose of the feature map Y.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the processor implements the steps corresponding to the network sparsification method based on inter-layer feature similarity described in any one of claims 1 to 4.

8. A computer storage medium having computer program instructions stored thereon, characterized in that: When the program instructions are executed by a processor, they are used to implement the steps corresponding to the network sparsification method based on inter-layer feature similarity described in any one of claims 1 to 4.

Images