Methods, computer programs, computer systems (adaptive node removal during training of artificial neural networks)

Abstract

Systems, methods, and computer program products for adaptive node removal during training of artificial neural networks are described herein. [Solution] The method comprises the steps of: normalizing each weight of an artificial neural network; determining the entropy value of each node in the artificial neural network based on the weights of the node's connections; identifying a set of candidate nodes, each of which has the maximum entropy value of its respective layer; selecting a subset of the set of candidate nodes; and removing each candidate node from the subset from the artificial neural network.

Description

[Technical Field]

[0001] Embodiments of this disclosure relate to training an artificial neural network, and more specifically, to adaptive node removal during training of an artificial neural network. [Overview of the project] [Problems that the invention aims to solve]

[0002] [Means for solving the problem]

[0003] Embodiments of this disclosure provide a system, method, and computer program product for adaptive node removal during training of an artificial neural network. A method for adaptive node removal during training of an artificial neural network may comprise a step of normalizing each weight of the artificial neural network. The artificial neural network may comprise a plurality of nodes. The nodes may be organized into a plurality of layers. Each of the plurality of nodes may be connected to at least one node in at least one adjacent layer. Each connection may have associated weights. The method may comprise a step of determining the entropy value of each node in the artificial neural network based on the weights of the connections of that node. The method may comprise a step of identifying a set of candidate nodes. The set of candidate nodes may comprise a node in each layer having the maximum entropy value of that layer. The method may comprise a step of selecting a subset of the set of candidate nodes. Each candidate node in the subset may be connected to at least two other candidate nodes in the set. The method may comprise a step of removing each candidate node in the subset from the artificial neural network. [Brief explanation of the drawing]

[0004] [Figure 1]This flowchart illustrates an exemplary method for adaptive node removal during training of an artificial neural network, according to one or more embodiments of the present disclosure.

[0005] [Figure 2-1] Figures 2A and 2B are illustrative diagrams of various stages of training of an artificial neural network according to one or more embodiments of the present disclosure. [Figure 2-2] Figure 2C is a diagram illustrating an exemplary artificial neural network at various stages of training according to one or more embodiments of the present disclosure.

[0006] [Figure 3] A computing node according to one embodiment of the present disclosure is illustrated. [Modes for carrying out the invention]

[0007] An artificial neural network is a collection of one or more nodes. Artificial neural networks are often divided into groups of nodes called layers. A layer is a collection of one or more nodes, all of which receive input from the same layer and all of which send output to the same layer. Layers in which all of the nodes receive input and all of the nodes send output are adjacent or neighboring layers. Two nodes through which information flows (for example, the output of one node is sent to the other node and received as input) are connected. Each connection may have associated weights.

[0008] In some implementations, the associated weights of a connection can characterize the strength of the connection. A layer in which all one or more nodes receive inputs is called a preceding layer. The connection of a node to a node in its preceding layer may be called an incoming connection. A layer in which all one or more nodes transmit outputs may be called a succeeding layer. The connection of a node to a node in its succeeding layer may be called an exit connection.

[0009] The input layer is the layer that receives input from an external source within the artificial neural network. The output layer is the layer that sends output to a target outside the artificial neural network. All other layers are intermediate processing layers (i.e., hidden layers). A multilayer neural network is an artificial neural network with one or more layers. A deep neural network is a multilayer neural network with many layers.

[0010] A sparse neural network is an artificial neural network in which nodes are connected to only some (one or more) of the nodes in their adjacent layers, rather than all of them. In a sparse neural network, the information flow from the input layer to the output layer can be isolated and form paths.

[0011] A tensor is a multidimensional array of numbers. A tensor block is a contiguous subarray of elements within a tensor. Each neural network layer is associated with a parameter tensor V, a weight tensor W, an input data tensor X, an output data tensor Y, and an intermediate data tensor Z. The parameter tensor contains all the parameters that control the node activation function σ within the layer. The weight tensor contains all the weights that connect the input to the layer. The input data tensor contains all the data that the layer consumes as input. The output data tensor contains all the data that the layer computes as output. The intermediate data tensor contains any data that the layer generates as intermediate computations, such as partial sums.

[0012] The data tensors for a layer (input, output, and intermediate) can be three-dimensional, where the first two dimensions can be interpreted as encoding spatial locations, and the third dimension as encoding different features. For example, if the data tensor represents a color image, the first two dimensions encode the vertical and horizontal coordinates in the image, and the third dimension encodes the color at each location. Every element of the input data tensor X can be connected to every node by distinct weights, and therefore the weight tensor W generally has six dimensions, concatenating the three dimensions of the input data tensor (input row a, input column b, input feature c) with the three dimensions of the output data tensor (output row i, output column j, output feature k). The intermediate data tensor Z has the same shape as the output data tensor Y.

[0013] The parameter tensor V concatenates the dimensions of the three output data tensors with an additional dimension ο that indexes the parameters of the activation function σ. In some embodiments, the activation function σ does not require an additional parameter, and in that case, the additional dimension is unnecessary. However, in some embodiments, the activation function σ requires at least one additional parameter appearing in dimension ο. The activation function can be a linear or nonlinear activation function. Activation functions that may be used by the artificial neural networks described herein may include sigmoid, tanh, ReLU, leaky ReLU, parametric ReLU, Softmax, and / or other activation functions. For example, the artificial neural network described herein may have a tanh activation function.

[0014] The elements Y of the layer's output data tensor can be calculated as shown in Equation 1, where the node activation function σ is constructed from a vector of activation function parameters V[i,j,k,:], and the weighted sum Z[i,j,k] can be calculated as shown in Equation 2.

number

number

[0015] For the sake of brevity of notation, the weighted sum in Formula 2 may be referred to as the output, which is equivalent to using the linear activation function Y[i, j, k]=σ(Z[i, j, k])=Z[i, j, k] on the understanding that the same statement applies without loss of generality even when different activation functions are used.

[0016] It will be clear from the above that the neural network is in a parallel structure. Nodes within a given layer receive inputs, i.e., X having elements x i or other inputs, from one or more layers. Each node calculates its state and y∈Y based on the inputs and the weights W having elements w i . In various embodiments, the weighted sum of the inputs is adjusted by a bias b and then the result is passed through a non-linearity F(·). For example, the activation of a single node can be expressed as y=F(b+Σx i w i ).

[0017] An artificial neural network can be trained to solve a particular problem (e.g., pattern recognition) by adjusting the connection weights so that a particular class of inputs yields a desired output. Various algorithms can be used for this learning process. A particular algorithm may be suitable for a particular task such as image recognition, speech recognition, or language processing. By training the algorithm, a pattern of weights that converges towards an optimal solution for a given problem is brought about during the learning process. The purpose of such learning is to obtain a system that generalizes data that was not available during training.

[0018] Referring now to FIG. 1, there is shown a flowchart depicting an exemplary method 100 for adaptive node removal during training of an artificial neural network. The operations of method 100 presented below are intended to be illustrative. In some implementations, method 100 may be implemented with one or more additional operations not described and / or without one or more of the operations discussed. The operations of method 100 may be executed in a different order. Additionally, the order in which the operations of method 100 are illustrated in FIG. 1 and described below is not intended to be limiting.

[0019] In some implementations, method 100 is implemented within one or more processing devices (e.g., digital processors, analog processors, digital circuits designed to process information, state machines, and / or other mechanisms for electronically processing information). The one or more processing devices may include one or more devices configured by hardware, firmware, and / or software to be specially designed for performing one or more of the operations of method 100. In some implementations, the operations of method 100 are performed with respect to layers of the artificial neural network rather than individual nodes.

[0020] Operation 104 may include the step of normalizing each weight of the artificial neural network. The artificial neural network may include a plurality of nodes. The nodes may be organized into a plurality of layers. Each of the plurality of nodes may be connected to at least one node of at least one adjacent layer. Each connection may have an associated weight. Normalization of each weight may include the step of determining the minimum-maximum normalization value of each weight.

[0021] Suitable artificial neural networks include, but are not limited to, feedforward neural networks, radial basis function networks, self-organizing maps, learning vector quantization, recurrent neural networks, Hopfield networks, Boltzmann machines, echo-state networks, long-term memory, bidirectional recurrent neural networks, hierarchical recurrent neural networks, probabilistic neural networks, modular neural networks, associative neural networks, deep neural networks, deep belief networks, convolutional neural networks, convolutional deep belief networks, large-scale memory storage and retrieval neural networks, deep Boltzmann machines, deep stacking networks, tensor deep stacking networks, spike and slab-restricted Boltzmann machines, complex hierarchical deep models, deep coding networks, multilayer kernel machines, or deep Q networks.

[0022] In some implementations, the artificial neural network is a feedforward neural network. Multiple layers may consist of an input layer, an output layer, and at least one hidden layer. In some implementations, each node in at least one hidden layer has at most four connections. In some implementations, each node in the input layer has at most one connection. In some implementations, each node in the output layer has at most as many connections as there are nodes in the preceding layer.

[0023] Figures 2A to 2C show the artificial neural network 280 in different periods of training. The artificial neural network 280 may be an artificial neural network suitable for adaptive node removal by the methods described herein (e.g., method 100 as described herein with reference to Figure 1). Training the artificial neural network may include training the artificial neural network for a predetermined number of epochs. Adaptive node removal is performed on the artificial neural network 280 during at least one of the epochs.

[0024] For example, adaptive node removal is performed after every k-th epoch during training of the artificial neural network 280. The number of epochs between each node removal can be chosen to allow the artificial neural network 280 to stabilize between removals. In some implementations, adaptive node removal is performed only a predetermined number of times. For example, adaptive node removal is performed only a total of 5 times. As a further example, adaptive node removal is performed every 10th epoch for the first 50 epochs of training. The artificial neural network 280 may be trained for a total of 100 epochs. The artificial neural network 280 may be allowed to stabilize for the remaining 50 epochs of training.

[0025] For example, the first period 200 represents the first k epochs of training the artificial neural network 280. Therefore, Figure 2A depicts the nodes of the artificial neural network 280 during the first k epochs of training. For example, the second period 225 represents the second k epochs of training the artificial neural network 280. Therefore, Figure 2B depicts the nodes of the artificial neural network 280 during the second k epochs of training. For example, adaptive node removal was performed between the first period 200 and the second period 225. For example, the third period 250 represents at least the third k epochs of training the artificial neural network 280. Therefore, Figure 2C depicts the nodes of the artificial neural network 280 during the third k epochs of training. For example, adaptive node removal was performed between the second period 225 and the third period 250.

[0026] Referring to Figure 2A, an exemplary artificial neural network 280 in the midst of a first training period 200 according to one or more embodiments of the present disclosure is shown. The first period 200 may be the period before adaptive node removal is performed on the artificial neural network 280. In a non-limiting example, the artificial neural network 280 is an untrained artificial neural network in the midst of the first period 200.

[0027] The artificial neural network 200 may have nodes 202-242. Each of nodes 202-242 may be organized into one of the following layers: input layer 256, hidden layer 252, hidden layer 254, hidden layer 252, hidden layer 250, hidden layer 248, hidden layer 246, and / or output layer 244. Nodes 238, 240, and 242 may be organized into input layer 256. Node 202 may be organized into output layer 244. Nodes 204, 206, and 208 may be organized into hidden layer 246. Nodes 210, 212, 214, and 216 may be organized into hidden layer 248. Nodes 218, 220, and 222 may be organized into hidden layer 250. Nodes 224, 226, 228, and 230 may be organized into the hidden layer 252. Nodes 232, 234, and 236 may be organized into the hidden layer 254.

[0028] Nodes 238, 240, and 242 may be connected to nodes 232, 234, and 236, respectively. Thus, each node in the input layer may have one connection during the first period 200. Nodes 232, 234, and 236 may each be connected to two nodes in the hidden layer 252. Nodes 224, 226, 228, and 230 may each be connected to two nodes in the hidden layer 254 and two nodes in the hidden layer 250. Nodes 218, 220, and 222 may each be connected to two nodes in the hidden layer 252 and two nodes in the hidden layer 248. Nodes 210, 212, 214, and 216 may each be connected to two nodes in the hidden layer 250 and two nodes in the hidden layer 246. Nodes 204, 206, and 208 can each be connected to two nodes and node 202 in the hidden layer 246. Thus, each node in the hidden layers 246, 248, 250, 252, and 254 can have at most four connections (at most two to incoming nodes and at most two to outgoing nodes). Each node in the output layer 244 can have connections to each node in the hidden layer 246. Thus, each node in the output layer 244 can have as many connections as the number of nodes organized into the hidden layer 246.

[0029] Referring back to Figure 1, operation 106 may include a step of determining the entropy value of each node in the artificial neural network based on the weights of the node's connections. The node's entropy value can characterize the magnitude of the information transmitted by that node. This information may relate to the generation of output by the artificial neural network. The step of determining the entropy value of a node in the artificial neural network may include a step of determining the product of the normalized weights over all connections of the node. The step of determining the entropy value may include a step of determining the logarithm of the product. The step of determining the entropy value may include a step of determining the absolute value of the logarithm. For example, each node n in layer l l The entropy value relating to the normalized incoming and outgoing connections is

number

number

number

[0030] Operation 108 may include a step of identifying a set of candidate nodes. The set of candidate nodes may include the node in each layer having the maximum entropy value in that layer, a node having an entropy value greater than or equal to the threshold entropy value, and / or other nodes. The step of identifying a set of candidate nodes may include a step of determining the current accuracy of the artificial neural network. The threshold entropy value may be determined based on the current accuracy of the artificial neural network. The step of determining the current accuracy may include a step of comparing the values ​​output by the nodes in the output layer with the predicted outputs. The step of identifying a set of candidate nodes may include a step of determining the node having the maximum entropy value in each layer.

[0031] Operation 110 may include a step of selecting a subset of a set of candidate nodes. Each candidate node in the subset may be connected to at least two eligible nodes in the set. Eligible nodes may be nodes included in the set of candidate nodes and / or nodes having an entropy value of at least a threshold entropy value.

[0032] Operation 112 may include a step of removing nodes from the artificial neural network. The step of removing nodes may include a step of identifying one or more orphan nodes. An orphan node may be a node that lacks any or specific connections as a result of the removal of a subset. For example, an orphan node is a node that lacks all incoming connections and / or all outgoing connections as a result of the removal of a subset. A node may be identified as an orphan node because it has only incoming and / or outgoing connections to candidate nodes. Each node in a subset of the set of candidate nodes, each orphan node, and / or one or more other nodes may be removed from the artificial neural network. In some implementations, one or more orphan nodes are identified after the removal of each node in the subset. In such implementations, one or more orphan nodes have no incoming connections and / or no outgoing connections.

[0033] Referring to Figure 2B, an artificial neural network 280 in the middle of a second training period 225 according to one or more embodiments of the present disclosure is shown. The second period 225 may be a training period after a first run of adaptive node elimination. For example, nodes 202, 204, 214, 220, 226, 234 and / or 242 are identified as candidate nodes during the first run of adaptive node elimination. Node 214 may be included in a subset of the set of candidate nodes because node 220 is a candidate node and / or node 206 has an entropy value of at least a threshold entropy value. Node 226 may be included in a subset of the set of candidate nodes because nodes 220 and 234 were identified as candidate nodes. Nodes 214 and 226 may be eliminated from the artificial neural network 280 between the first period 200 and the second period 225.

[0034] Referring to FIG. 2C, an artificial neural network 280 during a third period 250 of training according to one or more embodiments of the present disclosure is shown. The third period 250 can be a training period after a second execution of adaptive node removal. For example, nodes 208, 210, 222, 228, and / or 234 are identified as candidate nodes during the second execution. Node 228 can be included within a subset of the set of candidate nodes since nodes 222 and node 234 are identified as candidate nodes. Node 220 can be identified as an orphan node because it has only incoming connections with node 228. For example, after the removal of node 228, node 220 will have no incoming connections. Nodes 220 and 228 can be removed from the artificial neural network 280 between the second period 225 and the third period 250. The remaining nodes of the artificial neural network 280 can form a path for information flow through the remaining connections. During the training of the artificial neural network 280, nodes that contribute minimally to the generation of the output by the model can be removed.

[0035] Experiments were conducted to test the effectiveness of training an artificial neural network by adaptive node removal described herein. For the experiments, a test artificial neural network was trained to restore the output of the following polynomial function: Π i 8 (a i θ i 2 + 1) where a = [0, 0, 1.15, 2.25, 0, 1.5, 0.9, 0]. Due to the coefficients, for i ∈ {3, 4, 6, 7}, only the input θ i contributes to the training, while the others have no effect on the output. The inputs are random variables θ iIt was treated as ~U(0,1). The test artificial neural network was constructed using a structure suitable for node removal during training. The test artificial neural network was constructed using a sparse network with ultra-uniform connectivity. The test artificial neural network was constructed according to one or more embodiments described in U.S. Patent Application No. 2024 / 0202508, which is incorporated entirely herein. After every 10 epochs for the first 50 epochs, each weight was normalized, and the node entropy value was determined for each node of the test artificial neural network, and nodes with high entropy values ​​were removed according to the operation of Method 100 shown in Figure 1. The test artificial neural network was trained for a total of 100 epochs using Tanh activation. After training, the test artificial neural network had 107 connections, a mean squared error of 0.0056, and a relative error of 1.29%.

[0036] For comparison, a fully connected feedforward network with one hidden layer was trained for 100 epochs using Tanh activation without node removal. After training, the fully connected feedforward network had 108 connections, a mean squared error of 0.0201, and a relative error of 2.62%. For further comparison, an artificial neural network with the same starting structure as the test artificial neural network was trained for 100 epochs using Tanh activation without node removal. After training, the artificial neural network had 176 connections, a mean squared error of 0.0071, and a relative error of 1.45%.

[0037] Referring here to Figure 3, a schematic diagram of an example of a computing node is shown. Computing node 10 is merely one example of a suitable computing node and is not intended to imply any limitation on the scope or functionality of the embodiments described herein. In any case, computing node 10 is capable of implementing and / or performing any of the functions described above.

[0038] Computing node 10 contains computer systems / servers 12 capable of operating in many other general-purpose or dedicated computing system environments or configurations. Examples of well-known computing systems, environments, and / or configurations that may be suitable for use with computer systems / servers 12 include, but are not limited to, personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputer systems, mainframe computer systems, and distributed cloud computing environments and similar systems that include any of the above systems or devices.

[0039] The computer system / server 12 can be described in the general context of computer system executable instructions, such as program modules, that are executed by the computer system. Generally, a program module may include routines, programs, objects, components, logic, and data structures that perform a specific task or implement a specific abstract data type. The computer system / server 12 may be implemented in a distributed cloud computing environment where tasks are executed by remote processing devices linked through a communication network. In a distributed cloud computing environment, program modules may reside in both local and remote computer system storage media, including memory storage devices.

[0040] As shown in Figure 3, the computer system / server 12 within the computing node 10 is shown in the form of a general-purpose computing device. The components of the computer system / server 12 may include, but are not limited to, one or more processors or processing units 16, system memory 28, and a bus 18 that connects various system components, including the system memory 28, to the processor 16.

[0041] Bus 18 represents one or more of several types of bus structures, including memory buses or memory controllers, peripheral buses, accelerated graphics ports, and processor or local buses using any of the various bus architectures. Examples of such architectures include, but are not limited to, Industry Standard Architecture (ISA) buses, Microchannel Architecture (MCA) buses, Extended ISA (EISA) buses, Video Electronics Standards Association (VESA) local buses, Peripheral Component Interconnect (PCI) buses, Peripheral Component Interconnect Express (PCIe), and Advanced Microcontroller Bus Architecture (AMBA).

[0042] The computer system / server 12 typically includes various computer system-readable media. Such media may be any available media accessible by the computer system / server 12, and may include both volatile and non-volatile media, and removable and non-removable media.

[0043] The system memory 28 may include computer system-readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache memory 32. The computer system / server 12 may further include other removable / non-removable, volatile / non-volatile computer system storage media. As just one example, the storage system 34 may be provided for reading and writing to a non-removable non-volatile magnetic medium (not shown, typically called a “hard drive”). Not shown, a magnetic disk drive may be provided for reading and writing to a removable non-volatile magnetic disk (e.g., a “floppy disk”), and an optical disk drive may be provided for reading or writing to a removable non-volatile optical disk, such as a CD-ROM, DVD-ROM, or other optical media. In such cases, each may be connected to the bus 18 by one or more data medium interfaces. As further illustrated and described below, the memory 28 may include at least one program product having a set of program modules (e.g., at least one) configured to perform the functions of the embodiments of the present disclosure.

[0044] A program / utility 40 having a set (at least one) of program modules 42 may be stored in memory 28, as well as, but not limited to, an operating system, one or more application programs, other program modules, and program data. Each of the operating system, one or more application programs, other program modules, and program data, or any combination thereof, may include an implementation of a networking environment. The program modules 42 generally perform the functions and / or methods of the embodiments described herein.

[0045] The computer system / server 12 may communicate with one or more external devices 14, such as a keyboard, pointing device, display 24, etc.; one or more devices that enable a user to interact with the computer system / server 12; and / or any devices (e.g., network card, modem, etc.) that enable the computer system / server 12 to communicate with one or more other computing devices. Such communication may occur via the input / output (I / O) interface 22. Furthermore, the computer system / server 12 may communicate with one or more networks, such as a local area network (LAN), a general wide area network (WAN), and / or a public network (e.g., the Internet), via the network adapter 20. As shown in the illustration, the network adapter 20 communicates with other components of the computer system / server 12 via the bus 18. It should be understood that other hardware and / or software components, not shown, may be used in conjunction with the computer system / server 12. Examples include, but are not limited to, microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data archive storage systems.

[0046] This disclosure may be embodied as a system, method, and / or computer program product. The computer program product may include a computer-readable storage medium (or more mediums) having computer-readable program instructions for causing a processor to execute aspects of this disclosure.

[0047] A computer-readable storage medium can be a tangible device capable of holding and storing instructions used by an instruction execution device. A computer-readable storage medium may be, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exclusive list of more specific examples of computer-readable storage media is below: portable computer diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk read-only memory (CD-ROM), digital multipurpose disks (DVDs), memory sticks, floppy disks, mechanically encoded devices such as punch cards or grooves on which instructions are recorded, and any suitable combination of the foregoing. The computer-readable storage media used herein should not be interpreted as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses passing through optical fiber cables), or electrical signals transmitted through wires.

[0048] The computer-readable program instructions described herein may be downloaded from a computer-readable storage medium to each computing / processing device, or they may be downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network may include copper transmission cables, optical transmission fibers, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface of each computing / processing device receives computer-readable program instructions from the network and transfers the computer-readable program instructions for storage on a computer-readable storage medium within each computing / processing device.

[0049] The computer-readable program instructions that perform the operations of the Disclosure may be assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk®, C++ or similar, and conventional procedural programming languages ​​such as the "C" programming language or similar programming languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or wide area network (WAN), or such connection may be made to an external computer (for example, via the Internet using an Internet service provider). In some embodiments, an electronic circuit configuration, for example, including a programmable logic circuit configuration, a field-programmable gate array (FPGA), or a programmable logic array (PLA), can execute computer-readable program instructions by personalizing the electronic circuit configuration using state information of computer-readable program instructions in order to implement an aspect of the present disclosure.

[0050] Aspects of the present disclosure are described herein with reference to flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present disclosure. It will be understood that each block in a flowchart and / or block diagram, and combinations of blocks in a flowchart and / or block diagram, can be implemented by computer-readable program instructions.

[0051] These computer-readable program instructions may be provided to the processor of a general-purpose computer, a dedicated computer, or other programmable data processing device to generate a machine, and as a result, instructions executed via the processor of the computer or other programmable data processing device create means for implementing functions / operations specified in blocks or multiple blocks of a flowchart and / or block diagram. These computer-readable program instructions may also be stored in a computer-readable storage medium that can instruct a computer, a programmable data processing device, and / or other device to function in a particular manner, thereby including a product in which the computer-readable storage medium storing the instructions includes instructions that implement modes of functions / operations specified in blocks or multiple blocks of a flowchart and / or block diagram.

[0052] Computer-readable program instructions may be loaded onto a computer, another programmable device, or another device that generates a process implemented on a computer, another programmable device, or another device, so that instructions executed on a computer, another programmable device, or another device implement a function / operation specified in one or more blocks of a flowchart and / or block diagram.

[0053] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagram may represent a module, segment, or portion of instructions containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions described in a block may be performed in a different order than those shown in the figures. For example, two blocks shown consecutively may actually be executed substantially simultaneously, or blocks may, in some cases, be executed in reverse order depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart diagram, and combinations of blocks in the block diagram and / or flowchart diagram, may be implemented by a dedicated hardware-based system that performs a specified function or action or a combination of dedicated hardware and computer instructions.

[0054] The descriptions of the various embodiments of this disclosure are presented for illustrative purposes only and are not intended to be exhaustive or limitful to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein has been selected to best describe the principles of the embodiments, their practical applications, or technical improvements to the technology available on the market, or to enable other persons skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method for adaptive node removal during training of an artificial neural network, wherein the method is: In the step of normalizing each weight of an artificial neural network, the artificial neural network has a plurality of nodes, the nodes are organized into a plurality of layers, and each of the plurality of nodes is connected to at least one node in at least one adjacent layer, where each connection has associated weights; A step of determining the entropy value of each node in the artificial neural network based on the weights of the connections of that node; The step of identifying a set of candidate nodes, where the set of candidate nodes has a node in each layer that has the maximum entropy value of that layer; The step of selecting a subset of the set of candidate nodes, wherein each candidate node in the subset is connected to at least two other candidate nodes in the set; and Steps to remove each candidate node of the subset from the artificial neural network. A method that includes [a certain feature].

2. The step of identifying one or more orphan nodes lacking connectivity by removing the subset; and The step of removing one or more orphan nodes from the artificial neural network. The method according to claim 1, further comprising:

3. The method according to claim 1, wherein the artificial neural network is a feedforward neural network.

4. The plurality of layers include an input layer, an output layer, and at least one hidden layer. Each node in the aforementioned at least one hidden layer has at most four connections. Each node in the input layer has one connection, Each node in the output layer has at most as many connections as there are nodes in the preceding layer. The method according to claim 1.

5. The method according to claim 4, wherein each node in the at least one hidden layer has connections to at most two incoming nodes and at most two outgoing nodes.

6. The method according to claim 1, wherein the normalization of each weight includes the step of determining the minimum and maximum normalized values ​​of each weight.

7. The method according to claim 1, wherein the set of candidate nodes further comprises nodes having an entropy value of at least a threshold entropy value.

8. The method according to claim 7, further comprising the step of determining the current accuracy of the artificial neural network, wherein the threshold entropy value is determined based on the current accuracy of the artificial neural network.

9. A method for training an artificial neural network, wherein the method is: The method comprises a step of training the artificial neural network for a predetermined number of epochs, wherein during at least one of the epochs, adaptive node removal is performed according to the method of any one of claims 1 to 8. method.

10. A procedure for normalizing the weights of an artificial neural network, wherein the artificial neural network has a plurality of nodes, the nodes are organized into a plurality of layers, and each of the plurality of nodes is connected to at least one node in at least one adjacent layer, where each connection has associated weights; A procedure for determining the entropy value of each node in the artificial neural network based on the weights of the connections of that node; A procedure for identifying a set of candidate nodes, wherein the set of candidate nodes comprises the node in each layer having the maximum entropy value of that layer; A procedure for selecting a subset of the set of candidate nodes, wherein each candidate node in the subset is connected to at least two other candidate nodes in the set; and A procedure for removing each candidate node of the subset from the artificial neural network. Program instructions to cause the processor to perform an operation that has the following characteristics A computer program that includes the following features.

11. The aforementioned operation is: A procedure for identifying one or more orphan nodes lacking connectivity by removing the subset; and Procedure for removing one or more orphan nodes from the artificial neural network. The computer program according to claim 10, further comprising the above.

12. The plurality of layers include an input layer, an output layer, and at least one hidden layer. Each node in the aforementioned at least one hidden layer has at most four connections. Each node in the input layer has one connection, Each node in the output layer has at most as many connections as there are nodes in the preceding layer. The computer program according to claim 10.

13. The computer program according to claim 12, wherein each node of the at least one hidden layer has connections to at most two incoming nodes and at most two outgoing nodes.

14. The computer program according to claim 10, wherein the normalization of each weight includes a procedure for determining the minimum and maximum normalized values ​​of each weight.

15. The computer program according to any one of claims 10 to 14, wherein the operation is performed during at least one epoch of training the artificial neural network, and the artificial neural network is trained for a predetermined number of epochs.

16. Processor set; One or more computer-readable storage media; and Stored on one or more of the aforementioned computer-readable storage media: A procedure for normalizing the weights of an artificial neural network, wherein the artificial neural network has a plurality of nodes, the nodes are organized into a plurality of layers, and each of the plurality of nodes is connected to at least one node in at least one adjacent layer, where each connection has associated weights; A procedure for determining the entropy value of each node in the artificial neural network based on the weights of the connections of that node; A procedure for identifying a set of candidate nodes, wherein the set of candidate nodes comprises the node in each layer having the maximum entropy value of that layer; A procedure for selecting a subset of the set of candidate nodes, wherein each candidate node in the subset is connected to at least two other candidate nodes in the set; and A procedure for removing each candidate node of the subset from the artificial neural network. Program instructions for performing an operation having A computer system equipped with the following features.

17. The aforementioned operation is: A procedure for identifying one or more orphan nodes lacking connectivity by removing the subset; and Procedure for removing one or more orphan nodes from the artificial neural network. The computer system according to claim 16, further comprising:

18. The plurality of layers include an input layer, an output layer, and at least one hidden layer. Each node in the aforementioned at least one hidden layer has at most four connections. Each node in the aforementioned input layer has at most two connections. Each node in the output layer has at most as many connections as there are nodes in the preceding layer. The computer system according to claim 16.

19. The computer system according to claim 18, wherein each node of the at least one hidden layer has connections to at most two incoming nodes and at most two outgoing nodes.

20. The computer system according to any one of claims 16 to 19, wherein the operation is performed during at least one epoch of training the artificial neural network, and the artificial neural network is trained for a predetermined number of epochs.

Images