Dynamic sparse training method and device of artificial neural network, terminal and medium

Abstract

The invention provides a dynamic sparse training method and device of an artificial neural network, a terminal and a medium. The method comprises the following steps: constructing a loss function of a neural network based on activation point aggregation loss and regional center separation loss; wherein each layer of the neural network is provided with a plurality of activation areas; mapping the input features to a hyperbolic space represented by a Poincare disk model, and performing sparse training on a neural network based on a loss function; classifying each activation point to each activation region based on hyperbolic similarity between the activation points and the center of the region; and dynamically generating a sparse mask based on the clustering structure, and adding the sparse mask into an activation function of the neural network. According to the method, the model precision can be maintained or improved, the inference FLOPs and video memory occupation can be reduced by more than 70%, the training convergence time can be shortened by about 25%, the model interpretability can be obviously enhanced by means of hierarchically-friendly hyperbolic representation, and the requirements of edge terminals on high-efficiency and low-power AI deployment can be met.

Description

Technical Field

[0001] The present invention relates to the field of neural network technology, and in particular to a dynamic sparse training method, device, terminal and medium for artificial neural network. Background Art

[0002] As deep learning models continue to expand, traditional dense networks face significant challenges in terms of computational complexity, memory usage, and real-time response. Sparse connections help reduce model complexity, improve computational efficiency, and, to a certain extent, enhance model generalization. Furthermore, sparse neural networks can mimic the structure of the cerebral cortex, as biological neural systems are also highly sparse.

[0003] However, among currently used sparsification techniques, while random dropout can alleviate overfitting to some extent, its randomness disrupts feature continuity, making it difficult to form hierarchical sparse gradients, leading to decreased inference accuracy (statistically by 12%–15%). It also results in a loss of clustering between features, making efficient computation within a specific device difficult during physical deployment. The modular architecture employs a static TOP-K mechanism, assigning each instance to a fixed number of experts. These experts perform dense computations internally, making it difficult to flexibly adjust sparsity based on varying input and hierarchical requirements, limiting system performance. Current activation patterns differ significantly from the local aggregation and global separation structures of biological neural networks, with similarities below 65%. This lack of biological inspiration makes it ineffective for brain-inspired computing research. Traditional L1 regularization and pruning methods achieve sparsification by rigidly removing weights, lacking the necessary flexibility and dynamic adjustment, leading to information loss and reduced model interpretability. Consequently, current sparsification techniques suffer from low efficiency, high power consumption, and low interpretability. Summary of the Invention

[0004] Embodiments of the present invention provide a dynamic sparsification training method, device, terminal and medium for an artificial neural network to solve the problems of low efficiency, high power consumption and low interpretability of current neural network sparsification technology.

[0005] In a first aspect, an embodiment of the present invention provides a dynamic sparsification training method for an artificial neural network, comprising:

[0006] The loss function of the neural network is constructed based on the activation point aggregation loss and the region center separation loss. Each layer of the neural network has multiple activation regions, and each activation region includes a region center and multiple activation points.

[0007] Map the input features to the hyperbolic space represented by the Poincare disk model, and perform sparse training on the neural network based on the loss function;

[0008] Based on the hyperbolic similarity between the activation point and the region center, each activation point is classified into each activation region to form a cluster structure;

[0009] A sparse mask is dynamically generated based on the clustering structure, and the sparse mask is added to the activation function of the neural network to make the neural network have a sparse structure during the forward calculation process.

[0010] In one possible implementation, the activation point aggregation loss is:

[0011]

[0012] Among them, L agg is the activation point aggregation loss, c i is the center of the i-th activated region, i∈[1,N], N is the total number of activated regions, and p is c i The activation point in the activation area, d H (p,c i ) for p and c i The distance between them.

[0013] In one possible implementation, the region center separation loss is:

[0014]

[0015] Among them, L sep is the regional center separation loss, δ is the preset threshold, c i is the center of the i-th activated area, c j is the center of the jth activated region, i,j∈[1,N], N is the total number of activated regions, d H (c i ,c j ) is c i with c j The distance between them.

[0016] In one possible implementation, the formula for classifying each activation point into each activation region based on the hyperbolic similarity between the activation point and the region center is:

[0017]

[0018] Among them, assigned region(x j ) is the activation point x j Activation area classified as d H (x j ,C k ) is the activation point x j With Regional Center C k The hyperbolic distance betweenk (·) is the activation area corresponding to the minimum value among the k hyperbolic distances.

[0019] In one possible implementation, the formula for dynamically generating a sparse mask based on the clustering structure is:

[0020]

[0021]

[0022]

[0023]

[0024]

[0025] Among them, M final (x i ,x j ) is a sparse mask, M(x i ,c m ) is the mask in the activation area, M(c m ,c n ) is the cross-region connection mask, τ in , τ cross is the preset threshold, score in (x i ,c m ) is the activation point x i With regional center c m The relevance score, score cross (c m ,c n ) is the regional center c m With regional center c n The correlation score, d H (x i ,c m ) is the activation point x i With regional center c m The hyperbolic distance.

[0026] In one possible implementation, the activation function of the neural network is:

[0027]

[0028] Attention norm (x i ,x j )=softmax(A(x i ,x j ))·M final (x i,x j )

[0029] Among them, Attention norm (x i ,x j ) is the result of applying dynamic mask to calculate the weighted connection between each pair of activation points, softmax(A(x i ,x j )) is the activation point x i 、x j The normalized attention score, A(x i ,x j ) is the activation point x i 、x j The unnormalized attention score, M final (x i ,x j ) is a sparse mask.

[0030] In one possible implementation, sparse training of a neural network includes:

[0031] Based on back propagation and gradient descent, the activation center of each activation area is dynamically adjusted.

[0032] In a second aspect, an embodiment of the present invention provides a dynamic sparsification training device for an artificial neural network, comprising:

[0033] A function construction module for constructing a loss function of a neural network based on activation point aggregation loss and region center separation loss; wherein each layer of the neural network has multiple activation regions, each of which includes a region center and multiple activation points;

[0034] The sparse training module is used to map input features to the hyperbolic space represented by the Poincare disk model and perform sparse training on the neural network based on the loss function;

[0035] The clustering module is used to classify each activation point into each activation area based on the hyperbolic similarity between the activation point and the area center to form a cluster structure;

[0036] A mask generation module is used to dynamically generate a sparse mask based on the clustering structure, and add the sparse mask to the activation function of the neural network so that the neural network has a sparse structure during the forward calculation process.

[0037] In a third aspect, an embodiment of the present invention provides a terminal comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the steps of the method described in the first aspect or any possible implementation of the first aspect are implemented.

[0038] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the steps of the method described in the first aspect or any possible implementation of the first aspect.

[0039] The embodiment of the present invention provides a dynamic sparse training method, device, terminal and medium for artificial neural networks. By simultaneously introducing activation point aggregation loss and regional center separation loss in the loss function, all points in the same activation region are first allowed to converge toward the regional center to form a local aggregation effect, and then different regions are forced to move away from each other to prevent overlap of different regions, thereby improving global recognition and fundamentally ensuring that semantic boundaries can still be accurately expressed after sparsification; a sparse attention mask is added in the activation function stage to limit the scope of attention calculation and only calculate the most relevant connections, thereby reducing the amount of calculation and power consumption and strengthening the efficient aggregation of local and global information while retaining key paths. This closed-loop mechanism of "local compactness → global separation → sparse aggregation" enables the network to significantly reduce FLOPs and video memory usage while maintaining or improving accuracy, and gives the model an intuitive and interpretable hierarchical representation, thereby achieving dynamic sparse training with high efficiency, low power consumption and high interpretability. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0041] Figure 1 This is a flow chart for implementing a dynamic sparsification training method for an artificial neural network provided by one embodiment of the present invention;

[0042] Figure 2 1 is a schematic structural diagram of a dynamic sparsification training device for an artificial neural network provided by one embodiment of the present invention;

[0043] Figure 3 FIG. 4 is a schematic diagram of a terminal provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0044] In the following description, specific details such as particular system structures and techniques are provided for purposes of illustration, not limitation, to facilitate a thorough understanding of the embodiments of the present invention. However, it will be apparent to those skilled in the art that the present invention may be practiced in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted so as not to obscure the description of the present invention with unnecessary detail.

[0045] In order to make the purpose, technical solutions and advantages of the present invention more clear, specific embodiments will be described below with reference to the accompanying drawings.

[0046] See also Figure 1 , which shows a flow chart for implementing the dynamic sparse training method for an artificial neural network provided by an embodiment of the present invention, and is described in detail as follows:

[0047] Step 101: construct a loss function of a neural network based on activation point aggregation loss and region center separation loss; wherein each layer of the neural network is provided with multiple activation regions, and each activation region includes a region center and multiple activation points.

[0048] In this embodiment, the activation point clustering term is used to encourage all points within each activation region to be as close to the center of the region as possible, forming a local clustering effect. The region center separation term is used to ensure that the distance between the centers of each activation region is no less than a set threshold (such as 0.2), preventing overlap between different regions and improving global recognition.

[0049] After combining the two constraints, the total space loss is:

[0050] L spatial =λ agg ·L agg +λ sep ·L sep

[0051] Among them, L spatial is the total space loss, L agg is the activation point aggregation loss, L sep is the regional center separation loss, λ agg and λ sep is the weight coefficient, which can be determined through experimental tuning.

[0052] The spatial constraints composed of the above activation point aggregation loss and region center separation loss are combined with the traditional L1 regularization to form a structural loss, thereby maintaining key feature information during the sparsification process.

[0053] Combining the task loss with the above structural constraints forms the total loss function:

[0054] L total =L task +λ1·Lspatial + λ2 · ||W||1

[0055] where L total is the total loss function value, L task is the loss of a specific task (such as classification, segmentation or regression), ||W||1 is the L1 regularization of network weights, and λ1 and λ2 are regularization hyperparameters.

[0056] Step 102, map the input features to the hyperbolic space represented by the Poincare disk model, and train the neural network based on the loss function and the activation function.

[0057] In this embodiment, the Poincare disk model is used to describe the hyperbolic space, and the structural loss in step 101 is calculated by the hyperbolic distance. For any two points x, y in the space (including the activation point and the region center), the hyperbolic distance is defined as:

[0058]

[0059] where ||·|| represents the Euclidean norm. Mapping the input features to the hyperbolic space and calculating the structural loss using the hyperbolic distance not only compresses the model and reduces the computational overhead, but also provides a more human-cognitive hierarchical representation and interpretable structure for downstream tasks while maintaining or improving accuracy.

[0060] Mapping the input features to the hyperbolic space and calculating the structural loss using the hyperbolic distance can produce the following effects:

[0061] 1. Strengthen hierarchical expression

[0062] The negative curvature causes the volume of the space to expand exponentially with the radius, allowing large-scale tree-shaped or power-law distributed data to be accommodated with low distortion in the same dimension, maintaining the consistency of the "parent-child" hierarchical relationship.

[0063] 2. Improve clustering and separation performance

[0064] The structural loss encourages the compact distribution of intra-cluster samples around the center and the sufficient separation of inter-cluster centers. Under the hyperbolic metric, similar semantics are more easily brought together and different semantics are more easily separated than in the Euclidean metric, significantly improving the Top-k retrieval accuracy and the clarity of the classification boundary.

[0065] 3. Reduce dimensionality and parameters

[0066] Under the same capacity requirement, the hyperbolic embedding requires 30-50% less dimensionality than the Euclidean space, resulting in smaller model size and lower memory usage.

[0067] 4. Stabilize gradient and accelerate convergence

[0068] The natural gradient scaling effect of hyperbolic geometry (large gradient in the outer circle and small gradient in the inner circle) is consistent with the frequency distribution of data usage, which can alleviate the gradient vanishing / exploding problem, making training more stable and converging faster.

[0069] 5. Enhanced explainability

[0070] The geometric intuition of distance ≈ hierarchy depth allows node positions, cluster radius, and semantic levels to correspond one-to-one, making it easier to intuitively understand how the model organizes knowledge in visualization.

[0071] Step 103: Based on the hyperbolic similarity between the activation point and the region center, each activation point is classified into each activation region to form a cluster structure.

[0072] In this embodiment, the process of classifying the activation points into activation areas is as follows: first, the x value of each activation point is calculated. j With all regional centers C k The hyperbolic distance d between H (x j ,C k ), then select the distance from the activation point x j Nearest Regional Center C k , and classify the activation point into the corresponding activation area.

[0073] Step 104 : dynamically generate a sparse mask based on the clustering structure, and add the sparse mask to the activation function of the neural network so that the neural network has a sparse structure during the forward calculation process.

[0074] In this embodiment, the mask is calculated based on the similarity between each activation point and the center of the corresponding region. During the forward calculation process of the neural network, only the most relevant connections are retained, thereby achieving sparsification of the neural network structure.

[0075] The embodiment of the present invention adds activation point aggregation loss to the loss function to encourage all points in each activation area to be as close to the center of the area as possible, forming a local aggregation effect; adds regional center separation loss to prevent overlap of different areas and improve global recognition; introduces attention mask into the activation function to limit the scope of attention calculation, reduce computational complexity while enhancing the efficient aggregation of local and global information, thereby achieving efficient, low-power and highly interpretable neural network sparsification.

[0076] In one possible implementation, the activation point aggregation loss is:

[0077]

[0078] Among them, L agg is the activation point aggregation loss, c iis the center of the i-th activated region, i∈[1,N], N is the total number of activated regions, and p is c i The activation point in the activation area, d H (p,c i ) for p and c i The distance between them.

[0079] In this embodiment, L agg The smaller it is, the more samples (activation points) in the cluster are centered around the region c. i The tighter the clustering, the better. In hierarchical or power-law data distributions, hyperbolic distance can better maintain this clustering effect than Euclidean distance. This formula encourages activation points within a region to be as close to the region center as possible, thus forming tight feature clusters.

[0080] In one possible implementation, the region center separation loss is:

[0081]

[0082] Among them, L sep is the regional center separation loss, δ is the preset threshold, c i is the center of the i-th activated area, c j is the center of the jth activated region, i,j∈[1,N], N is the total number of activated regions, d H (c i ,c j ) is c i with c j The distance between them.

[0083] In this embodiment, to ensure that there is sufficient distance between the centers of different activated regions, this formula can be used to achieve that when the distance between any pair of region centers is less than a preset threshold δ (eg, 0.2), this term generates a penalty, thereby forcing the centers to dynamically separate.

[0084] In one possible implementation, the formula for classifying each activation point into each activation region based on the hyperbolic similarity between the activation point and the region center is:

[0085]

[0086] Among them, assigned region(x j ) is the activation point x j Activation area classified as d H (x j ,C k ) is the activation point x j With Regional Center C k The hyperbolic distance between k(·) is the activation area corresponding to the minimum value among the k hyperbolic distances.

[0087] In this embodiment, the activation area corresponding to each activation point is selected by using the hyperbolic distance, and each activation point is classified into the activation area, which can form a clear clustering structure and help streamline the model.

[0088] In one possible implementation, the formula for dynamically generating a sparse mask based on the clustering structure is:

[0089]

[0090]

[0091]

[0092]

[0093]

[0094] Among them, M final (x i ,x j ) is a sparse mask, M(x i ,c m ) is the mask in the activation area, M(c m ,c n ) is the cross-region connection mask, τ in , τ cross is the preset threshold, score in (x i ,c m ) is the activation point x i With regional center c m The relevance score, score cross (c m ,c n ) is the regional center c m With regional center c n The correlation score, d H (x i ,c m ) is the activation point x i With regional center c m The hyperbolic distance.

[0095] In this embodiment, the mask in the activation area is used to retain only the most relevant connections in the area, and the threshold τ in Used to control the ratio of activation points retained in each region. The cross-region connection mask is used to retain only the most relevant regions for each region, τ crossUsed to control the connection ratio between regions. The relevance score of the region center is calculated using the hyperbolic distance. The smaller the hyperbolic distance, the higher the relevance score of the region center.

[0096] In one possible implementation, the activation function of the neural network is:

[0097]

[0098] Attention norm (x i ,x j )=softmax(A(x i ,x j ))·M final (x i ,x j )

[0099] Among them, Attention norm (x i ,x j ) is the result of applying dynamic mask to calculate the weighted connection between each pair of activation points, softmax(A(x i ,x j )) is the activation point x i 、x j The normalized attention score, A(x i ,x j ) is the activation point x i 、x j The unnormalized attention score, M final (x i ,x j ) is a sparse mask.

[0100] In this embodiment, the activation function determines the nonlinear compression form of the output of each layer. In order to achieve efficient feature aggregation, sparse attention is used in the same layer. Sparse attention is not an activation function in the traditional sense, but imposes a mask or constraint on the attention weight calculation process, redistributes the feature importance based on a given Query-Key-Value, and makes most of the weights zero by masking, forming a dynamic sparse connection so that only a small number of Query-Key pairs participate in the weighted sum. Through a predefined binary mask, attention calculation can be limited to local or specified global positions. Therefore, sparse attention can be regarded as a learnable, input-dependent sparse activation gating, which complements static activation functions such as ReLU / Swish: the former controls connection sparsity, and the latter controls numerical sparsity.

[0101] During network training, the loss function simultaneously updates the normal weights, attention parameters, and the learnable part of the mask (if any), and further enforces sparsity through gradient clipping or L1 regularization.

[0102] Attention scores are normalized using softmax and a dynamic mask is applied to calculate the weighted connection between each pair of activation points. Unnormalized attention scores between activation points are calculated using hyperbolic distance, with smaller hyperbolic distances indicating higher similarity. In negatively curvatured hyperbolic space, the perimeter / volume expands exponentially with each increase in radius, making it ideal for representing tree or hierarchical structures. If two points are on the same hierarchical branch, they appear "closer" in hyperbolic space than in Euclidean space, ensuring that similarity estimates align with the true hierarchy.

[0103] In one possible implementation, sparse training of a neural network includes:

[0104] Based on back propagation and gradient descent, the activation center of each activation area is dynamically adjusted.

[0105] In this embodiment, during network training, the activation region centers are treated as model parameters. They serve not only as initial parameters but are also updated in a manner similar to network weights, continuously updated through backpropagation and gradient descent. This allows for dynamic regulation, allowing the centers of each layer to adapt to the changing distribution of input features. This not only captures changes in input features but also automatically optimizes local aggregation and global separation during training.

[0106] The initialization method of the region center adopts Xavier uniform sampling, and the mathematical description formula for dynamic adjustment of the activation region center is:

[0107]

[0108] Wherein, η is a preset parameter.

[0109] In a specific embodiment, this method can achieve tumor screening and auxiliary diagnosis based on medical images. The neural network is used for tumor screening and auxiliary diagnosis based on CT, MRI or ultrasound images. The specific process is as follows:

[0110] (1) Input preprocessing:

[0111] (a) Perform preprocessing operations such as normalization, denoising, and ROI segmentation on the original medical images;

[0112] (b) Use convolutional neural networks or wavelet transform to extract multidimensional features and preliminarily generate a set of activation points.

[0113] (2) Network construction and hierarchical configuration:

[0114] Construct a multi-layer sparse neural network based on image resolution and lesion scale. The example configuration is as follows:

[0115] Hierarchy Number of regions Activation rate Connection Strategy 1 32 <![CDATA[τ in =t cross =0.3]]> Preliminary integration of local features 3 16 <![CDATA[τ in =t cross =0.15]]> Aggregation of local features of suspected lesions 4 8 <![CDATA[τ in =t cross =0.05]]> Global information fusion and diagnostic decision support

[0116] Each layer is embedded:

[0117] Hyperbolic Embedding Layer: maps input features into a hyperbolic space;

[0118] Sparse attention module: aggregates local information;

[0119] Dynamic activation center adjustment module: updates the activation area centers ci to achieve better aggregation and separation.

[0120] The connection strategy itself is an implementation detail at the network structure design level. Its purpose is to use the connection methods between different layers or different regions to improve the efficiency of information flow or align the receptive fields. The goal of sparse training is to set weights or activations to zero (or close to zero) through loss functions and activation mechanisms, thereby reducing computational complexity and storage.

[0121] (3) Model training:

[0122] (a) Using the joint loss function L total Train the model;

[0123] (b) Simultaneously update the network weights and activate the regional center c i and other parameters;

[0124] (c) During the training process, the validation set is used to monitor indicators such as inference speed, memory usage, and diagnostic accuracy.

[0125] (4) Effect verification:

[0126] (a) Tested on a public medical imaging dataset, the results show that:

[0127] The inference speed is about 3.0 times faster than traditional fully connected or dense networks;

[0128] Memory usage is reduced to about 25% of the original model;

[0129] Tumor detection accuracy increased from approximately 92% with traditional methods to approximately 97%, with significantly improved recall rates.

[0130] The activation patterns are closer to biological neural networks (with over 80% similarity), facilitating clinical interpretation.

[0131] (5) Hardware acceleration and deployment:

[0132] (a) Design a fractal sparse coding hardware accelerator for edge medical devices to achieve efficient parallel computing;

[0133] (b) The dynamic load balancing controller further optimizes resource allocation based on real-time memory occupancy and response latency feedback to ensure stable operation in a clinical environment.

[0134] As can be seen from the above, the present invention adds activation point aggregation loss and regional center separation loss to the loss function, so that features are locally aggregated and globally separated in hyperbolic space; in the forward calculation, a dynamic sparse mask is generated for the attention mechanism, retaining only the most relevant connections; the network weights, sparse gating, and cluster centers are updated synchronously through joint backpropagation, and the mask is frozen for fine-tuning after reaching the target sparsity. While maintaining or improving model accuracy, the present invention can reduce inference FLOPs and video memory usage by more than 70%, shorten training convergence time by approximately 25%, and significantly enhance model interpretability with the help of hierarchical-friendly hyperbolic representation, meeting the edge terminal's demand for high-efficiency, low-power AI deployment.

[0135] This invention addresses the shortcomings of traditional neural networks in terms of feature continuity, computational efficiency, and biological plausibility by introducing a hierarchical sparse constraint mechanism based on hyperbolic manifolds, utilizing dual spatial constraints (clustering of activation points and separation of regional centers), dynamically adjusting activation centers, and a hierarchical activation rate attenuation strategy. In particular, in the implementation of tumor screening and auxiliary diagnosis, by optimizing activation patterns in suspected lesion areas in medical images, higher detection accuracy and real-time diagnostic performance are achieved, providing a reliable, intelligent, and efficient solution for clinical applications.

[0136] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0137] The following are device embodiments of the present invention. For details not fully described therein, reference may be made to the corresponding method embodiments described above.

[0138] Figure 2 A schematic diagram of the structure of a dynamic sparsification training device for an artificial neural network provided by an embodiment of the present invention is shown. For ease of explanation, only the parts related to the embodiment of the present invention are shown, which are described in detail as follows:

[0139] like Figure 2 As shown, the dynamic sparse training device 2 of the artificial neural network includes:

[0140] The function construction module 21 is configured to construct a loss function of the neural network based on the activation point aggregation loss and the region center separation loss, wherein each layer of the neural network is provided with a plurality of activation regions, and each activation region includes a region center and a plurality of activation points.

[0141] The sparse training module 22 is configured to map the input features to a hyperbolic space represented by a Poincare disk model, and perform sparse training on the neural network based on the loss function.

[0142] The clustering module 23 is configured to classify each activation point into each activation region based on a hyperbolic similarity between the activation point and the region center, and form a clustering structure.

[0143] The mask generation module 24 is configured to dynamically generate a sparse mask based on the clustering structure, and add the sparse mask to an activation function of the neural network, so that the neural network has a sparse structure in a forward calculation process.

[0144] In a possible implementation, the activation point aggregation loss is as follows:

[0145]

[0146] wherein L agg is the activation point aggregation loss, c i is a region center of an i-th activation region, i∈[1, N], N is a total number of the activation regions, p is an activation point in an activation region where c i is located, d H (p, c i ) is a distance between p and c i .

[0147] In a possible implementation, the region center separation loss is as follows:

[0148]

[0149] wherein L sep is the region center separation loss, δ is a preset threshold, c i is the region center of the i-th activation region, c j is the region center of a j-th activation region, i, j∈[1, N], N is the total number of the activation regions, d H (c i , c j ) is a distance between c i and c j .

[0150] In a possible implementation, based on the hyperbolic similarity between the activation point and the region center, a formula for classifying each activation point into each activation region is as follows:

[0151]

[0152] Among them, assigned region(x j ) is the activation point x j Activation area classified as d H (x j ,C k ) is the activation point x j With Regional Center C k The hyperbolic distance between k (·) is the activation area corresponding to the minimum value among the k hyperbolic distances.

[0153] In one possible implementation, the formula for dynamically generating a sparse mask based on the clustering structure is:

[0154]

[0155]

[0156]

[0157]

[0158]

[0159] Among them, M final (x i ,x j ) is a sparse mask, M(x i ,c m ) is the mask in the activation area, M(c m ,c n ) is the cross-region connection mask, τ in , τ cross is the preset threshold, score in (x i ,c m ) is the activation point x i With regional center c m The relevance score, score cross (c m ,c n ) is the regional center c m With regional center c n The correlation score, d H (x i ,c m ) is the activation point x i With regional center c m The hyperbolic distance.

[0160] In one possible implementation, the activation function of the neural network is:

[0161]

[0162] Attention norm (x i ,x j )=softmax(A(x i ,x j ))·M final (x i ,x j )

[0163] Among them, Attention norm (x i ,x j ) is the result of applying dynamic mask to calculate the weighted connection between each pair of activation points, softmax(A(x i ,x j )) is the activation point x i 、x j The normalized attention score, A(x i ,x j ) is the activation point x i 、x j The unnormalized attention score, M final (x i ,x j ) is a sparse mask.

[0164] In a possible implementation, the sparsification training module 22 is specifically configured to:

[0165] Based on back propagation and gradient descent, the activation center of each activation area is dynamically adjusted.

[0166] The embodiment of the present invention adds activation point aggregation loss to the loss function to encourage all points in each activation area to be as close to the center of the area as possible, forming a local aggregation effect; adds regional center separation loss to prevent overlap of different areas and improve global recognition; introduces attention mask into the activation function to limit the scope of attention calculation, reduce computational complexity while enhancing the efficient aggregation of local and global information, thereby achieving efficient, low-power and highly interpretable neural network sparsification.

[0167] Figure 3 Schematic diagram of a terminal provided by an embodiment of the present invention. Figure 3As shown, the terminal 3 of this embodiment includes: a processor 30, a memory 31, and a computer program 32 stored in the memory 31 and executable on the processor 30. When the processor 30 executes the computer program 32, the steps of the above-mentioned dynamic sparsification training method of each artificial neural network are implemented, such as Figure 1 Alternatively, when the processor 30 executes the computer program 32, the functions of the modules / units in the above-mentioned device embodiments are realized, for example, Figure 3 Functions of the modules / units 21 to 23 are shown.

[0168] Exemplarily, the computer program 32 may be divided into one or more modules / units, which are stored in the memory 31 and executed by the processor 30 to implement the present invention. The one or more modules / units may be a series of computer program instruction segments capable of implementing specific functions, which are used to describe the execution process of the computer program 32 in the terminal 3. For example, the computer program 32 may be divided into Figure 3 Modules / units 31 to 33 are shown.

[0169] The terminal 3 may be a computing device such as a desktop computer, a notebook, a PDA, or a cloud server. The terminal 3 may include, but is not limited to, a processor 30 and a memory 31. Those skilled in the art will understand that Figure 3 It is only an example of terminal 3 and does not constitute a limitation on terminal 3. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the terminal may also include input and output devices, network access devices, buses, etc.

[0170] The processor 30 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0171] The memory 31 may be an internal storage unit of the terminal 3, such as a hard disk or memory of the terminal 3. The memory 31 may also be an external storage device of the terminal 3, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash memory card, etc. equipped on the terminal 3. Furthermore, the memory 31 may include both an internal storage unit of the terminal 3 and an external storage device. The memory 31 is used to store the computer program and other programs and data required by the terminal. The memory 31 may also be used to temporarily store data that has been output or is about to be output.

[0172] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0173] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant description of other embodiments.

[0174] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present invention.

[0175] In the embodiments provided by the present invention, it should be understood that the disclosed devices / terminals and methods can be implemented in other ways. For example, the device / terminal embodiments described above are merely illustrative. For example, the division of the modules or units is merely a logical functional division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be through some interface, indirect coupling or communication connection of devices or units, and can be electrical, mechanical, or other forms.

[0176] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0177] In addition, the functional units in the various embodiments of the present invention may be integrated into a single processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0178] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the process in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, it can implement the steps of the above-mentioned dynamic sparsification training method embodiments of each artificial neural network. Wherein, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device that can carry the computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practices in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practices, computer-readable media does not include electrical carrier signals and telecommunication signals.

[0179] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention, and should all be included in the scope of protection of the present invention.

Claims

1. A dynamic sparsification training method for an artificial neural network, characterized in that: include: A loss function of a neural network is constructed based on activation point aggregation loss and region center separation loss; wherein each layer of the neural network is provided with multiple activation regions, each activation region including a region center and multiple activation points; Mapping the input features to a hyperbolic space represented by a Poincare disk model, and performing sparsification training on the neural network based on the loss function; Based on the hyperbolic similarity between the activation point and the region center, each activation point is classified into each activation region to form a cluster structure; A sparse mask is dynamically generated based on the clustering structure, and the sparse mask is added to the activation function of the neural network, so that the neural network has a sparse structure during the forward calculation process.

2. The dynamic sparsification training method for an artificial neural network according to claim 1, wherein: The activation point aggregation loss is: Among them, L agg is the activation point aggregation loss, c i is the center of the i-th activated region, i∈[1,N], N is the total number of activated regions, and p is c i The activation point in the activation area, d H (p,c i ) for p and c i The distance between them.

3. The dynamic sparsification training method for an artificial neural network according to claim 1, wherein: The regional center separation loss is: Among them, L sep is the regional center separation loss, δ is the preset threshold, c i is the center of the i-th activated area, c j is the center of the jth activated region, i,j∈[1,N], N is the total number of activated regions, d H (c i ,c j ) is c i with c j The distance between them.

4. The dynamic sparsification training method for an artificial neural network according to claim 1, wherein: Based on the hyperbolic similarity between the activation point and the region center, the formula for classifying each activation point into each activation region is: Among them, assigned region(x j ) is the activation point x j Activation area classified as d H (x j ,C k ) is the activation point x j With Regional Center C k The hyperbolic distance between k (·) is the activation area corresponding to the minimum value among the k hyperbolic distances.

5. The dynamic sparsification training method for an artificial neural network according to claim 4, characterized in that: The formula for dynamically generating sparse masks based on the clustering structure is: Among them, M final (x i ,x j ) is a sparse mask, M(x i ,c m ) is the mask in the activation area, M(c m ,c n ) is the cross-region connection mask, τ in , τ cross is the preset threshold, score in (x i ,c m ) is the activation point x i With regional center c m The relevance score, score cross (c m ,c n ) is the regional center c m With regional center c n The correlation score, d H (x i ,c m ) is the activation point x i With regional center c m The hyperbolic distance.

6. The dynamic sparsification training method for an artificial neural network according to claim 1, wherein: The activation function of the neural network is: Among them, Attention norm (x i ,x j ) is the result of applying dynamic mask to calculate the weighted connection between each pair of activation points, softmax(A(x i ,x j )) is the activation point x i 、x j The normalized attention score, A(x i ,x j ) is the activation point x i 、x j The unnormalized attention score, M final (x i ,x j ) is a sparse mask.

7. The dynamic sparsification training method for an artificial neural network according to claim 1, characterized in that: The performing sparse training on the neural network includes: Based on back propagation and gradient descent, the activation center of each activation area is dynamically adjusted.

8. A dynamic sparsification training device for an artificial neural network, characterized in that: include: A function construction module for constructing a loss function of a neural network based on activation point aggregation loss and region center separation loss; wherein each layer of the neural network is provided with multiple activation regions, each activation region including a region center and multiple activation points; A sparse training module, configured to map input features to a hyperbolic space represented by a Poincare disk model, and perform sparse training on the neural network based on the loss function; The clustering module is used to classify each activation point into each activation area based on the hyperbolic similarity between the activation point and the area center to form a cluster structure; A mask generation module is used to dynamically generate a sparse mask based on the clustering structure, and add the sparse mask to the activation function of the neural network so that the neural network has a sparse structure during the forward calculation process.

9. A terminal comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images