Dynamic sparsity training methods, devices, terminals and media for artificial neural networks

Abstract

This invention provides a dynamic sparse training method, apparatus, terminal, and medium for artificial neural networks. The method includes: constructing a loss function for the neural network based on activation point clustering loss and region center separation loss; wherein each layer of the neural network has multiple activation regions; mapping input features to a hyperbolic space represented by a Poincaré disk model, and performing sparse training on the neural network based on the loss function; classifying each activation point into a specific activation region based on the hyperbolic similarity between activation points and region centers; dynamically generating a sparse mask based on the clustering structure, and adding the sparse mask to the activation function of the neural network. This invention can reduce inference FLOPs and GPU memory usage by more than 70% while maintaining or improving model accuracy, shorten training convergence time by approximately 25%, and significantly enhance model interpretability through a hierarchical-friendly hyperbolic representation, meeting the needs of edge terminals for high-efficiency, low-power AI deployment.

Description

Technical Field

[0001] This invention relates to the field of neural network technology, and in particular to a dynamic sparse training method, apparatus, terminal, and medium for artificial neural networks. Background Technology

[0002] As deep learning models continue to expand, traditional dense networks face significant challenges in terms of computational cost, memory usage, and real-time response. Sparse connections help reduce model complexity, improve computational efficiency, and can enhance the model's generalization ability to some extent. Furthermore, sparse neural networks can mimic the structure of the cerebral cortex, as biological nervous systems also exhibit high sparsity.

[0003] However, among commonly used sparsity techniques, while random dropout strategies can alleviate overfitting to some extent, their randomness disrupts feature continuity, making it difficult to form hierarchical sparse gradients, thus leading to a decrease in inference accuracy (statistically, the decrease can reach 12%–15%). Furthermore, it results in the loss of clustering effects between features, making it difficult to achieve efficient computation within local devices during physical deployment (on specific devices). Modular architectures employ a static TOP-K mechanism, assigning each instance to a fixed number of experts, while experts engage in dense computation, making it impossible to flexibly adjust sparsity according to different inputs and hierarchical requirements, thus limiting system performance improvement. Current activation patterns differ significantly from the structure of local clustering and global separation in biological neural networks, with a similarity of less than 65%, lacking sufficient biological inspiration and failing to effectively support neuromorphic computing research. Traditional L1 regularization and pruning methods achieve sparsity by hard-removing weights, lacking necessary flexibility and dynamic adjustment, easily causing information loss and reducing model interpretability. Therefore, current sparsity techniques suffer from inefficiency, high power consumption, and low interpretability. Summary of the Invention

[0004] This invention provides a method, apparatus, terminal, and medium for dynamic sparsification training of artificial neural networks, in order to solve the problems of inefficiency, high power consumption, and low interpretability of current neural network sparsification techniques.

[0005] In a first aspect, embodiments of the present invention provide a method for dynamic sparsity training of an artificial neural network, comprising:

[0006] The loss function of the neural network is constructed based on activation point clustering loss and region center separation loss; each layer of the neural network has multiple activation regions, and each activation region includes a region center and multiple activation points;

[0007] The input features are mapped to the hyperbolic space represented by the Poincaré disk model, and the neural network is trained to be sparsified based on the loss function.

[0008] Based on the hyperbolic similarity between activation points and region centers, each activation point is classified into a different activation region, forming a cluster structure.

[0009] Sparse masks are dynamically generated based on clustering structures and then added to the activation function of the neural network to give the neural network a sparse structure during forward computation.

[0010] In one possible implementation, the activation point aggregation loss is:

[0011]

[0012] Among them, L agg For the clustering loss of activation points, c i Let p be the center of the i-th activated region, i∈[1,N], N is the total number of activated regions, and p is c i The activation point in the activation area, d H (p,c i ) for p and c i The distance between them.

[0013] In one possible implementation, the regional center separation loss is:

[0014]

[0015] Among them, L sep The region center separation loss is δ, which is a preset threshold, and c is the region center separation loss. i c is the region center of the i-th activated region. j Let d be the center of the j-th activation region, i,j∈[1,N], where N is the total number of activation regions. H (c i ,c j ) is c i With c j The distance between them.

[0016] In one possible implementation, the formula for classifying each activation point into its respective activation region based on the hyperbolic similarity between the activation point and the region center is as follows:

[0017]

[0018] Among them, assigned region(x j ) is the activation point x j The activation region to which it is classified, d H (x j C k ) is the activation point x j With regional center C k hyperbolic distance between them, arg mink (·) represents the activation region corresponding to the minimum value among the k hyperbolic distances.

[0019] In one possible implementation, the formula for dynamically generating a sparse mask based on the clustering structure is:

[0020]

[0021]

[0022]

[0023]

[0024]

[0025] Among them, M final (x i ,x j M(x) is a sparse mask. i ,c m M(c) is the mask within the active region. m ,c n ) is the cross-region connection mask, τ in τ cross The score is the preset threshold. in (x i ,c m ) is the activation point x i With regional center c m Relevance score cross (c m ,c n ) is the regional center c m With regional center c n Relevance score, d H (x i ,c m ) is the activation point x i With regional center c m The hyperbolic distance.

[0026] In one possible implementation, the activation function of the neural network is:

[0027]

[0028] Attention norm (x i ,x j ) = softmax(A(x i ,x j ))·M final (x i,x j )

[0029] Among them, Attention norm (x i ,x j The result of applying dynamic masking to calculate the weighted connection between each pair of activation points is softmax(A(x)). i ,x j )) is the activation point x i x j Inter-normalized attention score, A(x) i ,x j ) is the activation point x i x j Unnormalized attention score, M final (x i ,x j ) is a sparse mask.

[0030] One possible implementation involves sparse training of the neural network, including:

[0031] Based on backpropagation and gradient descent, the activation center of each activation region is dynamically adjusted.

[0032] Secondly, embodiments of the present invention provide a dynamic sparsity training apparatus for an artificial neural network, comprising:

[0033] The function building module is used to construct the loss function of the neural network based on the activation point clustering loss and the region center separation loss; wherein each layer of the neural network has multiple activation regions, and each activation region includes a region center and multiple activation points;

[0034] The sparse training module is used to map input features to the hyperbolic space represented by the Poincaré disk model and to perform sparse training on the neural network based on the loss function.

[0035] The clustering module is used to classify each activation point into each activation region based on the hyperbolic similarity between the activation point and the region center, thus forming a cluster structure.

[0036] The mask generation module is used to dynamically generate a sparse mask based on the clustering structure and add the sparse mask to the activation function of the neural network so that the neural network has a sparse structure during the forward computation process.

[0037] Thirdly, embodiments of the present invention provide a terminal, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the method as described in the first aspect or any possible implementation thereof.

[0038] Fourthly, embodiments of the present invention provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the method as described in the first aspect or any possible implementation thereof.

[0039] This invention provides a dynamic sparsity training method, apparatus, terminal, and medium for artificial neural networks. By simultaneously introducing activation point clustering loss and region center separation loss into the loss function, all points within the same activation region are first allowed to converge towards the region center, forming a local clustering effect. Then, different regions are forced to move away from each other to prevent overlap, thereby improving global discriminability and ensuring that semantic boundaries can still be accurately expressed after sparsification. A sparse attention mask is added during the activation function stage to limit the scope of attention calculation, calculating only the most relevant connections. This reduces computational load and power consumption while enhancing the efficient aggregation of local and global information while preserving key pathways. This closed-loop mechanism of "local compaction → global separation → sparse aggregation" enables the network to maintain or improve accuracy while significantly reducing FLOPs and memory usage, and provides the model with an intuitive and interpretable hierarchical representation, thus achieving efficient, low-power, and highly interpretable dynamic sparsity training. Attached Figure Description

[0040] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0041] Figure 1 This is a flowchart illustrating the implementation of a dynamic sparsity training method for artificial neural networks according to an embodiment of the present invention.

[0042] Figure 2 This is a schematic diagram of the structure of a dynamic sparsity training device for an artificial neural network provided in an embodiment of the present invention;

[0043] Figure 3 This is a schematic diagram of a terminal provided in an embodiment of the present invention. Detailed Implementation

[0044] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of the invention. However, those skilled in the art will understand that the invention can be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods are omitted so as not to obscure the description of the invention with unnecessary detail.

[0045] To make the objectives, technical solutions, and advantages of the present invention clearer, specific embodiments will be described below in conjunction with the accompanying drawings.

[0046] See Figure 1 The flowchart illustrating the implementation of the dynamic sparsity training method for artificial neural networks provided in this embodiment of the invention is described in detail below:

[0047] Step 101: Construct the loss function of the neural network based on the activation point clustering loss and the region center separation loss; wherein, each layer of the neural network has multiple activation regions, and each activation region includes a region center and multiple activation points.

[0048] In this embodiment, the activation point clustering term encourages all points within each activation region to be as close as possible to the center of that region, creating a local clustering effect. The region center separation term ensures that the distance between the centers of each activation region is not less than a set threshold (e.g., 0.2), preventing different regions from overlapping and improving global recognizability.

[0049] After considering both constraints, the total space loss is:

[0050] L spatial =λ agg ·L agg +λ sep ·L sep

[0051] Among them, L spatial For the total space loss, L agg For the clustering loss of activation points, L sep For the separation loss of the regional center, λ agg and λ sep These are weighting coefficients, which can be determined through experimental optimization.

[0052] The spatial constraints formed by the above activation point clustering loss and region center separation loss are combined with traditional L1 regularization to form structural loss, thereby preserving key feature information during the sparsification process.

[0053] Combining the task loss with the above structural constraints, we construct the total loss function:

[0054] L total =L task +λ1·Lspatial +λ2·‖W‖1

[0055] Among them, L total L represents the total loss function value. task The loss is the result for a specific task (such as classification, segmentation, or regression), where ||W||1 is the L1 regularization of the network weights, and λ1 and λ2 are regularization hyperparameters.

[0056] Step 102: Map the input features to the hyperbolic space represented by the Poincaré disk model, and perform sparse training on the neural network based on the loss function and activation function.

[0057] In this embodiment, a Poincaré disk model is used to describe the hyperbolic space, and the structural loss in step 101 is calculated using the hyperbolic distance. For any two points x and y in the space (including the activation point and the region center), the hyperbolic distance is defined as:

[0058]

[0059] Here, ||·|| represents the Euclidean norm. Mapping input features to a hyperbolic space and using hyperbolic distance to calculate structural loss not only compresses the model and computational overhead, but also provides a more human-like hierarchical representation and interpretable structure for downstream tasks while maintaining or improving accuracy.

[0060] Mapping input features to a hyperbolic space and using hyperbolic distance to calculate structural loss can produce the following effects:

[0061] 1. Strengthen hierarchical expression

[0062] Negative curvature causes the spatial volume to expand exponentially with the radius, allowing large-scale tree-like or power-law distributed data to be accommodated with low distortion within the same dimension, maintaining the consistency of the "parent-child" hierarchical relationship.

[0063] 2. Improve clustering and separation performance

[0064] Structural loss encourages samples within clusters to be compactly distributed around the center and that the centers between clusters maintain sufficient spacing. Under hyperbolic metric, similar semantics are more likely to be brought closer than Euclidean metric, while different semantics are more likely to be brought further apart, thus significantly improving the accuracy of Top-k retrieval and the clarity of classification boundaries.

[0065] 3. Reduce dimensionality and parameters

[0066] For the same capacity requirements, hyperbolic embedding can reduce the required dimensions by 30-50% compared to Euclidean space, resulting in smaller model size and lower memory usage.

[0067] 4. Stabilize gradients and accelerate convergence

[0068] The natural gradient scaling effect of hyperbolic geometry (large gradient on the outer edge and small gradient on the inner edge) matches the data usage frequency distribution, which can alleviate the gradient vanishing / exploding problem, making training more stable and convergent faster.

[0069] 5. Enhance interpretability

[0070] The geometric intuition that distance is approximately equal to hierarchical depth allows for a one-to-one correspondence between node positions, cluster radii, and semantic levels, making it easy to intuitively understand how the model organizes knowledge in visualization.

[0071] Step 103: Based on the hyperbolic similarity between the activation point and the region center, classify each activation point into each activation region to form a cluster structure.

[0072] In this embodiment, the process of classifying activation points into activation regions is as follows: First, calculate x for each activation point. j With all regional centers C k The hyperbolic distance d between H (x j C k Then select the distance x from the activation point. j The nearest regional center C k And classify the activation point into the corresponding activation area.

[0073] Step 104: Dynamically generate a sparse mask based on the clustering structure, and add the sparse mask to the activation function of the neural network so that the neural network has a sparse structure during the forward computation process.

[0074] In this embodiment, the mask is calculated based on the similarity between each activation point and the center of the corresponding region. This allows the most relevant connections to be retained during the forward computation of the neural network, thereby achieving sparsity in the neural network structure.

[0075] This invention incorporates an activation point clustering loss into the loss function to encourage all points within each activation region to be as close as possible to the center of the region, thus forming a local clustering effect; it also incorporates a region center separation loss to prevent overlap between different regions and improve global discriminability; and it introduces an attention mask into the activation function to limit the scope of attention calculation, thereby reducing computational complexity while enhancing the efficient aggregation of local and global information, thereby achieving efficient, low-power, and highly interpretable neural network sparsification.

[0076] In one possible implementation, the activation point aggregation loss is:

[0077]

[0078] Among them, L agg For the clustering loss of activation points, c iLet p be the center of the i-th activated region, i∈[1,N], N is the total number of activated regions, and p is c i The activation point in the activation area, d H (p,c i ) for p and c i The distance between them.

[0079] In this embodiment, L agg The smaller the value, the more likely the samples (activation points) within the cluster are to surround the region center c. i The higher the degree of clustering, the better the clustering effect is maintained when using hyperbolic distance compared to Euclidean distance in hierarchical or power-law data distributions. This formula encourages activation points within a region to be as close to the region center as possible, thereby forming dense feature clusters.

[0080] In one possible implementation, the regional center separation loss is:

[0081]

[0082] Among them, L sep The region center separation loss is δ, which is a preset threshold, and c is the region center separation loss. i c is the region center of the i-th activated region. j Let d be the center of the j-th activation region, i,j∈[1,N], where N is the total number of activation regions. H (c i ,c j ) is c i With c j The distance between them.

[0083] In this embodiment, to ensure that there is a sufficient distance between the centers of different active regions, this formula can be used to penalize the center when the distance between any pair of region centers is less than a preset threshold δ (e.g., 0.2), thereby causing the centers to dynamically separate.

[0084] In one possible implementation, the formula for classifying each activation point into its respective activation region based on the hyperbolic similarity between the activation point and the region center is as follows:

[0085]

[0086] Among them, assigned region(x j ) is the activation point x j The activation region to which it is classified, d H (x j C k ) is the activation point x j With regional center C k hyperbolic distance between them, arg min k(·) represents the activation region corresponding to the minimum value among the k hyperbolic distances.

[0087] In this embodiment, the activation region corresponding to each activation point is selected by hyperbolic distance, and each activation point is classified into an activation region, which can form a clear clustering structure and help simplify the model.

[0088] In one possible implementation, the formula for dynamically generating a sparse mask based on the clustering structure is:

[0089]

[0090]

[0091]

[0092]

[0093]

[0094] Among them, M final (x i ,x j M(x) is a sparse mask. i ,c m M(c) is the mask within the active region. m ,c n ) is the cross-region connection mask, τ in τ cross The score is the preset threshold. in (x i ,c m ) is the activation point x i With regional center c m Relevance score cross (c m ,c n ) is the regional center c m With regional center c n Relevance score, d H (x i ,c m ) is the activation point x i With regional center c m The hyperbolic distance.

[0095] In this embodiment, an active region mask is used to retain only the most relevant connections within the region, with a threshold τ. in Used to control the ratio of active points retained within each region. A cross-region connectivity mask is used to retain only the most relevant regions for each region, τ. crossThis is used to control the connectivity ratio between regions. The relevance score of the region centers is calculated using hyperbolic distance; the smaller the hyperbolic distance, the higher the relevance score of the region centers.

[0096] In one possible implementation, the activation function of the neural network is:

[0097]

[0098] Attention norm (x i ,x j ) = softmax(A(x i ,x j ))·M final (x i ,x j )

[0099] Among them, Attention norm (x i ,x j The result of applying dynamic masking to calculate the weighted connection between each pair of activation points is softmax(A(x)). i ,x j )) is the activation point x i x j Inter-normalized attention score, A(x) i ,x j ) is the activation point x i x j Unnormalized attention score, M final (x i ,x j ) is a sparse mask.

[0100] In this embodiment, the activation function determines the nonlinear compression form of the output of each layer. To achieve efficient feature aggregation, sparse attention is used within the same layer. Sparse attention is not an activation function in the traditional sense, but rather applies a mask or constraint to the attention weight calculation process. Based on a given query-key-value pair, it redistributes feature importance, making most weights zero through masking, forming dynamic sparse connections, so that only a small portion of query-key pairs participate in the weighted summation. Using a predefined binary mask, attention calculation can be limited to local or specified global locations. Therefore, sparse attention can be viewed as a learnable, input-dependent sparse activation gating, complementing static activation functions such as ReLU / Swish: the former controls connection sparsity, while the latter controls numerical sparsity.

[0101] During network training, the loss function simultaneously updates the ordinary weights, attention parameters, and the learnable part of the mask (if any), and further enhances sparsity through gradient clipping or L1 regularization.

[0102] Attention scores are normalized using softmax, and weighted connectivity between each pair of activations is calculated using dynamic masks. Hyperbolic distance is used to calculate the unnormalized attention scores between activations; a smaller hyperbolic distance indicates higher similarity. In hyperbolic space with negative curvature, the perimeter / volume expands exponentially with each increase in radius, making it ideal for representing tree or hierarchical structures. If two points are on the same hierarchical branch, they will be "closer" in hyperbolic space than in Euclidean space, thus ensuring that the similarity assessment aligns with the true hierarchy.

[0103] One possible implementation involves sparse training of the neural network, including:

[0104] Based on backpropagation and gradient descent, the activation center of each activation region is dynamically adjusted.

[0105] In this embodiment, during network training, the activation region center is considered one of the model parameters. It is not only used as an initial parameter setting, but its update method is similar to that of network weights, continuously updated through backpropagation and gradient descent to achieve dynamic adjustment. This allows the centers of each layer to adapt to changes in the distribution of different input features. This not only captures changes in input features but also automatically optimizes local clustering and global separation effects during training.

[0106] The initialization method for the region centers uses Xavier uniform sampling, and the mathematical description of the dynamic adjustment of the active region centers is as follows:

[0107]

[0108] Where η is a preset parameter.

[0109] In one specific embodiment, this method can achieve tumor screening and assisted diagnosis based on medical images. The neural network is used for tumor screening and assisted diagnosis using CT, MRI, or ultrasound images, and the specific process is as follows:

[0110] (1) Input preprocessing:

[0111] (a) Perform preprocessing operations such as normalization, noise reduction, and ROI segmentation on the original medical images;

[0112] (b) Use convolutional neural networks or wavelet transform to extract multidimensional features and initially generate a set of activation points.

[0113] (2) Network construction and hierarchical configuration:

[0114] A multi-layer sparse neural network is constructed based on image resolution and lesion scale. An example configuration is as follows:

[0115] hierarchy Number of regions Activation rate Connection strategy 1 32 <![CDATA[τ in =t cross =0.3]]> Preliminary integration of local features 3 16 <![CDATA[τ in =t cross =0.15]]> Convergence of local features of suspected lesions 4 8 <![CDATA[τ in =t cross =0.05]]> Global information fusion and diagnostic decision support

[0116] Each layer is embedded with:

[0117] • Hyperbolic embedding layer: maps input features to a hyperbolic space;

[0118] • Sparse attention module: Aggregates local information;

[0119] • Dynamic Activation Center Adjustment Module: Updates the activation region center CI to achieve better aggregation and separation.

[0120] The connection strategy mentioned above is an implementation detail at the network structure design level. Its purpose is to improve the efficiency of information flow or align the receptive field by utilizing the connection methods between different layers or different regions. The goal of sparse training is to reduce the amount of computation and storage by setting the weights or activations to zero (or close to zero) through loss functions and activation mechanisms.

[0121] (3) Model training:

[0122] (a) Using the joint loss function L total Train the model;

[0123] (b) Simultaneously update network weights and activate regional centers. i Other parameters;

[0124] (c) During training, the inference speed, memory usage and diagnostic accuracy are monitored using the validation set.

[0125] (4) Effect verification:

[0126] (a) Testing on publicly available medical image datasets showed that:

[0127] • Inference speed is approximately 3.0 times faster than traditional fully connected or dense networks;

[0128] • Memory usage reduced to approximately 25% of the original model;

[0129] • The accuracy of tumor detection has been improved from approximately 92% using traditional methods to approximately 97%, and the recall rate has also been significantly improved;

[0130] • The activation pattern is closer to that of biological neural networks (with a similarity of over 80%), which is helpful for clinical interpretation.

[0131] (5) Hardware acceleration and deployment:

[0132] (a) Design a fractal sparse coding hardware accelerator for edge medical devices to achieve efficient parallel computing;

[0133] (b) The dynamic load balancer further optimizes resource allocation based on real-time memory usage and response latency feedback to ensure stable operation in the clinical environment.

[0134] As shown above, this invention incorporates activation point clustering loss and region center separation loss into the loss function, enabling features to aggregate locally and separate globally within the hyperbolic space. In the forward computation, a dynamic sparse mask is generated for the attention mechanism, retaining only the most relevant connections. Network weights, sparse gating, and cluster centers are updated synchronously through joint backpropagation, and the mask is frozen for fine-tuning after reaching the target sparsity. This invention can reduce inference FLOPs and GPU memory usage by more than 70% while maintaining or improving model accuracy, shorten training convergence time by approximately 25%, and significantly enhance model interpretability through hierarchical-friendly hyperbolic representation, meeting the needs of edge terminals for high-efficiency, low-power AI deployment.

[0135] This invention addresses the shortcomings of traditional neural networks in terms of feature continuity, computational efficiency, and biological rationality by introducing a hierarchical sparsity constraint mechanism based on hyperbolic manifolds. It utilizes dual spatial constraints (activation point clustering and region center separation), dynamic adjustment of activation centers, and a hierarchical decay activation rate strategy. Particularly in the embodiment for tumor screening and assisted diagnosis, by optimizing the activation patterns of suspected lesion regions in medical images, it achieves higher detection accuracy and real-time diagnostic performance, providing a reliable, intelligent, and efficient solution for clinical applications.

[0136] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0137] The following are device embodiments of the present invention. For details not described in detail, please refer to the corresponding method embodiments described above.

[0138] Figure 2 A schematic diagram of the structure of the dynamic sparsity training device for artificial neural networks provided in an embodiment of the present invention is shown. For ease of explanation, only the parts related to the embodiment of the present invention are shown, and are described in detail below:

[0139] like Figure 2 As shown, the dynamic sparsity training device 2 for artificial neural networks includes:

[0140] Function construction module 21 is used to construct the loss function of the neural network based on activation point clustering loss and region center separation loss; wherein, each layer of the neural network has multiple activation regions, and each activation region includes a region center and multiple activation points;

[0141] The sparse training module 22 is used to map the input features to the hyperbolic space represented by the Poincaré disk model and to perform sparse training on the neural network based on the loss function.

[0142] Clustering module 23 is used to classify each activation point into each activation region based on the hyperbolic similarity between the activation point and the region center, thus forming a clustering structure.

[0143] The mask generation module 24 is used to dynamically generate a sparse mask based on the clustering structure and add the sparse mask to the activation function of the neural network so that the neural network has a sparse structure during the forward computation process.

[0144] In one possible implementation, the activation point aggregation loss is:

[0145]

[0146] Among them, L agg For the clustering loss of activation points, c i Let p be the center of the i-th activated region, i∈[1,N], N is the total number of activated regions, and p is c i The activation point in the activation area, d H (p,c i ) for p and c i The distance between them.

[0147] In one possible implementation, the regional center separation loss is:

[0148]

[0149] Among them, L sep The region center separation loss is δ, which is a preset threshold, and c is the region center separation loss. i c is the region center of the i-th activated region. j Let d be the center of the j-th activation region, i,j∈[1,N], where N is the total number of activation regions. H (c i ,c j ) is c i With c j The distance between them.

[0150] In one possible implementation, the formula for classifying each activation point into its respective activation region based on the hyperbolic similarity between the activation point and the region center is as follows:

[0151]

[0152] Among them, assigned region(x j ) is the activation point x j The activation region to which it is classified, d H (x j C k ) is the activation point x j With regional center C k hyperbolic distance between them, arg min k (·) represents the activation region corresponding to the minimum value among the k hyperbolic distances.

[0153] In one possible implementation, the formula for dynamically generating a sparse mask based on the clustering structure is:

[0154]

[0155]

[0156]

[0157]

[0158]

[0159] Among them, M final (x i ,x j M(x) is a sparse mask. i ,c m M(c) is the mask within the active region. m ,c n ) is the cross-region connection mask, τ in τ cross The score is the preset threshold. in (x i ,c m ) is the activation point x i With regional center c m Relevance score cross (c m ,c n ) is the regional center c m With regional center c n Relevance score, d H (x i ,c m ) is the activation point x i With regional center c m The hyperbolic distance.

[0160] In one possible implementation, the activation function of the neural network is:

[0161]

[0162] Attention norm (x i ,x j ) = softmax(A(x i ,x j ))·M final (x i ,x j )

[0163] Among them, Attention norm (x i ,x j The result of applying dynamic masking to calculate the weighted connection between each pair of activation points is softmax(A(x)). i ,x j )) is the activation point x i x j Inter-normalized attention score, A(x) i ,x j ) is the activation point x i x j Unnormalized attention score, M final (x i ,x j ) is a sparse mask.

[0164] In one possible implementation, the sparse training module 22 is specifically used for:

[0165] Based on backpropagation and gradient descent, the activation center of each activation region is dynamically adjusted.

[0166] This invention incorporates an activation point clustering loss into the loss function to encourage all points within each activation region to be as close as possible to the center of the region, thus forming a local clustering effect; it also incorporates a region center separation loss to prevent overlap between different regions and improve global discriminability; and it introduces an attention mask into the activation function to limit the scope of attention calculation, thereby reducing computational complexity while enhancing the efficient aggregation of local and global information, thereby achieving efficient, low-power, and highly interpretable neural network sparsification.

[0167] Figure 3 This is a schematic diagram of a terminal provided in an embodiment of the present invention. Figure 3As shown, the terminal 3 in this embodiment includes: a processor 30, a memory 31, and a computer program 32 stored in the memory 31 and executable on the processor 30. When the processor 30 executes the computer program 32, it implements the steps in the above embodiments of the dynamic sparse training methods for various artificial neural networks, for example... Figure 1 Steps 101 to 103 are shown. Alternatively, when the processor 30 executes the computer program 32, it implements the functions of each module / unit in the above-described device embodiments, for example... Figure 3 The functions of modules / units 21 to 23 shown.

[0168] For example, the computer program 32 can be divided into one or more modules / units, which are stored in the memory 31 and executed by the processor 30 to complete the present invention. The one or more modules / units can be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of the computer program 32 in the terminal 3. For example, the computer program 32 can be divided into... Figure 3 Modules / units 31 to 33 are shown.

[0169] The terminal 3 can be a desktop computer, laptop, handheld computer, or cloud server, etc. The terminal 3 may include, but is not limited to, a processor 30 and a memory 31. Those skilled in the art will understand that... Figure 3 This is merely an example of terminal 3 and does not constitute a limitation on terminal 3. It may include more or fewer components than shown, or combine certain components, or different components. For example, the terminal may also include input / output devices, network access devices, buses, etc.

[0170] The processor 30 may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0171] The memory 31 can be an internal storage unit of the terminal 3, such as a hard disk or memory of the terminal 3. The memory 31 can also be an external storage device of the terminal 3, such as a plug-in hard disk, smart media card (SMC), secure digital card (SD), flash card, etc., equipped on the terminal 3. Furthermore, the memory 31 can include both internal storage units and external storage devices of the terminal 3. The memory 31 is used to store the computer program and other programs and data required by the terminal. The memory 31 can also be used to temporarily store data that has been output or will be output.

[0172] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0173] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0174] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0175] In the embodiments provided by this invention, it should be understood that the disclosed devices / terminals and methods can be implemented in other ways. For example, the device / terminal embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0176] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0177] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0178] If the integrated module / unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the above embodiments of the present invention can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the above embodiments of the dynamic sparse training method for various artificial neural networks. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc. It should be noted that the content contained in the computer-readable medium may be appropriately added to or subtracted from the content as required by the legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer-readable medium may not include electrical carrier signals and telecommunication signals.

[0179] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A method for dynamic sparsification training of an artificial neural network, characterized in that, The artificial neural network is used for tumor screening and auxiliary diagnosis based on medical images; the method comprises: a multi-layer sparse neural network is constructed according to image resolution and lesion scale; a loss function of the neural network is constructed based on an activation point aggregation loss and a region center separation loss; wherein each layer of the neural network is provided with a plurality of activation regions, and each activation region comprises a region center and a plurality of activation points; input features are mapped to hyperbolic space represented by a Poincare disk model, and the neural network is trained based on the loss function; based on hyperbolic similarity between activation points and region centers, each activation point is classified into each activation region to form a clustering structure; a sparse mask is dynamically generated based on the clustering structure, and the sparse mask is added to an activation function of the neural network, so that the neural network has a sparse structure in a forward calculation process. 2.The method of claim 1, wherein, The activation point aggregation loss is: wherein, is the activation point gathering loss, is the region center of the th activation region, , is the total number of activation regions, is the activation point in the activation region where is located, is the hyperbolic distance between and . 3.The method of claim 1, wherein, The region center separation loss is: wherein, is a region center separation loss, is a preset threshold, is a region center of the th activated region, is a region center of the th activated region, , is a total number of activated regions, is a distance between and . 4.The method of claim 1, wherein, The formula for classifying each activation point into each activation region based on hyperbolic similarity between activation points and region centers is: wherein, is the activation point is the activation region is the activation point is the hyperbolic distance between the region center is the hyperbolic distance between the region center is the activation region is the activation region 5.The method of claim 4, wherein, The formula for dynamically generating a sparse mask based on the clustering structure is: in, For sparse masking, To activate the mask within the region, For cross-regional connection mask, , For the preset threshold, Activation point With regional center Relevance score, As regional center With regional center Relevance score, Activation point With regional center The hyperbolic distance. 6.The method of claim 1, wherein, The activation function of the neural network is: wherein, the result of the weighted connection between each pair of activated points is computed for the application of the dynamic mask, the attention score between activated points is normalized, the attention score between activated points is normalized, the attention score between activated points is normalized, the attention score between activated points is not normalized, the sparse mask. 7.The method of claim 1, wherein, The sparse training of the neural network comprises: based on back propagation and gradient descent, the activation center of each activation region is dynamically adjusted.

8. A dynamic sparsity training device for an artificial neural network, characterized in that, The artificial neural network is used for tumor screening and auxiliary diagnosis based on medical images; the device comprises: a function construction module, configured to construct a multi-layer sparse neural network according to image resolution and lesion scale, and construct a loss function of the neural network based on an activation point aggregation loss and a region center separation loss; wherein each layer of the neural network is provided with a plurality of activation regions, and each activation region comprises a region center and a plurality of activation points; a sparse training module, configured to map input features to hyperbolic space represented by a Poincare disk model, and train the neural network based on the loss function; a clustering module, configured to classify each activation point into each activation region based on hyperbolic similarity between activation points and region centers, to form a clustering structure; a mask generation module, configured to dynamically generate a sparse mask based on the clustering structure, and add the sparse mask to an activation function of the neural network, so that the neural network has a sparse structure in a forward calculation process.

9. A terminal comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, The processor executes the computer program to implement the steps of the method of any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, the computer program comprising instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 1-9. The computer program is executed by the processor to implement the steps of the method of any one of claims 1 to 7.

Images