A neural network construction and optimization method and system based on structure entropy minimization and a storage medium thereof

Abstract

This invention discloses a method, system, and storage medium for constructing and optimizing neural networks based on minimizing structural entropy, belonging to the fields of deep learning and structural information theory. The method first constructs the input unstructured data into a graph structure using a preset strategy; then, it constructs an encoding tree for the graph and calculates the structural entropy of the graph under the encoding tree; next, it calculates the k-dimensional structural entropy, the structural information of a given vertex, and the structural information of a given set of vertices by restricting the encoding tree type; finally, with minimizing structural entropy as the optimization objective, it jointly optimizes and trains the neural network with the task objective, and decodes the substantial structure of the data from the optimal encoding tree. This method can enhance the neural network's ability to perceive the data topology, improve model interpretability, and is applicable to data processing tasks in multiple fields such as image classification, text mining, and time series analysis.

Description

Technical Field

[0001] This invention relates to the fields of deep learning and structural information theory, specifically to a method, system, and storage medium for constructing and optimizing neural networks based on minimizing structural entropy. Background Technology

[0002] With the rapid development of deep learning technology, neural networks have achieved remarkable results in various data processing tasks. However, existing neural networks have the following technical problems: First, there is a lack of structure awareness. The loss functions of existing neural networks only focus on the difference between predicted and true values, and cannot perceive the internal topological structure and hierarchical relationships of the input data. For example, in image classification tasks, traditional convolutional neural networks can only learn the distribution of pixel values ​​and cannot understand the component structure of objects; in text classification tasks, recurrent neural networks can only learn word order and cannot capture the thematic hierarchical structure of documents.

[0003] Second, there is a lack of interpretability. In 1953, Shannon pointed out that his information measurement did not support communication network analysis and proposed establishing a structural theory of information. Brooks further stated in 2003 that "there is no theory to measure information embedded in a structure." The decision-making process of existing neural networks lacks structural explanations and cannot answer the fundamental question of "what structure does the model base its judgments on?"

[0004] Third, the theoretical foundation is weak. Current deep learning lacks a provable theoretical basis, and model performance improvement relies on empirical parameter tuning rather than theoretical guidance. Structural entropy, proposed by Li and Pan in 2016, can measure the dynamic information embedded in a graph and decode the graph's actual structure while measuring the information, but there is currently no technical solution to introduce structural entropy into neural network training.

[0005] Therefore, there is an urgent need for a new method that can introduce structural information measurement into neural network training, enhance the model's ability to perceive the data topology, and provide structural interpretability. Summary of the Invention

[0006] To overcome the above-mentioned technical problems, the present invention provides a method, system and storage medium for constructing and optimizing neural networks based on minimizing structural entropy.

[0007] To achieve the above technical solutions, the present invention provides three aspects of technology. The first aspect is a neural network construction and optimization method based on minimizing structural entropy. This method transforms unstructured data into a graph structure. After constructing an encoding tree for the graph structure and calculating the structural entropy of the graph structure under the encoding tree, the encoding tree type is restricted to calculate multiple structural information. The neural network is then jointly optimized and trained with minimizing structural entropy as the optimization objective and task. Finally, the actual structure of the data is decoded from the optimal encoding tree obtained from the training, thus completing the optimization. The unstructured data includes: image data, text data, and time-series data.

[0008] Furthermore, the method for transforming unstructured data into a graph structure is as follows: the input unstructured data is transformed into a graph structure using a preset graph construction strategy. ,in, For a set of nodes, Let the set be an edge set, wherein the preset graph construction strategies include: For image data, edges are constructed using pixels or superpixels as nodes through spatial proximity strategy, feature similarity strategy, or saliency strategy; For text data, edges are constructed using words or phrases as nodes, through co-occurrence strategies, syntactic dependency strategies, or semantic similarity strategies. For time-series data, edges are constructed using time points as nodes through visibility graph strategies, recursive graph strategies, or phase space reconstruction strategies.

[0009] Furthermore, the method for constructing the encoding tree for the graph structure is as follows: Set up the encoding tree root node tags For the entire node set ; Set up the encoding tree intermediate nodes tags And based on the feature similarity or structural importance of nodes, the labels are... Divide into a preset number of subsets, each subset corresponding to a child node. And satisfy ,constitute A division; Set up the encoding tree leaf nodes tags It is a single-point set Corresponding diagram structure a certain vertex .

[0010] Furthermore, the expression for the structural entropy of the computation graph structure under the coding tree is as follows: In the formula, Indicates label The number of cut edges; Graph structure The number of sides in; Indicates label Volume; Indicates the volume of the parent node.

[0011] Furthermore, the restriction of the coding tree type to compute various structural information includes: Limit the height of the coding tree to Resulting in a graph structure of 3D structural entropy, expressed as: Among them, when When, it degenerates into Shannon entropy with a degree distribution, i.e., static information content; when At that time, the amount of dynamic information is measured; For a given vertex ,calculate: in, Indicates a partitioning operation; Corresponding from The initial issue with personalized search; For a given set of vertices ,calculate: The optimization problem of corresponding learning or transfer learning.

[0012] Furthermore, the expression for minimizing structural entropy is as follows: In the formula, , and These represent the weight coefficients for each item.

[0013] Furthermore, the overall objective function for jointly optimizing and training the neural network by minimizing structural entropy, which is the optimization objective and task, is expressed as: in This is the objective function for the task.

[0014] The parameters in the neural network were analyzed using gradient descent. Optimize: In the formula, These are the optimized parameters; During training, the coding tree With neural network parameters Joint optimization and alternating updates are implemented, and specific methods include: Every fixed number of rounds, the coding tree structure is updated based on the current feature representation to minimize the structural entropy; With the coding tree fixed, network parameters are updated through backpropagation; The process is repeated alternately until convergence.

[0015] Furthermore, the actual structure of the data decoded from the optimal encoding tree obtained through training includes the following decoding methods: Single semantic decoding: When the optimal coding tree is unique, then each intermediate node For a given semantic unit, its label A set of nodes contained in a semantic unit; Multi-semantic decoding: When multiple optimal coding trees exist, each optimal coding tree corresponds to a semantic interpretation. These can be selected or combined according to application requirements to reveal different perspectives on the data.

[0016] Secondly, this invention provides a neural network construction and optimization system based on structural entropy minimization, comprising: a data structuring module for converting input unstructured data into a graph structure; a coding tree generation module for constructing a coding tree for the graph structure; a structural entropy calculation module for calculating the structural entropy of the graph structure under the coding tree and various types of structural information; an optimization target setting module for setting an optimization target with the goal of minimizing structural entropy; a network training module for jointly optimizing the structural entropy minimization target and the task target to train the neural network; and a structural decoding module for decoding the actual structure of the data from the optimal coding tree.

[0017] Thirdly, the present invention provides a computer-readable storage medium deployed on a memory, on which a computer program is stored, and which, when executed by a processor, implements the method of the first aspect.

[0018] Beneficial effects of the present invention This invention introduces structural entropy minimization into the neural network construction and optimization process. First, by converting input data into a graph structure and calculating its structural entropy under the encoding tree, a total optimization objective is constructed. This enables the neural network to actively perceive the internal topological structure and hierarchical relationships of the data while being task-driven, and achieves multi-granularity structural perception from static to dynamic information through k-dimensional structural entropy. Second, the substantive structure is decoded from the optimal encoding tree, with each intermediate node corresponding to a semantic unit, supporting single and multiple semantic interpretations, and enabling structural-level attribution analysis of prediction errors, providing structural interpretability for neural network decision-making. Finally, guided by the theory of structural entropy minimization, a unified framework for various structural information is used to shift structural analysis from empirical parameter tuning to provable theoretical optimization. Differentiated graph construction strategies are designed for various data types such as images, text, and time series, which can be widely applied to image classification, text mining, time series analysis, social network analysis, and other fields, effectively solving the three major problems of lack of structural perception, insufficient interpretability, and weak theoretical foundation in existing technologies. Attached Figure Description

[0019] Figure 1 This is a flowchart of the steps of the present invention; Figure 2 This is a schematic diagram of the data structuring of the present invention; Figure 3 This is a schematic diagram of the coding tree structure of the present invention; Figure 4 This is a schematic diagram illustrating the calculation of entropy for multiple structural types in this invention; Figure 5 This is a schematic diagram illustrating the multi-semantic interpretation of the present invention. Detailed Implementation

[0020] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] See Figure 1 A method for constructing and optimizing neural networks based on minimizing structural entropy includes the following steps: S1. Data structuring; The input unstructured data (images, text, and time-series data) is transformed into a graph structure using a predefined graph construction strategy. ,in, For a set of nodes, This is a set of edges; for image data, edges are constructed using pixels or superpixels as nodes, employing spatial proximity, feature similarity, or saliency strategies; for text data, edges are constructed using words or phrases as nodes, employing co-occurrence, syntactic dependency, or semantic similarity strategies; for time-series data, edges are constructed using time points as nodes, employing visibility graph, recursive graph, or phase space reconstruction strategies. The processing structure is as follows: Figure 2 As shown.

[0022] S2. Construct the coding tree; Graph constructed using S1 Used as input to build a coding tree coding tree It is a picture The hierarchical partitioning is constructed by setting up a coding tree. root node tags For the entire node set Set up the coding tree intermediate nodes tags And based on the feature similarity or structural importance of nodes, the labels are... Divide into several subsets, each subset corresponding to a child node. And these child nodes satisfy constitute A partition; setting up a coding tree leaf nodes tags It is a single-point set Corresponding diagram a certain vertex The final encoded tree For the image Each vertex in the code defines a unique codeword, encoded using a prefix-free encoding method, which can be uniquely decoded, such as... Figure 3 As shown.

[0023] S3. Calculate the structural entropy; Calculation graph In the coding tree The structural entropy is calculated using the following formula: In the formula, Indicates label The number of cut edges, i.e., the number of connections. The number of edges between internal and external nodes. ; For the image The number of sides in; Indicates label The volume, that is The sum of the degrees of all nodes in the array. ; for In the coding tree The parent node in Indicates the volume of the parent node; Indicates random walk entering substructure The probability of; Substructure The structural uncertainty relative to its parent node; The structural entropy measures the amount of information required to determine the endpoint of a random walk given the starting point information, i.e., the dynamic uncertainty of the graph.

[0024] S4. Entropy calculation for multiple types of structures; See Figure 4 According to structural information theory, various structural information can be computed by restricting the type of coding tree. The steps include: S4.1 3D structural entropy calculation Limit the height of the coding tree to , to obtain the image of 3D structural entropy, expressed as: Among them, when At that time, the one-dimensional structural entropy degenerates into a graph. The Shannon entropy of the degree distribution, i.e., the static information content; when At that time, the structural entropy measures the dynamic information content of the graph, and as... The size is increased, and the structural details are more refined; S4.2 Calculation of structural information for a given vertex; For a given vertex ,calculate: in, Indicates a partitioning operation, such as ,in, Indicates that it contains a given vertex The set, the metric corresponding to from The initial issue with personalized search; S4.3 Calculation of structural information for a given set of vertices; For a given set of vertices ,calculate: in, Indicates a partitioning operation, such as This metric can be interpreted as an optimization problem of learning or transfer learning.

[0025] S5. The optimization objective is to minimize structural entropy. Based on multi-type structural entropy, and with minimizing structural entropy as the optimization objective, a weighted sum form is constructed: in, Restrictions for different types of coding trees, These are the corresponding weighting coefficients; That is, the present invention adopts the following combination form: In the formula, , and These represent the weight coefficients of each item; This optimization objective measures the structural uncertainty under the current feature representation of the model. Minimizing this objective helps the model learn the inherent structural patterns of the data.

[0026] S6, Joint Optimization Training; The complete execution process includes the following 7 steps: S6.1 Data Structuring: According to the method in S1 of this invention, the input unstructured data is converted into a graph structure through a preset strategy. ; S6.2, Encoding Tree Construction: According to the method in S2 of this invention, the graph is constructed... Building a coding tree This creates a hierarchical division of the data; S6.3, Structural entropy calculation: According to the method in S3 of this invention, the graph is calculated. In the coding tree Structural entropy To measure the dynamic uncertainty of data; S6.4, Multi-type structural entropy calculation: According to the method in S4 of this invention, by restricting the type of the coding tree, multiple structural information is calculated, including: 3D structural entropy Given a vertex Structural information Given a set of vertices Structural information ; S6.5 Setting the structural entropy minimization target: According to the method described in S5 of this invention, based on the above-mentioned multiple types of structural entropy, a structural entropy minimization optimization target is set. ; S6.6 Joint Optimization Training: Constructing the Overall Optimization Objective ,in The objective function for the task (e.g., cross-entropy for a classification task); Use gradient descent to optimize neural network parameters. Optimize: In the formula, These are the optimized parameters; During training, the coding tree With neural network parameters Joint optimization, alternating updates: 1. At fixed intervals, update the coding tree structure based on the current feature representation to minimize the structural entropy; 2. With the coding tree fixed, update the network parameters through backpropagation; 3. Repeat the above steps alternately until convergence; S6.7, Substantive Structure Decoding: According to the method in S7 of this invention, after training is completed, the optimal encoding tree is used for decoding. The actual structure of the decoded data.

[0027] S7, Decoding the Substantive Structure; After training, from the optimal encoding tree The actual structure of the decoded data and the decoding methods include: Single semantic decoding: When the optimal encoding tree is unique, decode the hierarchical structure corresponding to that encoding tree, including each intermediate node. For a given semantic unit, its label This is the set of nodes contained in the semantic unit; by traversing the encoding tree, the hierarchical organization pattern of the data from the whole to the part can be obtained; Multi-semantic decoding: when multiple optimal coding trees exist satisfy Each optimal coding tree corresponds to a semantic interpretation. Different coding trees reveal different functional perspectives or organizational methods of the data. They can be selected or combined according to application requirements. See [link to relevant documentation]. Figure 5 ; Decision Interpretation: For a test sample, calculate its structural entropy under the optimal coding tree for each category. The category with the smallest structural entropy is the structural affiliation of the sample. When the model makes a prediction error, analyze the high-entropy region of the sample relative to the true category coding tree, locate the structural anomaly that caused the error, decode the coding tree node corresponding to the high-entropy region, and output the structural-level error interpretation.

[0028] S8, Output representation; Finally, the feature representations of all the graphs processed above are concatenated to form a matrix, with each row representing the feature representation of a graph, which can be directly used for subsequent downstream tasks such as classification, regression, and clustering. At the same time, the optimal encoding tree obtained from decoding is output as a byproduct, providing structural interpretability for model decision-making.

[0029] The above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. The available medium can be a magnetic medium (e.g., floppy disk, ATA hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. The semiconductor medium can be a solid-state ATA hard disk.

[0030] It should be understood that in the various embodiments of this application, the order of the above-mentioned processes does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0031] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0032] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0033] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0034] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0035] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable ATA hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0036] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for constructing and optimizing neural networks based on minimizing structural entropy, characterized in that, The method transforms unstructured data into a graph structure. After constructing an encoding tree for the graph structure and calculating the structural entropy of the graph structure under the encoding tree, the encoding tree type is restricted to calculate multiple structural information. The neural network is jointly optimized and trained with minimizing structural entropy as the optimization objective and task. Finally, the actual structure of the data is decoded from the optimal encoding tree obtained by training to complete the optimization. The unstructured data includes: image data, text data, and time-series data.

2. The method for constructing and optimizing a neural network based on minimizing structural entropy according to claim 1, characterized in that: The method for transforming unstructured data into a graph structure is as follows: the input unstructured data is transformed into a graph structure using a preset graph construction strategy. ,in, For a set of nodes, Let the set be an edge set, wherein the preset graph construction strategies include: For image data, edges are constructed using pixels or superpixels as nodes through spatial proximity strategy, feature similarity strategy, or saliency strategy; For text data, edges are constructed using words or phrases as nodes, through co-occurrence strategies, syntactic dependency strategies, or semantic similarity strategies. For time-series data, edges are constructed using time points as nodes through visibility graph strategies, recursive graph strategies, or phase space reconstruction strategies.

3. The method for constructing and optimizing a neural network based on minimizing structural entropy according to claim 2, characterized in that: The method for constructing the encoding tree from the graph structure is as follows: Set up the encoding tree root node tags For the entire node set ; Set up the encoding tree intermediate nodes tags And based on the feature similarity or structural importance of nodes, the labels are... Divide into a preset number of subsets, each subset corresponding to a child node. And satisfy ,constitute A division; Set up the encoding tree leaf node tags It is a single-point set Corresponding diagram structure a certain vertex .

4. The method for constructing and optimizing a neural network based on minimizing structural entropy according to claim 3, characterized in that: The expression for the structural entropy of the computation graph structure under the coding tree is as follows: In the formula, Indicates label The number of cut edges; Graph structure The number of sides in; Indicates label Volume; Indicates the volume of the parent node.

5. The method for constructing and optimizing a neural network based on minimizing structural entropy according to claim 4, characterized in that: The restricted coding tree type for calculating various structural information includes: Limit the height of the coding tree to Resulting in a graph structure of 3D structural entropy, expressed as: Among them, when When, it degenerates into Shannon entropy with a degree distribution, i.e., static information content; when At that time, the amount of dynamic information is measured; For a given vertex ,calculate: in, Indicates a partitioning operation; Corresponding from The initial issue with personalized search; For a given set of vertices ,calculate: The optimization problem of corresponding learning or transfer learning.

6. The method for constructing and optimizing a neural network based on minimizing structural entropy according to claim 5, characterized in that: The expression for minimizing structural entropy is as follows: In the formula, , and These represent the weight coefficients for each item.

7. The method for constructing and optimizing a neural network based on minimizing structural entropy according to claim 6, characterized in that: The overall objective function for jointly optimizing and training the neural network by minimizing structural entropy, which is the optimization objective and task, is expressed as follows: in This is the objective function for the task. The parameters in the neural network were analyzed using gradient descent. Optimize: In the formula, These are the optimized parameters; During training, the coding tree With neural network parameters Joint optimization and alternating updates are implemented, and specific methods include: Every fixed number of rounds, the coding tree structure is updated based on the current feature representation to minimize the structural entropy; With the coding tree fixed, network parameters are updated through backpropagation; The process is repeated alternately until convergence.

8. The method for constructing and optimizing a neural network based on minimizing structural entropy according to claim 7, characterized in that: The actual structure of the data decoded from the optimal encoding tree obtained through training, and the decoding method includes: Single semantic decoding: When the optimal coding tree is unique, then each intermediate node For a given semantic unit, its label A set of nodes contained in a semantic unit; Multi-semantic decoding: When multiple optimal coding trees exist, each optimal coding tree corresponds to a semantic interpretation. These can be selected or combined according to application requirements to reveal different perspectives on the data.

9. A neural network construction and optimization system based on structural entropy minimization, used to implement the method as described in any one of claims 1-8, characterized in that, The system includes: The data structuring module is used to convert input unstructured data into a graph structure. The coding tree generation module is used to construct coding trees from graph structures. The structural entropy calculation module is used to calculate the structural entropy of the graph structure under the coding tree and various types of structural information. The optimization objective setting module is used to set an optimization objective that minimizes structural entropy. The network training module is used to jointly optimize the structural entropy minimization objective and the task objective to train the neural network. The structure decoding module is used to decode the actual structure of data from the optimal coding tree.

10. A computer-readable storage medium, characterized in that, The storage medium is deployed on the memory, and a computer program is stored on the storage medium, which, when executed by a processor, implements the method of any one of claims 1-8.

Images