Matrix pre-sorting method and system based on reinforcement learning and storage medium

Abstract

The invention relates to the technical field of data processing, and particularly discloses a matrix pre-sequencing method and system based on reinforcement learning and a storage medium, which are used for modeling a matrix pre-sequencing problem into a sequence decision process, and learning an optimal sequencing strategy by utilizing a reinforcement learning agent through interaction with a matrix environment. The state of the agent is defined by the structural features of the matrix, and the action corresponds to the selection of nodes that have not been eliminated. The reward function aims at minimizing the ratio of the number of filling elements to the matrix dimension in the calculation process. The performance of the method is compared with that of a traditional heuristic algorithm in an experiment, and the result shows that in specific test cases such as a large-scale circuit model, the pre-sequencing method based on reinforcement learning has the advantages in the aspects of reducing the number of filling elements and shortening the calculation time.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a matrix pre-sorting method, system, and storage medium based on reinforcement learning. Background Technology

[0002] As a fundamental and powerful mathematical tool, matrices play a crucial role in numerous fields such as scientific computing, engineering analysis, data science, machine learning, and graph theory. Whether solving large-scale linear equations, performing eigenvalue decomposition, or conducting sparse matrix-vector multiplication (SpMV), the structural properties of matrices directly affect computational efficiency, numerical stability, and storage requirements. These challenges are particularly pronounced when dealing with large-scale sparse matrices derived from complex system modeling or massive datasets.

[0003] To address these challenges, matrix pre-ordering techniques have emerged. Matrix pre-ordering is a process of transforming a given matrix A by changing the order of its rows and / or columns. The aim is to optimize the efficiency, numerical stability, or storage requirements of subsequent numerical computations, such as solving systems of linear equations or calculating eigenvalues. The core idea is that although the reordered matrix mathematically represents the same linear transformation, its structural properties (such as the distribution pattern of non-zero elements) are altered, thus affecting algorithm performance. For example, in solving sparse linear systems using direct methods, appropriate pre-ordering can significantly reduce the number of "fill-ins" in the LU or Cholesky decomposition process, thereby reducing computational cost and memory consumption. For iterative solvers, pre-ordering helps improve the condition number of the matrix or enhance the locality of the data, thus accelerating convergence. Furthermore, in graph computation and network analysis, the ordering of the adjacency matrix directly affects the execution efficiency and visualization of graph algorithms.

[0004] Traditional matrix presorting algorithms, such as the Minimum Degree (MD), Approximate Minimum Degree (AMD), Cuthill-McKee (CM) algorithm and its variants (e.g., Reverse Cuthill-McKee, RCM), and Nested Dissection (ND), are mostly based on heuristic rules or graph theory partitioning strategies. These methods have achieved significant results in specific types of matrices and applications. However, they typically rely on fixed, manually designed heuristics, and their performance can vary considerably for matrices with diverse structures, making it difficult to find a globally optimal sorting scheme. Furthermore, many heuristic methods are inherently greedy, potentially getting trapped in local optima, and have limited capabilities in multi-objective optimization (e.g., simultaneously optimizing padding and bandwidth). Summary of the Invention

[0005] This invention provides a matrix pre-sorting method, system, and storage medium based on reinforcement learning. The technical problem it solves is that traditional pre-sorting algorithms, such as minimum degree (MD) and nested partitioning (ND), rely on fixed heuristic rules, which may limit their performance and lack adaptability when dealing with matrices with diverse structures.

[0006] To address the above technical problems, this invention provides a matrix pre-sorting method based on reinforcement learning, comprising the following steps:

[0007] Transform the original matrix into an adjacency matrix of an undirected graph, where the nodes of the undirected graph represent variables in the matrix and the edges represent non-zero elements;

[0008] Input the adjacency matrix and node mask of the undirected graph into a node rating calculation network based on a graph neural network to obtain the rating of each node;

[0009] An adaptive mask is calculated based on the score of each node and the undirected graph, and candidate nodes are selected based on the adaptive mask.

[0010] Based on the candidate nodes, a maskable near-end policy optimization algorithm based on reinforcement learning agents is used to adjust the allowed action space, and the action with the highest reward is selected as the elimination node for this time.

[0011] Furthermore, the state of the reinforcement learning agent is defined by the structural features of the matrix, the action corresponds to selecting nodes that have not been eliminated, and the reward function aims to minimize the ratio of the number of filler elements to the matrix dimension during the computation process.

[0012] Furthermore, the state space of the reinforcement learning agent consists of the adjacency matrix of the current undirected graph and a mask vector that identifies which nodes have not yet been eliminated; the action space of the reinforcement learning agent consists of the nodes that have not been eliminated and are masked.

[0013] Furthermore, the calculation process of the node scoring network includes:

[0014] Calculate the initial features of each node based on the adjacency matrix and node mask of the undirected graph;

[0015] The initial features and adjacency matrix are input into the graph neural network for multi-scale feature extraction and fusion to obtain the multi-scale fused features of each node;

[0016] The score for each node is calculated based on the multi-scale fusion features and mask of each node.

[0017] Furthermore, the graph neural network includes an L-layer network structure. Except for the first layer, whose input features are the initial features, the input features of the other layers are the fused features of the input and output features of the previous layer after residual connection. The adjacency matrix is ​​input into each layer.

[0018] The processing flow of the i-th node in the l-th layer network includes the following steps:

[0019] The attention of the i-th node in this layer of the network is calculated based on the adjacency matrix and the input features of this layer.

[0020] Calculate the fusion weight of the i-th node in the network layer based on the attention and input features of the i-th node in the network layer;

[0021] The fusion features of the i-th node in this layer of the network are calculated based on the attention, input features, and fusion weights of the i-th node in this layer of the network.

[0022] The fused features are subjected to a nonlinear transformation to obtain the transformed features;

[0023] Connect the transformation features and input features of the i-th node in this layer of the network to obtain the input features of the next layer of the network;

[0024] The initial features of the i-th node and the fused features obtained by concatenating the output and input features of all layers of the network are fused to obtain the multi-scale fused features of the i-th node;

[0025] The score for each node is calculated based on the multi-scale fusion features and mask of each node.

[0026] Furthermore, an adaptive mask is calculated based on the score of each node and the undirected graph, and candidate nodes are selected according to the adaptive mask. The specific steps include:

[0027] The node scoring computation network is used as the feature extraction layer of the policy network in the maskable near-end policy optimization algorithm based on reinforcement learning agents.

[0028] Find the node with the minimum fill cost and its corresponding node mask;

[0029] Calculate the training progress based on the current number of training rounds and the total number of training rounds;

[0030] If the training progress is in the initial stage, select the node with the minimum fill cost as the candidate node; otherwise, proceed to the next step.

[0031] Calculate exploration factors based on training progress;

[0032] Calculate the adaptive threshold based on the current threshold and the exploration factor;

[0033] Nodes whose filling cost is lower than the adaptive threshold are selected as candidate nodes.

[0034] Furthermore, the exploration factor is calculated based on the training progress. Specifically, if the training progress is in the middle stage, the exploration factor is equal to an increasing function value that increases with the training progress; otherwise, the exploration factor is equal to a decreasing function value that decreases with the training progress.

[0035] Furthermore, when the dimension of the original matrix exceeds the preset dimension, a nested segmentation method is first used to divide the original matrix into multiple sub-blocks.

[0036] The present invention also provides a matrix pre-sorting system based on reinforcement learning, the key of which is: it includes a processor and a memory, the memory stores instructions, and when the processor executes the instructions, it is configured to execute the matrix pre-sorting method based on reinforcement learning.

[0037] The present invention also provides a storage medium, the key feature of which is that the storage medium stores machine-executable instructions, and when the machine-executable instructions are executed, the matrix pre-sorting method based on reinforcement learning or the matrix pre-sorting system based on reinforcement learning is used.

[0038] This invention provides a matrix pre-sorting method, system, and storage medium based on reinforcement learning (RL). It models the matrix pre-sorting problem as a sequential decision-making process, utilizing a reinforcement learning agent to learn the optimal sorting strategy through interaction with the matrix environment. The agent's state is defined by the structural features of the matrix, and the action corresponds to selecting nodes that have not been eliminated. The reward function aims to minimize the ratio of the number of fill-in elements to the matrix dimension during computation, guiding the agent's learning. Experiments compare the performance of this invention with traditional heuristic algorithms (such as AMD and nested partitioning). Results show that in specific large-scale circuit models (matrix dimension N = 35324), the reinforcement learning-based pre-sorting method exhibits significant advantages in reducing the number of fill-in elements and shortening computation time. This demonstrates the potential of reinforcement learning in exploring and learning the underlying patterns in complex matrix structures, thereby discovering better pre-sorting strategies, providing a new perspective and effective tool for solving challenges in large-scale scientific and engineering computing. Attached Figure Description

[0039] Figure 1 This is a schematic diagram of the principle of a simple matrix four-step elimination and filling method provided in the embodiment of the present invention;

[0040] Figure 2 This is a schematic diagram of the principle of simple matrix four-step elimination without generating fill provided by the embodiment of the present invention;

[0041] Figure 3This is a flowchart of a matrix pre-sorting method based on reinforcement learning provided in an embodiment of the present invention;

[0042] Figure 4 This is a diagram illustrating the process of converting a matrix into an adjacency matrix, as provided in an embodiment of the present invention.

[0043] Figure 5 This is a diagram illustrating the calculation process of the node scoring calculation network provided in this embodiment of the invention.

[0044] Figure 6 This is a comparison diagram of a 13×13 matrix before and after nested segmentation, as provided in an embodiment of the present invention.

[0045] Figure 7 This is a reward comparison diagram with adaptive mask and mask provided in an embodiment of the present invention;

[0046] Figure 8 This is a test connection diagram of the equipment motor drive system provided in an embodiment of the present invention;

[0047] Figure 9 This is a circuit equivalent model diagram provided in an embodiment of the present invention;

[0048] Figure 10 This is a comparison diagram of two sorting algorithms provided in the embodiments of the present invention. Detailed Implementation

[0049] The embodiments of the present invention are described in detail below with reference to the accompanying drawings. The embodiments are given for illustrative purposes only and should not be construed as limiting the present invention. The accompanying drawings are for reference and illustration only and do not constitute a limitation on the scope of patent protection of the present invention, because many changes can be made to the present invention without departing from the spirit and scope of the present invention.

[0050] Matrix rearrangement is achieved by left-multiplying or right-multiplying by one or more permutation matrices. An n×n permutation matrix P is a matrix composed of the identity matrix I. n A matrix obtained by arranging rows (or columns) in a certain way. It has the following properties:

[0051] Each row has only one element that is 1, and the rest of the elements are 0.

[0052] Each column has only one element that is 1, and the rest are 0.

[0053] The permutation matrix is ​​an orthogonal matrix, i.e., P T P = PP T =I n Therefore P -1 =P T T represents the transpose operation.

[0054] For a symmetric matrix, reordering the rows and columns in the same way preserves the eigenvalues ​​of the matrix.

[0055] Below is an example of solving a linear system using matrix rearrangement. The solution process includes the following steps:

[0056] 1. Model the linear system as follows:

[0057] Ax = b

[0058] Where x represents the input parameter, and the coefficient matrix of x is A∈R n×n It is a sparse matrix, and b represents the system output.

[0059] 2. Introduce the permutation matrix

[0060] Using the permutation matrix P∈R n×n (orthogonal matrix, P) -1 =P T Perform symmetric reordering:

[0061] PAP T ·(Px)=Pb

[0062] Let: A′ = PAP T The system after reordering is: x′=Px, b′=Pb.

[0063] A′x′=b′

[0064] 3. ① Cholesky decomposition (symmetric positive definite)

[0065] If A′ is symmetric and positive definite, Cholesky decomposition can be performed:

[0066] A′=LL T

[0067] Where L is a lower triangular matrix.

[0068] Reordering affects the sparsity structure of L, and a reasonable P can significantly reduce the number of non-zero elements in L.

[0069] ②LU decomposition

[0070] Generally, A is not symmetrical. Let A = A + A. T Or A = AA T Then, it is decomposed:

[0071] A′=LU

[0072] At this point, L and U are not necessarily symmetrical.

[0073] 4. Reduction Solution

[0074] Once the reordered system solution x′ is obtained, the original system solution can be restored:

[0075] x = P T x′

[0076] Figure 1 The Cholesky four-step elimination process for a simple matrix is ​​shown. From Figure 1 As can be seen, each step of the Cholesky decomposition eliminates a row and a column from the matrix. The non-zero elements in row a above the diagonal result in new non-zero elements in those rows that already have non-zero elements in column a. These new non-zero values ​​are called padding, and they cause problems for the sparse Cholesky algorithm by requiring more floating-point operations. A noteworthy phenomenon is that padding depends on the order in which the matrix elements are placed.

[0077] Figure 2 It shows the relationship with Figure 1 No padding elimination process is generated during the same matrix elimination process. From Figure 2 As can be seen, except that the first and last rows and columns have been swapped, no padding is generated during elimination, and sparse Cholesky completes the process faster. If the order of columns and rows that produce the least padding in the matrix can be found, the inverse of any sparse symmetric matrix can be minimized by reordering the matrix using permutation matrices. Based on this, this invention proposes a solution to the minimum padding problem based on deep reinforcement learning, providing an optimized elimination order for sparse matrix factorization. The core of the minimum padding problem is to find a variable elimination order that minimizes the padding elements generated during Gaussian elimination.

[0078] Specifically, embodiments of the present invention provide a matrix pre-sorting method based on reinforcement learning, such as... Figure 3 As shown in the flowchart, the method includes the following steps:

[0079] S1. Convert the original matrix into an adjacency matrix of an undirected graph, where the nodes of the undirected graph represent variables in the matrix and the edges represent non-zero elements;

[0080] S2. Input the adjacency matrix and node mask of the undirected graph into a node rating calculation network based on a graph neural network to obtain the rating of each node.

[0081] S3. Calculate the adaptive mask based on the score of each node and the undirected graph, and filter candidate nodes according to the adaptive mask;

[0082] S4. Based on the candidate nodes, the maskable proximal policy optimization algorithm based on reinforcement learning agents is used to adjust the allowed action space, and the action with the most reward is selected as the elimination node for this time.

[0083] (1) Step S1

[0084] Since the input is a sparse matrix structure, in step S1, the matrix is ​​converted into an adjacency matrix of an undirected graph G = (V, E), where the set of nodes V represents variables and the set of edges E represents non-zero elements. Eliminating a node means connecting all its neighbors to form a complete subgraph, which may introduce new edges (filling elements). Figure 4 This illustrates the process of converting the original matrix into an adjacency matrix. The original matrix is ​​in COO format, where each row represents the row, column, and element value (only elements with a value of 1 are represented). The rows and columns with a value of 1 (non-zero elements) in the COO format are extracted, as shown below. Figure 4 As shown in the left part, the rows and columns 1, 2, 3, 4, and 5 are respectively regarded as nodes v1, v2, v3, v4, and v5 of the undirected graph. Connections between nodes are established based on the row and column numbers of each row. For example, "1 2", "1 3", and "1 4" in the first three rows represent nodes v1 connected to nodes v2, v3, and v5. Similarly, we can obtain nodes v2 connected to v1, v3 connected to v5, v4 connected to v5, and v5 connected to nodes v3 and v4, finally resulting in... Figure 4 The middle section shows the undirected graph. Finally, using v1, v2, v3, v4, and v5 in the undirected graph as rows and columns, and based on the connection relationships between the nodes in the undirected graph, we obtain the following... Figure 4 The adjacency matrix shown on the right.

[0085] (2) Step S2

[0086] The calculation process of the node scoring network is as follows: Figure 5As shown, the network first receives input data and decomposes it into an adjacency matrix and a node mask. In the feature initialization phase, the normalized degree feature and neighborhood density of the adjacency matrix are calculated, concatenated with the node mask, and input into a feedforward neural network to obtain initial features. These initial features are then input into a graph convolutional network for graph convolution processing. The graph convolution processing stage is the core of the node scoring calculation network, containing multiple network layers, each including multiple functional components. The attention message passing component calculates the importance relationships between nodes using learnable weights, achieving intelligent aggregation of neighbor information; the feature transformation module converts the aggregated information into a more expressive representation; the dynamic feature fusion component uses a gating mechanism to adaptively balance the contributions of node features and neighbor features; the nonlinear transformation and regularization component ensures the model's expressiveness and generalization ability; and the residual connection alleviates the gradient problem in deep network training. These components are applied cyclically in a multi-layer graph convolutional structure, enabling the network to gradually expand its receptive field and capture a wider range of graph structure information. After completing multi-layer graph convolution processing, the system integrates multi-scale features captured at different levels through a multi-layer feature averaging fusion mechanism to form comprehensive node representations. Finally, these representations are converted into node scores through an output mapping layer, and a mask is applied to ensure that only the contributions of valid nodes are considered. Specifically, the computation process of the node score calculation network includes the following steps:

[0087] S21. Based on the adjacency matrix of the undirected graph G=(V,E) And node mask M∈{0,1} n×1 Calculate the initial characteristics of each node.

[0088] The i-th node v i The initial features are calculated as follows:

[0089]

[0090] Where, d norm (v i ) represents the i-th node v i The normalized degree, M i Let ρ(v) represent the i-th node in M. i ) represents the i-th node v i The neighborhood density, [;] denotes the vector concatenation operation, and MLP() represents a feedforward neural network operation used to map input features to a higher-dimensional representation space.

[0091] d norm (v i The calculation is as follows:

[0092]

[0093] Where |V| represents the number of nodes in the node set V, and Aij This represents the element in the i-th row and j-th column of the adjacency matrix.

[0094] ρ(v i The calculation is as follows:

[0095]

[0096] Where N(i) represents the i-th node v i The neighbor set is ε, which is a small constant to prevent division by zero, and |N(i)| represents the number of nodes in the neighbor set N(i).

[0097] S22. Input the initial features and adjacency matrix into the graph neural network for multi-scale feature extraction and fusion to obtain multi-scale fused features.

[0098] Graph neural networks consist of an L-layer network structure. Except for the first layer, which uses initial features as input, the input features of each of the other layers are fused features resulting from residual connections between the input and output features of the previous layer. The adjacency matrix is ​​input to each layer.

[0099] The processing flow of the i-th node in the l-th (l = 1, 2, ..., L) layer of the network includes the following steps:

[0100] S221. Calculate the attention of the i-th node in this layer of the network based on the adjacency matrix and the input features of this layer:

[0101]

[0102] in, This represents the feature vector of node j in the input features of this network layer. This represents the element in the i-th row and j-th column of the weighted adjacency matrix of layer l.

[0103] The calculation is as follows:

[0104]

[0105] in, This represents the attention weights of the l-th layer network.

[0106] The calculation is as follows:

[0107]

[0108] Where σ() is the sigmoid function, It is the learnable weight vector of the l-th layer (d is the vector dimension).

[0109] S222, Based on the attention of the i-th node in this layer of the network. and input features Calculate the fusion weight of the i-th node in this layer of the network:

[0110]

[0111] in, These are learnable parameters, σ() is the sigmoid function, and [;] represents the vector concatenation operation.

[0112] S223. Based on the attention of the i-th node in this layer of the network. Input features The fusion weights are used to calculate the fusion features of the i-th node in this layer of the network.

[0113]

[0114] S224, regarding the fusion feature f i (l) Perform a nonlinear transformation to obtain the transformation characteristics:

[0115]

[0116] in, b (l) These are the learnable parameters and bias vector of the network layer, respectively. ReLU(·) is a non-linear activation function, and LayerNorm(·) refers to the layer normalization operation.

[0117] S225. Connect the transformation features and input features of the i-th node in this layer of the network to obtain the input features of the next layer of the network:

[0118]

[0119] Finally, the initial features of the i-th node are... The fused features are obtained by concatenating the output and input features of all layers (L layers). to The fusion process yields multi-scale fusion features.

[0120] The final multi-scale fusion feature of the i-th node is calculated as follows:

[0121]

[0122] S23. Calculate the score of each node based on the multi-scale fusion features and mask of each node.

[0123] The score for the i-th node is calculated as follows:

[0124]

[0125] Among them, W out This represents the learnable weight matrix of the output layer of the node scoring calculation network.

[0126] This embodiment uses an enhanced graph neural network with high attention efficiency as a feature extractor. Through innovative structural design, it fully extracts the dynamic graph structure and improves the network's expressive power. The scores of each learned node are further sent to the policy / evaluation network as a preprocessor for the entire reinforcement learning process.

[0127] This enhanced graph neural network employs a lightweight attention mechanism for message passing, enabling the network to dynamically assign weights to different neighbor nodes, highlighting the influence of key nodes. Residual skip connections address the vanishing gradient problem in deep GNNs, ensuring effective information preservation during multi-layer propagation. An innovative dynamic feature fusion mechanism intelligently integrates node features with neighbor information through gating units, achieving adaptive learning. During feature extraction, basic graph features (degree, neighborhood density, etc.) are efficiently computed first, and then node representations are updated through multi-layer attention-enhanced message passing. The output of each layer is preserved through residual connections, and multi-scale feature fusion is performed in the final stage to ensure the model simultaneously captures local and global graph structure information. This design gives the model three major advantages in the minimum fill problem: 1) efficient identification of key nodes and potential fill regions; 2) adaptive handling of graphs of different sizes and structures; and 3) significantly reduced computational complexity.

[0128] (3) Step S3

[0129] The reinforcement learning minimum fill environment is a reinforcement learning environment specifically designed for the graph elimination problem. Its core lies in introducing an adaptive action masking strategy. The goal of graph elimination is to progressively remove nodes from the graph until no cliques remain, while minimizing fill (the extra edges introduced during node removal). Traditional masking strategies employ a greedy approach, selecting the node with the minimum fill cost (number of fill elements) at each step. While simple and efficient, this approach is prone to getting trapped in local optima.

[0130] This invention seeks a balance between exploration and exploitation by adaptively adjusting the action selection strategy, thereby more effectively exploring the solution space and potentially finding better graph elimination schemes. The key innovation of this environment lies in its adaptive threshold mechanism. The adaptive threshold is the core parameter controlling the agent's exploration level. Instead of rigidly selecting the node with the lowest filling cost, the environment allows for the selection of nodes with slightly higher filling costs within a certain range to increase the randomness of exploration. This threshold is not fixed but dynamically adjusted based on the training progress and the agent's past successes. Specifically, if the agent achieves better results in previous steps or complete paths, such as finding an elimination sequence with lower filling costs, the adaptive threshold will be appropriately increased to encourage the agent to explore more and try different deletion strategies in order to find better solutions. Conversely, if the agent performs poorly, such as failing to find an effective elimination path multiple times consecutively, or finding a path with high filling costs, the adaptive threshold will be decreased to reduce the exploration level, making the agent rely more on known better strategies, thereby accelerating convergence. Typically, the training process involves adaptively selecting which node to eliminate based on the threshold, further reducing matrix filling costs, and optimizing the matrix pre-sorting. In addition, the environment introduces the concept of an exploration factor, which is usually in the form of a bell curve. In the early stage of training, the exploration factor gradually increases, encouraging the agent to explore the solution space extensively; in the later stage of training, the exploration factor gradually decreases, making the agent make more use of the learned knowledge and tend to choose the node with the lowest filling cost, thereby improving the quality of the solution.

[0131] Specifically, step S3 includes the following steps:

[0132] S31. The node scoring computation network serves as the feature extraction layer of the policy network in the maskable near-end policy optimization (PPO) algorithm, and is also the input to the PPO policy network.

[0133] S32. Find the node with the minimum filling cost and its corresponding node mask. That is, the node selection strategy is restricted to the node with the minimum degree or the node with the minimum filling cost, thereby generating the minimum filling cost.

[0134] S33. Calculate the training progress based on the current number of training rounds and the total number of training rounds.

[0135] The training progress is the smaller value between the current training round number / the total training round number and 1.

[0136] S34. If the training progress is in the initial stage, select the node with the minimum filling cost as the candidate node; otherwise, proceed to the next step.

[0137] S35. Calculate the exploration factor based on the training progress.

[0138] The specific calculation method is as follows:

[0139] If the training progress is in the middle stage, the exploration factor is equal to an increasing function value that increases with the training progress (e.g., the increasing function is a quadratic function); otherwise, the exploration factor is equal to a decreasing function value that decreases with the training progress (e.g., the decreasing function is a cosine function).

[0140] S36. Calculate the adaptive threshold based on the current threshold and the exploration factor.

[0141] The specific calculation method is as follows:

[0142] The adaptive threshold is equal to the current minimum padding cost of training * (1 + exploration factor * 0.5). Generally speaking, strict masking restricts the action space. Reducing the restriction within a certain range will help to explore a better strategy. However, if the range is too large, it will get stuck in a local optimum or converge for a long time. Therefore, multiplying the exploration factor function by a coefficient of 0.5 can effectively adjust this situation while maintaining good exploration.

[0143] S37. Select nodes whose filling cost is lower than the adaptive threshold as candidate nodes. The adaptive threshold changes with the training progress throughout the training process, providing better exploratory capabilities. The training process improves towards a better strategy, and after a long training period, when the filling of the entire matrix remains essentially unchanged, a stable filling strategy can be obtained.

[0144] (5) Step S4

[0145] This step is modeled as a reinforcement learning task: a reinforcement learning agent learns the optimal ranking strategy by interacting with a matrix environment. The agent's state is defined by the structural features of the matrix, and actions correspond to selecting principal components. The reward function aims to minimize the negative number of fill-in elements in the computation process divided by the normalized fill-in of the matrix dimensions. At each decision, the agent tends to choose actions with larger reward values ​​to obtain higher rewards, thus gradually reducing the fill-in elements in the matrix.

[0146] ①State Space

[0147] The state space consists of the adjacency matrix of the current undirected graph and a mask vector indicating which nodes have not yet been eliminated, and is modeled as follows:

[0148] S = (B, M)

[0149] B is the adjacency matrix of the current undirected graph, and M is the mask vector (indicating which nodes have not yet been eliminated; when a node is eliminated, its mask value is 0, and a value of 1 indicates that the current node can be deleted). For a matrix of dimension N, its state space is represented as: S = (N, N+1), B = (N, N), M = (N, 1).

[0150] ② Action Space

[0151] The action space consists of nodes that have not been eliminated and are modeled using adaptive masks:

[0152] A={a∈Λ|M[a]=1&Mask[a]=1}

[0153] 'a' represents the selected action, M[a] = 1 indicates that the selected node has not been eliminated, and Mask[a] = 1 indicates that a masking strategy is used to restrict its action space, limiting it to minimum degree or minimum fill. The masking strategy can be dynamically processed. A strict restriction strategy will miss the optimal solution. During execution, the restriction strategy can be dynamically processed, and an adaptive masking strategy can be adopted according to the progress of reinforcement learning.

[0154] ③ Reward function

[0155] The reward function is set as follows:

[0156] R(S,a)=-fillin / N

[0157] Fillin refers to the number of filler elements introduced in each action selection during reinforcement learning. Normalizing the number of filler elements by dividing it by the matrix dimension helps the agent find a smaller filler number.

[0158] The node scoring computation network is jointly trained with the policy and value networks of reinforcement learning, and the weights are automatically updated via gradient backpropagation of PPO. Reinforcement learning (RL) has achieved remarkable success in matrix preprocessing, but it faces severe computational challenges when applied to problems involving large-scale matrices (e.g., matrices with dimension N > 10000). In RL problems, the sheer size of the state space can become intractable if it is directly derived from or represented by such a matrix. For example, if every element of the matrix or any combination thereof constitutes part of the state representation, the number of possible states grows exponentially or in higher-order polynomials as N increases. This phenomenon is often associated with the curse of dimensionality and leads to several key problems. First, processing these massive states becomes exceptionally slow during training and inference, becoming a bottleneck in the learning process. Second, neural networks, often used as function approximators in RL, require a huge number of input units. This, in turn, leads to an explosive growth in network parameters, requiring massive amounts of memory for storage and significantly increasing the computational cost per training iteration. Therefore, effectively training such a large model becomes a daunting task, often requiring an unbearable amount of data and time.

[0159] Finally, by iteratively executing steps S22 to S24, the elimination order of the original matrix is ​​obtained.

[0160] To address these challenges when processing extremely large matrices, a robust strategy is to divide the original matrix into multiple sub-blocks using nested partitioning (or nested dissection). Therefore, when the original matrix is ​​large (exceeding a preset dimension, such as 500), the matrix pre-sorting method based on reinforcement learning provided in this embodiment of the invention further includes the following step before step S1:

[0161] S0. The original matrix is ​​divided into multiple sub-blocks using a nested segmentation method.

[0162] This "divide and conquer" strategy systematically decomposes a large and difficult-to-manage original matrix into a hierarchical set of smaller and more manageable submatrices. It starts with an N×N matrix, where N can reach tens of thousands. The first step might be to divide this matrix into several larger blocks. Then, each of these blocks is further subdivided, and this process can be done recursively—this is the meaning of "nesting." The ultimate goal is to decompose the original matrix into a series of much smaller submatrices; for example, the target dimension could be set to around m×m, where m is much smaller than N.

[0163] This segmentation of the transformation offers profound benefits. By operating on these smaller submatrices, the local state representation of the RL agent becomes significantly simplified and reduced. This directly translates to faster processing speeds when computations involving these subproblems are performed. Furthermore, the neural network architecture can be tailored accordingly. Instead of having a single, large network struggle with massive inputs, small, dedicated networks can be used for each type of submatrix, or a modular network can be employed to process these sub-components sequentially or hierarchically. This inherently limits the growth of network parameters and reduces overall memory consumption.

[0164] For a matrix of size N, the state space of reinforcement learning is (N, N+1). For larger matrices, such as N>1000, a nested segmentation method is used to process the matrix into a series of sub-matrices, dividing it into m (<500) sub-matrices. The entire segmentation process includes:

[0165] S01. Obtain the dimension (number of rows and columns) and recursion depth of the current matrix;

[0166] S02. If the dimension is less than or equal to the set m×m or the recursion depth is greater than or equal to the maximum recursion depth, then store the current matrix as the completed submatrix; otherwise, proceed to the next step.

[0167] S03. Divide the current matrix into multiple sub-blocks based on the minimum vertex segmentation.

[0168] S04. Perform S01 and S02 for each sub-block;

[0169] S05, until the dimension of all sub-blocks is less than or equal to the set m×m or the recursion depth is greater than or equal to the maximum recursion depth;

[0170] S06. Output all the partitioned submatrices.

[0171] Figure 6 This example diagram illustrates nested partitioning of a 13×13 matrix, where the submatrices are defined as 5×5. From the undirected graph of the original matrix, we can see that columns 6, 7, and 8 are the partitioning matrices. After matrix reordering, their rows and columns are swapped, and these rows and columns are now located at the bottom and rightmost edges of the original matrix. Figure 6 As can be seen, the 13×13 matrix is ​​divided into 3 submatrices, including two submatrices along the diagonal, a bottom right corner dividing matrix along the diagonal, and the rest are the filling elements of the submatrices relative to the dividing matrix.

[0172] Traditional nested partitioning algorithms typically employ the AMD algorithm when processing submatrices because it offers high efficiency while obtaining approximate solutions. However, this embodiment solves submatrices using the aforementioned reinforcement learning algorithm. This combination fully leverages the advantages of each algorithm, making matrix preprocessing more efficient and finding a better sorting method.

[0173] It should be noted that the various processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved. This embodiment does not impose any limitations on these steps.

[0174] Based on the above-described reinforcement learning-based matrix pre-sorting method, this invention also provides a reinforcement learning-based matrix pre-sorting system, which includes a processor and a memory. The memory stores instructions, and when the processor executes the instructions, it is configured to execute the above-described method.

[0175] The present invention also provides a storage medium storing machine-executable instructions, which, when executed, utilize the aforementioned reinforcement learning-based matrix pre-sorting method or system. The storage medium is a computer-readable storage medium, which may be a tangible medium that may contain or store computer programs for use by or in conjunction with an instruction execution system, apparatus, or device. The computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, the computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, optical disc read-only memory (CD ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0176] In summary, the matrix pre-sorting method, system, and storage medium based on reinforcement learning provided in this invention model the matrix pre-sorting problem as a sequential decision-making process. It utilizes a reinforcement learning agent to learn the optimal sorting strategy through interaction with the matrix environment, thereby reducing the number of fill elements and shortening computation time. The agent's state is defined by the structural features of the matrix, and the action corresponds to selecting nodes that have not been eliminated. The reward function aims to minimize the number of fill elements (fill-in) divided by the normalized fill size of the matrix dimension, guiding the agent's learning.

[0177] The performance of the Reinforcement Learning (RL) based algorithm ReFill in optimizing the total filling cost of node deletion is evaluated below. The experiment uses data with N<500 from the PACE 2017 Track-B challenge. The number of filling elements on the present invention (DRL), Minimum Degree Filling Algorithm (MDH), and Minimum Filling Heuristic Algorithm (MFillH) are shown in Table 1.

[0178] Table 1 Comparison of dataset filling effects

[0179]

[0180] As can be seen from Table 1, for each graph, the DRL-based method is on par with or improves upon the MDH and MFILLH methods, with an average improvement of more than 5%.

[0181] like Figure 7In section 2.GRAPH, the reward changes in reinforcement learning are compared between adaptive masking and masked methods. The orange curve represents the adaptive masking method, and the blue curve represents the masked method. Adaptive masking is highly exploratory in the early stages, maintains a lower level for a period during training, but subsequently learns a better policy and becomes more exploratory. In the mid-to-late stages, it maintains a high level of exploratory ability, exceeding the exploratory performance of masked methods.

[0182] Below is an example of applying matrix reordering in the electromagnetic interference path topology model of electric drive equipment.

[0183] Based on the equivalent model of all internal and external interference sources of electric drive equipment and the topology model of conducted electromagnetic interference paths, a system-level electromagnetic simulation platform is built using an improved nodal analysis method. This study investigates the solution method for the hybrid current and voltage responses of nodes in complex system network topologies, and realizes a transient electromagnetic interference prediction algorithm. The entire test system is as follows: Figure 8 As shown, it mainly includes: power supply cabinet, LISN, DC cable, DC / AC converter, AC cable, and permanent magnet synchronous motor.

[0184] The low-voltage electromagnetic interference (EMI) measurement experiment first connects the power supply to the line impedance stabilization network (LISN), which isolates external interference and provides a standard impedance. The devices under test (inverter and motor) are connected to the system, and interference signals are acquired through an EMI receiver. A spectrum analyzer monitors and displays the interference intensity at different frequencies in real time, while an oscilloscope observes the time-domain waveform characteristics. The motor speed is adjusted, and interference data under different operating conditions is recorded. Finally, the data is analyzed to determine the interference source and propagation path, and to assess whether it complies with EMC standards. The equivalent circuit model is as follows: Figure 9 As shown, this model is based on the multi-port Thevenin equivalence principle to model electromagnetic interference (EMI) in multi-electric systems. Firstly, it includes the impedance Z of the multi-electric system. INV And the impedance Z of the power supply network alone L V SP and V SN These represent the equivalent noise sources at the positive and negative power supply ports, respectively. The final model equates the inverter input to a value containing the impedance matrix Z. INV and voltage source V SThe equivalent circuit is obtained, and the impedance frequency characteristics are approximated by vector fitting. Simultaneously, the frequency-domain voltage source is converted to a time-domain voltage source using inverse Fourier transform, thus constructing a high-precision behavioral model suitable for EMC simulation. Solving the circuit model of an electric drive system involves solving a large-scale circuit matrix. The circuit model is first analyzed into different devices, then the parameters of the devices are transformed into elements in the matrix, and finally the matrix is ​​solved. The first step in matrix solving is preprocessing. Different matrix sortings result in different padding elements, which in turn affect the subsequent matrix processing time and memory usage. The padding elements and time (s) of the traditional nested dissection algorithm and the reinforcement learning-based preprocessing algorithm (DRL) are compared. Figure 10 As shown, the matrix dimension N = 35324 formed by the circuit model requires fewer padding elements for pre-sorting using reinforcement learning, resulting in a shorter running time.

[0185] Experiments compared the performance of the proposed RL method with traditional heuristic algorithms (such as AMD and nested partitioning). The results show that, in specific test cases such as large-scale circuit models (matrix dimension N = 35324), the reinforcement learning-based pre-sorting method demonstrates advantages in reducing the number of padding elements and shortening computation time. This proves the potential of reinforcement learning in exploring and learning the underlying patterns in complex matrix structures, thereby discovering better pre-sorting strategies, and provides a new perspective and effective tool for solving challenges in large-scale scientific and engineering computing.

[0186] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.

Claims

1. A matrix pre-sorting method based on reinforcement learning, characterized in that, Including the following steps: Transform the original matrix into an adjacency matrix of an undirected graph, where the nodes of the undirected graph represent variables in the matrix and the edges represent non-zero elements; Input the adjacency matrix and node mask of the undirected graph into a node rating calculation network based on a graph neural network to obtain the rating of each node; An adaptive mask is calculated based on the score of each node and the undirected graph, and candidate nodes are selected based on the adaptive mask. Based on the candidate nodes, a maskable near-end policy optimization algorithm based on reinforcement learning agents is used to adjust the allowed action space, and the action with the highest reward is selected as the elimination node for this time.

2. The matrix pre-sorting method based on reinforcement learning according to claim 1, characterized in that: The state of a reinforcement learning agent is defined by the structural features of the matrix, the action corresponds to selecting nodes that have not been eliminated, and the reward function aims to minimize the ratio of the number of filler elements to the matrix dimension during the computation process.

3. The matrix pre-sorting method based on reinforcement learning according to claim 2, characterized in that: The state space of a reinforcement learning agent consists of the adjacency matrix of the current undirected graph and a mask vector that identifies which nodes have not yet been eliminated. The action space of a reinforcement learning agent consists of nodes that have not been eliminated and are masked using an adaptive mask.

4. The matrix pre-sorting method based on reinforcement learning according to claim 3, characterized in that, The calculation process of the node scoring network includes: Calculate the initial features of each node based on the adjacency matrix and node mask of the undirected graph; The initial features and adjacency matrix are input into the graph neural network for multi-scale feature extraction and fusion to obtain the multi-scale fused features of each node; The score for each node is calculated based on the multi-scale fusion features and mask of each node.

5. The matrix pre-sorting method based on reinforcement learning according to claim 4, characterized in that, Graph neural networks consist of an L-layer network structure. Except for the first layer, which uses the initial features as input, the other layers use the fused features of the input and output features of the previous layer after residual connections. The adjacency matrix is ​​input into each layer. The processing flow of the i-th node in the l-th layer network includes the following steps: The attention of the i-th node in this layer of the network is calculated based on the adjacency matrix and the input features of this layer. Calculate the fusion weight of the i-th node in the network layer based on the attention and input features of the i-th node in the network layer; The fusion features of the i-th node in this layer of the network are calculated based on the attention, input features, and fusion weights of the i-th node in this layer of the network. The fused features are subjected to a nonlinear transformation to obtain the transformed features; Connect the transformation features and input features of the i-th node in this layer of the network to obtain the input features of the next layer of the network; The initial features of the i-th node and the fused features obtained by concatenating the output and input features of all layers of the network are fused to obtain the multi-scale fused features of the i-th node; The score for each node is calculated based on the multi-scale fusion features and mask of each node.

6. The matrix pre-sorting method based on reinforcement learning according to any one of claims 1 to 5, characterized in that, During training, an adaptive mask is calculated based on the score of each node and the undirected graph, and candidate nodes are selected according to the adaptive mask. The specific steps include: The node scoring computation network is used as the feature extraction layer of the policy network in the maskable near-end policy optimization algorithm based on reinforcement learning agents. Find the node with the minimum fill cost and its corresponding node mask; Calculate the training progress based on the current number of training rounds and the total number of training rounds; If the training progress is in the initial stage, then select the node with the minimum fill cost as the candidate node. Otherwise, proceed to the next step; Calculate exploration factors based on training progress; Calculate the adaptive threshold based on the current threshold and the exploration factor; Nodes whose filling cost is lower than the adaptive threshold are selected as candidate nodes.

7. The matrix pre-sorting method based on reinforcement learning according to claim 6, characterized in that, The exploration factor is calculated based on the training progress. Specifically, if the training progress is in the middle stage, the exploration factor is an increasing function value that increases with the training progress; otherwise, the exploration factor is a decreasing function value that decreases with the training progress.

8. The matrix pre-sorting method based on reinforcement learning according to claim 1, characterized in that, When the dimension of the original matrix exceeds the preset dimension, the nested segmentation method is first used to divide the original matrix into multiple sub-blocks.

9. A matrix pre-sorting system based on reinforcement learning, characterized in that: It includes a processor and a memory, the memory storing instructions, which, when executed by the processor, configure the processor to perform the matrix pre-sorting method based on reinforcement learning as described in any one of claims 1 to 8.

10. A storage medium, characterized in that: The storage medium stores machine-executable instructions, which, when executed, use the reinforcement learning-based matrix pre-sorting method according to any one of claims 1 to 8 or the reinforcement learning-based matrix pre-sorting system according to claim 9.

Images