Dynamic asset liability allocation optimization method based on deep reinforcement learning

Abstract

The application discloses a dynamic asset liability configuration optimization method based on deep reinforcement learning, relates to the field of deep learning, and is characterized in that: a state feature vector is constructed by synchronizing market and internal data; a compliance region vertex set is obtained by using a double description method to analyze a regulatory rule set; a strategy network generates a proportion weight vector based on the feature vector and mask processing; a barycentric coordinate system is projected and mapped, the weight is used as a coefficient to act on the vertex set, and a natural compliance asset allocation instruction is generated; a manifold deviation degree is calculated to correct the network parameters in reverse; meanwhile, an edge safety distance and its change rate are calculated to determine an execution strategy and monitor region drift, and dynamic response updating is realized. Through a constraint embedded mapping mechanism, the application solves the problems of gradient fracture and decision violation, significantly improves the convergence efficiency, dynamic adaptability and risk resistance of the configuration decision under the premise of ensuring compliance.

Description

Technical Field

[0001] This invention relates to the field of deep learning, and more specifically to a dynamic asset and liability allocation optimization method based on deep reinforcement learning. Background Technology

[0002] With the development of fintech, dynamic allocation of large-scale assets and liabilities has become a core task of data processing and decision-making systems in financial institutions. In recent years, deep reinforcement learning technology has been introduced into such decision-making systems, using deep neural networks to extract features from complex environmental observations and output corresponding asset allocation vectors. However, in practical applications, asset and liability allocation is subject to stringent and dynamically changing regulatory rules (such as upper limits on the proportion of various asset classes, solvency constraints, and liquidity coverage ratios). These regulatory rules mathematically constitute rigid boundary constraints in a high-dimensional space.

[0003] However, traditional reinforcement learning models often cause gradient information to break or disappear at geometric boundaries, severely impacting the model's convergence efficiency and policy quality near compliance boundaries. In practice, it is difficult to output accurate compliance weight assignments, making it highly susceptible to decision bias. Furthermore, it is difficult to provide early warnings of potential compliance risks and dynamically adjust the execution step size, lacking effective emergency response capabilities when faced with sudden shifts in the compliance zone. Summary of the Invention

[0004] The purpose of this application is to provide a dynamic asset and liability allocation optimization method based on deep reinforcement learning, which has the advantages of effectively reducing the interference of nonlinear fluctuations in financial markets and macroeconomic uncertainties on the stability of allocation decisions, improving the accuracy of dynamic matching of assets and liabilities and multi-objective trade-offs under complex constraints, thereby enhancing the intelligence level and risk resistance capability of long-term asset and liability management decisions.

[0005] The objective of this application can be achieved through the following technical solution: a dynamic asset-liability allocation optimization method based on deep reinforcement learning, comprising the following steps:

[0006] Simultaneously acquire first market data and second internal financial data, and construct a state feature vector representing the real-time status of assets and liabilities;

[0007] Based on the preset set of regulatory rules and the second internal financial data, the set of compliant region vertices corresponding to the current decision moment is parsed out. The coordinates of each vertex in the set of compliant region vertices represent the extreme point configuration ratio that satisfies the regulatory constraints.

[0008] The state feature vector is input into a preset policy network to generate a weighted vector that matches the number of elements in the vertex set of the compliant region.

[0009] Based on the projection mapping process of the centroid coordinate system, the allocated weight vector is used as a coefficient to calculate the coordinates of each vertex in the vertex set of the compliant area, and an asset allocation instruction within the compliant area is generated.

[0010] Calculate the manifold deviation between the asset allocation instruction and the original output of the strategy network, and perform reverse correction on the network parameters of the strategy network based on the manifold deviation.

[0011] Compared with the prior art, the beneficial effects of this application are:

[0012] 1. By introducing a projection mapping mechanism based on the centroid coordinate system, the output space of the policy network is deeply mathematically coupled with the feasible region of the convex polyhedron defined by regulatory constraints. Utilizing the convex combination theorem, the generated asset allocation instructions are strictly limited to the compliant region in the mathematical mechanism during the computation stage. This solves the technical defect of the policy output being disconnected from the constraint boundary in traditional reinforcement learning schemes, ensuring the inherent compliance of decision instructions.

[0013] 2. By calculating the manifold deviation between asset allocation instructions and the original action tendency, and using this deviation as a penalty weight to correct the network parameters, a continuous gradient feedback signal regarding the constraint manifold structure is provided to the policy network. This mechanism effectively overcomes the problem of gradient information breakage or disappearance caused by traditional hard constraint processing methods, guides the policy network to internalize compliance boundary features, and thus improves the search accuracy and training convergence speed of the model in highly constrained environments.

[0014] 3. By dynamically parsing the vertex set corresponding to regulatory rules using a dual-description method and combining it with a masking zeroing mechanism, adaptive matching between a fixed-dimensional deep neural network and the dynamically fluctuating topology of the compliance region is achieved. This ensures that when adjustments to regulatory policies or changes in the financial status of an institution cause shifts in the shape and dimension of the feasible region, the versatility and robustness of the decision-making system in dynamic policy environments are enhanced. Attached Figure Description

[0015] Figure 1 This is a flowchart illustrating the steps of the dynamic asset and liability allocation optimization method based on deep reinforcement learning proposed in this application. Detailed Implementation

[0016] It is readily understood that, based on the technical solution of this invention, those skilled in the art can propose various interchangeable structural methods and implementations without altering the essential spirit of the invention. Therefore, the following detailed embodiments and accompanying drawings are merely illustrative examples of the technical solution of this invention and should not be considered as the entirety of the invention or as limitations or restrictions on the technical solution of this invention.

[0017] Example 1,

[0018] like Figure 1 As shown, the dynamic asset-liability allocation optimization method based on deep reinforcement learning includes the following steps:

[0019] This system simultaneously acquires primary market data and secondary internal financial data, constructing a state feature vector representing the real-time state of assets and liabilities. This component is used to synchronously collect and process financial data from both market and internal dimensions. Specifically, it can be implemented using a multi-source data interface combined with a timestamp alignment algorithm. Its role is to ensure the spatiotemporal consistency of market and financial information, providing accurate input for subsequent decision-making. The state feature vector is a multi-dimensional numerical representation that comprehensively reflects the current market environment and the institution's financial situation. Specifically, it can be achieved by calculating and standardizing key indicators such as asset price statistics, debt-to-equity ratio, cash flow coverage ratio, and interest rate sensitivity. Its role is to provide a complete environmental state description for the reinforcement learning strategy network.

[0020] Based on a pre-defined set of regulatory rules and the second set of internal financial data, the set of compliant region vertices corresponding to the current decision-making moment is analyzed. The coordinates of each vertex in the compliant region vertex set represent the proportion of extreme points that satisfy regulatory constraints. The compliant region vertex set refers to the set of extreme point coordinates of a convex polyhedron formed under multi-dimensional constraints. Specifically, it can be realized by analyzing a system of linear inequalities using the double-description method. Its function is to transform abstract regulatory rules into feasible domain boundaries in geometric space, providing a mathematical basis for strategy mapping. The set of regulatory rules includes, but is not limited to, upper and lower limits for various asset proportions, leverage ratio restrictions, liquidity ratio requirements, etc. These rules will form compliant regions of different shapes and sizes under different market environments and institutional states.

[0021] The state feature vector is input into a preset policy network to generate an allocated weight vector that matches the number of elements in the vertex set of the compliant region. The allocated weight vector is a set of normalized coefficients that weight the vertices of the compliant region. Specifically, it can be achieved by masking and exponential normalization after outputting from the fully connected layer at the end of the policy network. Its function is to quantify the contribution of each pole configuration scheme to the final decision. The policy network adopts a deep neural network architecture, capable of learning the complex nonlinear mapping relationship between state features and optimal weight allocation.

[0022] The projection mapping process based on the centroid coordinate system uses the allocated weight vector as coefficients to perform a weighted summation calculation on the coordinates of each vertex in the compliant region vertex set, generating an asset allocation instruction within the compliant region. Centroid coordinate system mapping refers to a mathematical calculation method that uses convex geometry theory to transform the weight vector into points within the convex hull. Specifically, this can be achieved through a linear convex combination of vertex coordinates. Its function is to ensure that the output configuration instruction mathematically necessarily satisfies all regulatory constraints, eliminating the need for post-processing corrections. This mapping process guarantees the inherent compliance of the output action and is the core technical innovation of this application.

[0023] The manifold deviation between the asset allocation instruction and the original output of the policy network is calculated, and the network parameters of the policy network are then corrected in reverse based on the manifold deviation. The manifold deviation is a measure of the spatial distance between the original policy intention and the actual executed action, which can be calculated using Euclidean distance or Manhattan distance. Its function is to quantify the degree to which constraint mapping distorts the policy intention, provide feedback signals for gradient correction, and guide the policy network to learn inherent compliance.

[0024] The core innovation of this application lies in its use of a centroid coordinate mapping mechanism driven by the geometric structure of the compliance region to deeply mathematically couple the output space of the policy network with the feasible domain of regulatory constraints. This allows the policy network to directly learn the optimal configuration within the compliance region, fundamentally solving the technical shortcomings of traditional hard constraint processing methods, such as inability to adapt to dynamic regulatory environments, low training efficiency, and frequent policy violations. By transforming discrete constraint checks into continuous geometric mappings, this application implements a constraint-embedded learning architecture, significantly improving the application effect of reinforcement learning in constrained optimization problems.

[0025] This application further proposes that the specific steps for constructing the state feature vector include:

[0026] Based on the first market data, the mean and variance of asset prices within a preset sliding window are calculated. This process selects price data from the past 20 trading days, calculates the arithmetic mean of each asset class as the mean indicator, and calculates the square of its standard deviation as the variance indicator, to characterize the market volatility and price trend of that asset class. Specifically, for equity assets, the daily closing price of the corresponding stock index is used; for fixed-income assets, the market capitalization-weighted yield of the bond portfolio is used; and for cash assets, the benchmark interest rate of the money market is used.

[0027] Based on the second set of internal financial data, the ratio of total assets to total liabilities at the current moment is calculated, and the cash flow coverage ratio for the future preset period is calculated in conjunction with the preset cash flow time series data. The formula for calculating the asset-liability ratio is: Asset-liability ratio = Total assets / Total liabilities. This ratio reflects the institution's overall solvency and financial leverage level; a higher ratio indicates stronger solvency. The cash flow coverage ratio is calculated using the ratio of expected total cash inflows to total cash outflows over the next 6 months, with the formula: Cash flow coverage ratio = Expected cash inflows / Expected cash outflows. This indicator is used to assess short-term liquidity risk, and a ratio greater than 1.5 is generally considered to have a sufficient liquidity safety margin.

[0028] Based on the first market data and the second internal financial data, the interest rate sensitivity difference between assets and liabilities is calculated. This indicator is achieved through duration analysis, specifically by first calculating the adjusted duration of the asset portfolio. Modified duration of the debt portfolio Adjusted duration represents the percentage change in the value of an asset or liability when the interest rate changes by 1%. The formula is as follows:

[0029] Modified duration = Macaulay duration / (1 + yield to maturity).

[0030] Then calculate the interest rate sensitivity difference. This difference reflects the size and direction of interest rate risk exposure; a positive value indicates that assets are more sensitive to rising interest rates, while a negative value indicates that liabilities are more sensitive to rising interest rates.

[0031] The mean and variance, the ratio, and the interest rate sensitivity difference are standardized and combined in a multidimensional manner to generate a state feature vector representing the real-time state of the asset and liability system. The standardization process uses the Z-score method; for each indicator x, a standardized value is calculated. ,in, This is the historical average of the indicator. To determine the historical standard deviation, each indicator is mapped to a standard normal distribution with a mean of 0 and a standard deviation of 1, eliminating the influence of different units and numerical ranges. The standardized indicators are then concatenated according to a pre-defined logical order: the first 40 dimensions represent the mean and variance characteristics of various asset prices, the middle 40 dimensions represent financial ratio indicators (including debt-to-equity ratio, cash flow coverage ratio, and their derivatives), and the last 48 dimensions represent interest rate sensitivity-related features and other auxiliary features, together forming a 128-dimensional state feature vector.

[0032] Through the above technical solutions, this application achieves the effective integration and standardized representation of multi-dimensional financial data. The mean and variance statistical characteristics of market data accurately reflect the volatility and trend changes in the external market environment, while the asset-liability ratio and cash flow coverage ratio of internal financial data accurately reflect the institution's own financial health and liquidity level. The interest rate sensitivity difference indicator reveals the exposure of the asset-liability portfolio to interest rate risk. Z-score standardization of multi-dimensional features ensures the comparability of indicators with different dimensions and numerical ranges, reduces the adverse effects of numerical scale on neural network training, and accelerates model convergence. This comprehensive and standardized state feature vector provides the strategy network with comprehensive, accurate, and real-time environmental state input, serving as the data foundation for high-quality decision-making and contributing to improving the scientific, accurate, and adaptable nature of asset allocation decisions.

[0033] This application further proposes that the specific steps for parsing the set of compliant region vertices corresponding to the current decision-making moment include:

[0034] Based on the aforementioned set of regulatory rules, construct a matrix of linear inequalities concerning the asset allocation variable:

[0035] ;

[0036] in, Represents the constraint coefficient matrix. This represents the action vector to be generated. This represents the constraint boundary vector; each row of the constraint coefficient matrix corresponds to a regulatory rule or constraint condition, and the number of columns in the matrix equals the number of asset classes. Each element of the constraint boundary vector corresponds to a threshold limit for each rule, and these thresholds are dynamically adjusted based on changes in the current balance sheet, market environment, and regulatory policies. For example, in certain extreme market conditions, regulators may temporarily tighten the upper limit on the allocation of equity assets, in which case the corresponding elements in the constraint boundary vector will decrease accordingly.

[0037] The convex polyhedron defined by the linear inequality matrix is ​​analyzed using the bi-description method. The coordinates of the convex polyhedron's poles are extracted, and all extracted pole coordinates are used to determine the vertex set of the compliant region. The bi-description method is a classic algorithm in computational geometry used to convert convex polyhedra represented by inequalities (H-representation) into vertex representations (V-representation). The core idea of ​​this algorithm is to iteratively add constraints one by one, updating the current vertex set each time a constraint is added, ultimately obtaining a complete set of poles that satisfies all constraints.

[0038] The specific implementation process is as follows: Assuming the regulatory rules include the following 5 constraints, in the three-dimensional asset allocation space (equity asset ratio)... Fixed income assets as a percentage Cash and cash equivalents )middle:

[0039] Constraint 1: (Equity assets shall not exceed 30%)

[0040] Constraint 2: (Fixed income assets account for no less than 40%)

[0041] Constraint 3: (Cash and cash equivalents should account for no less than 5%)

[0042] Constraint 4: (Cash and cash equivalents shall not exceed 15%)

[0043] Constraint 5: (The sum of the proportions of all types of assets equals 100%)

[0044] Rewrite these constraints in standard linear inequality form and construct the constraint coefficient matrix. and constraint boundary vector Using the bi-description method, these constraints define a convex polyhedron in this 3D space, which can be calculated to have 7 vertices, each representing an extreme but fully compliant configuration. For example:

[0045] vertex Equity holdings are capped at 30%, cash holdings at 15%, and fixed-income holdings at 55%.

[0046] vertex Equity holdings should reach the upper limit of 30%, cash holdings should reach the lower limit of 5%, and fixed income holdings should reach 65%.

[0047] vertex Equity holdings are capped at 25%, cash holdings at the lower limit of 5%, and fixed-income holdings at 70%.

[0048] vertex Equity investments are capped at 15%, fixed-income investments at 70%, and cash investments at the maximum limit of 15%.

[0049] vertex Equity investments are capped at 5%, fixed-income investments at 80%, and cash investments at the maximum limit of 15%.

[0050] vertex No equity investments are allocated; fixed-income investments comprise 85%; and cash investments reach the maximum limit of 15%.

[0051] vertex No equity holdings; fixed-income holdings up to 95%; cash holdings up to 5%.

[0052] These seven vertices constitute the vertex set of the compliance region. Any convex combination of two or more vertices will produce a valid configuration scheme located within the compliance region. Vertex Represents an aggressive allocation (high equity), peak One vertex represents a conservative allocation (no stake), while the other vertices represent intermediate strategies of varying degrees.

[0053] It is worth noting that when regulatory rules change, such as lowering the equity asset cap from 30% to 25%, the constraint boundary vector will be updated accordingly, and the dual-description method will recalculate a new vertex set. This dynamic resolution mechanism ensures that the compliance area always reflects the latest regulatory requirements. Similarly, when an institution's balance sheet changes significantly, certain relative constraints (such as absolute limits based on total assets) will also lead to adjustments in the constraint boundaries, thereby affecting the composition of the vertex set.

[0054] Through the aforementioned technical solution, this application achieves a precise geometric representation of regulatory constraints. It transforms the originally abstract, textually described regulatory rules into a mathematically rigorous set of convex polyhedron vertices, changing the constraints from implicit and difficult-to-manipulate forms to explicit and directly computable geometric objects. The application of the dual-description method ensures the completeness and accuracy of vertex extraction, guaranteeing the full equivalence of the geometric representation of the compliance region with the original constraints. The dynamic parsing mechanism can respond in real-time to changes in regulatory rules and fluctuations in the institution's financial status, updating the shape and boundaries of the compliance region. This ensures that the strategy network is always optimized under the current effective constraints, effectively supporting the compliance, accuracy, and dynamic adaptability of asset allocation decisions, and providing key technical support for achieving robust asset-liability management in a complex and ever-changing regulatory environment.

[0055] This application further proposes that the specific steps for generating the allocated weight vector include:

[0056] The original score tensor is output by the fully connected layer at the end of the policy network. The preset output dimension of the fully connected layer is greater than the number of elements in the vertex set of the compliant region. The preset output dimension is usually set to 10 to 15 dimensions. This design is based on historical data statistical analysis: under most market environments and regulatory rule combinations, the number of vertices in the compliant region of the three-dimensional asset allocation space will not exceed 12. Setting a larger preset output dimension can cope with the dynamic changes in the number of vertices at different times, ensuring the universality and stability of the network architecture and avoiding frequent adjustments to the network structure due to fluctuations in the number of vertices.

[0057] Based on the actual number of elements in the compliant region vertex set at the current moment, invalid dimensions in the original score tensor that exceed the actual number of elements are masked and zeroed. The specific implementation method for this masking is as follows: assuming the output dimension of the final fully connected layer is 15, and the compliant region vertex set obtained at the current moment contains 7 vertices, then the 8th to 15th dimensions (a total of 8 invalid dimensions) of the output tensor need to be masked. The processing method involves replacing the original scores of these dimensions with a very small value, typically set to -1 × 10⁹. This value will approach 0 in subsequent exponential operations, thus corresponding to weight values ​​close to 0 in the normalized weight vector, ensuring that these invalid dimensions do not affect the final configuration decision.

[0058] Exponential normalization is performed on the masked tensor to obtain the amortized weight vector where the sum of all elements in each dimension is equal to one and all elements are non-negative. Exponential normalization is implemented using the standard softmax function for the masked score vector. The softmax function calculates the weights for each dimension:

[0059] ;

[0060] Where i and j range from 1 to 15. Since the scores for dimensions 8 to 15 have been set to -1 × 10⁹, exp(-1 × 10⁹) ≈ 0, these dimensions contribute almost nothing to the denominator, and their corresponding weights are... to It also approaches 0. The final weight vector has the first 7 dimensions as effective weights and the last 8 dimensions as ineffective weights close to 0. The first 7 dimensions can be directly extracted as the final distributed weight vector.

[0061] Through the above technical solutions, this application realizes a flexible adaptive matching mechanism between the policy network output and the dynamic number of vertices. The pre-designed large output dimension ensures the stability of the network architecture, avoiding frequent modifications to the network structure due to the time-varying nature of the number of vertices, and reducing system complexity and maintenance costs. The masking mechanism cleverly resolves the contradiction between the fixed output dimension and the dynamic number of vertices. By setting the scores of invalid dimensions to a minimum value, their weights after softmax normalization approach 0, achieving dynamic dimension adaptation without changing the network architecture. The exponential normalization operation mathematically guarantees that the weight vector satisfies the non-negativity and normalization constraints, ensuring the legality of subsequent centroid coordinate mapping calculations. Simultaneously, the exponential characteristic of the softmax function amplifies the differences between the original scores, making the policy network's preference for different vertices more explicit and improving the certainty of decision-making. This effectively supports the intelligent, adaptive, and optimization goals of asset allocation decision-making.

[0062] This application further proposes to generate asset allocation instructions within the compliant region, with the specific calculation formula as follows:

[0063] ;

[0064] in, This indicates the asset allocation instruction. The first element in the allocated weight vector represents the weight of the first element. Each weighted element The vertex set of the compliant region represents the first... Vertex coordinate vectors This indicates the actual number of elements in the vertex set of the compliant region.

[0065] The mathematical principle of this formula is based on a fundamental theorem in convex set theory: any point in a convex set can be represented as a convex combination of the extreme points of that set. Specifically, if If A represents all the poles of a convex set C, then for any point A in C, there exists a set of non-negative weights. ,satisfy and , making Since the compliant region is a convex polyhedron defined by a system of linear inequalities, and this application accurately extracts all poles using the dual-description method, the amortized weight vector satisfies the non-negativity and normalization constraints. and According to the above theorem, the calculated result is... It must be located inside or on the boundary of the convex polyhedron in the compliance area, thus mathematically guaranteeing that it naturally satisfies all regulatory constraints without any subsequent modification or trimming.

[0066] As a preferred embodiment, it is assumed that the compliant area vertex set contains 7 vertices, and the corresponding three-dimensional coordinates (equity%, fixed income%, cash%) are as follows: P1=(30, 55, 15), P2=(30, 60, 10), P3=(25, 65, 10), P4=(20, 70, 10), P5=(10, 75, 15), P6=(0, 85, 15), P7=(0, 90, 10).

[0067] The weighted distribution vector is w = [0.18, 0.12, 0.25, 0.08, 0.20, 0.10, 0.07]. The asset allocation instruction is then calculated as follows:

[0068] The calculation yielded:

[0069] Equity Fixed income Cash ;

[0070] The final configuration command is: .verify: (Meeting the maximum rights limit) (Meets the lower limit of fixed income), 12.40% is in the range of 5%-15% (meets the cash range), the total is 100%.

[0071] Through the above technical solutions, this application achieves a constraint-embedded configuration instruction generation mechanism. The centroid coordinate mapping method utilizes the fundamental properties of convex geometry theory to mathematically guarantee that the output configuration instructions must lie within the compliance region, solving the fundamental defect of frequent policy output violations of constraints in traditional reinforcement learning methods. The mathematical rigor of this method ensures that the policy network can focus its computational resources and learning capabilities on optimizing the return objective without being interfered with by the complex logic of constraint handling, thus improving training efficiency and policy quality. Simultaneously, through end-to-end neural network training, the policy network can learn how to dynamically adjust the weight allocation of various extreme configuration schemes under different market environments and financial states, thereby flexibly finding the optimal configuration point within the geometric boundaries of compliance constraints, achieving an organic unity of compliance and profitability. This provides key technical guarantees and compliance support for deploying deep reinforcement learning-based asset allocation systems in actual financial institutions.

[0072] This application further proposes that the specific steps for calculating the manifold deviation include:

[0073] Mapping the allocated weight vector to the asset allocation space yields the original action tendency vector of the policy network before compliance constraints. The purpose of this mapping is to reveal the true intent of the policy network, i.e., what configuration scheme the policy network would prefer to adopt if no compliance constraints existed. Specifically, assuming the output layer of the policy network directly outputs an asset allocation vector (rather than an allocated weight vector), an auxiliary fully connected layer is used to convert the allocated weight vector into a representation of the asset allocation space. The weight matrix of this auxiliary layer can be initialized as an identity matrix and remains unchanged or undergoes only slight adjustments during training; its role is merely space transformation rather than additional feature extraction.

[0074] Calculate the Euclidean distance between the original action tendency vector and the asset allocation instruction, and determine the Euclidean distance as the manifold deviation.

[0075] The formula for calculating Euclidean distance is:

[0076] ;

[0077] in, This is the original tendency vector. For the actual configuration instruction, i represents the i-th asset class (i=1 corresponds to equity, i=2 to fixed income, i=3 to cash, u=3). The manifold deviation d reflects the degree to which the compliance constraint mapping distorts the original intent of the strategy. The larger d is, the stronger the constraint restricts the strategy, and the more the strategy network needs to learn to adapt to the constraint boundary.

[0078] The reverse correction process includes:

[0079] The manifold deviation is coupled as a penalty weight into the loss function of the policy network to construct a comprehensive objective function that includes a target return reward term and a compliance deviation penalty term. The design concept of the comprehensive objective function is to maximize investment returns while minimizing the deviation between the original policy tendency and the actual execution, guiding the policy network to gradually learn to output actions that naturally conform to the constraints.

[0080] The overall objective function is in the following form: ;

[0081] Where R represents the reward, the negative sign indicates that maximizing the reward is equivalent to minimizing the negative reward, and d represents the manifold deviation, the penalty coefficient. The value needs to be optimized according to the specific application scenario, and is usually set between 0.01 and 0.1. If the value is too small, the policy network will not pay enough attention to compliance learning and will still frequently output actions that are close to the constraint boundary. An excessively large value can lead to an overly conservative policy network, sacrificing performance gains in order to reduce bias.

[0082] The gradient of the comprehensive objective function with respect to the network parameters is calculated using the gradient descent algorithm, and the neuron weights of the policy network are updated based on the calculated gradient. Specifically, the Adam optimizer is used, which combines momentum and adaptive learning rate adjustment mechanisms, enabling stable and efficient optimization on complex loss function surfaces. The learning rate is set to 0.0001 to ensure the stability and convergence of the training process. Gradient calculation employs automatic differentiation, using the backpropagation algorithm to calculate the partial derivatives with respect to the network parameters of each layer sequentially, starting from the comprehensive objective function L, and then updating the parameters according to the gradient direction.

[0083] ;

[0084] in, Represents network parameters, For learning rate, This represents the gradient of the loss function L with respect to the parameter θ.

[0085] The introduction of manifold bias has a dual function: on the one hand, it quantifies the degree to which the constraint mapping distorts the policy intent. When the bias is large, it indicates that the original tendency of the policy network output is far from the compliance requirements, and the learning intensity needs to be increased. On the other hand, by incorporating the bias into the loss function, the policy network will gradually learn to output actions that are closer to the center of the compliance region during the training process, reducing mapping distortion and improving the intrinsic compliance of the decision.

[0086] As a preferred embodiment, suppose that in a certain decision, the initial tendency of the policy network is to allocate 35% equity, 50% fixed income, and 15% cash. However, due to the equity cap of 30%, the actual allocation after centroid coordinate mapping is 18.85%, 68.75%, and 12.40% (as in the previous example). Calculate the manifold deviation:

[0087] ;

[0088] This deviation reflects the difference between the original tendency and the actual execution.

[0089] In the loss function, assuming the current period reward is R = 5.2 and the penalty coefficient λ = 0.1, the overall objective function is: By updating network parameters through backpropagation, the policy network gradually reduces its tendency to allocate equity in subsequent decisions, moving closer to the compliant zone.

[0090] Through the above technical solution, this application achieves adaptive learning and constraint internalization of the policy network. The manifold bias, as a precise quantitative feedback signal, propagates the impact of compliance constraints from the output layer back to each hidden layer of the network, guiding the parameter adjustments of the entire network towards reducing bias and improving inherent compliance. This enables the policy network to learn the inherent laws and structural characteristics of the compliance boundary from the training data. The effectiveness of the constraint internalization learning mechanism has been verified, providing an efficient solution for constrained optimization problems based on deep reinforcement learning. It has broad application value in many practical application fields that require meeting complex constraints, such as asset allocation, energy scheduling, and traffic management.

[0091] This application further proposes that, after generating asset allocation instructions within the compliant region, it also includes:

[0092] The vertical distance from the asset allocation instruction to each constraint plane defined by the linear inequality matrix is ​​calculated using the constraint coefficient matrix and constraint boundary vector in the linear inequality matrix.

[0093] The vertical distance is calculated using the formula for the distance from a point to a hyperplane: ;

[0094] in Let be the coefficient vector of the i-th constraint. For the corresponding boundary values, for The Euclidean norm. The formula calculates the configuration command. The shortest distance to the i-th constraint hyperplane reflects how much safety margin there is before the current configuration is triggered.

[0095] The minimum value is extracted from the calculated vertical distances and determined as the edge safety distance. The edge safety distance reflects the distance between the current configuration command and the nearest constraint boundary; the smaller the value, the closer the configuration is to a regulatory constraint.

[0096] Based on the edge safety distance and a preset safety threshold, where the safety threshold is determined according to the risk control experience of financial institutions in asset allocation, the execution strategy of the asset allocation instruction is determined as follows:

[0097] If the edge safety distance is less than the preset safety threshold, the asset configuration instruction is determined to be in step mode, and the step size is calculated based on the edge safety distance.

[0098] Step-by-step mode breaks down configuration adjustments into multiple small-scale operations, with the step size calculated using the following formula:

[0099] (Edge safety distance / 2, minimum step size);

[0100] Ensure that each adjustment does not touch the constraint boundaries.

[0101] If the edge safety distance is greater than or equal to the preset safety threshold, the asset configuration instruction is determined to be in single-use mode. Single-use mode directly executes the complete configuration adjustment, which is suitable for scenarios where the distance constraint boundary is far and the safety margin is sufficient.

[0102] As a preferred embodiment, assuming the current configuration instructions are (18.85%, 68.75%, 12.40%), the regulatory constraints include: equity ≤ 30%, fixed income ≥ 40%, cash ≥ 5%, and cash ≤ 15%. Calculate the vertical distance between each constraint:

[0103] (Distance from the maximum equity limit);

[0104] (Distance from the lower limit of fixed income);

[0105] ;

[0106] ;

[0107] Edge safety distance .

[0108] If the preset safety threshold is 3%, then This triggers the step-by-step mode.

[0109] Calculate the step size: The configuration adjustment will be carried out in multiple steps, with each step adjusting the cash ratio by no more than 1.30%, to ensure that the 15% cap is not reached.

[0110] Through the above technical solution, this application implements a risk control mechanism for configuration execution. The edge safety distance quantifies the compliance margin of configuration instructions, providing a scientific basis for execution strategy decisions. The step-by-step mode adopts a conservative strategy when approaching constraint boundaries, using small, gradual adjustments to avoid violations caused by minor market fluctuations or data errors; the single-transaction mode improves execution efficiency and reduces the number of transactions and costs when there is sufficient safety margin. This adaptive execution mechanism effectively balances configuration flexibility and compliance stability, reduces operational risks, and enhances the reliability and practicality of asset allocation management.

[0111] This application further proposes that the asset allocation process also includes:

[0112] The rate of change of the edge safety distance is monitored in real time; the rate of change is calculated by the difference between the edge safety distances at consecutive time points, using the following formula:

[0113] rate of change ;

[0114] in, The current edge safety distance, The edge safety distance at the previous moment. The time interval is defined as follows: a positive rate of change indicates that the safety distance is increasing and the configuration is moving away from the constraint boundary; a negative rate of change indicates that the safety distance is decreasing and the configuration is moving closer to the constraint boundary.

[0115] If the rate of change exceeds a preset mutation threshold, which is determined based on the three sigma (3σ) anomaly detection principle in statistics, it is determined that a momentary drift has occurred in the current compliance area, and the currently incomplete asset allocation instructions are stopped. The mutation threshold is set to three times the standard deviation of the historical rate of change. Exceeding this threshold indicates that there has been a significant change in the regulatory environment or internal financial status.

[0116] The system resynchronizes and acquires the first market data and the second internal financial data to re-parse the compliance region vertex set based on the updated state feature vector, and dynamically updates the asset allocation instructions. This re-parse process reflects the latest constraints, ensuring that the allocation instructions always adapt to current regulatory requirements.

[0117] The mechanism is designed because regulatory rules can change abruptly due to policy adjustments, and internal financial situations can shift dramatically due to large transactions. These changes can cause the shape and boundaries of the compliance zone to drift rapidly. If the original configuration instructions are still followed, new constraint boundaries may be triggered during execution, creating a risk of non-compliance. By monitoring the rate of change of the edge safety distance in real time, abnormal changes in the compliance zone can be detected in a timely manner, triggering an emergency response mechanism.

[0118] As a preferred embodiment, it is assumed that during the configuration adjustment process, the edge safety distance suddenly decreases from 2.60% to 0.80% over a time interval of 5 minutes (0.0833 hours). The rate of change is calculated as follows:

[0119] rate of change percentage points / hour

[0120] If the standard deviation of the historical rate of change is 5 percentage points / hour and the mutation threshold is 15 percentage points / hour, then This triggered a mutation detection. The system immediately paused the current configuration adjustment and reacquired market and financial data. The inspection revealed that the regulatory authorities had just issued an emergency notice raising the minimum cash-like assets ratio from 5% to 10%, causing a contraction in the compliance zone. The vertex set was re-parsed to generate new configuration instructions, adjusting the cash ratio to 10.5% to meet the new requirement.

[0121] Through the aforementioned technical solutions, this application achieves real-time response to dynamic changes in compliance areas. The change rate monitoring mechanism can keenly capture sudden changes in the regulatory environment, avoiding configurations based on outdated compliance assumptions and effectively preventing compliance risks caused by information lag. The dynamic update mechanism ensures that the configuration strategy is always synchronized with the latest constraints, improving the adaptability and security of asset allocation management and providing reliable technical support for a complex and ever-changing financial environment.

[0122] like Figure 1 As shown, in a preferred embodiment, the solution of this application is specifically implemented as follows:

[0123] Real-time market data is obtained from major stock indices (such as the CSI 300 Index and SSE 50 Index), bond yield curves (government bonds and corporate bonds), foreign exchange rates, and commodity prices via market data interfaces. Simultaneously, secondary internal financial data, including total assets, total liabilities, detailed asset lists, cash flow statements, and future projected revenue and expenditure plans, is extracted from the internal financial management system. Data collection is set to update every 5 minutes to ensure real-time decision-making. The collected raw data undergoes preprocessing steps such as data cleaning, outlier handling, and missing value imputation to ensure data quality.

[0124] Then, using feature engineering algorithms, the mean and variance of asset prices over a sliding window of the past 20 trading days are calculated based on the first market data to characterize market volatility. The ratio of total assets to total liabilities (debt-to-equity ratio) is calculated based on the second set of internal financial data, and the cash flow coverage ratio is calculated by combining expected cash inflows and outflows for the next 6 months. Simultaneously, the duration of the asset and liability portfolios is calculated, and the interest rate sensitivity difference index is obtained through duration differences. These indicators are Z-score standardized, mapped to a distribution with a mean of 0 and a standard deviation of 1, and then concatenated according to a preset dimensional order to generate a 128-dimensional state feature vector.

[0125] Based on a pre-defined set of regulatory rules, and combined with the current total assets and liabilities, a matrix of linear inequalities concerning asset allocation variables is constructed. For example, regulatory rules may include: equity assets should not exceed 30%, fixed-income assets should not be less than 40%, cash assets should be between 5% and 15%, and the sum of all asset proportions should equal 100%. These rules can be represented as a system of linear inequalities. Using a bi-descriptive method, the convex polyhedron determined by this system of inequalities is analyzed, and the coordinates of all extreme points are systematically enumerated to form a set of compliant vertex regions. In three asset allocation scenarios, typically 6 to 10 vertex coordinates are obtained, each vertex representing an extreme but compliant allocation scheme.

[0126] A 128-dimensional state feature vector is input into a pre-trained deep neural network policy model. This policy network adopts a three-layer fully connected architecture with hidden layer dimensions of 256 and 128, respectively, and uses ReLU as the activation function. The output dimension of the final fully connected layer is preset to 15 (greater than the upper limit of the actual number of vertices) to cope with the dynamic changes in the number of vertices at different times. Based on the actual number of elements in the vertex set of the current compliant region (assuming 7 vertices are parsed at the current time), the last 8 dimensions of the output tensor are masked and zeroed out. Specifically, the values ​​of these dimensions are set to negative infinity (-1 × 10⁹). Then, the masked tensor is exponentially normalized using the softmax function to obtain a 7-dimensional amortized weight vector. The sum of all elements in this vector is strictly equal to 1, and each element is non-negative, satisfying the mathematical requirements of convex combinations.

[0127] Based on the principle of the centroid coordinate system in convex geometry, each element of the allocated weight vector is used as a coefficient to perform a weighted summation of the coordinates of each vertex in the compliant region vertex set, generating the final asset allocation instruction. According to the convex combination theorem, since the weight vector satisfies the non-negativity and normalization constraints, the calculated allocation instruction is mathematically necessarily located inside the convex polyhedron of the compliant region, naturally satisfying all regulatory constraints. For example, if the allocated weight vector is [0.18, 0.12, 0.25, 0.08, 0.20, 0.10, 0.07], and the vertex set consists of 7 three-dimensional coordinates (equity%, fixed income%, cash%), then the final allocation is a convex combination of these 7 vertices with corresponding weights, requiring no subsequent correction or pruning.

[0128] To guide the policy network in learning inherent compliance, the Euclidean distance between the original action tendency vector (obtained by directly mapping the allocated weight vector to the vertex coordinates) before constraint mapping and the actually executed asset allocation instruction is calculated as the manifold deviation. This manifold deviation is coupled as a penalty term into the loss function of the policy network, constructing a comprehensive objective function that includes an investment return reward term and a compliance deviation penalty term. The gradient of the comprehensive objective function with respect to the network parameters is calculated using the Adam gradient descent optimization algorithm, and the neuron weights of the network are updated accordingly. This allows the policy network to gradually learn the optimal decision boundary under compliance constraints during iterative training, reducing the deviation between the original tendency and the actual execution.

[0129] The application provides a high-dimensional environmental awareness foundation for reinforcement learning by aligning and standardizing the features of multi-source heterogeneous market data with internal financial data in a spatiotemporal manner. Based on this, a dual-description method is used to dynamically parse abstract linear regulatory rules into a set of extreme coordinates of a convex polyhedron. Combined with adaptive weights generated by the policy network and a barycentric coordinate system projection mapping, this mathematically ensures that the output asset allocation instructions, while satisfying normalization and non-negativity, inevitably fall within the compliant and feasible domain, eliminating computational redundancy and decision-making violation risks caused by traditional hard constraint processing methods. Furthermore, through a reverse correction mechanism of manifold deviation and a real-time monitoring strategy of edge safety distance, a deep coupling learning of compliance constraints and return objectives, as well as precise risk control during execution, are achieved. Thus, under a volatile macroeconomic environment and stringent regulatory requirements, a logical unity is achieved between the stability of asset-liability allocation decisions, inherent compliance, and the goal of optimal returns.

[0130] The above embodiments are only used to illustrate the technical methods of this application and are not intended to limit it. Although this application has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical methods of this application without departing from the spirit and scope of the technical methods of this application.

Claims

1. A dynamic asset-liability allocation optimization method based on deep reinforcement learning, characterized in that, Includes the following steps: Simultaneously acquire first market data and second internal financial data, and construct a state feature vector representing the real-time status of assets and liabilities; Based on the preset set of regulatory rules and the second internal financial data, the set of compliant region vertices corresponding to the current decision moment is parsed out. The coordinates of each vertex in the set of compliant region vertices represent the extreme point configuration ratio that satisfies the regulatory constraints. The state feature vector is input into a preset policy network to generate a weighted vector that matches the number of elements in the vertex set of the compliant region. Based on the projection mapping process of the centroid coordinate system, the allocated weight vector is used as a coefficient to calculate the coordinates of each vertex in the vertex set of the compliant area, and an asset allocation instruction within the compliant area is generated. Calculate the manifold deviation between the asset allocation instruction and the original output of the strategy network, and perform reverse correction on the network parameters of the strategy network based on the manifold deviation.

2. The dynamic asset-liability allocation optimization method based on deep reinforcement learning according to claim 1, characterized in that, The process of constructing state feature vectors includes: Based on the first market data, calculate the mean and variance of the asset price within a preset sliding window; Based on the second internal financial data, calculate the ratio of total assets to total liabilities at the current moment, and combine it with the preset cash flow time series data to calculate the cash flow coverage ratio in the future preset period. Based on the first market data and the second internal financial data, calculate the difference in interest rate sensitivity of assets and liabilities; The mean and variance, the ratio and the interest rate sensitivity difference are standardized and multidimensionally combined to generate a state feature vector representing the real-time state of assets and liabilities.

3. The dynamic asset-liability allocation optimization method based on deep reinforcement learning according to claim 1, characterized in that, The process of parsing the set of compliant region vertices corresponding to the current decision moment includes: Based on the aforementioned set of regulatory rules, construct a matrix of linear inequalities concerning the asset allocation variable: ； in, Represents the constraint coefficient matrix. This represents the action vector to be generated. Represents the constraint boundary vector; The convex polyhedron determined by the linear inequality matrix is ​​analyzed using the bi-description method. The coordinates of the poles of the convex polyhedron are extracted, and all extracted pole coordinates are determined as the vertex set of the compliant region.

4. The dynamic asset-liability allocation optimization method based on deep reinforcement learning according to claim 1, characterized in that, The process of generating the distributed weight vector is as follows: The original score tensor is output by the fully connected layer at the end of the policy network, and the preset output dimension of the fully connected layer at the end is greater than the number of elements in the vertex set of the compliant region. Based on the actual number of elements in the compliant region vertex set at the current moment, the invalid dimensions in the original score tensor that exceed the actual number of elements are masked and zeroed. The tensor after masking is subjected to exponential normalization to obtain the amortized weight vector in which the sum of the elements of each dimension is equal to one and all elements are non-negative.

5. The dynamic asset-liability allocation optimization method based on deep reinforcement learning according to claim 1, characterized in that, Generate asset allocation instructions that are within the compliant zone. The specific calculation formula is as follows: ； in, This indicates the asset allocation instruction. Represents the first element in the allocated weight vector. Each weighted element The vertex set of the compliant region represents the first... Vertex coordinate vectors This indicates the actual number of elements in the vertex set of the compliant region.

6. The dynamic asset-liability allocation optimization method based on deep reinforcement learning according to claim 1, characterized in that, The process of calculating manifold deviation includes: Mapping the allocated weight vector to the asset allocation space yields the original action tendency vector of the strategy network before it is subject to compliance constraints. Calculate the Euclidean distance between the original action tendency vector and the asset allocation instruction, and determine the Euclidean distance as the manifold deviation.

7. The dynamic asset-liability allocation optimization method based on deep reinforcement learning according to claim 1, characterized in that, The reverse correction process includes: The manifold deviation is coupled as a penalty weight to the loss function of the policy network to construct a comprehensive objective function that includes a target return reward term and a compliance deviation penalty term; The gradient of the integrated objective function with respect to the network parameters is calculated using the gradient descent algorithm, and the neuron weights of the policy network are updated based on the calculated gradient.

8. The dynamic asset-liability allocation optimization method based on deep reinforcement learning according to claim 3, characterized in that, After generating asset allocation instructions that fall within the compliant zone, the following steps are also included: The vertical distance from the asset allocation instruction to each constraint plane defined by the linear inequality matrix is ​​calculated using the constraint coefficient matrix and constraint boundary vector in the linear inequality matrix. The minimum value is extracted from the calculated vertical distances and determined as the edge safety distance.

9. The dynamic asset-liability allocation optimization method based on deep reinforcement learning according to claim 8, characterized in that, The method further includes: The execution strategy for the asset configuration instruction is determined based on the edge safety distance and the preset safety threshold. If the edge safety distance is less than the preset safety threshold, the asset configuration instruction is determined to be in step mode, and the step size is calculated based on the edge safety distance. If the edge safety distance is greater than or equal to the preset safety threshold, the asset configuration instruction will be determined as a single-use mode.

10. The dynamic asset-liability allocation optimization method based on deep reinforcement learning according to claim 9, characterized in that, The method also includes the following in the process of asset allocation: Real-time monitoring of the rate of change characteristics of the edge safety distance; If the rate of change exceeds a preset mutation threshold, it is determined that a momentary drift has occurred in the current compliant area, and the currently incomplete asset allocation instructions are stopped. The system resynchronizes and acquires the first market data and the second internal financial data to re-parse the compliant region vertex set based on the updated state feature vector, and dynamically updates the asset allocation instructions.

Images