Project risk quantification and strategy adjustment method combining reinforcement learning with AHP (Analytic Hierarchy Process), electronic equipment and storage medium

Abstract

The invention provides a project risk quantification and strategy adjustment method combining reinforcement learning with AHP (Analytic Hierarchy Process), electronic equipment and a storage medium, and relates to the field of intelligent project risk management. According to the method, the project information is integrated into the state space, so that the reinforcement learning model can more accurately understand the change of the project state, and an optimal decision is made according to the change. It is ensured that the reinforcement learning model can comprehensively understand the project state, and an optimal decision is made accordingly; through detailed state space design, key information in project progress can be captured, and a rich decision basis is provided for the model. And the risk score is converted into the key parameter in the reward function, so that the model can intuitively understand the effect of risk management, and the strategy is adjusted according to the result to realize the target of risk minimization.

Description

Technical Field

[0001] The present invention relates to the field of intelligent project risk management, and in particular to a project risk quantification and strategy adjustment method, electronic equipment and storage medium that combines reinforcement learning with AHP hierarchical analysis. Background Art

[0002] At present, in the field of project management, common schemes for project risk assessment include: using the time series relationship analysis method and the entropy method to calculate the subjective weight and objective weight of risk assessment indicators, and using the AHP hierarchical analysis method to determine the weight of risk indicators at each level.

[0003] Existing risk assessment methods lack comprehensive evaluation dimensions and fail to fully consider the various risks a project faces, resulting in inaccurate assessment results. Furthermore, existing project risk management methods are limited to manually adjusting risk management strategies. Risk management strategies implemented as project status changes cannot guarantee the minimization of overall project risk. Summary of the Invention

[0004] Purpose of the invention: To propose a project risk quantification and strategy adjustment method, electronic device and storage medium combining reinforcement learning with AHP hierarchical analysis, combining AHP with reinforcement learning algorithm, determining the weight of various dimensions of risk in the evaluation process, that is, the degree of impact of risk on the project, and comprehensively considering all dimensions of risk in the project, solving the problem of inaccurate traditional project risk assessment, and then using reinforcement learning to dynamically adjust the risk management strategy, solving the problem of dynamically adjusting the strategy according to project information data to minimize the total project risk.

[0005] In a first aspect of the present invention, a method for project risk quantification and strategy adjustment is proposed that combines reinforcement learning with AHP hierarchical analysis, and the steps are as follows:

[0006] Define the first-level dimensions of project risk as cost risk, technical risk, quality risk, environmental risk, legal compliance risk, market trend risk, and resource risk, and divide each first-level dimension into sub-dimensions;

[0007] Determine quantitative evaluation indicators for each sub-dimension;

[0008] Construct a hierarchical model and determine the weights of each risk dimension and sub-dimension through expert judgment matrix, consistency test and weight calculation;

[0009] Collect project status data corresponding to each risk indicator;

[0010] Design a state space containing project status data, an action space containing management actions, and a reward function based on risk score. Use reinforcement learning algorithm to train the model and obtain a strategy parameter file for minimizing the total risk of the project.

[0011] In a further embodiment of the first aspect, the sub-dimensions of cost risk include cost overrun risk, cost variation risk, and financial resource availability risk;

[0012] The sub-dimensions of technical risk include technical feasibility risk, technical maturity risk and technical implementation time risk;

[0013] The sub-dimensions of quality risk include quality standard compliance risk, quality control effectiveness risk and customer satisfaction risk;

[0014] The sub-dimensions of environmental risk include natural disaster risk, regulatory change risk, and management structure change risk;

[0015] The sub-dimensions of legal compliance risk include compliance assessment risk, legal litigation risk, and license application risk;

[0016] The sub-dimensions of market trend risk include market demand change risk, competition situation risk and price fluctuation risk;

[0017] The sub-dimensions of resource risk include human resource stability risk, material supply reliability risk and equipment maintenance status risk.

[0018] In a further embodiment of the first aspect, the evaluation indicator of the cost overrun risk is the degree to which the actual cost exceeds the budget, the evaluation indicator of the cost change risk is the cost change rate, and the evaluation indicator of the financial resource availability risk is the speed at which funds are in place;

[0019] The assessment indicator for the technical feasibility risk is the difficulty of technology research and development, the assessment indicator for the technical maturity risk is the success rate of technology transfer, and the assessment indicator for the technical implementation time risk is the degree to which the time required for technology implementation is less than the scheduled time;

[0020] The assessment indicator for the quality standard compliance risk is the defective product rate, the assessment indicator for the quality control effectiveness risk is the severity of quality defects, and the assessment indicator for the customer satisfaction risk is the degree of customer satisfaction;

[0021] The assessment indicator for natural disaster risk is the probability of a natural disaster affecting the project implementation; the assessment indicator for regulatory change risk is the degree of impact of relevant regulatory changes on the project; and the assessment indicator for management structure change risk is the degree of impact of changes in the core management structure on the project implementation;

[0022] The assessment indicator for compliance assessment risk is the number of non-compliance incidents, the assessment indicator for legal litigation risk is the number of pending lawsuits, and the assessment indicator for license application risk is the extent to which the license application time is shorter than the scheduled time.

[0023] The assessment indicator for market demand change risk is the degree of understanding of market growth rate, the assessment indicator for competition situation risk is the market share size of major competitors, and the assessment indicator for price fluctuation risk is the fluctuation range of the ratio of product pricing to market price;

[0024] The assessment indicator for the human resource stability risk is the employee turnover rate, the assessment indicator for the material supply reliability risk is the reputation of key material suppliers, and the assessment indicator for the equipment maintenance status risk is the equipment failure rate.

[0025] In a further embodiment of the first aspect, the constructing of the hierarchical model, determining the weights of each risk dimension and sub-dimension through an expert judgment matrix, consistency check, and weight calculation, specifically includes:

[0026] Construct a hierarchical model including target layer, dimension layer and sub-dimension layer;

[0027] Invite experts to use a 1 to 9 scale to compare elements of the same level pairwise to form a judgment matrix;

[0028] Calculate the consistency ratio CR of the judgment matrix and ensure that CR < 0.1;

[0029] The geometric mean method is used to calculate the weights of each dimension and sub-dimension.

[0030] In a further embodiment of the first aspect, the state space is a vector composed of various risk indicators, and the action space is a vector composed of management actions for each risk dimension, wherein the management actions include adjusting budgetary manpower investment, adjusting the maturity ratio of the technology stack, increasing quality control investment, formulating emergency plans, adjusting the professional requirements of legal advisors, increasing market research investment, and improving employee satisfaction.

[0031] In a further embodiment of the first aspect, the reward function is designed as follows:

[0032] r=r1+r2+r3+r4+r5+r6+r7

[0033] r1=α1·(5-Score1)

[0034] r2=α2·(5-Score2)

[0035] r3=α3·(5-Score3)

[0036] r4=α4·(5-Score4)

[0037] r5=α5·(5-Score5)

[0038] r6=α6·(5-Score6)

[0039] r7=α7·(5-Score7)

[0040] Where r is the total reward value, r1, r2, r3, r4, r5, r6, and r7 are the reward values ​​of the seven dimensions of cost risk, technical risk, quality risk, environmental risk, legal and regulatory risk, market risk, and resource risk, respectively; α1, α2, α3, α4, α5, α6, and α7 are the importance weights of the seven dimensions determined by the AHP hierarchical analysis method, and Score1, Score2, and Score3 are the weights of the seven dimensions. core2 , Score3, Score4, Score5, Score6, S core7 They are the weighted sums of the sub-dimension risks of the seven dimensions in the current state of the state space, and the importance weight of each sub-dimension is determined according to the AHP hierarchical analysis method.

[0041] In a further embodiment of the first aspect, the reinforcement learning algorithm adopts the DDPG algorithm, comprising the following steps:

[0042] Randomly initialize the critic network Q(s,a|θ Q ), Actor network μ(s|θ μ ), target critic network Q′(s,a|θ Q′ ) and the target Actor network μ′(s|θ μ′ ), where s represents the environment state, a represents the action taken in state s, θ represents the network parameters, and the upper right corner of θ represents the network corresponding to the network parameters;

[0043] Initialize the experience replay buffer, loop through collecting project status, executing actions, calculating reward values, and storing experience in the buffer;

[0044] Sample data from the buffer and update the parameters of the Critic network and Actor network;

[0045] The target network parameters are updated using a soft update method until the strategy parameter file is obtained through training.

[0046] In a further embodiment of the first aspect, the loss function updated by the critic network is:

[0047]

[0048] Where N is the amount of sampled data and γ is the discount factor.

[0049] In a further embodiment of the first aspect, the policy gradient update formula of the Actor network is:

[0050]

[0051] Update the target network:

[0052] θ Q′ ←τθ Q +(1-τ)θ Q′

[0053] θ μ′ ←τθ μ +(1-τ)θ μ′

[0054] In the formula, the arrow represents the updating process, that is, the weighted sum of the Critic network parameters and the target Critic network parameters is used as the new target Critic network parameters, and the weighted sum of the Actor network parameters and the target Actor network parameters is used as the new target Actor network parameters, where τ is the weight of the Critic network parameters and the Actor network parameters when updating the target Critic network parameters and the target Actor network parameters; when the number of inner loops reaches T times, the inner loop is jumped out, and when the number of outer loops reaches M times, the outer loop is jumped out; finally, the strategy parameter file is obtained through training, and the project status is subsequently input. According to the parameters in the strategy parameter file, the optimal strategy under the current environment state is obtained, that is, the optimal strategy under the current project state is obtained to minimize the project risk.

[0055] Compared with the prior art, the present invention has at least the following beneficial effects:

[0056] (1) By constructing a hierarchical model and using the pairwise comparison method combined with consistency testing, we can comprehensively quantify project risks, ensure the scientificity and rationality of weight calculation, and improve the objectivity and accuracy of risk assessment.

[0057] (2) Using reinforcement learning algorithms (such as the DDPG algorithm), by defining the state space, action space, and reward function, the model can continuously trial and error in a simulated environment to learn the optimal risk management strategy. It can also dynamically adjust the risk management strategy based on project information, thereby improving the flexibility and efficiency of risk management.

[0058] (3) By integrating project information into the state space, the reinforcement learning model can more accurately understand the changes in project status and make optimal decisions accordingly. This ensures that the reinforcement learning model can fully understand the project status and make optimal decisions accordingly; through detailed state space design, it can capture key information in project progress and provide the model with rich decision-making basis.

[0059] (4) By converting the risk score into a key parameter in the reward function, the model can intuitively understand the effectiveness of risk management and adjust its strategy accordingly to achieve the goal of risk minimization. The reinforcement learning model can effectively learn risk management strategies and achieve the goal of risk minimization. By designing the main body of the reward function as the risk assessment score, the model's learning goal is clearer, namely, to reduce the risk assessment score. This reward function design improves learning efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 It is the overall flow chart of the project risk quantification and strategy adjustment method in the embodiment.

[0061] Figure 2 A diagram that divides risk dimensions and sub-dimensions.

[0062] Figure 3 Flowchart for determining risk assessment dimension weights for AHP.

[0063] Figure 4 Flowchart for modeling and training reinforcement learning algorithms. DETAILED DESCRIPTION

[0064] In the following description, numerous specific details are provided to provide a more thorough understanding of the present invention. However, it will be apparent to those skilled in the art that the present invention may be practiced without one or more of these details. In other instances, certain technical features well known in the art have not been described to avoid confusion with the present invention.

[0065] The main purpose of this invention is to provide a method that can comprehensively quantify project risks and dynamically adjust risk management strategies based on project status. The main process steps are as follows:

[0066] S1. Division of risk dimensions and sub-dimensions

[0067] The first-level dimensions include: cost risk, technology risk, quality risk, environmental risk, legal and compliance risk, market trend risk, and resource risk. The second-level dimensions include: sub-risks of cost risk include cost overrun risk, cost change risk, and financial resource availability risk; sub-risks of technology risk include technology feasibility risk, technology maturity risk, and technology implementation time risk; sub-risks of quality risk include quality standard compliance risk, quality control effectiveness risk, and customer satisfaction risk; environmental risk includes natural disaster risk, regulatory change risk, and management structure change risk; sub-risks of legal and compliance risk include compliance assessment risk, legal litigation risk, and license application risk; sub-risks of market trend risk include market demand change risk, competitive situation risk, and price fluctuation risk; and sub-risks of resource risk include human resource stability risk, material supply reliability risk, and equipment maintenance risk.

[0068] S2. Determination of risk indicators (project status)

[0069] The sub-risk assessment indicators for cost risk are the degree to which actual costs exceed budgets, the rate of cost change, and the speed of funding availability. The sub-risk assessment indicators for technology risk are the difficulty of technology R&D, the success rate of technology transfer, and the degree to which technology implementation time is shorter than the scheduled time. The sub-risk assessment indicators for quality risk are the defective product rate, the severity of quality defects, and customer satisfaction. The sub-risk assessment indicators for environmental risk are the probability of natural disasters impacting project implementation, the impact of changes in relevant regulations on the project, and the impact of changes in the core management structure on project implementation. The sub-risk assessment indicators for legal and compliance risk are the number of violations, the number of pending lawsuits, and the degree to which license application time is shorter than the scheduled time. The sub-risk assessment indicators for market trend risk are the degree of understanding of market growth rate, the market share of major competitors, and the fluctuation in the ratio of product pricing to market price. The assessment indicators for resource risk are employee turnover rate, the reputation of key material suppliers, and equipment failure rate.

[0070] S3. AHP analytic hierarchy process to determine the weights of risk assessment dimensions

[0071] Constructing a Hierarchical Structure: Based on the above divisions, a hierarchical model was constructed, including a target layer, a dimension layer, and a sub-dimension layer. Constructing an Expert Judgment Matrix: Invite experts in the project domain to use a scale of 1 to 9 to compare elements within the same hierarchy, forming a judgment matrix. Consistency Verification: Calculate the consistency ratio (CR) of the expert judgment matrix, ensuring that the CR is less than 0.1 to verify the consistency of the judgment matrix. Weight Calculation: Use the geometric mean method to calculate the weights of each dimension and sub-dimension.

[0072] Examples of expert judgment matrices are shown in Tables 1 to 8 below:

[0073] Table 1: First level dimensions of risk dimensions

[0074]

[0075] Table 2: Second level of risk dimensions

[0076] Cost overrun risk Cost change risk Financial resource availability risk Cost overrun risk 1 2 3 Cost change risk 1 / 2 1 2 Financial resource availability risk 1 / 3 1 / 2 1

[0077] Table 3: Sub-risks of technical risks

[0078] Technical feasibility risk Technology maturity risk Technical implementation time risk Technical feasibility risk 1 2 3 Technology maturity risk 1 / 2 1 2 Technical implementation time risk 1 / 3 1 / 2 1

[0079] Table 4: Sub-risks of quality risk

[0080] Quality standards compliance risk Quality control effectiveness risks Customer satisfaction risk Quality standards compliance risk 1 2 3 Quality control effectiveness risks 1 / 2 1 2 Customer satisfaction risk 1 / 3 1 / 2 1

[0081] Table 5: Environmental risks

[0082] Natural disaster risks Risk of regulatory changes Managing structural change risks Natural disaster risks 1 2 3 Risk of regulatory changes 1 / 2 1 2 Managing structural change risks 1 / 3 1 / 2 1

[0083] Table 6: Legal and Compliance Risk Sub-Risks

[0084] Compliance Assessment Risk Legal litigation risks License application risks Compliance Assessment Risk 1 2 3 Legal litigation risks 1 / 2 1 2 License application risks 1 / 3 1 / 2 1

[0085] Table 7: Sub-risks of market trend risk

[0086] Risk of changes in market demand Competitive situation risk Price volatility risk Risk of changes in market demand 1 2 3 Competitive situation risk 1 / 2 1 2 Price volatility risk 1 / 3 1 / 2 1

[0087] Table 8: Sub-risks of resource risk

[0088] Human resource stability risk Material supply reliability risk Equipment maintenance status risk Human resource stability risk 1 2 3 Material supply reliability risk 1 / 2 1 2 Equipment maintenance status risk 1 / 3 1 / 2 1

[0089] The numbers in the above expert judgment matrix mean the importance of the previous risk relative to the latter risk. For example, in the expert judgment matrix of the seven risks in the dimensional layer, the number in the first row and second column is 3. A number greater than 1 indicates that cost risk is more important than technical risk, and 3 indicates an importance value within the range of [1,9]. Therefore, the number in the second row and first column on the other side of the diagonal line is 1 / 3, indicating that the importance of technical risk relative to cost risk is 1 / 3. A number less than 1 indicates that technical risk is less important than cost risk, and 1 / 3 indicates a degree of unimportance within the range of [0,1].

[0090] The attribute weights obtained by the hierarchical analysis method need to be tested by a matrix to determine whether the importance coefficients between the attributes are reasonable. The essence of the matrix test is to determine the size of the random consistency ratio CR. If the ratio value is less than 0.1, the matrix is ​​judged to be reasonable. The calculation formulas for the consistency ratio CI and the random consistency ratio CR are:

[0091]

[0092] where λ max is the maximum eigenvalue of the expert judgment matrix, n is the dimension of the matrix, the dimension layer is 7 here, the sub-dimension layer is 3, and the average random consistency index is shown in Table 9:

[0093] Table 9: Average random consistency index

[0094]

[0095] According to the calculation formula, the CR of the dimension layer is 0.03127, and the CR of the sub-dimension layer is 0.00876, both of which are less than 0.1, meeting the consistency requirements.

[0096] Weight calculation: First, standardize each column of the expert judgment matrix, then sum the standardized elements by row, and standardize the summed results. The judgment matrix is:

[0097]

[0098] The normalized matrix is:

[0099]

[0100] The standardized elements are summed row by row, and the result of the normalized sum is: [0.35351150.1611645 0.0990762 0.2292428 0.0438589 0.0765332 0.0366129] T Therefore, the importance weights of each risk in the seven first-level risk assessment dimensions are: α1 = 35.35115%, α2 = 16.11645%, α3 = 9.90762%, α4 = 22.92428%, α5 = 4.38589%, α6 = 7.65332%, and α7 = 3.66129%. In the specific embodiment of the present invention, the expert judgment matrix for each sub-dimension of risk is set to the same matrix, and the same processing method as for the risk and assessment dimensions is used to obtain the importance weights of each sub-risk: β1 = 53.8961%, β2 = 29.72583%, and β3 = 16.37807%.

[0101] S4. Data Collection

[0102] Data Collection: Cost Data: Degree to which actual costs exceed budget (u1=1), Cost Change Rate (u2=3), Speed ​​of Funding Arrival (u3=5). Technical Data: Difficulty of Technology R&D (b1=1), Success Rate of Technology Transfer (b2=3), Degree to which the time required for technology implementation is shorter than the scheduled time (b3=5). Quality Data: Defective Product Rate (c1=1), Severity of Quality Defects (c2=3), Customer Satisfaction (c3=5). Environmental Data: Probability of Natural Disasters Impacting Project Implementation (d1=1), Impact of Changes in Relevant Regulations on Project Implementation (d2=3), Impact of Changes in Core Management Structure on Project Implementation (d3=5). Legal Compliance Data: Number of Non-Compliance Incidents (e1=1), Number of Pending Litigation (e2=3), Degree to which License Application Time is Shorter than the Scheduled Time (e3=5). Market Trend Data: Understanding of Market Growth Rate (f1=1), Market Share of Major Competitors (f2=3), Variation in Product Pricing to Market Price Ratio (f3=5). Resource Data: Employee Turnover Rate (g1=1), Reputation of Key Material Suppliers (g2=3), Equipment Failure Rate (g3=5).

[0103] S5. Reinforcement Learning Algorithm Modeling and Training

[0104] State space design: s = (u1,u2,u3,b1,b2,b3,c1,c2,c3,d1,d2,d3,e1,e2,e3,f1,f2,f3,g1,g2,g3)

[0105] Action space design: a = (h1, h2, h3, j1, j2, j3, k1, k2, k3, l1, l2, l3, o1, o2, o3, p1, p2, p3, q1, q2, q3). All variables in the action space range from integers in the interval [-2, 2]. Cost risk actions include: adjusting the manpower and time investment in budgeting to h1 = +2, adjusting the purchase order change frequency to h2 = +0, and adjusting the degree of control over resource availability to h3 = +2. Technology risk actions include: adjusting R&D investment to j1 = +2, adjusting the proportion of mature technologies in the technology stack to j2 = 0, and adjusting the technical level requirements of the technical consultants to j3 = +2. Quality risk actions include: adjusting the quality control level to k1 = +2, adjusting the testing frequency to k2 = 0, and adjusting the investment in customer communication training to k3 = +2. Environmental Risk Actions: Adjust the investment in emergency planning to l1 = +2, the frequency of legal-related project planning to l2 = 0, and the probability of changes to the core management structure to l3 = +2. Legal Compliance Risk Actions: Adjust the professionalism requirements of hired legal counsel to o1 = +2, the frequency of compliance training to o2 = +0, and the manpower required for license applications to o3 = +2. Market Trend Risk Actions: Adjust the manpower and material investment in market research to p1 = -2, adjust product positioning to p2 = +0, and adjust the frequency of pricing changes based on market prices to p3 = -2. Resource Risk Actions: Adjust employee satisfaction to q1 = +2, adjust the degree of supplier diversity to q2 = +0, and adjust the frequency of equipment maintenance to q3 = -2.

[0106] The impact of actions on the state (the process of interaction between the agent and the environment): As shown in Table 10, for example, when the action of "adjusting the manpower and time input of the budget work by +1" is taken, the "actual cost is greater than the budget" in the state vector will be "-1". At the same time, based on the actual cost is greater than the budget value after "-1", the corresponding risk value is determined in combination with the evaluation index range. For example, if the current actual cost is greater than the budget value is 2, and the actual cost is greater than the budget value after taking the "-1" action, the corresponding risk value is 1. This risk value is one of the parameters when calculating the reward function.

[0107] Table 10: Effects of actions on states (the process of interaction between the agent and the environment)

[0108]

[0109]

[0110]

[0111]

[0112] Reward function design:

[0113] r=r1+r2+r3+r4+r5+r6+r7

[0114] r1=α1·(5-Score1)

[0115] r2=α2·(5-Score2)

[0116] r3=α3·(5-Score3)

[0117] r4=α4·(5-Score4)

[0118] r5=α5·(5-Score5)

[0119] r6=α6·(5-Score6)

[0120] r7=α7·(5-Score7)

[0121] r is the total reward value, r1, r2, r3, r4, r5, r6, and r7 are the reward values ​​of the seven dimensions of cost risk, technical risk, quality risk, environmental risk, legal and regulatory risk, market risk, and resource risk, respectively.

[0122] α1=35.35115%, α2=16.11645%, α3=9.90762%, α4=22.92428%, α5=4.38589%, α6=7.65332%,

[0123] α7=3.66129% are the importance weights of the seven dimensions determined by AHP, Score1, Score2, Score3, Score4, Score5, Score6, Score7, Score8, Score9, Score10, Score11, Score12, Score13, Score14, Score15, Score16, Score17, Score18, Score19, Score19, Score11 core2 , Score3, Score4, Score5, Score6, S core7 The weighted sum of the sub-dimension risks of the seven dimensions in the current state of the state space is obtained. The importance weight of each sub-dimension is determined according to the AHP:

[0124] β1=53.8961%

[0125] β2=29.72583%

[0126] β3=16.37807%

[0127] In the current state t =(u1,u2,u3,b1,b2,b3,c1,c2,c3,d1,d2,d3,e1,e2,e3,f1,f2,f3,g1,g2,g3) = (1,3,5,1,3,5,1,3,5,1,3,5,1,3,5,1,3,5), take action:

[0128] a t =(h1,h2,h3,j1,j2,j3,k1,k2,k3,l1,l2,l3,o1,o2,o3,p1,p2,p3,q1,q2,q3)

[0129] =(2,0,2,2,0,2,2,0,2,2,0,2,2,0,2,-2,0,-2,2,0,-2)

[0130] The new state s t+1 =(u1,u2,u3,b1,b2,b3,c1,c2,c3,d1,d2,d3,e1,e2,e3,f1,f2,f3,g1,g2,g3)=(3,3,7,3,3,7,3,3,7,3,3,7,3,3,7,-1,3,3,3,3,3).

[0131] The corresponding risk value vector of this state is rv t+1 =(3,3,1,3,3,1,3,3,1,3,3,5,3,3,1,5,3,3,3,3,3). Taking Score 1 of cost risk as an example, the sub-risk values ​​are 3, 3, and 7 respectively. Therefore, Score 1 = 3×53.8961%+3×29.72583%+1×16.37807%=2.6724386. The scores of the other 6 risks are calculated in the same way as above, namely 2.6724386, 2.6724386, 3.3275614, 2.6724386, 4.077922, and 1. Therefore, we can get r t =2.6708016999.

[0132] After the three core designs of the state space, action space, and reward function are determined, a reinforcement learning algorithm is selected for training. For example, the DDPG algorithm is selected as the reinforcement learning algorithm of the present invention. The algorithm flow is as follows:

[0133] Randomly initialize the critic network Q(s,a|θ Q ), Actor network μ(s|θ μ ), target critic network Q′(s,a|θ Q′ ) and the target Actor network μ′(s|θ μ′ ), where s represents the environmental state.

[0134] In the present invention, it is the project status, that is:

[0135] s=(u1,u2,u3,b1,b2,b3,c1,c2,c3,d1,d2,d3,e1,e2,e3,f1,f2,f3,g1,g2,g3)=

[0136] (1,3,5,1,3,5,1,3,5,1,3,5,1,3,5,1,3,5,1,3,5).

[0137] a represents the action taken in state s, namely:

[0138] a=(h1,h2,h3,j1,j2,j3,k1,k2,k3,l1,l2,l3,o1,o2,o3,p1,p2,p3,q1,q2,q3)=

[0139] (2,0,2,2,0,2,2,0,2,2,0,2,2,0,2,-2,0,-2,2,0,-2).

[0140] θ represents the network parameter. The symbol in the upper right corner of θ represents the network corresponding to the network parameter. For example, the superscript of the network parameter corresponding to the Critic network is Q, and the superscript of the network parameter corresponding to the target Critic network is Q′. Initialize the experience playback buffer H and loop through it 10,000 times. The process of each loop traversal is as follows: initialize random noise to obtain the initial state s, which is the collected project state information in this invention. Loop through it 100,000 times, that is, execute 100,000 steps. The process of each traversal of this layer is as follows: execute action a t , represents the action taken at step t. Combining AHP and reward function, we can get the reward value r t , and can be based on a t and the current environment state s t (In this invention, it is the project status) to obtain the environmental status s t+1 , this experience (s t ,a t ,r t ,s t+1 ) is stored in the experience buffer H, and 1000 data (s) are sampled in batches from the experience buffer i ,a i ,r i ,s i+1 ), use the target Critic network to calculate the target Q value y i :

[0141] y i =r i +γQ′(s t+1 ,μ′(s i+1 |θ μ′ )|θQ′ )

[0142] The Q value is calculated using the Critic network. i ,a i |θ Q ), and then minimize the loss function L to update the Critic network:

[0143]

[0144] Sampling policy gradients to update the Actor policy network:

[0145]

[0146] Update the target network:

[0147] θ Q′ ←τθ Q +(1-τ)θ Q′

[0148] θ μ′ ←τθ μ +(1-τ)θ μ′

[0149] The arrows represent the updating process, i.e., the weighted sum of the Critic network parameters and the target Critic network parameters is used as the new target Critic network parameters, and the weighted sum of the Actor network parameters and the target Actor network parameters is used as the new target Actor network parameters, where τ is the weight of the Critic network parameters and the Actor network parameters when updating the target Critic network parameters and the target Actor network parameters. When the number of inner loops reaches 100,000, the inner loop is exited, and when the number of outer loops reaches 10,000, the outer loop is exited. The final training results in a strategy parameter file, and the environment state (i.e., the project state in the present invention) can be input later. The parameters in the strategy parameter file are used as reinforcement learning network parameters to obtain the optimal strategy under the current environment state, i.e., the optimal strategy under the current project state is obtained to minimize project risks.

[0150] The technical features of the above embodiments may be combined in any manner. To simplify the description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification. Although the present invention has been shown and described with reference to specific preferred embodiments, it should not be interpreted as limiting the invention itself. Various changes in form and details may be made without departing from the spirit and scope of the invention as defined in the appended claims.

Claims

1. A project risk quantification and strategy adjustment method based on reinforcement learning combined with AHP hierarchical analysis, characterized by: The steps include: Define the first-level dimensions of project risk as cost risk, technical risk, quality risk, environmental risk, legal compliance risk, market trend risk, and resource risk, and divide each first-level dimension into sub-dimensions; Determine quantitative evaluation indicators for each sub-dimension; Construct a hierarchical model and determine the weights of each risk dimension and sub-dimension through expert judgment matrix, consistency test and weight calculation; Collect project status data corresponding to each risk indicator; Design a state space containing project status data, an action space containing management actions, and a reward function based on risk score. Use reinforcement learning algorithm to train the model and obtain a strategy parameter file for minimizing the total risk of the project.

2. The project risk quantification and strategy adjustment method combining reinforcement learning with AHP hierarchical analysis according to claim 1 is characterized in that: The sub-dimensions of cost risk include cost overrun risk, cost variation risk, and financial resource availability risk; The sub-dimensions of technical risk include technical feasibility risk, technical maturity risk and technical implementation time risk; The sub-dimensions of quality risk include quality standard compliance risk, quality control effectiveness risk and customer satisfaction risk; The sub-dimensions of environmental risk include natural disaster risk, regulatory change risk, and management structure change risk; The sub-dimensions of legal compliance risk include compliance assessment risk, legal litigation risk, and license application risk; The sub-dimensions of market trend risk include market demand change risk, competition situation risk and price fluctuation risk; The sub-dimensions of resource risk include human resource stability risk, material supply reliability risk and equipment maintenance status risk.

3. The project risk quantification and strategy adjustment method combining reinforcement learning with AHP hierarchical analysis according to claim 1 is characterized in that: The hierarchical structure model is constructed to determine the weights of each risk dimension and sub-dimension through expert judgment matrix, consistency test and weight calculation, specifically including: Construct a hierarchical model including target layer, dimension layer and sub-dimension layer; Invite experts to use a 1 to 9 scale to compare elements of the same level pairwise to form a judgment matrix; Calculate the consistency ratio CR of the judgment matrix and ensure that CR < 0.1; The geometric mean method is used to calculate the weights of each dimension and sub-dimension.

4. The project risk quantification and strategy adjustment method combining reinforcement learning with AHP hierarchical analysis according to claim 1 is characterized in that: The state space is a vector composed of various risk indicators, and the action space is a vector composed of management actions for each risk dimension, where management actions include adjusting budget and human resources investment, adjusting the maturity ratio of the technology stack, increasing quality control investment, formulating emergency plans, adjusting the professional requirements of legal advisors, increasing market research investment, and improving employee satisfaction.

5. The project risk quantification and strategy adjustment method combining reinforcement learning with AHP hierarchical analysis according to claim 1 is characterized in that: The reward function is designed as: r＝r1+r2+r3+r4+r5+r6+r7 r1=α1·(5-Score1) r2=α2·(5-Score2) r3=α3·(5-Score3) r4=α4·(5-Score4) r5=α5·(5-Score5) r6=α6·(5-Score6) r7=α7·(5-Score7) Where r is the total reward value, r1, r2, r3, r4, r5, r6, and r7 are the reward values ​​of the seven dimensions of cost risk, technical risk, quality risk, environmental risk, legal and regulatory risk, market risk, and resource risk, respectively; α1, α2, α3, α4, α5, α6, and α7 are the importance weights of the seven dimensions determined by the AHP hierarchical analysis method, and Score1, Score2, and Score3 are the weights of the seven dimensions. core2 , Score3, Score4, Score5, Score6, S core7 They are the weighted sums of the sub-dimension risks of the seven dimensions in the current state of the state space, and the importance weight of each sub-dimension is determined according to the AHP hierarchical analysis method.

6. The project risk quantification and strategy adjustment method combining reinforcement learning with AHP hierarchical analysis according to claim 1 is characterized in that: The reinforcement learning algorithm adopts the DDPG algorithm. The following steps are involved: Randomly initialize the critic network Q(s,a|θ Q ), Actor network μ(s|θ μ ), target critic network Q ′ (s,a|θ Q′ ) and the target Actor network μ ′ (s|θ μ′ ), where s represents the environment state, a represents the action taken in state s, θ represents the network parameters, and the upper right corner of θ represents the network corresponding to the network parameters; Initialize the experience replay buffer, loop through collecting project status, executing actions, calculating reward values, and storing experience in the buffer; Sample data from the buffer and update the parameters of the Critic network and Actor network; The target network parameters are updated using a soft update method until the strategy parameter file is obtained through training.

7. The project risk quantification and strategy adjustment method combining reinforcement learning with AHP hierarchical analysis according to claim 6 is characterized in that: The loss function of the Critic network update is: y i =r i +γQ ′ (s i+1 ,m ′ (s i+1 |θ μ′ )|θ Q′ ) Where N is the amount of sampled data and γ is the discount factor.

8. The project risk quantification and strategy adjustment method combining reinforcement learning with AHP hierarchical analysis according to claim 6 is characterized in that: The policy gradient update formula of the Actor network is: Update the target network: θQ ′ ←τθQ+(1-τ)θQ ′ tm ′ ←tthm+(1-t)thm ′ In the formula, the arrow represents the updating process, that is, the weighted sum of the Critic network parameters and the target Critic network parameters is used as the new target Critic network parameters, and the weighted sum of the Actor network parameters and the target Actor network parameters is used as the new target Actor network parameters, where τ is the weight of the Critic network parameters and the Actor network parameters when updating the target Critic network parameters and the target Actor network parameters; when the number of inner loops reaches T times, the inner loop is jumped out, and when the number of outer loops reaches M times, the outer loop is jumped out; finally, the strategy parameter file is obtained through training, and the project status is subsequently input. According to the parameters in the strategy parameter file, the optimal strategy under the current environment state is obtained, that is, the optimal strategy under the current project state is obtained to minimize the project risk.

9. An electronic device, characterized in that: The device includes: a processor and a memory storing computer program instructions; when the processor executes the computer program instructions, the project risk quantification and strategy adjustment method combining reinforcement learning with AHP hierarchical analysis according to any one of claims 1 to 8 is implemented.

10. A computer-readable storage medium, characterized in that The storage medium stores at least one executable instruction, and when the executable instruction is executed on the electronic device, the electronic device executes the project risk quantification and strategy adjustment method combining reinforcement learning with AHP hierarchical analysis according to any one of claims 1 to 8.

Images