Shield tunneling machine cutter maintenance decision-making method, equipment, medium and computer program product

Abstract

The invention relates to the technical field of shield tunneling machine cutter maintenance, and discloses a shield tunneling machine cutter maintenance decision-making method and device, a medium and a computer program product, and the method comprises the steps: initializing a state; defining a sequence decision process in a simulation environment: according to a maintenance action and an updated state calculation moment award, storing an experience tuple into an experience playback buffer area; randomly sampling a batch of empirical tuples from an empirical playback buffer area, minimizing a loss function by using a gradient descent method, and updating policy network parameters of an inspection agent and a maintenance agent; and repeating the above process until a preset termination condition is reached. According to the method, the problem of passive agents in multi-agent reinforcement learning is solved by constructing a sequential simulation environment and adopting a value decomposition network architecture, and the inspection interval can be dynamically adjusted according to different degradation conditions of the shield tunneling machine cutter, so that the maintenance strategy is optimized, and the maintenance cost is reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of shield machine cutter maintenance, and in particular to a shield machine cutter maintenance decision-making method, equipment, medium and computer program product. Background Art

[0002] As the main propulsion system for shield machine construction and excavation, the status of shield machine cutters has a significant impact on the safety and performance of tunnel shield construction. Due to their complex operating conditions and design, the maintenance task of shield machine cutters is very arduous, resulting in high maintenance costs. To address this problem, maintenance strategies have gradually shifted from time-based maintenance (TBM) to condition-based maintenance (CBM). Time-based maintenance (TBM) performs maintenance according to a fixed schedule, while condition-based maintenance (CBM) recommends appropriate maintenance actions by continuously monitoring the degradation status of shield machine cutters. However, while frequent health monitoring ensures reliability, it also leads to a large amount of resource waste, further increasing maintenance costs.

[0003] In recent years, many researchers have made significant progress in maintenance strategy optimization. Mainstream methods usually use heuristic algorithms to transform maintenance problems into mathematical models by constructing cost-related optimization objectives. Although these methods are effective in specific scenarios, they usually require complex mathematical models and expert knowledge to determine maintenance thresholds, which limits their versatility. With the development of deep reinforcement learning (DRL), its application in maintenance strategy optimization has shown great potential. The advantage of deep reinforcement learning (DRL) is that it does not require preset maintenance thresholds and can optimize maintenance decisions through autonomous learning. However, these methods still rely on frequent inspections, resulting in inefficient resource allocation. Therefore, optimizing inspection intervals has become the focus of current research.

[0004] Existing inspection interval optimization methods typically use fixed intervals and are unable to adapt to the dynamic changes in equipment degradation. For example, Mendes and Lorenzoni optimized the inspection intervals of cold standby systems using Monte Carlo simulations, and Sharifi and Taghipour proposed an inspection interval optimization model for K-out-of-N systems. However, these methods struggle to address the complex degradation processes of high-risk equipment, such as shield machine cutters. Furthermore, most existing studies consider inspection and maintenance as the same step, lacking in-depth exploration of their coordinated optimization. Summary of the Invention

[0005] In order to solve the above technical problems, the present invention provides a shield machine tool maintenance decision-making method based on collaborative-sequential optimization. Taking into account the inspection interval and maintenance action at the same time, unlike directly extending the single-agent two-layer Deep Q Network (Deep Q-Network, DQN) structure to a multi-agent setting, a value decomposition network (VDN) architecture is used to handle passive agents. The method proposed in the present invention can provide dynamic inspection intervals for different degradation conditions instead of fixed inspection intervals. In view of the fact that multiple agents in the current multi-agent reinforcement learning algorithm make decisions simultaneously, the present invention establishes a sequential environment to convert sequential decisions into simultaneous decisions.

[0006] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0007] In a first aspect, the present invention provides a shield machine tool maintenance decision method based on collaborative-sequential optimization, comprising the following steps:

[0008] S1, initialization state, which includes the degradation value and operation state of all shield machine cutters; initialization of the strategy network parameters of the inspection agent and the maintenance agent;

[0009] S2, define the sequential decision-making process in the simulation environment: the checking agent is based on the state s at time t t Choose the checking interval τ t , after τ t The status of the shield machine cutter after a cycle is updated as follows: The maintenance agent is based on the updated state Select the maintenance action a at time t t ;

[0010] S3, according to maintenance action a t and the updated status Calculate the reward r at time t t ,The design goal of the reward function is to minimize the ,maintenance cost of the shield machine cutter and the ,maintenance cost of the shield machine cutter includes inspection ,cost, downtime cost and maintenance cost;

[0011] S4, the experience tuple Store to the experience replay buffer;

[0012] S5, randomly sampling a batch of experience tuples from the experience replay buffer, minimizing the loss function using gradient descent, and updating the policy network parameters of the inspection agent and the maintenance agent; wherein, the overall Q function of the team consisting of the inspection agent and the maintenance agent is decomposed into the sum of the sub-Q functions of each agent through the value decomposition network; the error between the overall Q function and the target value consisting of the weighted sum of the current reward and the expected future reward is used as the loss function;

[0013] S6, repeating steps S2 to S5 until a preset termination condition is reached.

[0014] In one embodiment, the state includes degradation values ​​and operating states of all shield machine cutters, specifically including:

[0015] s t =[o1,…,o n ,d1,…,d n ];

[0016] o n Indicates the operating status of the nth shield machine cutter, d n It represents the degradation value of the nth shield machine cutter, where n is the total number of shield machine cutters.

[0017] In one embodiment, the checking agent is based on the state s at time t. t Choose the checking interval τ t , after τ t The status of the shield machine cutter after a cycle is updated as follows: Specifically include:

[0018] The inspection interval is adjusted based on the degradation state of the shield machine tool. The degradation process of the shield machine tool is modeled using the Wiener process. The lower the degradation value of the shield machine tool, the longer the inspection interval is selected; the inspection interval τ t The value range of is {1, 2, …, H-1, H}, where H is the maximum inspection interval;

[0019] After τ t After running cycles, the update formula of degradation value is: n (t+τ t )=d n (t)+Δd n (τ t );Δd n (τ t ) is the degenerate increment and obeys the normal distribution.

[0020] In one embodiment, the maintenance agent is based on the updated state Select the maintenance action a at time t t , specifically including:

[0021] Maintain the agent according to the updated state The degradation value of determines the maintenance action a t :

[0022] a t =[m1,…,m n ];

[0023] Among them, m n ∈{0,1} indicates whether the nth shield machine cutter is maintained, m n =1 means to maintain the nth shield machine cutter, m n =0 means no maintenance is performed on the nth shield machine cutter, where n is the total number of shield machine cutters.

[0024] In one embodiment, the maintenance action a t and the updated status Calculate the reward r at time t t The design goal of the reward function is to minimize the shield machine tool maintenance cost. The shield machine tool maintenance cost includes inspection cost, downtime cost and maintenance cost, specifically including:

[0025] The inspection agent and the maintenance agent have the same reward function; the reward r t for:

[0026] r t =-cost;

[0027] The shield machine tool maintenance cost includes the inspection cost c ins , downtime cost c down and maintenance costs c rep :

[0028]

[0029] Among them, c fail is the additional maintenance cost after failure; I is an indicator variable indicating whether there is a shield machine cutter failure and no maintenance is performed; m i Indicates whether to perform maintenance on the i-th shield machine cutter; o i It represents the operating status of the i-th shield machine cutter, and n is the total number of shield machine cutters.

[0030] In one embodiment, the overall Q function of the team consisting of the inspection agent and the maintenance agent is decomposed into the sum of the sub-Q functions of each agent through the value decomposition network, specifically including:

[0031] The value decomposition network architecture is applied to the inspection agent and the maintenance agent;

[0032] Check the agent's sub-Q function Q Ins According to the current state t and the inspection interval τ t calculate:

[0033]

[0034] represents the calculation expectation, k represents the offset of the time step, γ k represents the discount factor, r t+k represents the immediate reward obtained at time t+k;

[0035] Maintain the agent's sub-Q function Q Act According to the updated status and maintenance action a t calculate:

[0036]

[0037] The overall Q function Q of the team consisting of the agent and the maintenance agent total Breaks down to:

[0038]

[0039] In one embodiment, the error between the overall Q function and a target value consisting of a weighted sum of current reward and future expected reward is used as the loss function, specifically including:

[0040] Target value y total for:

[0041]

[0042] γ is the discount factor, Q total is the overall Q function;

[0043] Overall Q function Q total It is defined as the sum of the sub-Q functions of all agents:

[0044]

[0045] s represents the global state, a=(a1,a2,…,a N ) represents the joint action of all agents, a N is the action of the Nth agent, N is the total number of agents, h i represents the local observation history of the i-th agent, Q i (h i ,a i ) represents the sub-Q function of the i-th agent;

[0046] Loss Function for:

[0047]

[0048] m is the number of samples randomly sampled from the experience replay buffer.

[0049] In a second aspect, the present invention provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the method of any one embodiment of the first aspect when executing the computer program.

[0050] In a third aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method of any one of the embodiments of the first aspect.

[0051] In a fourth aspect, the present invention provides a computer program product, comprising a computer program, which, when executed by a processor, implements the steps of the method of any one embodiment of the first aspect.

[0052] Compared with the prior art, the beneficial technical effects of the present invention are:

[0053] By constructing a sequential simulation environment and adopting a value decomposition network (VDN) architecture, the passive agent problem in multi-agent reinforcement learning is solved, and the inspection interval can be dynamically adjusted according to the different degradation conditions of the shield machine tool, thereby optimizing the maintenance strategy and reducing maintenance costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 is a flow chart of a method in an embodiment of the present invention;

[0055] Figure 2 FIG. 4 is a maintenance cost diagram for different maintenance intervals in an embodiment of the present invention. DETAILED DESCRIPTION

[0056] A preferred embodiment of the present invention will be described in detail below with reference to the accompanying drawings.

[0057] like Figure 1 As shown, the present invention provides a shield machine tool maintenance decision method based on collaborative-sequential optimization, comprising the following steps:

[0058] S1, initialization state, which includes the degradation value and operation state of all shield machine cutters; initialization of the strategy network parameters of the inspection agent and the maintenance agent;

[0059] S2, define the sequential decision-making process in the simulation environment: the checking agent is based on the state s at time t t Choose the checking interval τ t , after τ t The status of the shield machine cutter after a cycle is updated as follows: The maintenance agent is based on the updated state Select the maintenance action a at time t t ;

[0060] S3, according to maintenance action a t and the updated status Calculate the reward r at time t t ,The design goal of the reward function is to minimize the ,maintenance cost of the shield machine cutter and the ,maintenance cost of the shield machine cutter includes inspection ,cost, downtime cost and maintenance cost;

[0061] S4, the experience tuple Store to the experience replay buffer;

[0062] S5, randomly sampling a batch of experience tuples from the experience replay buffer, minimizing the loss function using gradient descent, and updating the policy network parameters of the inspection agent and the maintenance agent; wherein, the overall Q function of the team consisting of the inspection agent and the maintenance agent is decomposed into the sum of the sub-Q functions of each agent through the value decomposition network; the error between the overall Q function and the target value consisting of the weighted sum of the current reward and the expected future reward is used as the loss function;

[0063] S6, repeating steps S2 to S5 until a preset termination condition is reached.

[0064] In one embodiment, the status in step S1 includes degradation values ​​and operating status of all shield machine cutters, specifically including:

[0065] s t =[o1,…,o n ,d1,…,d n ];

[0066] o n Indicates the operating status of the nth shield machine cutter, d n It represents the degradation value of the nth shield machine cutter, where n is the total number of shield machine cutters.

[0067] In one embodiment, the checking agent in step S2 is based on the state s at time t. t Choose the checking interval τ t , after τ t The status of the shield machine cutter after a cycle is updated as follows: Specifically include:

[0068] The inspection interval is adjusted based on the degradation state of the shield machine tool. The degradation process of the shield machine tool is modeled using the Wiener process. The lower the degradation value of the shield machine tool, the longer the inspection interval is selected; the inspection interval τ t The value range of is {1,2,…,H-1,H}, where H is the maximum inspection interval;

[0069] After τ t After running cycles, the update formula of degradation value is:n (t+τ t )=d n (t)+Δd n (τ t );Δd n (τ t ) is the degenerate increment, which obeys the normal distribution: Δd i (τ t )~N(μτ t ,σ 2 τ t ).

[0070] In one embodiment, the maintenance agent in step S2 is based on the updated state Select maintenance action a t , specifically including:

[0071] Maintain the agent according to the updated state The degradation value of determines the maintenance action a t :

[0072] a t =[m1,…,m n ]

[0073] Among them, m n ∈{0,1} indicates whether the nth shield machine cutter is maintained, m n =1 means to maintain the nth shield machine cutter, m n =0 means no maintenance is performed on the nth shield machine cutter, where n is the total number of shield machine cutters.

[0074] Specifically, the present invention defines the following state space and action space:

[0075] State space S: state s t Indicates the degradation state of the shield machine cutter. Assuming that the shield machine cutter has four shield machine cutters, the state can be expressed as: s t =[o1,o2,o3,o4,d1,d2,d3,d4].

[0076] Action space A: The action of the inspection agent (Ins-Agent) is to choose the inspection interval τ t , its value range is: τ t ∈{1,2,…,H-1,H}; where H is the maximum inspection interval. The action of the maintenance agent (Act-Agent) is to select maintenance action a t :a t =[m1,m2,m3,m4].

[0077] The present invention constructs a sequential simulation environment in which multiple agents make decisions in sequence. First, the inspection agent (Ins-Agent) determines the inspection interval, and then the maintenance agent (Act-Agent) determines the maintenance action based on the inspection results.

[0078] The sequential decision process can be formalized as a Markov decision process (MDP), where: the state transition function P(s t+1 |s t ,a t ) means in state s t Next, perform action a t Then transfer to state (s t+1 ) probability. Reward function R(s t ,a t ):Indicates that in state s t Next, perform action a t The goal of the inspection agent (Ins-Agent) and the maintenance agent (Act-Agent) is to maximize the cumulative reward through collaboration: where γ∈[0,1] is a discount factor that balances the importance of current rewards and future rewards.

[0079] Specifically, the adjustment of the inspection interval of the present invention is based on the degradation state of the equipment. The degradation process of the shield machine tool can be modeled using a Wiener process, and the variation of the degradation value X(t) with time t satisfies the following formula:

[0080] X(t)=X(0)+μt+σW(t);

[0081] X(0) is the initial degradation value (usually set to 0), μ is the drift coefficient, which represents the average rate of degradation, σ is the diffusion coefficient, which represents the random fluctuation of degradation, and W(t) is the standard Brownian motion, which represents random noise.

[0082] The increment of degradation value ΔX(t) in the time interval Δt obeys the normal distribution: ΔX(t)=X(t+Δt)-X(t)~N(μΔt,σ 2 Δt).

[0083] The dynamic inspection interval is determined by the inspection agent (Ins-Agent) according to the current degradation state s t Make:

[0084] If the degradation value of a shield machine tool is close to the fault threshold D, a shorter inspection interval (such as τ t =1 for timely inspection.

[0085] If the degradation values ​​of all TBM cutters are low, a longer inspection interval (such as τ t =H to reduce unnecessary checks.

[0086] In one embodiment, the maintenance action a in step S3 t and the updated status Calculate the reward r at time t t The design goal of the reward function is to minimize the shield machine tool maintenance cost. The shield machine tool maintenance cost includes inspection cost, downtime cost and maintenance cost, specifically including:

[0087] The inspection agent and the maintenance agent have the same reward function; the reward r t for:

[0088] r t =-cost;

[0089] The shield machine tool maintenance cost includes the inspection cost c ins , downtime cost c down and maintenance costs c rep :

[0090]

[0091] Among them, c fail is the additional maintenance cost after failure; I is an indicator variable indicating whether there is a shield machine cutter failure and no maintenance is performed; m i Indicates whether to perform maintenance on the i-th shield machine cutter; o i It represents the operating status of the i-th shield machine cutter, and n is the total number of shield machine cutters.

[0092] In one embodiment, in step S5, the overall Q function of the team consisting of the inspection agent and the maintenance agent is decomposed into the sum of the sub-Q functions of each agent through the value decomposition network, specifically including:

[0093] The value decomposition network architecture is applied to the inspection agent and the maintenance agent;

[0094] Check the agent's sub-Q function Q Ins According to the current state t and the inspection interval τ t calculate:

[0095]

[0096] Maintain the agent's sub-Q function Q Act According to the updated status and maintenance action a t calculate:

[0097]

[0098] The overall Q function Q of the team consisting of the agent and the maintenance agent total Breaks down to:

[0099]

[0100] In one embodiment, the error between the overall Q function and the target value consisting of the weighted sum of the current reward and the future expected reward in step S5 is used as the loss function, specifically including:

[0101] Target value y total for:

[0102]

[0103] γ is the discount factor, Q total is the overall Q function;

[0104] Loss Function for:

[0105]

[0106] Specifically, the present invention adopts the value decomposition network (VDN) architecture to decompose the team's overall Q function into multiple sub-Q functions, each sub-Q function corresponds to an intelligent agent, in this way solving the problem of uniform response of passive intelligent agents to rewards.

[0107] The VDN architecture breaks down the team's overall Q function in the following way:

[0108] The team's overall Q function Q total It is defined as the sum of the sub-Q functions of all agents:

[0109]

[0110] Where: s is the global state, a=(a1,a2,…,a N ) is the joint action of all agents, h i is the local observation history of the i-th agent, Q i (h i ,a i ) is the sub-Q function of the i-th agent. Each sub-Q function Q i (h i ,a i ) depends only on the local observations and actions of the i-th agent, and does not depend on the states or actions of other agents. This independence ensures that the contribution of each agent can be evaluated separately.

[0111] The training goal of VDN is to minimize the error between the overall Q function and the target value. total Defined as:

[0112]

[0113] Where: r is the immediate reward, γ is the discount factor, s ′ is the next state, a ′ is the joint action in the next state, and the loss function Defined as:

[0114]

[0115] In the shield machine tool maintenance task, the VDN architecture is applied to two agents: the inspection agent (Ins-Agent) and the act-agent. The VDN training process minimizes the error between the overall Q function and the target value and uses gradient descent to update the policy network parameters.

[0116] like Figure 2 As shown, the present invention tests the maintenance costs of different maintenance intervals. It can be seen that as the maintenance interval increases, the maintenance consumption shows a trend of first decreasing and then increasing.

[0117] It should be understood that although the steps in the flowcharts of the accompanying drawings are shown in sequence as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least some of the steps in the flowcharts of the accompanying drawings may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times. The order of execution of these steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a portion of the steps or stages in other steps.

[0118] In one embodiment, the present invention provides a computer device, which may be a server. The computer device includes a processor, a memory, and a network interface connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The database of the computer device is used to store data used in the above method. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, the above method is implemented.

[0119] In one embodiment, the present invention provides a computer-readable storage medium, such as a memory, comprising instructions, wherein the instructions are executable by a processor to perform the method. The storage medium may be a computer-readable storage medium, such as a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, or the like.

[0120] In one embodiment, the present invention provides a computer program product, comprising a computer program, which is capable of implementing the above method when executed by a processor.

[0121] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0122] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above and that the invention can be embodied in other specific forms without departing from the spirit or essential characteristics of the invention. Therefore, the embodiments should be considered in all respects as illustrative and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. It is intended that all variations within the meaning and range of equivalents of the claims be embraced herein, and any reference signs in the claims should not be construed as limiting the claims to which they relate.

[0123] In addition, it should be understood that although this specification is described in terms of implementation methods, not every implementation method contains only one independent technical solution. This narrative method of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment can also be appropriately combined to form other implementation methods that can be understood by those skilled in the art.

Claims

1. A shield machine tool maintenance decision method based on collaborative-sequential optimization, characterized in that: The following steps are involved: S1, initialization state, which includes the degradation value and operation state of all shield machine cutters; initialization of the strategy network parameters of the inspection agent and the maintenance agent; S2, define the sequential decision-making process in the simulation environment: the checking agent is based on the state s at time t t Choose the checking interval τ t , after τ t The status of the shield machine cutter after a cycle is updated as follows: The maintenance agent is based on the updated state Select the maintenance action a at time t t ; S3, according to maintenance action a t and the updated status Calculate the reward r at time t t ,The design goal of the reward function is to minimize the ,maintenance cost of the shield machine cutter and the ,maintenance cost of the shield machine cutter includes inspection ,cost, downtime cost and maintenance cost; S4, the experience tuple Store to the experience replay buffer; S5, randomly sampling a batch of experience tuples from the experience replay buffer, minimizing the loss function using gradient descent, and updating the policy network parameters of the inspection agent and the maintenance agent; wherein, the overall Q function of the team consisting of the inspection agent and the maintenance agent is decomposed into the sum of the sub-Q functions of each agent through the value decomposition network; the error between the overall Q function and the target value consisting of the weighted sum of the current reward and the expected future reward is used as the loss function; S6, repeating steps S2 to S5 until a preset termination condition is reached.

2. A shield machine tool maintenance decision method based on collaborative-sequential optimization according to claim 1, characterized in that: The status includes the degradation value and operating status of all shield machine cutters, specifically including: s t =[o1,…,o n ,d1,…,d n ]; o n Indicates the operating status of the nth shield machine cutter, d n It represents the degradation value of the nth shield machine cutter, where n is the total number of shield machine cutters.

3. The shield machine tool maintenance decision method based on collaborative-sequential optimization according to claim 2 is characterized in that: The checking agent is based on the state s at time t t Choose the checking interval τ t , after τ t The status of the shield machine cutter after a cycle is updated as follows: Specifically include: The inspection interval is adjusted based on the degradation state of the shield machine tool. The degradation process of the shield machine tool is modeled using the Wiener process. The lower the degradation value of the shield machine tool, the longer the inspection interval is selected; the inspection interval τ t The value range of is {1, 2, …, H-1, H}, where H is the maximum inspection interval; After τ t After running cycles, the update formula of degradation value is: n (t+τ t )=d n (t)+Δd n (τ t );Δd n (τ t ) is the degenerate increment and obeys the normal distribution.

4. The shield machine tool maintenance decision method based on collaborative-sequential optimization according to claim 1 is characterized in that: The maintenance agent is based on the updated state Select the maintenance action a at time t t , specifically including: Maintain the agent according to the updated state The degradation value of determines the maintenance action a t : a t =[m1,…,m n ]; Among them, m n ∈{0,1} indicates whether the nth shield machine cutter is maintained, m n =1 means to maintain the nth shield machine cutter, m n =0 means no maintenance is performed on the nth shield machine cutter, where n is the total number of shield machine cutters.

5. The shield machine tool maintenance decision method based on collaborative-sequential optimization according to claim 1 is characterized in that: According to the maintenance action a t and the updated state s t+τt Calculate the reward r at time t t The design goal of the reward function is to minimize the shield machine tool maintenance cost. The shield machine tool maintenance cost includes inspection cost, downtime cost and maintenance cost, specifically including: The inspection agent and the maintenance agent have the same reward function; the reward r t for: r t =-cost; The shield machine tool maintenance cost includes the inspection cost c ins , downtime cost c down and maintenance costs c rep : Among them, c fail is the additional maintenance cost after failure; I is an indicator variable indicating whether there is a shield machine cutter failure and no maintenance is performed; m i Indicates whether to perform maintenance on the i-th shield machine cutter; o i It represents the operating status of the i-th shield machine cutter, and n is the total number of shield machine cutters.

6. The shield machine tool maintenance decision method based on collaborative-sequential optimization according to claim 1 is characterized in that: The value decomposition network decomposes the overall Q function of the team consisting of the inspection agent and the maintenance agent into the sum of the sub-Q functions of each agent, specifically including: The value decomposition network architecture is applied to the inspection agent and the maintenance agent; Check the agent's sub-Q function Q Ins According to the current state t and the inspection interval τ t calculate: represents the calculation expectation, k represents the offset of the time step, γ k represents the discount factor, r t+k represents the immediate reward obtained at time t+k; Maintain the agent's sub-Q function Q Act According to the updated status and maintenance action a t calculate: The overall Q function Q of the team consisting of the agent and the maintenance agent total Breaks down to: Q total (s t ,τ t ,a t )=Q Ins (s t ,τ t )+Q Act (s t+τt ,a t )。 7. The shield machine tool maintenance decision method based on collaborative-sequential optimization according to claim 1 is characterized in that: The error between the overall Q function and the target value consisting of the weighted sum of the current reward and the future expected reward is used as the loss function, specifically including: Target value y total for: γ is the discount factor, Q total is the overall Q function; Overall Q function Q total It is defined as the sum of the sub-Q functions of all agents: s represents the global state, a=(a1,a2,…,a N ) represents the joint action of all agents, a N is the action of the Nth agent, N is the total number of agents, h i represents the local observation history of the i-th agent, Q i (h i ,a i ) represents the sub-Q function of the i-th agent; The loss function L is: m is the number of samples randomly sampled from the experience replay buffer.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images