Underwater AUV service cache and switch state switching method based on D3QN

Abstract

This invention discloses a method for underwater AUV service cache and on / off state switching based on D3QN. First, for an ocean network composed of surface buoy nodes and AUVs, a coupled control model of AUV service cache state and dormant / active state is established. Next, by binding the target on / off state with the consistency of service cache write actions, implementing dormant isolation constraints, and setting dormant cache freezing rules, a feasible action mask satisfying cache capacity constraints and anti-duplicate write constraints is constructed. A comprehensive cost function is then built using cache update time overhead, write energy consumption, on / off state switching cost, and invalid activation penalty. Finally, D3QN is used to solve this function within the legal action subspace, outputting the AUV service cache and on / off state switching control strategy. This invention can effectively realize AUV service cache updates and on / off state switching under dynamic loads, reducing underwater network energy consumption and service cache update latency.

Description

Technical Field

[0001] This invention belongs to the field of marine edge computing and intelligent control. Specifically, it relates to a method for underwater AUV service caching and on / off state switching based on a Dueling Double Deep Q-Network (D3QN). Background Technology

[0002] With the continuous advancement of applications such as smart ocean, marine environmental monitoring, underwater target detection, and marine resource development, the number of Autonomous Underwater Vehicles (AUVs) deployed in marine Internet of Things (IoT) systems is constantly increasing. When performing tasks such as perception, identification, analysis, and collaborative operations, AUVs typically need to invoke different types of computing services, including target detection, data analysis, and intelligent identification. Due to the limited energy, computing power, and storage resources of AUVs themselves, it is difficult to store all services completely for extended periods, and it is also difficult to maintain efficient operation under dynamic workloads. Utilizing buoy nodes (BNs) as service support nodes to provide AUVs with service delivery and edge collaboration capabilities has become an important technical approach to improve the task processing capabilities of marine edge networks.

[0003] However, in the aforementioned marine edge networks, AUV service cache control and on / off state control still face significant challenges. On the one hand, AUV cache capacity is limited and cannot store all services simultaneously. When business demands change, if the service cache is not updated in a timely manner, it will lead to the loss of required services and increase system processing latency. Frequent service cache writes will result in high service delivery latency and cache write energy consumption. On the other hand, AUVs do not always need to remain active in the dynamic marine environment. Reasonable sleep and wake-up control can help reduce system energy consumption. However, if the on / off state switching is too frequent, or if AUVs are blindly activated when there is a lack of effective service demand, it will introduce additional switching costs and ineffective activation overhead, affecting system stability and resource utilization efficiency.

[0004] Furthermore, in the joint control process of AUV service cache updates and on / off state switching, there is a problem that the coupling relationship of actions is difficult to characterize. If an AUV in a dormant state is still allowed to perform service cache writes, it will cause control logic conflicts; if cached services are repeatedly written, it will cause unnecessary bandwidth and energy waste; if cache capacity constraints, state switching constraints, and action consistency are not uniformly constrained, a large number of illegal joint actions can easily be generated, increasing the scale of the decision space and reducing control efficiency.

[0005] To address the aforementioned issues, existing technologies typically focus on task offloading, computing resource allocation, or single-dimensional service deployment optimization. For example, some marine edge computing methods assume that edge nodes already possess the necessary services, primarily optimizing task transmission, link scheduling, or resource allocation without explicitly considering the dynamic update problem under limited AUV service caching conditions. While some service caching methods can adjust cache content according to business needs, they are mostly designed for terrestrial edge networks or static scenarios, failing to consider the switchable on / off states of AUV nodes and the requirement for service write actions to maintain consistency with the target on / off state, making it difficult to balance service cache update costs with node start / stop costs. Furthermore, some methods employ heuristic strategies, fixed rules, or ordinary reinforcement learning algorithms for caching or node control, which struggle to handle the high-dimensional discrete joint action space composed of service cache write actions and on / off state actions, easily leading to problems such as numerous illegal actions, slow strategy convergence, and insufficient adaptability to dynamic load scenarios.

[0006] In summary, existing technologies in marine network scenarios generally suffer from the following shortcomings: First, they often treat AUV service cache updates and on / off state switching as separate processes, lacking a unified modeling method for coupled control of the two; second, they fail to simultaneously consider the time overhead of service cache updates, service write energy consumption, on / off state switching costs, and the additional costs of invalid activation, making it difficult to accurately reflect the overall system overhead; third, they lack unified action filtering for constraints such as hibernation isolation, anti-duplication write, and cache capacity, making it difficult to achieve efficient adaptive control under dynamic business loads. Therefore, there is an urgent need to propose a method that can couple and optimize AUV service cache updates and on / off state switching for dynamic marine business scenarios, in order to improve the overall operational efficiency of marine networks. Summary of the Invention

[0007] Purpose of the Invention: Existing technologies in marine network scenarios suffer from several drawbacks, including separate control of AUV service cache updates and on / off state switching, insufficient consideration of overall system overhead, and numerous illegal joint actions and inefficient joint decision-making under dynamic service loads. These issues hinder the effective reduction of service cache update latency and underwater network energy consumption. To overcome these shortcomings, this invention proposes a D3QN-based method for underwater AUV service cache and on / off state switching. This method constructs a coupled control model that binds AUV service cache write actions to the consistency of the target on / off state. It sets hibernation isolation constraints, hibernation cache freezing rules, and anti-duplication write constraints. Based on a mask of feasible actions, it performs decision-making within the legal action subspace, thereby achieving AUV service cache update and on / off state switching control under dynamic loads. This reduces underwater network energy consumption and service cache update latency, ultimately improving the overall operational efficiency of the marine network.

[0008] To achieve the above-mentioned objectives, the present invention employs the following technical solution:

[0009] A method for service caching and on / off state switching of underwater AUVs based on D3QN includes the following steps:

[0010] S1: Construct a marine network system model consisting of surface buoy nodes and AUVs, where the AUV set is denoted as... The set of surface buoy nodes is denoted as The service set is denoted as The set of decision-making cycles is denoted as AUV The maximum storage capacity is ,Serve The storage size is denoted as AUV In the cycle For services The service cache state variable is denoted as Its current switch state variable is denoted as ,in Indicates AUV In active state Indicates AUV In hibernation state; service In the cycle The normalized request frequency is denoted as Used for characterization services The popularity of the requests;

[0011] S2: Establish a coupled control model for AUV service cache update and switch state switching, define the AUV target switch state and service cache write action, bind the service cache write action with the target switch state in a consistent manner, and set hibernation isolation constraints and hibernation cache freeze rules.

[0012] S3: Construct the system state space based on the current service cache state of the AUV, the switch state of the previous cycle, service request statistics, and system load statistics; construct the system action space based on the AUV target switch state vector and service cache update matrix, and generate action masks based on cache capacity constraints, consistency binding constraints, sleep isolation constraints, and anti-duplicate write constraints to dynamically block illegal joint actions; construct the reward function of the Markov decision process using a weighted combination of service cache update time overhead, service write energy consumption, switch state switching cost, and invalid activation penalty as the comprehensive system cost;

[0013] S4: Use D3QN to solve the Markov decision process in the legal action subspace after filtering by the action mask, and output the service cache and on / off state switching control strategy for each AUV.

[0014] Furthermore, step S1 is specifically as follows:

[0015] Surface buoy nodes are deployed in the target sea area, and underwater AUVs are deployed; the AUV ensemble is defined as follows. The set of surface buoy nodes is Let the service set be... The system decision cycle set is as follows: ;in, For the number of AUVs, The number of surface buoy nodes. The total number of service types, Total number of decision-making cycles; Pre-set service set for surface buoy nodes. All services within the AUV are used as the source node for AUV service cache updates; AUVs are limited by local storage resources, cache only a portion of services, and their cached services are obtained from pushes by surface buoy nodes; define AUVs. In the cycle The service cache state variable is ,in , indicating AUV In the cycle Cached service , Indicates uncached service ;

[0016] Define AUV The maximum storage capacity is ,Serve The storage size is The AUV cache capacity constraint is then:

[0017]

[0018] Define AUV In the cycle The current switch state variable is ,in Indicates AUV In active state Indicates AUV In a dormant state; define service In the cycle The normalized request frequency is denoted as Used for characterization services The popularity of the request.

[0019] Furthermore, step S2 is specifically as follows:

[0020] Define AUV In the cycle The target switch state variable is And the service cache write decision variable is The service cache is written to the decision variable and the target switch state variable for consistency binding, and a hibernation isolation constraint is set:

[0021]

[0022] That is, only when AUV In the cycle Service execution is only permitted when the target state is active. Cache writes; when When =0, AUV When in a dormant state, its service cache state remains frozen, that is:

[0023]

[0024] when When =1, AUV When active, its service cache status is updated based on the service cache write decision as follows:

[0025] ;

[0026] Define the unit storage energy consumption coefficient for AUV cache write as: Then the energy consumption of service write for:

[0027] ;

[0028] Define the update time overhead for the AUV to receive services from the surface buoy node and complete the write operation as follows: Then the period Service cache update cost for:

[0029]

[0030] in, , These are the tradeoff coefficients for balancing update time overhead and service write energy consumption, and they satisfy... ;

[0031] Define the wake-up switching cost of an AUV from sleep mode to active mode as follows: The cost of switching from active to dormant state is Define indicator functions The value is 1 when the condition within the parentheses is true, and 0 otherwise. The period is... Switching cost for:

[0032]

[0033] in, AUV In the cycle Switch state variables; for AUVs that satisfy the following equation, they are defined as invalid active AUVs:

[0034] ,

[0035] Accordingly, an invalid activation penalty term is introduced:

[0036]

[0037] in, AUV Invalid activation penalty coefficient.

[0038] Furthermore, step S3 is specifically as follows:

[0039] In the cycle Define the system state space for:

[0040]

[0041] in, For period Normalized system load statistics;

[0042] Define the system in the period The joint action is ,in, The target switch state vector; Update the service cache matrix;

[0043] The system is constructed in state space based on the following conditions. The global set of actions :

[0044] The cache update satisfies the capacity constraint:

[0045]

[0046] Satisfying consistency binding constraints and hibernation isolation constraints:

[0047]

[0048] Satisfy the anti-duplicate write constraint:

[0049]

[0050] Joint actions that do not meet any of the above conditions are judged as illegal joint actions and are masked by an action mask, thereby forming the global action set. ;

[0051] For the period Service cache update cost Switching costs and invalid activation penalty Normalization was performed, and the results are denoted as follows: , and The reward function is defined as follows:

[0052]

[0053] in, These are the weighting coefficients for balancing service cache update cost, on / off state switching cost, and invalid activation penalty, respectively, and satisfying the following conditions: + + =1.

[0054] Furthermore, step S4 is specifically as follows:

[0055] Construct an online network and a target network for D3QN, both employing the same duel-based dual-branch structure; the Q-value function of the online network is denoted as... ,in, This represents the parameters of the online network; the online network adopts a duel structure, decomposing the Q-value into a state value function and an action advantage function, expressed as:

[0056] ,

[0057] in, For the state value function, For state value branch parameters; For the action advantage function,

[0058] For the action advantage branch parameters; and ; For the system state space The set of global possible actions; The number of actions in the globally available action set; To perform traversal actions within the set of globally actionable actions; to reduce overestimation bias in the Q-value estimation process, with the target Q-value... With online network Q-value function Mean square time difference error as loss function By minimizing the loss function For online network parameters Iterative updates are represented as:

[0059]

[0060] in, This represents the mathematical expectation of a small batch of samples obtained from the experience replay pool.

[0061] In the decision-making cycle The maximum Q-value screening method is used in the global set of possible actions. Internal selection of joint actions After the system executes the joint action, it obtains the reward function. and the next state ; and the state transition quadruple Store in the experience replay pool; when calculating the target Q value, only in the next state. The corresponding global action set Internally, select actions and evaluate their value.

[0062] After each online network parameter update, a soft update strategy is used to update the target network parameters. Perform a smooth update:

[0063]

[0064] in, The soft update coefficient is used to improve the stability and convergence of the training process. Through the above training process, the AUV service cache update and on / off state switching control strategy is obtained.

[0065] The advantages and technical effects of this invention are as follows:

[0066] First, this invention integrates AUV service cache updates and on / off state switching into a unified framework for coupled control, establishing a consistent binding relationship between service cache write actions and target on / off states, overcoming the problems of separate processing and insufficient coordination in existing technologies. Second, this invention sets dormancy isolation constraints, dormancy cache freezing rules, and anti-duplication write constraints, prohibiting AUVs in a dormant state from performing service cache writes during the current decision cycle and preventing cached services from being repeatedly pushed and written, thereby reducing invalid cache updates and unnecessary resource consumption. Third, this invention simultaneously considers service cache update time overhead, service write energy consumption, on / off state switching costs, and invalid activation penalties, enabling a more comprehensive characterization of the overall system overhead in the AUV service cache control and on / off control process, and improving the adaptability of the control model to actual dynamic marine business scenarios. Finally, this invention dynamically masks illegal joint actions based on actionable action masks and uses D3QN to solve within the legal action subspace, reducing the search complexity brought by the high-dimensional discrete joint action space and improving policy learning capabilities and control efficiency under dynamic load scenarios.

[0067] In summary, this invention can effectively realize the control of AUV service cache update and on / off state switching under dynamic load, reduce underwater network energy consumption and service cache update latency, and improve the operating performance of marine networks. Attached Figure Description

[0068] Figure 1 This is an overall flowchart of one embodiment of the present invention;

[0069] Figure 2 This is a network architecture diagram of one embodiment of the present invention;

[0070] Figure 3 This is a diagram of the D3QN architecture used in one embodiment of the present invention;

[0071] Figure 4 This is a graph showing the network energy consumption comparison results under different service load conditions using the method and comparison method provided by this invention according to one embodiment of the present invention;

[0072] Figure 5 This is a graph showing the comparison results of service cache update latency under different service request arrival rate conditions using the method provided by this invention and a comparative method according to one embodiment of the present invention. Detailed Implementation

[0073] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0074] This embodiment proposes a method for underwater AUV service caching and on / off state switching based on D3QN, and its overall flowchart is as follows: Figure 1 As shown, it includes the following steps:

[0075] S1: Construct a marine network system model consisting of surface buoy nodes and AUVs, where the AUV set is denoted as... The set of surface buoy nodes is denoted as The service set is denoted as The set of decision-making cycles is denoted as AUV The maximum storage capacity is ,Serve The storage size is denoted as AUV In the cycle For services The service cache state variable is denoted as Its current switch state variable is denoted as ,in Indicates AUV In active state Indicates AUV In hibernation state; service In the cycle The normalized request frequency is denoted as Used for characterization services To determine the popularity of requests, the specific steps are as follows:

[0076] like Figure 2 As shown, three surface buoy nodes and six underwater AUVs are deployed in the target sea area; the AUV array is set as follows. The set of surface buoy nodes is Let the service set be... The system decision cycle set is as follows: ;in, For the number of AUVs, The number of surface buoy nodes. The total number of service types, Total number of decision-making cycles; Pre-set service set for surface buoy nodes. All services within the AUV are used as the source node for AUV service cache updates; AUVs are limited by local storage resources, cache only a portion of services, and their cached services are obtained from pushes by surface buoy nodes; define AUVs. In the cycle The service cache state variable is ,in , indicating AUV In the cycle Cached service , Indicates uncached service ;

[0077] Define AUV The maximum storage capacity is ,Serve The storage size is The AUV cache capacity constraint is then:

[0078]

[0079] Define AUV In the cycle The current switch state variable is ,in Indicates AUV In active state Indicates AUV In a dormant state; define service In the cycle The normalized request frequency is denoted as Used for characterization services The popularity of the request.

[0080] S2: Establish a coupled control model for AUV service cache updates and on / off state switching, define the AUV target on / off state and service cache write actions, and bind the service cache write actions to the target on / off state for consistency. Set hibernation isolation constraints and hibernation cache freeze rules. The specific steps are as follows:

[0081] Define AUV In the cycle The target switch state variable is And the service cache write decision variable is The service cache is written to the decision variable and the target switch state variable for consistency binding, and a hibernation isolation constraint is set:

[0082]

[0083] That is, only when AUV In the cycle Service execution is only permitted when the target state is active. Cache writes; when When =0, AUV When in a dormant state, its service cache state remains frozen, that is:

[0084]

[0085] when When =1, AUV When active, its service cache status is updated based on the service cache write decision as follows:

[0086] ;

[0087] Define the unit storage energy consumption coefficient for AUV cache write as: Then the energy consumption of service write for:

[0088] ;

[0089] Define the update time overhead for the AUV to receive services from the surface buoy node and complete the write operation as follows: Then the period Service cache update cost for:

[0090]

[0091] in, , These are the tradeoff coefficients for balancing update time overhead and service write energy consumption, and they satisfy... In this embodiment, we take =0.5, =0.5;

[0092] Define the wake-up switching cost of an AUV from sleep mode to active mode as follows: The cost of switching from active to dormant state is Define indicator functions The value is 1 when the condition within the parentheses is true, and 0 otherwise. The period is... Switching cost for:

[0093]

[0094] in, AUV In the cycle Switch state variables; for AUVs that satisfy the following equation, they are defined as invalid active AUVs:

[0095] ,

[0096] Accordingly, an invalid activation penalty term is introduced:

[0097]

[0098] in, AUV The invalid activation penalty coefficient is taken in this embodiment. =0.2.

[0099] S3: Construct the system state space based on the current service cache state of the AUV, the on / off state of the previous cycle, service request statistics, and system load statistics; construct the system action space based on the AUV target on / off state vector and service cache update matrix, and generate action masks based on cache capacity constraints, consistency binding constraints, sleep isolation constraints, and anti-duplicate write constraints to dynamically shield illegal joint actions; construct the reward function of the Markov decision process using a weighted combination of service cache update time overhead, service write energy consumption, on / off state switching cost, and invalid activation penalty as the overall system cost. The specific steps are as follows:

[0100] In the cycle Define the system state space for:

[0101]

[0102] in, For period Normalized system load statistics;

[0103] Define the system in the period The joint action is ,in, The target switch state vector; Update the service cache matrix;

[0104] The system is constructed in state space based on the following conditions. The global set of actions :

[0105] The cache update satisfies the capacity constraint:

[0106]

[0107] Satisfying consistency binding constraints and hibernation isolation constraints:

[0108]

[0109] Satisfy the anti-duplicate write constraint:

[0110]

[0111] Joint actions that do not meet any of the above conditions are judged as illegal joint actions and are masked by an action mask, thereby forming the global action set. ;

[0112] For the period Service cache update cost Switching costs and invalid activation penalty Normalization was performed, and the results are denoted as follows: , and The reward function is defined as follows:

[0113]

[0114] in, These are the weighting coefficients for balancing service cache update cost, on / off state switching cost, and invalid activation penalty, respectively, and satisfying the following conditions: + + =1, in this embodiment, we take... =0.4, =0.4, =0.2.

[0115] S4: Solve the Markov decision process using D3QN within the legal action subspace after filtering by the action mask, and output the service cache and on / off state switching control strategy for each AUV. The specific steps are as follows:

[0116] like Figure 3 As shown, an online network and a target network of D3QN are constructed, both employing the same duel-based dual-branch structure; the Q-value function of the online network is denoted as... ,in, This represents the parameters of the online network; the online network adopts a duel structure, decomposing the Q-value into a state value function and an action advantage function, expressed as:

[0117] ,

[0118] in, For the state value function, For state value branch parameters; For the action advantage function,

[0119] For the action advantage branch parameters; and ; For the system state space The set of global possible actions; The number of actions in the globally available action set; To perform traversal actions within the set of globally actionable actions; to reduce overestimation bias in the Q-value estimation process, with the target Q-value... With online network Q-value function Mean square time difference error as loss function By minimizing the loss function For online network parameters Iterative updates are represented as:

[0120]

[0121] in, This represents the mathematical expectation of a small batch of samples obtained from the experience replay pool.

[0122] In the decision-making cycle The maximum Q-value screening method is used in the global set of possible actions. Internal selection of joint actions After the system executes the joint action, it obtains the reward function. and the next state ; and the state transition quadruple Store in the experience replay pool; when calculating the target Q value, only in the next state. The corresponding global action set Internally, select actions and evaluate their value.

[0123] After each online network parameter update, a soft update strategy is used to update the target network parameters. Perform a smooth update:

[0124]

[0125] in, For soft update coefficients, in this embodiment, we take... =0.005, used to improve the stability and convergence of the training process. Through the above training process, the AUV service cache update and on / off state switching control strategy is obtained.

[0126] The underwater AUV service caching and on / off state switching method based on D3QN provided in this invention is compared with fixed rule methods and traditional Deep Q-Learning (DQN) methods. Simulation results are as follows: Figure 4 and Figure 5 As shown. Among them, Figure 4 The results show the network energy consumption comparison under different methods. Figure 5 This section compares service cache update latency under different service request arrival rates. Network energy consumption is measured using the average total energy consumption metric; a smaller value indicates lower energy expenditure for AUVs during activation, hibernation switching, and service cache update processes. Service cache update latency is measured using the average service cache update time metric; a smaller value indicates higher efficiency for AUVs in retrieving and writing services from surface buoy nodes.

[0127] This embodiment was simulated using Python 3.9 and the PyTorch framework. The experimental platform was Ubuntu 20.04 operating system, with an Intel(R) Xeon(R) Silver 4210R CPU and an NVIDIA GeForce RTX 3090 graphics processor. All simulations were performed under the same system parameters and training budget to ensure the comparability of the results. The main simulation parameters are listed in Table 1.

[0128] Table 1 Simulation Parameters

[0129]

[0130] The network energy consumption comparison results of the method of the present invention and the comparative method under different service load conditions are as follows: Figure 4 As shown. By Figure 4 As can be seen, the network energy consumption of all methods increases with the increase in business load. This is because the frequency of AUV service cache updates increases with the increase in business requests, and the number of AUVs in the active state also increases, leading to an increase in overall system energy consumption. Among them, the fixed rule method has the highest network energy consumption, indicating that it is less adaptable to service cache updates and on / off state switching under dynamic business load conditions, and is prone to frequent updates, invalid activations, and unnecessary node start-ups and shutdowns. Although the network energy consumption of the traditional DQN method is lower than that of the fixed rule method, it is still higher than that of the method of this invention. In contrast, the method of this invention maintains lower network energy consumption under various business load conditions, indicating that it can more reasonably coordinate AUV service cache updates and on / off state switching through the consistency binding of service cache write actions and target on / off states, sleep isolation constraints, and legal action filtering mechanisms, reducing the additional overhead caused by unnecessary service writes and invalid activations. In summary, this invention can effectively reduce network energy consumption and improve system performance under dynamic load conditions.

[0131] The comparison results of service cache update latency between the method of this invention and the comparative method under different service request arrival rates are as follows: Figure 5 As shown. By Figure 5As can be seen, the service cache update latency of each method gradually increases with the increase in service request arrival rate. This is because as the business load increases, the number of times the surface buoy node sends and writes services to the AUV increases, leading to increased queuing and transmission overhead during the update process. Among them, the fixed rule method has the largest service cache update latency, indicating that it is less adaptable to the timing and objects of cache updates under dynamic load conditions, and is prone to duplicate writes and invalid updates. Although the service cache update latency of the traditional DQN method is lower than that of the fixed rule method, it is still higher than that of the method of this invention. In contrast, the method of this invention maintains a lower service cache update latency under all service request arrival rate conditions, indicating that it can make more reasonable service cache update decisions based on sleep isolation constraints, anti-duplicate write constraints, and action masks, reducing unnecessary cache writes and duplicate updates, and improving AUV service update efficiency. In summary, this invention can effectively reduce service cache update latency and improve system performance under dynamic load conditions.

[0132] In summary, this invention can effectively realize AUV service cache update and on / off state switching control under dynamic load, reduce underwater network energy consumption and service cache update latency, and improve the operational performance of marine networks.

[0133] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions claimed by the present invention.

Claims

1. A method for service caching and on / off state switching of underwater AUVs based on D3QN, characterized in that, This method is applied to an ocean network consisting of surface buoy nodes and AUVs, wherein the surface buoy nodes are pre-configured with all services and serve as service update source nodes for the AUVs, and the AUVs have limited cache capacity and support dynamic switching between dormant and active states; the method includes the following steps: S1: Construct a marine network system model consisting of surface buoy nodes and AUVs, where the AUV set is denoted as... The set of surface buoy nodes is denoted as The service set is denoted as The set of decision-making cycles is denoted as AUV The maximum storage capacity is ,Serve The storage size is denoted as AUV In the cycle For services The service cache state variable is denoted as Its current switch state variable is denoted as ,in Indicates AUV In active state Indicates AUV In hibernation mode; service In the cycle The normalized request frequency is denoted as Used for characterization services The popularity of the requests; S2: Establish a coupled control model for AUV service cache update and switch state switching, define the AUV target switch state and service cache write action, bind the service cache write action with the target switch state in a consistent manner, and set hibernation isolation constraints and hibernation cache freeze rules. S3: Construct the system state space based on the current service cache state of the AUV, the switch state of the previous cycle, service request statistics, and system load statistics; construct the system action space based on the AUV target switch state vector and service cache update matrix, and generate action masks based on cache capacity constraints, consistency binding constraints, sleep isolation constraints, and anti-duplicate write constraints to dynamically block illegal joint actions; construct the reward function of the Markov decision process using a weighted combination of service cache update time overhead, service write energy consumption, switch state switching cost, and invalid activation penalty as the comprehensive system cost; S4: Use D3QN to solve the Markov decision process in the legal action subspace after filtering by the action mask, and output the service cache and on / off state switching control strategy for each AUV.

2. The method for service buffering and on / off state switching of an underwater AUV based on D3QN as described in claim 1, characterized in that, Step S2 is as follows: Define AUV In the cycle The target switch state variable is And the service cache write decision variable is The service cache is written to the decision variable and the target switch state variable for consistency binding, and a hibernation isolation constraint is set: That is, only when AUV In the cycle Service execution is only permitted when the target state is active. Cache writes; when When =0, AUV When in a dormant state, its service cache state remains frozen, that is: when When =1, AUV When active, its service cache status is updated based on the service cache write decision as follows: ； Define the unit storage energy consumption coefficient for AUV cache write as: Then the energy consumption of service write for: ; Define the update time overhead for the AUV to receive services from the surface buoy node and complete the write operation as follows: Then the period Service cache update cost for: in, , These are the tradeoff coefficients for balancing update time overhead and service write energy consumption, and they satisfy... ; Define the wake-up switching cost of an AUV from sleep mode to active mode as follows: The cost of switching from active to dormant state is Define indicator functions The value is 1 when the condition within the parentheses is true, and 0 otherwise. The period is... Switching cost for: in, AUV In the cycle Switch state variables; for AUVs that satisfy the following equation, they are defined as invalid active AUVs: , Accordingly, an invalid activation penalty term is introduced: in, AUV Invalid activation penalty coefficient.

3. The method for service caching and on / off state switching of an underwater AUV based on D3QN as described in claim 1, characterized in that, Step S3 is as follows: In the cycle Define the system state space for: in, For period Normalized system load statistics; Define the system in the period The joint action is ,in, The target switch state vector; Update the service cache matrix; The system is constructed in state space based on the following conditions. The global set of actions : The cache update satisfies the capacity constraint: Satisfying consistency binding constraints and hibernation isolation constraints: Satisfy the anti-duplicate write constraint: Joint actions that do not meet any of the above conditions are judged as illegal joint actions and are masked by an action mask, thereby forming the global action set. ; For the period Service cache update cost Switching costs and invalid activation penalty Normalization was performed, and the results are denoted as follows: , and The reward function is defined as follows: in, These are the weighting coefficients for balancing service cache update cost, on / off state switching cost, and invalid activation penalty, respectively, and satisfying the following conditions: + + =1.

4. The method for service caching and on / off state switching of an underwater AUV based on D3QN as described in claim 1, characterized in that, Step S4 is as follows: Construct an online network and a target network for D3QN, both employing the same duel-based dual-branch structure; during the decision cycle... , system state space Input the online network and only set the global actions. Q-value evaluation is performed on the legal joint actions in the process; The maximum Q-value screening method is used in the global action set. Internal selection of joint actions After the system executes the joint action, it obtains the reward function. and the next state ; Transition quadruple Store in the experience replay pool; When calculating the target Q value, only in the next state... The corresponding global action set The system performs action selection and action value evaluation. After each online network parameter update, a soft update strategy is used to smoothly update the target network parameters. Through the above training process, the AUV service cache update and on / off state switching control strategy is obtained.

Images