A vehicle-network interaction regulation method and system based on multi-task offline meta-reinforcement learning

Abstract

This invention discloses a vehicle-to-everything (V2X) interaction control method and system based on multi-task offline meta-reinforcement learning. The method includes: constructing a control policy training mechanism based on an offline meta-reinforcement learning framework; introducing a meta-network for task recognition; and achieving joint modeling of historical trajectory data and expert scheduling text knowledge through a semantic-physical fusion mechanism enhanced by a large language model. The meta-network generates task-specific embeddings to guide the downstream policy network to quickly adapt to dynamic control tasks, thereby formulating context-aware control policies. This invention effectively overcomes the bottlenecks of traditional reinforcement learning methods in terms of task transferability and generalization ability, significantly improving the policy network's adaptability to diverse EV distributions and control targets.

Description

Technical Field

[0001] This invention relates to a vehicle-to-grid (V2G) interaction control method and system based on multi-task offline meta-reinforcement learning, belonging to the interdisciplinary field of smart grid scheduling, electric vehicle charging and discharging optimization, and artificial intelligence technology. Background Technology

[0002] With the rapid development of new energy vehicles, electric vehicles (EVs) are increasingly becoming an important adjustable load resource in power distribution networks. In the context of large-scale grid integration, EVs are no longer merely passive loads of the power system, but can achieve functions such as orderly charging and vehicle-to-grid interaction through intelligent control, providing new support for the flexibility, stability, and economy of grid operation. However, the charging and discharging behavior of EVs exhibits significant individual heterogeneity, dynamic spatiotemporal distribution, and is constrained by both traffic patterns and user preferences, leading to numerous challenges when they participate in grid regulation, including high state dimensions, complex decision-making processes, and diverse conflicting objectives.

[0003] Traditional electric vehicle regulation methods largely rely on centralized optimization or rule-based scheduling strategies. While these methods are feasible for single-task scenarios, they often struggle to adapt and generalize flexibly when facing diverse regulation objectives such as peak shaving and valley filling, maximizing renewable energy utilization, and electricity price arbitrage, as well as high-frequency dynamic environments. Meanwhile, distributed reinforcement learning (RL) technology has been widely applied in the autonomous regulation of complex systems in recent years, demonstrating strong online decision-making capabilities. However, classic RL methods typically require large amounts of interactive data, making them difficult to directly apply to offline scenarios where data acquisition is limited and grid security requirements are high.

[0004] Meta-Reinforcement Learning (Meta-RL), as a policy optimization framework capable of "fast transfer learning," has shown promising promise in multi-task decision-making problems in recent years. Its core idea is to train the model on a task distribution, enabling it to quickly adapt to new tasks. However, most existing Meta-RL methods are limited to simple simulation environments and primarily rely on pure data-driven approaches. When modeling complex power system tasks, they suffer from insufficient generalization, poor physical interpretability, and weak task recognition capabilities. Furthermore, most methods do not consider how to integrate high-level information such as expert experience or semantic instructions, thus limiting their application potential in real-world scheduling scenarios. Summary of the Invention

[0005] The technical problem to be solved by this invention is: how to construct a group control model for electric vehicles in the process of interaction and control between electric vehicles and power distribution networks, which can integrate multimodal information to simulate the expert behavior of dispatchers, realize the control between electric vehicles and power distribution networks, and at the same time have the ability to generalize tasks and can be trained efficiently under offline data conditions.

[0006] To address the aforementioned technical problems, this invention provides a vehicle-to-network interaction control method based on multi-task offline meta-reinforcement learning, comprising:

[0007] Step S1: Collect distribution network operation and scheduling information, traffic network topology, spatiotemporal travel demand data of electric vehicles, and investment cost parameters of different types of charging infrastructure to construct a distribution network operation model that considers the access characteristics of electric vehicles;

[0008] Step S2: Construct a charging and discharging behavior model for electric vehicle groups to represent the charging demand, power constraints, and energy evolution patterns of each electric vehicle at different times and locations.

[0009] Step S3: Abstract the different operation and control objectives of the distribution network and their corresponding spatiotemporal configuration requirements into a task tuple, and define a corresponding economic cost item for each type of task; the task tuple is the objective function and the configuration scenario;

[0010] Step S4: The vehicle-grid interaction control problem established in steps S1-S3 is formalized into a Markov decision process to form an intelligent agent; the vehicle-grid interaction control problem includes a power distribution network operation model considering the access characteristics of electric vehicles, a charging and discharging behavior model of electric vehicle groups, and task tuples.

[0011] Step S5: Construct an offline meta-reinforcement learning model for the group control of electric vehicles in the power distribution network. The meta-reinforcement learning model includes an online interaction stage and an offline adaptive stage. In the online interaction stage, the agent performs interactive training in a single specific task environment with the goal of acquiring expert experience data that represents the characteristics of the task. In the offline adaptive stage, a unified meta-network is constructed to achieve collaborative optimization of task recognition and policy transfer.

[0012] The aforementioned vehicle-to-grid interaction control method based on multi-task offline meta-reinforcement learning also includes:

[0013] Step S6: Construct a task distribution set covering multiple operational objectives and control scenarios to uniformly represent diverse Markov decision processes;

[0014] Step S7: Based on the task distribution set in Step S6, enhance the interactive expression capability of semantic-state information using the cross-modal attention fusion module.

[0015] The aforementioned vehicle-to-grid interaction control method based on multi-task offline meta-reinforcement learning also includes:

[0016] Step S8: During the training of the meta-reinforcement learning model, a contrastive learning objective function is used to guide the embedding modules in the meta-network to construct a task-distinguished structured distribution in the latent embedding space.

[0017] The aforementioned method for vehicle-to-grid interaction control based on multi-task offline meta-reinforcement learning,

[0018] In step S1, in the distribution network operation model considering the characteristics of electric vehicle access, the distribution network operation model considering the characteristics of electric vehicle access is expressed as follows:

[0019] A linearized branch power flow model is used to model the radial structure of the distribution network to represent the active power balance, reactive power balance, voltage relationships, and physical constraints in the distribution network system, including the following constraints:

[0020] (1)

[0021] (2)

[0022] (3)

[0023] (4)

[0024] (5)

[0025] (6)

[0026] (7)

[0027] (8)

[0028] The set of electric vehicle aggregators is denoted as . j represents the node number, and t represents the time index; This represents the power distribution line from node i to node j. For the set of routes; and The lines are respectively Active and reactive power flows at time t , This represents the voltage magnitude at nodes i and j. and These are the line impedance parameters; and The active and reactive power injected into the substation. This is a parameter indicating the balance node; it takes the value of 1 if node j is a balance node, and 0 otherwise. and These represent the basic active and reactive loads of node j, respectively. and The portion that can be reduced from the base load; and This represents the aggregate active and reactive loads of electric vehicles. and This reduces the combined active and reactive load of electric vehicles. and These represent the active and reactive power outputs of the photovoltaic system, respectively. This represents the maximum available active power of photovoltaic power under ideal weather conditions. This refers to the reduction in photovoltaic output due to dispatch control. This represents the minimum power factor of a photovoltaic inverter; and These are the lower and upper limits of the node voltage, respectively. This represents the set of electric vehicle clusters connected to node j. Let k be the set of electric vehicle nodes in cluster k. and These represent the active power for charging and discharging electric vehicle i, respectively. Let be the reactive power of the electric vehicle at node i; This represents the active power transmitted from node s to node j on the branch during time period t. This represents the active power from node j to node s. and These represent the reactive power transmission amounts in the corresponding directions, with the power direction being positive from the branch start point to the end point.

[0029] The aforementioned vehicle-network interaction control method based on multi-task offline meta-reinforcement learning uses a two-stage power flow calculation method to solve the distribution network operation model considering the access characteristics of electric vehicles. First, the active and reactive power demand of the distribution network system nodes is initially estimated using the linearized power flow model shown in equations (1) to (6).

[0030] The preliminary estimation results are passed as input to the OpenDSS simulation platform to perform high-fidelity nonlinear power flow calculations and obtain more accurate voltage and power distribution results.

[0031] The aforementioned method for vehicle-to-grid interaction control based on multi-task offline meta-reinforcement learning,

[0032] In step S2, in the electric vehicle group charging and discharging behavior model, for each node Let I be the set of electric vehicle nodes, and at any time t, the charging and discharging behavior is expressed as equations (9)-(13):

[0033] (9)

[0034] (10)

[0035] (11)

[0036] (12)

[0037] (13)

[0038] in, and Let represent the charging and discharging active power of electric vehicle i at time t, respectively. This represents the reactive power of the electric vehicle at node t; and These are the maximum permissible active power and apparent power capacity, respectively; variables. This is a binary decision variable used to indicate whether an electric vehicle is charging. Indicates charging. Indicates that it is not charging; This is a vehicle connection status indicator; a value of 1 indicates that the vehicle is connected to the power grid at time t, and a value of 0 indicates that it is not connected. This represents the state of charge (SoC) of an electric vehicle, with values ​​subject to upper and lower limits. and limit; For charge and discharge efficiency; It represents the duration of a time segment.

[0039] The aforementioned method for vehicle-to-grid interaction control based on multi-task offline meta-reinforcement learning,

[0040] In step S3, each regulation task is defined as a binary tuple. ,in Indicates the type of regulatory target. This represents the spatiotemporal distribution pattern of electric vehicles; To regulate the number of target types, The number of typical distribution patterns of electric vehicles;

[0041] The definitions of various economic cost items are as follows:

[0042] (14)

[0043] This indicates the cost incurred by reducing active and reactive power for normal loads;

[0044] (15)

[0045] This indicates the cost reduction associated with improving the charging and discharging capabilities of electric vehicles.

[0046] (16)

[0047] This indicates the loss of abandoned electricity when photovoltaic power cannot be connected to the grid;

[0048] (17)

[0049] This indicates the cost of electricity purchased by the substation;

[0050] (18)

[0051] in, For time period Electricity price, , and These are the penalty coefficients for base load, EV load, and photovoltaic output reduction, respectively. I represents the set of ordinary load nodes, and I represents the set of electric vehicle nodes. A set of photovoltaic access nodes. The active power injected into the substation. Let be the voltage magnitude of node i at time t;

[0052] and The portion that can be reduced from the base load; and This reduces the combined active and reactive load of electric vehicles.

[0053] Define the following energy shortage cost:

[0054] (19)

[0055] (20)

[0056] in, This indicates that electric vehicle i will depart at the scheduled time. The minimum state of charge that is expected to be achieved at that time. Penalty weights for insufficient battery power in electric vehicles;

[0057] Each task The objective function at time t The model is as follows:

[0058] (twenty one)

[0059] in, to To target the first The weight coefficients of each task for the regulatory objectives.

[0060] The aforementioned method for vehicle-to-grid interaction control based on multi-task offline meta-reinforcement learning,

[0061] In step S4, the Markov decision process is defined as a quintuple. ,in, Representing the state space, Represents the action space. Represents the state transition probability. For the reward function, Discount factor;

[0062] (1) State space :

[0063] At any given time t, the state of the distribution network system includes information from both the grid level and the electric vehicle level:

[0064] 1) Power grid status It is used to characterize the operating status of the distribution network, including node voltage amplitude, line power flow, photovoltaic output and load shedding information:

[0065] (twenty two)

[0066] 2) Electric vehicle status This is used to describe the energy state and scheduling information of individual electric vehicles.

[0067] (twenty three)

[0068] in, This represents the initial charge of the i-th electric vehicle when it is connected to the power distribution network. This indicates the start time of the vehicle's connection to the charging station. Indicates the time the vehicle is scheduled to leave;

[0069] 3) Global state vector:

[0070] (twenty four)

[0071] (2) Action space :

[0072] At every moment Appropriate charging and discharging power needs to be allocated to electric vehicles:

[0073] (25)

[0074] (3) Reward function :

[0075] At any time t, the immediate reward of the distribution network system is determined by the task. The corresponding overall operating cost is determined as follows:

[0076] (26)

[0077] in, This represents the system cost under a multi-objective weighted configuration, including load reduction, electric vehicle reduction, solar curtailment, electricity purchase cost, and voltage deviation.

[0078] (4) State transition function :

[0079] The transition of the distribution network system state from time t to t+1 is determined by control actions. With random disturbances The following decision was made jointly:

[0080] (27)

[0081] in, The charging and discharging power calculation module calculates the allowable power interaction amount based on the current state of charge, electric vehicle connection status, and control actions. Then, the obtained aggregated charging and discharging load is passed as input to the power flow solution module. Update system status, including node voltages. Branch current , satisfying the physical constraints defined by formulas (1) to (6); This represents external random disturbance variables, used to simulate system disturbance factors such as the uncertainty of electric vehicle arrival and departure, load fluctuations, and changes in photovoltaic power.

[0082] It represents the state of the system at time step t, including the operating status of the distribution network, the access information of electric vehicles and their state of charge; Indicates the state The regulatory actions taken below; This represents the state transition function, used to describe the system's transition from state to state under given actions and disturbances. Evolving to the next state The process.

[0083] In the aforementioned vehicle-to-network interaction control method based on multi-task offline meta-reinforcement learning, in step S5,

[0084] During the online interaction phase, perform the following steps:

[0085] S5.1: Construct a vehicle-to-grid coupling environment representing a specific task, including node load, photovoltaic output, electric vehicle access behavior, and power grid flow;

[0086] S5.2: Based on specific task objectives, reinforcement learning algorithms are used to train the agent to obtain expert policies;

[0087] S5.3: Collect trajectory data of the expert agent during the training process, including state transition sequences, reward information, and behavioral statistical features;

[0088] S5.4: Extract the scheduling behavior pattern of each task, including typical charging and discharging strategy sequences, as reference signals for subsequent task identification and meta-network;

[0089] During the offline adaptive phase, perform the following steps:

[0090] S5.5: Randomly sample Markov decision process samples and semantic task descriptions of the samples from expert trajectories of multiple tasks;

[0091] S5.6: Use a pre-trained language model to extract vector features of semantic descriptions, and input them together with the trajectory behavior context into the meta-network to learn task embedding representations;

[0092] S5.7: The low-dimensional task embedding is concatenated with the physical observation state in the current task and used as input to the policy network for decision generation;

[0093] S5.8: Optimize the meta-policy network so that when faced with new tasks, the meta-policy network can quickly transfer based on the embedding representation and output an effective control policy.

[0094] In the aforementioned vehicle-network interaction control method based on multi-task offline meta-reinforcement learning, in step S8, the contrastive learning objective function aggregates positive sample pairs of trajectory-semantic instructions from the same task and separates negative sample pairs between different tasks, thereby representing the feature differences between negative sample pairs of different tasks, enabling the meta-network to learn to capture the potential differences between tasks.

[0095] The positive sample pairs consist of trajectory embeddings and corresponding semantic instructions under the same task. They are used to train the meta-reinforcement learning model to bring the semantic instructions and the matched trajectory representations closer to each other in the latent embedding space, thereby achieving the feature mapping capability of semantic-behavior alignment.

[0096] The aforementioned vehicle-to-grid interaction control method based on multi-task offline meta-reinforcement learning, wherein the contrastive learning objective function is defined as follows:

[0097] (34)

[0098] in, The loss function represents distance metric learning; There are two input samples; This indicates the corresponding category label or task identifier; and This is an indicator function that takes the value 1 when samples come from the same task, and 0 otherwise. This is the sample embedding vector obtained through the context encoder; The square of the Euclidean distance; This is the nth power of the Euclidean distance, used to control the strength of the similarity penalty for negative samples, where... It is a hyperparameter that adjusts the power; It is a constant used to avoid division by zero errors; It is a hyperparameter that balances the loss between positive and negative samples;

[0099] For each task Given its pre-collected dataset and accompanying semantic instructions ,in: The initial state of the j-th sample in the i-th task; : The corresponding action; The next state after the action is performed; The instant reward received; This indicates the number of samples for each task trajectory data; Indicates task The corresponding semantic task description comes from the semantic space. ;

[0100] Context encoder Based on the sampled trajectory context With semantic information Output task representation Capture multimodal task features; any task embedding After being concatenated with the current state s, it is input into the policy network. With Q network They generate control actions and action value assessments respectively.

[0101] A computer system includes a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the method described above.

[0102] The beneficial effects achieved by this invention are as follows: The vehicle-to-grid interaction control method based on multi-task offline meta-reinforcement learning of this invention enables the model to adapt to various control objectives and operating environments by constructing a task distribution and introducing a meta-network, without the need for retraining for each new task; it introduces semantic task instructions and fuses them with physical states through a cross-modal attention mechanism to simulate the decision-making logic and attention mechanism of human dispatchers when handling complex tasks; and by constructing a contrastive learning target and task embedding representation based on offline data, it guides the meta-network to form a structured feature space, effectively improving policy transfer and generalization effects. Attached Figure Description

[0103] Figure 1 This is a flowchart of the offline meta-reinforcement learning framework for multi-task vehicle-to-network interaction regulation in Embodiment 1 of the present invention;

[0104] Figure 2 This is a convergence graph of the training reward curve in Embodiment 1 of the present invention. Detailed Implementation

[0105] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings.

[0106] Example 1

[0107] like Figure 1 As shown, this embodiment provides a vehicle-to-network interaction control method based on multi-task offline meta-reinforcement learning, including:

[0108] Step S1: Collect distribution network operation and scheduling information, traffic network topology, spatiotemporal travel demand data of electric vehicles, and investment cost parameters of different types of charging infrastructure. Construct a distribution network operation model that considers the access characteristics of electric vehicles to provide basic data support for the power-transportation-storage system for subsequent modeling.

[0109] In step S1, in the distribution network operation model considering the characteristics of electric vehicle access, to accurately describe the power flow distribution and operational constraints of the distribution network under electric vehicle access, the following mathematical model is constructed, including:

[0110] A linearized branch power flow model is used to model the radial structure of the distribution network to represent the active power balance, reactive power balance, voltage relationships, and physical constraints in the distribution network system, including the following constraints:

[0111] (1)

[0112] (2)

[0113] (3)

[0114] (4)

[0115] (5)

[0116] (6)

[0117] (7)

[0118] (8)

[0119] The set of electric vehicle aggregators is denoted as . j represents the node number, and t represents the time index; This represents the power distribution line from node i to node j. For the set of routes; and The lines are respectively Active and reactive power flows at time t , This represents the voltage magnitude at nodes i and j. and These are the line impedance parameters; and The active and reactive power injected into the substation. This is a parameter indicating the balance node; it takes the value of 1 if node j is a balance node, and 0 otherwise. and These represent the basic active and reactive loads of node j, respectively. and The portion that can be reduced from the base load; and This represents the aggregate active and reactive loads of electric vehicles. and This reduces the combined active and reactive load of electric vehicles. and These represent the active and reactive power outputs of the photovoltaic system, respectively. This represents the maximum available active power of photovoltaic power under ideal weather conditions. This refers to the reduction in photovoltaic output due to dispatch control. This represents the minimum power factor of a photovoltaic inverter; and These are the lower and upper limits of the node voltage, respectively. This represents the set of electric vehicle clusters connected to node j. Let k be the set of electric vehicle nodes in cluster k. and These represent the active power for charging and discharging electric vehicle i, respectively. Let be the reactive power of the electric vehicle at node i. This represents the active power transmitted from node s to node j on the branch during time period t. This represents the active power from node j to node s. and These represent the reactive power transmission amounts in the corresponding directions, with the power direction being positive from the branch start point to the end point.

[0120] Due to the nonlinear and nonconvex nature of power flow equations, directly solving the optimal power flow problem in a distribution network including electric vehicles often involves high computational complexity, making it difficult to provide efficient and stable feedback for reinforcement learning agents. To overcome these difficulties, this invention employs a two-stage power flow calculation method to balance computational efficiency and physical accuracy.

[0121] First, the active and reactive power demands of the nodes in the distribution network system are initially estimated using the linearized power flow model shown in equations (1) to (6).

[0122] Next, the preliminary estimation results are passed as input to the OpenDSS simulation platform to perform high-fidelity nonlinear power flow calculations and obtain more accurate voltage and power distribution results.

[0123] Step S2: Construct a charging and discharging behavior model for electric vehicle groups to represent the charging demand, power constraints, and energy evolution patterns of each electric vehicle at different times and locations, thereby quantifying the adjustability at any given time and ensuring the physical feasibility of subsequent scheduling models.

[0124] In step S2, for each node At any time t, the charging and discharging behavior is expressed as equations (9)-(13):

[0125] (9)

[0126] (10)

[0127] (11)

[0128] (12)

[0129] (13)

[0130] in, and Let represent the charging and discharging active power of electric vehicle i at time t, respectively. This represents the reactive power of the electric vehicle at node i. and These are the maximum permissible active power and apparent power capacity, respectively. Variables This is a binary decision variable used to indicate whether an electric vehicle is charging. Indicates charging. Indicates that it is not charging; This is a vehicle connectivity status indicator; a value of 1 indicates that the vehicle is connected. Always connected to the power grid; 0 indicates not connected. This represents the state of charge (SoC) of an electric vehicle, with values ​​subject to upper and lower limits. and limit; For charge and discharge efficiency; It represents the duration of a time segment.

[0131] The above-mentioned electric vehicle group charging and discharging behavior model ensures that the power of electric vehicles participating in regulation does not exceed the inverter's capacity, the state of charge (SoC) remains within a reasonable range, and it is physically feasible.

[0132] Step S3: Abstract the different operation and control objectives of the distribution network and their corresponding spatiotemporal configuration requirements into task tuples, and define the corresponding economic cost item for each type of task, laying the foundation for multi-task generalization modeling; the operation and control objectives include peak shaving and valley filling, voltage support, load balancing, etc.; the task tuple is the objective function and the configuration scenario.

[0133] In step S3, to meet the diverse control objectives in the operation of the distribution network, each control task is defined as a binary tuple. ,in Indicates the type of regulatory target. This represents the spatiotemporal distribution pattern of electric vehicles. The number of control targets, such as load shedding, photovoltaic power consumption, and voltage control. This represents the number of typical distribution patterns for electric vehicles.

[0134] The definitions of various economic cost items are as follows:

[0135] (14)

[0136] This indicates the cost incurred by reducing active and reactive power for normal loads;

[0137] (15)

[0138] This indicates the cost reduction associated with improving the charging and discharging capabilities of electric vehicles.

[0139] (16)

[0140] This indicates the loss of abandoned electricity when photovoltaic power cannot be connected to the grid;

[0141] (17)

[0142] This indicates the cost of electricity purchased by the substation;

[0143] (18)

[0144] This represents the squared error of the node voltage deviating from 1.0 pu (per unit).

[0145] in, Let t be the electricity price for time period t. , and These are the penalty coefficients for base load, EV load, and photovoltaic output reduction, respectively. I represents the set of ordinary load nodes, and I represents the set of electric vehicle nodes. A set of photovoltaic access nodes. Injecting active power into the substation Let be the voltage amplitude of node i at time t, i.e., the per-unit value.

[0146] and The portion that can be reduced from the base load; and This is a portion of the combined active and reactive load that can be reduced for electric vehicles.

[0147] In addition, to penalize electric vehicles for failing to meet their energy demands when leaving a parking lot, the following energy shortfall cost is defined:

[0148] (19)

[0149] (20)

[0150] in, This indicates that electric vehicle i will depart at the scheduled time. The minimum state of charge that is expected to be achieved at that time. Penalty weights for insufficient battery power in electric vehicles.

[0151] Based on the above definition, each task The objective function at time t The model is as follows:

[0152] (twenty one)

[0153] in, to The task weight coefficients for the m-th type of control objective reflect the operator's priority preferences for different cost items. The corresponding electric vehicle distribution pattern is represented by a matrix. The representation, as shown in equations (9)-(13), indicates whether each electric vehicle is connected to the grid node at different times. Equation (21) enables unified modeling under multi-task control objectives.

[0154] Step S4: The vehicle-grid interaction regulation problem established in steps S1-S3 is formalized into a Markov decision process (MDP), and the state space, action space, transition probability, reward function, and discount factor are determined; the vehicle-grid interaction regulation problem includes a distribution network operation model considering the characteristics of electric vehicle access, a charging and discharging behavior model of electric vehicle groups, and task tuples.

[0155] In step S4, to effectively support the multi-task control strategy for electric vehicle groups in the distribution network, the vehicle-grid interaction control problem described in steps S1-S3 is modeled as a Markov Decision Process (MDP) to form an intelligent agent. The Markov Decision Process is defined as a quintuple. ,in, Representing the state space, Represents the action space. Represents the state transition probability. For the reward function, This represents the discount factor. The Markov decision process described above can accurately describe the dynamic evolution and system operation characteristics during the vehicle-to-grid (V2G) interaction control process.

[0156] (1) State space :

[0157] At any time t, the system state includes information from both the power grid level and the electric vehicle level:

[0158] 1) Power grid status It is used to characterize the operating status of the distribution network, including information such as node voltage amplitude, line power flow, photovoltaic output and load shedding:

[0159] (twenty two)

[0160] 2) Electric vehicle status This is used to describe the energy state and scheduling information of individual electric vehicles.

[0161] (twenty three)

[0162] in, This represents the initial charge of the i-th electric vehicle when it is connected to the power distribution network. This indicates the start time of the vehicle's connection to the charging station. This indicates the planned departure time of the vehicle. Together, these three factors describe the electric vehicle's access status and charging demand window.

[0163] 3) Global state vector:

[0164] (twenty four)

[0165] (2) Action space :

[0166] At each time t, corresponding charging and discharging power needs to be allocated to the electric vehicle group:

[0167] (25)

[0168] Directly controlling the actions of a large number of individual electric vehicles could lead to the curse of dimensionality, significantly increasing computational complexity. To reduce the problem size and improve computational efficiency, a cluster-based control approach is adopted. This involves using an information aggregation center as a proxy unit for the electric vehicle group to achieve dimensionality reduction control and scheduling optimization.

[0169] (3) Reward function

[0170] At any time t, the immediate reward of the distribution network system is determined by the task. The corresponding overall operating cost is determined as follows:

[0171] (26)

[0172] in, This represents the system cost under a multi-objective weighting configuration, including loss indicators such as load reduction, electric vehicle reduction, photovoltaic curtailment, electricity purchase cost, and voltage deviation.

[0173] (4) State transition function

[0174] The transition of the system state from time t to t+1 is determined by the control action. With random disturbances The following decision was made jointly:

[0175] (27)

[0176] in, The charging / discharging power calculation module calculates the allowable power interaction based on the current state of charge, the electric vehicle's connection status, and regulatory actions. If the regulatory actions exceed physical limits, the module automatically reduces excessive charging / discharging power to ensure operational feasibility and equipment safety. The resulting aggregated charging / discharging load is then passed as input to the power flow solving module. Update system status, including node voltages. Branch current And other operating indicators, satisfying the physical constraints defined by formulas (1) to (6). This represents external random disturbance variables, used to simulate system disturbance factors such as the uncertainty of electric vehicle arrival and departure, load fluctuations, and photovoltaic power changes, including electric vehicle travel behavior, load fluctuations, or photovoltaic output uncertainty.

[0177] Indicates the system at time step The status includes the operating status of the distribution network, the access information of electric vehicles and their state of charge, etc. Indicates the state The regulatory actions taken include the allocation of charging and discharging power for electric vehicles, the amount of solar power curtailment, or the load reduction ratio. This represents the state transition function, used to characterize the system's transition from state to state under given actions and disturbances. Evolving to the next state The process.

[0178] Step S5: Construct an offline meta-reinforcement learning model for the regulation of electric vehicle groups in the power distribution network, including an online interaction stage and an offline adaptive stage.

[0179] During the online interaction phase, the agent undergoes interactive training in a single, specific task environment, aiming to acquire expert experience data representative of the task's characteristics. Each task is defined by a specific Markov decision process, differing in its objective function (e.g., voltage stability, peak shaving and valley filling) and the spatial-temporal distribution characteristics of electric vehicles. Through in-task reinforcement learning training, a policy agent with expert performance is obtained, and high-quality trajectory data, including state-action-reward triples, is collected.

[0180] The specific steps are as follows:

[0181] S5.1: Construct a vehicle-to-grid coupling environment representing a specific task, including factors such as node load, photovoltaic output, electric vehicle access behavior, and grid power flow;

[0182] S5.2: Based on specific task objectives, reinforcement learning algorithms are used to train the agent to obtain expert policies;

[0183] S5.3: Collect trajectory data of the expert agent during the training process, including state transition sequences, reward information, and behavioral statistical features;

[0184] S5.4: Extract the scheduling behavior pattern of each task, including typical charging and discharging strategy sequences, as reference signals for subsequent task identification and meta-network.

[0185] In the offline adaptation phase, a unified meta-network is constructed. Unlike traditional data-driven methods, the offline adaptation phase integrates physical runtime data with semantic information extracted from pre-trained language models to achieve collaborative optimization of task recognition and policy transfer. This enables rapid adaptation to unseen tasks.

[0186] The specific steps are as follows:

[0187] S5.5: Randomly sample Markov decision process samples and semantic task descriptions of the samples from expert trajectories of multiple tasks, such as "improve photovoltaic consumption, with emphasis on midday scheduling";

[0188] S5.6: Use a pre-trained language model to extract vector features of semantic descriptions, and input them together with the trajectory behavior context into the meta-network to learn task embedding representations;

[0189] S5.7: The low-dimensional task embedding is concatenated with the physical observation state in the current task and used as input to the policy network for decision generation;

[0190] S5.8: Optimize the meta-policy network so that when faced with new tasks, the meta-policy network can quickly transfer based on the embedding representation and output effective control policies, thereby improving generalization ability and practicality.

[0191] Step S6: Considering the significant differences in state transition functions and reward function mappings among different tasks, a task distribution set covering various operational objectives and control scenarios is constructed to uniformly represent diverse Markov decision processes. This task distribution not only provides a training sample basis for the offline adaptive phase but also provides structured support for the task representation learning and policy generalization of the meta-network, enhancing the modeling capabilities of the entire meta-reinforcement learning framework in terms of task diversity and transfer adaptability.

[0192] In step S6, in the electric vehicle population control problem, the task differences mainly stem from the following two types of changes:

[0193] (1) Due to the different spatiotemporal distribution of electric vehicles and the randomness of photovoltaic power output, there is heterogeneity in the dynamics of state transitions; (2) Different tasks have different objective functions, thus introducing differences in reward mapping. Therefore, even state-action pairs The same applies, but the transfer probability and rewards may differ under different tasks.

[0194] set up For task distribution, each task can be uniquely determined by the transition function and the reward function.

[0195] To describe the differences in state transitions and rewards between different tasks, the "task-transition correspondence" hypothesis is introduced:

[0196] (28)

[0197] in, This represents the state transition probability distribution after performing action a in state s under the i-th task. This represents the instant reward function. Let represent the overall definition of the task, s represent the system state, and a represent the optional action. This assumption states that if the transition probabilities of two tasks under all state-action pairs are completely consistent with the reward function, then they can be considered the same task.

[0198] Furthermore, each task It can be uniquely represented by a mapping function from its state-action pair to the "next state and reward":

[0199] (29)

[0200] in, Indicates task The state transition-reward function is as follows. Indicates the state The regulatory actions taken below For the next state, For immediate rewards. Therefore, the task set can be derived. This constitutes a family of functions on the state transition-reward space, which defines the task context space. It is used to characterize the structural differences in different regulatory tasks.

[0201] To achieve unsupervised representation learning for the task, a context encoder is introduced into the meta-network. This is used to extract potential task representations from interaction trajectories, and its mapping form is as follows:

[0202] (30)

[0203] in, Representing the state space, Represents the action space. Represents the reward value space. For context representation space, For parameters The context encoder. The network uses several trajectory segments. Using the input, the latent embedding representation z of the task is extracted and used as an auxiliary input for subsequent policy optimization, thereby improving the agent's generalization ability on unknown tasks.

[0204] In theory, trajectory information can uniquely correspond to a specific task. However, in practice, due to limited environmental interaction data, task recognition performance is poor, which in turn affects the model's adaptability to new tasks.

[0205] To address this issue, this invention proposes a meta-network model that integrates physical data and semantics. By introducing a pre-trained language model, prior task information is extracted from semantic instructions provided by human experts and deeply fused with physical trajectory data (step S7) to improve the accuracy of task recognition and the model's generalization ability. For example, in scenarios with severe voltage fluctuations, a dispatcher might issue dispatch instructions such as "prioritize voltage control, even if it requires significant load shedding and photovoltaic curtailment." This semantic information can be used to guide the model in determining the dispatch objective.

[0206] Step S7: Based on the task distribution set in Step S6, enhance the interactive expression capability of semantic-state information using the cross-modal attention fusion module.

[0207] Specifically, physical observation data is used as the query vector, and semantic task description is used as the key vector. An attention mechanism is employed to achieve conditional coupling modeling of the physical observation data and semantic task description, enabling the meta-reinforcement learning model to dynamically focus on task-related key information in the instructions based on the current state. The fused features of the attention mechanism's output are then passed as input to the policy network, enhancing task recognition capabilities and the contextual adaptability of the control policy.

[0208] In step S7, in order to effectively fuse heterogeneous modal information, physical observation data is used as the query vector of the cross-attention fusion module, and semantic features are used as the key vector and value vector to achieve a conditional association between task semantics and state information in formula 24.

[0209] Specifically, let the physical observation characteristics be... , indicating that it contains A local trajectory segment at each time step. The physical state feature dimension for each time step typically includes node voltage, line power flow, and electric vehicle state of charge; the semantic task is represented as... ,in Indicates the maximum length of the semantic instruction. The embedding dimension output by the language model. Semantic representation. It is obtained by encoding task instructions using a pre-trained language model, and can capture the target preferences and control intentions in the instructions.

[0210] Physical observation features and semantic representations are linearly projected onto a unified space:

[0211] (31)

[0212] in , These are trainable parameters.

[0213] Then, the multi-head attention was calculated:

[0214] (32)

[0215] The output of multi-head attention is processed by residual connection and layer normalization:

[0216] (32-1)

[0217] Then processed by a feedforward network:

[0218] (33)

[0219] in , , This is the ReLU function.

[0220] Finally, pooling operations (such as average pooling, max pooling, or attention pooling) are applied to the fused features to generate the final representation vector. As a task representation that integrates semantics and physical priors, it is used for downstream policy generation and meta-policy training.

[0221] Step S8: To accurately identify the differences between different tasks while maintaining the similarity within the same task, during the training of the meta-reinforcement learning model, a contrastive learning objective function is used to guide the embedding modules in the meta-network to construct a task-discriminable structured distribution in the latent embedding space.

[0222] In step S8, the contrastive learning objective function aggregates trajectory-semantic instruction positive sample pairs from the same task and separates negative sample pairs between different tasks to represent the feature differences between negative sample pairs of different tasks. In other words, it maximizes the feature differences between negative sample pairs of different tasks so that the meta-network can learn to capture the potential differences between tasks.

[0223] The positive sample pairs consist of trajectory embeddings and corresponding semantic instructions under the same task. They are used to train the meta-reinforcement learning model to bring the semantic instructions and the matched trajectory representations closer to each other in the latent embedding space, thereby achieving the feature mapping capability of semantic-behavior alignment.

[0224] This invention introduces a contrastive learning objective function based on negative power form in task embedding learning, defined as follows:

[0225] (34)

[0226] in, This represents the loss function for distance metric learning. For two input samples (such as context trajectory pairs from different tasks); This indicates the corresponding category label or task identifier; and This is an indicator function that takes the value 1 when samples come from the same task, and 0 otherwise. This refers to the sample embedding vector obtained through the context encoder; The square of the Euclidean distance; Euclidean distance The power form is used to control the strength of the similarity penalty for negative samples, where It is a hyperparameter that adjusts the power; It is a small constant used to avoid division by zero errors; It is a hyperparameter that balances the loss of positive and negative samples.

[0227] This loss function encourages samples from the same task to be close together in the embedding space, while penalizing samples from different tasks, especially when their embeddings are too close, thereby enhancing the context encoder's ability to discriminate task semantics.

[0228] For each task Given its pre-collected dataset and accompanying semantic instructions ,in: The initial state of the j-th sample in the i-th task; : The corresponding action; The next state after the action is performed; The instant reward received; This indicates the number of samples for each task trajectory data; Indicates task The corresponding semantic task description comes from the semantic space. Such as scheduling objectives and instructions, like "prioritize voltage control" and "reduce the amount of wasted light".

[0229] Context encoder Based on the sampled trajectory context With semantic information Output task representation Capture multimodal task features. Any task embedding Compared with the current state After concatenation, input the policy network. With Q network They generate control actions and action value assessments respectively.

[0230] The contrastive learning objective function includes the following three loss terms:

[0231] Strategy Loss Used to update strategy parameters ;

[0232] Value function loss Used to update value network parameters ;

[0233] Embedded consistency loss Encourage the i-th task With the j-th task The embedding maintains structural stability.

[0234] The aforementioned loss terms are jointly optimized using gradient descent. During multi-task iterative training, a compact and generalizable task representation is gradually learned. Ultimately, they will possess the ability to quickly adapt to unknown mission environments.

[0235] The contrastive learning objective function, used as an auxiliary loss term in the training process of the meta-policy network, is optimized together with the reinforcement learning loss of the main task (such as policy gradient or PPO loss) to improve policy transfer and generalization performance. After training, the meta-network model generates a control policy that conforms to semantic intent and physical constraints based on the given task embedding, enabling autonomous scheduling decision output for new tasks. The contrastive learning objective function promotes a well-structured embedding space, allowing representations of similar tasks to naturally cluster together while representations of different tasks remain separate, thereby improving the model's ability to recognize tasks and generalize to unseen tasks.

[0236] The proposed control framework was validated on a dual NVIDIA RTX 4090 GPU server platform using Python 3.10 as the experimental platform. The large language model employed was Qwen1.5, whose main functions include extracting task semantic information and assisting in scheduling decision optimization. The scheduling objects were 2000 electric vehicles, divided into 10 clusters based on geographical location and charging behavior characteristics to achieve cluster-level collaborative control. The system model used an IEEE 33-node distribution network as a test case to verify the effectiveness and practicality of the proposed method in the electric vehicle cluster scheduling problem.

[0237] To comprehensively evaluate the adaptability of the proposed framework, ten test scenarios were constructed, combining two electric vehicle distribution patterns with five types of control task modes. The two electric vehicle distribution patterns differ significantly in temporal and energy characteristics: the first type is temporally concentrated (arrival times are concentrated in time slots 2 to 49), with long dwell times, low initial SOC (range 0.11–0.19), and high target SOC (above 0.63), representing a typical scenario with high demand and strong controllability; the second type is more dispersed in arrival time (time slots 0–57), exhibiting diverse charging demands and reflecting the uncertainty of user behavior in reality. The five control modes are weighted using the task cost coefficients in Table 1. Define the different priorities of each task in terms of operational objectives. For example, patterns. The emphasis is on load reduction and the amount of reduction in electric vehicles, as well as voltage deviation, and mode. Greater attention will be paid to the capacity to accommodate renewable energy, and a higher penalty will be imposed on curtailment of photovoltaic power.

[0238] Table 1. Task Cost Weighting Coefficients under Different Control Modes

[0239]

[0240] To verify the effectiveness of the proposed semantic-physical fusion meta-reinforcement learning framework, three representative baseline methods were designed and compared: (i) Meta-network-free: using standard policy networks (such as offline SAC) without introducing any task adaptation mechanisms; (ii) Pure physical input: using only structured physical observations as meta-network input, without containing any semantic information; (iii) Pure semantic input: relying only on semantic embeddings extracted from task descriptions, ignoring physical observation data; (iv) Semantic-physical fusion: fusing physical measurements and semantic information, achieving adaptive control under multiple tasks through the meta-network. These four methods were evaluated in 10 constructed task scenarios, and the differences in generalization ability among the methods were systematically analyzed.

[0241] Figure 2 shows the average evaluation reward curves of the four methods during training as a function of epochs. The results indicate that the proposed semantic-physical fusion method achieves higher rewards early in training and converges significantly faster than the baseline methods, demonstrating stronger stability and efficiency. In contrast, the pure physical method exhibits slower convergence and fluctuations, reflecting its insufficient robustness to environmental changes when task prior information is lacking. The pure semantic method, lacking the supplementation of actual trajectories, shows even slower convergence and higher variance in policy learning. The meta-network-less method performs the worst overall, with drastic fluctuations in reward levels during training and a lack of effective task differentiation and policy adaptation capabilities. In summary, the proposed meta-learning framework fusing semantic and physical perception can effectively improve policy generalization ability and significantly enhance the robustness and performance of electric vehicle swarms in complex control tasks.

[0242] Example 2

[0243] A computer system includes a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the method described above.

[0244] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the technical principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A vehicle-to-grid (V2G) interaction control method based on multi-task offline meta-reinforcement learning, characterized in that, include: Collect power distribution network operation and scheduling information, traffic network topology, spatiotemporal travel demand data of electric vehicles, and investment cost parameters of different types of charging infrastructure to construct a power distribution network operation model that takes into account the access characteristics of electric vehicles. A charging and discharging behavior model for electric vehicle groups is constructed to represent the charging demand, power constraints, and energy evolution of each electric vehicle at different times and locations. Different operation and control objectives of the distribution network and their corresponding spatiotemporal configuration requirements are abstracted into task tuples, and a corresponding economic cost item is defined for each type of task. The task tuple consists of the objective function and the configuration scenario. The established vehicle-grid interaction regulation problem is formalized as a Markov decision process, constituting an intelligent agent; the vehicle-grid interaction regulation problem includes a distribution network operation model considering the access characteristics of electric vehicles, a charging and discharging behavior model of electric vehicle groups, and task tuples. An offline meta-reinforcement learning model for the regulation of electric vehicle groups in a power distribution network is constructed, including an online interaction stage and an offline adaptive stage. During the online interaction phase, the agent undergoes interactive training in a single, specific task environment, aiming to acquire expert experience data representative of the task's characteristics. During the offline adaptive phase, a unified meta-network is constructed to achieve collaborative optimization of task recognition and policy transfer. The power distribution network operation model considering the characteristics of electric vehicle access is expressed as follows: A linearized branch power flow model is used to model the radial structure of the distribution network to represent the active power balance, reactive power balance, voltage relationships, and physical constraints in the distribution network system, including the following constraints: （1） （2） （3） （4） （5） （6） （7） （8） The set of electric vehicle aggregators is denoted as . , Indicates the node number. Indicates a time index; Indicates from node To the node The power distribution lines, For the set of routes; and The lines are respectively At any moment The active and reactive power flow on the surface , Represents a node , voltage amplitude, and These are the line impedance parameters; and The active and reactive power injected into the substation. For the balancing node indicator parameters, if the node If it is a balanced node, the value is 1; otherwise, it is 0. and They are nodes Basic active and reactive loads, and The portion that can be reduced from the base load; and This represents the aggregate active and reactive loads of electric vehicles. and This reduces the combined active and reactive load of electric vehicles. and These represent the active and reactive power outputs of the photovoltaic system, respectively. This represents the maximum available active power of photovoltaic power under ideal weather conditions. This refers to the reduction in photovoltaic output due to dispatch control. This represents the minimum power factor of a photovoltaic inverter; and These are the lower and upper limits of the node voltage, respectively; Indicates connection to node A collection of electric vehicle clusters, For clusters The set of electric vehicle nodes in the data. and Electric vehicles The active power of charging and discharging, For nodes The reactive power of electric vehicles; Indicates time Time period nodes To the node The active power transmitted on the branches between them Represents a node To the node Active power in the direction, and These represent the reactive power transmission amounts in the corresponding directions, with the power direction being positive from the branch start point to the end point; In the electric vehicle group charging and discharging behavior model, for each node , Let be the set of electric vehicle nodes, at any given time. The charging and discharging behavior is expressed as equations (9) to (13): （9） （10） （11） （12） （13） in, and They represent electric vehicles. At any moment The active power of charging and discharging, Represents a node The reactive power of electric vehicles; and These are the maximum permissible active power and apparent power capacity, respectively; variables. This is a binary decision variable used to indicate whether an electric vehicle is charging. Indicates charging. Indicates that it is not charging; This is a vehicle connection status indicator; if it is... This indicates that the car is in Always connected to the power grid, This indicates that the connection is not established. This represents the state of charge (SoC) of an electric vehicle, with a lower limit. and upper limit limit; For charge and discharge efficiency; It represents the duration of a time segment.

2. The vehicle-to-network interaction control method based on multi-task offline meta-reinforcement learning according to claim 1, characterized in that, Also includes: Construct a task distribution set covering multiple operational objectives and control scenarios to uniformly represent diverse Markov decision processes; Based on the task distribution set, a cross-modal attention fusion module is used to enhance the interactive expression capability of semantic-state information.

3. The vehicle-to-network interaction control method based on multi-task offline meta-reinforcement learning according to claim 1, characterized in that, Also includes: During the training of the meta-reinforcement learning model, a contrastive learning objective function is used to guide the embedding modules in the meta-network to construct a task-discriminable structured distribution in the latent embedding space.

4. The vehicle-to-network interaction control method based on multi-task offline meta-reinforcement learning according to claim 1, characterized in that, A two-stage power flow calculation method is used to solve the distribution network operation model considering the characteristics of electric vehicle access. First, the active and reactive power demand of the nodes of the distribution network system is initially estimated using the linearized power flow model shown in equations (1) to (6). The preliminary estimation results are passed as input to the OpenDSS simulation platform to perform high-fidelity nonlinear power flow calculations and obtain more accurate voltage and power distribution results.

5. The vehicle-to-network interaction control method based on multi-task offline meta-reinforcement learning according to claim 1, characterized in that, In the process of abstracting different operation and control objectives of the distribution network and their corresponding spatiotemporal configuration requirements into task tuples, each control task is defined as a tuple. ,in Indicates the type of regulatory target. This represents the spatiotemporal distribution pattern of electric vehicles; To regulate the number of target types, The number of typical distribution patterns of electric vehicles; The definitions of various economic cost items are as follows: （14） This indicates the cost incurred by reducing active and reactive power for normal loads; （15） This indicates the cost reduction associated with improving the charging and discharging capabilities of electric vehicles. （16） This indicates the loss of abandoned electricity when photovoltaic power cannot be connected to the grid; （17） This indicates the cost of purchasing electricity for the substation; （18） in, For time period Electricity price, , and These are the penalty coefficients for base load, EV load, and photovoltaic output reduction, respectively. For a set of ordinary load nodes, For electric vehicle nodes, A set of photovoltaic access nodes. The active power injected into the substation. For nodes exist Voltage amplitude at any given moment; and The portion that can be reduced from the base load; and This reduces the combined active and reactive load of electric vehicles. Define the following energy shortage cost: （19） （20） in, Indicates electric vehicles at the scheduled departure time The minimum state of charge that is expected to be achieved at that time. Penalty weights for insufficient battery power in electric vehicles; Each task At any moment objective function The model is as follows: （21） in, to To target the first The weight coefficients of each task for the regulatory objectives.

6. The vehicle-to-network interaction control method based on multi-task offline meta-reinforcement learning according to claim 1, characterized in that, In formalizing the vehicle-to-grid (V2G) interaction regulation problem into a Markov decision process, the Markov decision process is defined as a quintuple. ,in, Representing the state space, Represents the action space. Represents the state transition probability. For the reward function, Discount factor; (1) State space : At any time The status of the power distribution network system includes information at both the grid level and the electric vehicle level: 1) Power grid status It is used to characterize the operating status of the distribution network, including node voltage amplitude, line power flow, photovoltaic output and load shedding information: （22） 2) Electric vehicle status This is used to describe the energy state and scheduling information of individual electric vehicles. （23） in, Indicates the first The initial charge of an electric vehicle when connected to the power distribution network. This indicates the start time of the vehicle's connection to the charging station. Indicates the time the vehicle is scheduled to leave; 3) Global state vector: （24） (2) Action space : At every moment Appropriate charging and discharging power needs to be allocated to electric vehicles: （25） (3) Reward function : At any time The immediate rewards for the distribution network system are determined by the task. The corresponding overall operating cost is determined as follows: （26） in, This represents the system cost under a multi-objective weighted configuration, including load reduction, electric vehicle reduction, solar curtailment, electricity purchase cost, and voltage deviation. (4) State transition function : Distribution network system status from time 1 arrive The transfer is caused by regulatory actions With random disturbances The following decision was made jointly: （27） in, The charging and discharging power calculation module calculates the allowable power interaction amount based on the current state of charge, electric vehicle connection status, and control actions. Then, the obtained aggregated charging and discharging load is passed as input to the power flow solution module. Update system status, including node voltages. Branch current , satisfying the physical constraints defined by formulas (1) to (6); This represents external random disturbance variables, used to simulate the uncertainty of electric vehicle arrival and departure, load fluctuations, and photovoltaic power change system disturbance factors; Indicates the system at time step The status includes the operating status of the distribution network, the access information of electric vehicles, and their state of charge; Indicates the state The regulatory actions taken below; This represents the state transition function, used to describe the system's transition from state to state under given actions and disturbances. Evolving to the next state The process.

7. The vehicle-to-network interaction control method based on multi-task offline meta-reinforcement learning according to claim 1, characterized in that, During the online interaction phase, perform the following steps: S5.1: Construct a vehicle-to-grid coupling environment representing a specific task, including node load, photovoltaic output, electric vehicle access behavior, and power grid flow; S5.2: Based on specific task objectives, reinforcement learning algorithms are used to train the agent to obtain expert policies; S5.3: Collect trajectory data of the expert agent during the training process, including state transition sequences, reward information, and behavioral statistical features; S5.4: Extract the scheduling behavior pattern of each task, including typical charging and discharging strategy sequences, as reference signals for subsequent task identification and meta-network; During the offline adaptive phase, perform the following steps: S5.5: Randomly sample Markov decision process samples and semantic task descriptions of the samples from expert trajectories of multiple tasks; S5.6: Use a pre-trained language model to extract vector features of semantic descriptions, and input them together with the trajectory behavior context into the meta-network to learn task embedding representations; S5.7: The low-dimensional task embedding is concatenated with the physical observation state in the current task and used as input to the policy network for decision generation; S5.8: Optimize the meta-policy network so that when faced with new tasks, the meta-policy network can quickly transfer based on the embedding representation and output an effective control policy.

8. The vehicle-to-network interaction control method based on multi-task offline meta-reinforcement learning according to claim 3, characterized in that, In the contrastive learning objective function, by aggregating positive sample pairs of trajectory-semantic instructions from the same task and separating negative sample pairs between different tasks, the feature differences between negative sample pairs of different tasks are represented, enabling the meta-network to learn to capture the potential differences between tasks. The positive sample pairs consist of trajectory embeddings and corresponding semantic instructions under the same task. They are used to train the meta-reinforcement learning model to bring the semantic instructions and the matched trajectory representations closer to each other in the latent embedding space, thereby achieving the feature mapping capability of semantic-behavior alignment.

9. The vehicle-to-network interaction control method based on multi-task offline meta-reinforcement learning according to claim 8, characterized in that, The contrastive learning objective function is defined as follows: （34） in, The loss function represents distance metric learning; There are two input samples; This indicates the corresponding category label or task identifier; and This is an indicator function that takes the value 1 when samples come from the same task, and 0 otherwise. This is the sample embedding vector obtained through the context encoder; The square of the Euclidean distance; Euclidean distance The power form is used to control the strength of the similarity penalty for negative samples, where It is a hyperparameter that adjusts the power; It is a constant; It is a hyperparameter that balances the loss between positive and negative samples; For each task Given its pre-collected dataset and accompanying semantic instructions ,in: : No. The first task The initial state of each sample; : The corresponding action; The next state after the action is performed; The instant reward received; This indicates the number of samples for each task trajectory data; Indicates task The corresponding semantic task description comes from the semantic space. ; Context encoder Based on the sampled trajectory context With semantic information Output task representation Capture multimodal task features; any task embedding Compared with the current state After concatenation, input the policy network. With Q network They generate control actions and action value assessments respectively.

10. A computer system comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the steps of the vehicle-to-grid interaction control method based on multi-task offline meta-reinforcement learning as described in any one of claims 1-9.

Images