Cooperative inspection method and device based on multi-agent reinforcement learning in wind field environment

Abstract

This invention discloses a collaborative inspection method and device based on multi-agent reinforcement learning in a wind farm environment. In each decision cycle, the target inspection device considers each mobile supply device and / or each wind turbine to be inspected in the inspection area as candidate targets. It calculates the spatiotemporal attraction between the target inspection device and each candidate target. Based on the target inspection device's current remaining power and the obtained spatiotemporal attraction, the target task of the target inspection device is determined, whereby the target task is inspection or supply. An attention mechanism is employed, based on the target inspection device's local observation vector and the target feature vectors of each candidate target, to determine the attention weights between the target inspection device and the corresponding candidate targets, thus obtaining the corresponding spatiotemporal attraction. The local observation vector includes the flight energy consumption penalty distribution of the target inspection device in the current area. This invention can effectively improve the supply success rate of the inspection device.

Description

Technical Field

[0001] This invention relates to the field of automatic control, and in particular to a collaborative inspection method and device based on multi-agent reinforcement learning in a wind farm environment. Background Technology

[0002] Currently, the solutions for inspecting wind turbines in wind farm environments are often based on traditional heuristic algorithms for path planning, or manual / semi-automatic inspection. For manual / semi-automatic inspections, inspections are performed by manual remote control or preset waypoints; drones are usually carried to a designated area by mobile resupply equipment (unmanned vehicles or unmanned boats) and then take off, and inspections are performed by manual remote control or preset waypoints. Offshore wind farms have strong wake effects and rapidly changing sea conditions (wind and waves). Traditional static planning cannot be adjusted in real time to avoid high turbulence areas, resulting in higher-than-expected energy consumption of drones. In onshore wind farms, due to the undulating terrain (mountains, hills) and the arrangement of wind turbine arrays, the superposition of wind turbine wakes forms complex turbulence channels, causing drones to experience lift instability and a surge in energy consumption in the downstream area. Summary of the Invention

[0003] This invention addresses the shortcomings of existing technologies that rely on drones for inspection due to the susceptibility to dynamic wake interference from wind fields and the limited endurance of drones. It provides a collaborative inspection method and device based on multi-agent reinforcement learning in wind field environments.

[0004] To solve the above-mentioned technical problems, the present invention provides the following technical solution: Firstly, this invention proposes a collaborative inspection method based on multi-agent reinforcement learning in a wind field environment, for use on the inspection equipment side, comprising the following steps: In each decision cycle, the target inspection equipment takes each mobile supply device and / or each wind turbine to be inspected in the inspection area as candidate targets, calculates the spatiotemporal attraction between the target inspection equipment and each candidate target, and determines the target task of the target inspection equipment based on the current remaining power of the target inspection equipment and the obtained spatiotemporal attraction. The target task is inspection or supply. in: The local observation vector is constructed based on the three-dimensional coordinates, velocity vector, current remaining power, and local wake cost of the target inspection equipment. The local wake cost is used to indicate the distribution of flight energy consumption penalty of the target inspection equipment in the current area. Based on the three-dimensional coordinates, type encoding, and real-time status of the candidate targets, a target feature vector corresponding to each candidate target is constructed. An attention mechanism is employed to determine the attention weights between the target inspection equipment and the corresponding candidate targets based on the local observation vector and the target feature vector, thereby obtaining the corresponding spatiotemporal attraction.

[0005] As one possible implementation method: If the remaining power is less than the preset replenishment threshold, the target task is directly determined to be replenishment. Each mobile replenishment device is taken as a candidate target. Based on the local observation vector of the target inspection device and the spatiotemporal attraction between the target inspection device and each candidate target, a distress message is generated and broadcast. If the current remaining power is greater than or equal to the preset warning threshold, the target task is directly determined to be inspection, and each wind turbine to be inspected is taken as a candidate target. The target object to be inspected is determined based on the spatiotemporal attraction between the target inspection equipment and each candidate target; the warning threshold is greater than the replenishment threshold. If the current remaining power is greater than or equal to the replenishment threshold and less than the warning threshold, then each mobile replenishment device and each wind turbine to be inspected will be regarded as a candidate target, and the target task will be determined based on the spatiotemporal attraction between the target inspection device and each candidate target.

[0006] As one possible implementation, when the current remaining power is greater than or equal to the replenishment threshold and less than the warning threshold: The maximum inspection attraction is determined based on the spatiotemporal attraction of the target inspection equipment on each wind turbine to be inspected. The spatiotemporal attraction of the target inspection equipment to the mobile supply equipment is taken as the corresponding supply attraction. When the ratio of the supply attraction to the maximum inspection attraction exceeds a preset attention threshold, the target task is set to supply; otherwise, the target task is set to inspection. When the target task is inspection, the wind turbine to be inspected is based on the wind turbine with the maximum inspection attraction. When the objective is resupply: A distress signal is generated and broadcast globally. The distress signal includes the local observation vector corresponding to the target inspection equipment and the spatiotemporal attraction between the target inspection equipment and each mobile supply equipment. Receive spatiotemporal intersection points sent by external control devices or target mobile supply devices, and use them as target objects. The spatiotemporal intersection points are used to indicate the time and coordinates of the intersection with the target mobile supply device.

[0007] As one possible implementation method: The pre-trained inspection-side Actor policy network outputs corresponding first action commands based on the local observation vectors and target objects. These commands instruct the target inspection equipment on its three-dimensional flight heading angle and acceleration, guiding it to fly out of the current area with the goal of minimizing energy consumption when reaching the target object.

[0008] As one possible implementation method: An attention mechanism is employed to determine the intent vector of the target inspection device based on the local observation vector, determine the attribute vector corresponding to the candidate target based on each target feature vector, and determine the attention weight between the target inspection device and the corresponding candidate target based on the inner product of the intent vector and the attribute vector, thereby obtaining the corresponding spatiotemporal attraction.

[0009] As one possible implementation method: The corresponding query matrix is ​​determined based on the local observation vector and the corresponding first projection matrix. The query matrix is ​​used as the corresponding intent vector to indicate the current demand intent of the corresponding target inspection equipment. The corresponding key matrix is ​​determined based on the target feature vector and the corresponding second projection matrix. The key matrix is ​​used as the corresponding attribute vector to indicate the task attributes of the corresponding candidate target. The value matrix of the corresponding candidate target is determined based on the target feature vector and the corresponding third projection matrix; The first projection matrix, the second projection matrix, and the third projection matrix are obtained through pre-training.

[0010] As one possible implementation method: Local wake cost includes the energy cost scalar corresponding to each grid in the current region, wherein the current region is determined based on the preset region range and the three-dimensional coordinates of the target inspection equipment; The energy consumption cost scalar is an energy consumption penalty obtained by weighted calculation based on speed loss penalty, attitude maintenance penalty, and crosswind / headwind penalty; The velocity loss penalty is calculated based on the free-flow reference wind speed and the actual wind speed in the corresponding grid, resulting in the corresponding aerodynamic lift loss. The attitude maintenance penalty is calculated based on the corresponding high-frequency wind disturbance calculated according to the turbulence intensity in the corresponding grid. The crosswind / headwind penalty is based on the relative wind direction of the corresponding grid and the corresponding heading angle.

[0011] As one possible implementation method, the energy consumption cost scalar C corresponding to grid g. wake The formula for calculating (g) is: ; in: |u base -u g|As a penalty for speed loss, u base U represents the free-flow reference wind speed. g This represents the actual wind speed corresponding to grid g; This is the first weighting coefficient; This represents the high-frequency wind disturbance corresponding to grid g, which serves as an attitude maintenance penalty; This is the second weighting coefficient; Crosswind / headwind penalty, indicating the heading angle of the target inspection equipment. Wind direction relative to grid g The corresponding relative wind direction function; This is the third weighting coefficient.

[0012] As one possible implementation method: Wind field environments include offshore wind fields and onshore wind fields; The target inspection device is a drone; The mobile supply equipment includes unmanned boats, unmanned vehicles, and robots.

[0013] Secondly, this invention proposes a collaborative inspection method based on multi-agent reinforcement learning in a wind field environment for the supply side. When the current mobile supply device is the target mobile supply device to be inspected, the following steps are performed: The system receives the local observation vector corresponding to the equipment to be resupplyed and inspected, as well as the target spatiotemporal attraction of the current mobile resupply equipment to the equipment to be resupplyed and inspected. The local observation vector indicates the three-dimensional coordinates, velocity vector, current remaining power, and local wake cost of the target equipment. The local wake cost indicates the flight energy consumption penalty distribution of the target equipment in the current area. The target spatiotemporal attraction is an attention weight generated by the equipment to be resupplyed and inspected using an attention mechanism, based on the local observation vector and the target feature vector corresponding to the current mobile resupply equipment. The target feature vector indicates the three-dimensional coordinates, type encoding, and real-time status of the current mobile resupply equipment. Obtain the corresponding spatiotemporal intersection point, which is used to indicate the time and coordinates of the intersection between the current mobile supply equipment and the inspection equipment; Based on the local observation vector, the target spatiotemporal attraction, the spatiotemporal intersection point, and the three-dimensional coordinates and speed corresponding to the current mobile supply equipment, a second action command is generated. The second action command is used to indicate the follow-up intersection course and speed of the current mobile supply equipment, and guide the current mobile supply equipment to the spatiotemporal intersection point to meet the equipment to be supplied and inspected.

[0014] As one possible implementation method: The mobile supply equipment currently receives a response command from an external control device. This response command is used to indicate the equipment to be supplied and inspected, and the corresponding spatiotemporal rendezvous point. or, The current mobile supply equipment, based on its three-dimensional coordinates and maximum response speed, as well as the three-dimensional coordinates, maximum return speed, and local wake cost of the equipment to be supplied and inspected, solves the shortest encounter time and the corresponding rendezvous coordinates to obtain the corresponding spatiotemporal rendezvous point.

[0015] As one possible implementation method, the shortest encounter time that satisfies the dynamic constraints is determined; The dynamic constraints include: ; ; in: T dock Indicates the shortest meeting time; (X) c Y c () represents the intersection coordinates corresponding to the shortest meeting time; (X) u0 Y u0 () represents the three-dimensional coordinates of the equipment to be resupplyed and inspected; (X) s0 Y s0 () represents the three-dimensional coordinates of the current mobile supply equipment; V uav This indicates the maximum return speed of the equipment to be resupplyed and inspected; C wake This represents the local wake cost corresponding to the equipment to be resupplyed and inspected; It is the attenuation factor; V usv This indicates the maximum response speed of the current mobile supply equipment; T0 represents the current time.

[0016] As one possible implementation method: The pre-trained supply-side Actor policy network outputs a corresponding second action command based on the local observation vector, the target spatiotemporal attraction, the spatiotemporal intersection point, and the three-dimensional coordinates and speed corresponding to the current mobile supply equipment. Calculate the contact distance corresponding to the second action command; Based on the response distance, a corresponding positive joint reward is generated to drive the supply-side Actor strategy network to actively respond to the mobile supply device.

[0017] Thirdly, the present invention proposes a collaborative inspection device based on multi-agent reinforcement learning in a wind field environment, wherein the collaborative inspection device is an inspection device or a mobile supply device. The inspection equipment is used to execute the collaborative inspection method based on multi-agent reinforcement learning in a wind field environment as described in any of the above-mentioned methods. The mobile supply device is used to execute the collaborative inspection method based on multi-agent reinforcement learning in the wind field environment described in any of the above-mentioned methods.

[0018] This invention, by adopting the above technical solutions, has significant technical effects: In traditional unmanned swarm inspections, tasks are usually assigned using preset routes or heuristic rules (such as mandatory return when battery is below 30%), which can easily lead to situations where the battery runs out due to wind interference and the vehicle crashes. The present invention, through the design of local wake cost and spatiotemporal attraction, independently makes task decisions based on the current remaining power and the spatiotemporal attraction to each wind turbine and / or resupply equipment to be inspected. Compared with the existing scheme of returning to resupply based on a preset power threshold, it can reduce the risk of crash and improve the resupply success rate.

[0019] The present invention further improves the success rate of resupply by actively responding to the inspection equipment waiting to be resupplyed. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a schematic diagram of the workflow of the collaborative inspection method corresponding to the inspection equipment side of the present invention; Figure 2 This is a schematic diagram of the basic cost field and path avoidance in the case study; Figure 3 This is a schematic diagram illustrating the calculation of spatiotemporal attraction in the case study, and the determination of the target task based on spatiotemporal attraction; Figure 4 This is the convergence diagram of the spatiotemporal trajectory of the UAV-USV dynamic follow-up docking in the case study. Detailed Implementation

[0022] The present invention will be further described in detail below with reference to the embodiments. The following embodiments are explanations of the present invention, but the present invention is not limited to the following embodiments.

[0023] Existing technologies for using drones to inspect wind farms have the following drawbacks: Poor adaptability to environmental dynamics: Offshore wind farms have strong wake effects and rapidly changing sea conditions (wind and waves). Traditional static planning cannot be adjusted in real time to avoid high turbulence areas, leading to unexpected energy consumption of drones. In onshore wind farms, due to undulating terrain (mountains, hills) and the arrangement of wind turbine arrays, the superposition of wind turbine wakes forms complex turbulence channels, causing drones to experience lift instability and a surge in energy consumption in downstream areas. Rigid resupply logic: Existing technologies mostly use forced return when the battery is low, without considering the spatiotemporal coupling between mobile resupply equipment and drones, resulting in a high rate of operation interruption.

[0024] To address the shortcomings of existing technologies, this application proposes a collaborative inspection method based on multi-agent reinforcement learning in wind farm environments. By utilizing a multi-agent reinforcement learning framework with centralized training and distributed execution, the dynamic wake effect of the wind farm is transformed into a physical cost field in path planning. Furthermore, an attention mechanism is introduced to decouple task priorities between heterogeneous clusters in real time, thereby achieving deep spatiotemporal coupling between the inspection path of the inspection equipment and the follow-up supply trajectory of the mobile supply equipment. This application upgrades the fixed-point supply mode to a dynamic rendezvous mode, fundamentally solving the pain points of difficult heterogeneous cluster collaboration and high operation interruption rate in complex wind farm environments, ultimately achieving a dual guarantee of energy efficiency and safety in inspection operations.

[0025] The wind field environment described in this application includes offshore wind fields and onshore wind fields; the target inspection device is an unmanned aerial vehicle (UAV); and the mobile supply equipment includes unmanned surface vessels (USVs), unmanned vehicles (UAVs), and robots.

[0026] Reference Figure 1 Based on the workflow of a target inspection device in each decision cycle, an example is given to illustrate the collaborative inspection method corresponding to the inspection device. As one possible implementation method, the target inspection equipment performs the following steps in each decision cycle: S100. Select each mobile supply equipment and / or each fan to be inspected in the inspection area as candidate targets. As one possible implementation method, all mobile supply equipment and wind turbines to be inspected are directly used as candidate targets for spatiotemporal attraction calculation.

[0027] As another possible implementation method, the task is pre-classified based on the current remaining power and the preset power threshold. The pre-classification result is either replenishment, inspection or pending. Specifically: If the remaining power is less than the preset replenishment threshold, it means that the power is insufficient and needs to be replenished in time. At this time, the target task is directly determined to be replenishment, and each mobile replenishment device is selected as a candidate target. If the current remaining power is greater than or equal to the preset warning threshold, it means that the power is sufficient at this time. The target task is directly determined to be inspection, and each wind turbine to be inspected is taken as a candidate target. If the current remaining power is greater than or equal to the replenishment threshold and less than the warning threshold, it means that the decision is made based on the wind farm environment, remaining power, and the distribution of wind turbines to be inspected. In this case, each mobile replenishment device and each wind turbine to be inspected will be considered as a candidate target.

[0028] The warning threshold is greater than the supply threshold. Those skilled in the art can set the supply threshold and warning threshold according to actual needs. For example, the supply threshold can be set to 20% and the warning threshold to 50%. This specification does not limit them in detail.

[0029] S200: Calculate the spatiotemporal attraction between the target inspection equipment and each candidate target; The method for calculating the spatiotemporal attraction between the target inspection equipment and the current candidate target is as follows: S210. Construct the local observation vector corresponding to the target inspection equipment; The local observation vector is constructed based on the three-dimensional coordinates, velocity vector, current remaining power, and local wake cost of the target inspection equipment. The local wake cost is used to indicate the distribution of flight energy consumption penalty of the target inspection equipment in the current area.

[0030] Among them, the three-dimensional coordinates, velocity vector, and current remaining power are obtained by the target inspection equipment itself through status monitoring. The local wake cost is based on the three-dimensional coordinates and velocity vector, and the flight energy consumption penalty distribution within the preset area where the target inspection equipment is located is obtained from the external control equipment.

[0031] In this embodiment, the local wake cost includes the energy consumption cost scalar corresponding to each grid in the current region, wherein the current region is determined based on a preset region range (e.g., 5×5×5 around the target inspection device) and the three-dimensional coordinates of the target inspection device. The energy consumption cost scalar is an energy consumption penalty obtained by weighted calculation based on speed loss penalty, attitude maintenance penalty, and crosswind / headwind penalty; The velocity loss penalty is calculated based on the free-flow reference wind speed and the actual wind speed in the corresponding grid, resulting in the corresponding aerodynamic lift loss. The attitude maintenance penalty is calculated based on the corresponding high-frequency wind disturbance calculated according to the turbulence intensity in the corresponding grid. The crosswind / headwind penalty is based on the relative wind direction of the corresponding grid and the corresponding heading angle.

[0032] Specifically, the energy consumption cost scalar C corresponding to grid g. wake The formula for calculating (g) is: ; in: |u base -u g |As a penalty for speed loss, u base U represents the free-flow reference wind speed. g This represents the actual wind speed corresponding to grid g; This is the first weighting coefficient; This represents the high-frequency wind disturbance corresponding to grid g, which serves as an attitude maintenance penalty; this embodiment introduces a turbulence term. This characterizes the intense eddies generated at the edge of the wake due to the rotation of the blades (which cause significant energy consumption for maintaining the flight attitude of the UAV).

[0033] This is the second weighting coefficient; Crosswind / headwind penalty, indicating the heading angle of the target inspection equipment. Wind direction relative to grid g The corresponding relative wind direction function; This is the third weighting coefficient.

[0034] in: Free-flow reference wind speed u base and wind direction The data is obtained based on measurements from on-site meteorological sensors, which can be mounted on mobile supply equipment or directly installed on-site; this specification does not impose detailed limitations on them. Actual wind speed u g The flow field tensor data obtained from the calculation of high-frequency wind disturbance can be obtained by those skilled in the art through existing publicly available wake models (Jensen wake model, Gaussian wake model, Frandsen model), or by directly accessing the flow field tensor data output by on-site wind measurement lidar (LiDAR) and fluid dynamics (CFD) software in real time. This specification does not limit it in detail. Those skilled in the art can set the first weighting coefficient, the second weighting coefficient, and the third weighting coefficient according to actual needs, and this specification does not limit them in detail.

[0035] S220. Construct the target feature vector corresponding to each candidate target; Based on the three-dimensional coordinates, type encoding, and real-time status of the candidate targets, a target feature vector corresponding to each candidate target is constructed. The type code is a one-hot code, used to distinguish whether the candidate target is a wind turbine to be inspected or a mobile supply equipment; When the candidate target is a wind turbine, its real-time status may include, for example, the inspection status (to be inspected and completed), to indicate the wind turbine to be inspected; When the candidate target is a mobile supply device, its real-time status may include, for example, supply conditions (responsive to supply tasks / unresponsive to supply tasks) to indicate mobile supply devices that can meet the supply requirements.

[0036] S230, Calculate the spatiotemporal attraction between the target inspection equipment and each candidate target; An attention mechanism is employed to determine the attention weights between the target inspection equipment and the corresponding candidate targets based on the local observation vector and the target feature vector, thereby obtaining the corresponding spatiotemporal attraction.

[0037] In this embodiment, an attention mechanism is adopted to determine the intention vector of the target inspection device based on the local observation vector, to determine the attribute vector corresponding to the candidate target based on each target feature vector, and to determine the attention weight between the target inspection device and the corresponding candidate target based on the inner product of the intention vector and the attribute vector, thereby obtaining the corresponding spatiotemporal attraction. Specifically: Three projection matrices are pre-trained to indicate attention preferences; The corresponding query matrix is ​​determined based on the local observation vector and the corresponding first projection matrix. The query matrix is ​​used as the corresponding intent vector to indicate the current demand intent of the corresponding target inspection equipment. The corresponding key matrix is ​​determined based on the target feature vector and the corresponding second projection matrix. The key matrix is ​​used as the corresponding attribute vector to indicate the task attributes of the corresponding candidate target. The value matrix of the corresponding candidate target is determined based on the target feature vector and the corresponding third projection matrix.

[0038] The larger the inner product of the intent vector and the attribute vector, the better the match between the target inspection equipment's requirements and the candidate target's task attributes. In other words, the greater the spatiotemporal attraction, the better the match between the target inspection equipment and the candidate target.

[0039] S300: Determine the target task of the target inspection equipment based on the current remaining power of the target inspection equipment and the obtained spatiotemporal attractive forces. The target task is inspection or resupply.

[0040] When pre-classification is performed first, the target task needs to be judged based on the obtained spatiotemporal attraction when the pre-classification result is pending. When no pre-classification has been performed beforehand, the target task is determined based on the current remaining power, the preset replenishment threshold, and the preset warning threshold after pre-classification. When the pre-classification result is pending, the target task is determined based on the obtained spatiotemporal attraction. Specifically: S310, The pre-classification result is pending; The maximum inspection attraction is determined based on the spatiotemporal attraction of the target inspection equipment on each wind turbine to be inspected. The spatiotemporal attraction of the target inspection equipment to the mobile supply equipment is taken as the corresponding supply attraction. When the ratio of the supply attraction to the maximum inspection attraction exceeds a preset attention threshold, the target task is set to supply; otherwise, the target task is set to inspection. In this embodiment, Set the target task to resupply if the target task is active, otherwise set it to inspection, where β i,j The maximum value represents the spatiotemporal attraction between the i-th inspection device and the j-th supply device, i.e., the corresponding supply attraction. k (α i,k ); represents the maximum inspection attraction corresponding to the i-th inspection device. The attention threshold is preset, and those skilled in the art can set it themselves according to actual needs.

[0041] By designing the spatiotemporal attraction, when the power is critical or a strong wake causes a surge in power consumption, the supply attraction surges. When the ratio of the supply attraction to the maximum inspection attraction exceeds the preset attention threshold, the task target is automatically switched to supply, realizing real-time dynamic decoupling of task scheduling without manual intervention.

[0042] S400, Execute the target task based on the obtained spatiotemporal attraction: S410. Determine the target object based on the objective task and the obtained spatiotemporal attractiveness; S411. When the target task is inspection, the wind turbine to be inspected is based on the wind turbine corresponding to the maximum inspection attraction. S412. When the objective mission is resupply: A distress signal is generated and broadcast globally. The distress signal includes the local observation vector corresponding to the target inspection equipment and the spatiotemporal attraction between the target inspection equipment and each mobile supply equipment. Receive spatiotemporal intersection points sent by external control devices or target mobile supply devices, and use them as target objects. The spatiotemporal intersection points are used to indicate the time and coordinates of the intersection with the target mobile supply device.

[0043] S420. Generate a corresponding first action command based on the target object and the corresponding local observation vector: The pre-trained inspection-side Actor policy network outputs corresponding first action commands based on the corresponding local observation vectors and target objects. These commands instruct the target inspection equipment to fly out of the current area with the goal of reaching the target object with the lowest energy consumption, thereby driving the target inspection equipment to generate a smooth trajectory with the lowest energy consumption and actively avoiding high turbulence areas.

[0044] Those skilled in the art can directly use the three-dimensional coordinates corresponding to the target object as input data for the inspection-side Actor policy network according to actual needs, or guide the UAV to fly towards the target coordinate point through the design of attention mechanism and reward function. This specification does not limit it in detail.

[0045] As one possible implementation, when the target mission is resupply and the target object is a spatiotemporal intersection point, a landing judgment step is also included before generating the first action command; Since each decision cycle involves a decision on the target task and the target object, and the decision cycle here does not refer to the period from determining the target object to arriving at the target object, but rather the duration for which the target inspection equipment maintains its current state of motion. That is, each decision cycle determines the three-dimensional flight heading angle and acceleration of the target inspection equipment within this cycle. The next decision cycle updates the three-dimensional flight heading angle and acceleration based on the status of the target inspection equipment and changes in the environment. Therefore, there are situations where the current target mission is resupply and is already very close to the spatiotemporal rendezvous point. In this embodiment: After obtaining the spatiotemporal intersection point, calculate the difference between the current time and the intersection time; When the difference is less than or equal to the preset preparation threshold, it is determined to prepare for landing. At this time, the target inspection equipment switches to the end vision guidance mode and completes dynamic and precise landing by recognizing the QR code / ArUco code on the corresponding supply equipment. When the difference is greater than the preset preparation threshold, a corresponding first action command is generated based on the target object and the corresponding local observation vector, and the aircraft continues to fly towards the spatiotemporal intersection point.

[0046] Reference Figure 2 Based on a supply device, an example is given to illustrate a collaborative inspection method for distress messages sent by inspection devices. In response to distress messages broadcast by equipment awaiting resupply and inspection, each mobile resupply equipment may independently determine whether to respond, or an external control device may determine which mobile resupply equipment will respond to the distress message and issue a response command. The innovation of this application lies in the fact that the mobile supply equipment responding to distress messages actively greets the equipment to be supplied and inspected. Those skilled in the art can make a response judgment based on actual needs using existing technologies, such as directly selecting the mobile supply equipment with the greatest appeal that meets the supply requirements as the target inspection equipment. This specification does not limit it in detail.

[0047] As one possible implementation, if the current mobile supply equipment is the target mobile supply equipment to be supplied and inspected, the following steps are performed: S500: Receive the local observation vector corresponding to the equipment to be resupplyed and inspected, and the spatiotemporal attraction of the current mobile resupply equipment to the equipment to be resupplyed and inspected. The local observation vector is used to indicate the three-dimensional coordinates, velocity vector, current remaining power, and local wake cost of the target inspection equipment. The local wake cost is used to indicate the distribution of flight energy consumption penalty of the target inspection equipment in the current area. The target spatiotemporal attraction is the attention weight generated by the attention mechanism of the equipment to be resupplyed and inspected, based on the local observation vector and the target feature vector corresponding to the current mobile resupply equipment. The target feature vector is used to indicate the three-dimensional coordinates, type code and real-time status of the current mobile resupply equipment. The current mobile supply equipment can obtain its local observation vector and the corresponding spatiotemporal attraction of the target from the distress message broadcast by the supply inspection equipment.

[0048] S600, Obtain the corresponding spatiotemporal intersection point; The spatiotemporal intersection point is used to indicate the time and coordinates at which the current mobile supply equipment and the inspection equipment meet. As one possible implementation, an external control device sends instructions containing spatiotemporal intersection points to the corresponding resupply inspection device and mobile inspection device. For example, the current mobile supply device receives a response instruction from the external control device. The response instruction is used to indicate the resupply inspection device and the corresponding spatiotemporal intersection point, so that the current mobile supply device can obtain the local observation vector, the target spatiotemporal attraction force, and the spatiotemporal intersection point.

[0049] As another possible implementation method, the current mobile supply equipment calculates the shortest encounter time and corresponding rendezvous coordinates based on its three-dimensional coordinates and maximum response speed, as well as the three-dimensional coordinates, maximum return speed, and local wake cost of the equipment to be supplied and inspected, to obtain the corresponding spatiotemporal rendezvous point, and sends the spatiotemporal rendezvous point to the corresponding supply equipment to be inspected.

[0050] Note that, given the above parameters, calculating the shortest meeting time is existing technology, so this specification will not elaborate on it.

[0051] In this embodiment, the shortest encounter time that satisfies the dynamic constraints is solved by an external control device or the current mobile device based on the corresponding mobile supply device's three-dimensional coordinates and maximum response speed, as well as the three-dimensional coordinates, maximum return speed, and local wake cost of the equipment to be supplied and inspected. The dynamic constraints include: ; ; in: T dock Indicates the shortest meeting time; (X) c Y c () represents the intersection coordinates corresponding to the shortest meeting time; (X) u0 Y u0 () represents the three-dimensional coordinates of the equipment to be resupplyed and inspected; (X) s0 Y s0 () represents the three-dimensional coordinates of the current mobile supply equipment; V uav This indicates the maximum return speed of the equipment to be resupplyed and inspected; C wake This represents the local wake cost corresponding to the equipment to be resupplyed and inspected; It is the attenuation factor; V usv This indicates the maximum response speed of the current mobile supply equipment; T0 represents the current time.

[0052] S700: Based on the local observation vector, the target spatiotemporal attraction, the spatiotemporal intersection point, and the three-dimensional coordinates and speed corresponding to the current mobile supply equipment, generate a second action command; The second action command is used to indicate the current mobile supply equipment's rendezvous course and speed, guiding the current mobile supply equipment to actively head to the spatiotemporal rendezvous point to meet the equipment awaiting supply and inspection, thereby achieving dynamic prediction and response in space.

[0053] In this embodiment, the pre-trained supply-side Actor policy network outputs a corresponding second action command based on the local observation vector, the target spatiotemporal attraction, the spatiotemporal intersection point, and the three-dimensional coordinates and speed corresponding to the current mobile supply device. Calculate the contact distance corresponding to the second action command; Based on the response distance, a corresponding positive joint reward is generated to drive the supply-side Actor strategy network to actively respond to the mobile supply device.

[0054] In this embodiment, the positive joint reward r dock The calculation formula is , where d t Let t be the distance between the two parties at time t. If the current mobile supply equipment waits in place, its reward value is 0. However, if its supply-side Actor policy network outputs an action angle instruction to move towards the spatiotemporal intersection point (three-dimensional coordinates), this reward function will give it a large positive joint reward. The design of this reward function can drive the supply equipment-side Actor policy network to make the mobile supply equipment break through the standby zone patrol and learn to actively maneuver in order to shorten the rescue time.

[0055] To enable those skilled in the art to clearly understand the technical solutions disclosed in this application, a detailed description is given using a scheme for coordinated inspection of offshore wind farms by drones and unmanned surface vessels as an example. The workflow for this case is as follows: 100. Real-time updates of the three-dimensional wake cost field corresponding to the target wind field environment; In complex, highly turbulent environments such as offshore wind farms, traditional UAV path planning typically treats wind turbine towers and blades as static geometric obstacles for obstacle avoidance, neglecting the wake vortices generated by aerodynamics behind the turbine rotor. This invention introduces a physics-based aerodynamic model, transforming it into a dynamic spatial cost matrix that can be directly read by reinforcement learning algorithms, thereby achieving a technological leap from geometric obstacle avoidance to energy efficiency obstacle avoidance.

[0056] In this case, the detailed implementation process of accurately modeling the three-dimensional wake cost field and integrating it with environmental dynamics is as follows: 110. Construct a three-dimensional spatial mesh set corresponding to the target wind field environment; In this case, the three-dimensional airspace of the offshore wind field to be inspected is discretized with a preset resolution (e.g., Δx=Δy=Δz=5m) to construct a three-dimensional spatial grid set G.

[0057] For any independent grid point g(x,y,z) in the grid set, initialize its environment attribute vector Eg=[u base ,θ base In this case, u base θ is the free-flow reference wind speed transmitted in real time by the USV weather sensor. base Used as the baseline wind direction.

[0058] 120. Obtain the actual wind speed corresponding to each grid point; The actual wind speed can be the measured wind speed obtained in real time based on the on-site wind measurement lidar (LiDAR) and fluid dynamics (CFD) software, or it can be the equivalent wind speed calculated based on the wake model. In this case, the equivalent wind speed is calculated using a wake model. The specific implementation scheme is as follows: 121. Traverse the set F of all wind turbine nodes in the wind farm, and for each operating wind turbine... Calculate the aerodynamic disturbance it causes to downstream grid point g, and obtain the actual wind speed corresponding to grid point g: 1) Coordinate system transformation: Project the global coordinates of grid point g onto the coordinate system of wind turbine f. k In a local wake coordinate system with the hub center as the origin and the reference wind direction as the X-axis, the downstream distance x and the radial distance r are obtained.

[0059] 2) Local wind speed attenuation calculation: If the downstream distance x>0 and the radial distance r is within the wake influence radius R wake =D / 2+Ax (where D is the impeller diameter and A is the wake expansion coefficient), then the fan f in grid point g can be calculated using the wake equation. k The resulting wind speed loss; The formula is as follows: Δu k =u base -u k (g); In the formula: Δu k Indicates fan f k The corresponding wind speed loss; u base This indicates the corresponding free-flow reference wind speed; u k (g) represents the grid point g combined with the fan f. k The wake affects the calculated equivalent wind speed; u k (g) Calculated based on the wake equation, the specific formula is as follows: In the formula: CT is the real-time thrust coefficient of the wind turbine, which is provided synchronously by the wind farm system; D is the impeller diameter; A is the wake expansion coefficient; x represents the downstream distance.

[0060] 121. Turbulence Intensity Energy Fusion: After traversing all wind turbines, for any grid point g, summarize the wake effects of multiple wind turbines and base the results on the wind speed loss Δu generated by each wind turbine. k The equivalent wind speed at that point is obtained by performing turbulence intensity-energy fusion calculation; Based on the wind speed loss Δu corresponding to each wind turbine k Calculate the total wind speed loss Δu at the corresponding grid point. total Specifically: ; Free-flow reference wind speed u based on grid point g base Total wind speed loss Δu total The formula for calculating the actual wind speed ug at the corresponding grid point g is as follows: u g =u base -Δu total .

[0061] 130. Real-time updates to the basic cost field: After completing the global wind speed calculation, the system maps the aerodynamic vector characteristics at grid point g into a single energy cost scalar, reflecting the additional battery energy penalty required for the UAV to stay or travel a unit distance at that grid point.

[0062] This case study uses a speed loss penalty. and posture to maintain punishment By constructing the basic energy consumption cost and determining the basic cost field, the original physical three-dimensional space is transformed into a purely mathematical three-dimensional tensor matrix. Each element in the matrix represents the basic energy consumption cost, forming a corresponding basic cost field Mcost. The basic cost field Mcost is refreshed locally and globally at a fixed frequency (e.g., 1Hz) based on the latest wind direction, wind speed, and turbine yaw angle transmitted back by the USV. The basic cost field is defined on spatial grid nodes and is used to characterize the impact of environmental factors on flight costs, and does not depend on any specific UAV.

[0063] The basic cost field and path avoidance diagram are as follows: Figure 2 As shown, during the path planning process, the basic energy consumption cost of each grid in the current area is corrected according to the direction of the UAV's movement and the local wind field direction, that is, crosswind / headwind penalties are introduced. Thus, the corresponding energy consumption cost scalar is obtained.

[0064] In this embodiment, after the cost matrix Mcost is generated, it is fed into the reinforcement learning model through a dual-channel approach: observation and reward / penalty. 1) State Observation Channel (State Input): The Actor policy network on the inspection equipment side not only receives its own (x,y,z) coordinates, but also performs local slice extraction of Mcost (for example, extracting the 5×5×5 local cost sub-matrix around itself). Combined with the crosswind / headwind penalty, the energy consumption cost scalar of each grid point is generated and concatenated into the input feature vector of the Actor policy network on the inspection equipment side. This gives the network the ability to predict high energy consumption areas ahead.

[0065] 2) Reward Penalty: During the training phase, whenever the UAV performs an action a t A displacement p is generated t The simulator performs line integration on the traversed mesh, extracts the sum of costs, and uses it as a negative reward r. wake The total expected return is directly deducted, thereby driving the policy network to converge to an action policy that bypasses or tangentially crosses the optimal cost grid through gradient descent.

[0066] 200. Pre-train reinforcement learning models; The complex dynamic environment of offshore wind fields (such as wake interference) is mapped to a Markov decision process (MDP) in collaboration with UAV-USV heterogeneous agents, and then solved using a multi-agent proximal policy optimization algorithm with attention mechanism.

[0067] In this case, a dynamic environment biomimetic corresponding to the wind field is constructed during the pre-training process. In this embodiment, an ocean-atmosphere dynamic environment biomimetic is constructed to provide a real-time state machine digital twin mapping of wind / wave / current. In this case, the multi-agent network architecture includes: Inspection-side Actor policy network: Deployed on the UAV, it takes local observation vectors (position, velocity, battery level, local cost matrix slices) as input and outputs continuous action commands (heading angle, acceleration).

[0068] Supply-side Actor strategy network: Deployed on the unmanned surface vessel, it takes into account the global status and supply request signals, and outputs follow-up rendezvous commands (heading, speed).

[0069] Centralized Critic Network: Used only during the training phase, it receives the global state and joint actions, evaluates the value of the current policy to guide policy updates.

[0070] Attention mechanism decoupling module: Employs a multilayer perceptron to calculate spatiotemporal attraction based on the corresponding local observation vectors and target feature vectors.

[0071] In this case, those skilled in the art can perform state steps based on the action sequence output by each Actor network in a physical environment or digital twin simulation environment. The system calculates the joint reward for this action: if the UAV successfully avoids the turbulent area and completes the shooting, or the heterogeneous cluster successfully completes the dynamic rendezvous, a positive reward is given; if the flight energy consumption exceeds the limit, the aircraft crashes, or the rendezvous fails, a negative penalty is given. This is a conventional reinforcement learning reward, which is not described in detail in this case. Those skilled in the art can set it themselves. In addition to the aforementioned rewards, this case study also establishes a negative reward r calculated based on the energy consumption cost scalar corresponding to the grid traversed by the UAV's flight path. wakeBy directly penalizing energy consumption in the design of this negative reward, the network is guided to learn to avoid high-energy-consuming turbulent regions. This case also establishes a positive joint reward r based on the distance reduction when the unmanned surface vessel performs a movement toward the spatiotemporal intersection point (three-dimensional coordinates). dock This is to guide the network to learn how to respond to drones awaiting resupply.

[0072] The generated transition states and reward data are stored in the experience replay pool. The central server uses this data to periodically update the weight parameters of the Critic and Actor networks using the gradient descent principle, and synchronously distributes the latest parameters to each execution end, thus forming a complete data closed loop of "perception-decision-evaluation-evolution".

[0073] Given the known network architecture and the aforementioned negative reward r wake and positive joint reward r dock Given the design, those skilled in the art can train and update the corresponding inspection-side Actor policy network, supply-side Actor policy network, and attention mechanism decoupling module based on existing reinforcement learning techniques. This case will not be described in detail.

[0074] 300. Each UAV independently makes decisions at the edge based on the training of the attention mechanism decoupling module and the inspection-side Actor policy network. In traditional unmanned aerial vehicle (UAV) swarm inspections, tasks are typically assigned using preset routes or heuristic rules (such as mandatory return when battery level drops below 30%). However, offshore wind farms exhibit dynamic, high-energy-consumption wake zones, and fixed thresholds can easily cause UAVs to run out of power and crash into the sea during their return journey due to sudden strong headwinds or high turbulence. To address this, this invention introduces a multi-head attention mechanism from the Transformer architecture. Through continuous feature space mapping, it achieves isomorphism and decoupling between inspection and resupply tasks, thereby transforming discrete rule-based decisions into a differentiable weight shifting process within the neural network. This avoids state jitter based on rule thresholds and outputs a smoother yaw control law.

[0075] The independent decision-making steps for each drone in each decision-making cycle are as follows: 310. Construct its own local observation vector h i .

[0076] h i =Concat(p i ,v i E i C wake ); Where, p i For three-dimensional coordinates, v i E is the velocity vector. iC represents the current remaining battery power. wake This is the corresponding local wake cost.

[0077] 320. Construct the target feature vector z for each candidate target. j .

[0078] Candidate targets include uninspected wind turbines and mobile supply drones: z j =Concat(p j Type j Status j ); Where Typej is a one-hot encoded value used to distinguish whether the target is a wind turbine or an unmanned vessel; Status j This represents the real-time status of the corresponding candidate target.

[0079] 330. Calculation of spatiotemporal attraction; Reference Figure 3 To evaluate the spatiotemporal attractiveness of the UAV's current state to each candidate target, the algorithm decouples the weight matrix W corresponding to the module through an attention mechanism. Q W K W V Project the state into the common hidden space; Based on the local observation vector h i and the corresponding first projection matrix W Q Determine the corresponding query matrix Q i =W Q ·h i The query matrix is ​​used as the corresponding intent vector to indicate the current demand intent of the corresponding target inspection equipment (such as addressing a fan or seeking power). Based on the target feature vector z j and the corresponding second projection matrix W K Determine the corresponding bond matrix K j =W K ·z j The key matrix is ​​used as the corresponding attribute vector to indicate the task attributes of the corresponding candidate target; Based on the target feature vector z j and the corresponding third projection matrix W V Determine the value matrix V of the corresponding candidate target j =W V ·z j .

[0080] Subsequently, computational unmanned aerial vehicles (UAVs) i With candidate targets inner product e i , jThe calculation formula is: (This indicates that...) ; Among them, Q i K represents the intent vector of the corresponding drone; j K represents the attribute vector of the corresponding candidate target; j Indicates the dimension of the attribute vector.

[0081] In this case, the resulting inner product e i , j Normalization is performed using the Softmax function to obtain the assigned weights for all targets, i.e., attention weights, or spatiotemporal attraction. In this case, the weight pointing towards the wind turbine target is defined as the inspection attraction α. i,j The weight pointing to the USV target is the supply attraction β. i,j .

[0082] When the UAV is at a high charge level and there is no extremely strong wake in the vicinity, its eigenvector h i After the query is mapped, it will produce the maximum inner product with the wind turbine target key. At this point, the network output α i,k Dominant (e.g.) Figure 3 In the UAV (1, α=0.85), when the UAV's power decreases, or when the local cost matrix Cwake surges despite having a significant amount of power (detecting high turbulence truncates the homing path), h i A sudden change in the data distribution causes a sharp shift in the query vector, resonating with the key of the USV target. At this point, the network automatically outputs a surge in β weights (as shown in the attached image). Figure 3 UAV 2, β=0.92).

[0083] 340. Determine the target task based on the current remaining power and spatiotemporal attraction; 350. When the target task is inspection: The wind turbines with the greatest spatiotemporal attraction to be inspected will be the target objects; The inspection-side Actor policy network generates corresponding action commands based on the local observation vectors and the three-dimensional coordinates of the wind turbine to be inspected, controlling the UAV's heading and acceleration.

[0084] 370. When the objective is resupply: 371. Broadcasts a distress message to the global system via a low-latency communication link. The distress message includes the corresponding local observation matrix h. i And the attraction of different times and spaces.

[0085] 372. It also receives the spatiotemporal intersection point that responds to its distress message; 373. Based on the difference between the rendezvous time indicated by the spatiotemporal rendezvous point and the current time, a landing determination is made; 374. When landing is determined, switch to visual guidance mode and identify the markings on the USV deck (such as QR codes / ArUco codes) to complete a dynamic and precise landing.

[0086] 375. When it is determined that the drone will not land, the spatiotemporal intersection point is taken as the target object. The Actor strategy network on the inspection side generates corresponding action commands to control the drone's heading and acceleration based on the corresponding local observation vector and the three-dimensional coordinates of the spatiotemporal intersection point.

[0087] 400. Each unmanned surface vessel (USV) independently makes decisions at the edge based on training, decoupling the attention mechanism module and the inspection-side Actor policy network. In traditional swarm operations, the USV (UAV) resupply is typically considered a static landing origin. When the UAV's power is insufficient, it unilaterally performs a return maneuver. However, in windy sea conditions, headwinds or wind turbine wakes can significantly reduce the UAV's effective range, causing it to crash into the sea due to power depletion during its one-way return journey. This invention utilizes a collaborative reward mechanism based on multi-agent reinforcement learning to achieve bidirectional collaborative docking of the USV towards the UAV.

[0088] 410. Receive a response command issued by an external control device, wherein the response command is used to indicate the equipment to be resupplyed and inspected and the corresponding spatiotemporal rendezvous point; 420. Obtain the corresponding local observation vectors and target spatiotemporal attraction based on the distress message broadcast by the equipment to be resupplyed and inspected; 430. The supply-side Actor strategy network outputs corresponding second action commands to control the unmanned surface vessel's heading and acceleration based on the local observation vector, the target's spatiotemporal attraction, the spatiotemporal intersection point, and the three-dimensional coordinates and speed corresponding to the current mobile supply equipment.

[0089] 440. The receiving distance corresponding to the second action command; generate a corresponding positive joint reward based on the receiving distance.

[0090] In this case, the solid lines of the drone and unmanned surface vessel are shown in Figure 4: USV spatial trajectory (lower short line): Although the USV has a slow speed and short spatial displacement, its active yaw action towards the UAV tangential direction saves the UAV a significant amount of travel distance.

[0091] UAV spatial trajectory (upper long line): The UAV rapidly converges towards (Xc,Yc) along the curved trajectory arc that avoids the high resistance region.

[0092] In summary, this case study breaks through the limitations of traditional offshore wind farm cluster inspections, which rely on pre-set static routes and passive one-way return of UAVs. By transforming the three-dimensional aerodynamic wake field into a reinforcement learning cost matrix and combining it with dynamic decoupling through a deep attention mechanism, the system not only endows UAVs with the autonomous risk avoidance capability to anticipate and bypass high-energy-consuming turbulent areas under complex wind and sea conditions, but also features a two-way collaborative rendezvous mechanism where unmanned surface vessels actively leave their standby areas when facing sudden power and weather emergencies. This closed-loop strategy eliminates the risk of UAVs crashing into the sea due to sudden headwinds, enables continuous operation of heterogeneous clusters in extreme dynamic environments, and improves the safety and overall efficiency of intelligent inspection of the entire offshore wind farm.

[0093] This application also proposes a collaborative inspection device based on multi-agent reinforcement learning in a wind field environment, wherein the collaborative inspection device is an inspection device or a mobile supply device. The collaborative inspection equipment is used in the collaborative inspection method based on multi-agent reinforcement learning in the wind field environment described in any of the above embodiments.

[0094] As the device embodiment is basically similar to the method embodiment, the description is relatively simple, and relevant parts can be found in the description of the method embodiment.

[0095] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0096] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, apparatus, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0097] This invention is described with reference to flowchart illustrations and / or block diagrams of the method, terminal device (system), and computer program product according to the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1A device that provides the functions specified in one or more boxes.

[0098] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0099] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal equipment, causing a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0100] It should be noted that: The phrase "an embodiment" or "an embodiment" used in this specification means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the invention. Therefore, the phrase "an embodiment" or "an embodiment" appearing in various places throughout the specification does not necessarily refer to the same embodiment.

[0101] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0102] Furthermore, it should be noted that the shapes and names of the parts and components described in the specific embodiments described in this specification may differ. All equivalent or simple variations made to the structure, features, and principles described in this patent concept are included within the protection scope of this patent. Those skilled in the art to which this invention pertains may make various modifications or additions to the described specific embodiments or use similar methods to replace them, as long as they do not depart from the structure of this invention or exceed the scope defined in these claims, they should all fall within the protection scope of this invention.

Claims

1. A collaborative inspection method based on multi-agent reinforcement learning in a wind farm environment, used for inspecting equipment, characterized in that, Includes the following steps: In each decision cycle, the target inspection equipment takes each mobile supply device and / or each wind turbine to be inspected in the inspection area as candidate targets, calculates the spatiotemporal attraction between the target inspection equipment and each candidate target, and determines the target task of the target inspection equipment based on the current remaining power of the target inspection equipment and the obtained spatiotemporal attraction. The target task is inspection or supply. in: The local observation vector is constructed based on the three-dimensional coordinates, velocity vector, current remaining power, and local wake cost of the target inspection equipment. The local wake cost is used to indicate the distribution of flight energy consumption penalty of the target inspection equipment in the current area. Based on the three-dimensional coordinates, type encoding, and real-time status of the candidate targets, a target feature vector corresponding to each candidate target is constructed. An attention mechanism is employed to determine the attention weights between the target inspection equipment and the corresponding candidate targets based on the local observation vector and the target feature vector, thereby obtaining the corresponding spatiotemporal attraction.

2. The collaborative inspection method based on multi-agent reinforcement learning in a wind field environment according to claim 1, characterized in that: If the remaining power is less than the preset replenishment threshold, the target task is directly determined to be replenishment. Each mobile replenishment device is taken as a candidate target. Based on the local observation vector of the target inspection device and the spatiotemporal attraction between the target inspection device and each candidate target, a distress message is generated and broadcast. If the current remaining power is greater than or equal to the preset warning threshold, the target task is directly determined to be inspection, and each wind turbine to be inspected is taken as a candidate target. The target object to be inspected is determined based on the spatiotemporal attraction between the target inspection equipment and each candidate target; the warning threshold is greater than the replenishment threshold. If the current remaining power is greater than or equal to the replenishment threshold and less than the warning threshold, then each mobile replenishment device and each wind turbine to be inspected will be regarded as a candidate target, and the target task will be determined based on the spatiotemporal attraction between the target inspection device and each candidate target.

3. The collaborative inspection method based on multi-agent reinforcement learning in a wind field environment according to claim 2, characterized in that, When the current remaining battery power is greater than or equal to the replenishment threshold and less than the warning threshold: The maximum inspection attraction is determined based on the spatiotemporal attraction of the target inspection equipment on each wind turbine to be inspected. The spatiotemporal attraction of the target inspection equipment to the mobile supply equipment is taken as the corresponding supply attraction. When the ratio of the supply attraction to the maximum inspection attraction exceeds a preset attention threshold, the target task is set to supply; otherwise, the target task is set to inspection. When the target task is inspection, the wind turbine to be inspected is based on the wind turbine with the maximum inspection attraction. When the objective is resupply: A distress signal is generated and broadcast globally. The distress signal includes the local observation vector corresponding to the target inspection equipment and the spatiotemporal attraction between the target inspection equipment and each mobile supply equipment. Receive spatiotemporal intersection points sent by external control devices or target mobile supply devices, and use them as target objects. The spatiotemporal intersection points are used to indicate the time and coordinates of the intersection with the target mobile supply device.

4. The collaborative inspection method based on multi-agent reinforcement learning in a wind field environment according to claim 3, characterized in that: The pre-trained inspection-side Actor policy network outputs corresponding first action commands based on the local observation vectors and target objects. These commands instruct the target inspection equipment on its three-dimensional flight heading angle and acceleration, guiding it to fly out of the current area with the goal of minimizing energy consumption when reaching the target object.

5. A collaborative inspection method based on multi-agent reinforcement learning in a wind field environment according to any one of claims 1-4, characterized in that: An attention mechanism is employed to determine the intent vector of the target inspection device based on the local observation vector, determine the attribute vector corresponding to the candidate target based on each target feature vector, and determine the attention weight between the target inspection device and the corresponding candidate target based on the inner product of the intent vector and the attribute vector, thereby obtaining the corresponding spatiotemporal attraction.

6. The collaborative inspection method based on multi-agent reinforcement learning in a wind field environment according to claim 5, characterized in that: The corresponding query matrix is ​​determined based on the local observation vector and the corresponding first projection matrix. The query matrix is ​​used as the corresponding intent vector to indicate the current demand intent of the corresponding target inspection equipment. The corresponding key matrix is ​​determined based on the target feature vector and the corresponding second projection matrix. The key matrix is ​​used as the corresponding attribute vector to indicate the task attributes of the corresponding candidate target. The value matrix of the corresponding candidate target is determined based on the target feature vector and the corresponding third projection matrix; The first projection matrix, the second projection matrix, and the third projection matrix are obtained through pre-training.

7. A collaborative inspection method based on multi-agent reinforcement learning in a wind field environment according to any one of claims 1-4, characterized in that: Local wake cost includes the energy cost scalar corresponding to each grid in the current region, wherein the current region is determined based on the preset region range and the three-dimensional coordinates of the target inspection equipment; The energy consumption cost scalar is an energy consumption penalty obtained by weighted calculation based on speed loss penalty, attitude maintenance penalty, and crosswind / headwind penalty; The velocity loss penalty is calculated based on the free-flow reference wind speed and the actual wind speed in the corresponding grid, resulting in the corresponding aerodynamic lift loss. The attitude maintenance penalty is calculated based on the turbulence intensity in the corresponding grid. The crosswind / headwind penalty is calculated from the relative wind direction of the wind direction and the corresponding heading angle in the corresponding grid.

8. A collaborative inspection method based on multi-agent reinforcement learning in a wind field environment according to claim 7, characterized in that, The energy cost scalar C corresponding to grid g wake The formula for calculating (g) is: ； in: |u base -u g |As a penalty for speed loss, u base U represents the free-flow reference wind speed. g This represents the actual wind speed corresponding to grid g; This is the first weighting coefficient; This represents the high-frequency wind disturbance corresponding to grid g, which serves as an attitude maintenance penalty; This is the second weighting coefficient; Crosswind / headwind penalty, indicating the heading angle of the target inspection equipment. Wind direction relative to grid g The corresponding relative wind direction function; This is the third weighting coefficient.

9. A collaborative inspection method based on multi-agent reinforcement learning in a wind field environment according to any one of claims 1-4, characterized in that: Wind field environments include offshore wind fields and onshore wind fields; The target inspection device is a drone; The mobile supply equipment includes unmanned boats, unmanned vehicles, and robots.

10. A collaborative inspection method based on multi-agent reinforcement learning in a wind field environment, used on the resupply side, characterized in that, If the current mobile supply equipment is the target mobile supply equipment to be supplied and inspected, perform the following steps: The system receives the local observation vector corresponding to the equipment to be resupplyed and inspected, as well as the target spatiotemporal attraction of the current mobile resupply equipment to the equipment to be resupplyed and inspected. The local observation vector indicates the three-dimensional coordinates, velocity vector, current remaining power, and local wake cost of the target equipment. The local wake cost indicates the flight energy consumption penalty distribution of the target equipment in the current area. The target spatiotemporal attraction is an attention weight generated by the equipment to be resupplyed and inspected using an attention mechanism, based on the local observation vector and the target feature vector corresponding to the current mobile resupply equipment. The target feature vector indicates the three-dimensional coordinates, type encoding, and real-time status of the current mobile resupply equipment. Obtain the corresponding spatiotemporal intersection point, which is used to indicate the time and coordinates of the intersection between the current mobile supply equipment and the inspection equipment; Based on the local observation vector, the target spatiotemporal attraction, the spatiotemporal intersection point, and the three-dimensional coordinates and speed corresponding to the current mobile supply equipment, a second action command is generated. The second action command is used to indicate the follow-up intersection course and speed of the current mobile supply equipment, and guide the current mobile supply equipment to the spatiotemporal intersection point to meet the equipment to be supplied and inspected.

11. The collaborative inspection method based on multi-agent reinforcement learning in a wind field environment according to claim 10, characterized in that: The mobile supply equipment currently receives a response command from an external control device. This response command is used to indicate the equipment to be supplied and inspected, and the corresponding spatiotemporal rendezvous point. or, The current mobile supply equipment, based on its three-dimensional coordinates and maximum response speed, as well as the three-dimensional coordinates, maximum return speed, and local wake cost of the equipment to be supplied and inspected, solves the shortest encounter time and the corresponding rendezvous coordinates to obtain the corresponding spatiotemporal rendezvous point.

12. The collaborative inspection method based on multi-agent reinforcement learning in a wind field environment according to claim 11, characterized in that, Find the shortest meeting time that satisfies the dynamic constraints; The dynamic constraints include: ； ； in: T dock Indicates the shortest meeting time; (X) c Y c () represents the intersection coordinates corresponding to the shortest meeting time; (X) u0 Y u0 () represents the three-dimensional coordinates of the equipment to be resupplyed and inspected; (X) s0 Y s0 () represents the three-dimensional coordinates of the current mobile supply equipment; V uav This indicates the maximum return speed of the equipment to be resupplyed and inspected; C wake This represents the local wake cost corresponding to the equipment to be resupplyed and inspected; It is the attenuation factor; V usv This indicates the maximum response speed of the current mobile supply equipment; T0 represents the current time.

13. The collaborative inspection method based on multi-agent reinforcement learning in a wind field environment according to claim 10, characterized in that: The pre-trained supply-side Actor policy network outputs a corresponding second action command based on the local observation vector, the target spatiotemporal attraction, the spatiotemporal intersection point, and the three-dimensional coordinates and speed corresponding to the current mobile supply equipment. Calculate the contact distance corresponding to the second action command; Based on the response distance, a corresponding positive joint reward is generated to drive the supply-side Actor strategy network to actively respond to the mobile supply device.

14. A collaborative inspection device based on multi-agent reinforcement learning in a wind field environment, characterized in that, The collaborative inspection equipment is either an inspection device or a mobile supply device; The inspection equipment is used to execute the collaborative inspection method based on multi-agent reinforcement learning in a wind field environment as described in any one of claims 1 to 9; The mobile supply device is used to execute the collaborative inspection method based on multi-agent reinforcement learning in a wind field environment as described in any one of claims 10 to 13.

Images