Air-ground cluster coordination method based on visual target behavior intention prediction

Abstract

The present application relates to a kind of air-ground cluster cooperation method based on visual target behavior intention prediction, belong to cluster air-ground cooperation field, solve the problem of low efficiency of unmanned aerial vehicle and unmanned vehicle cooperation mode in air-ground cooperation method Artificial design difficulty. Including obtaining the experience data set of unmanned aerial vehicle and unmanned vehicle;Experience data is randomly selected from experience data set, and the Q value of unmanned aerial vehicle and unmanned vehicle is obtained by inputting experience data into double value network, and the target value of unmanned aerial vehicle and unmanned vehicle is calculated based on the Q value of unmanned aerial vehicle and unmanned vehicle, and the loss function is constructed based on the target value and Q value of unmanned aerial vehicle and unmanned vehicle, and the weight of air-ground cooperation model is updated, to obtain the trained air-ground cooperation model after reaching predetermined training iteration number;Real-time acquisition unmanned aerial vehicle and unmanned vehicle image is preprocessed as state input in real environment Training of good air-ground cooperation model, the action control unmanned aerial vehicle unmanned vehicle of model output unmanned aerial vehicle unmanned vehicle movement. Realize the adaptability of air-ground cooperation model.

Description

Technical Field

[0001] This invention relates to the field of cluster air-ground collaboration technology, and in particular to a visual-based method for predicting target behavior intentions in air-ground cluster collaboration. Background Technology

[0002] Visual prediction is an important means for unmanned platforms to perceive and understand their environment. Compared with other sensors (such as LiDAR, millimeter-wave radar, and infrared cameras), it is inexpensive and provides richer information, making it one of the most widely used sensors on various unmanned platforms. Furthermore, vision-based understanding and prediction of target behavior and intent is unaffected by communication and positioning devices in the scene, forming a more efficient vision-based air-ground collaborative system.

[0003] Existing air-to-ground swarm collaboration methods mainly focus on the situational awareness of the ground by unmanned aerial platforms at a specific altitude, and the fusion of the perception information with that of ground platforms to assist ground platforms in navigation and positioning.

[0004] Existing air-ground collaborative methods mainly utilize the high-angle advantage of aerial drones to obtain information about ground scenes and obstacles, assisting ground-based unmanned vehicles in target localization and path planning. However, there is a lack of research on whether the images captured by drones at their flight altitude and position are optimal for collaboration with unmanned vehicle tasks in different scenarios. In existing air-ground swarm collaborative methods, the flight altitude of aerial drones is manually set, making it more difficult to find the optimal air-ground collaborative method manually. Furthermore, drones need to autonomously determine their flight actions based on the real-time position and scene of the unmanned vehicle. Summary of the Invention

[0005] Based on the above analysis, the embodiments of the present invention aim to provide an air-ground swarm collaboration method based on vision-based target behavior intention prediction, in order to solve the technical problems of low efficiency and difficulty in manual design of existing air-ground collaboration methods, so as to improve the adaptability of air-ground collaboration models in different scenarios and expand their application scope.

[0006] This invention provides a vision-based method for predicting target behavior and intent in air-to-ground cluster collaboration, comprising the following steps:

[0007] Images collected by drones and unmanned vehicles are preprocessed. The preprocessed images are used as state inputs to the air-ground cooperative model to obtain the actions of drones and unmanned vehicles. The actions of drones and unmanned vehicles are applied to drones and unmanned vehicles to obtain the next state and the reward for interaction with the environment. The state, action, reward and next state are used as an experience to obtain an experience dataset.

[0008] Experience data is randomly selected from the experience dataset and input into a dual-value network to obtain the Q-values ​​of the drone and the unmanned vehicle. The target values ​​of the drone and the unmanned vehicle are calculated based on the Q-values ​​of the drone and the unmanned vehicle, respectively. A loss function is constructed based on the target values ​​and Q-values ​​of the drone and the unmanned vehicle to update the weights of the air-ground cooperative model. After reaching a predetermined number of training iterations, the trained air-ground cooperative model is obtained.

[0009] Interacting with the real environment, the preprocessed images of drones and unmanned vehicles collected in real time are used as state inputs to the trained air-ground cooperative model, and the action control of drones and unmanned vehicles output by the air-ground cooperative model controls the movement of drones and unmanned vehicles.

[0010] Furthermore, the images acquired by the UAV include circumferential images perceived by the UAV and ground images;

[0011] The images collected by the unmanned vehicle are circumferential images perceived by the unmanned vehicle.

[0012] Furthermore, the air-ground cooperative model includes a UAV circumferential perception and ground scene prediction model and an unmanned vehicle circumferential perception and target area search model, both of which are constructed using deep reinforcement neural networks.

[0013] The input state of the UAV circumferential perception and ground scene prediction model is the preprocessed circumferential image and ground image perceived by the UAV, and the output is the UAV's action.

[0014] The input state of the unmanned vehicle circumferential perception and target area search model is the preprocessed ground image perceived by the UAV and the circumferential image perceived by the unmanned vehicle, and the output is the action of the unmanned vehicle.

[0015] Furthermore, each experience also includes a flag indicating whether it is the last time step in the current round; multiple experiences are accumulated to form an experience dataset.

[0016] One empirical data point e at each time step i ,as follows:

[0017]

[0018]

[0019] in, This refers to the circumferential and ground-facing states of the UAV. This refers to the circumferential state of the autonomous vehicle. For drone and unmanned vehicle actions, r i UAV r i UGV Rewards for drones and driverless cars, This is the next state after the UAV's circumferential and ground-facing states. This is the next state in the circumferential state of the drone;

[0020] The empirical dataset is E = {e0, e1, ..., e} i ,...};

[0021] A round refers to the period from the start of the first state until the drone or unmanned vehicle collides or the unmanned vehicle reaches the target area. Each round includes multiple time steps, and each time step corresponds to one experience.

[0022] Furthermore, the calculation of the target value of the drone based on the drone's Q-value includes:

[0023] If end is the last time step of the current round, then the drone's target value Y i UAV ,as follows:

[0024] Y i UAV =r i UAV +r i UGV

[0025] Otherwise, if end is not the last time step of the current round, the drone's target value... as follows:

[0026]

[0027]

[0028] Where γ is the discount factor, and Q() is the expected value of taking a specific action in a certain state. To maximize the Q value, Let be the parameters of the UAV circumferential perception and ground scene prediction model at the i-th time step. These are the parameters of the target network in a dual-value network.

[0029] Furthermore, the calculation of the target value of the autonomous vehicle based on the dual-value network includes:

[0030] If end is the last time step of the current round, then the target value Y of the autonomous vehicle is... i UGV ,as follows:

[0031] Y i UGV =r i UGV

[0032] Otherwise, if end is not the last time step of the current round, then the target value Y of the autonomous vehicle is... i UGV ,as follows:

[0033]

[0034]

[0035] in, Let be the parameters of the autonomous vehicle's circumferential perception and target area search model at the i-th time step. These are the parameters of the target network in a dual-value network.

[0036] Furthermore, the loss function of the drone as follows:

[0037]

[0038] Based on this loss function, gradient descent is used to train and update the weights of the UAV circumferential perception and ground scene prediction model.

[0039] Furthermore, the loss function of autonomous vehicles as follows:

[0040]

[0041] Based on this loss function, gradient descent is used to train and update the weights of the autonomous vehicle's circumferential perception and target area search model.

[0042] Furthermore, the drone receives a reward of -1 for a collision and a reward of 0 for not colliding with another drone.

[0043] The reward for the unmanned vehicle to collide is set to -1, and the reward for reaching the target location is set to 1.

[0044] The drone's actions include six actions: moving forward, turning left, turning right, ascending, descending, and hovering.

[0045] The unmanned vehicle's actions include four actions: moving forward, turning left, turning right, and stopping.

[0046] Furthermore, the UAV circumferential perception and ground scene prediction model and the unmanned vehicle circumferential perception and target area search model respectively include:

[0047] Two input layers are used to receive an image and process the image into a tensor;

[0048] Two feature extraction modules are used to extract features from one type of image, respectively;

[0049] The feature fusion layer is used to combine the features of two images to obtain complete features;

[0050] The fully connected layer receives the complete features and outputs actions.

[0051] Compared with the prior art, the present invention can achieve at least one of the following beneficial effects:

[0052] 1. A method for air-ground cluster collaboration based on visual target behavior intention prediction is proposed. It does not rely on the positioning signal in the scene, but achieves air-ground cluster collaboration by perceiving and predicting the behavior intention of other unmanned platforms in the vicinity.

[0053] 2. The drone uses circumferential perception to avoid obstacles between itself and other obstacles;

[0054] 3. By sensing the direction of the ground, drones can understand and predict the behavioral intentions of unmanned vehicles on the ground;

[0055] 4. By combining circumferential perception with ground-oriented perception, and superimposing the rewards for unmanned vehicles with those for drones, a better collaborative approach with ground-based unmanned vehicles can be achieved, including drone flight position and altitude.

[0056] 5. Ground-based unmanned vehicles use circumferential perception to avoid obstacles between themselves and other obstacles;

[0057] 6. Ground-based unmanned vehicles combine circumferential perception with UAV ground perception to achieve rapid target area search and reach the target area.

[0058] In this invention, the above-described technical solutions can be combined with each other to achieve more preferred combinations. Other features and advantages of this invention will be set forth in the following description, and some advantages may become apparent from the description or be learned by practicing the invention. The objects and other advantages of this invention can be realized and obtained from what is particularly pointed out in the description and drawings. Attached Figure Description

[0059] The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Throughout the drawings, the same reference numerals denote the same parts.

[0060] Figure 1 The flowchart shows the air-to-ground cluster collaboration method based on visual target behavior intention prediction.

[0061] Figure 2 A schematic diagram of a deep augmented neural network used for UAV circumferential perception and ground scene prediction models and unmanned vehicle circumferential perception and target area search models.

[0062] Figure 3This is a schematic diagram illustrating the interaction between drones and unmanned vehicles and their environment. Detailed Implementation

[0063] Preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings, which form part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not intended to limit the scope of the present invention.

[0064] To improve the adaptability and optimal collaborative efficiency of unmanned platforms (including aerial drones and ground unmanned vehicles) in different scenarios during air-ground collaboration, a vision-based target behavior intention prediction air-ground swarm collaboration method is proposed. By perceiving and recognizing the current scenario, the flight altitude and position of aerial drones are obtained based on the behavioral intentions of other unmanned platforms within the perception range, effectively improving collaboration efficiency.

[0065] This is achieved primarily through two aspects: First, during mission execution, the drone can autonomously adjust its flight altitude and position based on the perceived location of the drone and the state of the environment. Second, the drone can combine its own circumferential images with the ground images collected by the drone to achieve autonomous maneuvering to the target area with the assistance of an aerial perspective.

[0066] This invention provides an air-ground swarm collaboration method based on visual target behavior intention prediction. It mainly utilizes end-to-end multi-agent deep reinforcement learning technology to predict the behavior intention of ground unmanned vehicles by using drones in a collision-free environment. Combined with the unmanned vehicle's reward, the drone's actions are optimized. The unmanned vehicle uses ground-view images collected by the drone in the ground direction and its own circumferential images to achieve autonomous maneuvering with aerial perspective assistance and reach the target area.

[0067] like Figure 1 As shown in the figure, a specific embodiment of the present invention, an air-to-ground cluster collaboration method based on visual target behavior intention prediction, includes the following steps:

[0068] Step S1: Preprocess the images collected by the UAV and the unmanned vehicle, use the preprocessed images as the state input to the air-ground cooperative model to obtain the actions of the UAV and the unmanned vehicle, apply the actions of the UAV and the unmanned vehicle to obtain the next state and the reward for interacting with the environment, and use the state, action, reward and next state as an experience to obtain an experience dataset.

[0069] Step S2: Randomly select empirical data from the empirical dataset, input the empirical data into the dual value network to obtain the Q values ​​of the drone and the unmanned vehicle, and calculate the target values ​​of the drone and the unmanned vehicle based on the Q values ​​of the drone and the unmanned vehicle respectively. Based on the target values ​​and Q values ​​of the drone and the unmanned vehicle, construct a loss function to update the weights of the air-ground cooperative model. After reaching a predetermined number of training iterations, the trained air-ground cooperative model is obtained.

[0070] Step S3: Interact with the real environment by preprocessing the real-time collected images of the drone and unmanned vehicle and using them as state inputs to the trained air-ground cooperative model. The air-ground cooperative model outputs the motion control of the drone and unmanned vehicle.

[0071] Step S1 includes steps S11-S14, specifically.

[0072] Step S11: Preprocess the images collected by the drone and unmanned vehicle.

[0073] The images acquired by the UAV include circumferential images perceived by the UAV and ground images;

[0074] The images collected by the unmanned vehicle are circumferential images perceived by the unmanned vehicle.

[0075] Both drones and unmanned vehicles acquire RGB images through their onboard cameras, including circumferential and ground-view images perceived by the drones and circumferential images perceived by the unmanned vehicles.

[0076] The circumferential and ground images perceived by the drone are converted to grayscale, and the size of the grayscale images is adjusted to 64×64 pixels.

[0077] The circumferential images perceived by the autonomous vehicle are converted to grayscale, and the size of the grayscale images is adjusted to 64×64 pixels.

[0078] In this regard, regardless of the original size of the circumferential image and the ground image perceived by the drone, both are adjusted to 64×64 pixels. Similarly, regardless of the original size of the circumferential image perceived by the autonomous vehicle, its size is also adjusted to 64×64 pixels.

[0079] The air-ground collaborative model includes a UAV circumferential perception and ground scene prediction model and an unmanned vehicle circumferential perception and target area search model, both of which are constructed using deep reinforcement neural networks.

[0080] The input state of the UAV circumferential perception and ground scene prediction model is the preprocessed circumferential image and ground image perceived by the UAV, and the output is the UAV's action.

[0081] The input state of the unmanned vehicle circumferential perception and target area search model is the preprocessed ground image perceived by the UAV and the circumferential image perceived by the unmanned vehicle, and the output is the action of the unmanned vehicle.

[0082] The preprocessed circumferential image perceived by the UAV serves as the circumferential state of the UAV.

[0083] The pre-processed ground images perceived by the UAV serve as the UAV's ground-view status.

[0084] The preprocessed circumferential image perceived by the autonomous vehicle serves as the circumferential state of the autonomous vehicle.

[0085] A method for air-to-ground cluster collaboration based on visual target behavior intention prediction (corresponding to) Figure 1 The air-ground coordination model in the context of air-ground communication, specifically, such as... Figure 2 As shown;

[0086] The UAV circumferential perception and ground scene prediction model and the UAV circumferential perception and target area search model respectively include:

[0087] Two input layers are used to receive an image and process it into a tensor;

[0088] Two feature extraction modules are used to extract features from one type of image, respectively;

[0089] The feature fusion layer is used to combine the features of two images to obtain complete features;

[0090] The fully connected layer receives the complete features and outputs actions.

[0091] Specifically as follows:

[0092] In the UAV circumferential perception and ground scene prediction model:

[0093] (1) Two input layers: receive the preprocessed UAV-sensed circumferential image and ground image respectively, process them into two 64×64×4 tensors, each tensor is 64×64 in size, where 4 indicates that 4 consecutive frames of grayscale images of size 64×64 are used as input.

[0094] The model includes two feature extraction modules, each consisting of three convolutional layers, used to extract the UAV's circumferential state features and the UAV's ground-facing state features, respectively.

[0095] (2) First convolutional layer: 64 convolutional kernels of size 5×5 are used. The number of convolutional kernels represents the number of features learned by this convolutional layer.

[0096] The activation function is ReLU (The Rectified Linear Unit).

[0097] (3) Second and third convolutional layers: The second convolutional layer uses 128 convolutional kernels of size 3×3, and the third convolutional layer uses 256 convolutional kernels of size 3×3;

[0098] The activation function remains ReLU.

[0099] (4) Feature fusion layer: The features from the first and second feature extraction modules are combined to obtain complete features.

[0100] (5) Fully connected layer: including the first fully connected layer and the second fully connected layer. The sizes of the two fully connected layers are 512 and 6 respectively. The function of these two fully connected layers is to map the complete features of the output of the feature fusion layer to the final action output of the UAV.

[0101] The second fully connected layer contains 6 units, each corresponding to one action of the drone, including forward, left turn, right turn, ascent, descent and hover.

[0102] The output of the UAV circumferential perception and ground scene prediction model is the action of the UAV with the largest corresponding unit value.

[0103] In the autonomous vehicle's circumferential perception and target area search model:

[0104] (1) Two input layers: receive the preprocessed ground image perceived by the UAV and the circumferential image perceived by the unmanned vehicle respectively, and process them into two 64×64×4 tensors. Each tensor is 64×64 in size, where 4 indicates that four consecutive frames of grayscale images of size 64×64 are used as input.

[0105] (2) The settings of the three convolutional layers in the neural network of the unmanned vehicle circumferential perception and target area search model are consistent with the settings of the UAV circumferential perception and ground scene prediction model.

[0106] The model includes three convolutional layers in the third and fourth feature extraction modules, which are used to extract the UAV's ground state features and the unmanned vehicle's circumferential state features, respectively.

[0107] (3) Feature fusion layer: The features of the third and fourth feature extraction modules are combined and merged to obtain the complete features.

[0108] (3) Fully connected layer: including the third fully connected layer and the fourth fully connected layer. The sizes of the two fully connected layers are 512 and 4 respectively. The function of these two fully connected layers is to map the complete features output by the feature fusion layer to the final action output of the autonomous vehicle.

[0109] The fourth fully connected layer uses a unit size of 4, with each unit corresponding to one action of the autonomous vehicle, including four actions: moving forward, turning left, turning right, and stopping.

[0110] Step 12: Input the preprocessed UAV-perceived circumferential image and the ground image of 4 consecutive grayscale frames into the UAV circumferential perception and ground scene prediction model to obtain the UAV's actions. Apply the UAV's actions to the UAV to obtain the UAV's next state and the reward for environmental interaction.

[0111] Obtain the state formed by the image data sensed by the UAV at the i-th time step and the j-th time step. The input is fed into the UAV's circumferential perception and ground scene prediction model to obtain the current reward for the UAV. and the actions of drones

[0112] The model's output of the drone's actions is sent to the drone for execution to obtain the state at the next time step.

[0113] in, A ring view is created by stitching together circumferential images acquired from the front, rear, left, and right perspectives of the drone. This is used for obstacle avoidance between the drone and other moving and static obstacles. This represents a ground scene image acquired from a downward view by the drone. It is used, on the one hand, to perceive the behavioral intentions of the unmanned ground vehicle, and on the other hand, for the global perception portion of the unmanned ground vehicle's input. The reward indicates whether a drone has collided with another drone. This is a ring view formed by stitching together the circumferential images acquired from the front, back, left, and right perspectives of the UAV at the next time step. This is a ground scene image obtained from the drone's downward view at the next time step.

[0114] The drone's reward is determined by environmental feedback; a drone receives a reward of -1 for a collision and a reward of 0 for no collision.

[0115] Step 13: Input the four consecutive grayscale images of the ground image perceived by the UAV and the circumferential image perceived by the unmanned vehicle into the circumferential perception and target area search model of the unmanned vehicle, output the action of the unmanned vehicle, apply the action of the unmanned vehicle to the unmanned vehicle, and obtain the next state and reward of the unmanned vehicle.

[0116] Obtain the state composed of the circumferential image data perceived by the ground-based unmanned vehicle at the i-th time step and the j-th time step. The input is fed into the autonomous vehicle's circumferential perception and target area search model, and the output is the autonomous vehicle's actions. Apply the actions of the autonomous vehicle to the autonomous vehicle and obtain the current reward for the autonomous vehicle. and the state at the next time step

[0117] in, This represents a panoramic view stitched together from images acquired from the front, rear, left, and right perspectives of the autonomous vehicle. It serves two purposes: obstacle avoidance between the autonomous vehicle and other moving and static obstacles, and target task point search. The reward for whether the autonomous vehicle has reached the target area or caused a collision is obtained through environmental feedback. The reward for a collision is set to -1, and the reward for reaching the target location is set to 1.

[0118] Step 14: Combine the state, actions, rewards, the state of the UAV and the unmanned vehicle in the next time step predicted by the simulation system, and the end flag indicating whether it is the last time step in the current episode to form an experience. Multiple experiences constitute an experience dataset.

[0119] One empirical data point for each time step is shown below:

[0120]

[0121]

[0122] in, This refers to the circumferential and ground-facing states of the UAV. This refers to the circumferential state of the autonomous vehicle. For drone and unmanned vehicle actions, r i UAV r i UGV Rewards for drones and driverless cars, This is the next state after the UAV's circumferential and ground-facing states. This is the next state in the circumferential state of the drone.

[0123] If it's the last time step, end=1; otherwise, end=0. Each experience is added to the experience dataset E={e0,e1,...,e...} i In the dataset, since the experience of all drones and unmanned vehicles is stored in the experience dataset, it is no longer necessary to use the subscript j to distinguish which drone or unmanned vehicle it is.

[0124] In this invention, a round refers to the period from the start of the first state until the drone or unmanned vehicle collides or the unmanned vehicle reaches the target area. Each round includes multiple time steps, and each time step corresponds to one experience.

[0125] Step S2, including S21-S24, specifically.

[0126] Multiple batches of empirical data are randomly selected from the empirical dataset and input into a dual-value network to obtain the Q-values ​​of the UAV and the unmanned vehicle. Based on the Q-values ​​of the UAV and the unmanned vehicle, the target values ​​of the UAV and the unmanned vehicle are calculated respectively. A loss function is constructed based on the target values ​​and Q-values ​​of the UAV and the unmanned vehicle. Gradient descent iterative training is performed using a deep reinforcement neural network to update the weights of the air-ground cooperative model. After reaching a predetermined number of training iterations, a trained air-ground cooperative model is obtained. This model includes a UAV circumferential perception and ground scene prediction model and an unmanned vehicle circumferential perception and target area search model. The interaction process between the UAV, the unmanned vehicle, and the environment is as follows: Figure 3 As shown;

[0127] In the training process of UAV circumferential perception and ground scene prediction models, randomly selected empirical data are used. Training was conducted to enable drones to cooperate with unmanned vehicles in maneuvering by combining rewards with those for drones and unmanned vehicles.

[0128] Step S21: The empirical data is input into a dual-value network to obtain the Q values ​​of the drone and the unmanned vehicle;

[0129] Randomly selected empirical data is input into a Double Value Network (DDQN) to obtain the Q-values ​​of drones and unmanned vehicles. The Double Value Network is used to solve the problem of overestimation of the target values ​​of drones and unmanned vehicles.

[0130] Step S22: Calculate the target value of the UAV based on the UAV's Q value.

[0131] The UAV circumferential perception and ground scene prediction model and the unmanned vehicle circumferential perception and target area search model are built based on deep reinforcement neural networks. During the training of the deep reinforcement neural network, the target value of the UAV is calculated based on empirical data randomly selected from the self-empirical dataset.

[0132] The calculation of the target value of the drone based on the drone's Q-value includes:

[0133] If end is the last time step of the current round, then the drone's target value Y i UAV ,as follows:

[0134] Y i UAV =r i UAV +r i UGV Formula (2)

[0135] Otherwise, if end is not the last time step of the current round, the drone's target value Y is... i UAV ,as follows:

[0136]

[0137]

[0138] Where γ is the discount factor, and Q() is the Q value for taking a specific action in a certain state. The action that maximizes Q value, Let be the parameters of the UAV circumferential perception and ground scene prediction neural network at the i-th time step. For the parameters of the target network in a dual-value network, at regular time intervals, the steps are from... Obtained by copying.

[0139] For example, the discount factor is preset to 0.99, and the interval time step is set to 5000 time steps.

[0140] The problem of overestimation of UAV target values ​​is solved by using Double DQN (DDQN). By periodically updating the target network in the Double DQN, the stability of the UAV's circumferential perception and ground scene prediction neural network during the training process is improved.

[0141] Step S23: Based on the UAV loss function, use gradient descent to train and update the weights of the UAV circumferential perception and ground scene prediction model.

[0142] Neural networks use drone target values The error between the Q-value and the target Q-value is used to perform gradient descent training to update the weights. The drone loss function is as follows:

[0143]

[0144] In the training process of the autonomous vehicle's circumferential perception and target area search model, randomly selected empirical data are used. Training was conducted to combine the drone's perception of the ground with the unmanned vehicle's perception, enabling a joint feedback mechanism where rewards are given to both the drone and the unmanned vehicle after the unmanned vehicle performs actions under joint air-ground perception.

[0145] Step S24: Calculate the target value of the unmanned vehicle based on the Q value of the unmanned vehicle.

[0146] During the training of a deep reinforcement neural network, if `end` is the last time step of the current round, then the target value Y of the autonomous vehicle is... i UGV ,as follows:

[0147] Yi UGV =r i UGV Formula (5)

[0148] If end is not the last time step of the current round, then the target value Y of the autonomous vehicle is... i UGV ,as follows:

[0149]

[0150]

[0151] in, Let be the weights of the neural network for the autonomous vehicle's circumferential perception and target region search at the i-th time step. For the parameters of the target network in a dual-value network, at regular time intervals from θ i UGV Obtained by copying.

[0152] For example, the interval time step is set to 5000 time steps.

[0153] Step S25: Based on the unmanned vehicle loss function, use gradient descent to train and update the weights of the unmanned vehicle circumferential perception and target area search model.

[0154] The autonomous vehicle's circumferential perception and target region search neural network also uses gradient descent to train and update weights based on the error between the autonomous vehicle's target value and Q-value. The autonomous vehicle's loss function is as follows:

[0155]

[0156] Step S3, specifically.

[0157] Interacting with the real environment, the preprocessed images of drones and unmanned vehicles collected in real time are used as state inputs to the trained air-ground cooperative model, and the action control of drones and unmanned vehicles output by the air-ground cooperative model controls the movement of drones and unmanned vehicles.

[0158] Drone Action Execution: Actions obtained from the drone's circumferential perception and ground scene prediction model are sent to the actual drone. The drone system executes corresponding flight operations based on the actions output by the model, including forward movement, left turn, right turn, ascent, descent, and hovering.

[0159] Autonomous vehicle action execution: Actions obtained from the autonomous vehicle's circumferential perception and target area search model are sent to the actual autonomous vehicle. The autonomous vehicle system executes corresponding movement operations based on the actions output by the model, including forward movement, left turn, right turn, and stop.

[0160] Looping execution: The two steps above are executed cyclically, meaning the model periodically outputs actions, and then the drone and autonomous vehicle systems execute these actions. This process continues until the autonomous vehicle reaches the predetermined target area.

[0161] Collision detection: At each time step, the system performs collision detection to ensure that the drone and unmanned vehicle do not collide with obstacles or other obstructions. If a collision is detected, the unmanned vehicle stops moving.

[0162] Target area detection: At each time step, it is detected whether the unmanned vehicle has reached the target area. If the target area is detected, the drone stops flying and the unmanned vehicle stops moving.

[0163] In summary, the actions output by the air-ground cooperative model are applied to real drones and unmanned vehicles, and the drones and unmanned vehicles are controlled to reach the target area by continuously executing actions.

[0164] Compared with the prior art, the present invention can achieve at least one of the following beneficial effects:

[0165] 1. A method for air-ground cluster collaboration based on visual target behavior intention prediction is proposed. It does not rely on the positioning signal in the scene, but achieves air-ground cluster collaboration by perceiving and predicting the behavior intention of other unmanned platforms in the vicinity.

[0166] 2. The drone uses circumferential perception to avoid obstacles between itself and other obstacles;

[0167] 3. By sensing the direction of the ground, drones can understand and predict the behavioral intentions of unmanned vehicles on the ground;

[0168] 4. By combining circumferential perception with ground-oriented perception, and superimposing the rewards for unmanned vehicles with those for drones, a better collaborative approach with ground-based unmanned vehicles can be achieved, including drone flight position and altitude.

[0169] 5. Ground-based unmanned vehicles use circumferential perception to avoid obstacles between themselves and other obstacles;

[0170] 6. Ground-based unmanned vehicles combine circumferential perception with UAV ground perception to achieve rapid target area search and reach the target area.

[0171] Those skilled in the art will understand that all or part of the processes of the methods described in the above embodiments can be implemented by a computer program instructing related hardware, and the program can be stored in a computer-readable storage medium. The computer-readable storage medium may be a disk, optical disk, read-only memory, or random access memory, etc.

[0172] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

Claims

1. A visual-based method for predicting target behavior and intent in air-to-ground cluster collaboration, characterized in that, Includes the following steps: The images collected by the drone and the unmanned vehicle are preprocessed; wherein, the images collected by the drone include the circumferential images perceived by the drone and the ground images; the images collected by the unmanned vehicle are the circumferential images perceived by the unmanned vehicle. The preprocessed image is used as the state input to the air-ground cooperative model to obtain the actions of the UAV and the unmanned vehicle. The actions of the UAV and the unmanned vehicle are applied to the UAV and the unmanned vehicle to obtain the next state and the reward for interacting with the environment. The state, action, reward and the next state are used as an experience to obtain an experience dataset. The air-ground cooperative model includes a UAV circumferential perception and ground scene prediction model and an unmanned vehicle circumferential perception and target area search model. The two models are constructed by deep reinforcement neural networks. The input state of the UAV circumferential perception and ground scene prediction model is the preprocessed circumferential image and ground image perceived by the UAV, and the output is the UAV's action. The input state of the unmanned vehicle circumferential perception and target area search model is the preprocessed ground image perceived by the UAV and the circumferential image perceived by the unmanned vehicle, and the output is the action of the unmanned vehicle. Experience data is randomly selected from the experience dataset and input into a dual-value network to obtain the Q-values ​​of the drone and the unmanned vehicle. The target values ​​of the drone and the unmanned vehicle are calculated based on the Q-values ​​of the drone and the unmanned vehicle, respectively. A loss function is constructed based on the target values ​​and Q-values ​​of the drone and the unmanned vehicle to update the weights of the air-ground cooperative model. After reaching a predetermined number of training iterations, the trained air-ground cooperative model is obtained. Interacting with the real environment, the preprocessed images of drones and unmanned vehicles collected in real time are used as state inputs to the trained air-ground cooperative model, and the action control of drones and unmanned vehicles output by the air-ground cooperative model controls the movement of drones and unmanned vehicles.

2. The method according to claim 1, characterized in that, The experience also includes a flag indicating whether it is the last time step in the current round. Multiple experiences accumulate to form an experience dataset; One empirical data point at each time step ,as follows: in, , This refers to the circumferential and ground-facing states of the UAV. This refers to the circumferential state of the autonomous vehicle. , For drone and unmanned vehicle actions, , Rewards for drones and driverless cars, , This is the next state after the UAV's circumferential and ground-facing states. This is the next state in the circumferential state of the drone; The empirical dataset is ; A round refers to the period from the start of the first state until the drone or unmanned vehicle collides or the unmanned vehicle reaches the target area. Each round includes multiple time steps, and each time step corresponds to one experience.

3. The method according to claim 2, characterized in that, The target value of a drone is calculated based on its Q-value, including: if If this is the last time step of the current round, then the drone's target value... ,as follows: Otherwise, if The drone's target value is not the last time step of the current round. ,as follows: in, As a discount factor, The expected value of taking a specific action in a certain state. To maximize the Q value of the drone's actions, For the first Parameters of a time-step UAV circumferential perception and ground scene prediction model. These are the parameters of the target network in a dual-value network.

4. The method according to claim 3, characterized in that, The target value of the autonomous vehicle is calculated based on the dual-value network, including: if If it is the last time step of the current round, then the target value of the autonomous vehicle is... ,as follows: Otherwise, if If it is not the last time step of the current round, then the target value of the autonomous vehicle... ,as follows: in, For the first The parameters of the time-step autonomous vehicle's circumferential perception and target area search model. These are the parameters of the target network in a dual-value network; To maximize the Q value of the autonomous vehicle's actions.

5. The method according to claim 4, characterized in that, Loss function of drones ,as follows: Based on this loss function, gradient descent is used to train and update the weights of the UAV circumferential perception and ground scene prediction model.

6. The method according to claim 5, characterized in that, Loss function of autonomous vehicles ,as follows: Based on this loss function, gradient descent is used to train and update the weights of the autonomous vehicle's circumferential perception and target area search model.

7. The method according to any one of claims 1-6, characterized in that, The drone receives a reward of -1 for colliding with another drone and a reward of 0 for not colliding with another drone. The reward for the unmanned vehicle to collide is set to -1, and the reward for reaching the target location is set to 1. The drone's actions include six actions: moving forward, turning left, turning right, ascending, descending, and hovering. The unmanned vehicle's actions include four actions: moving forward, turning left, turning right, and stopping.

8. The method according to claim 7, characterized in that, Both the UAV circumferential perception and ground scene prediction model and the UAV circumferential perception and target area search model include: Two input layers are used to receive an image and process it into a tensor; Two feature extraction modules are used to extract features from one type of image, respectively; The feature fusion layer is used to combine the features of two images to obtain complete features; The fully connected layer receives the complete features and outputs actions.

Images