Building scene recognition method and linkage system based on multi-unmanned-aerial-vehicle visual collaboration

Abstract

The invention discloses a multi-unmanned aerial vehicle visual collaborative building scene recognition method and linkage system. The invention relates to the technical field of machine vision. And a bidirectional Bi-GRU neural network model is used as a coding layer of the Seq2Seq model. The coding layer is responsible for receiving the feature sequence S and capturing context information in the feature sequence S through forward and backward transmission of bidirectional Bi-GRU; the attention mechanism dynamically pays attention to information at different positions in a coding layer when a decoding layer generates an output sequence S, and outputs an intermediate vector C; the Seq2Seq model naturally supports unequal input and output sequence lengths, so that the technical scheme of the invention can flexibly cope with the challenge of different track lengths in cooperation of multiple unmanned aerial vehicles, and the effect of cooperative planning is improved. In combination with the Bi-GRU recurrent neural network and the attention mechanism, the model can comprehensively consider the flight path of the unmanned aerial vehicle, the building scene characteristics and the external environment change, and a more reasonable and efficient flight path planning scheme is generated.

Description

Technical Field

[0001] The present invention relates to the field of machine vision technology, and specifically to the direction of UAV (unmanned aerial vehicle)-based collaborative building recognition and trajectory planning technology, and more particularly to a multi-unmanned aerial vehicle visual collaborative building scene recognition method and linkage system. Background Art

[0002] The neural network-based trajectory planning technology for multi-UAV collaborative building recognition combines the latest advances in UAV technology, computer vision and deep learning, aiming to improve the efficiency and accuracy of UAVs in building recognition tasks.

[0003] RNN (recurrent neural network) and LSTM (long short-term memory network) are powerful tools for processing sequence data and have important applications in multi-UAV collaborative trajectory planning. For example, the following documents disclose relevant technical solutions:

[0004] [1] Yu Yang, Shi Guowu, Liu Bin, et al. Building extraction from UAV images based on fully convolutional neural network[J]. Water Resources and Hydropower Technology, 2020, 51(7): 31-38.

[0005] [2] Tang Ying, Liu Zhengjun, Yang Yi, et al. Research on neural network building extraction based on feature enhancement and ELU[J]. Journal of Geo-information Science, 2021, 23(4): 692-709.

[0006] [3] Liu Qiang, Jie Jiafen, Chen Jianzhong, et al., Building scene recognition in high-resolution remote sensing images based on convolutional neural networks[J]. Bulletin of Surveying and Mapping, 2021(1):124-128.

[0007] [4] Guo Wen, Zhang Yang. Building extraction from high-resolution satellite images based on attention-enhanced fully convolutional neural network[J]. Remote Sensing of Land and Resources, 2021, 33(2): 100-107.

[0008] The above-mentioned prior art introduces a loop structure through RNN or LSTM, so that the network can process sequence data. At each time step, they receive an input and generate an output, while maintaining an internal state to remember past information. This mechanism enables RNN and LSTM to capture long-term dependencies in sequences, but since RNN and LSTM receive inputs and generate outputs in a fixed manner at each time step, this limits their ability to process sequences of unequal lengths. In multi-UAV collaborative trajectory planning, the flight trajectories of different UAVs have different lengths and are greatly affected by environmental factors. Traditional RNN and LSTM models encounter difficulties in dealing with such problems, resulting in inflexible or inaccurate trajectory planning.

[0009] To this end, the present invention proposes a multi-UAV visual collaborative building scene recognition method and linkage system. Summary of the invention

[0010] In view of this, the present invention hopes to provide a multi-UAV visual collaborative building scene recognition method and linkage system to solve or alleviate the technical problems existing in the prior art, that is, how to optimize the existing neural network model in the multi-UAV visual collaborative building scene recognition scenario to achieve the processing of unequal length sequences to improve the collaborative planning effect and flexibility, and at least provide a beneficial option for this; the technical solution of the present invention is implemented as follows:

[0011] First, the building scene recognition method of multi-UAV visual collaboration:

[0012] 1. Overview:

[0013] The present invention aims to combine the Seq2Seq model with the bidirectional Bi-GRU recurrent neural network model to solve the key technical problems in multi-UAV visual collaborative building scene recognition. By introducing the Seq2Seq model, the scheme can handle the situation where the input and output sequences are of unequal length and adapt to the complex needs of multi-UAV collaborative trajectory planning. At the same time, the use of bidirectional Bi-GRU as the internal neurons of the encoding layer and the decoding layer effectively alleviates the gradient vanishing or exploding problem of the traditional RNN model when processing long sequence data, and improves the model's ability to process long sequence data. In addition, by introducing the attention mechanism, the scheme further enhances the model's attention to key information and improves the accuracy and flexibility of trajectory planning.

[0014] (II) Technical solution:

[0015] To achieve the above technical objectives, flight data is collected from the multi-UAV system. After obtaining the flight trajectory t(i) of the i-th UAV, the following steps are performed;

[0016] 2.1 Step S1, data preprocessing:

[0017] Each flight trajectory t(i) is planned as a feature sequence S; where S = {s1, s2, ..., s T}, where T represents the length of the sequence.

[0018] 2.2 Step S2, activate the encoding layer of the Seq-Bi-GRU neural network model:

[0019] The bidirectional Bi-GRU neural network model is used as the encoding layer of the Seq2Seq model. The encoding layer is responsible for receiving the feature sequence S and capturing the context information in the feature sequence S through the forward and backward passes of the bidirectional Bi-GRU.

[0020] The bidirectional Bi-GRU neural network model includes two transmission directions: forward and backward.

[0021] 2.2.1 Step S200, forward propagation:

[0022] For the forward pass, the hidden state Updated to:

[0023]

[0024] Among them, s t represents the input features at time t, Represents the hidden state at the previous moment.

[0025] 2.2.2 Step S201, backward propagation:

[0026] For the backward pass, the hidden state Updated to:

[0027]

[0028] Among them, s t It also represents the input features at time t, and Represents the hidden state at the next moment.

[0029] 2.2.3 Step S202, output:

[0030] The encoding layer output h at time t t It is composed of the concatenation of the forward and backward hidden states:

[0031] H={h1,h2,...,h T};

[0032]

[0033] Among them, [·;·] represents the concatenation operation of the vectors, and H is a sequence containing the coded information at all times.

[0034] 2.3 Step S3, activate the attention mechanism:

[0035] The attention mechanism dynamically focuses on the information at different positions in the encoding layer when the decoding layer generates the output sequence H, and outputs the intermediate vector C;

[0036] The attention mechanism aims to calculate a weight distribution α t , which reflects the degree of attention paid by the decoding layer to the information at each position of the encoding layer when generating the current output. Specifically, the weight distribution α t Indicates the attention paid to the i-th position information of the coding layer at decoding time t:

[0037]

[0038] Among them, e t,i is an alignment model that evaluates the match between the decoding time t and the i-th position of the encoding layer. It can be calculated by a small neural network:

[0039]

[0040] Among them, v a , W a and U a is a learnable parameter and \tanh(\cdot) is the hyperbolic tangent activation function.

[0041] Once the weight distribution α is calculated t , we can use it to generate the intermediate vector c t , which is the weighted sum of all position information in the encoding layer:

[0042] C={c1,c2,...,c T};

[0043]

[0044] Among them, the tth intermediate vector c t Contains all the important information that the decoding layer cares about when generating the current output.

[0045] Finally, the decoding layer will combine the hidden state s from the previous moment t-1 and the intermediate vector c t , or add other inputs (such as the output of the previous moment) to generate the hidden state s at the current moment t and output.

[0046] 2.4 Step S4, activate the decoding layer of the Seq-Bi-GRU neural network model:

[0047] The intermediate vector C is input into the decoding layer. The decoding layer is based on the fixed-length vector output by the Bi-GRU structure, and gradually generates each element of the output sequence and terminates through the termination condition, and finally outputs the output sequence S' of the planned flight trajectory.

[0048] The decoding layer also uses the Bi-GRU structure and receives the intermediate vector c at each decoding time t t As input. At the same time, the decoding layer also receives the hidden state s of the previous moment t-1 and other inputs (such as the output at the previous moment).

[0049] 2.4.1 Step S400, forward pass:

[0050] The Bi-GRU structure of the decoding layer is responsible for updating the hidden state according to the input information and generating the output at the current moment. Specifically, for the forward pass, the hidden state Updated to:

[0051]

[0052] in, Indicates additional input information.

[0053] 2.4.2 Step S401, backward pass:

[0054] For the backward pass, since the decoding process is usually unidirectional, the hidden state of the backward In fact, this does not apply in standard Seq2Seq decoding. But in the variant of bidirectional decoding, the backward pass can be defined similarly. So here we can focus on the forward pass.

[0055] Therefore, the decoding layer combines the forward hidden state and in the backward decoding variant To generate the output y at the current time t Output y t Through a fully connected layer and a softmax activation function, we get:

[0056]

[0057] Among them, W o and b o are learnable parameters.

[0058] 2.4.3 Step S402, iterative decoding:

[0059] The decoding process is repeated until a certain termination condition is met, such as reaching the preset output sequence length. Finally, the decoding layer will output an output sequence of the planned flight trajectory:

[0060] S'={y1,y2,...y T '};

[0061] Where T' is the length of the output sequence.

[0062] 2.5Seq-Bi-GRU neural network model training method:

[0063] Although the above steps S2 to S4 use the Seq-Bi-GRU recurrent neural network model, the model still needs to be trained before deployment. The training steps include:

[0064] P1. Collect and divide the historical data set D':

[0065] The flight trajectory t(i) of the drone and its building identification label are collected from the historical data to form a dataset D', and then the preprocessed dataset is divided into a training set D train and the test set D test ;

[0066] P2. Set training parameters:

[0067] Including learning rate, batch size, number of iterations, and Adam optimizer.

[0068] P3. Training model:

[0069] Using the training set D train The model is trained and the model parameters are updated through the back propagation algorithm to minimize the mean square error (MSE) between the predicted value and the actual value, and parameter tuning and model structure adjustment are performed as needed.

[0070] P4. Model evaluation and deployment:

[0071] Repeat forward propagation, loss calculation and back propagation, and continuously iterate the training process until the stopping condition is met (the maximum number of iterations is reached or the loss function value is less than the threshold).

[0072] 2.6 Step S5, online adjustment:

[0073] The output sequence S' is decoded to form the flight trajectory t(i)' of each UAV and input into the multi-UAV system to achieve real-time building scene recognition and trajectory planning.

[0074] The decoding function is decode(·), which converts the output sequence S′ into a flight trajectory t(i)′:

[0075] t(i)′=decode(S′);

[0076] The decoding function decode(·) can convert the elements in the output sequence into position control instructions or speed control instructions of the UAV.

[0077] (III) Mechanism for solving technical problems:

[0078] 3.1 Model selection and combination

[0079] (1) Seq2Seq model: This model naturally supports situations where the input and output sequence lengths are unequal, and is very suitable for dealing with the challenge of different trajectory lengths in multi-UAV collaboration. The Seq2Seq model consists of two parts: the encoding layer and the decoding layer. The encoding layer is responsible for encoding the input sequence (the flight trajectory of the UAV) into a vector representation of a fixed length, and the decoding layer gradually generates the output sequence (the planned flight trajectory) based on the vector representation.

[0080] (2) Bidirectional Bi-GRU recurrent neural network: Compared with traditional RNN or LSTM, Bi-GRU effectively alleviates the gradient vanishing or exploding problem of the traditional Seq2Seq model by introducing a gating mechanism (update gate and reset gate), and can better process long sequence data. The bidirectional structure allows the model to consider both the forward and backward information of the sequence, thereby more comprehensively capturing the contextual relationship in the sequence.

[0081] 3.2 Introduction of attention mechanism:

[0082] The attention mechanism is introduced in the Seq2Seq model, so that the decoding layer can dynamically focus on the information at different positions in the encoding layer when generating each element of the output sequence. This mechanism enhances the model's ability to capture key information and improves the accuracy and flexibility of trajectory planning.

[0083] Secondly, the multi-UAV visual collaborative building scene recognition linkage system:

[0084] like Figure 2 As shown, the system includes a processor and a memory connected to the processor, wherein the memory stores program instructions, and when the program instructions are executed by the processor, the processor executes the building scene recognition method as described above, and the processor is connected to:

[0085] (1) A data preprocessing module responsible for converting the raw flight data collected in the multi-UAV system into feature sequences: It uses existing computer vision technology to extract features from high-resolution images to obtain feature representations of building scenes, and standardizes flight trajectories and feature sequences to ensure the consistency and comparability of input data.

[0086] (2) The Seq-Bi-GRU neural network model module combines and implements the Seq2Seq model and the bidirectional Bi-GRU recurrent neural network: The Seq2Seq model naturally supports processing input and output sequences of unequal lengths, which is very suitable for the scenario of multi-UAV collaborative trajectory planning. The bidirectional Bi-GRU, as the internal neuron of the encoding layer and the decoding layer, effectively alleviates the gradient vanishing or exploding problem of the traditional RNN model when processing long sequence data. This module is responsible for receiving the preprocessed feature sequence, and outputting the planned flight trajectory through the processing of the encoding layer and the decoding layer.

[0087] (3) The attention mechanism module allows the decoding layer to dynamically focus on information at different positions in the encoding layer when generating the output sequence: This enhances the model’s ability to capture key information and improves the accuracy and flexibility of trajectory planning. Through the attention mechanism, the model can more accurately focus on feature information that has an important impact on trajectory planning.

[0088] (4) Model training module responsible for training the Seq-Bi-GRU neural network model: Use the preprocessed data set to train the model and adjust the model parameters through the back propagation algorithm to minimize the error between the predicted value and the actual value. During the training process, this module can also adjust the learning rate, batch size, number of iterations and other parameters as needed to improve the performance and stability of the model.

[0089] (5) A deployment module for deploying the model into a multi-UAV system for practical application: For example, a communication module that communicates with the multi-UAV system or a separate UAV flight control system based on the flight trajectory t(i)' of each UAV to achieve real-time trajectory planning for building scenes.

[0090] Compared with the prior art, the present invention has the following beneficial effects:

[0091] 1. Enhance collaborative planning capabilities: The Seq2Seq model naturally supports situations where the input and output sequence lengths are unequal, so that the technical solution of the present invention can flexibly cope with the challenges of different trajectory lengths in multi-UAV collaboration, improving the effect of collaborative planning. Combined with the Bi-GRU recurrent neural network and attention mechanism, the model can comprehensively consider the flight trajectory of the UAV, the characteristics of the building scene, and the changes in the external environment to generate a more reasonable and efficient flight trajectory planning solution.

[0092] 2. Improved accuracy and efficiency: After the introduction of the attention mechanism, the technical solution of the present invention can dynamically focus on the information of key positions in the encoding layer, thereby more accurately planning the targets in the building scene and improving the recognition accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0093] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0094] Figure 1 It is a schematic diagram of the method flow of the present invention;

[0095] Figure 2 It is a schematic diagram of the system composition of the present invention;

[0096] Figure 3 Schematic diagram of a training method according to Embodiment 2 of the present invention;

[0097] Figure 4 It is a schematic diagram of the aerial photography trajectory of the experimental group of the test example of the present invention;

[0098] Figure 5 It is a schematic diagram of the aerial trajectory of the control group of the test example of the present invention;

[0099] Figure 6 This is a schematic diagram of the Python architecture program of the Seq-Bi-GRU neural network model of the first embodiment of the present invention;

[0100] Figure 7 This is a schematic diagram of a Python training program for the Seq-Bi-GRU neural network model of the second embodiment of the present invention;

[0101] Figure 8 It is a schematic diagram of the deviation comparison between the experimental group and the control group of the test example of the present invention; the horizontal axis in the figure is the key real estate boundary point, and the vertical axis is the deviation level. DETAILED DESCRIPTION

[0102] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the accompanying drawings. In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below;

[0103] It should be noted that the various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same or similar parts between the various embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description.

[0104] Embodiment 1: Figure 1 As shown, this embodiment provides a method for multi-UAV visual collaborative building scene recognition, including the following execution steps:

[0105] In this embodiment, regarding step S1, data preprocessing: each flight trajectory t(i) is planned as a feature sequence S; where S = {s1, s2, ..., s T}, where T represents the length of the sequence.

[0106] In this embodiment, regarding step S2, the encoding layer of the Seq-Bi-GRU neural network model is activated: the bidirectional Bi-GRU neural network model is used as the encoding layer of the Seq2Seq model. The encoding layer is responsible for receiving the feature sequence S and capturing the context information in the feature sequence S through the forward and backward passes of the bidirectional Bi-GRU.

[0107] The bidirectional Bi-GRU neural network model includes two transmission directions: forward and backward.

[0108] Specifically, step S200, forward propagation: In the forward propagation, the Bi-GRU neural network model processes the input feature sequence S in chronological order. For the forward propagation, the hidden state Updated to:

[0109]

[0110] Among them, s t represents the input features at time t, Represents the hidden state at the previous moment.

[0111] That is, use these inputs to update the hidden state at the current moment This state captures the sequence information up to the current moment. Through the forward pass, the encoding layer can capture the forward context information in the input feature sequence S. This helps the model understand the order and dependencies of the elements in the sequence.

[0112] Specifically, step S201, backward propagation: For backward propagation, the Bi-GRU neural network model processes the input feature sequence S in reverse time order, and the hidden state Updated to:

[0113]

[0114] Among them, s t It also represents the input features at time t, and Represents the hidden state at the next moment, which captures the information from the current moment to the end of the sequence.

[0115] It can be understood that through the backward pass, the encoding layer is able to capture the backward context information in the input feature sequence S. It helps the model understand the reverse order dependencies and global context of the elements in the sequence.

[0116] Specifically, step S202, output: the coding layer output h at time t t It is composed of the concatenation of the forward and backward hidden states:

[0117] H={h1,h2,...,h T};

[0118]

[0119] Among them, [·;·] represents the concatenation operation of the vectors, and H is a sequence containing the coded information at all times. That is, the output of the coding layer at all times constitutes a sequence H containing the coded information at all times.

[0120] It can be understood that by concatenating the forward and backward hidden states, the encoding layer can simultaneously capture the forward and backward contextual information in the input feature sequence S. This enables the model to more comprehensively understand the complex dependencies and contextual relationships in the sequence, thereby improving the accuracy of subsequent decoding and sequence generation.

[0121] In this embodiment, regarding step S3, the attention mechanism is activated: the attention mechanism dynamically pays attention to the information at different positions in the encoding layer when the decoding layer generates the output sequence H, and outputs the intermediate vector C;

[0122] Among them, the attention mechanism aims to calculate a weight distribution α t To reflect the degree of attention paid by the decoding layer to the information of each position in the encoding layer when generating the current output. Weight α t,i It is calculated by a softmax function, which represents the attention paid to the i-th position information of the encoding layer at the decoding time t. Alignment model e t,i It is used to evaluate the match between the decoding time t and the i-th position of the encoding layer. It is achieved through a small neural network (containing learnable parameters v a , W a and U a And hyperbolic tangent activation function) calculated:

[0123] Among them, e t,i is an alignment model that evaluates the match between the decoding time t and the i-th position of the encoding layer. It can be calculated by a small neural network:

[0124]

[0125] Among them, v a , W a and U a is a learnable parameter and \tanh(\cdot) is the hyperbolic tangent activation function.

[0126] Once the weight distribution α is calculated t , we can use it to generate the intermediate vector c t , which is the weighted sum of all position information in the encoding layer:

[0127] C={c1,c2,...,c T};

[0128]

[0129] Among them, the tth intermediate vector c t Contains all the important information that the decoding layer cares about when generating the current output.

[0130] Finally, the decoding layer will combine the hidden state s from the previous moment t-1 and the intermediate vector c t , or add other inputs (such as the output of the previous moment) to generate the hidden state s at the current moment t and output.

[0131] It can be understood that by calculating the weight distribution, the attention mechanism enables the decoding layer to pay more attention to the relevant information in the encoding layer when generating the current output, thereby improving the accuracy and relevance of the generation. t As an important reference for the decoding layer to generate the current output, it enables the decoding layer to more accurately capture the relevant information in the encoding layer, thereby improving the accuracy and fluency of the generation.

[0132] Furthermore, the decoding layer will combine the hidden state s of the previous moment t-1 and the intermediate vector c t (also includes other inputs, such as the output of the previous moment) to generate the hidden state st and output of the current moment. This process is implemented through the gating mechanism in the neural network, which can control the flow and update of information. Furthermore, by combining the hidden state and intermediate vectors of the previous moment, the decoding layer can more accurately capture the temporal dependencies and contextual information in the sequence, thereby generating a more accurate and coherent output sequence. This process also enables the decoding layer to flexibly handle variable-length input and output sequences, improving the versatility and applicability of the model.

[0133] In this embodiment, regarding step S4, the decoding layer of the Seq-Bi-GRU neural network model is activated:

[0134] The intermediate vector C is input into the decoding layer. The decoding layer is based on the fixed-length vector output by the Bi-GRU structure, and gradually generates each element of the output sequence and terminates through the termination condition, and finally outputs the output sequence S' of the planned flight trajectory.

[0135] The decoding layer also uses the Bi-GRU structure and receives the intermediate vector c at each decoding time t t As input. At the same time, the decoding layer also receives the hidden state s of the previous moment t-1 and other inputs (such as the output at the previous moment).

[0136] Specifically, step S400, forward pass: the Bi-GRU structure of the decoding layer is responsible for updating the hidden state according to the input information and generating the output at the current moment. Specifically, for the forward pass, the hidden state Updated to:

[0137]

[0138] in, Indicates additional input information.

[0139] It can be understood that through the forward pass, the decoding layer is able to capture the temporal dependencies and contextual relationships in the input information, thereby generating more accurate and coherent outputs. Updating the hidden state helps the decoding layer take into account previous information when generating the current output, improving the accuracy of the generation.

[0140] Specifically, step S401, backward pass: For backward pass, since the decoding process is usually unidirectional, the backward hidden state In fact, this does not apply in standard Seq2Seq decoding. But in the variant of bidirectional decoding, the backward pass can be defined similarly. So here we can focus on the forward pass.

[0141] Therefore, the decoding layer combines the forward hidden state and in the backward decoding variant To generate the output y at the current time t Output y t It is obtained through a fully connected layer and a softmax activation function to ensure that the output is in the form of a probability distribution:

[0142]

[0143] Among them, W o and b o are learnable parameters.

[0144] It is understandable that in the variant of bidirectional decoding, the backward pass is able to capture reverse contextual information and further improve the accuracy of generation. Even in standard unidirectional decoding, understanding the nature of the backward pass helps explore more complex model structures.

[0145] Specifically, step S402, iterative decoding: the decoding process is repeated continuously until a certain termination condition is met, such as reaching a preset output sequence length. Finally, the decoding layer will output an output sequence of the planned flight trajectory:

[0146] S'={y1,y2,...y T '};

[0147] Where T' is the length of the output sequence.

[0148] It is understandable that through iterative decoding, the model is able to gradually generate a complete output sequence, with each element taking into account the previous context information. The termination condition ensures that the length of the output sequence is controllable, which is convenient for subsequent processing and application. The use of the softmax activation function ensures that the output is a valid probability distribution, which is convenient for subsequent classification or decision-making tasks.

[0149] In this embodiment, the Python architecture program of the Seq-Bi-GRU neural network model in the above steps S2 to S4 is as follows: Figure 6 As shown in the figure, the principle is: using a bidirectional GRU (Bi-GRU) as an encoder, it is able to consider both forward and reverse temporal dependencies. The encoder receives an input sequence and returns an output sequence as well as the final forward and reverse hidden states. The decoder is a unidirectional GRU layer that receives the encoder's output sequence as input and uses the encoder's final hidden state (by concatenating the forward and reverse states) as the initial state. The decoder generates each element of the output sequence step by step. The output of the decoder passes through a fully connected layer and a softmax activation function to obtain the final output, which is a probability distribution representing the nature of each output category. The model is trained using a standard cross entropy loss function and an Adam optimizer. During the training process, the model learns to map the input sequence to the output sequence.

[0150] In this embodiment, regarding step S5, online adjustment is performed: the output sequence S' is decoded to form the flight trajectory t(i)' of each UAV and input it into the multi-UAV system to achieve real-time building scene recognition and trajectory planning.

[0151] Preferably, the decoding function is decode(·), which converts the output sequence S′ into a flight trajectory t(i)′: t(i)′=decode(S′);

[0152] The decoding function decode(·) can convert the elements in the output sequence into position control instructions or speed control instructions of the UAV.

[0153] Embodiment 2: Figure 3 , 7 As shown, this embodiment further provides a training method for the Seq-Bi-GRU neural network model based on the first embodiment, and the training steps include:

[0154] P1. Collect and divide the historical data set D': Collect the flight trajectory t(i) of the drone and its building identification label from the historical data and form the data set D'. Then divide the preprocessed data set into the training set D train and the test set D test ;

[0155] P2. Set training parameters: including learning rate, batch size, number of iterations, and Adam optimizer.

[0156] P3. Training model: using training set D trainThe model is trained and the model parameters are updated through the back propagation algorithm to minimize the mean square error (MSE) between the predicted value and the actual value, and parameter tuning and model structure adjustment are performed as needed.

[0157] P4, model evaluation and deployment: Repeat forward propagation, loss calculation and back propagation, and continuously iterate the training process until the stopping condition is met (the maximum number of iterations is reached or the loss function value is less than the threshold).

[0158] Specifically, Figure 7 In this paper, the flight trajectories of drones and their building identification labels are collected from historical data to form a dataset D'. The dataset D' is then divided into a training set Dtrain and a test set Dtest for model training and evaluation. At the same time, training parameters such as learning rate, batch size, and number of iterations are set, which will affect the training process and performance of the model. train Train the Seq-Bi-GRU neural network model. Update the model parameters through the back propagation algorithm to minimize the mean square error (MSE) between the predicted value and the actual value. Perform parameter tuning and model structure adjustment as needed to improve the performance of the model.

[0159] Test example:

[0160] (I) Preface:

[0161] This experiment aims to compare the performance of the Seq-Bi-GRU neural network model and the traditional parallel convolutional neural network in UAV trajectory planning and cruise photography tasks, and evaluate the ability of the two models in processing sequences of unequal lengths and planning smooth and intuitive flight trajectories.

[0162] (II) Experimental Materials and Methods

[0163] 2.1 Experimental Group:

[0164] Model: Seq-Bi-GRU neural network model.

[0165] UAV systems: Three sets of UAV systems with the same configuration.

[0166] 2.2 Control group:

[0167] Model: Traditional parallel convolutional neural network.

[0168] UAV systems: Three sets of UAV systems with the same configuration.

[0169] 2.3 UAV system parameters:

[0170] Three groups of the same type of D350 multi-rotor drones, the parameters are shown in Table 1:

[0171] Table 1 UAV parameters

[0172]

[0173] The D350 multi-rotor drone is equipped with a D-IP550 oblique camera module, which is compatible with a variety of drone platforms and is equipped with a full-frame micro-single camera and a five-lens system, of which four lenses are distributed at a 45° inclination angle. The specific parameters are shown in Table 2. The camera-by-camera marking and synchronous exposure technology is used to accurately capture the exposure position information:

[0174] Table 2 Camera parameters

[0175]

[0176] 2.4 Experimental content:

[0177] Based on GIS remote sensing system, a remote sensing image of a ground building complex is obtained (such as Figures 4-5 shown).

[0178] The experimental group and control group were trained using the Seq-Bi-GRU neural network model and the traditional parallel convolutional neural network, respectively.

[0179] After training, the two drone systems performed trajectory planning and cruise photography tasks based on the building complex and recorded their trajectory data.

[0180] In order to verify the accuracy of the integrated real estate measurement technology, a field test was conducted by combining high-precision GNSS with a laser scanner. 76 key real estate boundary points were selected at the boundaries of important buildings and plots, and accuracy was checked. The accuracy of boundary points is divided into two levels, and the error of each level of boundary points relative to the benchmark point shall not exceed the preset standard. For points with a distance of more than 100m, a comparison chart of the photographic effect was recorded and drawn.

[0181] (III) Experimental results:

[0182] 3.1 Trajectory comparison:

[0183] like Figure 4 As shown, the experimental group (Seq-Bi-GRU neural network model): the flight trajectory is more stable and intuitive, and can effectively process sequences of unequal lengths and adapt to the differences in flight trajectory lengths of different drones.

[0184] like Figure 5 As shown in the figure, the control group (traditional parallel convolutional neural network): Since RNNs receive inputs and generate outputs in a fixed manner at each time step, their ability to process sequences of unequal lengths is limited. In multi-UAV collaborative trajectory planning, the flight trajectories of different UAVs have different lengths, resulting in relatively chaotic trajectories.

[0185] 3.2 Photo effect:

[0186] like Figure 8 As shown in the figure, the experimental group had a more stable and intuitive trajectory planning, better cruise photography effect, and could capture the characteristics of the building complex more accurately. The control group had a chaotic trajectory, resulting in poor photography effect, and some building complex features were not accurately captured.

[0187] (IV) Conclusion and discussion:

[0188] This example compares the performance of the Seq-Bi-GRU neural network model with the traditional parallel convolutional neural network in UAV trajectory planning and cruise photography tasks, and finds that the Seq-Bi-GRU neural network model has significant advantages in processing sequences of unequal lengths and planning smooth and intuitive flight trajectories. This finding is of great significance for improving the efficiency and accuracy of multi-UAV collaborative trajectory planning. Future research can further explore the application potential of the Seq-Bi-GRU neural network model in other UAV tasks and optimize the model structure to improve performance.

[0189] All the above embodiments only express the implementation methods of the relevant practical applications of the present invention, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be based on the attached claims.

[0190] For those skilled in the art, it can be further appreciated that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0191] At the same time, those skilled in the art can understand that all or part of the processes in all the above-mentioned embodiments can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media provided in this application and used in the embodiments may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (SSRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

Claims

1. A method for recognizing building scenes by visual collaboration of multiple UAVs, comprising collecting the flight trajectory t(i) of the i-th UAV from a multi-UAV system, characterized in that: The steps include: S1, plan each flight trajectory t(i) as a feature sequence S; S2, uses a bidirectional Bi-GRU neural network model as the encoding layer of the Seq2Seq model; the encoding layer is responsible for receiving the feature sequence S and capturing the context information in the feature sequence S through the forward and backward passes of the bidirectional Bi-GRU; S3, when generating the output sequence H, dynamically pays attention to the information at different positions in the coding layer and outputs the intermediate vector C; S4, input the intermediate vector C into the decoding layer, the decoding layer is based on the fixed-length vector output by the Bi-GRU structure, and gradually generates each element of the output sequence and terminates through the termination condition, and finally outputs the output sequence S' of the planned flight trajectory; S5, decode the output sequence S', form the flight trajectory t(i)' of each UAV and input it into the multi-UAV system to achieve real-time building scene recognition and trajectory planning.

2. The building scene recognition method according to claim 1, characterized in that: In S1, S={s1,s2,...,s T }, where T represents the length of the sequence.

3. The building scene recognition method according to claim 2, characterized in that: In S2, the bidirectional Bi-GRU neural network model includes two transmission directions: forward and backward: For the forward pass, the hidden state Updated to: Among them, s t represents the input features at time t, Represents the hidden state at the previous moment; For the backward pass, the hidden state Updated to: Among them, s t It also represents the input features at time t, and Represents the hidden state at the next moment.

4. The building scene recognition method according to claim 3, characterized in that: The output of the encoding layer h t It is composed of the concatenation of the forward and backward hidden states: H={h1,h2,...,h T }; Among them, [·;·] represents the concatenation operation of the vectors, and H is a sequence containing the coded information at all times.

5. The building scene recognition method according to claim 2, characterized in that: In S3, the dynamics aims to calculate a weight distribution α t , which reflects the degree of attention paid by the decoding layer to the information at each position of the encoding layer when generating the current output.

6. The building scene recognition method according to claim 5, characterized in that: The weight distribution α t The calculation method is: Among them, e t,i It is an alignment model used to evaluate the matching degree between the decoding time t and the i-th position of the encoding layer; Based on the weight distribution α t Generate intermediate vector c t , which is the weighted sum of all position information in the coding layer: C = {c1, c2, ..., c T }; 7. The building scene recognition method according to claim 2, characterized in that: In S4, the decoding layer adopts a Bi-GRU structure and receives an intermediate vector c at each decoding time t t As input; at the same time, the decoding layer also receives the hidden state s of the previous moment t-1 And other inputs: S400, for the forward pass, the hidden state Updated to: in, Indicates other input information; S401, for the backward pass, combined with the forward hidden state and in the backward decoding variant To generate the output y at the current time t : Among them, softmax is the softmax activation function.

8. The building scene recognition method according to claim 7, characterized in that: In S4, the decoding process is repeated until a certain termination condition is met, and the decoding layer outputs an output sequence of a planned flight trajectory: S' = {y1, y2, ...y T '}; Where T' is the length of the output sequence.

9. The building scene recognition method according to claim 1, 2, 5 or 7, characterized in that: In S5, the decoding function is decode(·), which converts the output sequence S′ into a flight trajectory t(i)′: t(i)′=decode(S′); The decoding function decode(·) converts the elements in the output sequence into position control instructions or speed control instructions for the UAV.

10. The multi-UAV visual collaborative building scene recognition linkage system is characterized by: The system includes a processor and a memory connected to the processor, wherein the memory stores program instructions, and when the program instructions are executed by the processor, the processor executes the building scene recognition method according to any one of claims 1 to 9, and the processor is connected to, A data preprocessing module responsible for converting the raw flight data collected in the multi-UAV system into feature sequences; The Seq-Bi-GRU neural network model module combines and implements the Seq2Seq model and the bidirectional Bi-GRU recurrent neural network; An attention mechanism module that allows the decoding layer to dynamically focus on information at different locations in the encoding layer when generating an output sequence; Model training module responsible for training the Seq-Bi-GRU neural network model; A deployment module to deploy the model into a multi-UAV system for practical application.

Images