UAV positioning method and device, electronic device, and storage medium

Abstract

The present invention relates to the field of artificial intelligence technology, and in particular to a method for positioning a drone and its device, electronic device, and storage medium. In the drone positioning method of the present invention, a group of images to be identified is first obtained, and the group of images to be identified includes multiple environmental images collected during the navigation of the drone, and the multiple environmental images have spatial continuity features. Further, the spatial continuity features of the group of images to be identified are extracted based on a memory network model to obtain a continuous feature vector of the group of images to be identified, and then based on the preset initial position of the drone and the continuous feature vector, the real-time position information of the drone is finally determined. Through the drone positioning method provided by the present invention, the drone can obtain the current real-time position information of the drone based on the preset initial position and the continuous feature vector extracted by the memory network model during navigation, thereby improving the positioning efficiency of the drone.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a drone positioning method and device, electronic equipment, and storage medium. Background Art

[0002] In recent years, drones have played an increasingly important role in aerial photography, disaster relief, agricultural plant protection, forest firefighting, power inspection and other fields. The positioning system is an indispensable and important component in the use of drones. As the use of drones becomes more and more widespread, the environment faced by drones during operation is complex and diverse. How to achieve accurate positioning of drones in complex outdoor environments or indoor environments has become a difficulty in drone positioning. Among the related technologies, drone positioning technology mainly relies on the Global Positioning System (GPS) and inertial navigation system. However, GPS positioning is restricted by factors such as terrain and man-made buildings, which can easily cause the system signal to be interrupted by interference. Although the inertial navigation system does not rely on external information, the positioning error increases over time, and calibration time is required before each use, resulting in low positioning efficiency. Therefore, how to improve the efficiency of drone positioning has become a difficult problem that needs to be solved urgently by technical personnel in the industry. Summary of the Invention

[0003] The present invention aims to solve at least one of the technical problems existing in the prior art. To this end, the present invention provides a method and device for positioning a drone, an electronic device, and a storage medium thereof, which can improve the positioning efficiency of the drone.

[0004] A method for positioning a drone according to a first embodiment of the present invention includes:

[0005] Acquire a set of images to be identified, where the set of images to be identified includes multiple environmental images collected during the navigation of the UAV, and the multiple environmental images have spatial continuity characteristics;

[0006] Extracting spatial continuity features of the set of images to be identified based on a memory network model to obtain a continuous feature vector of the set of images to be identified;

[0007] Based on the preset initial position of the UAV and the continuous feature vector, the real-time position information of the UAV is determined.

[0008] Optionally, according to some embodiments of the present invention, before extracting spatial continuity features of the to-be-identified group of images based on the memory network model to obtain continuous feature vectors of the to-be-identified group of images, the method further includes:

[0009] Acquire a training data set, the training data set including a plurality of preset environment images having spatial continuity characteristics and preset displacement data matching each of the preset environment images;

[0010] The basic recognition model is optimized and trained based on the training data set to obtain the memory network model.

[0011] Optionally, according to some embodiments of the present invention, the optimizing and training the basic recognition model based on the training data set to obtain the memory network model includes:

[0012] Iteratively training the basic recognition model based on the training data set, wherein in each round of iterative training, a plurality of the preset environment images are input into the basic recognition model for current round training processing, and current round displacement data corresponding to each preset environment image is obtained;

[0013] After each round of iterative training, multiple sets of displacement data of the current round are compared with multiple sets of preset displacement data, the recognition accuracy of the basic recognition model is calculated, and the parameters of the basic recognition model are updated;

[0014] The change of the recognition accuracy after each round of the iterative training is counted, and when the recognition accuracy converges to a constant value, the trained memory network model is obtained.

[0015] Optionally, according to some embodiments of the present invention, the iterative training of the basic recognition model based on the training data set, wherein in each round of the iterative training, multiple preset environment images are input into the basic recognition model for current round training processing, and multiple sets of current round displacement data corresponding to the multiple preset environment images are obtained, including:

[0016] In the current round of training, multiple sets of current round movement directions and multiple sets of current round movement distances of the drone's perspective between the multiple preset environment images are obtained;

[0017] Based on the multiple sets of the current wheel movement directions and the multiple sets of the current wheel movement distances, multiple sets of the current wheel displacement data are obtained.

[0018] Optionally, according to some embodiments of the present invention, after each round of iterative training, comparing multiple sets of displacement data of the current round with multiple sets of preset displacement data, calculating the recognition accuracy of the basic recognition model and updating the parameters of the basic recognition model, including:

[0019] Obtaining displacement estimation errors of each group based on the multiple groups of current-round displacement data and the multiple groups of preset displacement data;

[0020] Obtaining the number of accurate results in each group of displacement estimation errors that are less than a preset threshold;

[0021] The recognition accuracy is calculated based on the number of precise results and the total number of displacement data in this round.

[0022] Optionally, according to some embodiments of the present invention, extracting spatial continuity features of the set of images to be identified based on a memory network model to obtain a continuous feature vector of the set of images to be identified includes:

[0023] Performing a pooling operation on the group of images to be identified to generate reduced feature map data;

[0024] Performing dimensionality reduction processing on the reduced feature map data to obtain a reduced-dimensionality feature vector;

[0025] The spatial continuity feature is extracted from the dimensionality reduction feature vector through the memory network model to obtain the continuous feature vector.

[0026] Optionally, according to some embodiments of the present invention, determining the real-time position information of the drone based on the preset initial position of the drone and the continuous feature vector includes:

[0027] Inputting the continuous feature vector into the fully connected layer of the memory network model;

[0028] Performing regression processing on the continuous feature vector based on the fully connected layer to obtain actual displacement data corresponding to the navigation process of the UAV;

[0029] The real-time position information of the UAV is determined based on the preset initial position and the actual displacement data.

[0030] The drone positioning device according to the second embodiment of the present invention includes:

[0031] An image acquisition module is used to acquire a set of images to be identified, wherein the set of images to be identified includes multiple environmental images collected during the navigation of the UAV, and the multiple environmental images have spatial continuity characteristics;

[0032] An image processing module, configured to extract spatial continuity features of the set of images to be identified based on a memory network model, and obtain a continuous feature vector of the set of images to be identified;

[0033] The positioning calculation module is used to determine the real-time position information of the drone based on the preset initial position of the drone and the continuous feature vector.

[0034] In a third aspect, an embodiment of the present invention provides an electronic device, comprising: a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, it implements the drone positioning method as described in any one of the embodiments of the first aspect of the present invention.

[0035] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, wherein the storage medium stores a program, and the program is executed by a processor to implement the drone positioning method as described in any one of the embodiments of the first aspect of the present invention.

[0036] The drone positioning method, device, electronic device, and storage medium provided by the embodiments of the present invention have at least the following beneficial effects:

[0037] In the drone positioning method of the present invention, a group of images to be identified is first obtained, the group of images to be identified includes multiple environmental images collected during the drone's navigation process, and the multiple environmental images have spatial continuity features. Further, the spatial continuity features of the group of images to be identified are extracted based on a memory network model to obtain a continuous feature vector of the group of images to be identified. Further, based on the preset initial position of the drone and the continuous feature vector, the real-time position information of the drone is finally determined. Through the drone positioning method and its device, electronic device, and storage medium provided by the present invention, the drone can obtain its current real-time position information based on the preset initial position and the continuous feature vector extracted by the memory network model during navigation, thereby improving the positioning efficiency of the drone.

[0038] Additional aspects and advantages of the present invention will be set forth in part in the description which follows and, in part, will be obvious from the description which follows, or may be learned by practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments with reference to the accompanying drawings, in which:

[0040] Figure 1 A flowchart of a method for positioning a drone according to an embodiment of the present invention;

[0041] Figure 2 A flow chart of another UAV positioning method provided by an embodiment of the present invention;

[0042] Figure 3 A flow chart of another UAV positioning method provided by an embodiment of the present invention;

[0043] Figure 4 A flow chart of another UAV positioning method provided by an embodiment of the present invention;

[0044] Figure 5 A flow chart of another UAV positioning method provided by an embodiment of the present invention;

[0045] Figure 6 A flow chart of another UAV positioning method provided by an embodiment of the present invention;

[0046] Figure 7 A flow chart of another UAV positioning method provided by an embodiment of the present invention;

[0047] Figure 8 A schematic diagram of a memory network model provided by an embodiment of the present invention;

[0048] Figure 9 A schematic diagram of coarse feature extraction provided by an embodiment of the present invention;

[0049] Figure 10 A schematic diagram of another memory network model provided by an embodiment of the present invention;

[0050] Figure 11 A flowchart of a drone positioning method provided by a more specific embodiment of the present invention;

[0051] Figure 12 A flowchart of another UAV positioning method provided in a more specific embodiment of the present invention;

[0052] Figure 13 A schematic diagram of a module of a drone positioning device provided by an embodiment of the present invention;

[0053] Figure 14 Schematic diagram of an electronic device for executing a drone positioning method provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0054] The following describes embodiments of the present invention in detail. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended only to explain the present invention and are not to be construed as limiting the present invention.

[0055] In the description of the present invention, "several" means one or more, "many" means more than two, "greater than," "less than," and "exceed" are understood to exclude the number itself, while "above," "below," and "within" are understood to include the number itself. The use of "first" and "second" in the description is solely for the purpose of distinguishing technical features and should not be construed as indicating or implying relative importance, implicitly specifying the number of the indicated technical features, or implicitly specifying the order of the indicated technical features.

[0056] In the description of the present invention, it should be understood that descriptions involving orientations, such as up, down, left, right, front, and back, etc., indicating orientations or positional relationships, are based on the orientations or positional relationships shown in the accompanying drawings. They are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation. Therefore, they cannot be understood as limitations on the present invention.

[0057] Throughout this specification, reference to terms such as "one embodiment," "some embodiments," "illustrative embodiments," "examples," "specific examples," or "some examples" means that a specific feature, structure, material, or characteristic described in conjunction with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, illustrative uses of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

[0058] In the description of the present invention, it should be noted that, unless otherwise explicitly defined, terms such as "set," "install," and "connect" should be interpreted broadly. Those skilled in the art can reasonably determine the specific meanings of these terms in the present invention based on the specific content of the technical solution. Furthermore, the identification of specific steps below does not constitute a limitation on the order or execution logic of the steps. The order and logic of the execution of the steps should be understood and inferred with reference to the contents described in the embodiments.

[0059] In recent years, drones have played an increasingly important role in aerial photography, disaster relief, agricultural plant protection, forest firefighting, power inspection and other fields. The positioning system is an indispensable and important component in the use of drones. As the use of drones becomes more and more widespread, the environment faced by drones during operation is complex and diverse. How to achieve accurate positioning of drones in complex outdoor environments or indoor environments has become a difficulty in drone positioning. Among the related technologies, drone positioning technology mainly relies on the Global Positioning System (GPS) and inertial navigation system. However, GPS positioning is restricted by factors such as terrain and man-made buildings, which can easily cause the system signal to be interrupted by interference. Although the inertial navigation system does not rely on external information, the positioning error increases over time, and calibration time is required before each use, resulting in low positioning efficiency. Therefore, how to improve the efficiency of drone positioning has become a difficult problem that needs to be solved urgently by technical personnel in the industry.

[0060] The present invention aims to solve at least one of the technical problems existing in the prior art. To this end, the present invention provides a method and device for positioning a drone, an electronic device, and a storage medium thereof, which can improve the positioning efficiency of the drone.

[0061] The following is further described with reference to the accompanying drawings.

[0062] Reference Figure 1 According to the first aspect of the present invention, the UAV positioning method includes:

[0063] Step S101: obtaining a set of images to be identified, where the set of images to be identified includes multiple environmental images collected during the flight of the drone, and the multiple environmental images have spatial continuity characteristics;

[0064] According to some embodiments provided by the present invention, a drone is provided with an image acquisition device. During the drone's flight, the image acquisition device collects environmental images of the drone's environment as the drone navigates, thereby obtaining a group of images to be identified. It should be noted that the environmental images collected at adjacent time intervals in each group have a common spatial connection area. For example, environmental image A includes scene m, scene n, and scene i. Environmental image B collected at a time interval adjacent to environmental image A includes scene n, scene i, and scene o. Environmental image C collected at a time interval adjacent to environmental image B includes scene o, scene p, and scene q. Among them, the area where scene n and scene i are located in environmental image A and environmental image B is the first spatial connection area, and the area where scene o is located in environmental image B and environmental image C is the second spatial connection area. The spatial continuity feature between multiple environmental images refers to the characteristics of the continuous space reflected by the spatial connection areas between each group of environmental images.

[0065] Step S102: extracting spatial continuity features of the image set to be identified based on the memory network model to obtain a continuous feature vector of the image set to be identified;

[0066] It should be noted that continuous feature vectors reflect the spatial continuity relationship between the environmental images in the set of images to be identified. For example, if several environmental images are collected consecutively, and since these images exhibit spatial continuity, the motion of the image acquisition device (drone) can be determined based on the environmental changes in these images. It should be understood that as the drone's flight progresses, the multiple environmental images with spatial continuity characteristics contained in the set of images to be identified can reflect the evolution of the spatial environment during the drone's flight. Therefore, extracting spatial continuity features from the set of images to be identified based on a memory network model and obtaining continuous feature vectors for the set of images to be identified facilitates subsequent steps in determining the drone's real-time location. Memory networks, also known as memory neural networks (MemNNs), refer to artificial intelligence technologies that utilize memory components to store scene information for long-term memory. Memory network models include recurrent neural networks (RNNs), long short-term memory networks (LSTMs), and their variant, the gated recurrent unit (GRU). In some preferred embodiments of the present invention, a long short-term memory network (LSTM) is selected as the memory network model of the drone positioning method of the present invention.

[0067] Step S103 : determining the real-time position information of the UAV based on the preset initial position of the UAV and the continuous feature vector.

[0068] It should be noted that, by extracting the spatial continuity features of the group of images to be identified based on the memory network model, a continuous feature vector reflecting the spatial continuity features in the group of images to be identified can be obtained, and the actual displacement data of the UAV from the start of navigation to the current position can be determined. It should be understood that if the real-time position of the UAV needs to be determined, it is also necessary to combine the continuous feature vector with the preset initial position for analysis to clarify the starting point and movement process of the UAV navigation, and then determine the real-time position information of the UAV.

[0069] In the drone positioning method of the present invention, a set of images to be identified is first obtained. The set of images to be identified includes multiple environmental images collected during the drone's navigation process, and the multiple environmental images have spatial continuity characteristics. Further, spatial continuity features are extracted from the set of images to be identified based on a memory network model to obtain a continuous feature vector of the set of images to be identified. Further, based on the drone's preset initial position and the continuous feature vector, the real-time position information of the drone is finally determined. Through the drone positioning method and its device, electronic device, and storage medium provided by the present invention, the drone's current real-time position information can be obtained based on the preset initial position and the continuous feature vector extracted from the memory network model during navigation, thereby improving the drone's positioning efficiency.

[0070] Reference Figure 2 According to some embodiments of the present invention, before extracting spatial continuity features of the set of images to be identified based on the memory network model and obtaining continuous feature vectors of the set of images to be identified, the method further includes:

[0071] Step S201: obtaining a training data set, where the training data set includes a plurality of preset environment images having spatial continuity features and preset displacement data matching each preset environment image;

[0072] Step S202: Optimize and train the basic recognition model based on the training data set to obtain a memory network model.

[0073] It should be noted that in order to obtain a memory network model that can analyze the spatial continuity characteristics of the image group, the basic recognition model needs to be optimized and trained. Among them, the basic recognition model refers to a neural network model with basic memory function. The basic recognition model can be a recurrent neural network (RNN), a long short-term memory network (LSTM), a gated recurrent unit structure (GRU) or other neural network models with basic memory function. It should be understood that the training data set includes multiple preset environment images with spatial continuity characteristics and preset displacement data matching each preset environment image. For example, the training data set is acquired by aerial photography, and a preset environment image is collected at every preset time interval, and it is ensured that each set of preset environment images collected at adjacent preset time intervals has spatial continuity characteristics. When the navigation reaches the southeast direction of the preset displacement data and 10KM, the acquisition of a set of training data sets is completed. It should be understood that obtaining a training dataset is not limited to the specific embodiments listed above. Based on the concepts of the above embodiments, the preset displacement data can be set to any direction and any preset value, thereby collecting preset environmental images that match the preset displacement data and have spatial continuity characteristics. It should be emphasized that the preset displacement data includes a preset direction and a preset distance. Since the preset displacement data is pre-set displacement data, multiple preset environmental images in a set of training datasets are correspondingly matched with a set of preset displacement data, which serves as the label data supporting the training dataset.

[0074] refer to Figure 3 According to some embodiments of the present invention, optimizing and training a basic recognition model based on a training data set to obtain a memory network model includes:

[0075] Step S301, iteratively training the basic recognition model based on the training data set. In each round of iterative training, multiple preset environment images are input into the basic recognition model for this round of training processing to obtain the displacement data corresponding to each preset environment image;

[0076] It should be noted that this round of training processing means that in each round of iterative training, multiple preset environment images need to be input into the basic recognition model to estimate the displacement data, and then obtain the displacement data of this round.

[0077] Step S302: After each round of iterative training, multiple sets of displacement data from the current round are compared with multiple sets of preset displacement data, the recognition accuracy of the basic recognition model is calculated, and the parameters of the basic recognition model are updated;

[0078] It should be noted that comparing multiple sets of current-round displacement data with multiple sets of preset displacement data is equivalent to comparing the estimated values ​​(multiple sets of current-round displacement data) with the label data (multiple sets of preset displacement data). Therefore, by comparing multiple sets of current-round displacement data with multiple sets of preset displacement data, the recognition accuracy of the basic recognition model can be calculated. After obtaining the recognition accuracy of the basic recognition model for this round, the parameters of the basic recognition model are further updated to adjust the memory network model's ability to extract spatial continuity features (i.e., continuous feature vectors).

[0079] Step S303: Count the changes in recognition accuracy after each round of iterative training. When the recognition accuracy converges to a constant value, a trained memory network model is obtained.

[0080] It should be noted that when the recognition accuracy converges to a constant value, it means that the basic recognition model has reached the application standard, and finally a trained memory network model is obtained.

[0081] Reference Figure 4 According to some embodiments of the present invention, a basic recognition model is iteratively trained based on a training data set. In each round of iterative training, multiple preset environment images are input into the basic recognition model for current round training processing, and multiple sets of current round displacement data corresponding to the multiple preset environment images are obtained, including:

[0082] Step S401: In this round of training, multiple sets of current round movement directions and multiple sets of current round movement distances of the drone's perspective between multiple preset environment images are obtained;

[0083] Step S402 : acquiring multiple sets of current wheel displacement data based on multiple sets of current wheel movement directions and multiple sets of current wheel movement distances.

[0084] It should be noted that during this round of training, multiple sets of current-round motion directions and distances are obtained from the drone's perspective across multiple preset environmental images, allowing for the spatial continuity between these sets of preset environmental images. Therefore, multiple sets of current-round displacement data can be obtained based on these multiple sets of current-round motion directions and distances.

[0085] Reference Figure 5 According to some embodiments of the present invention, after each round of iterative training, multiple sets of displacement data of the current round are compared with multiple sets of preset displacement data, the recognition accuracy of the basic recognition model is calculated, and the parameters of the basic recognition model are updated, including:

[0086] Step S501, obtaining displacement estimation errors of each group based on multiple groups of current-round displacement data and multiple groups of preset displacement data;

[0087] Step S502, obtaining the number of accurate results in each group of displacement estimation errors that are smaller than a preset threshold;

[0088] Step S503: Calculate the recognition accuracy rate based on the number of accurate results and the total number of displacement data in this round.

[0089] It should be noted that when a set of displacement estimation errors is less than a preset threshold, the displacement data corresponding to that set of displacement estimation errors is estimated more accurately. Therefore, the displacement data corresponding to that set of displacement estimation errors is considered accurate. The recognition accuracy after this round of training can be calculated based on the number of accurate results and the total amount of displacement data in this round.

[0090] According to some more specific embodiments provided by the present invention, a Uav_pic (UAV picture) training data set for training a memory network model is constructed, such as Figure 8 As shown. Specifically, it includes deploying the PyTorch (PyTorch is an open source Python machine learning library) deep learning framework based on Python (Python is a widely used interpreted, high-level and general programming language) language and the machine vision library Opencv (an open source computer vision library), and constructing the environmental images collected by the historical drone camera into a Uav_pic dataset. Based on the constructed Uav_pic dataset training model, a local feedback neural network (LRNN) is further deployed as the memory network model in the present invention. In some embodiments of the present invention, a group of images with a larger proportion in the Uav_pic dataset is used for training, and a group of images with a smaller proportion is used to test the training process. The training method of LRNN can adopt a transfer learning method. According to some more specific embodiments provided by the present invention, LRNN consists of two residual blocks, an LSTM network and three fully connected layers. First, each residual block consists of a convolution layer with a convolution kernel of 3*3 and a 3*3 pooling layer, and the input of the residual block and the output of the pooling layer are spliced ​​as the input of the next network layer. First, the convolution layer performs convolution operations on the image, as follows Figure 9 As shown in the figure, a coarse feature map of the image is obtained. The maximum pooling layer is then used to pool the coarse features of the image to reduce the size of the feature map. Since edges and other objects in the environment are considered small objects, using a 3*3 convolution kernel not only obtains more detailed edge features but also reduces the amount of computation. The residual block connects the input to the output for splicing, solving the vanishing gradient problem caused by stacking the network. This can be expressed as:

[0091] X l+1 =x1+F(x l ,W l )

[0092] Among them, xl is the output of the l-layer network, F(x l ,W l ) is the output of the feedforward neural network, W l The convolution and pooling operations are performed. The feature vector obtained by the residual block is fully connected through the fully connected layer to obtain the feature vector. The purpose is to extract the features of the image as a continuous sequence as the input of the LSTM neural network, as follows Figure 10 shown.

[0093] The 1000-dimensional feature vector extracted above is input into the LSTM neural network to extract features. LSTM can learn the spatial continuity information of the image and compress the feature vector. The 1000-dimensional feature vector extracted above is used as a continuous sequence x t , the hidden state h of the previous sequence t-1 and cell state C t-1 As the input of the LSTM unit at the current moment, the features to be discarded are selected by the formula as follows:

[0094] f t =σ(W f [h t-1 ,x t ]+b f )

[0095] Among them, W f and b f are weight matrix and bias respectively, σ represents Sigmoid function. f [□ t-1 ,x t ]+b f ) to determine the value that needs to be updated in the sequence, by calculating tanh(W C [h t-1 ,x t ]+b C ) creates a sequence and sets the new sequence value f t *C t-1 +i t *a t Add to the sequence. Finally, by calculating o t =σ(W o [h t-1 ,x t ]+b o ) Select the sequence value to be output, and finally use the tan□ function, formula o t *tanh(C t ) processes the sequence to obtain the final output result of LSTM.

[0096] Considering that a large dimensional feature vector output by the LSTM will increase the computational overhead of the two subsequent fully connected layers, while a low dimensional output vector will lose continuity information in the image features, we chose to output a 108-dimensional feature vector from the LSTM to extract the spatial continuity information of the image, namely, the continuous feature vector in the image.

[0097] Reference Figure 6 According to some embodiments of the present invention, spatial continuity feature extraction is performed on the image set to be identified based on the memory network model to obtain a continuous feature vector of the image set to be identified, including:

[0098] Step S601: performing a pooling operation on the image group to be identified to generate reduced feature map data;

[0099] Step S602: performing dimensionality reduction processing on the reduced feature map data to obtain a reduced dimensionality feature vector;

[0100] Step S603: extracting spatial continuity features from the dimensionality-reduced feature vector through a memory network model to obtain a continuous feature vector.

[0101] According to some embodiments provided by the present invention, spatial continuity feature extraction is performed on the group of images to be identified based on the memory network model to obtain a continuous feature vector of the group of images to be identified, specifically including performing a pooling operation on the group of images to be identified in a convolutional neural network (CNN) to generate reduced feature map data, and generating a reduced-dimensionality feature vector based on the reduced feature map data. It should be noted that by reducing the dimensionality of image pixels based on the above-mentioned "convolution" and "pooling", the group of images to be identified can be converted into data that is convenient for processing by the memory network model, so as to assist in the subsequent further continuous feature extraction process.

[0102] Reference Figure 7 According to some embodiments of the present invention, determining the real-time position information of the drone based on a preset initial position and a continuous feature vector of the drone includes:

[0103] Step S701, inputting the continuous feature vector into the fully connected layer of the memory network model;

[0104] Step S702: performing regression processing on the continuous feature vector based on the fully connected layer to obtain actual displacement data corresponding to the navigation process of the UAV;

[0105] Step S703: Determine the real-time position information of the UAV based on the preset initial position and the actual displacement data.

[0106] According to some more specific embodiments provided by the present invention, the embodiments of the present invention have the characteristic of high positioning accuracy, because the residual convolutional neural network model based on LSTM proposed in the embodiments of the present invention combines the residual network and the LSTM network, uses the residual network to extract the coarse features of the image, and inputs the feature vector into the LSTM as a continuous sequence after processing to learn the spatial continuity information of the image. Finally, the fully connected layer in the network output layer is regressed to obtain the current position of the drone, which includes the three-dimensional position of the target drone and the rotation angle position of the target drone. During model training, not only the transfer learning method is used to prevent the model from overfitting, but also the model introduces a new loss function to adjust the model parameters, thereby improving the accuracy of drone positioning.

[0107] According to some specific embodiments provided by the present invention, two parallel fully connected layers are added after the output layer of an LSTM neural network. The purpose is to have these two parallel fully connected layers perform regression operations on the continuous feature vectors input to the LSTM neural network on the three-dimensional direction and three-dimensional rotation angle of the drone, respectively, to obtain a three-dimensional output vector. The three-dimensional output vector reflects the actual displacement data of the drone during navigation. Ultimately, based on the preset initial position and the actual displacement data, the real-time position information of the drone is determined, completing the positioning of the drone.

[0108] According to some embodiments provided by the present invention, the drone positioning method further includes generating a sharpened environment image based on the environment image. Specifically, it includes:

[0109] Generate a grayscale environment image according to the environment image, and restore the grayscale environment image to generate a clear environment image;

[0110] According to some embodiments provided by the present invention, a grayscale environment image is generated based on an environment image, including: calculating each pixel point in the environment image to generate a corresponding grayscale value, and generating a grayscale environment image based on the grayscale value corresponding to each pixel point. According to a more specific embodiment provided by the present invention, a color image is preprocessed, that is, grayscaled, and the color image is grayscaled using the average method. Each pixel in the color image is composed of components of three channels: R, G, and B. A grayscale image is a special color image with the same R, G, and B components. The color image pixel point (x, y) is calculated using the formula:

[0111] f(x,y)=(R(x,y)+G(x,y)+B(x,y)) / 3

[0112] Calculate the grayscale value f(x, y) of a pixel in a grayscale image, where x and y are the values ​​of the R, G, and B channels of the pixel in row x and column y, respectively. The purpose of grayscaling an image is to reduce the amount of raw image data, facilitating computational complexity during subsequent processing.

[0113] Generate a sharpened environment image based on a clear environment image.

[0114] In some specific embodiments, the image preprocessing-restoration process uses Wiener filtering to filter motion blur and noise present in the grayscale image f(x, y). This method ensures that the mean square error E between the original image f(x, y) and its restored image f'(x, y) is minimized. This can be expressed as:

[0115] min:e 2 =E{[f(x,y)-f′(x,y)] 2}

[0116] Among them, E{...} is the mathematical expectation factor, e 2 It represents the mean square error between the restored image and the original image. The purpose of the restoration process is to minimize the mean square error between the original grayscale image and the restored grayscale image.

[0117] According to a more specific embodiment provided by the present invention, the step of generating a sharpened environment image according to the clear environment image is performed by sharpening processing, and the Laplacian operator is used to sharpen the improved grayscale image. Implementation. The Laplacian operator (x, y) can describe grayscale mutations in an image (g(x, y)). When the Laplacian operator is negative, the sharpened image is obtained by subtracting the clear environment image from the Laplacian image. When the Laplacian operator is non-negative, the sharpened image is obtained by superimposing the clear environment image and the Laplacian image. The purpose of image sharpening is to further reduce the grayscale of the central pixel whose grayscale is lower than the average grayscale of the other pixels in the neighborhood; and to further increase the grayscale of the central pixel whose grayscale is higher than the average grayscale of the other pixels in the neighborhood, thereby causing grayscale mutations at the edges and contours of the grayscale image.

[0118] According to a more specific embodiment provided by the present invention, the embodiment of the present invention has the characteristic of fast positioning speed. The visual image captured by the airborne camera is not only fully preprocessed, but the original color image is also grayscaled, which greatly reduces the burden of network model calculation and improves the recognition speed.

[0119] According to a more specific embodiment provided by the present invention, the camera of the drone is connected to a computer deployed with a machine vision library OpenCV (OpenCV is a cross-platform computer vision and machine learning software library released based on the Apache2.0 license) and a Pytorch deep learning framework based on the Python language, and the collected color image is transmitted to the computer for image algorithm calculation. After receiving the color image, the computer uses the average method to grayscale the color image and converts the color image into a grayscale image. The grayscale image is restored using Wiener filtering, and the motion blur and noise in the grayscale image are restored to restore the original clear image. The Laplace operation is used to sharpen the grayscale image after restoration, that is, the clear environment image, so that the edges, contours and details of the image become clearer.

[0120] According to a more specific embodiment provided by the present invention, the embodiment of the present invention has a strong anti-interference characteristic, because the embodiment of the present invention does not rely on satellite positioning information and only collects ground visual images through an onboard camera, and processes the image through a pre-trained neural network to complete the positioning of the drone, thereby improving the anti-interference performance of the drone positioning.

[0121] According to some more specific embodiments provided by the present invention, as shown in the following steps, step 1 is the drone camera installation part, step 2 is the digital image acquisition part, steps 3 to 6 are the digital image preprocessing part, steps 7 to 8 are the digital image feature extraction part, and step 9 is the part that outputs the predicted drone positioning data based on the extracted features.

[0122] Step 1: Install a high-definition camera on the drone. The camera should be installed at an angle that allows the drone to capture images at a 45-degree angle downward while in flight.

[0123] Step 2: Collect the environmental images captured by the camera of the drone during navigation and transmit the captured environmental images to the image processing system inside the drone.

[0124] Step 3: The drone's camera is connected to a computer equipped with the OpenCV machine vision library and the Python-based Pytorch deep learning framework, and the captured environmental images are transmitted to the computer for image algorithm calculations.

[0125] Step 4: After receiving the environmental image, the computer uses the average value method to grayscale the environmental image and convert the environmental image into a grayscale image.

[0126] Step 5: Use Wiener filtering to restore the grayscale image, restore the motion blur and noise in the grayscale image, and restore the original clear image.

[0127] Step 6: Use Laplace operation to sharpen the restored grayscale image to make the edges, contours and details of the image clearer.

[0128] Step 7: Input the grayscale image preprocessed in steps 4 to 6 into the LRNN. First, extract the features of the image through the residual block and output a 1000-dimensional feature vector through the fully connected layer. The purpose is to reduce the dimensionality of the image and reduce redundant data. The features of the image are extracted as a continuous sequence as the input of the LSTM neural network in step 8.

[0129] Step 8: Modify the feature vector output by the fully connected layer into a dimension that can be input into the LSTM neural network, input it into the LSTM neural network to extract features again, extract the spatial continuity information of the image, and obtain a continuous feature vector.

[0130] Step 9: Add two parallel fully connected layers to the output layer of the LSTM neural network, input the continuous feature vectors extracted from the LSTM neural network into the fully connected layers, and regress the feature vectors in the fully connected layers to obtain the actual displacement data of the UAV, which includes the actual displacement direction and actual displacement distance.

[0131] In some specific embodiments of the present invention, Figure 11 、 Figure 12 As shown, the embodiment of the present invention aims to solve the problem that a UAV can still be effectively positioned through an internal system when it cannot obtain external positioning information, and proposes a UAV visual positioning method based on LRNN. The vision-based UAV positioning method of the present invention does not rely on external positioning information. It only uses the visual image captured by the onboard camera and the residual block deployed in the internal system and the LSTM combined neural network to perform visual algorithm operations on the image to complete the positioning of the UAV. It has the characteristics of strong anti-interference, high positioning accuracy, and faster positioning speed, and has certain engineering application value.

[0132] Reference Figure 13 According to the second embodiment of the present invention, the drone positioning device 1300 includes:

[0133] The image acquisition module 1301 is used to acquire a set of images to be identified, which includes multiple environmental images collected during the flight of the drone, and the multiple environmental images have spatial continuity characteristics;

[0134] An image processing module 1302 is configured to extract spatial continuity features of the image set to be identified based on a memory network model to obtain a continuous feature vector of the image set to be identified;

[0135] The positioning calculation module 1303 is used to determine the real-time position information of the UAV based on the preset initial position of the UAV and the continuous feature vector.

[0136] Figure 14 An electronic device 1400 provided by an embodiment of the present invention is shown. The electronic device 1400 includes: a processor 1401, a memory 1402, and a computer program stored in the memory 1402 and executable on the processor 1401. When the computer program is executed, it is used to execute the above-mentioned drone positioning method.

[0137] The processor 1401 and the memory 1402 may be connected via a bus or other means.

[0138] Memory 1402, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs, such as the drone positioning method described in the embodiments of the present invention. Processor 1401 implements the drone positioning method by executing the non-transitory software programs and instructions stored in memory 1402.

[0139] The memory 1402 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and application programs required for at least one function. The data storage area may store data for executing the above-mentioned drone positioning method. In addition, the memory 1402 may include a high-speed random access memory 1402, and may also include a non-volatile memory 1402, such as at least one storage device memory device, a flash memory device or other non-volatile solid-state memory device. In some embodiments, the memory 1402 may optionally include a memory 1402 remotely located relative to the processor 1401, and these remote memories 1402 may be connected to the electronic device 1400 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0140] The non-transient software programs and instructions required to implement the above-mentioned UAV positioning method are stored in the memory 1402. When executed by one or more processors 1401, the above-mentioned UAV positioning method is executed, for example, Figure 1 Steps S101 to S103 of the method, Figure 2 Steps S201 to S202 of the method, Figure 3 Steps S301 to S303 of the method, Figure 4 Steps S401 to S402 of the method, Figure 5Steps S501 to S503 of the method, Figure 6 Steps S601 to S603 of the method, Figure 7 Steps S701 to S703 of the method and Figure 11 、 Figure 12 Steps shown.

[0141] An embodiment of the present invention further provides a computer-readable storage medium storing computer-executable instructions, which are used to execute the above-mentioned drone positioning method.

[0142] In one embodiment, the computer readable storage medium stores computer executable instructions, which are executed by one or more control processors, for example, Figure 1 Steps S101 to S103 of the method, Figure 2 Steps S201 to S202 of the method, Figure 3 Steps S301 to S303 of the method, Figure 4 Steps S401 to S402 of the method, Figure 5 Steps S501 to S503 of the method, Figure 6 Steps S601 to S603 of the method, Figure 7 Steps S701 to S703 of the method and Figure 11 、 Figure 12 Steps shown.

[0143] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, i.e., they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of this embodiment.

[0144] Those skilled in the art will appreciate that all or some of the steps and systems in the method disclosed above can be implemented as software, firmware, hardware, and appropriate combinations thereof. Some physical components or all physical components can be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software can be distributed on a computer-readable medium, and the computer-readable medium can include computer storage media (or non-transitory media) and communication media (or temporary media). As known to those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory, or other memory technology, CD-ROM, digital versatile disks (DVD), or other optical disk storage, magnetic cassettes, magnetic tapes, storage device storage, or other magnetic storage devices, or any other medium that can be used to store desired information and can be accessed by a computer. Furthermore, it is well known to those skilled in the art that communication media generally include computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transmission mechanism, and may include any information delivery medium. It should also be understood that the various implementations provided in the embodiments of the present invention may be arbitrarily combined to achieve different technical effects.

[0145] The above is a specific description of the preferred implementation of the present invention, but the present invention is not limited to the above implementation. Those skilled in the art can also make various equivalent modifications or substitutions under the shared conditions that do not violate the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of the present invention.

Claims

1. A method for positioning a drone, characterized in that: include: During the drone's flight, images of the drone's environment are collected to obtain a set of images to be identified, wherein the set of images to be identified includes multiple environmental images collected during the drone's flight, wherein the multiple environmental images have spatial continuity characteristics, and are used to characterize the evolution of the spatial environment during the drone's flight; wherein the environmental images collected at adjacent time intervals have a common spatial connection area; Extracting spatial continuity features of the set of images to be identified based on a memory network model to obtain a continuous feature vector of the set of images to be identified; wherein the continuous feature vector is used to characterize the motion of the drone during navigation; Inputting the continuous feature vector into the fully connected layer of the memory network model; Performing regression processing on the continuous feature vector based on the fully connected layer to obtain actual displacement data corresponding to the navigation process of the UAV; The real-time position information of the UAV is determined based on the preset initial position and the actual displacement data.

2. The method according to claim 1, characterized in that Before extracting spatial continuity features of the to-be-identified group of images based on the memory network model to obtain continuous feature vectors of the to-be-identified group of images, the method further includes: Acquire a training data set, the training data set including a plurality of preset environment images having spatial continuity characteristics and preset displacement data matching each of the preset environment images; The basic recognition model is optimized and trained based on the training data set to obtain the memory network model.

3. The method according to claim 2, characterized in that The optimizing and training the basic recognition model based on the training data set to obtain the memory network model includes: Iteratively training the basic recognition model based on the training data set, wherein in each round of iterative training, a plurality of the preset environment images are input into the basic recognition model for current round training processing, and current round displacement data corresponding to each preset environment image is obtained; After each round of iterative training, multiple sets of displacement data of the current round are compared with multiple sets of preset displacement data, the recognition accuracy of the basic recognition model is calculated, and the parameters of the basic recognition model are updated; The change of the recognition accuracy after each round of the iterative training is counted, and when the recognition accuracy converges to a constant value, the trained memory network model is obtained.

4. The method according to claim 3, characterized in that The iterative training of the basic recognition model based on the training data set, wherein in each round of the iterative training, multiple preset environment images are input into the basic recognition model for current round training processing, and multiple sets of current round displacement data corresponding to the multiple preset environment images are obtained, including: In the current round of training, multiple sets of current round movement directions and multiple sets of current round movement distances of the drone's perspective between the multiple preset environment images are obtained; Based on the multiple sets of the current wheel movement directions and the multiple sets of the current wheel movement distances, multiple sets of the current wheel displacement data are obtained.

5. The method according to claim 3, characterized in that After each round of iterative training, multiple sets of displacement data of the current round are compared with multiple sets of preset displacement data, the recognition accuracy of the basic recognition model is calculated, and the parameters of the basic recognition model are updated, including: Obtaining displacement estimation errors of each group based on the multiple groups of current-round displacement data and the multiple groups of preset displacement data; Obtaining the number of accurate results in each group of displacement estimation errors that are less than a preset threshold; The recognition accuracy is calculated based on the number of precise results and the total number of displacement data in this round.

6. The method according to any one of claims 1 to 5, characterized in that The extracting spatial continuity features of the to-be-identified group of images based on the memory network model to obtain a continuous feature vector of the to-be-identified group of images includes: Performing a pooling operation on the group of images to be identified to generate reduced feature map data; Performing dimensionality reduction processing on the reduced feature map data to obtain a reduced-dimensionality feature vector; The spatial continuity feature is extracted from the dimensionality reduction feature vector through the memory network model to obtain the continuous feature vector.

7. A UAV positioning device, characterized in that: include: An image acquisition module is configured to capture images of the environment in which the drone is located during navigation to obtain a set of images to be identified. The set of images to be identified includes multiple environmental images captured during navigation, wherein the multiple environmental images have spatial continuity characteristics and are used to characterize the evolution of the spatial environment during navigation. The environmental images captured at adjacent time intervals have a common spatial connection area. an image processing module, configured to extract spatial continuity features of the set of images to be identified based on a memory network model, and obtain a continuous feature vector of the set of images to be identified; wherein the continuous feature vector is used to characterize the motion of the drone during navigation; A positioning operation module is used to input the continuous feature vector into the fully connected layer of the memory network model, perform regression processing on the continuous feature vector based on the fully connected layer, obtain actual displacement data corresponding to the navigation process of the UAV, and determine the real-time position information of the UAV based on a preset initial position and the actual displacement data.

8. An electronic device, characterized in that: include: A memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the drone positioning method according to any one of claims 1 to 6 is implemented.

9. A computer-readable storage medium, characterized in that The storage medium stores a program, and the program is executed by a processor to implement the drone positioning method according to any one of claims 1 to 6.

Images