Unmanned aerial vehicle positioning method, device and equipment based on scene identification matching

Abstract

The invention relates to the technical field of unmanned aerial vehicles, and discloses an unmanned aerial vehicle positioning method, device and equipment based on scene recognition matching, and the method comprises the steps: carrying out the marking of a collected view field image of an unmanned aerial vehicle flight region through a target detection model, obtaining an unmanned aerial vehicle view angle scene data set, taking the data set as training data, and training a scene perception model; marking the satellite image, and constructing a visual knowledge base; training the scene recognition algorithm model by using a visual knowledge base and taking FaceNet as a base model, and carrying out specific scene classification; and carrying out similarity calculation on a scene perception result identified by using the trained scene perception model and an instance under a corresponding scene category in the visual knowledge base to obtain the position information of the unmanned aerial vehicle. According to the invention, a visual identification matching technology is adopted to get rid of the dependence of an unmanned aerial vehicle navigation technology on terminals such as satellites and the like; the image matching execution efficiency is improved, unmanned aerial vehicle positioning is realized, and the unmanned aerial vehicle navigation efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of unmanned aerial vehicles, and in particular to a method, device and equipment for positioning unmanned aerial vehicles based on scene recognition and matching. Background Art

[0002] As one of the core technologies in the process of drones performing flight missions, drone navigation technology is applied in aerial photography, smart agriculture, logistics and distribution, and has important research significance. Drone navigation technology can help drones avoid obstacles, avoid collisions, ensure the safety of flight paths, and thus reduce the risk of accidents. Navigation algorithms can achieve optimal flight path planning, improve flight efficiency, and reduce energy costs. Research on drone navigation technology will help improve the intelligence and reliability of drones and promote the application and development of drones in various industries.

[0003] In the existing technology, the flight positioning and navigation capabilities of drones mainly come from the interaction of real-time satellite information. In the case of short missions and simple mission scenarios, this navigation and positioning technology has high accuracy and high reliability. However, in the process of real mission execution, drones often face various human and natural interferences, which affect the information transmission between satellites and drones, resulting in service unavailability, resulting in the failure of navigation capabilities, and affecting the flight and decision-making of drones. Summary of the invention

[0004] In view of this, the present invention provides a method, device and equipment for positioning a UAV based on scene recognition and matching to solve the problem of navigation capability failure caused by unavailable satellite services.

[0005] In a first aspect, the present invention provides a method for positioning a UAV based on scene recognition and matching, the method comprising:

[0006] The collected UAV flight area field images are annotated using the pre-selected target detection model to obtain the UAV perspective scene dataset;

[0007] The scene perception model is trained using the drone-view scene dataset as training data;

[0008] Use image annotation tools to annotate the collected satellite images and build a visual knowledge base. The altitude of the satellite image is consistent with the flight altitude of the drone. The visual knowledge base includes instance category information and latitude and longitude information.

[0009] Using the visual knowledge base, FaceNet is used as the base model to train the scene recognition algorithm model and classify specific scenes;

[0010] The similarity between the perceived scene results identified by the trained scene perception model and the instances of the corresponding scene category in the visual knowledge base is calculated to obtain the location information of the UAV.

[0011] The present invention uses the selected target detection model to annotate the field of view image of the UAV flight area to establish a UAV perspective scene data set, and uses the UAV perspective scene data set as training data to train a scene perception model, so as to use visual recognition and matching technology to get rid of the dependence of UAV navigation technology on terminals such as satellites, use a visual knowledge base to train the scene recognition algorithm model, use similarity calculation to perform image matching, improve the execution efficiency of image matching, realize UAV positioning according to the similarity calculation result, and improve the efficiency and performance of UAV navigation.

[0012] In an optional implementation, the collected UAV flight area field image is annotated using a pre-selected target detection model, including:

[0013] The yolo perception model is pre-selected as the target detection model;

[0014] Collect the field of view images of the UAV flight area in real time, and form the UAV perspective video stream data based on the real-time collected field of view images of the UAV flight area;

[0015] Perform frame extraction on the video stream data to obtain an image data set;

[0016] Use the labelme image annotation tool to annotate images in the image dataset.

[0017] The present invention forms video stream data according to the real-time collected UAV flight area field of view images, extracts frames from the video stream data to obtain an image data set, so as to retain key frames while reducing the data volume, and annotates the collected UAV flight area field of view images with a pre-selected perception model to facilitate subsequent image analysis.

[0018] In an optional implementation, the scene perception model is trained using the drone perspective scene data set as training data, including:

[0019] Upload the drone-perspective scene image dataset to the perception model project, modify the training script, and add the drone-perspective scene image dataset path to the scene perception model training script;

[0020] The scale, rotation angle and imaging pitch angle of the scene to be identified are used to augment the image data of the scene from the drone's perspective;

[0021] The scene perception model is trained using the original loss function.

[0022] The present invention adds the path of the drone-perspective scene image dataset to the scene perception model training script, adjusts the scale, rotation angle, and imaging pitch angle, etc. to improve the generalization ability of the model, and uses the original loss function to train the scene perception model to ensure the stability and high compatibility of the training process.

[0023] In an optional implementation, an image annotation tool is used to annotate the collected satellite images to construct a visual knowledge base, including:

[0024] According to the flight altitude of the UAV, collect satellite images that are consistent with the flight altitude of the UAV;

[0025] Use the labelme image annotation tool to annotate the collected satellite images with categories and instance information to obtain annotation information, which includes the bounding box coordinates of the target in the image.

[0026] The scene in the image is cropped according to the bounding box coordinates in the image where the target is located to obtain the target slice, record the category information and longitude and latitude information of the instance target, and build a visual knowledge base.

[0027] The present invention collects satellite images that are highly consistent with the flight altitude of the drone, so that the size of the drone observation target and the target in the satellite image are close. The labelme image annotation tool is used to annotate the collected satellite images with category and instance information to record the instance category and longitude and latitude information, and a visual knowledge base is constructed, thereby facilitating image matching using the visual knowledge base.

[0028] In an optional implementation, the scene recognition algorithm model is trained using the visual knowledge base and FaceNet as a base model, including:

[0029] Extract the FaceNet training data set from the visual knowledge base according to the FaceNet network model training data requirements;

[0030] Use the FaceNet training data set to train the scene recognition algorithm model;

[0031] The Adam optimizer is used to optimize the training process of the scene recognition algorithm model.

[0032] The present invention trains a scene recognition algorithm model by extracting a FaceNet training data set from a visual knowledge base to improve the robustness and accuracy of the scene recognition algorithm model. The Adam optimizer is used to optimize the training process of the scene recognition algorithm model to dynamically adjust the model training and improve the convergence and stability of the model.

[0033] In an optional implementation, similarity calculation is performed between the perceived scene result identified by the trained scene perception model and the instance of the corresponding scene category in the visual knowledge base to obtain the location information of the drone, including:

[0034] The trained scene perception model is used to identify scene targets, and the identified targets are intercepted to obtain scene slices;

[0035] Calculate the similarity between the scene slice and the visual knowledge base slice, and select the knowledge base slice with the highest similarity to the scene slice;

[0036] The location information of the drone is determined based on the annotation information corresponding to the selected knowledge base slice.

[0037] The present invention utilizes a trained scene perception model to identify scene targets, calculates similarity between captured scene slices and visual knowledge base slices, and uses contrastive learning to convert image matching tasks into slice learning tasks, thereby greatly improving the execution efficiency of matching tasks. UAV positioning is achieved based on the annotation information corresponding to the selected knowledge base slices for UAV navigation.

[0038] In a second aspect, the present invention provides a UAV positioning device based on scene recognition and matching, the device comprising:

[0039] The annotation module is used to annotate the collected UAV flight area field images using a pre-selected target detection model to obtain a UAV perspective scene dataset;

[0040] The first model training module is used to train the scene perception model using the drone perspective scene data set as training data;

[0041] A construction module is used to annotate the collected satellite images using image annotation tools and build a visual knowledge base. The altitude of the satellite image is consistent with the flight altitude of the drone. The visual knowledge base includes instance category information and latitude and longitude information;

[0042] The second model training module is used to train the scene recognition algorithm model using the visual knowledge base and FaceNet as the base model to classify specific scenes;

[0043] The positioning module is used to calculate the similarity between the perceived scene results identified by the trained scene perception model and the instances under the corresponding scene category in the visual knowledge base to obtain the location information of the drone.

[0044] In a third aspect, the present invention provides a computer device, comprising: a memory and a processor, the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the drone positioning method based on scene recognition and matching of the above-mentioned first aspect or any corresponding embodiment thereof by executing the computer instructions.

[0045] In a fourth aspect, the present invention provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the drone positioning method based on scene recognition and matching of the above-mentioned first aspect or any corresponding embodiment thereof.

[0046] In a fifth aspect, the present invention provides a computer program product, comprising computer instructions, which are used to enable a computer to execute the drone positioning method based on scene recognition and matching of the above-mentioned first aspect or any corresponding embodiment thereof. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0048] Figure 1 is a flow chart of a method for positioning a UAV based on scene recognition and matching according to an embodiment of the present invention;

[0049] Figure 2 is a schematic diagram of drone scene detection according to an embodiment of the present invention;

[0050] Figure 3 is a schematic diagram of a process of UAV visual navigation based on scene recognition and matching according to an embodiment of the present invention;

[0051] Figure 4 is a schematic diagram of scene matching according to an embodiment of the present invention;

[0052] Figure 5 is a structural block diagram of a UAV positioning device based on scene recognition and matching according to an embodiment of the present invention;

[0053] Figure 6 It is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0054] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0055] According to an embodiment of the present invention, an embodiment of a UAV positioning method based on scene recognition and matching is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0056] In this embodiment, a method for positioning a UAV based on scene recognition and matching is provided. Figure 1 is a flow chart of a method for positioning a UAV based on scene recognition and matching according to an embodiment of the present invention. Figure 1 As shown, the process includes the following steps:

[0057] Step S101, annotating the collected drone flight area field image with a pre-selected target detection model to obtain a drone perspective scene dataset.

[0058] In an embodiment of the present invention, field of view images of the flight area during the flight of the drone are collected, a target detection model is pre-selected, and the pre-selected target detection model algorithm input format is used as a data production target to produce a drone perspective scene data set.

[0059] Step S102: training a scene perception model using the drone perspective scene data set as training data.

[0060] In an embodiment of the present invention, the prepared drone perspective scene data set is used as training data to train the scene perception model.

[0061] Step S103: annotate the collected satellite images using an image annotation tool to build a visual knowledge base.

[0062] In an embodiment of the present invention, the altitude of the satellite image is consistent with the flight altitude of the drone, and the visual knowledge base includes instance category information and latitude and longitude information.

[0063] The height of the satellite image is set according to the actual flight altitude of the drone, and the two are kept at the same height, so that the size of the target observed by the drone is close to that of the target in the satellite image. Image annotation tools such as Labelme are used to perform detailed classification and instance information on the collected satellite images to build a visual knowledge base.

[0064] Step S104, using the visual knowledge base and FaceNet as the base model to train the scene recognition algorithm model and perform specific scene classification.

[0065] In an embodiment of the present invention, a visual knowledge base is used to organize the visual knowledge base according to the training data requirements of the FaceNet network model, and a training data set for FaceNet is produced to train the scene recognition algorithm model to achieve detailed classification of specific scenes.

[0066] Step S105 , performing similarity calculation on the perceived scene result identified by using the trained scene perception model and the instance under the corresponding scene category in the visual knowledge base to obtain the position information of the UAV.

[0067] In an embodiment of the present invention, when a drone performs a flight mission, a trained scene perception model is used to perform scene recognition targets to obtain perceived scene results, and similarity calculations are performed between the perceived scene results and instances under corresponding scene categories in a visual knowledge base to obtain specific instances of the drone's field of view content scene and obtain the current drone's location information.

[0068] The UAV positioning method based on scene recognition and matching provided in this embodiment uses the selected target detection model to annotate the field of view image of the UAV flight area to establish a UAV perspective scene data set, and uses the UAV perspective scene data set as training data to train the scene perception model, so as to use visual recognition and matching technology to get rid of the dependence of UAV navigation technology on satellite terminals, etc., use the visual knowledge base to train the scene recognition algorithm model, use the similarity calculation method to perform image matching, improve the execution efficiency of image matching, realize UAV positioning according to the similarity calculation result, and improve the efficiency and performance of UAV navigation.

[0069] In this embodiment, a method for positioning a UAV based on scene recognition and matching is provided, and the process includes the following steps:

[0070] Step S201: annotate the collected drone flight area field image using a pre-selected target detection model to obtain a drone perspective scene dataset.

[0071] Specifically, the above step S201 includes:

[0072] Step S2011, pre-select the YOLO perception model as the target detection model.

[0073] Step S2012, real-time acquisition of the drone flight area field image, and the drone perspective video stream data is formed according to the real-time acquired drone flight area field image, and the video stream data is subjected to frame extraction processing to obtain an image data set.

[0074] Step S2013, performing frame extraction processing on the video stream data to obtain an image data set.

[0075] Step S2014: label the images in the image dataset using the labelme image labeling tool.

[0076] In the embodiment of the present invention, the target detection model yolov5m.pt pre-trained on the COCO dataset is downloaded, and the yolov5 perception model is selected as the target detection model. The drone carries a photoelectric sensor, which can capture and collect images in real time, collect the field of view images of the drone's flight area in real time to form a video stream, and save the video stream data as video data to initially form the drone's perspective video stream data.

[0077] Use ffmpeg or opencv and other frame extraction tools to extract frames from the video stream data to form an image dataset. Use labelme and other image annotation tools to annotate the images in the image dataset based on the selected perception model training data format requirements to form a drone scene dataset. The specific pouring category is determined according to the drone shooting scene, and the annotation target has positioning properties, such as Figure 2 As shown, common targets include airports, ports, roundabouts, and bridges.

[0078] Video stream data is formed according to the real-time collected UAV flight area field of view images, and frames are extracted from the video stream data to obtain image data sets, so as to retain key frames while reducing the amount of data. The collected UAV flight area field of view images are annotated with a pre-selected perception model to facilitate subsequent image analysis.

[0079] Step S202: training a scene perception model using the drone perspective scene data set as training data.

[0080] Specifically, the above step S202 includes:

[0081] Step S2021, upload the drone perspective scene image dataset to the perception model project, modify the training script, and add the drone perspective scene image dataset path to the scene perception model training script.

[0082] Step S2022: augment the drone's perspective scene image data using the scale, rotation angle, and imaging pitch angle of the scene to be identified.

[0083] Step S2023, training the scene perception model using the original loss function.

[0084] In the embodiment of the present invention, the drone perspective scene image dataset is uploaded to the perception model project, and the relevant training script is modified. The modification is determined according to the drone dataset, and the modification content includes the path of the prepared dataset, the number of categories to be detected, and which categories are included, etc. The drone perspective scene image dataset path is added to the scene perception model training script. According to the number of scene categories to be detected, the number of target categories in the training script is modified, and the perception model input size is set to 640*640*3.

[0085] Since the UAV has flight altitude, flight angle, imaging distortion and other conditions during flight, data augmentation based on factors such as the scale, rotation angle, and imaging pitch angle of the scene to be identified is added during the training of the perception model. The details are as follows:

[0086]

[0087] Where (x′, y′) is the pixel position of the scaled image I′, and the corresponding pixel in the original image I is (x, y), where x = x′ / s, y = y′ / s, s is the image scaling factor, and w ij are the weights of bilinear interpolation, It means to round x down. The specific calculation formula for rounding y down is as follows:

[0088]

[0089] Image rotation augmentation is mainly implemented around the image center point (cx, cy). Assuming the image size is H×W, the image center point (cx, cy) satisfies cx=W / 2, cy=H / 2, and the random rotation angle θ is used. The pixel transformation relationship before (x, y) and after (x′, y′) rotation is as follows:

[0090]

[0091] Assuming that the drone is perpendicular to the ground plane, the mathematical relationship between the original pixel (x, y) and the transformed pixel (x′, y′) at different aircraft sensor pitch angles β can be approximately summarized as:

[0092] x′=x(4)

[0093] y′=ycosβ-sinβ(5)

[0094] In the scene perception model training process, the scene perception model is trained using a loss function, and the loss function includes classification loss, coordinate loss, and confidence loss, etc. In the embodiment of the present invention, the original loss function is used, and the gradient backpropagation model update strategy and the common model training method are used to train the scene perception model.

[0095] By adding the path of the drone's perspective scene image dataset to the scene perception model training script, adjusting the scale, rotation angle, and imaging pitch angle, etc., the generalization ability of the model is improved. The scene perception model is trained using the original loss function to ensure the stability and high compatibility of the training process.

[0096] Step S203: annotate the collected satellite images using an image annotation tool to build a visual knowledge base.

[0097] Specifically, the above step S203 includes:

[0098] Step S2031, based on the flight altitude of the UAV, collect satellite images that are highly consistent with the flight altitude of the UAV.

[0099] Step S2031, use the labelme image annotation tool to annotate the collected satellite image with category and instance information to obtain annotation information, where the annotation information includes the coordinates of the bounding box in the image where the target is located.

[0100] Step S2031, cropping the scene in the image according to the bounding box coordinates of the target in the image to obtain target slices, recording the category information and longitude and latitude information of the instance target, and building a visual knowledge base.

[0101] In the embodiment of the present invention, the data map is downloaded and produced, and the height of the satellite image is set according to the actual flight height of the drone, and the heights of the two are kept consistent, so that the target observed by the drone and the target in the satellite image have similar sizes. After setting the resolution, the image of the possible flight area of ​​the drone is downloaded, and the latitude and longitude coordinate information of the image is retained.

[0102] Use labelme and other image annotation tools to annotate the collected satellite images with detailed category and instance information, and use the longitude and latitude information of the target center point coordinates as the target's longitude and latitude information. Satellite images can query the corresponding longitude and latitude information based on the pixel coordinates. According to the location of the target in the image in the labelme annotation information, the scene in the image is cropped according to the target bounding box information, and the pixel area containing the target is selected for saving. The target slices generated by cropping record the detailed category information and longitude and latitude information of the corresponding instance target. At this point, the construction of the visual knowledge base is completed.

[0103] Among them, the standard information includes the bounding box coordinates of the target in the image, such as the upper left corner point (x min ,y min ) and the lower right corner (x max ,y max ), the relationship between the cropped image and the original image satisfies the following formula:

[0104] cropped_img = image[y min :y max ,x min :x max ].

[0105] By collecting satellite images that are highly consistent with the UAV's flight altitude, the size of the UAV's observed targets and the targets in the satellite images are made close. The collected satellite images are annotated with category and instance information using the labelme image annotation tool to record instance categories and longitude and latitude information, and a visual knowledge base is constructed to facilitate image matching using the visual knowledge base.

[0106] Step S204, using the visual knowledge base and FaceNet as the base model to train the scene recognition algorithm model and perform specific scene classification.

[0107] Specifically, the above step S204 includes:

[0108] Step S2041, extracting a FaceNet training data set from a visual knowledge base according to the FaceNet network model training data requirements.

[0109] Step S2042: Use the FaceNet training data set to train the scene recognition algorithm model.

[0110] Step S2043, using the Adam optimizer to optimize the training process of the scene recognition algorithm model.

[0111] In the embodiment of the present invention, a visual knowledge base is used, and the visual knowledge base is sorted according to the training data requirements of the FaceNet network model to produce a training data set for FaceNet. In the FaceNet network model setting, MobileNet V1 is used as the feature extractor of the input image I to obtain the embedding vector f(I) of the input image, where the dimension of the embedding vector is set to 128.

[0112] The training loss function of the FaceNet network model adopts the triplet loss function (Triplet Loss), which is defined as follows:

[0113] A triplet usually consists of three input images: an anchor image, a positive sample image, and a negative sample image. The anchor image and the positive sample image belong to the same scene, while the negative sample image belongs to a different scene. The purpose of the triplet loss is to make the distance between the anchor and the positive sample image smaller than the distance between the anchor and the negative sample, and at least smaller by a margin value α:

[0114]

[0115] in, is the i-th anchor image, is the i-th positive sample image, is the i-th negative sample image, [] + Indicates a positive value.

[0116] During the network training process, the Adam optimizer was used, the FaceNet network model input size was set to 160*160*3, and the number of training cycles was set to 100.

[0117] The scene recognition algorithm model is trained by extracting the FaceNet training data set from the visual knowledge base to improve the robustness and accuracy of the scene recognition algorithm model. The Adam optimizer is used to optimize the training process of the scene recognition algorithm model to dynamically adjust the model training and improve the convergence and stability of the model.

[0118] In step S205, similarity calculation is performed between the scene perception result identified by the trained scene perception model and the instance of the corresponding scene category in the visual knowledge base to obtain the position information of the UAV.

[0119] Specifically, the above step S205 includes:

[0120] Step S2051, using the trained scene perception model to identify scene targets, intercepting the identified targets to obtain scene slices.

[0121] Step S2052: perform similarity calculation on the scene slice and the visual knowledge base slice, and select the knowledge base slice with the highest similarity to the scene slice.

[0122] Step S2053, determining the location information of the drone based on the annotation information corresponding to the selected knowledge base slice.

[0123] In an embodiment of the present invention, when the UAV performs a flight mission, the trained scene perception model is first used to complete the identification of relevant scene targets in the field of view. The category of the scene target is consistent with the category marked and set during model training, for example, there is an airport target in the field of view.

[0124] The target recognized by the perception model is intercepted, and the slice is scaled while maintaining the aspect ratio. The size of the long side of the scaled slice is 160. The scaled image is then pasted to a 160*160 blank background image to form a uniform-sized scene slice.

[0125] According to the category information of the current slice, the visual images of the corresponding large category in the visual knowledge base are selected, and the scene slices and the visual knowledge base slices are compared and learned based on the trained FaceNet to select the knowledge base slice with the highest similarity.

[0126] Based on the annotation information of the filtered knowledge base slice with the maximum similarity, the information is returned to the drone. In addition to the detailed category of the scene itself, the information also includes coordinate information such as the longitude and latitude of the target, so that the current approximate location information of the drone can be inferred.

[0127] The UAV positioning method based on scene recognition and matching provided in this embodiment uses a trained scene perception model to identify scene targets, calculates the similarity between the captured scene slices and the visual knowledge base slices, and uses contrastive learning to convert the image matching task into a slice learning task, thereby greatly improving the execution efficiency of the matching task, and realizes UAV positioning according to the annotation information corresponding to the selected knowledge base slices to perform UAV navigation.

[0128] like Figure 3 As shown, Figure 3 This is a flowchart of UAV visual navigation based on scene recognition and matching. The perception slices are extracted from the drawings taken by the UAV, input into the perception model, output the perception results, and the results are cropped. Scene slices are made based on satellite images, and a visual knowledge base is constructed to form visual knowledge. Based on the comparison of the scene knowledge base, if a target is found in the visual library, the target position is returned, and the UAV position information is inverted based on the scene position information. Figure 4 As shown in the figure, when performing scene matching, if the target is a roundabout, embedding vector A is extracted from the visual knowledge base of roundabout A and position A, and embedding vector B is extracted from the visual knowledge base of roundabout B and position B. If the similarity between embedding vector A and the target embedding vector is lower than the similarity between embedding vector B and the target embedding vector, the channel recognizes the target as roundabout B and position B.

[0129] The embodiments of the present invention have the following beneficial effects:

[0130] (1) The use of visual recognition and matching technology for embedded computing platforms eliminates the reliance of drone navigation technology on satellite terminals.

[0131] (2) The image matching task is converted into a slice learning task by using contrastive learning, which greatly improves the execution efficiency of the matching task and the information transmission efficiency of navigation;

[0132] (3) Using satellite images to assist in identification and matching tasks greatly reduces the reliance on terminals such as drones during data preparation, and lowers the threshold for the development of navigation technology;

[0133] (4) A visual scene knowledge base was constructed, and the visual slices in the knowledge base were sorted and divided according to scene category information, thereby improving the execution efficiency of downstream tasks.

[0134] In this embodiment, a drone positioning device based on scene recognition and matching is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and will not be repeated hereafter. As used below, the term "module" may be a combination of software and / or hardware that implements a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0135] This embodiment provides a UAV positioning device based on scene recognition and matching, such as Figure 5 As shown, including:

[0136] The labeling module 501 is used to label the collected drone flight area field of view images using a pre-selected target detection model to obtain a drone perspective scene data set.

[0137] The first model training module 502 is used to train the scene perception model using the drone perspective scene data set as training data.

[0138] The construction module 503 is used to annotate the collected satellite images using an image annotation tool to construct a visual knowledge base. The height of the satellite image is consistent with the flight height of the drone. The visual knowledge base includes instance category information and latitude and longitude information.

[0139] The second model training module 504 is used to train the scene recognition algorithm model using the visual knowledge base and FaceNet as the base model to perform specific scene classification.

[0140] The positioning module 505 is used to perform similarity calculation between the scene perception results identified by the trained scene perception model and the instances under the corresponding scene category in the visual knowledge base to obtain the location information of the drone.

[0141] In some optional implementations, the marking module 501 includes:

[0142] The model selection unit is used to pre-select the YOLO perception model as the target detection model.

[0143] The first acquisition unit is used to acquire the field of view image of the UAV flight area in real time, and form the UAV perspective video stream data according to the real-time acquired field of view image of the UAV flight area.

[0144] The frame extraction unit is used to extract frames from the video stream data to obtain an image data set.

[0145] The first labeling unit is used to label images in the image dataset using the labelme image labeling tool.

[0146] In some optional implementations, the first model training module 502 includes:

[0147] The data uploading unit is used to upload the drone's perspective scene image dataset to the perception model project, modify the training script, and add the drone's perspective scene image dataset path to the scene perception model training script.

[0148] The data augmentation unit is used to augment the image data of the scene from the drone's perspective by using the scale, rotation angle and imaging pitch angle of the scene to be identified.

[0149] The first model training unit is used to train the scene perception model using the original loss function.

[0150] In some optional implementations, the construction module 503 includes:

[0151] The second acquisition unit is used to acquire satellite images that are highly consistent with the flight altitude of the UAV according to the flight altitude of the UAV.

[0152] The second labeling unit is used to label the collected satellite images with categories and instance information using the labelme image labeling tool to obtain labeling information, where the labeling information includes the coordinates of a bounding box in the image where the target is located.

[0153] The construction unit is used to crop the scene in the image according to the bounding box coordinates in the image where the target is located, obtain the target slice, record the category information and longitude and latitude information of the instance target, and build a visual knowledge base.

[0154] In some optional implementations, the second model training module 504 includes:

[0155] The extraction unit is used to extract the FaceNet training data set from the visual knowledge base according to the FaceNet network model training data requirement.

[0156] The second model training unit is used to train the scene recognition algorithm model using the FaceNet training data set.

[0157] The optimization unit is used to optimize the training process of the scene recognition algorithm model using the Adam optimizer.

[0158] In some optional implementations, the positioning module 505 includes:

[0159] The interception unit is used to identify scene targets using the trained scene perception model, intercept the identified targets, and obtain scene slices.

[0160] The similarity calculation unit is used to calculate the similarity between the scene slice and the visual knowledge base slice, and select the knowledge base slice with the highest similarity to the scene slice.

[0161] The determination unit is used to determine the location information of the drone based on the annotation information corresponding to the selected knowledge base slice.

[0162] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.

[0163] The drone positioning device based on scene recognition and matching in this embodiment is presented in the form of a functional unit, where the unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0164] The embodiment of the present invention also provides a computer device having the above Figure 5 The drone positioning device based on scene recognition and matching is shown.

[0165] See also Figure 6 , Figure 6 is a schematic diagram of the structure of a computer device provided by an optional embodiment of the present invention, such as Figure 6 As shown, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process the instructions executed in the computer device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as, a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 6 A processor 10 is taken as an example.

[0166] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.

[0167] The memory 20 stores instructions executable by at least one processor 10, so that at least one processor 10 executes the method shown in the above embodiment.

[0168] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0169] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.

[0170] The computer device also includes an input device 30 and an output device 40. The processor 10, the memory 20, the input device 30 and the output device 40 may be connected via a bus or other means. Figure 6 The example of connecting through bus is taken in the following.

[0171] The input device 30 can receive input digital or character information and generate key signal input related to user settings and function control of the computer device, such as a touch screen, etc. The output device 40 can include a display device, etc.

[0172] The embodiment of the present invention also provides a computer-readable storage medium. The method according to the embodiment of the present invention can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium through a network download, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state hard disk, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.

[0173] A part of the present invention may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present invention through the operation of the computer. Those skilled in the art should understand that the existence of the computer program instruction in a computer-readable medium includes, but is not limited to, a source file, an executable file, an installation package file, etc., and accordingly, the way in which the computer program instruction is executed by the computer includes, but is not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Here, the computer-readable medium may be any available computer-readable storage medium or communication medium accessible to the computer.

[0174] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations all fall within the scope of the present application.

Claims

1. A method for positioning a UAV based on scene recognition and matching, characterized in that: The method comprises: The collected UAV flight area field images are annotated using the pre-selected target detection model to obtain the UAV perspective scene dataset; Using the drone perspective scene data set as training data, training the scene perception model; Annotate the collected satellite images using an image annotation tool to build a visual knowledge base, where the altitude of the satellite images is consistent with the flight altitude of the drone, and the visual knowledge base includes instance category information and latitude and longitude information; Using the visual knowledge base, the scene recognition algorithm model is trained with FaceNet as the base model to classify specific scenes; The similarity between the perceived scene results identified by the trained scene perception model and the instances of the corresponding scene category in the visual knowledge base is calculated to obtain the location information of the drone.

2. The method according to claim 1, characterized in that The method of labeling the collected drone flight area field image using the pre-selected target detection model includes: The yolo perception model is pre-selected as the target detection model; Collect the UAV flight area field image in real time, and form the UAV perspective video stream data based on the UAV flight area field image collected in real time; Performing frame extraction processing on the video stream data to obtain an image data set; Use the labelme image annotation tool to annotate images in the image dataset.

3. The method according to claim 1, characterized in that: The method of training the scene perception model using the drone perspective scene data set as training data includes: Upload the drone-perspective scene image dataset to the perception model project, modify the training script, and add the drone-perspective scene image dataset path to the scene perception model training script; The scale, rotation angle and imaging pitch angle of the scene to be identified are used to augment the image data of the scene from the drone's perspective; The scene perception model is trained using the original loss function.

4. The method according to claim 1, characterized in that: The method of using an image annotation tool to annotate the collected satellite images and construct a visual knowledge base includes: According to the flight altitude of the UAV, collect satellite images that are consistent with the flight altitude of the UAV; Use the labelme image annotation tool to annotate the collected satellite images with categories and instance information to obtain annotation information, where the annotation information includes the coordinates of the bounding box in the image where the target is located; The scene in the image is cropped according to the bounding box coordinates in the image where the target is located to obtain target slices, record the category information and longitude and latitude information of the instance target, and build a visual knowledge base.

5. The method according to claim 1, characterized in that The method of using the visual knowledge base and taking FaceNet as the base model to train the scene recognition algorithm model includes: Extract the FaceNet training data set from the visual knowledge base according to the FaceNet network model training data requirements; Use the FaceNet training data set to train the scene recognition algorithm model; The Adam optimizer is used to optimize the training process of the scene recognition algorithm model.

6. The method according to claim 1, characterized in that The method of calculating the similarity between the scene perception result identified by the trained scene perception model and the instance of the corresponding scene category in the visual knowledge base to obtain the position information of the drone includes: The trained scene perception model is used to identify scene targets, and the identified targets are intercepted to obtain scene slices; Calculate the similarity between the scene slice and the visual knowledge base slice, and select the knowledge base slice with the highest similarity to the scene slice; The location information of the drone is determined based on the annotation information corresponding to the selected knowledge base slice.

7. A drone positioning device based on scene recognition and matching, characterized in that: The device comprises: The annotation module is used to annotate the collected UAV flight area field images using a pre-selected target detection model to obtain a UAV perspective scene data set; A first model training module is used to train a scene perception model using the drone perspective scene data set as training data; A construction module is used to annotate the collected satellite images using an image annotation tool to construct a visual knowledge base, wherein the altitude of the satellite image is consistent with the flight altitude of the drone, and the visual knowledge base includes instance category information and latitude and longitude information; A second model training module is used to train a scene recognition algorithm model using the visual knowledge base and FaceNet as a base model to perform specific scene classification; The positioning module is used to calculate the similarity between the perceived scene results identified by the trained scene perception model and the instances under the corresponding scene category in the visual knowledge base to obtain the location information of the drone.

8. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the drone positioning method based on scene recognition and matching as described in any one of claims 1 to 6 by executing the computer instructions.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the drone positioning method based on scene recognition and matching according to any one of claims 1 to 6.

10. A computer program product, characterized in that The method comprises computer instructions for causing a computer to execute the method for positioning a UAV based on scene recognition and matching as described in any one of claims 1 to 6.

Images