Unmanned aerial vehicle ground target positioning method and system based on multi-modal information alignment

Abstract

The invention discloses an unmanned aerial vehicle ground target positioning method and system based on multi-modal information alignment. Multi-modal information is adopted to carry out super voxel segmentation, and matching alignment is carried out on a satellite reference map and an aerial image, so that ground target positioning is carried out according to the unmanned aerial vehicle. And respectively carrying out super voxel segmentation on a plurality of two-dimensional feature maps in a satellite feature base map formed by the satellite reference map. And after arranging the plurality of loss values from small to large, combining the two-dimensional regions segmented by the super voxels in the feature direction to obtain a three-dimensional region. Different features are segmented into areas with similar positions, which means that the features have the same effect on detecting the positions, and the areas can be used as a three-dimensional area for detection. The features are subjected to judgment of the same effect according to the influence from large to small, and the best representation importance region can be subjected to relation construction. The combination of the three-dimensional area and the two-dimensional area enables the subsequent matching to be more accurate, so that the target positioning is more accurate.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method and system for positioning a ground target of an unmanned aerial vehicle based on multi-modal information alignment. Background Art

[0002] At present, due to the current use of supervoxel segmentation of two-dimensional images or three-dimensional point clouds, and then matching them. The main use is to segment similar points under the position relationship between images or point clouds. At the same time, the feature map of extracted features can also be segmented by supervoxel and then position matched. However, when multiple feature maps representing different features are segmented by supervoxel separately, the relationship between the features is not obtained, and the effect of the features on the positioning of the detection target is not detected, which will lead to inaccurate matching and target detection. Summary of the invention

[0003] The purpose of the present invention is to provide a method and system for UAV ground target positioning based on multimodal information alignment to solve the above-mentioned problems existing in the prior art.

[0004] In a first aspect, an embodiment of the present invention provides a method for positioning a ground target of an unmanned aerial vehicle based on multimodal information alignment, comprising:

[0005] Obtaining multimodal information; the multimodal information includes satellite reference images, aerial images, drone altitude, drone attitude and aerial elevation information; the aerial elevation information is the elevation information of the corresponding area of ​​the aerial image;

[0006] Constructing a satellite feature base map according to the satellite reference map and the aerial elevation information; the satellite feature base map includes a plurality of two-dimensional feature maps;

[0007] Based on the altitude, attitude and aerial image of the UAV, scene feature extraction is performed on the aerial image to obtain a scene feature map;

[0008] Performing supervoxel segmentation on the multiple two-dimensional feature maps in the satellite feature base map to obtain supervoxel segmentation feature maps;

[0009] Based on the supervoxel segmentation feature map, different three-dimensional regions are selectively constructed according to the two-dimensional feature map to obtain the segmentation satellite feature map;

[0010] Based on the segmented satellite feature map and scene feature map, ground scene matching and target positioning are performed.

[0011] Optionally, the supervoxel-based segmentation feature map selectively constructs different three-dimensional regions according to the two-dimensional feature map to obtain the segmented satellite feature map, including:

[0012] Adjusting the arrangement of the two-dimensional feature map in the supervoxel segmentation feature map to obtain a supervoxel feature map; the supervoxel feature map is a three-dimensional feature map;

[0013] The supervoxel feature map includes a plurality of segmented two-dimensional regions; the segmented two-dimensional regions are regions obtained by segmenting the two-dimensional feature map in the supervoxel segmentation feature map;

[0014] If the area of ​​the segmented two-dimensional region is greater than the segmentation threshold, the segmented two-dimensional region is marked as a two-dimensional region to be fused;

[0015] Perform three-dimensional fusion on multiple two-dimensional regions to be fused to obtain multiple fused three-dimensional regions;

[0016] Multiple fused three-dimensional areas are used to replace the corresponding two-dimensional areas to be fused to form a segmented satellite feature map.

[0017] Optionally, adjusting the arrangement of the two-dimensional feature map in the supervoxel segmentation feature map to obtain the supervoxel feature map includes:

[0018] Inputting the two-dimensional feature maps in the supervoxel segmentation feature map into the trained first object detection network respectively to obtain multiple loss values; one loss value corresponds to one two-dimensional feature map;

[0019] Arrange the multiple loss values ​​from small to large to obtain a supervoxel feature map;

[0020] Obtain an important two-dimensional feature map; the important two-dimensional feature map is a two-dimensional feature map whose loss value is less than the loss value of other two-dimensional feature maps.

[0021] Optionally, the three-dimensional fusion of the multiple two-dimensional regions to be fused to obtain multiple fused three-dimensional regions includes:

[0022] Based on the two-dimensional region to be fused, a plurality of labeled regions are numbered according to the position; the positions of the two-dimensional regions to be fused with the same number overlap; the supervoxel segmentation feature maps corresponding to the two-dimensional regions to be fused with the same number are different;

[0023] Obtain multiple first labeled regions; the first labeled regions are regions with the same label and smaller than other labels in the supervoxel segmentation feature map; one supervoxel segmentation feature map corresponds to one first labeled region;

[0024] A histogram is established with the arranged supervoxel segmentation feature map as the abscissa and the area of ​​the first labeled region as the ordinate, thereby obtaining a histogram of the first labeled region;

[0025] Based on the first labeled region histogram, determining whether a three-dimensional region is formed, and obtaining a plurality of first three-dimensional fusion regions;

[0026] The above method is repeated to judge multiple numbered regions with the same number for multiple times, so as to obtain multiple fused three-dimensional regions; the multiple fused three-dimensional regions include multiple first three-dimensional fused regions.

[0027] Optionally, judging whether a three-dimensional region is formed based on the first labeled region histogram to obtain a plurality of first three-dimensional fused regions includes:

[0028] Obtaining a fusion set; the fusion set is an empty set;

[0029] Add the values ​​corresponding to the important two-dimensional feature maps in the histogram of the first labeled area to the fusion set;

[0030] Obtaining adjacent values; the adjacent values ​​are values ​​in the first numbered region histogram that are not added to the fusion set and are adjacent to the values ​​that are most recently added to the fusion set in the horizontal coordinate;

[0031] Determine whether the absolute value of the value newly added to the fusion set and the adjacent value is less than the fusion threshold;

[0032] If the absolute value of the value newly added to the fusion set and the adjacent value is less than or equal to the fusion threshold, the adjacent ordinate that has not been added to the fusion set is added to the fusion set;

[0033] The corresponding two-dimensional region to be fused in the fusion set is constructed into a three-dimensional region to obtain a first three-dimensional fusion region;

[0034] If the absolute value of the value newly added to the fusion set and the adjacent value is greater than the fusion threshold, a new fusion set is obtained, and the adjacent vertical coordinates not added to the fusion set are added to the new fusion set. The above method is repeated to obtain multiple first three-dimensional fusion areas; the new fusion set is an empty set.

[0035] Optionally, the two-dimensional region to be fused is numbered by position to obtain a plurality of numbered regions, including:

[0036] Label the two-dimensional regions to be fused in the important two-dimensional feature map to obtain multiple important two-dimensional regions;

[0037] The overlapping area of ​​the important two-dimensional region and the two-dimensional region to be fused except the important two-dimensional feature map is calculated to obtain the overlapping area;

[0038] Divide the overlap area by the area of ​​the important two-dimensional region to obtain the overlap value;

[0039] If the overlap value is greater than the overlap threshold, the two-dimensional region to be fused corresponding to the overlap value is marked with the same label as the important two-dimensional region.

[0040] Optionally, the labels of the two-dimensional regions to be fused at different positions in each important two-dimensional feature map are different.

[0041] Optionally, the extracting scene features of the aerial image based on the altitude of the drone, the attitude of the drone and the aerial image to obtain a scene feature map includes:

[0042] Ortho-rectifying the aerial image according to the altitude and attitude of the drone to obtain a corrected aerial image;

[0043] The rectified aerial image is passed through the directional gradient histogram operator to obtain the directional gradient histogram;

[0044] The rectified aerial image is passed through the LBP operator to obtain the LBP feature map;

[0045] The size of the directional gradient histogram and the LBP feature map is the same;

[0046] Superimpose the directional gradient histogram and the LBP feature map to obtain a superimposed feature map;

[0047] The directional gradient histogram and the LBP feature map in the superposition feature map are respectively subjected to supervoxel segmentation and constituted into different three-dimensional regions to obtain a scene feature map.

[0048] Optionally, performing ground scene matching and target positioning based on segmenting the satellite feature map and the scene feature map includes:

[0049] Matching the segmented satellite feature map and the scene feature map to obtain a matching position;

[0050] The segmented satellite feature map and the scene feature map corresponding to the matching position are located through the second target detection network to obtain the target position.

[0051] In a second aspect, an embodiment of the present invention provides a UAV ground target positioning system based on multimodal information alignment, comprising:

[0052] Acquisition module: obtain multimodal information; the multimodal information includes satellite reference map, aerial image, drone altitude, drone attitude and aerial elevation information; the aerial elevation information is the elevation information of the corresponding area of ​​the aerial image;

[0053] Satellite feature base map construction module: construct a satellite feature base map according to the satellite reference map and aerial elevation information; the satellite feature base map includes multiple two-dimensional feature maps;

[0054] Scene feature map acquisition module: based on the altitude, attitude and aerial image of the drone, scene feature extraction is performed on the aerial image to obtain a scene feature map;

[0055] A two-dimensional feature map segmentation module: performing supervoxel segmentation on the multiple two-dimensional feature maps in the satellite feature base map to obtain a supervoxel segmentation feature map;

[0056] 3D region segmentation module: Based on the supervoxel segmentation feature map, different 3D regions are selectively constructed according to the 2D feature map to obtain the segmentation satellite feature map;

[0057] Ground scene matching and target positioning module: Based on the segmented satellite feature map and scene feature map, ground scene matching and target positioning are performed

[0058] Compared with the prior art, the embodiments of the present invention achieve the following beneficial effects:

[0059] The embodiment of the present invention also provides a method and system for positioning a ground target of a UAV based on multimodal information alignment, the method comprising: obtaining multimodal information; the multimodal information comprising a satellite reference map, an aerial image, a UAV height, a UAV attitude and aerial elevation information; the aerial elevation information is the elevation information of the ground area corresponding to the aerial image; constructing a satellite feature base map according to the satellite reference map and the aerial elevation information; the satellite feature base map comprises a plurality of two-dimensional feature maps; based on the UAV height, the UAV attitude and the aerial image, performing scene feature extraction on the aerial image to obtain a scene feature map; performing supervoxel segmentation on the plurality of two-dimensional feature maps in the satellite feature base map to obtain a supervoxel segmentation feature map; based on the supervoxel segmentation feature map, selectively constructing different three-dimensional regions according to the two-dimensional feature map to obtain a segmented satellite feature map; performing ground scene matching and target positioning based on the segmented satellite feature map and the scene feature map.

[0060] The present invention uses multimodal information to perform supervoxel segmentation, matches and aligns satellite reference images and aerial images, and thus locates ground targets based on drones. Multiple two-dimensional feature maps in the satellite feature base map formed by the satellite reference map are respectively subjected to supervoxel segmentation. After arranging the multiple loss values ​​from small to large, the two-dimensional regions segmented by supervoxels are combined in the feature direction to obtain a three-dimensional region. Regions with similar segmentation conditions can be fused, and other regions are ignored. Different features are segmented into regions with similar positions, indicating that these features have the same function in detecting this position, and can be detected as a three-dimensional region. Arranging the features from large to small in influence to judge the same function can construct a relationship between the regions that can best represent importance. The combination of three-dimensional and two-dimensional regions can make subsequent matching more accurate, thereby locating the target more accurately. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 The present invention provides a flowchart of a method for positioning a ground target by a UAV based on multimodal information alignment.

[0062] Figure 2 It is a schematic diagram of the block structure of an electronic device provided by an embodiment of the present invention.

[0063] Labels in the figure: bus 500; receiver 501; processor 502; transmitter 503; memory 504; bus interface 505. DETAILED DESCRIPTION

[0064] The present invention will be described in detail below in conjunction with the accompanying drawings.

[0065] Example 1

[0066] like Figure 1 As shown, an embodiment of the present invention provides a method for positioning a ground target of a UAV based on multimodal information alignment, the method comprising:

[0067] S101: Obtain multimodal information; the multimodal information includes satellite reference maps, aerial images, drone altitude, drone attitude and aerial elevation information; the aerial elevation information is the elevation information of the ground area corresponding to the aerial image.

[0068] The aerial photography elevation information is the elevation information of the ground area of ​​the aerial image taken by the drone at the current position, current drone altitude and current drone posture.

[0069] S102: constructing a satellite feature base map according to the satellite reference map and the aerial elevation information; the satellite feature base map includes a plurality of two-dimensional feature maps.

[0070] The satellite feature base map is a three-dimensional image, and each layer of the two-dimensional feature map represents different features.

[0071] Among them, the satellite feature base map refers to a map that is marked and drawn using feature points in satellite images and aerial elevation information, and contains multiple feature information.

[0072] S103: Based on the drone height, drone posture and aerial image, scene feature extraction is performed on the aerial image to obtain a scene feature map

[0073] S104: performing supervoxel segmentation on each of the plurality of two-dimensional feature maps in the satellite feature base map to obtain a supervoxel segmentation feature map.

[0074] Wherein, the supervoxel segmentation feature map is a three-dimensional map.

[0075] S105: Based on the supervoxel segmentation feature map, different three-dimensional regions are selectively constructed according to the two-dimensional feature map to obtain a segmentation satellite feature map.

[0076] S106: performing ground scene matching and target positioning based on the segmented satellite feature map and the scene feature map;

[0077] Therein, the positions of the segmented satellite feature map and the scene feature map are matched.

[0078] Optionally, the supervoxel-based segmentation feature map selectively constructs different three-dimensional regions according to the two-dimensional feature map to obtain the segmented satellite feature map, including:

[0079] The arrangement of the two-dimensional feature map in the supervoxel segmentation feature map is adjusted to obtain a supervoxel feature map; the supervoxel feature map is a three-dimensional feature map.

[0080] Among them, because the arrangement of feature maps is different, the three-dimensional areas formed by adding and fusion are different, so it is necessary to arrange them according to the importance of the two-dimensional feature maps, so that the constructed three-dimensional area can be based on the most important features, and features of similar importance can be fused together for subsequent judgment.

[0081] The supervoxel feature map includes a plurality of segmented two-dimensional regions; the segmented two-dimensional regions are regions obtained by segmenting the two-dimensional feature map in the supervoxel segmentation feature map;

[0082] If the area of ​​the segmented two-dimensional region is greater than the segmentation threshold, the segmented two-dimensional region is marked as a two-dimensional region to be fused.

[0083] In this embodiment, the segmentation threshold is 1 / 400 of the area of ​​the satellite feature base map.

[0084] The number of the segmented two-dimensional regions is greater than or equal to the number of the two-dimensional regions to be fused. One two-dimensional region to be fused corresponds to one segmented two-dimensional region, but one segmented two-dimensional region does not necessarily correspond to one two-dimensional region to be fused.

[0085] The multiple two-dimensional regions to be fused are three-dimensionally fused to obtain multiple fused three-dimensional regions.

[0086] The number of the two-dimensional regions to be fused is greater than or equal to the number of the three-dimensional regions to be fused. One fused three-dimensional region corresponds to one or more two-dimensional regions to be fused, and one two-dimensional region to be fused corresponds to one fused three-dimensional region.

[0087] Multiple fused three-dimensional areas are used to replace the corresponding two-dimensional areas to be fused to form a segmented satellite feature map.

[0088] The segmented satellite feature map is a three-dimensional map. The two-dimensional region to be fused corresponding to the fused three-dimensional region is used to form a fused three-dimensional region, and other segmented two-dimensional regions are not changed to obtain a segmented satellite feature map.

[0089] Optionally, adjusting the arrangement of the two-dimensional feature map in the supervoxel segmentation feature map to obtain the supervoxel feature map includes:

[0090] The two-dimensional feature maps in the supervoxel segmentation feature map are respectively input into the trained first object detection network to obtain multiple loss values; one loss value corresponds to one two-dimensional feature map.

[0091] Among them, the trained first target detection network is the YOLOv5 model.

[0092] The multiple loss values ​​are arranged from small to large to obtain a super voxel feature map.

[0093] Optionally, the three-dimensional fusion of the multiple two-dimensional regions to be fused to obtain multiple fused three-dimensional regions includes:

[0094] Based on the two-dimensional region to be fused, a plurality of labeled regions are numbered according to the position; the positions of the two-dimensional regions to be fused with the same number overlap; the two-dimensional feature maps corresponding to the two-dimensional regions to be fused with the same number are different;

[0095] Among them, according to the corresponding position relationship of the two-dimensional area to be fused in the supervoxel segmentation feature map, the feature with the smallest loss value, that is, the largest influence value, that is, the most important feature is used as the basis for labeling. There is no repeated label in a two-dimensional feature map, and the label in each two-dimensional feature map starts from 0.

[0096] The numbers reflect the positional relationship of the two-dimensional regions to be fused, and whether fusion can be performed is determined by the positional relationship. In this embodiment, different features are divided into regions with similar positions, indicating that these features have the same function in detecting this position, and can be detected as a three-dimensional region. Arranging these features from large to small in terms of influence to determine the same function can construct a relationship between the most important regions.

[0097] Obtain multiple first labeled regions; the first labeled regions are regions with the same label and smaller than other labels in the supervoxel segmentation feature map; one supervoxel segmentation feature map corresponds to one first labeled region;

[0098] A histogram is established with the arranged supervoxel segmentation feature map as the horizontal coordinate and the area of ​​the first labeled region as the vertical coordinate to obtain a histogram of the first labeled region.

[0099] In this embodiment, the area is represented by the number of feature points in the first numbered area.

[0100] Based on the first labeled region histogram, it is determined whether a three-dimensional region is formed, and a plurality of first three-dimensional fusion regions are obtained.

[0101] The above method is repeated to judge multiple numbered regions with the same number for multiple times, so as to obtain multiple fused three-dimensional regions; the multiple fused three-dimensional regions include multiple first three-dimensional fused regions.

[0102] Optionally, judging whether multiple first labeled regions are fused based on the first labeled region histogram to obtain multiple first three-dimensional fused regions includes:

[0103] Obtaining a fusion set; the fusion set is an empty set;

[0104] The values ​​corresponding to the important two-dimensional feature maps in the histogram of the first labeled area are added to the fusion set.

[0105] The value corresponding to the important two-dimensional feature map is the area of ​​the first numbered region corresponding to the important two-dimensional feature map.

[0106] Obtaining adjacent values; the adjacent values ​​are values ​​in the first numbered region histogram that are not added to the fusion set and are adjacent to the values ​​that are most recently added to the fusion set in the horizontal coordinate;

[0107] Determine whether the absolute value of the value newly added to the fusion set and the adjacent value is greater than the fusion threshold.

[0108] In this embodiment, the fusion threshold is 1 / 1600 of the area of ​​the satellite feature base map.

[0109] If the absolute value of the value newly added to the fusion set and the adjacent value is less than or equal to the fusion threshold, the adjacent ordinate that has not been added to the fusion set is added to the fusion set.

[0110] Among them, if the absolute value of the value newly added to the fusion set and the adjacent value is less than or equal to the fusion threshold, it means that the area change between the first labeled regions corresponding to the two adjacent two-dimensional feature maps is small, so the impact of the two features on the segmentation of this position is not much different.

[0111] The corresponding two-dimensional regions to be fused in the fusion set are constructed into a three-dimensional region to obtain a first three-dimensional fusion region.

[0112] The method of constructing the three-dimensional region is to mark the corresponding two-dimensional region to be fused in the fusion set with a label corresponding to the three-dimensional region.

[0113] If the absolute value of the value newly added to the fusion set and the adjacent value is greater than the fusion threshold, a new fusion set is obtained, and the adjacent vertical coordinates not added to the fusion set are added to the new fusion set. The above method is repeated to obtain multiple first three-dimensional fusion areas; the new fusion set is an empty set.

[0114] Optionally, the two-dimensional region to be fused is numbered by position to obtain a plurality of numbered regions, including:

[0115] The two-dimensional regions to be fused in the important two-dimensional regions are labeled to obtain multiple important two-dimensional regions.

[0116] Among them, the labels of the two-dimensional regions to be fused in the important supervoxel segmentation feature map are not repeated.

[0117] The overlapping area of ​​the important two-dimensional region and the two-dimensional region to be fused except the important two-dimensional feature map is calculated to obtain the overlapping area.

[0118] In this embodiment, the area is represented by the number of feature points of the important two-dimensional area and the two-dimensional area to be fused. The overlap area is the feature points that represent the overlap between the important two-dimensional area and the two-dimensional area to be fused.

[0119] The overlap value was obtained by dividing the overlap area by the area of ​​the significant two-dimensional region.

[0120] Among them, the important two-dimensional area is used as a benchmark, so it is necessary to divide it by the important two-dimensional area to obtain the overlap ratio.

[0121] If the overlap value is greater than the overlap threshold, the two-dimensional region to be fused corresponding to the overlap value is marked with the same label as the important two-dimensional region.

[0122] In this embodiment, the overlap threshold is 0.5.

[0123] Optionally, the labels of the two-dimensional regions to be fused at different positions in each important two-dimensional feature map are different.

[0124] Optionally, the extracting scene features of the aerial image based on the altitude of the drone, the attitude of the drone and the aerial image to obtain a scene feature map includes:

[0125] The aerial image is orthorectified according to the altitude and attitude of the drone to obtain a corrected aerial image.

[0126] In this embodiment, the collinear model performs orthorectification. In this embodiment, because the aerial image is obtained by the sensor on the drone, the orthorectification is performed by the drone height and the drone attitude. If a camera is used for shooting, the orthorectification is performed by the camera intrinsic parameters and camera extrinsic parameters.

[0127] The rectified aerial image is passed through the oriented gradient histogram operator to obtain the oriented gradient histogram.

[0128] Among them, the Histogram of Oriented Gradient (HOG) feature is a feature descriptor used for object detection in computer vision and image processing. The HOG feature is constructed by calculating and counting the gradient direction histogram of the local area of ​​the image.

[0129] The rectified aerial image is passed through the LBP operator to obtain the LBP feature map.

[0130] Among them, Local Binary Pattern (LBP), LBP operator is an operator used to describe the local texture features of an image.

[0131] The size of the directional gradient histogram and the LBP feature map is the same;

[0132] Superimpose the directional gradient histogram and the LBP feature map to obtain a superimposed feature map;

[0133] The superposition direction is a characteristic direction, and the superposition characteristic map is three-dimensional.

[0134] The directional gradient histogram and the LBP feature map in the superposition feature map are respectively subjected to supervoxel segmentation and constituted into different three-dimensional regions to obtain a scene feature map.

[0135] The method of performing supervoxel segmentation and forming different three-dimensional regions is the same as the method of performing supervoxel segmentation and forming different three-dimensional regions on the satellite feature base map to obtain a segmented satellite feature map.

[0136] Optionally, performing ground scene matching and target positioning based on the second segmented satellite feature map and the scene feature map includes:

[0137] The segmented satellite feature map and the scene feature map are matched to obtain the matching position.

[0138] The similarity is judged based on the region segmented by supervoxels, and then matching is performed to obtain the matching position.

[0139] The segmented satellite feature map and the scene feature map corresponding to the matching position are located through the second target detection network to obtain the target position.

[0140] Among them, the second target detection network in this embodiment is a YOLOV5 model trained by multiple historical segmented satellite feature maps and corresponding scene feature maps.

[0141] Optional, multi-modal information: (1) satellite reference map + elevation information (2) drone aerial photography + drone altitude / point cloud information / drone attitude.

[0142] High-precision positioning information is obtained through scene matching of satellite reference images. To meet the demand for high-precision positioning of ground targets by UAVs under GNSS denial conditions, technologies such as multi-scale weighted satellite feature base map construction, real-time extraction of aerial scene features, ground scene matching and target positioning are studied to achieve high-precision positioning capability of ground targets under denial conditions and provide reliable pointing information for our equipment.

[0143] Step 1: Construction of multi-scale weighted satellite feature basemap

[0144] Combined with elevation information

[0145] Create satellite feature basemap (grayscale, Gaussian filtering, sobe1 convolution, multi-scale filtering, normalized weighting, feature direction).

[0146] Organize and manage satellite feature base maps.

[0147] Step 2: Real-time extraction of aerial scene features,

[0148] Orthorectification of aerial images: drone altitude, drone attitude angle, payload attitude angle, camera intrinsic parameters, and camera extrinsic parameters.

[0149] Step 3: Perform supervoxel segmentation on both images.

[0150] Step 4: Ground scene matching and target positioning,

[0151] Similarity measurement based on cosine, search for points with the same name in satellite feature basemap, registration of aerial images with satellite feature basemap, and calculation of longitude and latitude of ground targets.

[0152] Example 2

[0153] Based on the above-mentioned UAV ground target positioning method based on multimodal information alignment, an embodiment of the present invention also provides a UAV ground target positioning system based on multimodal information alignment, the system including an acquisition module, a satellite feature base map construction module, a scene feature map acquisition module, a two-dimensional feature map segmentation module, a three-dimensional region segmentation module and a ground scene matching and target positioning module.

[0154] The acquisition module is used to obtain multimodal information; the multimodal information includes satellite reference map, aerial image, drone altitude, drone attitude and aerial elevation information; the aerial elevation information is the elevation information of the corresponding area of ​​the aerial image;

[0155] The satellite feature base map construction module is used to construct a satellite feature base map according to the satellite reference map and the aerial elevation information; the satellite feature base map includes multiple two-dimensional feature maps;

[0156] The scene feature map acquisition module is used to extract scene features from the aerial image based on the altitude, attitude and aerial image of the drone to obtain a scene feature map;

[0157] The two-dimensional feature map segmentation module is used to perform supervoxel segmentation on the multiple two-dimensional feature maps in the satellite feature base map to obtain a supervoxel segmentation feature map;

[0158] The three-dimensional region segmentation module is used to selectively construct different three-dimensional regions according to the two-dimensional feature map based on the supervoxel segmentation feature map to obtain the segmentation satellite feature map;

[0159] The ground scene matching and target positioning module is used to perform ground scene matching and target positioning based on the segmented satellite feature map and the scene feature map.

[0160] The embodiment of the present invention further provides an electronic device, such as Figure 2 As shown, it includes a memory 504, a processor 502, and a computer program stored in the memory 504 and executable on the processor 502. When the processor 502 executes the program, the steps of any method of the UAV ground target positioning method based on multimodal information alignment described above are implemented.

[0161] Among them, Figure 2 In the embodiment of the present invention, a bus architecture (represented by bus 500) is shown, which may include any number of interconnected buses and bridges, and bus 500 links various circuits including one or more processors represented by processor 502 and memory represented by memory 504. Bus 500 may also link various other circuits such as peripherals, voltage regulators, and power management circuits, which are well known in the art and are therefore not further described herein. Bus interface 505 provides an interface between bus 500 and receiver 501 and transmitter 503. Receiver 501 and transmitter 503 may be the same element, namely a transceiver, providing a unit for communicating with various other devices over a transmission medium. Processor 502 is responsible for managing bus 500 and general processing, while memory 504 may be used to store data used by processor 502 when performing operations.

[0162] An embodiment of the present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the methods for positioning a ground target by an unmanned aerial vehicle based on multimodal information alignment as described above and the data involved above.

[0163] The algorithm and display provided herein are not inherently related to any particular computer, virtual system or other device. Various general purpose systems can also be used together with the teachings based on this. According to the above description, it is obvious that the structure required for constructing such systems. In addition, the present invention is not directed to any specific programming language either. It should be understood that various programming languages ​​can be utilized to realize the content of the present invention described herein, and the description of the above specific languages ​​is for disclosing the best mode of the present invention.

[0164] In the description provided herein, a large number of specific details are described. However, it is understood that embodiments of the present invention can be practiced without these specific details. In some instances, well-known methods, structures and techniques are not shown in detail so as not to obscure the understanding of this description.

[0165] Similarly, it should be understood that in order to streamline the present disclosure and aid in understanding one or more of the various inventive aspects, in the above description of exemplary embodiments of the present invention, various features of the present invention are sometimes grouped together into a single embodiment, figure, or description thereof. However, this disclosed method should not be interpreted as reflecting the intention that the claimed invention requires more features than those explicitly recited in each claim. More specifically, as reflected in the claims below, inventive aspects lie in less than all the features of the individual embodiments disclosed above. Therefore, the claims that follow the specific embodiment are hereby expressly incorporated into the specific embodiment, with each claim itself serving as a separate embodiment of the present invention.

[0166] Those skilled in the art will appreciate that the modules in the devices in the embodiments may be adaptively changed and arranged in one or more devices different from the embodiments. The modules or units or components in the embodiments may be combined into one module or unit or component, and in addition they may be divided into a plurality of submodules or subunits or subcomponents. Except that at least some of such features and / or processes or units are mutually exclusive, all features disclosed in this specification (including the accompanying claims, abstracts and drawings) and all processes or units of any method or device disclosed in this manner may be combined in any combination. Unless otherwise expressly stated, each feature disclosed in this specification (including the accompanying claims, abstracts and drawings) may be replaced by an alternative feature providing the same, equivalent or similar purpose.

[0167] In addition, those skilled in the art will appreciate that, although some embodiments herein include certain features included in other embodiments but not other features, the combination of features of different embodiments is meant to be within the scope of the present invention and form different embodiments. For example, in the claims below, any one of the claimed embodiments may be used in any combination.

[0168] The various component embodiments of the present invention may be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. It should be understood by those skilled in the art that a microprocessor or digital signal processor (DSP) may be used in practice to implement some or all of the functions of some or all of the components in the apparatus according to an embodiment of the present invention. The present invention may also be implemented as a device or device program (e.g., a computer program and a computer program product) for executing part or all of the methods described herein. Such a program implementing the present invention may be stored on a computer-readable medium, or may have the form of one or more signals. Such a signal may be downloaded from an Internet website, or provided on a carrier signal, or provided in any other form.

[0169] It should be noted that the above embodiments illustrate the present invention rather than limit it, and that those skilled in the art may devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference symbol between brackets shall not be construed as a limitation on the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "one" or "an" preceding an element does not exclude the presence of a plurality of such elements. The present invention may be implemented by means of hardware comprising a number of different elements and by means of a suitably programmed computer. In a unit claim enumerating a number of devices, several of these devices may be embodied by the same hardware item. The use of the words first, second, and third, etc., does not indicate any order. These words may be interpreted as names.

Claims

1. A method for positioning a ground target using a UAV based on multimodal information alignment, characterized in that: include: Obtain multimodal information; The multimodal information includes satellite reference images, aerial images, drone altitude, drone attitude, and aerial elevation information; The aerial photography elevation information is the elevation information of the corresponding area of ​​the aerial photography image; Constructing a satellite feature base map according to the satellite reference map and the aerial elevation information; the satellite feature base map includes a plurality of two-dimensional feature maps; Based on the altitude, attitude and aerial image of the drone, scene features of the aerial image are extracted to obtain a scene feature map; Performing supervoxel segmentation on the multiple two-dimensional feature maps in the satellite feature base map to obtain supervoxel segmentation feature maps; Based on the supervoxel segmentation feature map, different three-dimensional regions are selectively constructed according to the two-dimensional feature map to obtain the segmentation satellite feature map; Based on the segmented satellite feature map and scene feature map, ground scene matching and target positioning are performed.

2. The method for positioning a ground target using a UAV based on multimodal information alignment according to claim 1, characterized in that: The method of selectively forming different three-dimensional regions based on the supervoxel segmentation feature map according to the two-dimensional feature map to obtain the segmentation satellite feature map includes: Adjusting the arrangement of the two-dimensional feature map in the supervoxel segmentation feature map to obtain a supervoxel feature map; the supervoxel feature map is a three-dimensional feature map; The supervoxel feature map includes a plurality of segmented two-dimensional regions; the segmented two-dimensional regions are regions obtained by segmenting the two-dimensional feature map in the supervoxel segmentation feature map; If the area of ​​the segmented two-dimensional region is greater than the segmentation threshold, the segmented two-dimensional region is marked as a two-dimensional region to be fused; Perform three-dimensional fusion on multiple two-dimensional regions to be fused to obtain multiple fused three-dimensional regions; Multiple fused three-dimensional areas are used to replace the corresponding two-dimensional areas to be fused to form a segmented satellite feature map.

3. The method for positioning a ground target using a UAV based on multimodal information alignment according to claim 2, characterized in that: The adjusting the arrangement of the two-dimensional feature map in the supervoxel segmentation feature map to obtain the supervoxel feature map includes: Inputting the two-dimensional feature maps in the supervoxel segmentation feature map into the trained first object detection network respectively to obtain multiple loss values; one loss value corresponds to one two-dimensional feature map; Arrange the multiple loss values ​​from small to large to obtain a supervoxel feature map; Obtain an important two-dimensional feature map; the important two-dimensional feature map is a two-dimensional feature map whose loss value is less than the loss value of other two-dimensional feature maps.

4. The method for positioning a ground target using a UAV based on multimodal information alignment according to claim 2, characterized in that: The step of performing three-dimensional fusion on the multiple two-dimensional regions to be fused to obtain multiple fused three-dimensional regions includes: Based on the two-dimensional region to be fused, a plurality of labeled regions are numbered according to the position; the positions of the two-dimensional regions to be fused with the same number overlap; the supervoxel segmentation feature maps corresponding to the two-dimensional regions to be fused with the same number are different; Obtain multiple first labeled regions; the first labeled regions are regions with the same label and smaller than other labels in the supervoxel segmentation feature map; one supervoxel segmentation feature map corresponds to one first labeled region; A histogram is established with the arranged supervoxel segmentation feature map as the abscissa and the area of ​​the first labeled region as the ordinate, thereby obtaining a histogram of the first labeled region; Based on the first labeled region histogram, determining whether a three-dimensional region is formed, and obtaining a plurality of first three-dimensional fusion regions; The above method is repeated to judge multiple numbered regions with the same number for multiple times, so as to obtain multiple fused three-dimensional regions; the multiple fused three-dimensional regions include multiple first three-dimensional fused regions.

5. The method for positioning a ground target using a UAV based on multimodal information alignment according to claim 4, characterized in that: The determining whether a three-dimensional region is formed based on the first labeled region histogram to obtain a plurality of first three-dimensional fusion regions includes: Obtaining a fusion set; the fusion set is an empty set; Add the values ​​corresponding to the important two-dimensional feature maps in the histogram of the first labeled area to the fusion set; Obtaining adjacent values; the adjacent values ​​are values ​​in the first numbered region histogram that are not added to the fusion set and are adjacent to the values ​​that are most recently added to the fusion set in the horizontal coordinate; Determine whether the absolute value of the value newly added to the fusion set and the adjacent value is less than the fusion threshold; If the absolute value of the value newly added to the fusion set and the adjacent value is less than or equal to the fusion threshold, the adjacent ordinate that has not been added to the fusion set is added to the fusion set; The corresponding two-dimensional region to be fused in the fusion set is constructed into a three-dimensional region to obtain a first three-dimensional fusion region; If the absolute value of the value newly added to the fusion set and the adjacent value is greater than the fusion threshold, a new fusion set is obtained, and the adjacent vertical coordinates not added to the fusion set are added to the new fusion set. The above method is repeated to obtain multiple first three-dimensional fusion areas; the new fusion set is an empty set.

6. The method for positioning a ground target using a UAV based on multimodal information alignment according to claim 4, characterized in that: The two-dimensional region to be fused is numbered according to the position to obtain multiple numbered regions, including: Label the two-dimensional regions to be fused in the important two-dimensional feature map to obtain multiple important two-dimensional regions; The overlapping area of ​​the important two-dimensional region and the two-dimensional region to be fused except the important two-dimensional feature map is calculated to obtain the overlapping area; Divide the overlap area by the area of ​​the important two-dimensional region to obtain the overlap value; If the overlap value is greater than the overlap threshold, the two-dimensional region to be fused corresponding to the overlap value is marked with the same label as the important two-dimensional region.

7. The method for positioning a ground target using a UAV based on multimodal information alignment according to claim 6, characterized in that: The labels of the two-dimensional regions to be fused at different positions in each important two-dimensional feature map are different.

8. The method for positioning a ground target using a UAV based on multimodal information alignment according to claim 1, characterized in that: The method of extracting scene features from the aerial image based on the altitude, attitude and aerial image of the drone to obtain a scene feature map includes: Ortho-rectifying the aerial image according to the altitude and attitude of the drone to obtain a corrected aerial image; The rectified aerial image is passed through the directional gradient histogram operator to obtain the directional gradient histogram; The rectified aerial image is passed through the LBP operator to obtain the LBP feature map; The size of the directional gradient histogram and the LBP feature map is the same; Superimpose the directional gradient histogram and the LBP feature map to obtain a superimposed feature map; The directional gradient histogram and the LBP feature map in the superposition feature map are respectively subjected to supervoxel segmentation and constituted into different three-dimensional regions to obtain a scene feature map.

9. The method for positioning a ground target using a UAV based on multimodal information alignment according to claim 1, characterized in that: The method of performing ground scene matching and target positioning based on segmenting the satellite feature map and the scene feature map includes: Matching the segmented satellite feature map and the scene feature map to obtain a matching position; The segmented satellite feature map and the scene feature map corresponding to the matching position are located through the second target detection network to obtain the target position.

10. A UAV ground target positioning system based on multi-modal information alignment, characterized in that: include: Acquisition module: obtain multimodal information; The multimodal information includes satellite reference images, aerial images, drone altitude, drone attitude, and aerial elevation information; The aerial photography elevation information is the elevation information of the corresponding area of ​​the aerial photography image; Satellite feature base map construction module: construct a satellite feature base map according to the satellite reference map and aerial elevation information; the satellite feature base map includes multiple two-dimensional feature maps; Scene feature map acquisition module: based on the altitude, attitude and aerial image of the drone, scene feature extraction is performed on the aerial image to obtain a scene feature map; A two-dimensional feature map segmentation module: performing supervoxel segmentation on the multiple two-dimensional feature maps in the satellite feature base map to obtain a supervoxel segmentation feature map; 3D region segmentation module: Based on the supervoxel segmentation feature map, different 3D regions are selectively constructed according to the 2D feature map to obtain the segmentation satellite feature map; Ground scene matching and target positioning module: Based on the segmented satellite feature map and scene feature map, ground scene matching and target positioning are performed.

Images