A semantic segmentation method for feature images captured by a drone and a neural network structure

Abstract

The application discloses a semantic segmentation method of feature images captured by a UAV and a neural network structure. The method comprises the following steps: predicting indexes corresponding to a plurality of inflation rates according to a feature image; initializing full-scale shared convolution according to the inflation rates corresponding to the indexes, and performing feature grabbing on the feature image after the initialization; predicting a plurality of angle indexes for each pixel of the feature image in an effective scale; selecting an angle corresponding to an index for each pixel, and extracting the feature image in the effective scale corresponding to the selected index; decomposing an n*n convolution into n*n 1*1 convolutions, and inputting the extracted feature image in the effective scale into the 1*1 convolutions; and performing rotation sampling on the spatial basic features of each pixel according to the selected angle and the original coordinates of the n*n convolution relative to the center, and performing feature aggregation after the rotation sampling. The application avoids the problem of sparsity of useful information and effectively reduces the calculation cost.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of unmanned aerial vehicle image processing, and particularly relates to a semantic segmentation method for a feature image captured by an unmanned aerial vehicle and a neural network structure. BACKGROUND

[0002] Existing semantic segmentation methods are mainly designed for ground view images (such as street view and natural scenes), and have the following significant limitations when applied to unmanned aerial vehicle images, especially aerial images captured by unmanned aerial vehicles, which have extremely high pixel sizes (for example, 6000x4000 pixels).

[0003] Due to the size of the object itself, the shooting resolution and the shooting distance, the problem of different scales of objects in the unmanned aerial vehicle image is very serious. Superimposing multiple scale results together may cause useful information to be sparse. If the number of scales is reduced, scale information will be lost. Moreover, by changing the space of the convolution kernel to cope with different resolution inputs, the computational cost increases dramatically with the number of dilated convolutions, and the computational overhead becomes infeasible when processing unmanned aerial vehicle scale images.

[0004] Therefore, there is an urgent need for new technical solutions to solve the above technical problems. SUMMARY

[0005] The present application relates to the technical field of unmanned aerial vehicle image processing, and particularly relates to a semantic segmentation method for a feature image captured by an unmanned aerial vehicle and a neural network structure.

[0006] To achieve the above object, the present application provides the following technical solutions:

[0007] The present application provides a semantic segmentation method for a feature image captured by an unmanned aerial vehicle, comprising the following steps:

[0008] According to the feature image, an index corresponding to a plurality of dilation rates is predicted;

[0009] A full-scale shared convolution is initialized according to the dilation rate corresponding to the index, and a feature image under an effective scale is obtained by performing feature extraction on the feature image using the initialized full-scale shared convolution;

[0010] A plurality of angle indexes are predicted for each pixel of the feature image under the effective scale;

[0011] An angle corresponding to an angle index is selected for each pixel, and the feature image under the effective scale corresponding to the selected angle index is extracted;

[0012] The nxn convolution is decomposed into nxn 1x1 convolutions, the feature image at the effective scale extracted is input into the 1x1 convolution, and the spatial basis feature of each pixel at the nxn kernel position is obtained;

[0013] According to the selected angle and the obtained original coordinates of the nxn convolution relative to the center, the spatial basis feature of each pixel is rotationally sampled, and after rotationally sampling, feature aggregation is performed to obtain the semantic feature of the feature image at the effective scale and the effective angle of the current center pixel.

[0014] In one or more embodiments, the method further comprises:

[0015] A four-dimensional tensor of the feature image is obtained, wherein the four-dimensional tensor comprises an inflation rate range of the feature image;

[0016] According to the inflation rate range in the four-dimensional tensor, the probability of each inflation rate of each pixel in the feature image within the inflation rate range is calculated;

[0017] The index corresponding to the first k inflation rates with the highest probability is selected.

[0018] In one or more embodiments, the full-scale shared convolution is performed by using one convolution kernel and cooperating with different inflation rates to capture object information in the feature image.

[0019] In one or more embodiments, the method further comprises:

[0020] A four-dimensional tensor of the feature image at the effective scale is obtained, wherein the four-dimensional tensor comprises a plurality of preset angles corresponding to the feature image at the effective scale;

[0021] The probability of each angle corresponding to each pixel of the feature image at the effective scale is calculated;

[0022] The index corresponding to the first k angles with the highest probability for each pixel is selected.

[0023] In one or more embodiments, the method further comprises:

[0024] According to the selected angle, the original coordinates of the nxn convolution relative to the center, the actual offset of the sampling point is calculated;

[0025] According to the actual offset of the sampling point, the spatial basis feature of each pixel is moved to the center relative to the nxn convolution.

[0026] In one or more embodiments, bilinear interpolation or translation is further included for the coordinates of the sampling points after rotation sampling.

[0027] According to another aspect of the present application, a neural network structure is also provided, based on the semantic segmentation method of a feature image captured by a UAV as described above, comprising:

[0028] a full-scale dilated convolution network, comprising a dilation rate strategy decision subnetwork and a multi-scale convolution subnetwork; the dilation rate strategy decision subnetwork is configured to predict indexes corresponding to a plurality of dilation rates according to a feature image; the multi-scale convolution subnetwork is a full-scale shared convolution layer, configured to initialize the full-scale shared convolution layer according to the dilation rates corresponding to the indexes, and perform feature extraction on the feature image by using the initialized full-scale shared convolution layer to obtain a feature image at an effective scale;

[0029] a full-directional convolution network, comprising an angle strategy decision subnetwork and a full-directional shared convolution subnetwork; the angle strategy decision subnetwork is configured to predict a plurality of angle indexes for each pixel of the feature image at the effective scale; the full-directional shared convolution subnetwork is configured to select an angle corresponding to each index for each pixel, extract the feature image at the effective scale corresponding to the selected index, decompose an nxn convolution into nxn 1x1 convolutions, input the extracted feature image at the effective scale into the 1x1 convolutions to obtain spatial basis features of nxn kernel positions of each pixel, and perform rotation sampling on the spatial basis features of each pixel according to the selected angle and the original coordinates of the nxn convolution relative to the center, and perform feature aggregation after the rotation sampling to obtain semantic features of the feature image at the effective scale and the effective angle of the current center pixel.

[0030] In one or more embodiments, the dilation rate strategy decision subnetwork comprises a first multi-layer perceptron, a first Softmax layer and a dilation rate index selection layer connected in sequence, and the dilation rate index selection layer is connected with the full-scale shared convolution.

[0031] The angle strategy decision subnetwork comprises a second multi-layer perceptron, a second Softmax layer and an angle index selection layer connected in sequence.

[0032] The full-directional shared convolution subnetwork comprises an angle rotation layer, an angle convolution layer and a feature aggregation layer connected in sequence, and the angle index selection layer is connected with the angle rotation layer.

[0033] In one or more embodiments, further comprising a subsequent network layer for training and updating the full-scale dilated convolution network and the full-direction convolution network, the subsequent network layer comprising a reward module, an advantage function module and a loss function module connected in sequence, the reward module being connected to the output of the full-scale shared convolution and the output of the full-direction shared convolution subnetwork respectively, and the loss function module being connected to the dilated rate policy decision subnetwork and the angle policy decision subnetwork respectively.

[0034] In one or more embodiments, when the resolution of the feature image does not meet the set requirements, the complete high-resolution feature image is loaded into the video memory at one time before training the dilated rate policy decision subnetwork, and then the feature image is randomly cropped, and the randomly cropped feature image is used as the sample image for training the dilated rate policy decision subnetwork.

[0035] When the resolution of the feature image does not meet the set requirements, when training the angle policy decision subnetwork, a random angle is randomly generated in the angle range of [0, 360], and the result of the affine transformation performed on the feature image in the training sample by the full-direction shared convolution subnetwork is rotated according to the random angle.

[0036] Implementing one of the technical solutions in the above technical solutions of the present application has the following advantages or beneficial effects:

[0037] The present application covers all possible scales of the input feature map by predicting multiple optimal dilated rates, while only retaining the scales effective for the current center pixel, thereby comprehensively covering the scale range in which various objects may appear, while avoiding the problem of sparsity of useful information. In the selection of effective visual angles, multiple optimal angle indexes are predicted and the convolution kernel is decomposed, which can avoid complex calculation problems such as derivation, effectively reduce the calculation overhead, and improve the running efficiency. BRIEF DESCRIPTION OF DRAWINGS

[0038] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can also be obtained according to these drawings without creative labor for those skilled in the art. In the drawings:

[0039] Figure 1 is a semantic segmentation method flow of a feature image captured by a UAV according to an embodiment of the present application;

[0040] Figure 2 is a neural network structure diagram according to an embodiment of the present application. DETAILED DESCRIPTION

[0041] In order to make the objects, technical solutions and advantages of the present application clearer, the various exemplary embodiments to be described below will be described with reference to the corresponding drawings, which constitute a part of the exemplary embodiments and in which various exemplary embodiments that can be used to implement the present application are described. The same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all the implementations consistent with the present disclosure. It should be understood that they are only examples of processes, methods and apparatuses, etc. consistent with some aspects of the present disclosure as detailed in the appended claims, and other implementations can be used or structural and functional modifications can be made to the implementations listed herein without departing from the scope and spirit of the present application.

[0042] In the description of the present application, it should be understood that the terms "center", "longitudinal", "transverse" and the like indicate the orientation or positional relationship shown in the drawings, which are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the elements referred to must have a specific orientation, be constructed and operated in a specific orientation. The terms "first", "second" and the like are only for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. The term "a plurality of" means two or more. The terms "connected", "connected" should be interpreted broadly, for example, it can be fixed connection, detachable connection, integral connection, mechanical connection, electrical connection, communication connection, direct connection, indirect connection through intermediate medium, internal communication of two elements or interaction relationship between two elements. The term "and / or" includes any and all combinations of one or more related listed items. For those skilled in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0043] In order to illustrate the technical solutions of the present application, the following specific embodiments are described, only showing the parts related to the embodiments of the present application.

[0044] Embodiment one: as shown, the present application provides a semantic segmentation method for feature images captured by a UAV, comprising the following steps: Figure 1

[0045] S100, according to the feature image, predict a plurality of inflation rates corresponding to the index.

[0046] In this embodiment, semantic segmentation is defined as a computer vision basic task, which aims to assign a class label to each pixel in the feature image to achieve pixel-level scene understanding.

[0047] ​The feature image refers to an aerial image captured by a high-resolution unmanned aerial vehicle (UAV) and having a very high pixel size (for example, 6000x4000), which usually contains a wide field of view and fine spatial details.

[0048] It can be understood that, before the image is processed by the method, the image captured by the UAV is processed by feature extraction (such as convolutional network processing) to obtain a feature image having batch size, channel number, height, and width.

[0049] In one or more embodiments, predicting, according to the feature image, indexes corresponding to a plurality of dilation rates comprises:

[0050] Obtaining a four-dimensional tensor of the feature image, wherein the four-dimensional tensor comprises batch size, height, width, and a range of dilation rates of the feature image;

[0051] According to the range of dilation rates in the four-dimensional tensor, calculating a probability of each dilation rate within the range of dilation rates for each pixel in the feature image;

[0052] Selecting indexes corresponding to the top-k dilation rates with the highest probabilities.

[0053] In specific embodiments, a three-layer multilayer perceptron (three-layer MLP) can be used to obtain the four-dimensional tensor (logits dimension) of the feature image. A Softmax function can be used to calculate the probability of the dilation rate of each pixel position in the feature image.

[0054] For ease of understanding, the top-k (k is an integer greater than 1) dilation rates with the highest probabilities are directly selected without weighting, and the indexes are directly taken as parameters to be passed to the next convolution, thereby achieving no need to derive the convolution and reducing the calculation overhead.

[0055] S200, initializing a full-scale shared convolution according to the dilation rate corresponding to the index, and performing feature extraction on the feature image using the initialized full-scale shared convolution to obtain a feature image at an effective scale.

[0056] In one or more embodiments, the full-scale shared convolution extracts object information in the feature image by using a convolution kernel in combination with different dilation rates.

[0057] It should be noted that the object information refers to common objects on the ground, such as cars, people, houses, bicycles, ships, and trees.

[0058] For ease of understanding, the unique feature of the full-scale convolution is that it covers all possible scales of the input feature map, while only retaining the scale effective for the current center pixel, thereby comprehensively covering the scale range of various objects that may occur, while avoiding the sparsity problem of useful information. In terms of selection of effective scales, a three-layer multilayer perceptron (MLP) is trained by a reinforcement learning method to make decisions, which can avoid the problem of derivation.

[0059] S300, predicting a plurality of angle indexes for each pixel of the feature image under the effective scale.

[0060] In one or more embodiments, predicting a plurality of angle indexes for each pixel of the feature image under the effective scale comprises:

[0061] obtaining a four-dimensional tensor of the feature image under the effective scale, wherein the four-dimensional tensor comprises a batch size, a height and a width of the feature image under the effective scale, and a plurality of preset angles of the feature image under the effective scale;

[0062] calculating a probability of each pixel of the feature image under the effective scale corresponding to each angle (preset angle);

[0063] selecting, for each pixel, an index corresponding to the angle with the highest probability from the k (k is an integer greater than 1) angles.

[0064] In specific embodiments, the preset angles (discrete angles) can be established in a list, for example, [0, 45, 90, 135, 180, 225, 270, 315]. Each discrete angle corresponds to an index (for example, index 0 corresponds to 0 degrees, index 1 corresponds to 45 degrees, and the rest follow the same pattern).

[0065] It should be noted that, unlike conventional street view segmentation, the image captured by the UAV can be captured at any angle from 0 degrees to 360 degrees, and the above-mentioned preset discrete angles correspond to the possible capture angles of the UAV from 0 degrees to 360 degrees.

[0066] In specific embodiments, a trained three-layer multilayer perceptron (three-layer MLP) can be used to obtain the four-dimensional tensor (logits dimension) of the feature image under the effective scale. The probability of each pixel in the feature image under the effective scale belonging to each preset angle can be calculated by a Softmax function.

[0067] S400, selecting an angle corresponding to an angle index for each pixel, and extracting the feature image under the effective scale corresponding to the selected angle index.

[0068] For ease of understanding, the feature image under the effective scale corresponding to the angle index can be obtained by a trained three-layer multilayer perceptron (three-layer MLP).

[0069] S500, decompose the nxn convolution into nxn 1x1 convolutions, input the extracted feature image at the effective scale into the 1x1 convolution to obtain the spatial basis feature of each pixel at the nxn kernel position.

[0070] In a specific embodiment, the 3x3 convolution is decomposed into 9 1x1 convolutions, and the spatial feature basis of 9 orientations (kernel positions) is extracted.

[0071] It can be understood that the decomposition of the convolution kernel in this step only needs to be calculated once.

[0072] S600, according to the selected angle and the obtained original coordinates of the nxn convolution relative to the center, rotate and sample the spatial basis feature of each pixel, and then aggregate the features after rotation and sampling to obtain the semantic feature of the feature image at the effective scale and the effective angle of the current center pixel.

[0073] In a specific embodiment, the original coordinates can be the coordinates of the upper left corner of the angle index map (-1, -1).

[0074] In one or more embodiments, according to the selected angle and the obtained original coordinates of the nxn convolution relative to the center, rotating and sampling the spatial basis feature of each pixel includes:

[0075] According to the selected angle and the original coordinates of the nxn convolution relative to the center, calculating the actual offset of the sampling point;

[0076] According to the actual offset of the sampling point, moving the spatial basis feature of each pixel to the center relative to the nxn convolution.

[0077] In a specific embodiment, the actual offset is calculated according to the following formula:

[0078] Actual offset = x cos(theta) - y sin(theta);

[0079] Wherein, theta is the angle corresponding to the selected angle index, x and y are the x coordinate and y coordinate corresponding to the original coordinates.

[0080] It should be noted that this step constructs a sampling grid for the selected angle by selecting the sampling step, which is equivalent to moving all the above basis features to the center according to the calculated rotation offset.

[0081] In one or more embodiments, since the coordinates after rotation may not be integers, bilinear interpolation or translation is needed for each pixel coordinate of each angle index map after rotation.

[0082] The embodiment covers all possible scales of the input feature map by predicting multiple optimal dilation rates, while only retaining the scales effective for the current center pixel, thereby comprehensively covering the scale range of various objects that can occur, while avoiding the sparsity problem of useful information. In the selection of the effective visual angle, multiple optimal angle indexes are predicted, and the convolution kernel is decomposed, which can avoid complex calculation problems such as derivation, effectively reduce the calculation overhead, and improve the running efficiency.

[0083] Embodiment two: as shown in the embodiment, the embodiment provides a neural network structure based on the semantic segmentation method of the feature image captured by the unmanned aerial vehicle in embodiment one, which comprises: Figure 2

[0084] The full-scale dilated convolution network comprises a dilation rate strategy decision subnetwork and a multi-scale convolution subnetwork. The dilation rate strategy decision subnetwork is used to predict multiple dilation rate corresponding indexes according to the feature image. The multi-scale convolution subnetwork is a full-scale shared convolution, which is used to initialize the full-scale shared convolution according to the dilation rate corresponding to the index, and to obtain the feature image under the effective scale by using the initialized full-scale shared convolution to capture the features of the feature image;

[0085] The full-direction convolution network comprises an angle strategy decision subnetwork and a full-direction shared convolution subnetwork. The angle strategy decision subnetwork is used to predict multiple angle indexes for each pixel of the feature image under the effective scale. The full-direction shared convolution subnetwork is used to select an angle corresponding to an index for each pixel, extract the feature image under the effective scale corresponding to the selected index, decompose the nxn convolution into nxn 1x1 convolutions, input the extracted feature image under the effective scale into the 1x1 convolution, and obtain the spatial basis features of the nxn kernel positions of each pixel. According to the selected angle and the original coordinates of the nxn convolution relative to the center, the spatial basis features of each pixel are rotated and sampled, and after the rotation and sampling, the features are aggregated to obtain the semantic features of the feature image under the effective scale and the effective angle of the current center pixel.

[0086] In one or more embodiments, the dilation rate strategy decision subnetwork comprises a first multi-layer perceptron, a first Softmax layer and a dilation rate index selection layer connected in sequence, and the dilation rate index selection layer is connected with the full-scale shared convolution.

[0087] The angle strategy decision subnetwork comprises a second multi-layer perceptron, a second Softmax layer and an angle index selection layer connected in sequence.

[0088] In specific embodiments, the first multi-layer perceptron comprises three first MLP layers, and the second multi-layer perceptron comprises three second MLP layers.

[0089] ​The omnidirectional shared convolutional subnetwork consists of an angle rotation layer, an angle convolutional layer, and a feature aggregation layer connected in sequence; the angle index selection layer is connected to the angle rotation layer.

[0090] In one or more embodiments, the system further includes subsequent network layers for training and updating the full-scale dilated convolutional network and the omnidirectional convolutional network. The subsequent network layers include a reward module, an advantage function module, and a loss function module connected in sequence. The reward module is connected to the output of the full-scale shared convolutional network and the output of the omnidirectional shared convolutional subnetwork, respectively. The loss function module is connected to the dilation rate policy decision subnetwork and the angle policy decision subnetwork, respectively.

[0091] Based on the above embodiments, the method further includes training and updating the full-scale dilated convolutional network using subsequent network layers according to the following steps:

[0092] The feature image with labeled pixel indexes is input into the initialized policy network to obtain the labeled indexes corresponding to the predicted optimal dilation rates.

[0093] The optimal dilation rate corresponding to the predicted labeled index and the feature image with labeled pixel index are input into the initialized omnidirectional convolutional network to obtain the segmentation result;

[0094] Calculate the average intersection-union ratio (mIoU) between the semantic segmentation index corresponding to the feature image with labeled pixel index in the segmentation result and the true labeled index.

[0095] The dominance is calculated based on the average cross-union ratio and the moving average cross-union ratio corresponding to the feature images;

[0096] Based on the probability and dominance corresponding to the optimal dilation rate, the loss value corresponding to the feature image is calculated, and the loss value is backpropagated to the initialized policy network to update the initialized policy network.

[0097] Return to the step of "inputting the feature image with labeled pixel indexes into the initialized policy network" and continue until the number of iterations is satisfied.

[0098] The aforementioned dominance is obtained through the following functional expression:

[0099] advantage i = reward i - baseline;

[0100] Among them, advantage i The reward is the advantage score of the i-th labeled indexed pixel in the feature image. i is the cross-union ratio of the i-th labeled pixel, and is the moving average of the cross-union ratios of all labeled pixels in the baseline.

[0101] The loss value is obtained by the following function expression:

[0102] Loss = -E[log(p i ) advantage i ];

[0103] Wherein, log(p i ): the prediction probability corresponding to the inflation rate (pixel angle) index selected by the pixel with the i-th labeled index in the feature image, E is the expectation function.

[0104] The above initialization inflation rate strategy decision subnetwork is updated, which actually only updates the parameters of the inflation rate strategy decision subnetwork, and the parameters of SharedConv are updated by the conventional segmentation loss.

[0105] It can be understood that if the model selects the correct angle, the convolution can extract the complete object feature, and the mIoU will be high naturally.

[0106] Based on the above embodiment, further comprising training and updating the omnidirectional convolutional network according to the following steps by using subsequent network layers:

[0107] A preset discrete angle list, the angle strategy decision subnetwork predicts multiple potential orientations of the object to which each pixel belongs according to the input feature image;

[0108] Select one potential orientation from the multiple potential orientations, and rotate and segment the feature image according to the selected potential orientation through the omnidirectional shared convolutional subnetwork to obtain the rotated feature and the segmentation accuracy,

[0109] Take the segmentation accuracy as the reward, and calculate the advantage of the selected potential orientation;

[0110] Extract the probability of the selected potential orientation, calculate the loss value corresponding to the feature image according to the probability, and propagate the loss value to the trained strategy network to update the trained strategy network;

[0111] Return to execute the corresponding steps of the preset discrete angle list until the number of iterations is met.

[0112] The above preset discrete angle list is [0, 45, 90, 135, 180, 225, 270, 315].

[0113] It can be understood that if the model selects the correct angle, the convolution can extract the complete object feature, and the segmentation accuracy will be high naturally. Of course, the segmentation accuracy can be replaced by mIoU as the reward.

[0114] To further effectively process the input of extreme resolution and further improve the robustness of the neural network structure to the change of the shooting angle of the unmanned aerial vehicle, before training, the complete high-resolution (such as a pixel greater than 1920x1080) image is loaded into the display memory at one time, and then random cropping is performed, so as to maximize the diversity of the training samples and enable the neural network structure to directly learn to process large-size image blocks.

[0115] Unlike conventional street view segmentation, the unmanned aerial vehicle image can be captured from any angle. Therefore, the embodiment introduces random rotation data enhancement in training to improve the ability of the model to process images of different shooting angles. Specifically,

[0116] In one or more embodiments, when the resolution of the feature image (such as a 4K or 8K image) does not meet the set requirements (such as the resolution corresponding to the above-mentioned pixel greater than 1920x1080), the complete high-resolution feature image is loaded into the display memory at one time before training the strategy network, and then random cropping is performed, and the randomly cropped feature image is used as the sample image for training the inflation rate strategy decision subnetwork.

[0117] Further, when cropping the high-resolution feature image loaded at one time, the cropping coordinates are randomly generated, and the cropping coordinates of the cropping frame within the boundary of the loaded feature image are used as the cropping coordinates.

[0118] It can be understood that directly loading the entire super-large image into the GPU (CUDA) display memory can utilize the high bandwidth of the GPU for fast slicing, avoiding repeated IO reading.

[0119] It can be understood that the classification labels of the feature image after segmentation are cropped in the same way.

[0120] In one or more embodiments, when the resolution of the feature image does not meet the set requirements, when training the angle strategy decision subnetwork, a random angle is randomly generated within the angle range of [0, 360], and the feature image in the training sample of the omnidirectional shared convolution subnetwork is rotated by performing affine transformation according to the random angle.

[0121] For ease of understanding, generating a random angle is different from the street view with only a slight tilt, and the unmanned aerial vehicle scene needs to be fully covered within [0, 360].

[0122] Based on the above embodiment, in the semantic segmentation of the omnidirectional shared convolution subnetwork, the classification labels of the feature image after segmentation are rotated synchronously.

[0123] The embodiment solves the main bottleneck of the existing semantic segmentation model when applied to high-resolution, multi-scale, multi-angle unmanned aerial vehicle images through a targeted architecture and training design. Compared with the prior art, the neural network structure effectively improves the processing capability of the model for complex scenes under aerial view, extreme resolution changes, multi-scale problems and multi-angle problems, and can realize more fine and robust aerial scene understanding, thereby providing a reliable technical foundation for large-scale monitoring, precision agriculture and infrastructure inspection and other practical applications.

[0124] It should be understood that, although each step in the flowchart involved in each of the above embodiments is displayed in sequence according to the arrow, these steps are not necessarily executed in the order indicated by the arrow. Unless otherwise specified herein, the execution of these steps is not strictly limited in sequence, and these steps can be executed in other orders. Moreover, at least part of the steps in the flowchart involved in each of the above embodiments can include multiple steps or stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution sequence of these steps or stages is not necessarily sequential, but can be alternately or alternately executed with at least part of other steps or steps or stages in other steps.

[0125] The above only describes the preferred embodiments of the present application, and those skilled in the art know that various changes or equivalent replacements can be made to the features and embodiments without departing from the spirit and scope of the present application. In addition, under the guidance of the present application, the features and embodiments can be modified to adapt to specific conditions and materials without departing from the spirit and scope of the present application. Therefore, the present application is not limited by the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of the present application are within the protection scope of the present application.

Claims

1. A method for semantic segmentation of feature images captured by a drone, the method comprising: The method comprises the following steps: According to the feature image, the index corresponding to a plurality of inflation rates is predicted; The full-scale shared convolution is initialized according to the inflation rate corresponding to the index, the feature image is captured by using the initialized full-scale shared convolution, and the feature image under the effective scale is obtained; For each pixel of the feature image under the effective scale, a plurality of angle indexes are predicted; For each pixel, an angle corresponding to an angle index is selected, and the feature image under the effective scale corresponding to the selected angle index is extracted; The n*n convolution is decomposed into n*n 1*1 convolution, the extracted feature image under the effective scale is input into the 1*1 convolution, and the spatial basis feature of each pixel in the n*n kernel position is obtained; According to the selected angle and the original coordinates of the n*n convolution relative to the center, the spatial basis feature of each pixel is rotated and sampled, and after the rotation and sampling, the semantic feature of the feature image under the effective scale and the effective angle of the current center pixel is obtained by feature aggregation; The index corresponding to a plurality of inflation rates is predicted according to the feature image, comprising: A four-dimensional tensor of the feature image is obtained, wherein the four-dimensional tensor comprises an inflation rate range of the feature image; According to the inflation rate range in the four-dimensional tensor, the probability of each inflation rate of each pixel in the feature image within the inflation rate range is calculated; The indexes corresponding to the top k inflation rates with the highest probability are selected; For each pixel of the feature image under the effective scale, a plurality of angle indexes are predicted, comprising: A four-dimensional tensor of the feature image under the effective scale is obtained, wherein the four-dimensional tensor comprises a plurality of preset angles corresponding to the feature image under the effective scale; The probability of each angle corresponding to each pixel of the feature image under the effective scale is calculated; For each pixel, the indexes corresponding to the top k angles with the highest probability are selected. 2.The method of claim 1, wherein, The full-scale shared convolution is initialized according to the inflation rate corresponding to the index, the feature image is captured by using the initialized full-scale shared convolution, and the feature image under the effective scale is obtained; 3.The method of claim 1, wherein, According to the selected angle and the original coordinates of the n*n convolution relative to the center, the spatial basis feature of each pixel is rotated and sampled, and after the rotation and sampling, the semantic feature of the feature image under the effective scale and the effective angle of the current center pixel is obtained by feature aggregation; The index corresponding to a plurality of inflation rates is predicted according to the feature image, comprising: A four-dimensional tensor of the feature image is obtained, wherein the four-dimensional tensor comprises an inflation rate range of the feature image; 4.The method of claim 3, wherein, According to the inflation rate range in the four-dimensional tensor, the probability of each inflation rate of each pixel in the feature image within the inflation rate range is calculated; 5. A method of constructing a neural network structure, characterized by, The indexes corresponding to the top k inflation rates with the highest probability are selected; For each pixel of the feature image under the effective scale, a plurality of angle indexes are predicted, comprising: A four-dimensional tensor of the feature image under the effective scale is obtained, wherein the four-dimensional tensor comprises a plurality of preset angles corresponding to the feature image under the effective scale; The probability of each angle corresponding to each pixel of the feature image under the effective scale is calculated; For each pixel, the indexes corresponding to the top k angles with the highest probability are selected. The full-scale shared convolution is initialized according to the inflation rate corresponding to the index, the feature image is captured by using the initialized full-scale shared convolution, and the feature image under the effective scale is obtained; According to the selected angle and the original coordinates of the n*n convolution relative to the center, the spatial basis feature of each pixel is rotated and sampled, and after the rotation and sampling, the semantic feature of the feature image under the effective scale and the effective angle of the current center pixel is obtained by feature aggregation; The index corresponding to a plurality of inflation rates is predicted according to the feature image, comprising: A four-dimensional tensor of the feature image is obtained, wherein the four-dimensional tensor comprises an inflation rate range of the feature image; According to the inflation rate range in the four-dimensional tensor, the probability of each inflation rate of each pixel in the feature image within the inflation rate range is calculated; The indexes corresponding to the top k inflation rates with the highest probability are selected; For each pixel of the feature image under the effective scale, a plurality of angle indexes are predicted, comprising: A four-dimensional tensor of the feature image under the effective scale is obtained, wherein the four-dimensional tensor comprises a plurality of preset angles corresponding to the feature image under the effective scale; The probability of each angle corresponding to each pixel of the feature image under the effective scale is calculated; For each pixel, the indexes corresponding to the top k angles with the highest probability are selected. The full-scale shared convolution is initialized according to the inflation rate corresponding to the index, the feature image is captured by using the initialized full-scale shared convolution, and the feature image under the effective scale is obtained; According to the selected angle and the original coordinates of the n*n convolution relative to the center, the spatial basis feature of each pixel is rotated and sampled, and after the rotation and sampling, the semantic feature of the feature image under the effective scale and the effective angle of the current center pixel is obtained by feature aggregation; The index corresponding to a plurality of inflation rates is predicted according to the feature image, comprising: A four-dimensional tensor of the feature image is obtained, wherein the four-dimensional tensor comprises an inflation rate range of the feature image; According to the inflation rate range in the four-dimensional tensor, the probability of each inflation rate of each pixel in the feature image within the inflation rate range is calculated; The indexes corresponding to the top k inflation rates with the highest probability are selected; For each pixel of the feature image under the effective scale, a plurality of angle indexes are predicted, comprising: A four-dimensional tensor of the feature image under the effective scale is obtained, wherein the four-dimensional tensor comprises a plurality of preset angles corresponding to the feature image under the effective scale; The probability of each angle corresponding to each pixel of the feature image under the effective scale is calculated; For each pixel, the indexes corresponding to the top k angles with the highest probability are selected. The full-scale shared convolution is initialized according to the inflation rate corresponding to the index, the feature image is captured by using the initialized full-scale shared convolution, and the feature image under the effective scale is obtained; The omnidirectional convolution network comprises an angle policy decision subnetwork and an omnidirectional shared convolution subnetwork; the angle policy decision subnetwork is configured to predict a plurality of angle indexes for each pixel of a feature image at an effective scale; the omnidirectional shared convolution subnetwork is configured to select an angle corresponding to an index for each pixel, extract a feature image at an effective scale corresponding to the selected index, decompose an n*n convolution into n*n 1*1 convolutions, input the extracted feature image at the effective scale into the 1*1 convolution to obtain a spatial basis feature of each pixel at n*n kernel positions, and perform rotation sampling on the spatial basis feature of each pixel according to the selected angle and the original coordinates of the n*n convolution relative to a center, and perform feature aggregation after the rotation sampling to obtain a semantic feature of the feature image at the effective scale and the effective angle of the current center pixel; The expansion rate policy decision subnetwork comprises a first multilayer perceptron, a first Softmax layer and an expansion rate index selection layer connected in sequence, and the expansion rate index selection layer is connected with the full-scale shared convolution; The angle policy decision subnetwork comprises a second multilayer perceptron, a second Softmax layer and an angle index selection layer connected in sequence; The omnidirectional shared convolution subnetwork comprises an angle rotation layer, an angle convolution layer and a feature aggregation layer connected in sequence, and the angle index selection layer is connected with the angle rotation layer; The subsequent network layers for training and updating the full-scale expansion convolution network and the omnidirectional convolution network comprise a reward module, an advantage function module and a loss function module connected in sequence, the reward module is connected with the output of the full-scale shared convolution and the output of the omnidirectional shared convolution subnetwork, and the loss function module is connected with the expansion rate policy decision subnetwork and the angle policy decision subnetwork.

6. The method of claim 5, wherein, When the resolution of the feature image does not meet the set requirement, the complete high-resolution feature image is loaded into the video memory at one time before training the expansion rate policy decision subnetwork, and then the feature image is randomly cropped, and the randomly cropped feature image is used as a sample image for training the expansion rate policy decision subnetwork; When the resolution of the feature image does not meet the set requirement, a random angle is randomly generated in the angle range of [0, 360] when the angle policy decision subnetwork is trained, and the result of the affine transformation performed on the feature image in the training sample by the omnidirectional shared convolution subnetwork is rotated according to the random angle.

Images