A multi-scale rotation bounding box regression method and device for aerial images

Abstract

The present invention discloses a method and device for multi-scale rotation bounding box regression of aerial images, the method comprising: obtaining an aerial image sample data set; preprocessing the aerial image sample data set to obtain an aerial image rotation bounding box regression data set; processing the aerial image rotation bounding box regression data set based on an aerial image rotation bounding box regression model to obtain aerial image object category and position information. The present invention provides a method and device for multi-scale rotation bounding box regression of aerial images, which solves the problem of difficulty in regressing densely arranged objects of different scales in aerial images, improves the perception ability of multi-scale objects in aerial images and the accuracy of rotation bounding box regression.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence theory and algorithm technology, and in particular to a method and device for multi-scale rotation bounding box regression of aerial images. Background Art

[0002] With the gradual development and mature application of horizontal box position regression methods, regression methods using rotated bounding boxes are also gradually applied to various images. Especially in aerial images with a bird's-eye view, most objects are not arranged horizontally. Especially for objects with large aspect ratios and densely arranged objects, traditional horizontal box regression methods cannot accurately describe the edge position information of objects. Rotated bounding boxes can reduce the overlap between object bounding boxes, remove redundant background information, and accurately regress the edge information of objects. Current methods can be divided into the following four categories: directly regressing the position of the rotated box by extracting the rotation-invariant features of the object; directly predicting the direction of the object through independent angle regression and using a separate loss function for constraint; setting a rotated anchor box, presetting multiple directional anchor boxes at each pixel position at a fixed angle interval to match the different directions of the object's rotation; and obtaining position information through feature enhancement and regression using the anchor-free method. The mask is generated by the segmentation method, and the rotated bounding box is obtained using the minimum circumscribed quadrilateral area of ​​the mask.

[0003] However, due to the different resolutions of aerial images and the large differences in the sizes of objects in the images, it is difficult for general feature extraction structures to accurately extract multi-scale objects, which increases the difficulty of position regression. In addition, there are a large number of densely arranged objects in the overhead image, and the background is relatively cluttered, making it difficult to determine the exact position. Especially for objects with large aspect ratios, slight angle deviations during bounding box regression will lead to large changes in the intersection-and-union ratio. In addition, the definition method and angle periodicity of the rotated bounding box will lead to discontinuity in the intersection-and-union ratio of the bounding box, affecting the regression performance. Therefore, it is necessary to further optimize the rotated bounding box regression method. Summary of the invention

[0004] The technical problem to be solved by the present invention is that the present invention provides a method and device for multi-scale rotated bounding box regression of aerial images, which solves the problem of difficulty in regressing densely arranged objects of different scales in aerial images, improves the perception ability of multi-scale objects in aerial images and the accuracy of rotated bounding box regression.

[0005] In order to solve the above technical problems, the first aspect of the embodiments of the present invention discloses a multi-scale rotation bounding box regression method for aerial images, the method comprising:

[0006] S1, obtain the aerial image sample dataset;

[0007] S2, preprocessing the aerial image sample dataset to obtain an aerial image rotation bounding box regression dataset;

[0008] S3, based on the aerial image rotation bounding box regression model, processing the aerial image rotation bounding box regression dataset to obtain aerial image object category and position information.

[0009] As an optional implementation manner, in the first aspect of the embodiment of the present invention, the preprocessing of the aerial image sample dataset to obtain the aerial image rotation bounding box regression dataset includes:

[0010] S21, traversing the aerial image sample data set to obtain all aerial image sample data;

[0011] The aerial image sample data set includes a plurality of aerial image sample data;

[0012] S22, identifying and cutting any of the aerial image sample data to obtain a target aerial image object set;

[0013] The target aerial image object set includes a plurality of target aerial image objects;

[0014] S23, labeling any of the target aerial image objects to obtain a target aerial image object label;

[0015] S24, processing the target aerial image object based on the target aerial image object label to obtain aerial image rotation bounding box regression data;

[0016] S25, combining all the aerial image rotation bounding box regression data in order to obtain an aerial image rotation bounding box regression data set;

[0017] The aerial image rotation bounding box regression data set includes a plurality of aerial image rotation bounding box regression data.

[0018] As an optional implementation manner, in the first aspect of the embodiment of the present invention, the processing of the target aerial image object based on the target aerial image object label to obtain the aerial image rotation bounding box regression data includes:

[0019] S241, segmenting the target aerial image object to obtain a target aerial image object slice information set;

[0020] The target aerial image object slice information set includes a plurality of target aerial image object slice information;

[0021] S242, based on the target aerial image object label information, processing any of the target aerial image object slice information to obtain the target aerial image object slice label information;

[0022] S243, combining the target aerial image object slice label information in order to obtain a target aerial image object label information set;

[0023] S244, performing screening processing on the target aerial image object label information set to obtain a processed aerial image object label information set;

[0024] S245, using a data conversion model, converting the processed aerial image object label information set to obtain aerial image rotation bounding box regression data;

[0025] The aerial image rotation bounding box regression data includes aerial image object rotation bounding box information and aerial image object slice label data information.

[0026] As an optional implementation, in the first aspect of the embodiment of the present invention, the converting the processed aerial image object label information set to obtain the aerial image rotation bounding box regression data includes:

[0027] S2451, preset minimum bounding rectangle information set;

[0028] S2452, obtaining the four-point coordinate information of the aerial image object slice and the aerial image object label information of the processed aerial image object label information set;

[0029] S2453, determining whether the four-point coordinate frame of the four-point coordinate information of the aerial image object slice is a rectangle, and obtaining a shape determination result of the aerial image object slice;

[0030] When the result of determining the shape of the aerial image object slice is yes, the four-point coordinate information of the aerial image object slice is converted to obtain the rotation bounding box information of the aerial image object, and S2456 is executed;

[0031] When the result of determining the shape of the aerial image object slice is no, executing S2454;

[0032] S2454, acquiring the four-point coordinate information of the minimum bounding rectangle information set, and obtaining the four-point coordinate information of the minimum bounding rectangle;

[0033] S2455, converting the coordinate information of the four points of the minimum circumscribed rectangle to obtain the rotation bounding box information of the aerial image object;

[0034] S2456: Combine the aerial image object rotation bounding box information and the aerial image object label information in order to obtain aerial image rotation bounding box regression data.

[0035] As an optional implementation, in the first aspect of the embodiment of the present invention, the aerial image rotation bounding box regression model is used to process the aerial image rotation bounding box regression dataset to obtain the aerial image object category and position information, including:

[0036] S31, dividing the aerial image rotation bounding box regression dataset to obtain an aerial image regression training set, an aerial image regression verification set, and an aerial image regression test set;

[0037] S32, using the aerial image regression training set and the aerial image regression verification set, training the aerial image rotation bounding box regression model to obtain an aerial image rotation bounding box regression optimization model;

[0038] S33, using the aerial image rotation bounding box regression optimization model to process the aerial image regression test set to obtain aerial image object category and position information.

[0039] As an optional implementation, in the first aspect of the embodiment of the present invention, the aerial image rotation bounding box regression model is trained using the aerial image regression training set and the aerial image regression verification set to obtain the aerial image rotation bounding box regression optimization model, including:

[0040] S321, set the total number of iterations a, and initialize the number of iterations b=0;

[0041] S322, using the aerial image rotation bounding box regression model, processing the aerial image regression training set to obtain first aerial image regression detection information;

[0042] The first aerial image regression detection information includes first aerial image object category information and first aerial image object position information;

[0043] S323, using a first loss function, calculating the first aerial image object category information and the aerial image object label information of the aerial image regression training set to obtain a first difference value;

[0044] Using a second loss function, calculating the first aerial image object position information and the aerial image object rotation bounding box information of the aerial image regression training set to obtain a second difference value;

[0045] S324, determining whether the first difference value is less than a first preset threshold, and obtaining a second determination result;

[0046] Determine whether the second difference value is less than a second preset threshold, and obtain a third determination result;

[0047] S325, when the second judgment result and the third judgment result are both yes, execute S326;

[0048] When the second judgment result or the third judgment result is no, increasing the number of training times by 1;

[0049] Determine whether the training times value is greater than a preset training times threshold, and obtain a training times determination result;

[0050] When the result of the training times determination is yes, executing S326;

[0051] When the result of the training number determination is no, executing S322;

[0052] S326, updating parameters of the aerial image rotation bounding box regression model to obtain an updated aerial image rotation bounding box regression model;

[0053] S327, using the updated aerial image rotation bounding box regression model, processing the aerial image regression verification set to obtain second aerial image regression detection information;

[0054] The second aerial image regression detection information includes second aerial image object category information and second aerial image object position information;

[0055] S328, comparing the second aerial image object category information and the second aerial image object position information with the aerial image object rotation bounding box information and the aerial image object label information of the aerial image regression validation set respectively to obtain prediction accuracy information;

[0056] S329, determining whether the prediction accuracy information is greater than a preset accuracy threshold, and obtaining an accuracy determination result;

[0057] When the accuracy determination result is no, executing S321;

[0058] If the accuracy judgment result is yes, the training process of the aerial image rotation bounding box regression model is completed, and the updated aerial image rotation bounding box regression model is updated to the aerial image rotation bounding box regression optimization model.

[0059] As an optional implementation, in the first aspect of the embodiment of the present invention, the aerial image rotation bounding box regression model includes: a data input module, a feature extraction module, a classification module, a regression module and a data output module;

[0060] The output end of the data input module is data-connected to the input end of the feature extraction module; the output end of the feature extraction module is data-connected to the input end of the classification module and the input end of the regression module respectively; the output end of the classification module and the output end of the regression module are both connected to the input end of the data output module;

[0061] The data input module is used to obtain and send the aerial image sample data set to the feature extraction module;

[0062] The feature extraction module is used to receive and process the aerial image sample data set, obtain and send the aerial image object feature information to the classification module and the regression module;

[0063] The classification module is used to receive and process the aerial image object feature information, obtain and send the aerial image object classification information to the data output module;

[0064] The regression module is used to receive and process the aerial image object feature information, obtain and send the aerial image object rotation bounding box regression information to the data output module;

[0065] The data output module is used to receive and fuse the aerial image object classification information and the aerial image object rotation bounding box regression information to obtain the aerial image object rotation bounding box regression information.

[0066] As an optional implementation, in the first aspect of the embodiment of the present invention, the feature extraction module includes: a first feature extraction submodule, a dense connection submodule, a first fusion submodule, and a second fusion submodule;

[0067] The output end of the first feature extraction submodule is data-connected to the input end of the dense connection submodule, the input end of the first fusion submodule, and the input end of the second fusion submodule; the output end of the dense connection submodule is data-connected to the input end of the first fusion submodule and the input end of the second fusion submodule;

[0068] The first feature extraction submodule is used to perform feature extraction and upsampling processing on the aerial image sample data set to obtain first feature information, second feature information, third feature information and fourth feature information;

[0069] The dense connection submodule is used to perform upsampling, downsampling, convolution and fusion processing on the fourth feature information to obtain fifth feature information, sixth feature information, seventh feature information, eighth feature information and ninth feature information;

[0070] The first fusion submodule is used to perform a first fusion process on the first feature information and the fifth feature information, the second feature information and the sixth feature information, the third feature information, the seventh feature information, the eighth feature information and the ninth feature information to obtain aerial image classification information;

[0071] The second fusion submodule is used to perform a second fusion process on the first feature information, the fifth feature information, the second feature information, the sixth feature information, the third feature information, the seventh feature information, the eighth feature information and the ninth feature information to obtain aerial image regression information.

[0072] A second aspect of the present invention discloses a multi-scale rotation bounding box regression device for aerial images, the device comprising:

[0073] A memory storing executable program code;

[0074] a processor coupled to the memory;

[0075] The processor calls the executable program code stored in the memory to execute part or all of the steps in the multi-scale rotation bounding box regression method for aerial images disclosed in the first aspect of the embodiment of the present invention.

[0076] The third aspect of the present invention discloses a computer-readable storage medium, which stores computer instructions. When the computer instructions are called, they are used to execute some or all of the steps in the multi-scale rotated bounding box regression method for aerial images disclosed in the first aspect of an embodiment of the present invention.

[0077] Compared with the prior art, the embodiments of the present invention have the following beneficial effects:

[0078] In feature extraction, the present invention uses a densely connected multi-path hole convolution layer to cover objects of different sizes in aerial images, extract feature maps adaptively at multiple scales, and use an upsampling skip connection in parallel with the hole convolution layer to directly transmit accurate feature positions. This network structure design can not only extract dense and rich multi-scale spatial semantic information, but also perform feature transmission and fusion in the deep network, further improving the perception of objects of different sizes in aerial images.

[0079] The present invention uses a power-transformed Gaussian distribution metric as a first loss function and uses optimized relative entropy divergence to perform position regression, thereby improving the accuracy of position and angle regression of a rotation bounding box. BRIEF DESCRIPTION OF THE DRAWINGS

[0080] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0081] Figure 1 It is a flowchart of a multi-scale rotation bounding box regression method for aerial images disclosed in an embodiment of the present invention;

[0082] Figure 2 It is a structural schematic diagram of a multi-scale rotation bounding box regression device for aerial images disclosed in an embodiment of the present invention. DETAILED DESCRIPTION

[0083] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0084] The terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, device, product or equipment that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units that are not listed, or may optionally include other steps or units that are inherent to these processes, methods, products or equipment.

[0085] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present invention. The appearance of the phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0086] The present invention discloses a method and device for multi-scale rotation bounding box regression of aerial images, which improves the search speed and positioning accuracy, solves the problem of difficulty in regressing densely arranged objects of different scales in aerial images, and improves the perception ability of multi-scale objects in aerial images and the accuracy of rotation bounding box regression.

[0087] Embodiment 1

[0088] See also Figure 1 , Figure 1 1 is a flow chart of a multi-scale rotation bounding box regression method for aerial images disclosed in an embodiment of the present invention. Figure 1 The described multi-scale rotation bounding box regression method for aerial images is applied to an aerial image processing system, such as a local server or cloud server for spatial region search and positioning processing, and the embodiments of the present invention are not limited thereto. Figure 1 As shown, the spatial area search and positioning method may include the following operations:

[0089] S1, obtain the aerial image sample dataset;

[0090] It should be noted that, in this embodiment, the DOTA1.0 dataset is used as the aerial image sample dataset;

[0091] It should be noted that the DOTA1.0 dataset is a large-scale dataset of aerial and remote sensing images. The images come from different sensors and platforms. The image sizes range from 800×800 to 20000×20000 pixels, and contain objects of various scales, directions and shapes.

[0092] It should be noted that the DOTA 1.0 version contains 15 common categories, 2,806 images and 188,282 instances, including airplanes, ships, oil storage tanks, baseball fields, tennis courts, swimming pools, athletic fields, harbors, bridges, large vehicles, small vehicles, helicopters, roundabouts, football fields, and basketball courts;

[0093] S2, preprocessing the aerial image sample dataset to obtain an aerial image rotation bounding box regression dataset;

[0094] It should be noted that the annotation format of the image objects in the aerial image sample dataset is an arbitrary quadrilateral;

[0095] S3, based on the aerial image rotation bounding box regression model, processing the aerial image rotation bounding box regression dataset to obtain aerial image object category and position information.

[0096] It can be seen that the implementation of the multi-scale rotated bounding box regression method for aerial images described in the embodiment of the present invention solves the problem of difficulty in regressing densely arranged objects of different scales in aerial images, and improves the perception ability of multi-scale objects in aerial images and the accuracy of rotated bounding box regression.

[0097] In an optional embodiment, in the above step S2, the preprocessing of the aerial image sample dataset to obtain the aerial image rotation bounding box regression dataset includes:

[0098] S21, traversing the aerial image sample data set to obtain all aerial image sample data;

[0099] It should be noted that the aerial image sample data set includes a number of aerial image sample data;

[0100] It should be noted that the aerial image sample data is DOTA1.0 aerial image data;

[0101] S22, identifying and cutting any of the aerial image sample data to obtain a target aerial image object set;

[0102] It should be noted that the target aerial image object set includes a number of target aerial image objects;

[0103] It should be noted that the target aerial image object refers to an object contained in the aerial image;

[0104] S23, labeling any of the target aerial image objects to obtain label information of the target aerial image object;

[0105] It should be noted that the step of labeling any of the target aerial image objects to obtain the target aerial image object label information includes:

[0106] S231, acquiring the target aerial image object;

[0107] S232, marking the edge and orientation of the target aerial image object to obtain a non-horizontal bounding box with direction;

[0108] S233, obtaining the four-point coordinates and object category of the non-horizontal bounding box to obtain the target aerial image object label information;

[0109] S24, processing the target aerial image object based on the target aerial image object label to obtain aerial image rotation bounding box regression data;

[0110] S25, combining all the aerial image rotation bounding box regression data in order to obtain an aerial image rotation bounding box regression data set;

[0111] It should be noted that the aerial image rotation bounding box regression dataset includes a plurality of aerial image rotation bounding box regression data.

[0112] It can be seen that the implementation of the multi-scale rotated bounding box regression method for aerial images described in the embodiment of the present invention solves the problem of difficulty in regressing densely arranged objects of different scales in aerial images, and improves the perception ability of multi-scale objects in aerial images and the accuracy of rotated bounding box regression.

[0113] In another optional embodiment, in the above step S24, the processing of the target aerial image object based on the target aerial image object label information to obtain the aerial image rotation bounding box regression data includes:

[0114] S241, segmenting the target aerial image object to obtain a target aerial image object slice information set;

[0115] It should be noted that the target aerial image object slice information set includes a plurality of target aerial image object slice information;

[0116] It should be noted that, in order for the target aerial image objects to be segmented, the segmentation sliding window leaves a distance of 1.5 to 2 times the minimum object width;

[0117] It should be noted that the segmentation of the target aerial image object to obtain the target aerial image object slice information set includes:

[0118] S2411, for the entire original aerial image, the region containing the object is cut out, and the region without the object is discarded, so as to obtain the object region of the target aerial image;

[0119] S2412, according to the size of the target aerial image object, slidingly cutting the target aerial image object region from left to right and from top to bottom to obtain a target aerial image object slice set;

[0120] It should be noted that the target aerial image object slice set includes a plurality of target aerial image object slices;

[0121] S242, based on the target aerial image object label information, processing any of the target aerial image object slice information to obtain the target aerial image object slice label information;

[0122] It should be noted that processing any of the target aerial image object slice information includes subtracting the target aerial image object label coordinates from the target aerial image object slice upper left corner coordinates to obtain a new target aerial image object slice label;

[0123] S243, combining the target aerial image object slice label information in order to obtain a target aerial image object label information set;

[0124] It should be noted that the combination in order means arranging in the order of slices;

[0125] S244, performing screening processing on the target aerial image object label information set to obtain a processed aerial image object label information set;

[0126] It should be noted that the screening process refers to screening out incomplete target aerial image object label information caused by imaging conditions in the target aerial image object label information set;

[0127] S245, using a data conversion model, converting the processed aerial image object label information set to obtain aerial image rotation bounding box regression data;

[0128] It should be noted that the data conversion model expression is:

[0129]

[0130] θ is obtained by the rotation angle returned by OpenCV's cv2.minAreaRect function;

[0131] Wherein, i represents the endpoint coordinate index of the target aerial image object slice; (x i ,y i ) represents the four-point coordinates of the target aerial image object slice; (x, y) represents the coordinate value of the aerial image rotation bounding box; W represents the long side value of the aerial image rotation bounding box; h represents the short side value of the aerial image rotation bounding box; θ represents the angle between the aerial image rotation bounding box and the horizontal axis;

[0132] It should be noted that, through this processing, the four-point coordinates (x1, y1, x2, y2, x3, y3, x4, y4) of the target aerial image object slice are converted into the aerial image rotation bounding box label of the five-parameter coordinates (x, y, w, h, θ);

[0133] It should be noted that, in this embodiment, i≤4, N=4;

[0134] The aerial image rotation bounding box regression data includes aerial image object rotation bounding box information and aerial image object slice label data information.

[0135] It can be seen that the implementation of the multi-scale rotated bounding box regression method for aerial images described in the embodiment of the present invention solves the problem of difficulty in regressing densely arranged objects of different scales in aerial images, and improves the perception ability of multi-scale objects in aerial images and the accuracy of rotated bounding box regression.

[0136] In yet another optional embodiment, in the above step S245, the converting process of the processed aerial image object label information set to obtain the aerial image rotation bounding box regression data includes:

[0137] S2451, preset minimum bounding rectangle information set;

[0138] S2452, obtaining the four-point coordinate information of the aerial image object slice and the aerial image object label information of the processed aerial image object label information set;

[0139] S2453, determining whether the four-point coordinate frame of the four-point coordinate information of the aerial image object slice is a rectangle, and obtaining a shape determination result of the aerial image object slice;

[0140] When the result of determining the shape of the aerial image object slice is yes, the four-point coordinate information of the aerial image object slice is converted to obtain the rotation bounding box information of the aerial image object, and S2456 is executed;

[0141] When the result of determining the shape of the aerial image object slice is no, executing S2454;

[0142] S2454, acquiring the four-point coordinate information of the minimum bounding rectangle information set, and obtaining the four-point coordinate information of the minimum bounding rectangle;

[0143] S2455, converting the coordinate information of the four points of the minimum circumscribed rectangle to obtain the rotation bounding box information of the aerial image object;

[0144] S2456: Combine the aerial image object rotation bounding box information and the aerial image object label information in order to obtain aerial image rotation bounding box regression data.

[0145] It can be seen that the implementation of the multi-scale rotated bounding box regression method for aerial images described in the embodiment of the present invention solves the problem of difficulty in regressing densely arranged objects of different scales in aerial images, and improves the perception ability of multi-scale objects in aerial images and the accuracy of rotated bounding box regression.

[0146] In another optional embodiment, in the above step S3, the aerial image rotation bounding box regression data set is processed based on the aerial image rotation bounding box regression model to obtain the aerial image object category and position information, including:

[0147] S31, dividing the aerial image rotation bounding box regression dataset to obtain an aerial image regression training set, an aerial image regression verification set, and an aerial image regression test set;

[0148] S32, using the aerial image regression training set and the aerial image regression verification set, training the aerial image rotation bounding box regression model to obtain an aerial image rotation bounding box regression optimization model;

[0149] S33, using the aerial image rotation bounding box regression optimization model to process the aerial image regression test set to obtain aerial image object category and position information.

[0150] It can be seen that the implementation of the multi-scale rotated bounding box regression method for aerial images described in the embodiment of the present invention solves the problem of difficulty in regressing densely arranged objects of different scales in aerial images, and improves the perception ability of multi-scale objects in aerial images and the accuracy of rotated bounding box regression.

[0151] In another optional embodiment, in the above step S32, the use of the aerial image regression training set and the aerial image regression verification set to train the aerial image rotation bounding box regression model to obtain the aerial image rotation bounding box regression optimization model includes:

[0152] S321, set the total number of iterations a, and initialize the number of iterations b=0;

[0153] It should be noted that the total number of iterations is set to 20 and the learning rate is set to 0.0005;

[0154] S322, using the aerial image rotation bounding box regression model, processing the aerial image regression training set to obtain first aerial image regression detection information;

[0155] The first aerial image regression detection information includes first aerial image object category information and first aerial image object position information;

[0156] S323, using a first loss function, calculating the first aerial image object category information and the aerial image object label information of the aerial image regression training set to obtain a first difference value;

[0157] Using a second loss function, calculating the first aerial image object position information and the aerial image object rotation bounding box information of the aerial image regression training set to obtain a second difference value;

[0158] S324, determining whether the first difference value is less than a first preset threshold, and obtaining a second determination result;

[0159] It should be noted that the first difference value includes a first foreground difference value and a first background difference value;

[0160] It should be noted that the first preset threshold includes a first foreground threshold value and a first background threshold value;

[0161] It should be noted that the first foreground threshold is set to 0.5, and the first background threshold is set to 0.4;

[0162] Determine whether the second difference value is less than a second preset threshold, and obtain a third determination result;

[0163] It should be noted that the second difference value includes a second foreground difference value and a second background difference value;

[0164] It should be noted that the second preset threshold includes a second foreground threshold value and a second background threshold value;

[0165] It should be noted that the second foreground threshold is set to 0.7, and the second background threshold is set to 0.6;

[0166] S325, when the second judgment result and the third judgment result are both yes, execute S326;

[0167] When the second judgment result or the third judgment result is no, increasing the number of training times by 1;

[0168] Determine whether the training times value is greater than a preset training times threshold, and obtain a training times determination result;

[0169] When the result of the training times determination is yes, executing S326;

[0170] When the result of the training number determination is no, executing S322;

[0171] S326, updating parameters of the aerial image rotation bounding box regression model to obtain an updated aerial image rotation bounding box regression model;

[0172] S327, using the updated aerial image rotation bounding box regression model, processing the aerial image regression verification set to obtain second aerial image regression detection information;

[0173] The second aerial image regression detection information includes second aerial image object category information and second aerial image object position information;

[0174] S328, comparing the second aerial image object category information and the second aerial image object position information with the aerial image object rotation bounding box information and the aerial image object label information of the aerial image regression validation set respectively to obtain prediction accuracy information;

[0175] S329, determining whether the prediction accuracy information is greater than a preset accuracy threshold, and obtaining an accuracy determination result;

[0176] It should be noted that the accuracy threshold is set to 1;

[0177] When the accuracy determination result is no, executing S321;

[0178] If the accuracy judgment result is yes, the training process of the aerial image rotation bounding box regression model is completed, and the updated aerial image rotation bounding box regression model is updated to the aerial image rotation bounding box regression optimization model.

[0179] It can be seen that the implementation of the multi-scale rotated bounding box regression method for aerial images described in the embodiment of the present invention solves the problem of difficulty in regressing densely arranged objects of different scales in aerial images, and improves the perception ability of multi-scale objects in aerial images and the accuracy of rotated bounding box regression.

[0180] In another optional embodiment, in the above step S323, the first loss function expression is:

[0181]

[0182] Among them, L l represents the first loss function value; n represents the category index; ρ represents the classification loss weight; c represents the category index; N represents the number of categories; M represents the number of samples; i represents the sample index; y ic represents the sample attribution value; P ic represents the probability that sample i belongs to category c;

[0183] It should be noted that the classification loss weight is set by the user based on experience. In this embodiment, ρ is set to 1;

[0184] It should be noted that the sample attribution value has two values. When sample i belongs to category c, y ic =1; when sample i does not belong to category c, y ic =0;

[0185] The second loss function expression is:

[0186]

[0187] Among them, L r represents the second loss function value; n represents the category index; μ represents the regression loss weight; (q t ,∑ t ) represents the reference frame of the aerial image object (x t ,y t ,ω t ,h t ,θ t )’s Gaussian distribution; (q p ,∑ p ) represents the predicted box of the aerial image object (x p ,y p ,ω p ,h p ,θ p)’s Gaussian distribution; τ represents a hyperparameter; α represents a power parameter; ε represents a regularization term;

[0188] It should be noted that the regression loss weight is set by the user based on experience. In this embodiment, μ is set to 2;

[0189] It should be noted that the rotation rectangle (x, y, ω, h, θ) of the aerial image object conforms to the Gaussian distribution, and The hyperparameter τ = 1; the power parameter α = 0.5; the regularization term ε = 1×10 -3 ;

[0190] It can be seen that the implementation of the multi-scale rotated bounding box regression method for aerial images described in the embodiment of the present invention solves the problem of difficulty in regressing densely arranged objects of different scales in aerial images, and improves the perception ability of multi-scale objects in aerial images and the accuracy of rotated bounding box regression.

[0191] In yet another optional embodiment, in the above step S3, the aerial image rotation bounding box regression model comprises: a data input module, a feature extraction module, a classification module, a regression module and a data output module;

[0192] The output end of the data input module is data-connected to the input end of the feature extraction module; the output end of the feature extraction module is data-connected to the input end of the classification module and the input end of the regression module respectively; the output end of the classification module and the output end of the regression module are both connected to the input end of the data output module;

[0193] The data input module is used to obtain and send the aerial image sample data set to the feature extraction module;

[0194] The feature extraction module is used to receive and process the aerial image sample data set, obtain and send the aerial image object feature information to the classification module and the regression module;

[0195] The classification module is used to receive and process the aerial image object feature information, obtain and send the aerial image object classification information to the data output module;

[0196] The regression module is used to receive and process the aerial image object feature information, obtain and send the aerial image object rotation bounding box regression information to the data output module;

[0197] The data output module is used to receive and fuse the aerial image object classification information and the aerial image object rotation bounding box regression information to obtain the aerial image object rotation bounding box regression information.

[0198] It can be seen that the implementation of the multi-scale rotated bounding box regression method for aerial images described in the embodiment of the present invention solves the problem of difficulty in regressing densely arranged objects of different scales in aerial images, and improves the perception ability of multi-scale objects in aerial images and the accuracy of rotated bounding box regression.

[0199] In an optional embodiment, the feature extraction module includes: a first feature extraction submodule, a dense connection submodule, a first fusion submodule, and a second fusion submodule;

[0200] The output end of the first feature extraction submodule is data-connected to the input end of the dense connection submodule, the input end of the first fusion submodule, and the input end of the second fusion submodule; the output end of the dense connection submodule is data-connected to the input end of the first fusion submodule and the input end of the second fusion submodule;

[0201] The first feature extraction submodule is used to perform feature extraction and upsampling processing on the aerial image sample data set to obtain first feature information, second feature information, third feature information and fourth feature information;

[0202] It should be noted that the first feature extraction submodule adopts a traditional feature pyramid structure, uses a bottom-up structure to extract feature maps of different scales and outputs the first feature information, the second feature information, the third feature information and the fourth feature information;

[0203] It should be noted that the first feature information represents the feature information obtained by first performing convolution processing with a convolution kernel of 3×3 and a step size of 2 and a pooling processing with a pooling kernel of 3×3 and a pooling kernel stride of 2 on the input aerial image sample data set, obtaining a first feature map, and then performing convolution processing with a convolution kernel of 1×1, a convolution kernel of 3×3 and a step size of 1 and a convolution kernel of 1×1 on the first feature map;

[0204] It should be noted that the second feature information refers to feature information obtained after performing convolution processing with a convolution kernel of 1×1, a convolution kernel of 3×3 and a step size of 2 and downsampling processing on the first feature information;

[0205] It should be noted that the third feature information refers to feature information obtained after performing convolution processing with a convolution kernel of 1×1, a convolution kernel of 3×3 and a step size of 2 and downsampling processing on the second feature information;

[0206] It should be noted that the fourth feature information refers to feature information obtained after performing convolution processing with a convolution kernel of 1×1, a convolution kernel of 3×3 and a step size of 2 and downsampling processing on the third feature information;

[0207] The dense connection submodule is used to perform upsampling, downsampling, convolution and fusion processing on the fourth feature information to obtain fifth feature information, sixth feature information, seventh feature information, eighth feature information and ninth feature information;

[0208] It should be noted that the densely connected submodule uses a top-down structure to perform bilinear interpolation upsampling processing on the high-resolution fourth feature information to obtain the tenth feature information;

[0209] It should be noted that the dense connection submodule performs bilinear difference upsampling on the tenth feature information to obtain the eleventh feature information;

[0210] It should be noted that the dense connection submodule performs bilinear difference upsampling on the eleventh feature information to obtain the twelfth feature information;

[0211] It should be noted that the tenth characteristic information is element-wise added to the first characteristic information to obtain the fifth characteristic information;

[0212] It should be noted that the tenth characteristic information and the second characteristic information are element-wise added to obtain the fifth characteristic information;

[0213] It should be noted that the eleventh characteristic information is element-wise added to the third characteristic information to obtain the sixth characteristic information;

[0214] It should be noted that the twelfth characteristic information and the fourth characteristic information are element-wise added to obtain the seventh characteristic information;

[0215] It should be noted that the dense connection submodule performs 1x1 convolution on the tenth feature information to obtain the eighth feature information;

[0216] It should be noted that the dense connection submodule performs 1x1 convolution on the eighth feature information to obtain the ninth feature information;

[0217] It should be noted that the convolution processing of the densely connected submodule is implemented by a 5-level cascade of dilated convolutional layers. The dilation rate increases with the number of layers. The neurons of the intermediate feature map can encode semantic information from different scales. Each layer of neurons is densely connected to solve the problem of gradient disappearance in deep convolutional neural networks. At the same time, it covers the semantic information of multi-scale large receptive fields. Each layer can be expressed as:

[0218] y l =A k，r (Concat(y l-1 ,y l-2 , ..., y0))

[0219] Among them A k，r In (), k represents the convolution kernel size; r represents the dilation rate; y l represents the lth convolutional layer;

[0220] It should be noted that the size of k is 3; the void ratios of layers y1 to y5 are set to 3, 6, 12, 18, and 24, respectively.

[0221] An upsampling skip connection U() is connected in parallel with the dilated convolutional layer to directly transfer the accurate feature location, which can be expressed as:

[0222] D = Cony(∑(U(c), A(c)))

[0223] Where C represents the feature map extracted from the bottom-up pyramid structure;

[0224] The first fusion submodule is used to perform a first fusion process on the first feature information and the fifth feature information, the second feature information and the sixth feature information, the third feature information, the seventh feature information, the eighth feature information and the ninth feature information to obtain aerial image classification information;

[0225] It should be noted that the first fusion processing means extracting a nonlinear representation of learning data through features, and using a softmax activation function to convert the network output into a probability distribution, output the probability of each category, predict the object category, and obtain the aerial image classification information;

[0226] The second fusion submodule is used to perform a second fusion process on the first feature information, the fifth feature information, the second feature information, the sixth feature information, the third feature information, the seventh feature information, the eighth feature information and the ninth feature information to obtain the aerial image regression information;

[0227] It should be noted that the second fusion processing means performing feature extraction and nonlinear transformation, obtaining predicted object bounding box coordinates through the second loss function regression, and obtaining the aerial image regression information;

[0228] It should be noted that in this embodiment, a TITAN RTX graphics card with a video memory of 24G is used, and the DOTA1.0 training set is used to optimize the network weights of the rotation bounding box regression network. The number of training iterations is set to 540,000, the number of training times is set to 20, the initial learning rate is set to 0.0005, and it is decayed in the 8th and 14th rounds to facilitate the convergence of the object function to the minimum value.

[0229] The accuracy of the results of different classification regression methods in DOTA1.0 was compared. The multi-scale rotation bounding box regression method for aerial images according to the embodiment of the present invention achieved better performance, as shown in the following table.

[0230]

[0231] It can be seen that the implementation of the multi-scale rotated bounding box regression method for aerial images described in the embodiment of the present invention solves the problem of difficulty in regressing densely arranged objects of different scales in aerial images, and improves the perception ability of multi-scale objects in aerial images and the accuracy of rotated bounding box regression.

[0232] Embodiment 2

[0233] See also Figure 2 , Figure 2 : is a schematic diagram of the structure of a multi-scale rotation bounding box regression device for aerial images disclosed in an embodiment of the present invention. Figure 2 The described device can be applied to an aerial image processing system, such as a local server or a cloud server for spatial area search and positioning processing, and the embodiments of the present invention are not limited thereto. Figure 2 As shown, the device may include:

[0234] A memory 201 storing executable program codes;

[0235] a processor 202 coupled to the memory 201;

[0236] The processor 202 calls the executable program code stored in the memory 201 to execute the steps in the multi-scale rotation bounding box regression method for aerial images described in the first embodiment.

[0237] Embodiment 3

[0238] An embodiment of the present invention discloses a computer-readable storage medium storing a computer program for electronic data exchange, wherein the computer program enables a computer to execute the steps of the multi-scale rotated bounding box regression method for aerial images described in the first embodiment.

[0239] The device embodiments described above are only illustrative, wherein the modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules, i.e., they may be located in one place, or they may be distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Those of ordinary skill in the art may understand and implement it without creative work.

[0240] Through the specific description of the above embodiments, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on such an understanding, the above technical solution can be essentially or partly contributed to the prior art in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, and the storage medium includes a read-only memory (ROM), a random access memory (RAM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a one-time programmable read-only memory (OTPROM), an electronically erasable rewritable read-only memory (EEPROM), a compact disc (CD-ROM) or other optical disc storage, a magnetic disk storage, a magnetic tape storage, or any other computer-readable medium that can be used to carry or store data.

[0241] Finally, it should be noted that the spatial area search and positioning method and device disclosed in the embodiments of the present invention disclose only the preferred embodiments of the present invention, which are only used to illustrate the technical solution of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments can still be modified, or some of the technical features therein can be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multi-scale rotation bounding box regression method for aerial images, characterized in that: The method comprises: S1, obtain the aerial image sample dataset; S2, preprocessing the aerial image sample dataset to obtain an aerial image rotation bounding box regression dataset; S3, based on the aerial image rotation bounding box regression model, processing the aerial image rotation bounding box regression dataset to obtain the aerial image object category and position information; The step of preprocessing the aerial image sample dataset to obtain an aerial image rotation bounding box regression dataset includes: S21, traversing the aerial image sample data set to obtain all aerial image sample data; The aerial image sample data set includes a plurality of aerial image sample data; S22, identifying and cutting any of the aerial image sample data to obtain a target aerial image object set; The target aerial image object set includes a plurality of target aerial image objects; S23, labeling any of the target aerial image objects to obtain label information of the target aerial image object; S24, processing the target aerial image object based on the target aerial image object label information to obtain aerial image rotation bounding box regression data; S25, combining all the aerial image rotation bounding box regression data in order to obtain an aerial image rotation bounding box regression data set; The aerial image rotation bounding box regression data set includes a plurality of aerial image rotation bounding box regression data; The aerial image rotation bounding box regression model comprises: a data input module, a feature extraction module, a classification module, a regression module and a data output module; The output end of the data input module is data-connected to the input end of the feature extraction module; the output end of the feature extraction module is data-connected to the input end of the classification module and the input end of the regression module respectively; the output end of the classification module and the output end of the regression module are both connected to the input end of the data output module; The data input module is used to obtain and send the aerial image sample data set to the feature extraction module; The feature extraction module is used to receive and process the aerial image sample data set, obtain and send the aerial image object feature information to the classification module and the regression module; The classification module is used to receive and process the aerial image object feature information, obtain and send the aerial image object classification information to the data output module; The regression module is used to receive and process the aerial image object feature information, obtain and send the aerial image object rotation bounding box regression information to the data output module; The data output module is used to receive and fuse the aerial image object classification information and the aerial image object rotation bounding box regression information to obtain the aerial image object rotation bounding box regression information; The feature extraction module includes: a first feature extraction submodule, a dense connection submodule, a first fusion submodule and a second fusion submodule; The output end of the first feature extraction submodule is data-connected to the input end of the dense connection submodule, the input end of the first fusion submodule, and the input end of the second fusion submodule; the output end of the dense connection submodule is data-connected to the input end of the first fusion submodule and the input end of the second fusion submodule; The first feature extraction submodule is used to perform feature extraction and upsampling processing on the aerial image sample data set to obtain first feature information, second feature information, third feature information and fourth feature information; The first feature extraction submodule adopts a traditional feature pyramid structure, uses a bottom-up structure to extract feature maps of different scales and outputs the first feature information, the second feature information, the third feature information and the fourth feature information; The dense connection submodule is used to perform upsampling, downsampling, convolution and fusion processing on the fourth feature information to obtain fifth feature information, sixth feature information, seventh feature information, eighth feature information and ninth feature information; The first fusion submodule is used to perform a first fusion process on the first feature information and the fifth feature information, the second feature information and the sixth feature information, the third feature information, the seventh feature information, the eighth feature information and the ninth feature information to obtain aerial image classification information; The second fusion submodule is used to perform a second fusion process on the first feature information, the fifth feature information, the second feature information, the sixth feature information, the third feature information, the seventh feature information, the eighth feature information and the ninth feature information to obtain the aerial image regression information; The densely connected submodule uses a top-down structure to perform bilinear interpolation upsampling processing on the fourth feature information with high resolution to obtain tenth feature information; The dense connection submodule performs bilinear difference upsampling on the tenth feature information to obtain an eleventh feature information; The dense connection submodule performs bilinear difference upsampling on the eleventh feature information to obtain twelfth feature information; Performing element-wise addition of the tenth characteristic information and the first characteristic information to obtain the fifth characteristic information; Performing element-wise addition of the tenth characteristic information and the second characteristic information to obtain the fifth characteristic information; The eleventh characteristic information is added element-wise to the third characteristic information to obtain the sixth characteristic information; Performing element-wise addition of the twelfth characteristic information and the fourth characteristic information to obtain the seventh characteristic information; The dense connection submodule performs a 1x1 convolution on the tenth feature information to obtain the eighth feature information; The dense connection submodule performs 1x1 convolution on the eighth feature information to obtain the ninth feature information; The first fusion processing means extracting a nonlinear representation of learning data through features, and using a softmax activation function to convert the network output into a probability distribution, output the probability of each category, predict the object category, and obtain the aerial image classification information; The second fusion processing means performing feature extraction and nonlinear transformation, obtaining predicted object bounding box coordinates through second loss function regression, and obtaining the aerial image regression information.

2. The method for multi-scale rotation bounding box regression of aerial images according to claim 1, characterized in that: The step of processing the target aerial image object based on the target aerial image object label information to obtain aerial image rotation bounding box regression data includes: S241, segmenting the target aerial image object to obtain a target aerial image object slice information set; The target aerial image object slice information set includes a plurality of target aerial image object slice information; S242, based on the target aerial image object label information, processing any of the target aerial image object slice information to obtain the target aerial image object slice label information; S243, combining the target aerial image object slice label information in order to obtain a target aerial image object label information set; S244, performing screening processing on the target aerial image object label information set to obtain a processed aerial image object label information set; S245, using a data conversion model, converting the processed aerial image object label information set to obtain aerial image rotation bounding box regression data; The aerial image rotation bounding box regression data includes aerial image object rotation bounding box information and aerial image object slice label data information.

3. The method for multi-scale rotation bounding box regression of aerial images according to claim 2, characterized in that: The converting process of the processed aerial image object label information set to obtain the aerial image rotation bounding box regression data includes: S2451, preset minimum bounding rectangle information set; S2452, obtaining the four-point coordinate information of the aerial image object slice and the aerial image object label information of the processed aerial image object label information set; S2453, determining whether the four-point coordinate frame of the four-point coordinate information of the aerial image object slice is a rectangle, and obtaining a shape determination result of the aerial image object slice; When the result of determining the shape of the aerial image object slice is yes, the four-point coordinate information of the aerial image object slice is converted to obtain the rotation bounding box information of the aerial image object, and S2456 is executed; When the result of determining the shape of the aerial image object slice is no, executing S2454; S2454, acquiring the four-point coordinate information of the minimum bounding rectangle information set, and obtaining the four-point coordinate information of the minimum bounding rectangle; S2455, converting the coordinate information of the four points of the minimum circumscribed rectangle to obtain the rotation bounding box information of the aerial image object; S2456: Combine the aerial image object rotation bounding box information and the aerial image object label information in order to obtain aerial image rotation bounding box regression data.

4. The method for multi-scale rotation bounding box regression of aerial images according to claim 1, characterized in that: The aerial image rotation bounding box regression model is based on which the aerial image rotation bounding box regression dataset is processed to obtain the aerial image object category and position information, including: S31, dividing the aerial image rotation bounding box regression dataset to obtain an aerial image regression training set, an aerial image regression verification set, and an aerial image regression test set; S32, using the aerial image regression training set and the aerial image regression verification set, training the aerial image rotation bounding box regression model to obtain an aerial image rotation bounding box regression optimization model; S33, using the aerial image rotation bounding box regression optimization model to process the aerial image regression test set to obtain aerial image object category and position information.

5. The method for multi-scale rotation bounding box regression of aerial images according to claim 4, characterized in that: The method of using the aerial image regression training set and the aerial image regression verification set to train the aerial image rotation bounding box regression model to obtain an aerial image rotation bounding box regression optimization model includes: S321, set the total number of iterations a, and initialize the number of iterations b=0; S322, using the aerial image rotation bounding box regression model, processing the aerial image regression training set to obtain first aerial image regression detection information; The first aerial image regression detection information includes first aerial image object category information and first aerial image object position information; S323, using a first loss function, calculating the first aerial image object category information and the aerial image object label information of the aerial image regression training set to obtain a first difference value; Using a second loss function, calculating the first aerial image object position information and the aerial image object rotation bounding box information of the aerial image regression training set to obtain a second difference value; S324, determining whether the first difference value is less than a first preset threshold, and obtaining a second determination result; Determine whether the second difference value is less than a second preset threshold, and obtain a third determination result; S325, when the second judgment result and the third judgment result are both yes, execute S326; When the second judgment result or the third judgment result is no, increasing the number of iterations by 1; Determine whether the number of iterations is greater than a preset total number of iterations, and obtain a result of determining the number of iterations; When the result of the iteration number judgment is yes, execute S326; When the result of the iteration number judgment is no, executing S322; S326, updating parameters of the aerial image rotation bounding box regression model to obtain an updated aerial image rotation bounding box regression model; S327, using the updated aerial image rotation bounding box regression model, processing the aerial image regression verification set to obtain second aerial image regression detection information; The second aerial image regression detection information includes second aerial image object category information and second aerial image object position information; S328, comparing the second aerial image object category information and the second aerial image object position information with the aerial image object rotation bounding box information and the aerial image object label information of the aerial image regression validation set respectively to obtain prediction accuracy information; S329, determining whether the prediction accuracy information is greater than a preset accuracy threshold, and obtaining an accuracy determination result; When the accuracy determination result is no, executing S321; If the accuracy judgment result is yes, the training process of the aerial image rotation bounding box regression model is completed, and the updated aerial image rotation bounding box regression model is updated to the aerial image rotation bounding box regression optimization model.

6. A multi-scale rotation bounding box regression device for aerial images, characterized in that: The device comprises: A memory storing executable program code; a processor coupled to the memory; The processor calls the executable program code stored in the memory to execute the multi-scale rotation bounding box regression method for aerial images according to any one of claims 1 to 5.

7. A computer-readable storage medium storing computer instructions, which, when called, are used to execute some or all of the steps in the multi-scale rotated bounding box regression method for aerial images according to any one of claims 1 to 5.

Images