A multi-scale morphological object detection method for highly oblique images

Abstract

The present invention discloses a method for multi-scale morphology of target detection in large oblique images, including obtaining a large oblique image to be detected and performing preprocessing; inputting the preprocessed large oblique image into a trained target detection model to obtain a final target detection result; wherein the target detection model includes a first feature extraction structure, a shallow feature extraction structure, a deep feature extraction structure and an additional structure. The method for multi-scale morphology of target detection in large oblique images uses a target detection model to perform target detection on the large oblique image to be detected, and designs an additional structure for the large oblique image, so that the target detection of the large oblique image is more accurate; through a staged training method, it can effectively deal with the difficulties of small amount of large oblique image data, large differences in target scale and morphology.

Description

Technical Field

[0001] The present invention belongs to the technical field of target detection, and in particular relates to a method for detecting targets in multi-scale forms in highly oblique images. Background Art

[0002] Image target detection refers to the use of computer software algorithms to automatically discover the location and category of specific targets in an image. The current mainstream target detection algorithms in the industry are mainly based on the technical route of deep learning. By building a deep convolutional neural network model and training and learning on the basis of massive manually annotated data, it can automatically extract image features and identify specific targets in the image. This type of method has a strong dependence on data. When there are insufficient training data samples, the algorithm performance will drop sharply or even become invalid.

[0003] Multi-scale morphological target detection and training in highly oblique images is to use a high oblique angle to shoot long-distance images in aerial imaging scenarios. Affected by different parameters such as flight altitude, imaging tilt angle, and camera focal length, the imaged targets will show huge differences in scale and morphology, which brings difficulties to target detection. In addition, there are already a large number of datasets for conventional scenes in the existing public datasets, but there are fewer datasets for highly oblique imaging scenes, which further increases the difficulty of training target detection algorithms. Summary of the invention

[0004] The purpose of the present invention is to solve the problems raised in the background technology and to propose a multi-scale morphological target detection method for large oblique images.

[0005] To achieve the above object, the technical solution adopted by the present invention is:

[0006] The present invention proposes a method for detecting objects in multi-scale forms in large oblique images, comprising:

[0007] Obtain the large tilt image to be detected and perform preprocessing;

[0008] Input the preprocessed large tilted image into the trained target detection model to obtain the final target detection result;

[0009] Wherein, the target detection model includes a first feature extraction structure, a shallow feature extraction structure, a deep feature extraction structure and an additional structure;

[0010] The preprocessed large tilted image is used as an input of a first feature extraction structure, and the output of the first feature extraction structure is respectively input into a shallow feature extraction structure and a deep feature extraction structure, and the shallow feature extraction structure outputs a first target detection result, and the preprocessed large tilted image is also used as an input of the additional structure, and the additional structure is connected to the deep feature extraction structure and performs feature fusion to obtain a second target detection result;

[0011] The repeated targets in the first target detection result and the second target detection result are removed to obtain a final target detection result, that is, the target category and target position in the large tilted image to be detected.

[0012] Preferably, the preprocessing includes: first scaling the large tilted image to be detected to a preset size, and then performing a normalization operation.

[0013] Preferably, the first feature extraction structure is a convolutional neural network;

[0014] The shallow feature extraction structure includes A series of shallow convolutional neural network layers connected in sequence;

[0015] The deep feature extraction structure includes The first deep convolutional neural network layers are connected sequentially, and ;

[0016] The additional structure includes Additional feature extraction layers connected in sequence, each of which includes a first zero-initialized convolutional layer, a second deep convolutional neural network layer, and a second zero-initialized convolutional layer connected in sequence from data input to output, and ;

[0017] Should Additional feature extraction layers The first deep convolutional neural network layers correspond one by one. Additional feature extraction layers and front A first deep convolutional neural network layer is formed, the output of the second zero-initialized convolutional layer in each additional feature extraction layer is first fused with the output of the corresponding first deep convolutional neural network layer to obtain a fused feature, the fused feature is used as the input of the next first deep convolutional neural network layer, and the fused feature is second fused with the output of the second zero-initialized convolutional layer in the corresponding additional feature extraction layer to obtain a fused additional feature, and the fused additional feature is used as the input of the next additional feature extraction layer;

[0018] For Additional feature extraction layers and The first deep convolutional neural network layer, The output of the second zero-initialized convolution layer of the additional feature extraction layer is first fused with the output of the corresponding first deep convolutional neural network layer to obtain fused features, and then the fused features are sequentially input to the subsequent A first deep convolutional neural network layer is added to obtain the second target detection result.

[0019] Preferably, the structures of the shallow convolutional neural network layer, the first deep convolutional neural network layer and the second deep convolutional neural network layer are the same, including a first convolutional structure, a second convolutional structure and a third convolutional structure in parallel, and an activation layer, the first convolutional structure is a convolutional layer, and the second convolutional structure and the third convolutional structure are two convolutional layers connected in sequence;

[0020] The input of each convolutional neural network layer passes through the first convolutional structure, the second convolutional structure and the third convolutional structure in parallel. The outputs of the first convolutional structure, the second convolutional structure and the third convolutional structure are spliced ​​and input into the activation layer. Then the input of each convolutional neural network layer is added to the output of the activation layer to obtain the output of each convolutional neural network layer.

[0021] Preferably, each of the zero-initialized convolutional layers includes a convolutional layer, a batch normalization layer and an activation layer which are sequentially connected from data input to output.

[0022] Preferably, ,or .

[0023] Preferably, when When, before The first fusion and the front The second fusion is weighted fusion, and the The first fusion and the subsequent The second fusion is concatenation, and the calculation formula of the weighted fusion is as follows:

[0024] ;

[0025] in, Indicates fusion features or fusion additional features. When fusion features are represented, represents the output of the second zero-initialized convolutional layer, represents the output of the first deep convolutional neural network layer, and represents the weight of the output of the second zero-initialized convolutional layer and the weight of the output of the first deep convolutional neural network layer, respectively. When fused with additional features, represents the output of the second zero-initialized convolutional layer, represents the fusion feature, and They represent the output weight of the second zero-initialized convolutional layer and the weight of the fused features respectively.

[0026] Preferably, when When, before The first fusion and the front The second fusion is weighted fusion.

[0027] Preferably, when training the target detection model, the shallow feature extraction structure is first pre-trained separately, and the loss update parameter is calculated using the first loss function to obtain a pre-trained shallow feature extraction structure;

[0028] The pre-trained shallow feature extraction structure The parameters of the shallow convolutional neural network layers correspond one by one to the parameters of the previous layers in the deep feature extraction structure. Initialization parameters of the first deep convolutional neural network layer are randomly initialized, and the remaining parameters of the first deep convolutional neural network layer in the deep feature extraction structure are randomly initialized, and then the deep feature extraction structure is pre-trained, and the second loss function is used to calculate the loss to update the parameters of the deep feature extraction structure, so as to obtain a pre-trained deep feature extraction structure;

[0029] The parameters of the pre-trained shallow feature extraction structure and the parameters of the pre-trained deep feature extraction structure are used one by one as initialization parameters of the shallow feature extraction structure and the deep feature extraction structure, the shallow feature extraction structure and the deep feature extraction structure are jointly trained, and the average value of the first loss function and the second loss function is taken to simultaneously update the parameters of the shallow feature extraction structure and the deep feature extraction structure, so as to obtain the trained shallow feature extraction structure and the deep feature extraction structure;

[0030] The trained deep feature extraction structure is The parameters of the first deep convolutional neural network layers are used one by one as the initialization parameters of the second deep convolutional neural network layers in the additional structure, and the additional structure is trained. The parameters of the shallow feature extraction structure and the deep feature extraction structure are kept unchanged, and the average value of the first loss function and the second loss function is taken to update the parameters of the additional structure to obtain the trained additional structure.

[0031] Preferably, the formula of the first loss function is as follows:

[0032] ;

[0033] in, and represents the first target category loss The weight and first target position loss The weight of the first target category loss is the cross entropy loss, the first target position loss The calculation formula is as follows:

[0034] ;

[0035] The formula of the second loss function is as follows:

[0036] ;

[0037] in, and represents the second target category loss The weights and the second target position loss The weight of the second target category loss is the cross entropy loss, the second target position loss The calculation formula is as follows:

[0038] ;

[0039] in, and Indicates The center coordinates of the predicted positions, and Indicates The center coordinates of the true value positions, and Sequentially indicates The predicted position width and height, and In order The width and height of the true value position, Represents the number of truth value locations.

[0040] Compared with the prior art, the present invention has the following beneficial effects:

[0041] This method for multi-scale morphology of highly oblique images uses a target detection model to perform target detection on highly oblique images to be detected, and designs an additional structure for highly oblique images to make target detection on highly oblique images more accurate; through a staged training method, it can effectively deal with the difficulties of small amounts of highly oblique image data and large differences in target scale and morphology. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 It is a flow chart of the object detection method of multi-scale morphology of large oblique images of the present invention;

[0043] Figure 2 An embodiment of the additional structure in the target detection model of the present invention;

[0044] Figure 3 Another embodiment of the additional structure in the target detection model of the present invention;

[0045] Figure 4 Schematic diagram of the structure of the convolutional neural network layer of the present invention. DETAILED DESCRIPTION

[0046] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0047] like Figure 1-Figure 4 As shown, a method for multi-scale morphological target detection in a large oblique image is provided, comprising:

[0048] Step 1, obtaining a large tilted image to be detected and performing preprocessing;

[0049] The preprocessing includes: firstly scaling the large tilted image to be detected to a preset size and then performing a normalization operation;

[0050] The input of the target detection model is preset to a size of W×H (W represents the width of the input image, and H represents the height of the input image), and the width and height are both integer multiples of 32. The large oblique image to be detected is scaled to a size of W×H (for example, it is scaled to a size of 640×640 in this embodiment), and then a normalization operation is performed, that is, each pixel value of the large oblique image to be detected is normalized to a value between [0,1].

[0051] Step 2: Input the preprocessed large tilt image into the trained target detection model to obtain the final target detection result;

[0052] Step 2.1, the target detection model includes a first feature extraction structure, a shallow feature extraction structure, a deep feature extraction structure and an additional structure. The preprocessed large oblique image is used as the input of the first feature extraction structure, and the output of the first feature extraction structure is respectively input to the shallow feature extraction structure and the deep feature extraction structure. The shallow feature extraction structure outputs the first target detection result. The preprocessed large oblique image is also used as the input of the additional structure, and the additional structure is connected to the deep feature extraction structure and performs feature fusion to obtain the second target detection result. The repeated targets in the first target detection result and the second target detection result are removed to obtain the final target detection result, that is, the target category and target position in the large oblique image to be detected (add the targets in the first target detection result and the second target detection result, and then remove the same targets. The obtained target is the detection result, and the first target detection result and the second target detection result both contain target category and target position information). The specific description is as follows:

[0053] The first feature extraction structure is a convolutional neural network, such as a ResNet18 network.

[0054] Shallow feature extraction structures include A series of shallow convolutional neural network layers connected in sequence, such as , 6 shallow convolutional neural network layers are connected in series, where the shallow feature extraction structure downsamples the input by 8 times to obtain the first target detection result (which can better cope with the needs of small target detection), that is, the size of the first target detection result is The feature map, that is, in this embodiment, the output size is 80 80.

[0055] The deep feature extraction structure includes The first deep convolutional neural network layers are connected sequentially, and ;like , 12 first deep convolutional neural network layers are connected in series in sequence, where the deep feature extraction structure downsamples the input by 32 times to obtain the second target detection result (with a larger receptive field, which can better cope with the detection needs of large targets and take into account the detection needs of multi-scale targets), that is, the size of the second target detection result is The feature map, that is, in this embodiment, the output size is 20 20.

[0056] Additional structures include Additional feature extraction layers are connected in sequence, each of which includes a first zero-initialized convolutional layer, a second deep convolutional neural network layer, and a second zero-initialized convolutional layer connected in sequence from data input to output, and ; (In this embodiment 11, when the scene is more difficult, the additional structure adopts this structure), or (In this embodiment 6. When the scene is relatively simple, the additional structure adopts this structure).

[0057] Should Additional feature extraction layers The first deep convolutional neural network layers correspond one by one. Additional feature extraction layers and front A first deep convolutional neural network layer is formed, the output of the second zero-initialized convolutional layer in each additional feature extraction layer is first fused with the output of the corresponding first deep convolutional neural network layer to obtain a fused feature, the fused feature is used as the input of the next first deep convolutional neural network layer, and the fused feature is second fused with the output of the second zero-initialized convolutional layer in the corresponding additional feature extraction layer to obtain a fused additional feature, and the fused additional feature is used as the input of the next additional feature extraction layer;

[0058] For Additional feature extraction layers and The first deep convolutional neural network layer, The output of the second zero-initialized convolution layer of the additional feature extraction layer is first fused with the output of the corresponding first deep convolutional neural network layer to obtain a fused feature, and then the fused feature is sequentially passed through the following A first deep convolutional neural network layer is added to obtain the second target detection result.

[0059] It should be noted that the number of additional feature extraction layers in the additional structure is selected based on the complexity of the scene according to past experience. An additional feature extraction layer is used when the scene is relatively simple. An additional feature extraction layer. Figure 2 As shown, when At 11, the 11 additional feature extraction layers correspond to the first 11 first deep convolutional neural network layers one by one in sequence. For the first 10 additional feature extraction layers and the first 10 first deep convolutional neural network layers, the output of the second zero-initialized convolutional layer in each additional feature extraction layer is first fused with the output of the corresponding first deep convolutional neural network layer to obtain a fused feature, which is used as the input of the next first deep convolutional neural network layer. At the same time, the fused feature is second fused with the output of the second zero-initialized convolutional layer in the corresponding additional feature extraction layer to obtain a fused additional feature, and the fused additional feature is used as the input of the next additional feature extraction layer (the fused additional feature corresponding to the 10th additional feature extraction layer is used as the input of the 11th additional feature extraction layer);

[0060] For the 11th additional feature extraction layer and the last two first deep convolutional neural network layers (i.e., the 11th and 12th first deep convolutional neural network layers), the output of the second zero-initialized convolutional layer of the 11th additional feature extraction layer is first fused with the output of the corresponding first deep convolutional neural network layer (i.e., the 11th first deep convolutional neural network layer) to obtain a fused feature, and then the fused feature passes through the last first deep convolutional neural network layer (i.e., the 12th first deep convolutional neural network layer) to obtain the second target detection result.

[0061] like Figure 3 As shown, when 6, the 6 additional feature extraction layers correspond to the first 6 first deep convolutional neural network layers in sequence. For the first 5 additional feature extraction layers and the first 5 first deep convolutional neural network layers, the output of the second zero-initialized convolutional layer in each additional feature extraction layer is first fused with the output of the corresponding first deep convolutional neural network layer to obtain fused features, which are used as the input of the next first deep convolutional neural network layer. At the same time, the fused features are second fused with the output of the second zero-initialized convolutional layer in the corresponding additional feature extraction layer to obtain fused additional features, and the fused additional features are used as the input of the next additional feature extraction layer (the fused additional features corresponding to the 5th additional feature extraction layer are used as the input of the 6th additional feature extraction layer);

[0062] For the 6th additional feature extraction layer and the last 7 first deep convolutional neural network layers, the output of the second zero-initial value convolutional layer of the 6th additional feature extraction layer is first fused with the output of the corresponding first deep convolutional neural network layer (i.e., the 6th first deep convolutional neural network layer) to obtain a fused feature, and then the fused feature passes through the last 6 first deep convolutional neural network layers in sequence to obtain the second target detection result.

[0063] Among them, when When (that is, in this embodiment, 11), the first 6 first fusions and the first 6 second fusions are all weighted fusions, and the fifth first fusion and the fourth second fusion are all splicing;

[0064] when When (that is, in this embodiment, 6), the first 6 first fusions and the first 5 second fusions are all weighted fusions;

[0065] And the calculation formula of weighted fusion is as follows:

[0066] ;

[0067] in, Indicates fusion features or fusion additional features. When it represents the fusion feature (that is, the output of the first fusion), represents the output of the second zero-initialized convolutional layer, represents the output of the first deep convolutional neural network layer, and represents the weight of the output of the second zero-initialized convolutional layer and the weight of the output of the first deep convolutional neural network layer, respectively. When fused with additional features (i.e., the output of the second fusion), represents the output of the second zero-initialized convolutional layer, represents the fusion feature, and represents the output weight of the second zero-initialized convolutional layer and the weight of the fusion feature; , .

[0068] Among them, Figure 4 As shown in the figure, the structures of the shallow convolutional neural network layer, the first deep convolutional neural network layer and the second deep convolutional neural network layer are the same, including the first convolutional structure, the second convolutional structure and the third convolutional structure in parallel, and the activation layer (such as using the GELU activation function), the first convolutional structure is a convolutional layer (using a convolution with a convolution kernel of 1×1), and the second convolutional structure and the third convolutional structure are both two convolutional layers connected in sequence (the second convolutional structure uses convolutions with convolution kernels of 1×1 and 3×3 connected in sequence from the data input to the output direction, and the third convolutional structure uses convolutions with convolution kernels of 1×1 and 7×7 connected in sequence from the data input to the output direction); different sizes of convolution kernels can take into account receptive fields of different sizes, further improving the feature extraction capability;

[0069] The input of each convolutional neural network layer (i.e., the shallow convolutional neural network layer, the first deep convolutional neural network layer, and the second deep convolutional neural network layer) passes through the first convolutional structure, the second convolutional structure, and the third convolutional structure in parallel, and the outputs of the first convolutional structure, the second convolutional structure, and the third convolutional structure are spliced ​​and input into the activation layer, and then the input of each convolutional neural network layer is added to the output of the activation layer to obtain the output of each convolutional neural network layer.

[0070] Among them, each zero-initialized convolution layer includes a convolution layer (using a convolution kernel of 3×3, and the parameter initialization values ​​of the zero-initialized convolution layer are all 0 during the training process), a batch normalization layer and an activation layer (such as using a GELU activation function) connected in sequence from data input to output.

[0071] Step 2.2, build training data set: The training data set includes pre-training data set and tuning data set. The number of samples in the pre-training data set is more than 1000 times that of the tuning data set, which is a large-scale training data set; the tuning data set is a scarce data set with only a small number of samples, such as large oblique image data. For example, the number of samples in the pre-training data set is 500,000 images and the target label corresponding to each image; the tuning data set composed of large oblique images is only 437 images and the target label corresponding to each image.

[0072] Step 2.3: Train the target detection model in stages:

[0073] Step 2.3.1: First, pre-train the shallow feature extraction structure separately, and use the first loss function to calculate the loss update parameter to obtain the pre-trained shallow feature extraction structure, specifically:

[0074] Step 2.3.1.1, randomly initialize the parameters of the shallow feature extraction structure, and the deep feature extraction structure and additional structure do not participate in the training;

[0075] Step 2.3.1.2: Select 20% of the data from the pre-training dataset, i.e. 100,000 samples;

[0076] Step 2.3.1.3. The first loss function is used to calculate the loss value in each training iteration, and the learning rate is set to 0.01. If the loss value of the first loss function is stable at less than the preset threshold or the number of training iterations reaches 100 epochs, the training is stopped to obtain the pre-trained shallow feature extraction structure.

[0077] Among them, the formula of the first loss function is as follows:

[0078] ;

[0079] in, and represents the first target category loss The weight and first target position loss The weight of , , the first target category loss is the cross entropy loss, the first target position loss The calculation formula is as follows:

[0080] .

[0081] Step 2.3.2: Extract the pre-trained shallow feature into the structure The parameters of the shallow convolutional neural network layers correspond one by one to the parameters of the previous layers in the deep feature extraction structure. The initialization parameters of the first deep convolutional neural network layer are randomly initialized, and the parameters of the remaining first deep convolutional neural network layers in the deep feature extraction structure are randomly initialized, and then the deep feature extraction structure is pre-trained (wherein the pre-training process requires the participation of the shallow feature extraction structure, but the parameters of the shallow feature extraction structure are kept unchanged), and the second loss function is used to calculate the loss to update the parameters of the deep feature extraction structure, and the pre-trained deep feature extraction structure is obtained, specifically:

[0082] Step 2.3.2.1, based on the parameters of the pre-trained shallow feature extraction structure, the parameters of the deep feature extraction structure are initialized, wherein the initialization method is that for the network layer with the same shallow feature extraction structure and deep feature extraction structure, the parameters of the pre-trained shallow feature extraction structure are directly used as the initial values ​​of the parameters of the network layer with the same deep feature extraction structure, and the parameters of the remaining network layers in the deep feature extraction structure are randomly initialized;

[0083] In this embodiment, the parameters of the six shallow convolutional neural network layers in the pre-trained shallow feature extraction structure are used one by one as the initialization parameters of the first six first deep convolutional neural network layers in the deep feature extraction structure, and the parameters of the last six first deep convolutional neural network layers in the deep feature extraction structure are randomly initialized;

[0084] Step 2.3.2.2, select 60% of the data from the pre-training data set for parameter training of the deep feature extraction structure;

[0085] Step 2.3.2.3, freeze the parameter value of the shallow feature extraction structure, input the pre-training data to the shallow feature extraction structure and the deep feature extraction structure for forward reasoning, obtain the target detection result, and use the second loss function to calculate the loss value;

[0086] Step 2.3.2.4, according to the loss value of the second loss function, using the gradient descent method to update the network parameters of the deep feature extraction structure;

[0087] Step 2.3.2.5. Repeat steps 2.3.2.2 to 2.3.2.4, set the learning rate to 0.01, and stop training when the number of training iterations reaches 300 epochs to obtain the pre-trained deep feature extraction structure.

[0088] Among them: The formula of the second loss function is as follows:

[0089] ;

[0090] in, and represents the second target category loss The weights and the second target position loss The weight of , , the second target category loss is the cross entropy loss, the second target position loss The calculation formula is as follows:

[0091] ;

[0092] in, and Indicates The center coordinates of the predicted positions, and Indicates The center coordinates of the true value positions, and Sequentially indicates The predicted position width and height, and In order The width and height of the true value position, Represents the number of truth value locations.

[0093] Step 2.3.3, the parameters of the pre-trained shallow feature extraction structure and the parameters of the pre-trained deep feature extraction structure are used as the initialization parameters of the shallow feature extraction structure and the deep feature extraction structure in sequence, and the shallow feature extraction structure and the deep feature extraction structure are jointly trained, and the average value of the first loss function and the second loss function is taken to simultaneously update the parameters of the shallow feature extraction structure and the deep feature extraction structure, so as to obtain the trained shallow feature extraction structure and the deep feature extraction structure, specifically:

[0094] Step 2.3.3.1, the parameters of the pre-trained shallow feature extraction structure and deep feature extraction structure obtained in step 2.3.1 and step 2.3.2 are used as initialization parameters of the shallow feature extraction structure and the deep feature extraction structure in one-to-one correspondence;

[0095] Step 2.3.3.2, the pre-training dataset is used for joint training;

[0096] Step 2.3.3.3, use the first loss function to calculate the loss value of the shallow feature extraction structure, input the training data for forward reasoning, and use the second loss function to calculate the loss value of the deep feature extraction structure;

[0097] Step 2.3.3.4, take the average of the first loss function and the second loss function, set the learning rate to 0.1 times the learning rate of step 2.3.1 and step 2.3.2, that is, 0.001, and use the gradient descent method to simultaneously update the parameters of the shallow feature extraction structure and the deep feature extraction structure;

[0098] Step 2.3.3.5. Repeat steps 2.3.3.2 to 2.3.3.4. When the number of training iterations reaches 300 epochs, stop training and obtain the trained shallow feature extraction structure and deep feature extraction structure.

[0099] Step 2.3.4: Extract the trained deep feature structure The parameters of the first deep convolutional neural network layers are used as the initialization parameters of the second deep convolutional neural network layers in the additional structure one by one, and the additional structure is trained. The parameters of the shallow feature extraction structure and the deep feature extraction structure are kept unchanged, and the average value of the first loss function and the second loss function is taken to update the parameters of the additional structure to obtain the trained additional structure, which is specifically:

[0100] Step 2.3.4.1, initializing the parameters of the second deep convolutional neural network layer of the additional structure with the parameters of the deep feature extraction structure trained in step 2.3.3 (in this embodiment, when =11, then the parameters of the first 11 first deep convolutional neural network layers in the trained deep feature extraction structure are used one by one as the initialization parameters of each second deep convolutional neural network layer in the additional structure; when =6, then the parameters of the first 6 first deep convolutional neural network layers in the trained deep feature extraction structure are used one by one as the initialization parameters of each second deep convolutional neural network layer in the additional structure), and the initialization parameters of each zero-initialized convolutional layer are all set to 0;

[0101] Step 2.3.4.2, use the tuned dataset for parameter training of the additional structure;

[0102] Step 2.3.4.3, use the first loss function to calculate the loss value of the shallow feature extraction structure, input the training data for the previous inference, and use the second loss function to calculate the loss value of the deep feature extraction structure;

[0103] Step 2.3.4.4, take the average of the first loss function and the second loss function, set the learning rate to 0.5 times the learning rate of step 2.3.3, that is, 0.0005, freeze the network parameters of the shallow feature extraction structure and the deep feature extraction structure, do not participate in the update, and use the gradient descent method to update the network parameters of the additional structure;

[0104] Step 2.3.4.5. Repeat steps 2.3.4.2 to 2.3.4.4. When the number of training iterations reaches 300 epochs, stop training and obtain the trained additional structure.

[0105] This method for multi-scale morphology of highly oblique images uses a target detection model to perform target detection on highly oblique images to be detected, and designs an additional structure for highly oblique images to make target detection on highly oblique images more accurate; through a staged training method, it can effectively deal with the difficulties of small amounts of highly oblique image data and large differences in target scale and morphology.

[0106] Unless otherwise specified herein, there is no strict order restriction on the execution of the steps in the present application, and the steps may be executed in other orders.

[0107] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A method for multi-scale morphological object detection in large oblique images, characterized by: The multi-scale morphological target detection method of the large oblique image comprises: Obtain the large tilt image to be detected and perform preprocessing; Input the preprocessed large tilted image into the trained target detection model to obtain the final target detection result; Wherein, the target detection model includes a first feature extraction structure, a shallow feature extraction structure, a deep feature extraction structure and an additional structure; The preprocessed large tilted image is used as an input of a first feature extraction structure, and the output of the first feature extraction structure is respectively input into a shallow feature extraction structure and a deep feature extraction structure, and the shallow feature extraction structure outputs a first target detection result, and the preprocessed large tilted image is also used as an input of the additional structure, and the additional structure is connected to the deep feature extraction structure and performs feature fusion to obtain a second target detection result; Removing duplicate targets in the first target detection result and the second target detection result to obtain a final target detection result, that is, a target category and a target position in the large oblique image to be detected; Wherein, the first feature extraction structure is a convolutional neural network; The shallow feature extraction structure includes A series of shallow convolutional neural network layers connected in sequence; The deep feature extraction structure includes The first deep convolutional neural network layers are connected sequentially, and ; The additional structure includes Additional feature extraction layers connected in sequence, each of which includes a first zero-initialized convolutional layer, a second deep convolutional neural network layer, and a second zero-initialized convolutional layer connected in sequence from data input to output, and ; Should Additional feature extraction layers The first deep convolutional neural network layers correspond one by one. Additional feature extraction layers and front A first deep convolutional neural network layer is formed, the output of the second zero-initialized convolutional layer in each additional feature extraction layer is first fused with the output of the corresponding first deep convolutional neural network layer to obtain a fused feature, the fused feature is used as the input of the next first deep convolutional neural network layer, and the fused feature is second fused with the output of the second zero-initialized convolutional layer in the corresponding additional feature extraction layer to obtain a fused additional feature, and the fused additional feature is used as the input of the next additional feature extraction layer; For Additional feature extraction layers and The first deep convolutional neural network layer, The output of the second zero-initialized convolutional layer of the additional feature extraction layer is first fused with the output of the corresponding first deep convolutional neural network layer to obtain fused features, and then the fused features are sequentially input to the subsequent A first deep convolutional neural network layer is added to obtain the second target detection result.

2. The method for multi-scale morphological object detection in large oblique images according to claim 1, characterized in that: The preprocessing includes: firstly scaling the large tilted image to be detected to a preset size, and then performing a normalization operation.

3. The method for multi-scale morphological object detection in large oblique images according to claim 1, characterized in that: The structures of the shallow convolutional neural network layer, the first deep convolutional neural network layer and the second deep convolutional neural network layer are the same, including a first convolutional structure, a second convolutional structure and a third convolutional structure in parallel, and an activation layer, the first convolutional structure is a convolutional layer, and the second convolutional structure and the third convolutional structure are two convolutional layers connected in sequence; The input of each convolutional neural network layer passes through the first convolutional structure, the second convolutional structure and the third convolutional structure in parallel. The outputs of the first convolutional structure, the second convolutional structure and the third convolutional structure are spliced ​​and input into the activation layer. Then the input of each convolutional neural network layer is added to the output of the activation layer to obtain the output of each convolutional neural network layer.

4. The method for multi-scale morphological object detection in large oblique images according to claim 1, characterized in that: Each of the zero-initialized convolutional layers includes a convolutional layer, a batch normalization layer, and an activation layer that are sequentially connected from data input to output.

5. The method for multi-scale morphological object detection in large oblique images according to claim 1, characterized in that: ,or .

6. The method for multi-scale morphological object detection in large oblique images according to claim 5, characterized in that: when When, before The first fusion and the front The second fusion is weighted fusion, and the The first fusion and the subsequent The second fusion is concatenation, and the calculation formula of the weighted fusion is as follows: ； in, Indicates fusion features or fusion additional features. When fusion features are represented, represents the output of the second zero-initialized convolutional layer, represents the output of the first deep convolutional neural network layer, and represents the weight of the output of the second zero-initialized convolutional layer and the weight of the output of the first deep convolutional neural network layer, respectively. When fused with additional features, represents the output of the second zero-initialized convolutional layer, represents the fusion feature, and They represent the output weight of the second zero-initialized convolutional layer and the weight of the fused features respectively.

7. The method for multi-scale morphological object detection in large oblique images according to claim 6, characterized in that: when When, before The first fusion and the front The second fusion is weighted fusion.

8. The method for multi-scale morphological object detection in large oblique images according to claim 1, characterized in that: When training the target detection model, the shallow feature extraction structure is first pre-trained separately, and the first loss function is used to calculate the loss update parameter to obtain the pre-trained shallow feature extraction structure; The pre-trained shallow feature extraction structure The parameters of the shallow convolutional neural network layers correspond one by one to the parameters of the previous layers in the deep feature extraction structure. Initialization parameters of the first deep convolutional neural network layer are randomly initialized, and the remaining parameters of the first deep convolutional neural network layer in the deep feature extraction structure are randomly initialized, and then the deep feature extraction structure is pre-trained, and the second loss function is used to calculate the loss to update the parameters of the deep feature extraction structure, so as to obtain a pre-trained deep feature extraction structure; The parameters of the pre-trained shallow feature extraction structure and the parameters of the pre-trained deep feature extraction structure are used one by one as initialization parameters of the shallow feature extraction structure and the deep feature extraction structure, the shallow feature extraction structure and the deep feature extraction structure are jointly trained, and the average value of the first loss function and the second loss function is taken to simultaneously update the parameters of the shallow feature extraction structure and the deep feature extraction structure, so as to obtain the trained shallow feature extraction structure and the deep feature extraction structure; The trained deep feature extraction structure is The parameters of the first deep convolutional neural network layers are used one by one as the initialization parameters of the second deep convolutional neural network layers in the additional structure, and the additional structure is trained. The parameters of the shallow feature extraction structure and the deep feature extraction structure are kept unchanged, and the average value of the first loss function and the second loss function is taken to update the parameters of the additional structure to obtain the trained additional structure.

9. The method for multi-scale morphological object detection in large oblique images according to claim 8, characterized in that: The formula of the first loss function is as follows: ； in, and Denotes the first target category loss The weight and first target position loss The weight of the first target category loss is the cross entropy loss, the first target position loss The calculation formula is as follows: ； The formula of the second loss function is as follows: ； in, and represents the second target category loss The weights and the second target position loss The weight of the second target category loss is the cross entropy loss, the second target position loss The calculation formula is as follows: ； in, and Indicates The center coordinates of the predicted positions, and Indicates The center coordinates of the true value positions, and Sequentially indicates The predicted position width and height, and In order The width and height of the true value position, Represents the number of truth value locations.

Images