Modeling method of single-stage target detection model for high-resolution remote sensing image
By adopting a single-stage object detection model in high-score remote sensing images, combining backbone network, feature pyramid and separation detection head, the problem of low target detection accuracy in high-score satellite remote sensing images is solved, and efficient detection of small targets is achieved.
Patent Information
- Application Number
- CN202411345545.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-25
- Publication Date
- 2025-02-07
AI Technical Summary
In high-score remote sensing images, the target size is small, the direction is diverse, and the influence of shadows and occlusions, resulting in poor detection performance of weak targets.
Using a single-stage object detection model, the image is divided into multiple sub-images by preprocessing the remote sensing image of high-score satellites, and a target detection model is established, including backbone network, feature pyramid and separation detection head, and shallow features are introduced to improve detection. Accuracy.
The accuracy of target detection in high-score satellite remote sensing images has been improved, and it is suitable for disaster assessment, urban planning and environmental monitoring.
Smart Images
Figure CN119399643A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of remote sensing target detection, and in particular to a modeling method of a single-stage target detection model for high-resolution remote sensing images. Background Art
[0002] Remote sensing image target detection refers to the identification and location of ground targets of interest in remote sensing images, such as aircraft, ships, cars, etc. Remote sensing image target detection is widely used in civil and military fields, such as disaster assessment, urban planning, environmental monitoring, etc. With the rapid development of remote sensing technology, the spatial resolution of high-resolution satellite remote sensing images has been greatly improved, and the total amount of data has increased rapidly, providing rich data for target detection. At the same time, with the breakthrough progress of deep learning in the field of computer vision, deep neural networks have become the mainstream of target detection tasks. Neural network models based on convolutional neural networks can automatically extract image features and perform efficient and accurate target recognition and positioning.
[0003] Object detection technology based on convolutional neural networks has developed rapidly and has achieved performance that exceeds traditional computer vision methods. Currently, object detection technology based on convolutional neural networks is divided into two mainstream frameworks: one is a two-stage detection algorithm represented by the FasterR-CNN series and FPN, and the other is a single-stage object detection algorithm represented by the YOLO series. The two-stage detection algorithm first generates candidate target regions and then converts the detection problem into a classification and bounding box regression problem. The single-stage detection algorithm does not generate candidate target regions but directly estimates the location and category of the target. Therefore, the two-stage detector model is often more accurate, while the single-stage detector has higher real-time operation efficiency.
[0004] Satellite remote sensing images have a large coverage area and are less affected by ground factors, which makes them advantageous in disaster assessment, environmental monitoring and other fields. However, the ground objects of interest in satellite remote sensing images are small in size and easily affected by shadows, occlusions, etc. They lack obvious texture and boundary features, which poses a challenge to the detection performance of weak targets. The ground objects in satellite remote sensing images have directional diversity, and the aspect ratio of the bounding box of the same type of ground object varies greatly.
[0005] Considering the spatial resolution of satellite remote sensing images, small targets with small pixel sizes (usually less than 32×32 pixels) are widely distributed in satellite remote sensing images. Existing target detection models usually use multi-scale representation and feature pyramid (FPN) technologies to solve the problem of small target detection by enhancing multi-scale target detection capabilities, but lack detection technologies that take small targets as the main optimization target. Summary of the invention
[0006] The main purpose of the present invention is to provide a modeling method of a single-stage target detection model for high-resolution remote sensing images, which is used to establish a single-stage target detection model.
[0007] To achieve the above objectives, the technical solution adopted by the present invention is: a modeling method for a single-stage target detection model for high-resolution remote sensing images, which specifically includes the following steps:
[0008] Step 1: preprocess the original images of the pre-collected high-resolution satellite remote sensing image data set, divide each high-resolution satellite remote sensing image into multiple sub-images of predetermined sizes, obtain the location information of the sub-images and the category and location information of the targets in the sub-images, and name the sub-images;
[0009] Step 2: Generate a labeling file with the size of the sub-image, the location information of the sub-image, the category and location information of the target in the sub-image, and uniformly process the labeling file into the VOC format, and divide the sub-image into a training set, a validation set, and a test set according to a certain ratio;
[0010] Step 3: Establish target detection model;
[0011] Step 4: Input the training set obtained in step 2 into the target detection model for training, input the validation set into the trained target detection model for verification, and input the test set into the verified target detection model for testing.
[0012] Preferably, the sub-image is obtained as follows:
[0013] The pixels of the sub-image are preset to be m*n;
[0014] If the number of pixels in the length direction of the original image is not an integer multiple of m, then expand it on one side of the length direction until the number of pixels in the length direction of the original image is an integer multiple of m; if the number of pixels in the width direction of the original image is not an integer multiple of n, then expand it on one side of the width direction until the number of pixels in the length direction of the original image is an integer multiple of n, and fill the expanded area with pixels with a gray value of 0 as filling content to distinguish the original image from the expanded image;
[0015] The original image is divided into multiple sub-images of predetermined sizes.
[0016] Preferably, sub-images are named as follows:
[0017] When the satellite remote sensing image is divided into sub-images, the horizontal coordinate offset and the vertical coordinate offset of the sub-image in the satellite remote sensing image are recorded;
[0018] The sub-image is named ““original file name”_horizontal axis offset_vertical axis offset.jpg”, where the original file name is the file name of the corresponding satellite remote sensing image.
[0019] Preferably, the location information of the target in the sub-image is obtained by:
[0020] A rectangular bounding box is determined for each target on the sub-image. The target is located within the bounding box, and each side of the bounding box is tangent to at least one point on the edge of the target. The position of the bounding box on the sub-image is represented by the coordinates of the upper left corner vertex (x min ,y min ) and the coordinates of the lower right vertex (x max ,y max )express.
[0021] Preferably, step 2 comprises the following steps:
[0022] Step 21: Generate an XML file with the same file name as each sub-image. The XML file is a label file. The XML file records the size of the corresponding sub-image and the information of all objects in the sub-image. The object information includes the category of the object and the coordinates of the corresponding bounding box. Specifically, the object information is in the form of a file named <object>The linked list exists in the XML format file. <name>The element represents the category name of the target. <x min >、 <y min >、 <x max > and <y max > represents the coordinates of the upper left corner and lower right corner of the target bounding box.
[0023] Step 22: putting the obtained XML format file into a second predetermined folder;
[0024] Step 23, randomly select the obtained sub-images in a ratio of 7:2:1 and divide them into a training set, a validation set and a test set, and record the file names (without suffix) of the images in the training set, the validation set and the test set in the train.txt, val.txt and test.txt files respectively, and save these three txt files in a third predetermined folder.
[0025] Preferably, the target detection model in step three includes a backbone network, a feature pyramid, and a separation detection head. The backbone network has 5 CSP Darknet convolutional layers with the same structure, namely Dark 1 to Dark 5. Each CSP Darknet convolutional layer includes two sets of convolution kernels and a skip layer connection operation, wherein the parameters of the convolution kernel are obtained through training and learning. The image input to the backbone network passes through 5 convolutional layers step by step. Each time it passes through a CSP Darknet convolutional layer, the size of the image is reduced to a certain proportion of the original, while the number of channels remains unchanged, and feature maps of different levels are obtained.
[0026] Preferably, the feature pyramid has six Neck layers, three layers in a group, the first group is Neck1_1, Neck1_2, Neck1_3, the second group is Neck2_1, Neck2_2, Neck2_3, Neck1_1 is used to fuse the feature maps output by Dark5 and Dark4, Neck1_2 is used to fuse the feature map output by Neck1_1 and the feature map output by Dark3 magnified by a certain multiple, Neck1_3 is used to fuse the feature map output by Neck1_2 and the feature map output by Dark2 magnified by a certain multiple; Neck2_1 is used to fuse the feature map output by Neck1_2 and the feature map output by Neck1_3 reduced by a certain multiple, Neck2_2 is used to fuse the feature map output by Neck1_1 and the feature map output by Neck2_1 reduced by a certain multiple, and Neck2_3 is used to reduce the feature map output by Neck2_2 by a certain multiple.
[0027] Preferably, there are four separation detection heads, respectively denoted as DecoupledHead1, DecoupledHead2, Decoupled Head3, and Decoupled Head4. Each separation detection head includes two branches, which respectively predict the category and bounding box coordinates of the target. Decoupled Head1 is used to perform convolution layer processing on the feature map output by Neck1_3, DecoupledHead2 is used to perform convolution layer processing on the feature map output by Neck2_1, Decoupled Head3 is used to perform convolution layer processing on the feature map output by Neck2_2, and DecoupledHead4 is used to perform convolution layer processing on the feature map output by Neck2_3.
[0028] Compared with the prior art, the present invention has the following beneficial effects:
[0029] Compared with the existing methods, the present invention fully considers the characteristics that targets in high-resolution satellite remote sensing images are generally small and targets of the same category have similar scales, improves the YOLOX deep target detection model, and introduces shallow features into the feature fusion process, thereby achieving improved target detection accuracy in high-resolution satellite remote sensing images. The method can be applied to disaster assessment, urban planning, environmental monitoring, etc. BRIEF DESCRIPTION OF THE DRAWINGS
[0030] Figure 1 It is the main flow chart of the present invention;
[0031] Figure 2 It is a schematic diagram of the target detection model of the present invention. DETAILED DESCRIPTION
[0032] The following description is used to disclose the present invention so that those skilled in the art can implement the present invention. The preferred embodiments described below are only examples, and those skilled in the art may think of other obvious variations.
[0033] A modeling method for a single-stage target detection model for high-resolution remote sensing images specifically comprises the following steps:
[0034] Step 1: Perform raw image preprocessing on the pre-collected high-resolution satellite remote sensing image dataset, divide each high-resolution satellite remote sensing image into multiple sub-images of predetermined size, obtain the location information of the sub-image and the category and location information of the target in the sub-image, and name the sub-image and save it in the first predetermined folder. The sub-image naming method is as follows:
[0035] When a satellite remote sensing image is divided into sub-images, the horizontal coordinate offset and the vertical coordinate offset of the sub-image in the satellite remote sensing image are recorded, and the sub-image is named ""original file name"_horizontal coordinate offset_vertical coordinate offset.jpg", where the original file name is the corresponding satellite remote sensing image.
[0036] The location information of the target in the sub-image is obtained in the following way:
[0037] A rectangular bounding box is determined for each target on the sub-image. The length and width of the bounding box are parallel to the length and width of the corresponding sub-image. The target is located in the bounding box, and each side of the bounding box is tangent to at least one point on the edge of the target. The position of the bounding box on the sub-image adopts the coordinates of the upper left corner vertex (x min ,y min ) and the coordinates of the lower right vertex (x max ,y max )express.
[0038] Step 2: Generate a labeling file and uniformly process the labeling file into the VOC format, as follows:
[0039] Step 21: Generate an XML file with the same file name as each sub-image. The XML file is a label file. The XML file records the size of the corresponding sub-image and the information of all objects in the sub-image. The object information includes the category of the object and the coordinates of the corresponding bounding box. Specifically, the object information is in the form of a file named <object>The linked list exists in the XML format file. <name>The element represents the category name of the target. <x min >、 <y min >、 <x max > and <y max >Represents the coordinates of the upper left corner and lower right corner of the target's bounding box.
[0040] The coordinates of the target are obtained in the following manner: reading the horizontal coordinate offset and the vertical coordinate offset of the sub-image relative to the satellite remote sensing image in the file name, and transforming the coordinates of the bounding box of the target in the sub-image into the real coordinates corresponding to the original image according to the horizontal coordinate offset and the vertical coordinate offset;
[0041] Step 22: putting the obtained XML format file into a second predetermined folder;
[0042] Step 23, randomly select the obtained sub-images according to the ratio of 7:2:1 and divide them into a training set, a validation set and a test set, and record the file names (without suffix) of the images in the training set, the validation set and the test set in train.txt, val.txt and test.txt files respectively, and save these three txt files in a third predetermined folder;
[0043] Step 3: Establish target detection model;
[0044] Step 4: Input the training set obtained in step 2 into the target detection model in step 3 for training. After the training is completed, the trained target detection model is verified using the verification set, and after verification, the verified target detection model is tested using the test set. During training, the output results of the model can be compared with the XML format file of the corresponding sub-image, and the parameters in the model can be adjusted according to the comparison results.
[0045] The processed data set is input into the target detection model for training, and a usable target detection model is obtained after the training is completed.
[0046] The pre-collected high-resolution satellite remote sensing image dataset in step 1 may adopt an existing publicly available high-resolution satellite remote sensing image target detection dataset xView.
[0047] In step 1, the pixels of the sub-image are preset to be m*n. If the pixels in the length direction of the original image are not an integer multiple of m, then the original image is expanded on one side in the length direction until the pixels in the length direction of the original image are an integer multiple of m. Similarly, if the pixels in the width direction of the original image are not an integer multiple of n, then the original image is expanded on one side in the width direction until the pixels in the length direction of the original image are an integer multiple of n. The pixels with a gray value of 0 are filled in the expanded area as filling content to distinguish the original image from the expanded image, and then the original image is cut into multiple sub-images of predetermined sizes.
[0048] For step three, for step three, a target detection model based on the YOLOX target detection model is established. The target detection model includes a backbone network, a feature pyramid, and a separation detection head. The backbone network has 5 CSP Darknet convolutional layers (denoted as Dark 1 to Dark 5) with the same structure. Each CSPDarknet convolutional layer includes two sets of convolution kernels and a skip layer connection operation, wherein the parameters of the convolution kernel are the learnable parameters of the neural network used in the present invention. The image input to the backbone network passes through 5 convolutional layers step by step. Each time a layer of CSPDarknet convolutional layer is passed, the size of the image is reduced to half of the original size, while the number of channels remains unchanged at 80, thereby obtaining feature maps of different levels. The images that pass through a total of 5 convolutional layers of Dark 1 to 5 are named P1 to P5 feature maps respectively. If the size of the input image is a 3-channel square image with a side length of 640 pixels, the size of the P1 feature map is 320×320, the number of channels is 80 (its size is recorded as 320×320×80), the size of the P2 feature map is 160×160×80, and so on, the sizes of the P3, P4 and P5 feature maps are 80, 40 and 20 respectively, and the number of channels remains unchanged at 80. Among them, the P1 and P2 feature maps are shallow feature maps, mainly containing local detail features such as texture and edges of the image, while the P4 and P5 feature maps are deep features containing more scene and global features, and the feature information contained in the P3 feature map is mainly the appearance features of medium-sized targets.
[0049] The feature pyramid has six Neck layers, three layers in a group, the first group is Neck1_1, Neck1_2, Neck1_3, the second group is Neck2_1, Neck2_2, Neck2_3, Neck1_1 is used to fuse the feature maps output by Dark5 and Dark4, Neck1_2 is used to fuse the feature map output by Neck1_1 and the feature map output by Dark3 magnified by a certain multiple, Neck1_3 is used to fuse the feature map output by Neck1_2 and the feature map output by Dark2 magnified by a certain multiple. Neck2_1 is used to fuse the feature map output by Neck1_2 and the feature map output by Neck1_3 reduced by a certain multiple, Neck2_2 is used to fuse the feature map output by Neck1_1 and the feature map output by Neck2_1 reduced by a certain multiple, and Neck2_3 is used to reduce the feature map output by Neck2_2 by a certain multiple. Simply put, the upward and downward arrows in the feature pyramid module represent upsampling (the feature map is enlarged to twice the original size) and downsampling (the feature map is reduced to one-half). Through scale scaling and splicing operations, the feature maps of adjacent levels are fused to improve the multi-scale feature expression ability of the neural network. Taking Neck1_1 in the figure as an example, it accepts feature maps of two levels, P4 and P5. The size of the P4 feature map is 40×40×80, and the original size of the P5 feature map is 20×20×80. The P5 feature map is enlarged to twice its original size, that is, 40×40×80, through interpolation, so as to keep consistent with the size of the P4 feature map, and then splicing is performed at Neck1_1 to obtain a feature map of size 40×40×160.
[0050] There are four separation detection heads, which are respectively denoted as Decoupled Head1, Decoupled Head2, DecoupledHead3, and DecoupledHead4. Each separation detection head includes two branches, which respectively predict the category and bounding box coordinates of the target. DecoupledHead1 is used to perform convolution layer processing on the feature map output by Neck1_3, DecoupledHead2 is used to perform convolution layer processing on the feature map output by Neck2_1, DecoupledHead3 is used to perform convolution layer processing on the feature map output by Neck2_2, and DecoupledHead4 is used to perform convolution layer processing on the feature map output by Neck2_3. After the convolution layer processing, the feature vector is obtained, including a 1-dimensional vector for predicting the target category, a 4-dimensional vector for predicting the target bounding box coordinates, and a 1-dimensional vector for representing the confidence of the current predicted bounding box. There are 3 separation detection heads in the YOLOX neural network model, corresponding to the P3, P4, and P5 feature maps, respectively. After the above steps, the neural network obtains the outputs for predicting the category, position, and confidence of the target in the graphic according to the input image, which are respectively denoted as X cls , X box and X obj .
[0051] When executing step 3, after completing one training, the loss function is used to determine whether the training is completed. The loss function is set to:
[0052] L d =L cls +L obj +L reg ,
[0053] Among them, L d is the overall target detection loss function, L cls represents the classification loss, L obj represents the prediction loss, L reg Represents the regression loss, which is as follows:
[0054] L cls =CrossEntropy(X cls ,GT cls ),
[0055] L obj =CrossEntropy(X obj ,GT obj ),
[0056] L reg =CrossEntropy(X box ,GT box ),
[0057] CrossEntropy(p,q)=-[plog(q)+(1-p)log(1-q)],
[0058] Where CrossEntropy(·) represents the cross entropy loss function, X cls , X obj , X box Represent the target category, confidence and location predicted by the model, GT cls represents the true category of the target, L cls represents the cross entropy loss between the target category predicted by the model and the true target category, L obj A Boolean value GT indicating the model's confidence in the prediction and whether the target actually exists obj (0 or 1) cross entropy loss, L reg It represents the regression accuracy loss of the model, X box and GT box Represent the bounding box coordinates of the model prediction and the actual target respectively.
[0059] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The above embodiments and descriptions only describe the principles of the present invention. The present invention may be subject to various changes and improvements without departing from the spirit and scope of the present invention. These changes and improvements fall within the scope of the present invention. The scope of protection claimed by the present invention is defined by the attached claims and their equivalents.< / name> < / object> < / name> < / object>
Claims
1. A modeling method for a single-stage target detection model for high-resolution remote sensing images, comprising the following steps: Step 1: preprocess the original images of the pre-collected high-resolution satellite remote sensing image data set, divide each high-resolution satellite remote sensing image into multiple sub-images of predetermined sizes, obtain the location information of the sub-images and the category and location information of the targets in the sub-images, and name the sub-images; Step 2: Generate a labeling file with the size of the sub-image, the location information of the sub-image, the category and location information of the target in the sub-image, and uniformly process the labeling file into the VOC format, and divide the sub-image into a training set, a validation set, and a test set according to a certain ratio; Step 3: Establish target detection model; Step 4: Input the training set obtained in step 2 into the target detection model for training, input the validation set into the trained target detection model for verification, and input the test set into the verified target detection model for testing.
2. The modeling method according to claim 1, characterized in that: The sub-image is obtained as follows: The pixels of the sub-image are preset to be m*n; If the number of pixels in the length direction of the original image is not an integer multiple of m, then expand it on one side of the length direction until the number of pixels in the length direction of the original image is an integer multiple of m; if the number of pixels in the width direction of the original image is not an integer multiple of n, then expand it on one side of the width direction until the number of pixels in the length direction of the original image is an integer multiple of n, and fill the expanded area with pixels with a gray value of 0 as filling content to distinguish the original image from the expanded image; The original image is divided into multiple sub-images of predetermined sizes.
3. The modeling method according to claim 1, characterized in that: Sub-images are named as follows: When the satellite remote sensing image is divided into sub-images, the horizontal coordinate offset and the vertical coordinate offset of the sub-image in the satellite remote sensing image are recorded; The sub-image is named ""original file name"_horizontal axis offset_vertical axis offset.jpg", where the original file name is the file name of the corresponding satellite remote sensing image.
4. The modeling method according to claim 3, characterized in that: The location information of the target in the sub-image is obtained in the following way: A rectangular bounding box is determined for each target on the sub-image. The target is located within the bounding box, and each side of the bounding box is tangent to at least one point on the edge of the target. The position of the bounding box on the sub-image is represented by the coordinates of the upper left corner vertex (x min ,y min ) and the coordinates of the lower right corner (x max ,y max )express.
5. The modeling method according to claim 4, characterized in that: Step 2 includes the following steps: Step 21: Generate an XML file with the same file name as each sub-image. The XML file is a label file. The XML file records the size of the corresponding sub-image and the information of all objects in the sub-image. The object information includes the category of the object and the coordinates of the corresponding bounding box. Specifically, the object information is in the form of a file named <object>The linked list exists in the XML format file. <name>The element represents the category name of the target. <x min >、 <y min >、 <x max > and <y max > represents the coordinates of the upper left corner and lower right corner of the target bounding box. Step 22: putting the obtained XML format file into a second predetermined folder; Step 23, randomly select the obtained sub-images in a ratio of 7:2:1 and divide them into a training set, a validation set and a test set, and record the file names (without suffix) of the images in the training set, the validation set and the test set in the train.txt, val.txt and test.txt files respectively, and save these three txt files in a third predetermined folder.
6. The modeling method according to claim 1, characterized in that: The target detection model in step 3 includes a backbone network, a feature pyramid, and a separation detection head. The backbone network has five CSP Darknet convolutional layers with the same structure, namely Dark 1 to Dark 5. Each CSP Darknet convolutional layer includes two sets of convolutional kernels and a skip-layer connection operation. The parameters of the convolutional kernel are obtained through training and learning. The image input to the backbone network passes through five convolutional layers step by step. Each time it passes through a CSP Darknet convolutional layer, the size of the image is reduced to a certain proportion of the original, while the number of channels remains unchanged, and feature maps of different levels are obtained.
7. The modeling method according to claim 6, characterized in that: The feature pyramid has six Neck layers, three layers in a group, the first group is Neck1_1, Neck1_2, Neck1_3, the second group is Neck2_1, Neck2_2, Neck2_3, Neck1_1 is used to fuse the feature maps output by Dark5 and Dark4, Neck1_2 is used to fuse the feature map output by Neck1_1 and the feature map output by Dark3 magnified by a certain multiple, Neck1_3 is used to fuse the feature map output by Neck1_2 and the feature map output by Dark2 magnified by a certain multiple; Neck2_1 is used to fuse the feature map output by Neck1_2 and the feature map output by Neck1_3 reduced by a certain multiple, Neck2_2 is used to fuse the feature map output by Neck1_1 and the feature map output by Neck2_1 reduced by a certain multiple, and Neck2_3 is used to reduce the feature map output by Neck2_2 by a certain multiple.
8. The modeling method according to claim 7, characterized in that: There are four separation detection heads, which are respectively denoted as Decoupled Head1, Decoupled Head2, Decoupled Head3, and Decoupled Head4. Each separation detection head includes two branches, which respectively predict the category and bounding box coordinates of the target. Decoupled Head1 is used to perform convolution layer processing on the feature map output by Neck1_3, Decoupled Head2 is used to perform convolution layer processing on the feature map output by Neck2_1, Decoupled Head3 is used to perform convolution layer processing on the feature map output by Neck2_2, and Decoupled Head4 is used to perform convolution layer processing on the feature map output by Neck2_3.
9. The modeling method according to claim 7, characterized in that: When executing step 3, after completing one training, the loss function is used to determine whether the training is completed. The loss function is set to: L d =L cls +L obj +L reg , Among them, L d is the overall target detection loss function, L cls represents the classification loss, L obj represents the prediction loss, L reg Represents the regression loss, which is as follows: L cls =CrossEntropy(X cls ,GT cls ), L obj =CrossEntropy(X obj ,GT obj ), L reg =CrossEntropy(X box ,GT box ), CrossEntropy(p,q)=-[plog(q)+(1-p)log(1-q)], Where CrossEntropy(·) represents the cross entropy loss function, X cls , X obj , X box Represent the target category, confidence and location predicted by the model, GT cls represents the true category of the target, L cls represents the cross entropy loss between the target category predicted by the model and the true target category, L obj A Boolean value GT indicating the model's confidence in the prediction and whether the target actually exists obj The cross entropy loss, L reg It represents the regression accuracy loss of the model, X box and GT box Represent the bounding box coordinates of the model prediction and the actual target respectively. < / name> < / object>