A method for detecting ship parts from satellite images based on key point regression
Through a method based on key point regression, combined with data amplification and equalization processing, a target and component detection network is built, which solves the problem of unbalanced distribution and rotational changes of ship target detection in satellite images, and realizes the key points of ships and high-precision detection of components.
Patent Information
- Application Number
- CN202210175356.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-02-24
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2042-02-24
AI Technical Summary
The existing satellite image ship target detection methods cannot effectively detect key points and key components of the ship, and there are problems of data distribution imbalance and change in target rotation, resulting in low detection accuracy.
Using satellite image ship component detection method based on key point regression, the ship component detection training data set is constructed, data amplification and equalization is performed using the minimized coefficient of variation method, and the target detection network and component detection network are constructed, combined with Locally densely connect convolutional neural networks and feature pyramids, calculate the rotation frame cross-match loss function and key point regression loss function, and perform model training and component detection.
It realizes high-precision detection of ship targets and components, overcomes the long-tail effect and rotational changes of data distribution, can quickly locate key points and key components of the ship, and improves detection accuracy and fine-grainedness.
Smart Images

Figure CN114782800B_ABST
Abstract
Description
Technical Field
[0001] The invention belongs to the field of satellite image recognition, and more specifically relates to a satellite image ship component detection method based on key point regression. Background Art
[0002] Ships are important carriers of maritime transportation and are of great significance to military reconnaissance, environmental protection, marine detection and other fields. Ship target detection, as an important technology for maritime situational awareness, is the premise of ship target tracking and state estimation, and has broad application prospects in military and civilian fields. With the rapid development of remote sensing technology, the use of satellite images for ship target detection has the advantages of all-day, all-weather, and no airspace restrictions. However, compared with natural scene images, satellite images have the characteristics of large image size, insufficient resolution, poor imaging quality, target rotation changes, small targets, and few targets.
[0003] In recent years, the application of deep learning technology in the field of computer vision has greatly improved the target detection performance compared with the traditional manually designed feature algorithm. The current ship target detection method based on deep learning technology has been improved in data enhancement, feature extraction, feature reuse, loss function design, etc., which has alleviated the problem of insufficient satellite image resolution and small and few targets to a certain extent, but the detection accuracy of ship targets still needs to be improved. In addition, the existing detection method can only detect the attributes and location distribution of ship targets, and cannot determine the key points and key components of the ship. Summary of the invention
[0004] The purpose of the present invention is to address the problems of long-tail distribution of data, coarse granularity of detection result attributes, target rotation changes, etc. in satellite image ship detection, and propose a method that can simultaneously perform ship target detection and ship component detection.
[0005] The technical solution adopted by the present invention is:
[0006] A satellite image ship component detection method based on key point regression includes the following steps:
[0007] Step 1: construct a satellite remote sensing image ship component detection training data set, and manually annotate the component types and locations in the remote sensing images; the component types include ship location, ship type, naval gun, vertical radiation system, bridge, apron, hull number, and take-off and landing runway;
[0008] Step 2: Use the method of minimizing the coefficient of variation to perform data amplification and balancing on the training data with obvious long-tail effects;
[0009] Step 3, construct a ship target and component detection model, including a target detection network and a component detection network; wherein the feature extractor in the target detection network uses a local densely connected convolutional neural network and a feature pyramid, the local densely connected convolutional neural network is used to extract features from the input image, and the feature pyramid is used to perform multi-scale aggregation of the extracted features;
[0010] Step 4: Calculate the target detection loss function according to the calculation criterion of the rotation frame intersection-over-union ratio, calculate the component detection loss function according to the key point encoding representation method of the ship components, and incorporate the target detection loss function and the component detection loss function into the ship target and component detection loss function;
[0011] Step 5: Use the training data set after amplification and equalization to train the ship target and component detection model. During the training process, according to the target coordinates output by the target detection network, the input image of the target detection network and the fused feature map converged by the feature pyramid are aligned to obtain the interpolated and regularized target and feature map slices. The target and feature map slices are used as the input of the component detection network, and the back propagation optimization algorithm is iterated cyclically to obtain the trained ship target and component detection model.
[0012] Step 6: Input the image to be detected into the trained ship target and component detection model, decode the predicted coordinates of the ship target and the key point coordinates and sizes of the ship components, and mark the ship target and component detection results in the original image.
[0013] Furthermore, the data amplification and balancing method in step 2 is specifically as follows:
[0014] Set the data set amplification vector y, y = {(y1, y2, ...y n )|y i ∈N +},y i Indicates the magnification of the i-th image, N + Represents a positive integer, counting the number of target categories contained in all images in the training data set, represented by the matrix X, where X i,j represents the number of targets of type j in the i-th image, then the distribution of the number of targets of each type in the training data set is represented by the vector z=y T X indicates that the closer z is to uniform distribution, the closer the data set distribution is to balanced. For this purpose, the following mathematical model for minimizing the coefficient of variation is established to perform data amplification and balancing on the training data:
[0015]
[0016]
[0017] In the formula express n -dimensional positive integer vector, express n×m dimensional integer matrix, n represents the number of images in the dataset, m represents the number of target categories in the dataset, y is the decision variable, the feasible domain is an n-dimensional positive integer vector, X is a constant, and the objective function is It means z=y T The coefficient of variation of X, with the constraints S∈N + , which means that the number of images after constrained amplification is no more than S, and the amplified images are processed by random affine transformation, random contrast transformation, random cropping and noise addition.
[0018] Furthermore, in step 3, the local densely connected convolutional neural network adopts the CSPDarknet53 network model, and the feature pyramid adopts a three-layer FPN structure.
[0019] Furthermore, in step 4, the target detection loss includes three parts: position loss, confidence loss and category loss; namely:
[0020] loss(ship)=loss(object)+loss(unit)
[0021] loss(object)=loss(location)+loss(object_confidence)+loss(object_class)
[0022] Among them, loss(ship) is the loss function of ship target and component detection, loss(object) is the ship target detection loss, loss(unit) is the component detection loss, loss(location) is the target detection location loss, loss(object_confidence) is the confidence loss, and loss(object_class) is the category loss;
[0023] The target detection position loss is designed using a loss function that takes into account the rotation frame intersection-over-union ratio, center point distance, and angle deviation. Specifically:
[0024]
[0025] Among them, p represents the three layers of feature maps in the feature pyramid, S represents the grid size of each layer of feature maps, B represents the number of anchor boxes pre-set in each grid, IOU is the intersection-over-union ratio of the predicted box and the real box, dis(core) is the distance between the predicted box and the center coordinates of the real box, dis(diag) is the diagonal length of the smallest enclosing rectangle between the predicted box and the real box, sin(θ) is the sine value of the angle between the long side of the predicted box and the real box, and λ is the angle loss coefficient;
[0026] The target detection confidence loss uses cross entropy loss, and the target detection category loss uses binary cross entropy loss.
[0027] Furthermore, in step 5, the target slice alignment uses a region of interest pooling method to keep the sizes of all target and feature slices consistent.
[0028] Furthermore, in step 4, the key point regression loss is used as the component detection loss function, that is:
[0029]
[0030] Where N represents the number of all targets output by target detection, M represents the number of target components, dis(point1, point2) represents the Euclidean distance between the center coordinates of the predicted component and the true component, w1 and h1 represent the width and length of the predicted component, respectively, and w2 and h2 represent the width and length of the true component, respectively;
[0031] The Adam optimization algorithm is used for parameter training during training.
[0032] The advantages of the present invention over the prior art are:
[0033] (1) A satellite imagery ship component detection training dataset was constructed, providing data-driven support for component-level detection using deep learning methods;
[0034] (2) The method of minimizing the coefficient of variation is used to perform data amplification and balancing, which alleviates the long-tail effect of data distribution and expands the sample space;
[0035] (3) Component-level detection is performed by means of key point regression. The training phase can converge quickly, and the prediction phase can quickly locate the key points of the ship and the sizes of key components. BRIEF DESCRIPTION OF THE DRAWINGS
[0036] Figure 1 This is a diagram showing an example of labeling ship targets and components in satellite images of the present invention.
[0037] Figure 2 The overall technical flow chart of the method proposed in the present invention.
[0038] Figure 3 (a) is a distribution diagram of ship and component data of the data set used in the present invention, Figure 3 (b) is the data distribution diagram of ships and components after data amplification of the present invention.
[0039] Figure 4 (a) is a schematic diagram of the target detection position loss used in the present invention, Figure 4(b) is a schematic diagram of the key point regression loss function used in the present invention. DETAILED DESCRIPTION
[0040] At present, some ship target detection methods based on deep learning technology have been improved in data enhancement, feature extraction, feature reuse, loss function design, etc., which have alleviated the problems of insufficient satellite image resolution and small and few targets to a certain extent. However, the detection accuracy of ship targets still needs to be improved. In addition, the existing detection methods can only detect the attributes and location distribution of ship targets, and cannot determine the key points and key components of the ship. Therefore, the present invention proposes a satellite image ship component detection method based on key point regression. The specific flow chart is as follows: Figure 2 As shown in the figure. Figure 1 shown.
[0041] The specific implementation manner and basic principle of the present invention will be further described below in conjunction with the accompanying drawings.
[0042] Step 1: Dataset annotation. Construct a satellite image ship component detection training dataset, and manually annotate the ship position, ship type, and component types and locations such as naval guns, vertical radiation systems, bridges, aprons, hull numbers, and take-off and landing runways in remote sensing images. Figure 1 shown.
[0043] Step 2: Data augmentation. The method of minimizing the coefficient of variation is used to augment and balance the training data with obvious long-tail effects. First, the number of target categories contained in all images in the statistical data set is calculated, such as Figure 3 (a) shows the number of target categories in the data set used in the present invention, including 6 component categories and 23 ship model categories. The category distribution is extremely unbalanced and there is an obvious long-tail effect. Secondly, an optimization objective function that minimizes the coefficient of variation is established. The optimization problem is a mixed integer nonlinear programming problem. Then, the branch and bound method is used to solve the optimal solution. Finally, data augmentation and random transformation are performed according to the solution results, as shown in FIG. Figure 3 (b) is the data distribution diagram of ships and components after data amplification and equalization, and the long tail effect is significantly alleviated. Previous data enhancement methods such as random cropping and random flipping cannot solve the problem of unbalanced data distribution. The data enhancement method used in this invention can improve data diversity while alleviating the problem of unbalanced data distribution.
[0044] The data augmentation and balancing methods are as follows:
[0045] Set the data set amplification vector y, y = {(y1, y2, ...y n )|y i ∈N +},yi Indicates the magnification of the i-th image, N + Represents a positive integer, counting the number of target categories contained in all images in the training data set, represented by the matrix X, where X i,j represents the number of targets of type j in the i-th image, then the distribution of the number of targets of each type in the training data set is represented by the vector z=y T X indicates that the closer z is to uniform distribution, the closer the data set distribution is to balanced. For this purpose, the following mathematical model for minimizing the coefficient of variation is established to perform data amplification and balancing on the training data:
[0046]
[0047]
[0048] In the formula express n -dimensional positive integer vector, express n×m dimensional integer matrix, n represents the number of images in the dataset, m represents the number of target categories in the dataset, y is the decision variable, the feasible domain is an n-dimensional positive integer vector, X is a constant, and the objective function is It means z=y T The coefficient of variation of X, with the constraints S∈N + , which means that the number of images after constrained amplification is no more than S, and the amplified images are processed by random affine transformation, random contrast transformation, random cropping and noise addition.
[0049] Step 3, construct a ship target and component detection model, including a target detection network and a component detection network; the feature extractor in the target detection network adopts a local densely connected convolutional neural network and a feature pyramid, the local densely connected convolutional neural network is used to extract features from the input image, and the feature pyramid is used to perform multi-scale aggregation of the extracted features; the local densely connected convolutional neural network adopts the CSPDarknet53 network model, and the feature pyramid adopts a three-layer FPN structure.
[0050] Step 4: Calculate the target detection loss function according to the calculation criterion of the rotation frame intersection-over-union ratio, calculate the component detection loss function according to the key point encoding representation method of the ship components, and incorporate the target detection loss function and the component detection loss function into the ship target and component detection loss function;
[0051] The target detection loss includes three parts: position loss, confidence loss and category loss. The position loss is calculated according to the rotation box intersection over union (IOU) calculation criterion, such as Figure 4 As shown in (a), the calculation criterion of the rotation frame intersection and union ratio of the present invention takes into account both the rotation angle and the center point distance. The long side and the rotation box The angle θ of the long side is expressed, and the distance between the center points is expressed The center point A and The Euclidean distance between the center point B and AB and The minimum bounding rectangle of The ratio of the diagonal Euclidean distance DE is expressed as, IOU is expressed as and The intersection of Area and and Union Area The target detection confidence loss adopts cross entropy loss, and the target detection category loss adopts binary cross entropy loss. Compared with the previous method, the position loss function adopted by the present invention can speed up the training process and improve the detection position accuracy.
[0052] The target detection loss includes three parts: position loss, confidence loss and category loss; namely:
[0053] loss(ship)=loss(object)+loss(unit)
[0054] loss(object)=loss(location)+loss(object_confidence)+loss(object_class)
[0055] Among them, loss(ship) is the loss function of ship target and component detection, loss(object) is the ship target detection loss, loss(unit) is the component detection loss, loss(location) is the target detection location loss, loss(object_confidence) is the confidence loss, and loss(object_class) is the category loss;
[0056] The target detection position loss is designed using a loss function that takes into account the rotation frame intersection-over-union ratio, center point distance, and angle deviation. Specifically:
[0057]
[0058] Among them, p represents the three layers of feature maps in the feature pyramid, S represents the grid size of each layer of feature maps, B represents the number of anchor boxes pre-set in each grid, IOU is the intersection-over-union ratio of the predicted box and the real box, dis(core) is the distance between the predicted box and the center coordinates of the real box, dis(diag) is the diagonal length of the smallest enclosing rectangle between the predicted box and the real box, sin(θ) is the sine value of the angle between the long side of the predicted box and the real box, and λ is the angle loss coefficient;
[0059] The target detection confidence loss uses cross entropy loss, and the target detection category loss uses binary cross entropy loss.
[0060] The key point regression loss is used as the component detection loss function, that is:
[0061]
[0062] Where N represents the number of all targets output by target detection, M represents the number of target components, dis(point1, point2) represents the Euclidean distance between the center coordinates of the predicted component and the true component, w1 and h1 represent the width and length of the predicted component, respectively, and w2 and h2 represent the width and length of the true component, respectively;
[0063] The Adam optimization algorithm is used for parameter training during training.
[0064] The key point encoding method of the present invention adopts a normalized representation method, such as Figure 4 As shown in (b), the key point coordinates and the length and width of the component are divided by the length and width of the target slice to obtain the (x, y, w, h) four-element representation.
[0065] Step 5, using the training data set after amplification and equalization processing to train the ship target and component detection model, during the training process, according to the target coordinates output by the target detection network, the input image of the target detection network and the fused feature map converged by the feature pyramid are aligned to obtain interpolated and regularized target and feature map slices, and the alignment method of the present invention adopts the ROI pooling method to regularize all slices to a scale of 7×49; the target and feature map slices are used as the input of the component detection network, and the trained ship target and component detection model is obtained through a back propagation optimization algorithm.
[0066] Step 6: Input the image to be detected into the trained ship target and component detection model, decode the predicted coordinates of the ship target and the key point coordinates and sizes of the ship components, and mark the ship target and component detection results in the original image.
[0067] The present invention provides a method for simultaneously detecting ship targets and ship components. The method overcomes the long-tail effect of the distribution of ship target and component data through a data balancing processing method, thereby improving the detection accuracy. The method of key point regression is used to perform component-level fine-grained detection, thereby overcoming the problems of target occlusion and image blur, and can provide precise recognition results for the estimation of ship attributes and status.
Claims
1. A method for detecting ship parts from satellite images based on key point regression. It is characterized in that The following steps are involved: Step 1: construct a satellite remote sensing image ship component detection training data set, and manually annotate the component types and locations in the remote sensing images; the component types include ship location, ship type, naval gun, vertical radiation system, bridge, apron, hull number, and take-off and landing runway; Step 2: Use the method of minimizing the coefficient of variation to perform data amplification and balancing on the training data with obvious long-tail effects; Step 3, construct a ship target and component detection model, including a target detection network and a component detection network; wherein the feature extractor in the target detection network uses a local densely connected convolutional neural network and a feature pyramid, the local densely connected convolutional neural network is used to extract features from the input image, and the feature pyramid is used to perform multi-scale aggregation of the extracted features; Step 4: Calculate the target detection loss function according to the calculation criterion of the rotation frame intersection-over-union ratio, calculate the component detection loss function according to the key point encoding representation method of the ship components, and incorporate the target detection loss function and the component detection loss function into the ship target and component detection loss function; Step 5: Use the training data set after amplification and equalization to train the ship target and component detection model. During the training process, according to the target coordinates output by the target detection network, the input image of the target detection network and the fused feature map converged by the feature pyramid are aligned to obtain the interpolated and regularized target and feature map slices. The target and feature map slices are used as the input of the component detection network, and the back propagation optimization algorithm is iterated cyclically to obtain the trained ship target and component detection model. Step 6: Input the image to be detected into the trained ship target and component detection model, decode the predicted coordinates of the ship target and the key point coordinates and sizes of the ship components, and mark the ship target and component detection results in the original image; The data amplification and balancing method in step 2 is specifically as follows: Set the data set expansion factor vector y, y = {(y 1 ,y 2 ,...y n )|y i ∈N + },y i Indicates the magnification of the i-th image, N + Represents a positive integer, counting the number of target categories contained in all images in the training data set, represented by the matrix X, where X i,j represents the number of targets of type j in the i-th image, then the distribution of the number of targets of each type in the training data set is represented by the vector z=y Τ X indicates that the closer z is to uniform distribution, the closer the data set distribution is to balanced. For this purpose, the following mathematical model for minimizing the coefficient of variation is established to perform data amplification and balancing on the training data: In the formula express n -dimensional positive integer vector, express n×m dimensional integer matrix, n represents the number of images in the dataset, m represents the number of target categories in the dataset, y is the decision variable, the feasible domain is an n-dimensional positive integer vector, X is a constant, and the objective function is It means z=y Τ The coefficient of variation of X, with the constraints S∈N + , which means that the number of images after constraint amplification is no more than S, and the amplified images are subjected to random affine transformation, random contrast transformation, random cropping and noise processing; Among them, in step 4, the target detection loss includes three parts: position loss, confidence loss and category loss; that is: loss(ship)=loss(object)+loss(unit) loss(object)=loss(location)+loss(object_confidence)+loss(object_class) Among them, loss(ship) is the loss function of ship target and component detection, loss(object) is the ship target detection loss, loss(unit) is the component detection loss, loss(location) is the target detection location loss, loss(object_confidence) is the confidence loss, and loss(object_class) is the category loss; The target detection position loss adopts a loss function design that takes into account the rotation frame intersection-over-union ratio, center point distance, and angle deviation. Specifically: Among them, p represents the three layers of feature maps in the feature pyramid, S represents the grid size of each layer of feature maps, B represents the number of anchor boxes pre-set in each grid, IOU is the intersection-over-union ratio of the predicted box and the real box, dis(core) is the distance between the predicted box and the center coordinates of the real box, dis(diag) is the diagonal length of the smallest enclosing rectangle between the predicted box and the real box, sin(θ) is the sine value of the angle between the long side of the predicted box and the real box, and λ is the angle loss coefficient; The target detection confidence loss uses cross entropy loss, and the target detection category loss uses binary cross entropy loss.
2. According to the satellite image ship component detection method based on key point regression as described in claim 1, It is characterized in that In step 3, the local densely connected convolutional neural network adopts the CSPDarknet53 network model, and the feature pyramid adopts a three-layer FPN structure.
3. According to the satellite image ship component detection method based on key point regression as described in claim 1, It is characterized in that In step 5, the target slice alignment uses the region of interest pooling method to keep the sizes of all target and feature slices consistent.
4. According to the satellite image ship component detection method based on key point regression as described in claim 1, It is characterized in that In step 4, the key point regression loss is used as the component detection loss function, that is: Where N represents the number of all targets output by target detection, M represents the number of target components, dis(point1,point2) represents the Euclidean distance between the center point coordinates of the predicted component and the true component, and w 1 and h 1 Respectively represent the width and length of the predicted component, w 2 and h 2 Respectively represent the width and length of the real component; The Adam optimization algorithm is used for parameter training during training.
