Ship fine identification method and device

Abstract

The present invention provides a ship fine recognition method and device, which relates to the field of image recognition technology and is used to solve the problem that the existing detection method is difficult to finely recognize the outline of the ship. The method comprises: inputting a remote sensing image to be detected into a trained recognition model to obtain the coordinates of the pixels corresponding to the ship target detection frame and multiple components of the ship, wherein the multiple components include a main gun, a vertical launch system, a chimney and a flight deck; performing weighted fitting on the coordinates of the pixels corresponding to the multiple components of the ship to obtain a center line corresponding to the ship; taking the angle value between the target direction determined by the center line and the vertical direction as the direction angle, the ship target detection frame is posture-adjusted to the vertical direction; matching the ship target detection frame after posture adjustment with the template image in the template library to obtain the model and fine outline of the ship.

Description

Technical Field

[0001] The present invention relates to the technical field of image recognition, and in particular to a method and device for fine-grained ship recognition. Background Art

[0002] In the field of optical remote sensing image research, the detection and recognition of ship targets has always been an important research topic. Ship detection and recognition specifically refers to the detection of ship targets and their types, sizes and other information through remote sensing images. In practical applications, due to the low resolution of ships themselves in remote sensing images and their dense distribution in areas such as ports, commonly used target detection algorithms are prone to false detection and missed detection. Therefore, how to efficiently and accurately detect ship targets is an important research direction.

[0003] The traditional ship detection and recognition method is divided into three stages: extracting target candidate regions on a given remote sensing image, then extracting features from the target candidate regions, and finally using a classifier for classification. The purpose of extracting target candidate regions is to mark the areas that may contain the target to be detected from the remote sensing image for subsequent detection and classification. For remote sensing images, it is first necessary to perform land and sea segmentation to remove land areas that occupy too large an area and have rich texture information to prevent them from covering up the ship target in ship target detection. The main methods for land and sea segmentation are threshold segmentation and region growing. Then, target candidate regions are extracted. The main methods are threshold detection and fractal model detection. Feature extraction mainly extracts features from the separated target area image. Common ship features include: grayscale, texture, shape, etc. In addition, there are also extracted ship wake features, wavelet features, etc. When using a classifier for classification, the commonly used classifier is support vector machines (SVM).

[0004] Traditional ship detection methods use manually designed features to detect ships in images. This method has poor performance and is difficult to suppress false alarms. Compared with traditional methods, the performance of target detection methods based on deep learning has been greatly improved.

[0005] Current deep learning-based object detection methods can be roughly divided into two categories: single-stage methods and two-stage methods. Single-stage methods mainly include Region-Convolutional Neural Network (R-CNN), Fast R-CNN, Faster R-CNN, etc. Two-stage methods mainly include YOLO (You Only Look Once), etc. The single-stage method first generates high-quality candidate bounding boxes on the input image, and then classifies and regresses the candidate bounding boxes to obtain the final detection results. The two-stage method directly predicts the position bounding box and target category on the input image by regression.

[0006] The accuracy of deep learning-based ship detection algorithms depends on the quality and quantity of training data. If there are deviations or omissions in the training data, the prediction results of the algorithm may be affected. Accurately labeled data is often not easy to obtain. In addition, in areas with densely distributed ships such as ports, the above methods are prone to missed detections and false detections, and can only locate the target position of the ship, but it is difficult to finely identify the outline of the ship. Summary of the invention

[0007] In view of the above technical problems, the present invention provides a method and device for fine-grained ship identification, which are used to at least partially solve the above technical problems.

[0008] Based on this, the first aspect of the present invention provides a method for fine-grained ship identification, comprising: inputting a remote sensing image to be detected into a trained recognition model to obtain the coordinates of the pixels corresponding to the ship target detection frame and multiple components of the ship, wherein the multiple components include a main gun, a vertical launch system, a chimney and a flight deck; performing weighted fitting on the coordinates of the pixels corresponding to the multiple components of the ship to obtain a center straight line corresponding to the ship; taking the angle value between the target direction determined by the center straight line and the vertical direction as the direction angle, the ship target detection frame is posture-adjusted to the vertical direction; matching the ship target detection frame after posture adjustment with a template image in a template library to obtain the model and fine-grained contour of the ship.

[0009] According to an embodiment of the present invention, the equation of the center line is:

[0010] Y=aX+b

[0011]

[0012]

[0013] Among them, (x i ,y i ) is the coordinate of each pixel corresponding to the i-th component, P iis the accuracy of each pixel corresponding to the i-th component, n is the total number of components, and i takes values ​​of 1, 2, ..., n.

[0014] According to an embodiment of the present invention, matching the ship target detection frame after posture adjustment with the template image in the template library to obtain the model and refined contour of the ship includes: calculating the normalized sum of square differences between the first grayscale value of each pixel in the ship target detection frame after posture adjustment and the grayscale value of each pixel in the template image in the template library; determining the template image corresponding to the minimum normalized sum of square differences as the target template image of the ship; and fitting the refined contour of the ship through the target template image.

[0015] According to an embodiment of the present invention, calculating the normalized square sum of the difference between the first grayscale value of each pixel in the ship target detection frame after the posture adjustment and the grayscale value of each pixel in the template image in the template library includes:

[0016]

[0017] Calculate the normalized sum of squared differences R(x, y), where I(x, y) is the grayscale value of the pixel with coordinates (x, y) in the ship target detection frame, T(x′, y′) is the grayscale value of the pixel with coordinates (x′, y′) in the template image, and I(x+x′, y+y′) is the grayscale value of the pixel with coordinates (x+x′, y+y′) in the ship target detection frame.

[0018] According to an embodiment of the present invention, before inputting the remote sensing image to be detected into the trained recognition model, the method further includes: performing image denoising, contrast enhancement and land-sea segmentation on the remote sensing image to be detected.

[0019] According to an embodiment of the present invention, performing land and sea segmentation on a remote sensing image to be detected includes: performing land and sea area segmentation on a remote sensing image with image noise reduction and contrast enhancement using a grayscale histogram-based segmentation method, expanding the edge of the segmented ocean portion image using morphological methods such as dilation, corrosion, and opening and closing, and segmenting part of the land information and all of the ship information into the sea surface portion.

[0020] According to an embodiment of the present invention, the method also includes: training a recognition model; training the recognition model includes: obtaining historical remote sensing images containing ship targets; labeling multiple components of the ship in the historical remote sensing images, the multiple components including a main gun, a vertical launch system, a chimney and a flight deck; constructing a training picture set based on the labeled historical remote sensing images; inputting the training data set into a convolutional neural network for training to obtain a trained recognition model.

[0021] According to an embodiment of the present invention, a training data set is input into a convolutional neural network for training to obtain a trained recognition model, including: inputting a training picture set into a pre-trained region candidate network for training to obtain a trained pre-trained region candidate network; inputting a training picture set into a convolutional neural network model, using the trained pre-trained region candidate network to obtain a region of interest, fixing the weight parameters of the trained pre-trained region candidate network, repeatedly iterating the convolutional neural network model, and obtaining a trained convolutional neural network model as a trained recognition model.

[0022] According to an embodiment of the present invention, the convolutional neural network model includes a YOLO model or a Faster-RCNN model.

[0023] The second aspect of the present invention provides a device for fine-grained identification of ships, comprising: an identification module, which is used to input a remote sensing image to be detected into a trained identification model to obtain the coordinates of the pixels corresponding to the ship target detection frame and multiple components of the ship, wherein the multiple components include a main gun, a vertical launch system, a chimney and a flight deck; a fitting module, which is used to perform weighted fitting on the coordinates of the pixels corresponding to the multiple components of the ship to obtain a center straight line corresponding to the ship; an adjustment module, which is used to adjust the attitude of the ship target detection frame to the vertical direction with the angle value between the target direction determined by the center straight line and the vertical direction as the direction angle; a matching module, which is used to match the ship target detection frame after the attitude adjustment with a template image in a template library to obtain the model and fine-grained contour of the ship.

[0024] The ship fine identification method and device provided by the embodiment of the present invention at least include the following

[0025] Beneficial effects:

[0026] By identifying the coordinates of pixels corresponding to multiple parts of the same ship, since the parts are smaller than the main body of the ship, the features can be more uniform under different viewing angles. The center straight line obtained by weighted fitting based on the coordinates of the pixels corresponding to the parts is used to adjust the posture of the ship target detection frame, so that a more accurate ship target detection frame can be obtained. Combined with template matching, the outline of the ship can be fitted more accurately, and the detection efficiency is improved.

[0027] By labeling multiple parts of the same ship, we can obtain finely labeled data and increase the number of samples, thereby improving detection accuracy and results. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The above and other objects, features and advantages of the present invention will become more apparent through the following description of the embodiments of the present invention with reference to the accompanying drawings, in which:

[0029] Figure 1The flowchart of the ship fine identification method provided by an embodiment of the present invention is schematically shown.

[0030] Figure 2 Schematically showing the changes in remote sensing images corresponding to each operation of the ship fine identification method provided by an embodiment of the present invention

[0031] Figure 3 The flowchart of the recognition model training method provided by an embodiment of the present invention is schematically shown.

[0032] Figure 4 The flowchart of the device for fine-grained ship identification provided by an embodiment of the present invention is schematically shown. DETAILED DESCRIPTION

[0033] In order to make the purpose, technical scheme and advantages of the present invention more clearly understood, the present invention is further described in detail below in combination with specific embodiments and with reference to the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0034] The terms used herein are only for describing specific embodiments and are not intended to limit the present invention. The terms "comprise", "include", etc. used herein indicate the existence of the characteristic images, steps, operations and / or components, but do not exclude the existence or addition of one or more other characteristic images, steps, operations or components.

[0035] In the present invention, unless otherwise clearly specified and limited, the terms "installed", "connected", "connected", "fixed" and the like should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection, an electrical connection, or can communicate with each other; it can be a direct connection, or an indirect connection through an intermediate medium, it can be the internal connection of two elements or the interaction relationship between two elements. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0036] In the description of the present invention, it should be understood that the terms "longitudinal", "length", "circumferential", "front", "rear", "left", "right", "top", "bottom", "inside", "outside", etc., indicating the orientation or positional relationship are based on the orientation or positional relationship shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the subsystem or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention.

[0037] Throughout the drawings, the same elements are represented by the same or similar reference numerals. Conventional structures or configurations will be omitted when they may cause confusion in the understanding of the present invention. The shapes, sizes, and positional relationships of the components in the drawings do not reflect the actual size, proportion, and actual positional relationship. In addition, in the present invention, any reference symbol between brackets should not be constructed as a limitation to the present invention.

[0038] Similarly, in order to simplify the present invention and help understand one or more of the various disclosed aspects, in the above description of exemplary embodiments of the present invention, the various characteristic images of the present invention are sometimes grouped together into a single embodiment, figure or description thereof. The description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" and the like means that the specific characteristic images, structures, materials or features described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific characteristic images, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0039] In addition, the terms "first" and "second" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical feature images indicated. Therefore, the feature image defined as "first" or "second" may explicitly or implicitly include one or more of the feature images. In the description of the present invention, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0040] Figure 1 The flowchart of the ship fine identification method provided by an embodiment of the present invention is schematically shown. Figure 2 The diagram schematically shows the changes in the remote sensing image corresponding to each operation of the refined ship identification method provided by an embodiment of the present invention.

[0041] like Figure 1 and Figure 2 As shown, the ship fine-grained identification method may include, for example, operations S101 to S104.

[0042] In operation S101, the remote sensing image to be detected is input into a trained recognition model to obtain the coordinates of the pixels corresponding to the ship target detection frame and multiple components of the ship.

[0043] In an embodiment of the present invention, the plurality of components include a main gun, a vertical launching system, a chimney and a flight deck.

[0044] Since the acquired remote sensing images to be detected may contain noise, which may affect the recognition accuracy, the remote sensing images to be detected may be preprocessed by image denoising, contrast enhancement, and land-sea segmentation before being input into the trained recognition model.

[0045] Specifically, the image to be detected can be denoised using the wavelet threshold filtering method. The image contrast can be enhanced using the Gamma correction method for the denoised image. The remote sensing image that has completed denoising and contrast enhancement can be segmented into land and sea areas using a grayscale histogram-based segmentation method. The edge of the segmented ocean image can be expanded using morphological methods such as dilation, corrosion, and opening and closing, and some land information and all ship information can be segmented into the sea surface to avoid missed detection.

[0046] It should be noted that the recognition of the remote sensing image to be detected will correspond to the output accuracy, which can be used as the weight of weighted fitting later.

[0047] In operation S102, weighted fitting is performed on the coordinates of pixels corresponding to a plurality of components of the ship to obtain a center straight line corresponding to the ship.

[0048] In an embodiment of the present invention, weighted fitting is performed on the coordinates of all components detected on the same ship to obtain the center line equation. The regression equation of the center line is as follows:

[0049] Y=aX+b

[0050]

[0051]

[0052] Among them, (x i ,y i ) is the coordinate of each pixel corresponding to the i-th component, P i is the accuracy of each pixel corresponding to the i-th component, n is the total number of components, and i takes values ​​of 1, 2, ..., n.

[0053] In operation S103, the angle between the target direction determined by the central straight line and the vertical direction is used as the direction angle, and the ship target detection frame is adjusted to the vertical direction.

[0054] In an embodiment of the present invention, the obtained center straight line is determined as the direction of the target, and the angle value with the vertical direction is used as the direction angle, and the target posture is adjusted to a unified vertical direction using this angle.

[0055] In operation S104, the ship target detection frame after posture adjustment is matched with the template image in the template library to obtain the model and refined outline of the ship.

[0056] In an embodiment of the present invention, operation S104 may specifically include: calculating the normalized square sum of the difference between the first gray value of each pixel in the ship target detection frame after the posture adjustment and the gray value of each pixel in the template image in the template library. Determine the template corresponding to the minimum normalized square sum of the difference as the target template of the ship. Fit the refined contour of the ship through the target template.

[0057] Exemplarily, the template images in the template library can be collected from higher-resolution images (0.5-2m), and the relevant information, such as the specific model, actual length and width, etc., can be improved to establish the template library.

[0058] according to

[0059]

[0060] Calculate the normalized sum of squared differences R(x, y), where I(x, y) is the grayscale value of the pixel with coordinates (x, y) in the ship target detection frame, T(x′, y′) is the grayscale value of the pixel with coordinates (x′, y′) in the template image, and I(x+x′, y+y′) is the grayscale value of the pixel with coordinates (x+x′, y+y′) in the ship target detection frame.

[0061] For the normalized square difference matching method, the smaller the difference R is, the better the matching effect is. The difference between the target to be identified and all templates in the template library is calculated, and the template model with the best matching effect with the target is determined as the model of the target to be tested.

[0062] After determining the target template image, the detected ship components are used for positioning, and the refined contour of the ship target to be detected is fitted through the target template image.

[0063] Based on the above embodiment, the ship refined identification method also includes training an identification model.

[0064] Figure 3 The flowchart of the recognition model training method provided by an embodiment of the present invention is schematically shown.

[0065] like Figure 3 As shown, the recognition model training method may include, for example, operations S301 to S304.

[0066] In operation S301, a historical remote sensing image containing a ship target is acquired.

[0067] In an embodiment of the present invention, remote sensing images containing ship targets can be extracted from Google Earth, including categories such as Arleigh Burke-class destroyers, Perry-class frigates, San Antonio-class amphibious dock transport ships, and Ticonderoga-class cruisers.

[0068] In operation S302, multiple components of a ship in a historical remote sensing image are annotated, where the multiple components include a main gun, a vertical launch system, a chimney, and a flight deck.

[0069] In the embodiment of the present invention, the ship components may include, for example, ship categories such as Arleigh Burke-class destroyers, Perry-class frigates, San Antonio-class amphibious transport dock ships, and Ticonderoga-class cruisers, and four ship component categories such as main guns, vertical launch systems, chimneys, and flight decks. The specific marking content is the four categories of ship components: main guns, vertical launch systems, chimneys, and flight decks.

[0070] In operation S303, a training picture set is constructed based on the annotated historical remote sensing images.

[0071] In the embodiment of the present invention, after the labeling is completed, a training data set in VOC format is established. The format of the data set in VOC format is as follows:

[0072] ---VOC2007

[0073] ------Annotations

[0074] ------ImagesSets

[0075] ------JPEGImages

[0076] Annotations stores XML files, which correspond one-to-one to the images in JPEGImages and store the annotation information of the images.

[0077] ImagesSets stores txt files, each line of which contains the name of an image, including test.txt and train.txt, which are used as indexes for subsequent testing and training, respectively.

[0078] JPEGImages stores the images in the dataset.

[0079] In operation S304, the training data set is input into the convolutional neural network for training to obtain a trained recognition model.

[0080] In an embodiment of the present invention, the training image set (train.txt) is input into the pre-trained region candidate network for training to obtain a trained pre-trained region candidate network (RPN). The RPN obtains the region of interest, calculates the IoU with the annotated ground truth, divides the positive and negative samples, and calculates the loss function. Iterates repeatedly until the loss function is minimized to obtain a trained RPN.

[0081] Input the training picture set into the convolutional neural network model, use the trained pre-trained region candidate network to obtain the region of interest, fix the weight parameters of the trained pre-trained region candidate network, repeatedly iterate the convolutional neural network model, and obtain the trained convolutional neural network model as the trained recognition model. The above steps can be repeated once, and the parameters can be fine-tuned to finally obtain the trained ship target detection network. The convolutional neural network model can include a YOLO model or a Faster-RCNN model.

[0082] Figure 4 The flowchart of the device for fine-grained ship identification provided by an embodiment of the present invention is schematically shown.

[0083] like Figure 4 As shown, the ship fine-grained identification device 400 may include, for example, an identification module 410 , a fitting module 420 , an adjustment module 430 and a matching module 440 .

[0084] The recognition module 410 is used to input the remote sensing image to be detected into the trained recognition model to obtain the coordinates of the pixels corresponding to the ship target detection frame and multiple components of the ship, wherein the multiple components include the main gun, the vertical launch system, the chimney and the flight deck.

[0085] The fitting module 420 is used to perform weighted fitting on the coordinates of the pixels corresponding to the multiple components of the ship to obtain a center line corresponding to the ship.

[0086] The adjustment module 430 is used to adjust the ship target detection frame to a vertical direction by taking the angle between the target direction determined by the central straight line and the vertical direction as the direction angle.

[0087] The matching module 440 is used to match the ship target detection frame after posture adjustment with the template image in the template library to obtain the model and refined outline of the ship.

[0088] It should be understood that the implementation details and technical effects of the device embodiment are the same or similar to the implementation details and technical effects of the method embodiment, and will not be described in detail.

[0089] The specific embodiments described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A ship fine identification method, characterized in that: include: Input the remote sensing image to be detected into the trained recognition model to obtain the coordinates of the pixels corresponding to the ship target detection frame and multiple components of the ship, wherein the multiple components include the main gun, the vertical launch system, the chimney and the flight deck; Performing weighted fitting on the coordinates of pixels corresponding to the multiple components of the ship to obtain a center straight line corresponding to the ship; Taking the angle between the target direction determined by the central straight line and the vertical direction as the direction angle, the ship target detection frame is adjusted to a vertical direction; Match the ship target detection frame after posture adjustment with the template image in the template library to obtain the ship model and refined outline; The step of matching the posture-adjusted ship target detection frame with the template image in the template library to obtain the model and refined outline of the ship includes: Calculate the normalized square sum of the difference between the first gray value of each pixel in the ship target detection frame after the posture adjustment and the gray value of each pixel in the template image in the template library; Determine the template image corresponding to the minimum normalized square difference sum as the target template image of the ship; Fitting a refined profile of the ship through the target template image; The method further comprises: training the recognition model; The training of the recognition model comprises: Acquire historical remote sensing images containing ship targets; annotating a plurality of parts of the ship in the historical remote sensing image, the plurality of parts including a main gun, a vertical launch system, a chimney, and a flight deck; Construct a training image set based on the annotated historical remote sensing images; The training data set is input into a convolutional neural network for training to obtain the trained recognition model.

2. The ship fine identification method according to claim 1, characterized in that: The calculation of the normalized square sum of the difference between the first grayscale value of each pixel in the ship target detection frame after the posture adjustment and the grayscale value of each pixel in the template image in the template library includes: according to Calculate the normalized sum of squared differences R(x,y), where I(x,y) is the grayscale value of the pixel with coordinates (x,y) in the ship target detection frame, T(x',y') is the grayscale value of the pixel with coordinates (x',y') in the template image, and I(x+x',y+y') is the grayscale value of the pixel with coordinates (x+x',y+y') in the ship target detection frame.

3. The ship fine identification method according to claim 1, characterized in that: Before inputting the remote sensing image to be detected into the trained recognition model, the method further includes: The remote sensing image to be detected is subjected to image denoising, contrast enhancement and land-sea segmentation.

4. The ship fine identification method according to claim 1, characterized in that: The performing land-sea segmentation on the remote sensing image to be detected comprises: A grayscale histogram-based segmentation method is used to segment the land and sea areas of remote sensing images with image denoising and contrast enhancement. The edges of the segmented ocean images are expanded using morphological methods such as dilation, erosion, and opening and closing, and some land information and all ship information are segmented into the sea surface.

5. The ship fine identification method according to claim 1, characterized in that: Inputting the training data set into a convolutional neural network for training to obtain the trained recognition model comprises: Inputting the training picture set into the pre-trained region candidate network for training to obtain a trained pre-trained region candidate network; The training picture set is input into the convolutional neural network model, the region of interest is obtained using the trained pre-trained region candidate network, the weight parameters of the trained pre-trained region candidate network are fixed, the convolutional neural network model is repeatedly iterated, and the trained convolutional neural network model is obtained as the trained recognition model.

6. The ship fine identification method according to claim 5 is characterized in that: The convolutional neural network model includes a YOLO model or a Faster-RCNN model.

7. A ship fine identification device, characterized in that: include: A recognition module, used to input the remote sensing image to be detected into the trained recognition model to obtain the coordinates of the pixels corresponding to the ship target detection frame and multiple components of the ship, wherein the multiple components include the main gun, the vertical launch system, the chimney and the flight deck; A fitting module, used for performing weighted fitting on the coordinates of pixels corresponding to the multiple components of the ship to obtain a center straight line corresponding to the ship; An adjustment module is used to adjust the ship target detection frame to a vertical direction by taking the angle between the target direction determined by the central straight line and the vertical direction as the direction angle; The matching module is used to match the ship target detection frame after posture adjustment with the template image in the template library to obtain the ship model and refined outline; The step of matching the posture-adjusted ship target detection frame with the template image in the template library to obtain the model and refined outline of the ship includes: Calculate the normalized square sum of the difference between the first gray value of each pixel in the ship target detection frame after the posture adjustment and the gray value of each pixel in the template image in the template library; Determine the template image corresponding to the minimum normalized square difference sum as the target template image of the ship; Fitting a refined profile of the ship through the target template image; It also includes: training the recognition model; The training of the recognition model comprises: Acquire historical remote sensing images containing ship targets; annotating a plurality of parts of the ship in the historical remote sensing image, the plurality of parts including a main gun, a vertical launch system, a chimney, and a flight deck; Construct a training image set based on the annotated historical remote sensing images; The training data set is input into a convolutional neural network for training to obtain the trained recognition model.

Images