A ship image recognition method and system based on convolutional neural network

By combining YOLO-V3 and Faster R-CNN convolutional neural network methods, the problem of low ship recognition efficiency in traditional technology is solved, efficient recognition in complex backgrounds is achieved, and recognition rate is improved.

CN113850166BActive Publication Date: 2024-12-03ZNPL OCEAN DETECTION SYST ENG
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202111068434.1
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-09-13
Publication Date
2024-12-03
Estimated Expiration
2041-09-13

AI Technical Summary

Technical Problem

Traditional AI AI video image ship recognition technology has limited learning ability when processing natural data, requires professional knowledge to extract features, and has low recognition efficiency in complex contexts.

Method used

The method based on convolutional neural network is adopted, combined with YOLO-V3 one-stage algorithm for ship positioning and Faster R-CNN two-stage algorithm for fine feature extraction, and matched through the ship image database, reducing the impact of complex backgrounds and improving the recognition rate .

Benefits of technology

It effectively improves the recognition rate of ship features, reduces the calculation amount of refined recognition stage, and achieves efficient recognition in complex contexts.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113850166B_ABST
    Figure CN113850166B_ABST
Patent Text Reader

Abstract

The present invention relates to a ship image recognition method and system based on a convolutional neural network, and the method comprises the following steps: S1, inputting an original image collected by an optoelectronic video service; S2, calculating the probability of a ship being included in the image and the position coordinates of the ship in the image through a convolutional neural network one-stage algorithm; S3, outputting the probability that the image contains a ship and the position coordinates, cutting the ship image from the original image and performing normalization processing; S4, classifying and extracting features of the normalized ship image through a convolutional neural network two-stage algorithm; S5, searching the extracted features in a ship image database to find a ship with a feature matching degree greater than a certain ratio. The present invention uses two convolutional neural networks for ship positioning and recognition. Compared with single deep learning ship recognition, the influence of complex background is removed, the amount of calculation in the refined recognition stage is greatly reduced, and the recognition rate of ship features is effectively improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a ship image recognition method and system based on convolutional neural network. Background Art

[0002] Traditional AI video image ship recognition learning technology often uses raw forms to process natural data, and the learning ability of the model is greatly limited. Building a pattern recognition or machine learning system often requires considerable expertise to extract features from raw data (such as image pixel values) and convert them into an appropriate internal representation. Compared with traditional machine learning algorithms that extract features manually or rely on prior knowledge, deep learning has better adaptability and universal applicability. It can flexibly learn features autonomously driven by training data, and can use deep networks for feature description. Strategies such as weight sharing and layer-by-layer training greatly improve computing efficiency. Convolutional neural networks in deep learning have been successfully applied in fields such as face recognition and speech recognition. Using deep learning for ship recognition in natural scene images is of great significance to the construction of smart fishery administration, maritime traffic safety and other fields. Summary of the invention

[0003] The present invention aims at solving the technical problems existing in the prior art and provides a ship image recognition method and system based on convolutional neural network.

[0004] The technical solution of the present invention to solve the above technical problems is as follows:

[0005] In a first aspect, the present invention provides a ship image recognition method based on a convolutional neural network, wherein the convolutional neural network includes a first-stage algorithm consisting of a YOLO-V3 algorithm and a second-stage algorithm consisting of a Faster R-CNN, and comprises the following steps:

[0006] S1, the original image collected by the input optoelectronic video service;

[0007] S2, calculates the probability of the image containing a ship and the position coordinates of the ship in the image through a one-stage convolutional neural network algorithm;

[0008] S3, output the probability and position coordinates of the image containing the ship, cut the ship image from the original image and perform normalization processing;

[0009] S4, classification and feature extraction of normalized ship images by convolutional neural network two-stage algorithm;

[0010] S5, searching the extracted features in the ship image database to find ships with feature matching greater than a certain ratio.

[0011] Furthermore, before step 1, the process also includes building a ship image database, including:

[0012] The radar and AIS information are used to guide the camera to place the front, side and stern of the ship in the center of the video screen, and obtain the ship's AIS information and images of the ship in different directions;

[0013] Preprocess and annotate ship images and establish a ship image database.

[0014] Furthermore, the preprocessing of the ship image includes:

[0015] The ship is aligned to obtain a ship image with a correct position; the ship image is subjected to light compensation, grayscale transformation, histogram equalization and normalization to obtain a standardized ship image with consistent size and the same grayscale value range.

[0016] Furthermore, the preprocessing of the ship image also includes geometric correction, median filtering and sharpening processing of the ship image.

[0017] Furthermore, the annotation includes ship type, coordinate position, color and ship identity information.

[0018] Furthermore, the ship image database construction also includes increasing sample images in the ship image database by scaling, rotating, and changing the color of the images.

[0019] Furthermore, in the two-stage convolutional neural network algorithm Faster R-CNN, the detection targets are classified according to the ship's appearance characteristics, and an exclusive RPN network is configured for each type of ship. Each RPN network designs the scale and proportion of the anchor frame according to the corresponding size characteristics.

[0020] In a second aspect, the present invention provides a ship image recognition system based on a convolutional neural network, comprising:

[0021] An image input module, used to input the original image collected by the optoelectronic video service;

[0022] The positioning module calculates the probability of a ship being included in the image and the position coordinates of the ship in the image through a one-stage convolutional neural network algorithm;

[0023] The normalization module outputs the probability and position coordinates of the image containing the ship, cuts the ship image from the original image and performs normalization processing;

[0024] The classification and feature extraction module classifies and extracts features from normalized ship images through a two-stage convolutional neural network algorithm;

[0025] The retrieval module searches the extracted features in the ship image database to find ships with feature matching greater than a certain ratio.

[0026] According to a third aspect, an electronic device includes:

[0027] Memory for storing computer software programs;

[0028] The processor is used to read and execute the computer software program stored in the memory, thereby implementing the ship image recognition method based on convolutional neural network described in the first aspect of the present invention.

[0029] In a fourth aspect, a non-transitory computer-readable storage medium stores a computer software program for implementing a ship image recognition method based on a convolutional neural network as described in the first aspect of the present invention.

[0030] The beneficial effect of the present invention is that the present invention uses two convolutional neural networks to locate and identify ships. The basic network uses the YOLO V3 one-stage algorithm to locate ships based on the original image, and quickly calculates the probability of containing ships in the image, as well as the position coordinates of the ship in the image. The optimization network uses the Faster R-CNN two-stage algorithm to perform fine ship feature extraction in the ship image area selected by the basic network. By using the basic network to perform preliminary ship area positioning and the optimization network to perform fine ship feature recognition and extraction, compared with single deep learning ship recognition, the influence of complex background is eliminated, the amount of calculation in the refined recognition stage is greatly reduced, and the recognition rate of ship features is effectively improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 A schematic diagram of a process flow of a ship image recognition method based on a convolutional neural network provided by an embodiment of the present invention;

[0032] Figure 2 A structural schematic diagram of a ship image recognition system based on a convolutional neural network provided by an embodiment of the present invention;

[0033] Figure 3 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention;

[0034] Figure 4 A schematic diagram of the structure of a non-transitory computer-readable storage medium provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0035] The principles and features of the present invention are described below in conjunction with the accompanying drawings. The examples given are only used to explain the present invention and are not used to limit the scope of the present invention.

[0036] Deep learning allows multiple processing layers to form complex computational models, thereby automatically obtaining representations of data with multiple levels of abstraction. By using the BackPropagation algorithm, deep learning has the ability to discover hidden complex structures in large data sets. "Representation learning" can automatically discover the features to be detected from the original input data. Deep learning methods contain multiple layers, each of which completes a transformation (usually nonlinear) to represent a lower-level feature representation into a more abstract feature. As long as there are enough transformation levels, even very complex patterns can be automatically learned. For image classification tasks, neural networks will automatically remove irrelevant features, such as background color, object location, etc., but will automatically amplify useful features, such as shape. Images are often used as raw input in the form of pixel matrices, so the learning function of the first layer in the neural network is usually to detect the presence or absence of edges of specific directions and shapes, as well as the location of these edges in the image. The second layer often detects a specific layout of multiple edges, while ignoring small changes in edge location. The third layer can combine specific edge layouts into a part of the actual object. Subsequent layers will combine these parts to achieve object recognition, which is often done through fully connected layers. For deep learning, these features and levels do not need to be manually designed: they can all be obtained through a general learning process. Algorithm recognition effect requirements: For ships recorded in the ship sample data set, the on-site recognition rate is ≥ 50% and the false alarm rate is ≤ 20%.

[0037] The embodiment of the present invention uses an artificial intelligence-based video image ship recognition algorithm, relies on multiple sets of optoelectronic videos deployed on the smart ocean platform, and works with radar and AIS to identify ship targets. It trains models based on different features of the ship, such as type, size, color, and shape, accumulates a ship feature database, and performs ship feature recognition for subsequent illegal ship automatic identity confirmation, photography, comparison, early warning, and evidence collection.

[0038] Specifically, Figure 1 As shown, an embodiment of the present invention provides a ship image recognition method based on a convolutional neural network, wherein the convolutional neural network includes a one-stage algorithm consisting of a YOLO-V3 algorithm and a two-stage algorithm consisting of a Faster R-CNN, and includes the following steps:

[0039] S1, the original image collected by the input optoelectronic video service;

[0040] S2, calculates the probability of the image containing a ship and the position coordinates of the ship in the image through a one-stage convolutional neural network algorithm;

[0041] S3, output the probability and position coordinates of the image containing the ship, cut the ship image from the original image and perform normalization processing;

[0042] S4, classification and feature extraction of normalized ship images by convolutional neural network two-stage algorithm;

[0043] S5, searching the extracted features in the ship image database to find ships with feature matching greater than a certain ratio.

[0044] Before executing the above method, the convolutional neural network needs to be trained in advance. The training process is as follows:

[0045] Step 1: Ship image sample preparation.

[0046] 1. The radar AIS information guides the optoelectronics in different positions to place the front, side and stern of the ship in the center of the video screen. The video image analysis service pulls the video stream from the streaming service according to the optoelectronics specified by the optoelectronic guidance service, and saves the ship AIS information and ship pictures in different directions into a structured database.

[0047] 2. Image Preprocessing

[0048] Before using the algorithm for image recognition, good data preprocessing can quickly achieve twice the result with half the effort. Image preprocessing can not only make the original image conform to certain established rules for subsequent processing, but also help remove noise in the image, and can also help reduce the subsequent amount of calculations and accelerate convergence.

[0049] Commonly used image preprocessing operations include normalization, grayscale transformation, filtering transformation, and various morphological transformations. With the development of deep learning technology, some preprocessing methods have been integrated into deep learning models.

[0050] The preprocessing process adopted in this embodiment includes:

[0051] Ship alignment (obtaining an image of the ship in a correct position);

[0052] Light compensation, grayscale transformation, histogram equalization, and normalization of ship images (obtaining standardized ship images with consistent size and the same grayscale value range);

[0053] Geometric correction, median filtering (smoothing of the image to remove noise), and sharpening, etc.

[0054] 3. Sample library establishment

[0055] The preprocessed images are annotated and a ship image database is established.

[0056] The sample annotation database needs to include annotation information such as ship type, location coordinates, color, etc. Ships with identity information such as AIS numbers can be automatically associated and saved in the annotation database.

[0057] Since the number of images collected by different ships varies, some ships have a small number of samples, which can easily lead to overfitting during training. The number of sample images can be increased by scaling, rotating, changing the color of the image, etc.

[0058] Step 2: Ship image training. The ship image training uses the TensorFlow framework, and different algorithms can be configured and selected (mainly Two Stage (FasterRCNN) and One Stage (YOLO, SSD) algorithms) for training to obtain a ship feature recognition model.

[0059] (1) Faster R-CNN algorithm

[0060] Faster R-CNN replaces the traditional Selective Search method of extracting targets with network training, which greatly improves the detection and classification speed of the entire process. The following figure shows the basic structure of Faster R-CNN, which consists of the following four parts:

[0061] 1) Feature extraction: Use a series of convolution + pooling to extract the feature map from the original image;

[0062] 2) RPN (region proposal network): This part is a new structure proposed by Faster R-CNN. Its function is to obtain the approximate location of the target from the feature map through network training.

[0063] 3) Proposal Layer: Use the approximate position obtained by RPN to continue training and obtain a more accurate position;

[0064] 4) ROIPooling part: using the precise position obtained earlier, extract the target to be classified from the feature map and pool it into fixed-length data;

[0065] The training method of Faster R-CNN is to alternate between RPN training and R-CNN training, twice in total. During training, the network is initialized with the results of ImageNet pre-training, RPN is trained, R-CNN is trained with the obtained proposals, and then the network is initialized with the parameters trained by R-CNN, RPN is trained again, and finally the network is initialized with the parameters trained by RPN, and R-CNN is trained again to complete the entire training process.

[0066] The training process uses the following strategies to improve algorithm capabilities:

[0067] In terms of effect, the detection targets are classified according to their appearance features (for example, fishing boats, tugboats, cruise ships, sand dredging ships and other ships with large differences in appearance features are classified), and an exclusive RPN network is configured for each category. Each RPN designs the Ratio and Scale of the Anchor according to the corresponding size characteristics; different RPNs use feature maps of different layers as needed, and the design is more targeted.

[0068] In terms of effectiveness, a variety of sample enhancement methods are used to address the problem of uneven distribution of samples of various types, and OHEM and other methods are used to further adjust the sample distribution during training.

[0069] In terms of performance, the basic convolutional layers are shared between major categories to ensure that the detection time does not increase excessively.

[0070] In terms of scalability, for new types, ideally only a new RPN network needs to be added for separate iteration, which will not affect the effects of other types.

[0071] (2) YOLO-V3 algorithm

[0072] YOLOv3 uses the Darknet53 structure, which is a fully convolutional model.

[0073] Improvements of the YOLOv3 algorithm:

[0074] Feature extraction: YOLO v3 uses a 53-layer convolutional network, which is composed of residual units. According to experiments, this model performs better than ResNet-101, ResNet-152 and Darknet-19 in terms of the balance between classification accuracy and efficiency.

[0075] Bounding box prediction: YOLO v3 uses logistic regression to predict a score for each bounding box. If the prior bounding box overlaps the true box better than any other bounding box before, this value should be 1. If the prior bounding box is not the best, but does overlap the true object more than a certain threshold (here 0.5), then ignore this prediction. YOLO v3 only assigns one bounding box to each true object. If the prior bounding box does not match the true object, no coordinate or category prediction loss will be incurred, only object prediction loss will be incurred.

[0076] Category prediction: In order to achieve multi-label classification, the model no longer uses the softmax function as the final classifier, but uses logistic as the classifier and binary cross-entropy as the loss function.

[0077] Multi-scale prediction: YOLO v3 performs prediction tasks on feature maps of three different scales. Based on the feature map obtained by Darknet-53, the first feature map is obtained after 7 convolutions, and the first prediction is made on this feature map. Then, the output of the third-to-last convolution layer is obtained from back to front, and a convolution and a x2 upsampling are performed. The upsampled features are connected with the 43rd convolution features. After 7 convolutions, the second feature map is obtained, and the second prediction is made on this feature map. Then, the output of the third-to-last convolution layer is obtained from back to front, and a convolution and a x2 upsampling are performed. The upsampled features are connected with the 26th convolution features. After 7 convolutions, the third feature map is obtained, and the third prediction is made on this feature map.

[0078] Two convolutional neural networks are used for ship positioning and identification. The basic network uses the YOLO V3 one-stage algorithm to locate ships based on the original image, quickly calculating the probability of the image containing a ship and the position coordinates of the ship in the image. The optimized network uses the FasterR-CNN two-stage algorithm to perform fine ship feature extraction in the ship image area selected by the basic network. By using the basic network for preliminary ship area positioning and the optimized network for fine ship feature recognition and extraction, compared with single deep learning ship recognition, the influence of complex background is removed, the amount of calculation in the fine recognition stage is greatly reduced, and the recognition rate of ship features is effectively improved.

[0079] Figure 2 A structural diagram of a ship image recognition system based on a convolutional neural network provided by an embodiment of the present invention is shown in FIG. Figure 2 As shown, the identification system includes:

[0080] An image input module, used to input the original image collected by the optoelectronic video service;

[0081] The positioning module calculates the probability of a ship being included in the image and the position coordinates of the ship in the image through a one-stage convolutional neural network algorithm;

[0082] The normalization module outputs the probability and position coordinates of the image containing the ship, cuts the ship image from the original image and performs normalization processing;

[0083] The classification and feature extraction module classifies and extracts features from normalized ship images through a two-stage convolutional neural network algorithm;

[0084] The retrieval module searches the extracted features in the ship image database to find ships with feature matching greater than a certain ratio.

[0085] Figure 3 Schematic diagram of an electronic device provided by an embodiment of the present invention. Figure 3 As shown, an embodiment of the present invention provides an electronic device, including a memory 510, a processor 520, and a computer program 511 stored in the memory 510 and executable on the processor 520. When the processor 520 executes the computer program 511, the following steps are implemented:

[0086] S1, the original image collected by the input optoelectronic video service;

[0087] S2, calculates the probability of the image containing a ship and the position coordinates of the ship in the image through a one-stage convolutional neural network algorithm;

[0088] S3, output the probability and position coordinates of the image containing the ship, cut the ship image from the original image and perform normalization processing;

[0089] S4, classification and feature extraction of normalized ship images by convolutional neural network two-stage algorithm;

[0090] S5, searching the extracted features in the ship image database to find ships with feature matching greater than a certain ratio.

[0091] See also Figure 4 , Figure 4 A schematic diagram of an embodiment of a computer-readable storage medium provided in an embodiment of the present invention. Figure 4 As shown, this embodiment provides a computer-readable storage medium 600, on which a computer program 611 is stored. When the computer program 611 is executed by a processor, the following steps are implemented:

[0092] S1, the original image collected by the input optoelectronic video service;

[0093] S2, calculates the probability of the image containing a ship and the position coordinates of the ship in the image through a one-stage convolutional neural network algorithm;

[0094] S3, output the probability and position coordinates of the image containing the ship, cut the ship image from the original image and perform normalization processing;

[0095] S4, classification and feature extraction of normalized ship images by convolutional neural network two-stage algorithm;

[0096] S5, searching the extracted features in the ship image database to find ships with feature matching greater than a certain ratio.

[0097] It should be noted that in the above embodiments, the description of each embodiment has its own emphasis, and for parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0098] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0099] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded computer, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0100] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0101] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0102] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0103] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A ship image recognition method based on convolutional neural network, characterized in that: The convolutional neural network includes a first-stage algorithm consisting of a YOLO-V3 algorithm and a second-stage algorithm consisting of a Faster R-CNN algorithm. The following steps are involved: S1, the original image collected by the input optoelectronic video service; S2, calculates the probability of the image containing a ship and the position coordinates of the ship in the image through a one-stage convolutional neural network algorithm; S3, output the probability and position coordinates of the image containing the ship, cut the ship image from the original image and perform normalization processing; S4, classification and feature extraction of normalized ship images by convolutional neural network two-stage algorithm; S5, searching the extracted features in the ship image database to find ships with feature matching greater than a certain ratio.

2. The method according to claim 1, characterized in that Before step 1, the ship image database is also constructed, including: The radar and AIS information are used to guide the camera to place the front, side and stern of the ship in the center of the video screen, and obtain the ship's AIS information and images of the ship in different directions; Preprocess and annotate ship images and establish a ship image database.

3. The method according to claim 2, characterized in that The preprocessing of the ship image comprises: The ship is aligned to obtain a ship image with a correct position; the ship image is subjected to light compensation, grayscale transformation, histogram equalization and normalization to obtain a standardized ship image with consistent size and the same grayscale value range.

4. The method according to claim 3, characterized in that The preprocessing of the ship image also includes geometric correction, median filtering and sharpening processing of the ship image.

5. The method according to claim 2, characterized in that: The marking includes ship type, coordinate position, color and ship identity information.

6. The method according to claim 2, characterized in that The ship image database construction also includes increasing sample images in the ship image database by scaling, rotating, and changing the color of the images.

7. The method according to claim 1, characterized in that In the two-stage convolutional neural network algorithm Faster R-CNN, the detection targets are classified according to the ship's appearance features, and a dedicated RPN network is configured for each type of ship. Each RPN network designs the scale and proportion of the anchor frame according to the corresponding size characteristics.

8. A ship image recognition system based on convolutional neural network, characterized in that: The convolutional neural network includes a first-stage algorithm consisting of a YOLO-V3 algorithm and a second-stage algorithm consisting of a Faster R-CNN algorithm. The ship image recognition system includes: An image input module, used to input the original image collected by the optoelectronic video service; The positioning module calculates the probability of a ship being included in the image and the position coordinates of the ship in the image through a one-stage convolutional neural network algorithm; The normalization module outputs the probability and position coordinates of the image containing the ship, cuts the ship image from the original image and performs normalization processing; The classification and feature extraction module classifies and extracts features from normalized ship images through a two-stage convolutional neural network algorithm; The retrieval module searches the extracted features in the ship image database to find ships with feature matching greater than a certain ratio.

9. An electronic device, characterized in that: include: Memory for storing computer software programs; A processor, used to read and execute the computer software program stored in the memory, thereby implementing the ship image recognition method based on convolutional neural network as described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium, characterized in that: The storage medium stores a computer software program for implementing a ship image recognition method based on a convolutional neural network as described in any one of claims 1 to 7.