A remote sensing image marine ship recognition system and method based on improved YOLOv4 algorithm
By improving the YOLOv4 algorithm, combining high contrast retention method and K-means clustering, the problems of low recognition accuracy and slow calculation speed in marine ship target recognition are solved, and efficient recognition in harsh environments are achieved.
Patent Information
- Application Number
- CN202111381889.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-11-22
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2041-11-22
AI Technical Summary
The prior art has problems in the recognition of offshore ship targets with low recognition accuracy, slow calculation speed, and susceptible to environmental interference, especially in harsh environments and remote sensing images.
The improved YOLOv4 algorithm is adopted to enhance image contrast through the high contrast retention method, the anchor box is obtained using the K-means clustering algorithm, the network structure module is added, and the IOU threshold is adjusted to improve the detection accuracy of small objects.
It realizes rapid detection and automatic identification of surface ship targets, improves recognition probability and accuracy, and enhances recognition capabilities in harsh environments.
Smart Images

Figure CN113920436B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of ship target detection and relates to a remote sensing image marine ship recognition system and a method thereof based on an improved YOLOv4 algorithm. Background Art
[0002] The research on the recognition of surface ship targets is a key technology in both military and civilian applications, and has engineering application value. In the military, it can be used for real-time automatic reconnaissance, laying the foundation for threat estimation and situation assessment in the sea battlefield. At present, the intelligence level of optoelectronic reconnaissance systems is far lower than that of civilian electronic equipment. In the process of optoelectronic reconnaissance, operators are still required to manually interpret ship targets based on displayed images. The interpretation speed is slow and easily affected by subjective factors. The use of target detection and recognition technology can automatically analyze and process a large amount of accumulated image data. Real-time target detection and recognition technology can be applied to the reconnaissance of surface targets for real-time monitoring and alarm prompts, which greatly reduces the burden of image processing personnel after unloading data and operators during the execution of reconnaissance tasks, and lays a technical foundation for the realization of automatic reconnaissance. Therefore, the realization of intelligent detection and recognition of ship targets is of great significance to improving the automation level of military equipment. In the civilian field, intelligent detection and recognition of ship targets can be used for dynamic port monitoring, maritime traffic management, and maritime rescue. It can combat illegal smuggling and other illegal and criminal activities through the detection and recognition of ships.
[0003] The Chinese patent with publication number CN201810457334.X provides a SAR radar marine ship target recognition system. The method uses SAR radar to monitor the sea area in real time, and the image data obtained by the radar is stored in the database. The image is preprocessed and features are extracted by the host computer, and finally the target is identified; another example is the Chinese patent application with publication number CN201911156854.8, which discloses a method for automatic detection and recognition of marine ship targets. The method collects images through a visible light camera, uses the Faster-RCNN algorithm for training to obtain a target detector and detects the target. For example, the Chinese patent with publication number CN109255286B discloses a method for rapid optical detection and recognition of drones based on a deep learning network framework. The method trains the drone data set through the YOLO network, and then achieves the purpose of rapid detection.
[0004] In the Chinese patent application CN201810457334.X, the SAR radar screen display is not intuitive enough, and the surrounding environment cannot be directly judged, or in cold weather, it is affected by wind and waves, and it is difficult for ordinary monitoring equipment to identify key targets in harsh environments; in the Chinese patent CN201911156854.8, the two-stage detection model represented by Faster-RCNN needs to generate alternative boxes first and then identify and detect them respectively. Its calculation speed is slow, which is not conducive to practical applications. In addition, in ocean images, satellite images and camera images are often easily troubled by problems such as foggy weather, motion blur, and camera lens pollution, resulting in blurred shooting results, which greatly interferes with target recognition and has a lower target recognition accuracy. In the Chinese patent application CN109255286B, although the YOLO series of algorithms have achieved good results in real-time target detection and recognition, the YOLO series of algorithms have poor results in recognizing remote sensing images. Remote sensing images are different from natural images. In remote sensing images, due to the long shooting distance, most ship targets are small in size and occupy very few pixels in the entire image. If the IOU value (IOU=0.5) used to identify natural images is used to identify remote sensing images, missed detections may occur. Summary of the invention
[0005] The purpose of the present invention is to overcome the defects of the prior art and provide a remote sensing image marine ship recognition system and method based on the improved YOLOv4 algorithm, which can improve the contrast of the original image, enhance the image details, and eliminate noise; realize the rapid detection and automatic recognition of sea surface ship targets, and improve the probability and accuracy of ship recognition.
[0006] In order to solve the above technical problems, the present invention adopts the following technical solutions.
[0007] A remote sensing image marine ship recognition system based on an improved YOLOv4 algorithm of the present invention comprises:
[0008] The acquisition unit is used to collect visible light and infrared ship target images taken by optoelectronic reconnaissance equipment in the past when performing tasks, or satellite remote sensing images of sea scenes collected from Google Earth, including Google Earth App, satellite image collections, and ship collections photographed by optoelectronic equipment;
[0009] The calibration unit is used to use the data labeling software labelimg to label the preprocessed images, including marking the specific position (x, y) of the target in the image and the width and height (w, h) of the target;
[0010] The pre-processing unit is used to separate the ships in the remote sensing images from the islands, buildings and suspended objects next to them to eliminate the image noise, including the Python processing framework;
[0011] The clustering unit is used to obtain the estimated value of the anchor box of the YOLO algorithm, including clustering the annotation box through the K-means clustering algorithm, selecting k=1 to 18 to cluster the samples in the data set, and obtaining the relationship between k and IOU;
[0012] The generation unit is used to generate the YOLOv4 framework, including the target feature extraction network, SPP network, feature fusion network and target prediction network;
[0013] A detection unit, used to generate a detection frame of YOLOv4, including inputting the pictures in the picture set into the improved YOLOv4 network structure, generating a feature map, and obtaining position information, category information and object confidence of the prior frame corresponding to the feature map;
[0014] The screening unit is used to set the threshold of the candidate box and finally obtain the prediction box. In order to identify small targets such as ships in remote sensing images, a smaller prediction box must be used for recognition, so the IOU value must be reduced;
[0015] The training unit is used to calculate the target confidence loss function, the target position loss function, and the target type loss function, and use the feedback neural network to update the weights until the total value of the above three loss functions is minimized, thereby obtaining a trained improved YOLOv4 neural network;
[0016] The detection unit is used to input the images in the test subset into the trained improved YOLOv4 network structure to obtain the category of the target, the specific position of the target in the image, and the width and height of the target, thereby completing target detection.
[0017] A method for identifying marine ships in remote sensing images based on an improved YOLOv4 algorithm of the present invention comprises the following steps:
[0018] Step 1: Obtain a satellite remote sensing ship target sample library, use the data labeling software labelimg to annotate the images, obtain a data set, and allocate it in a training subset and a test subset ratio of 8:2;
[0019] Step 2: Preprocess the training subset using high contrast retention method;
[0020] Step 3: Build the YOLOv4 network framework and conduct training;
[0021] Step 4: Use non-maximum suppression to obtain the prediction box and improve the IOU threshold;
[0022] Step 5: Verify network feasibility.
[0023] Furthermore, the process of step one includes:
[0024] Step 1.1. Build a ship target sample library. The sources of the sample library include the following:
[0025] 1) Obtain existing ship target images on the Internet through web crawlers;
[0026] 2) Collect visible light and infrared ship target images taken by optoelectronic reconnaissance equipment during previous missions;
[0027] 3) Satellite remote sensing images of ocean scenes collected from Google Earth;
[0028] 4) Accumulate the newly detected ship target images during the subsequent mission execution of the optoelectronic reconnaissance equipment to maintain and expand the sample library;
[0029] Step 1.2. Perform target calibration on the images in the ship target sample library to make a data set; use the data labeling software labelimg to draw a target marking frame for the target area in each image; the position information of the target in the image refers to the width and height information of the target marking frame, and the coordinate information of the diagonal points of the target marking frame in the image coordinate system; set the label of each image as: ship, aircraft; the ratio of its training subset and test subset is 8:2; use the data labeling software labelimg to label the ships in the training set in the data set, generate an xml file, and use the python script voc_yolo to convert the xml file format to txt file format.
[0030] Furthermore, the process of step 2 includes:
[0031] The infrared image preprocessing method is used to retain the junction of the two parts with large color and light-dark contrast in the image; first, Gaussian filtering is used to separate the ship from the surrounding islands and buildings to enhance the contrast; then the following operations are used to sharpen the image to achieve the effect of eliminating image noise. Its expression is as follows:
[0032] scr=img-λ×guas(img) (1)
[0033] In formula (6), scr represents the processed result, img represents the original image, λ represents the gain, and guas represents the Gaussian filter used. In order to improve the infrared contrast and reduce the introduction of noise, λ=3.
[0034] The boundary between two parts with large color and light-dark contrast in the image is retained, making the ship target features more prominent.
[0035] Furthermore, the process of step three includes:
[0036] In the original network, the detect_image function is first used to scale the input image to a size of 416x416; then the image is processed by referencing three functions: convolution function, batch normalization function and activation function mish, which are respectively for feature extraction of image convolution, batch normalization of extracted features, and selection of appropriate activation function mish to prevent gradient disappearance, so that the number of channels changes from 3 to 32, speeding up the feature extraction process; then the input image undergoes multiple convolution and pooling operations in the network, and can generate three feature maps of different scales: a 52×52 scale feature layer upsampled 8 times the number of 256 channels, a 26×26 scale feature layer upsampled 16 times the number of 512 channels, and a 13×13 scale feature layer upsampled 32 times the number of 1024 channels; too much upsampling convolution pooling will lose some detailed features such as image position and texture when extracting features; therefore, the improved YOLOv4 network structure is adopted to generate more scale feature layers; some detailed features are extracted in the feature layer with less convolution pooling, thereby improving the accuracy of ship target detection.
[0037] Furthermore, the structure of the improved YOLOv4 network includes:
[0038] The improved YOLOv4 network adds two layers on the basis of the original feature pyramid to enhance the feature extraction function; the improved YOLOv4 network adds 4 data fusion modules, 2 upsampling modules, and 2 downsampling modules to the original path aggregation network; the improved YOLOv4 network changes the original 3 scale feature layers to 5 scale feature layers, namely 208*208 scale feature layer, 104*104 scale feature layer, 52*52 scale feature layer, 26*26 scale feature layer and 13*13 scale feature layer.
[0039] Furthermore, the process of step 4 includes:
[0040] The K-means clustering algorithm is used to obtain the anchor box size that meets the scale characteristics of the target to be detected in the infrared image. That is, the number of prior frames is set to 9, and the width and height of each prior frame are (17.4, 30.0), (26.6, 73.2), (49.0, 134.8), (50.8, 46.6), (89.9, 89.9), (97.3, 209.7), (176.4, 294.3), (204.7, 152.4), (348.4, 331.7); In order to obtain the optimal anchor box, the maximum IOU value is taken as a reference, and non-maximum suppression is used to remove redundant anchor boxes;
[0041] Then, in order to identify small targets such as ships in remote sensing images, a smaller prediction box must be used for recognition, so the IOU value must be reduced; however, the IOU value is proportional to the target detection confidence. If the confidence is too low, serious missed detection will occur; in order to improve the accuracy of small target detection, the IOU is reduced to 0.3 here.
[0042] Furthermore, the process of step five includes:
[0043] Import the preprocessed training set into Figure 6 In the improved YOLOv4 network shown, the weights are continuously updated until the loss function is minimized to obtain a trained network, where the loss function is as follows:
[0044] L=λ1L conf (o,c)+λ2L cla (O,C)+λ3L loc (l,g) (2)
[0045] In formula (2), the loss function consists of three parts: target confidence loss L conf (o,c), target position loss L loc (l,g), target category loss L cla (O, C), λ1, λ2, and λ3 are balance coefficients. λ3 is greater than 1, which increases the proportion of the target loss function and prevents excessive influence from external non-target objects;
[0046]
[0047] In formula (3), the target confidence loss L conf (o,c) uses binary cross entropy loss, where o i ∈{0,1}, indicates whether the target actually exists in the predicted target bounding box i, 0 means no target exists, 1 means yes target exists; Represents the Sigmoid probability of predicting whether there is a target in the target rectangular box i;
[0048]
[0049] In formula (4), the target position loss L loc (l,g) uses the square sum of the deviations between the true value and the predicted value, where i represents the coordinate offset of the predicted rectangular box, and g represents the coordinate offset between the matched GTbox and the default box;
[0050]
[0051] In formula (5), the target category loss L cla (O,C) uses binary cross entropy loss, where O i∈{0,1} indicates whether the target actually exists in the predicted target bounding box i, 0 means no target exists, 1 means yes target exists; Represents the Sigmoid probability of predicting whether there is a target in the target rectangular box i;
[0052] Then, the images in the test subset are input into the trained improved YOLOv4 network structure to obtain the category of the target, the specific position of the target in the image, and the width and height of the target; if the accuracy is high, the target detection is completed; if a large number of missed detections or false detections occur, return to step three to continue training until a network with higher detection accuracy is obtained.
[0053] Compared with the prior art, the present invention has the following advantages and beneficial effects:
[0054] 1. This paper takes satellite remote sensing ship images and aerial reconnaissance ship images as research objects, and proposes a satellite remote sensing image maritime ship recognition technology based on the improved YOLOv4 algorithm. The goal is to achieve rapid detection and automatic recognition of surface ship targets, and improve the probability and accuracy of ship recognition.
[0055] 2. Based on the YOLOv4 network algorithm, this invention proposes an improvement for the infrared image ship target detection task. The high contrast retention method is used to retain the junction of the two parts with large color and light-dark contrast in the image. First, the original image is Gaussian blurred, and then the Gaussian blurred data is removed with the original image data. The contrast of the original image is improved, the image details are enhanced, and the salt and pepper noise is eliminated, so that the infrared image ship target is highlighted.
[0056] 3. The present invention adds 4 data fusion modules, 2 upsampling modules, and 2 downsampling modules to the original YOLOv4 path aggregation network. Furthermore, the improved YOLOv4 network is changed from the original 3 scale feature layers to 5 scale feature layers, and the feature layer scales of 104×104 and 208*208 are added. The algorithm network structure is adjusted in combination with the attention mechanism idea, and then the anchor box size that meets the scale characteristics of the target to be detected in the infrared image is obtained through the K-means clustering algorithm, so as to improve the algorithm's detection performance for the infrared image hull.
[0057] 4. In view of the characteristics of remote sensing images, most ship targets are small in size due to the long shooting distance, and the small proportion of ship target images, the present invention proposes to reduce the IOU threshold to increase the detection accuracy of small targets. However, the IOU value is proportional to the target detection confidence. If the IOU is too small, serious missed detection will occur and the recall rate will be reduced. Therefore, reducing the IOU threshold to 0.3 can improve the detection accuracy of ship targets while having a small impact on the recall rate. BRIEF DESCRIPTION OF THE DRAWINGS
[0058] Figure 1It is a method flow chart of an embodiment of the present invention.
[0059] Figure 2a , Figure 2b All of them are anchor point frame calibration pictures according to an embodiment of the present invention, wherein: Figure 2a The anchor box marks 3 ship targets. Figure 2b 2 ship targets marked.
[0060] Figure 3 The present invention is a flowchart of ship target preprocessing according to an embodiment of the present invention.
[0061] Figure 4a , Figure 4b All of them are ship target preprocessing result diagrams of an embodiment of the present invention. Among them, Figure 4a It's the original picture. Figure 4b This is the processed picture.
[0062] Figure 5 It is a YOLOv4 network structure diagram of the prior art.
[0063] Figure 6 It is a diagram of an improved YOLOv4 network structure according to an embodiment of the present invention.
[0064] Figure 7 The figure is a relationship diagram between the number of anchor boxes and the IOU threshold value in an embodiment.
[0065] Figure 8a This is a YOLOv4 training result diagram of an embodiment of the present invention; Figure 8b This is an improved YOLOv4 training result diagram of an embodiment of the present invention.
[0066] Fig. 9 It is a YOLOv4 training process diagram of an embodiment of the present invention.
[0067] Fig.10 It is a diagram of the YOLO algorithm training process of an embodiment of the present invention.
[0068] Fig.11 4 is a relationship diagram between accuracy and recall rate of an embodiment of the present invention.
[0069] Fig.12a , Fig.12d , Figure 12b Figure 12e Fig.12c , and FIG. 12f are both test result diagrams of an embodiment of the present invention. Fig.12a , Fig.12d It's the original picture. Figure 12b Figure 12e is the YOLOv4 detection result diagram. Fig.12c,Figure 12f is the improved YOLOv4 result graph. DETAILED DESCRIPTION
[0070] The present invention is further described in detail below with reference to the accompanying drawings.
[0071] The present invention provides a remote sensing image marine ship recognition system based on an improved YOLOv4 algorithm. Figure 1 As shown, including:
[0072] The acquisition unit is used to collect visible light and infrared ship target images taken by optoelectronic reconnaissance equipment in the past when performing tasks, or satellite remote sensing images of sea scenes collected from Google Earth, including Google Earth App, satellite image collections, and ship collections photographed by optoelectronic equipment;
[0073] Calibration unit, such as Figure 2a , Figure 2b As shown, the data labeling software labelimg is used to label the preprocessed images, including marking the specific position (x, y) of the target in the image and the width and height (w, h) of the target;
[0074] Preprocessing units, such as Figure 3 and Figure 4a , Figure 4b As shown, it is used to separate the ships in the remote sensing images from the islands, buildings and suspended objects next to them, so as to eliminate the image noise, including Python and other processing frameworks;
[0075] The clustering unit is used to obtain the estimated value of the anchor box of the YOLO algorithm, including clustering the annotation box through the K-means clustering algorithm, selecting k=1~18 to cluster the samples in the data set, and obtaining the relationship between k and IOU (Intersection over Union) as follows Figure 7 As shown. As the k value increases, the change of the objective function becomes larger and more stable. When the k value is greater than 9, the curve begins to become smooth, so selecting 9 anchor boxes can not only accelerate the convergence of the loss function, but also reduce the error caused by the candidate box. In the experiment of the present invention, the anchor boxes corresponding to the ship target sample set are (17.4, 30.0), (26.6, 73.2), (49.0, 134.8), (50.8, 46.6), (89.9, 89.9), (97.3, 209.7), (176.4, 294.3), (204.7, 152.4), (348.4, 331.7);
[0076] The generation unit is used to generate the YOLOv4 framework, such as Figure 6As shown, it includes target feature extraction network, SPP network, feature fusion network and target prediction network.
[0077] A detection unit, used to generate a detection frame of YOLOv4. The detection unit includes inputting the images in the image set into the improved YOLOv4 network structure, generating a feature map, and obtaining the location information, category information and object confidence of the prior frame corresponding to the feature map;
[0078] The screening unit is used to set the threshold of the candidate box and finally obtain the prediction box. In order to identify small targets such as ships in remote sensing images, a smaller prediction box must be used for recognition, so the IOU value must be reduced. However, the IOU value is proportional to the confidence of target detection. If the confidence is too low, serious missed detection will occur. In order to improve the accuracy of small target detection, the IOU is set to 0.3 in this practice.
[0079] The training unit is used to calculate the target confidence loss function, target position loss function, and target type loss function, and use the feedback neural network to update the weights until the total value of the above three loss functions is minimized. In this way, the trained improved YOLOv4 neural network is obtained. Figure 8a , Figure 8b As shown in the training result diagram, the training samples for this training are 12731 satellite remote sensing images. These samples come from the CAESAR and RISD datasets and are divided into 50 groups for training. Figure 8a is the original YOLOv4 training result, Figure 8b The red curve is the training set loss function, and the orange curve is the test set loss function. When the training results are about 10 batches, the loss function is close to the minimum, and Figure 8b Compare Figure 8a The validation loss function is 0.3 lower, and the training and testing results are better.
[0080] The detection unit is used to input the images in the test subset into the trained improved YOLOv4 network structure to obtain the category of the target, the specific position of the target in the image, and the width and height of the target, thereby completing target detection.
[0081] In addition, the present invention provides a method for identifying marine ships in remote sensing images based on an improved YOLOv4 algorithm, comprising the following steps:
[0082] Step 1: Obtain a satellite remote sensing ship target sample library, use the data labeling software labelimg to label the images, obtain a data set, and allocate it in a training subset and test subset ratio of 8:2;
[0083] Step 2: Preprocess the training subset using high contrast retention method;
[0084] Step 3: Build the YOLOv4 network framework and conduct training;
[0085] Step 4: Use non-maximum suppression to obtain the prediction box and improve the IOU threshold;
[0086] Step 5: Verify network feasibility.
[0087] The specific process of step 1 is: obtain a sample library with ship targets. At present, there are very few pictures of ships taken from an aerial perspective, so it is necessary to build a sample library of ship targets. The basis for ship target detection and recognition is the construction of a sample library. At present, the sources of sample libraries are as follows:
[0088] 1) Obtain existing ship target images on the Internet through web crawlers;
[0089] 2) Collect visible light and infrared ship target images taken by optoelectronic reconnaissance equipment during previous missions;
[0090] 3) Satellite remote sensing images of ocean scenes collected from Google Earth;
[0091] 4) Accumulate the newly detected ship target images during the subsequent mission execution of the optoelectronic reconnaissance equipment to maintain and expand the sample library.
[0092] Furthermore, the images of the ship target sample library are calibrated to produce a data set. The data labeling software labelimg is used to draw a target marking frame for the target area in each image. The position information of the target in the image refers to the width and height information of the target marking frame, and the coordinate information of the diagonal points of the target marking frame in the image coordinate system. Optionally, the image set contains a total of 12620 frames of images, and the labels can be defined as: ships, aircraft, etc.; the training subset and the test subset have a ratio of 8:2; the data labeling software labelimg is used to annotate the ships in the training set in the data set, and an xml file is generated, and the python script voc_yolo is used to convert the xml file format to the txt file format.
[0093] The specific process of step 2 is as follows: The present invention provides an infrared image preprocessing method. The method mainly retains the intersection of two parts with large color and light-dark contrast in the image, and the effect is as follows Figure 3 As shown. First, Gaussian filtering is used to separate the ship from the surrounding islands and buildings to enhance the contrast; then the following operation is used to sharpen the image to eliminate image noise. Its expression is as follows:
[0094] scr=img-λ×guas(img) (6)
[0095] In formula (6), scr represents the processed result, img represents the original image, λ represents the gain, and guas represents the Gaussian filter used. In order to improve the infrared contrast and reduce the introduction of noise, λ=3.
[0096] The results are as follows Figure 4a , Figure 4b The preprocessing result diagram shown in Figure 4a is the original image, Figure 4b This is the image processed by the high contrast retention method. The junction of the two parts with large color and light-dark contrast in the image is retained, making the ship target features more prominent.
[0097] The specific process of step three is as follows: Figure 5 As shown, in the original network, the detect_image function is first used to scale the input image to a size of 416x416; then the image is preprocessed by referencing three functions, namely, convolution function, batch normalization function and activation function mish, which are respectively for feature extraction of image convolution, batch normalization of extracted features, and selection of appropriate activation function mish to prevent gradient disappearance, so that the number of channels changes from 3 to 32, and the feature extraction progress is accelerated; then the input image undergoes multiple convolution and pooling operations in the network, and three feature maps of different scales can be generated: 52×52 scale feature layer upsampled 8 times the number of 256 channels, 26×26 scale feature layer upsampled 16 times the number of 512 channels, and 13×13 scale feature layer upsampled 32 times the number of 1024 channels. Excessive upsampling convolution pooling will lose some image position, texture and other detail features when extracting features. Therefore, the present invention provides an improved YOLOv4 network structure, which generates a more scale feature layer. Some detailed features are extracted in the feature layer with less convolution pooling, thereby improving the accuracy of ship target detection.
[0098] The present invention provides an improved YOLOv4 network, which adds two layers to the original feature pyramid to enhance the feature extraction function such as Figure 5 As shown in the figure, the improved YOLOv4 network is as follows: 4 data fusion modules, 2 upsampling modules, and 2 downsampling modules are added to the original path aggregation network. In addition, the improved YOLOv4 network has 5 scale feature layers instead of the original 3 scale feature layers, namely 208*208 scale feature layer, 104*104 scale feature layer, 52*52 scale feature layer, 26*26 scale feature layer, and 13*13 scale feature layer.
[0099] The specific process of step 4 is: obtain the original value of the IOU value through clustering. Because the proportion of ship target images in remote sensing images is lower than that of ordinary images, the original value needs to be reduced for remote sensing image detection. If the IOU value is too low, missed detection will occur, so an IOU value suitable for detecting remote sensing ship targets should be obtained.
[0100] That is, the anchor box size that meets the scale characteristics of the target to be detected in the infrared image is obtained through the K-means clustering algorithm, that is, the number of the priori boxes is 9, and the width and height of each priori box are (17.4, 30.0), (26.6, 73.2), (49.0, 134.8), (50.8, 46.6), (89.9, 89.9), (97.3, 209.7), (176.4, 294.3), (204.7, 152.4), (348.4, 331.7). In order to obtain the most anchor box, non-maximum suppression is used to remove redundant anchor boxes.
[0101] Furthermore, in order to identify small targets such as ships in remote sensing images, a smaller prediction box must be used for recognition, so the IOU value must be reduced. However, the IOU value is proportional to the confidence of target detection. If the confidence is too low, serious missed detection will occur. In order to improve the accuracy of small target detection, the IOU is set to 0.3 in this practice.
[0102] The specific process of step 5 is: import the preprocessed training set into Figure 6 In the improved YOLOv4 network shown, the weights are continuously updated until the loss function is minimized to obtain a trained network, where the loss function is as follows:
[0103] L=λ1L conf (o,c)+λ2L cla (O,C)+λ3L loc (l,g) (7)
[0104] In formula (7), the loss function consists of three parts: target confidence loss L conf (o,c), target position loss L loc (l,g), target category loss L cla (O, C), λ1, λ2, and λ3 are balance coefficients. λ3 is greater than 1, which increases the proportion of the target loss function and prevents excessive influence from external non-target objects.
[0105]
[0106] In formula (8), the target confidence loss L conf (o,c) uses binary cross entropy loss; where o i∈{0,1} indicates whether there is a real target in the predicted target bounding box i, 0 means no target, and 1 means yes target. Represents the Sigmoid probability of predicting whether there is a target in the target rectangle i.
[0107]
[0108] In formula (9), the target position loss L loc (l,g) uses the squared sum of the deviations between the true value and the predicted value, where i represents the coordinate offset of the predicted rectangular box, and g represents the coordinate offset between the matching GTbox and the default box.
[0109]
[0110] In formula (10), the target category loss L cla (O,C) uses binary cross entropy loss; where O i ∈{0,1}, indicating whether there is a real target in the predicted target bounding box i, 0 means no target, 1 means yes target. Represents the Sigmoid probability of predicting whether there is a target in the target rectangular box i;
[0111] Further, if Fig. 9 As shown in the figure, the images in the test subset are input into the trained improved YOLOv4 network structure to obtain the category of the target, the specific position of the target in the image, and the width and height of the target. If the accuracy is high, the target detection is completed; if a large number of missed detections or false detections occur, return to step 3 to continue training until a network with high detection accuracy is obtained. The detection accuracy mPA in target detection is shown in the following table.
[0112] Actual situation Prediction is true Prediction is false Recall real TP (True) FN(True or False) Fake FP (False True) TN (false) Accuracy
[0113] The accuracy rate indicates how many of the samples predicted to be true are actually true samples. The expression is as follows
[0114]
[0115] The recall rate indicates how many true samples in the data set are predicted, and the expression is as follows
[0116]
[0117] Specific results such as Fig.11 , Fig.11 The relationship between precision and recall is shown in Figure 2. The mAP value of the improved YOLO algorithm is 0.67, and that of the original YOLO algorithm is 0.64.
[0118] Test results such as Fig.12a , Fig.12d , Figure 12b Figure 12e Fig.12c , as shown in Figure 12f, where Figure 12b Figure 12e is the recognition result after the original YOLOv4 network training; Fig.12c Figure 12e is the recognition result after the improved YOLOv4 network training; Fig.12c The recognition accuracy is slightly higher than that of Figure 12e, and Fig.12c Figure 12f Figure 12b , the ship that was missed in Figure 12e is detected.
[0119] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be equivalent replacement methods and are included in the protection scope of the present invention.
Claims
1. A remote sensing image marine ship recognition system based on improved YOLOv4 algorithm, characterized in that: include: An acquisition unit is used to collect visible light and infrared ship target images taken by optoelectronic reconnaissance equipment in the past when performing tasks, or to collect satellite remote sensing images of sea scenes from Google Earth, including satellite image collections and ship collections photographed by optoelectronic equipment; The calibration unit is used to use the data labeling software labelimg to label the preprocessed images, including marking the specific position (x, y) of the target in the image and the width and height (w, h) of the target; The pre-processing unit is used to separate the ships in the remote sensing images from the islands, buildings and suspended objects next to them to eliminate the image noise, including the Python processing framework; The clustering unit is used to obtain the estimated value of the anchor box of the YOLO algorithm, including clustering the annotation box through the K-means clustering algorithm, selecting k=1 to 18 to cluster the samples in the data set, and obtaining the relationship between k and IOU; The generation unit is used to generate the YOLOv4 framework, including the target feature extraction network, SPP network, feature fusion network and target prediction network; A detection unit, used to generate a detection frame of YOLOv4, including inputting the pictures in the picture set into the improved YOLOv4 network structure, generating a feature map, and obtaining position information, category information and object confidence of the prior frame corresponding to the feature map; The screening unit is used to set the threshold of the candidate box and finally obtain the prediction box. In order to identify small targets such as ships in remote sensing images, a smaller prediction box must be used for recognition, so the IOU value must be reduced; The training unit is used to calculate the target confidence loss function, the target position loss function, and the target type loss function, and use the feedback neural network to update the weights until the total value of the above three loss functions is minimized, thereby obtaining a trained improved YOLOv4 neural network; The detection unit is used to input the images in the test subset into the trained improved YOLOv4 network structure to obtain the category of the target, the specific position of the target in the image, and the width and height of the target, thereby completing target detection; The structure of the improved YOLOv4 network includes: The improved YOLOv4 network adds two layers to the original feature pyramid to enhance the feature extraction function; The improved YOLOv4 network adds 4 data fusion modules, 2 upsampling modules, and 2 downsampling modules to the original path aggregation network; the improved YOLOv4 network changes the original 3 scale feature layers to 5 scale feature layers, namely 208*208 scale feature layer, 104*104 scale feature layer, 52*52 scale feature layer, 26*26 scale feature layer and 13*13 scale feature layer.
2. A method for identifying marine ships in remote sensing images based on an improved YOLOv4 algorithm, characterized in that: The following steps are involved: Step 1: Obtain a satellite remote sensing ship target sample library, use the data labeling software labelimg to annotate the images, obtain a data set, and allocate it in a training subset and a test subset ratio of 8:2; Step 2: Preprocess the training subset using high contrast retention method; Step 3: Build the YOLOv4 network framework and conduct training; Step 4: Use non-maximum suppression to obtain the prediction box and improve the IOU threshold; Step 5: Verify network feasibility; The process of step three includes: In the original network, the detect_image function is first used to scale the input image to a size of 416x416; then the image is processed by referencing three functions including the convolution function, the batch normalization function and the activation function mish, which are respectively for feature extraction of the image convolution, batch normalization of the extracted features, and selection of the appropriate activation function mish to prevent the gradient from disappearing, so that the number of channels changes from 3 to 32, speeding up the feature extraction process; then the input image is subjected to multiple convolution and pooling operations in the network, which can generate three feature maps of different scales: a 52×52 scale feature layer upsampled 8 times the number of 256 channels, a 26×26 scale feature layer upsampled 16 times the number of 512 channels, and a 13×13 scale feature layer upsampled 32 times the number of 1024 channels; excessive upsampling convolution pooling will lose some image position and texture detail features when extracting features; therefore, the improved YOLOv4 network structure is adopted to increase the generated scale feature layer; some detail features are extracted in the feature layer with less convolution pooling, thereby improving the accuracy of ship target detection; The structure of the improved YOLOv4 network includes: The improved YOLOv4 network adds two layers on the basis of the original feature pyramid to enhance the feature extraction function; the improved YOLOv4 network adds 4 data fusion modules, 2 upsampling modules, and 2 downsampling modules to the original path aggregation network; the improved YOLOv4 network changes the original 3 scale feature layers to 5 scale feature layers, namely 208*208 scale feature layer, 104*104 scale feature layer, 52*52 scale feature layer, 26*26 scale feature layer and 13*13 scale feature layer.
3. A method for identifying marine ships in remote sensing images based on an improved YOLOv4 algorithm according to claim 2, characterized in that: The process of step one includes: Step 1.
1. Build a ship target sample library. The sources of the sample library include the following: 1) Obtain existing ship target images on the Internet through web crawlers; 2) Collect visible light and infrared ship target images taken by optoelectronic reconnaissance equipment during previous missions; 3) Satellite remote sensing images of ocean scenes collected from Google Earth; 4) Accumulate the newly detected ship target images during the subsequent mission execution of the optoelectronic reconnaissance equipment to maintain and expand the sample library; Step 1.
2. Perform target calibration on the images in the ship target sample library to produce a data set; use the data labeling software labelimg to draw a target marking frame for the target area in each image; the position information of the target in the image refers to the width and height information of the target marking frame, and the coordinate information of the diagonal points of the target marking frame in the image coordinate system; set the label of each image as: ship, aircraft; training subset and test subset, with a ratio of 8:2; use the data labeling software labelimg to label the ships in the training set in the data set, generate an xml file, and use the python script voc_yolo to convert the xml file format to txt file format.
4. A method for identifying marine ships in remote sensing images based on an improved YOLOv4 algorithm according to claim 2, characterized in that: The process of step 2 includes: The infrared image preprocessing method is used to retain the junction of the two parts with large color and light-dark contrast in the image; first, Gaussian filtering is used to separate the ship from the surrounding islands and buildings to enhance the contrast; then the following operations are used to sharpen the image to achieve the effect of eliminating image noise. Its expression is as follows: scr=img-λ×guas(img) (1) In formula (1), scr represents the processed result, img represents the original image, λ represents the gain, and guas represents the Gaussian filter used. In order to improve the infrared contrast and reduce the introduction of noise, λ=3. The boundary between two parts with large color and light-dark contrast in the image is retained, making the ship target features more prominent.
5. The method for identifying marine ships in remote sensing images based on an improved YOLOv4 algorithm according to claim 2, characterized in that: The process of step 4 includes: The K-means clustering algorithm is used to obtain the anchor box size that meets the scale characteristics of the target to be detected in the infrared image. That is, the number of prior frames is set to 9, and the width and height of each prior frame are (17.4, 30.0), (26.6, 73.2), (49.0, 134.8), (50.8, 46.6), (89.9, 89.9), (97.3, 209.7), (176.4, 294.3), (204.7, 152.4), (348.4, 331.7); In order to obtain the optimal anchor box, the maximum IOU value is taken as a reference, and non-maximum suppression is used to remove redundant anchor boxes; Then, in order to identify small targets such as ships in remote sensing images, a smaller prediction box must be used for recognition, so the IOU value must be reduced; however, the IOU value is proportional to the target detection confidence. If the confidence is too low, serious missed detection will occur; in order to improve the accuracy of small target detection, the IOU is reduced to 0.3 here.
6. The method for identifying marine ships in remote sensing images based on an improved YOLOv4 algorithm according to claim 2, characterized in that: The process of step five includes: Import the preprocessed training set into the improved YOLOv4 network, and continuously update the weights until the loss function is minimized to obtain a trained network. The loss function is as follows: L=λ1L conf (o,c)+λ2L cla (O,C)+λ3L loc (l,g) (2) In formula (2), the loss function consists of three parts: target confidence loss L conf (o,c), target position loss L loc (l,g), target category loss L cla (O, C), λ1, λ2, and λ3 are balance coefficients. λ3 is greater than 1, which increases the proportion of the target loss function and prevents excessive influence from external non-target objects; In formula (3), the target confidence loss L conf (o,c) uses binary cross entropy loss, where oi∈{0,1} indicates whether the target actually exists in the predicted target bounding box i, 0 means no target exists, and 1 means yes target exists; Represents the Sigmoid probability of predicting whether there is a target in the target rectangular box i; In formula (4), the target position loss L loc (l,g) uses the square sum of the deviations between the true value and the predicted value, where l represents the coordinate offset of the predicted rectangular box, and g represents the coordinate offset between the matched GTbox and the default box; In formula (5), the target category loss Lcla(O,C) adopts binary cross entropy loss, where O ij ∈{0,1} indicates whether there is a real target in the predicted target bounding box i, 0 means no target, 1 means yes target; Represents the Sigmoid probability of predicting whether there is a target in the target rectangular box i; Then, the images in the test subset are input into the trained improved YOLOv4 network structure to obtain the category of the target, the specific position of the target in the image, and the width and height of the target; if the accuracy is high, the target detection is completed; if a large number of missed detections or false detections occur, return to step three to continue training until a network with higher detection accuracy is obtained.
