Deep learning-based ship black smoke identification method and system

Abstract

The invention provides a ship black smoke recognition method and system based on deep learning, and belongs to the technical field of ship black smoke recognition, and the method comprises the steps: 1, obtaining a high-definition video of a ship according to a high-definition device, obtaining a key frame of the high-definition video based on a target detection algorithm, and carrying out the preprocessing; 2, designing a black smoke ship identification network model based on depth separable convolution, and performing optimization training on the black smoke ship identification network model according to an Adam algorithm; 3, inputting the preprocessing result into the network model for forward reasoning operation, obtaining the positions of all prediction frames, and obtaining a preliminary prediction result; and 4, performing non-maximum suppression filtering on a prediction result, and obtaining a final black smoke frame and Ringelmann blackness based on the limitation of a threshold value and the position of the ship to realize black smoke recognition of the ship. The problems that the detection effect is not satisfactory, the generalization performance is poor, a specific scene needs to be retrained, the black smoke coordinate accuracy cannot be improved, and the false detection occurrence probability is improved are solved.

Description

Technical Field

[0001] The present invention relates to the technical field of ship black smoke recognition, and in particular to a ship black smoke recognition method and system based on deep learning. Background Art

[0002] Due to the backward level of port pollution prevention and control in my country, pollutant emissions are becoming more and more serious, especially in coastal and riverside areas such as the Yangtze River Delta, the Pearl River Delta, and the Beijing-Tianjin-Hebei region. Ship port emissions have become one of the important sources of air pollution. Therefore, it is crucial for the maritime department to have a mature and reliable black smoke ship identification algorithm.

[0003] Early detection technology could only rely on manual feature construction and traditional machine learning. It used a sliding window to scan the image in blocks, extracted Haar features from the targets in the sliding window, and then used the AdaBoost algorithm for classification. This algorithm was not only time-consuming, but also had unsatisfactory detection results and poor generalization performance. It needed to be retrained for specific scenarios, and was unable to improve the accuracy of black smoke coordinates, increasing the probability of false detection.

[0004] Therefore, the present invention provides a ship black smoke recognition method and system based on deep learning. Summary of the invention

[0005] The present invention provides a ship black smoke recognition method and system based on deep learning, which are used to solve the defects of the prior art that the time consumption is huge, the detection effect is not satisfactory, the generalization performance is poor, retraining is required for specific scenes, the accuracy of black smoke coordinates cannot be improved, and the probability of false detection is increased.

[0006] On the one hand, the present invention provides a method for identifying ship black smoke based on deep learning, comprising:

[0007] Step 1: Obtain a high-definition video of the ship using a high-definition device, obtain key frames of the high-definition video of the ship based on a target detection algorithm, and preprocess the key frames to obtain a preprocessing result;

[0008] Step 2: Design a black smoke ship recognition network model based on deep separable convolution, and optimize and train the black smoke ship recognition network model according to the Adam algorithm;

[0009] Step 3: Input the preprocessing results into the trained network model for forward reasoning, obtain the positions of all prediction boxes, and obtain preliminary prediction results;

[0010] Step 4: Perform non-maximum suppression filtering on the prediction results, and obtain the final black smoke frame and Ringelmann blackness based on the threshold limit and the ship position to achieve black smoke recognition of the ship.

[0011] According to a ship black smoke recognition method based on deep learning provided by the present invention, a high-definition video of the ship is obtained according to a high-definition device, a key frame is obtained based on a target detection algorithm, and the key frame is preprocessed to obtain a preprocessing result, including:

[0012] Obtaining a high-definition camera with corresponding resolution and frame rate according to multiple weather conditions, and capturing a video stream of the ship based on the high-definition video data obtained by the high-definition camera;

[0013] The video stream is screened and cut according to the target detection algorithm to obtain a key frame containing the vessel;

[0014] Classifying the key frames with and without black smoke at different Ringelmann blackness levels, obtaining black smoke samples and samples without black smoke, and annotating the classified data frame by frame;

[0015] The black smoke samples are expanded according to the annotation results, and the expanded images are preprocessed and adjusted to obtain adjusted image data, and a training data set is determined according to the image data.

[0016] According to a method for identifying black smoke from a ship based on deep learning provided by the present invention, after obtaining a key frame containing a ship, the method further includes:

[0017] Obtaining the hull identification features of the ship, and judging whether the key frame contains a moving object other than the hull according to the hull identification features and the pixel features of the key frame;

[0018] If the key frame contains a moving object other than the hull, extract the motion vector in the key frame, and determine the motion rule of the moving object other than the hull according to the motion vector;

[0019] Determine the motion state of the moving object other than the hull based on the motion rule, and judge whether the moving object is black smoke according to the motion state;

[0020] If yes, determine the spatial characteristics of the black smoke according to the motion vector, and determine the flow gradient position direction of the black smoke based on the spatial characteristics;

[0021] Obtain the key frame pixel area in the direction of the flow gradient position, and extract the local fractal features of each key frame pixel area;

[0022] Determine the recognition loss weight of the black smoke in the key frame according to the local fractal features, and determine the tracking zoom factor for the key frame based on the recognition loss weight;

[0023] Determine an image processing method for the key frame based on the tracking zoom ratio of the key frame, and obtain a feature classification attribute of the image processing method;

[0024] Determine a feature capture form of the image processing method according to the feature classification attribute, and determine a black smoke concentration variation sequence under the feature capture form;

[0025] The pixel measurement accuracy of the key frame is determined according to the black smoke concentration change sequence, and the key frame is classified into different Ringelmann blackness categories of no black smoke and with black smoke based on the pixel measurement accuracy.

[0026] According to a ship black smoke recognition method based on deep learning provided by the present invention, a black smoke ship recognition network model based on deep separable convolution is designed, and the black smoke ship recognition network model is optimized and trained according to the Adam algorithm, including:

[0027] Using the depthwise separable convolution in MobileNet as a basic module, and determining a backbone network based on the basic module;

[0028] Adding a spatial attention mechanism and function replacement to the backbone network, and performing nonlinear operations;

[0029] The operated backbone network is artificially divided into three parts, and the features of the image data are extracted according to the three parts. The outputs of the three parts are connected to the back-end neck part of yolov4 and connected to the yolo Head layer to predict the features.

[0030] According to the k-means algorithm, the prior frames of black smoke and ships are obtained in the data set, and the black smoke ship recognition network is constructed based on the loss function according to the prior frames and feature values;

[0031] The black smoke ship recognition network is optimized and trained according to the Adam algorithm to obtain the trained black smoke ship recognition network model.

[0032] According to a ship black smoke recognition method based on deep learning provided by the present invention, the preprocessing result is input into the trained network model for forward reasoning operation, the positions of all prediction boxes are obtained, and preliminary prediction results are obtained, including:

[0033] Convert the preprocessed key video frame data into an appropriate tensor shape and input it into the trained network model for forward reasoning to obtain the operation result;

[0034] According to the operation results, features are extracted based on the backbone network and input into each layer of the pyramid for prediction to obtain the prediction results of each layer;

[0035] According to the prediction results of each layer and the feature value decoding of the current layer, the positions of all prediction boxes are obtained to obtain preliminary prediction results.

[0036] According to a ship black smoke recognition method based on deep learning provided by the present invention, non-maximum suppression filtering is performed on the prediction result, and the final black smoke frame and Ringelmann blackness are obtained based on the threshold limit and the ship position to realize the ship black smoke recognition, including:

[0037] The prediction results are sorted according to probability, and non-maximum suppression filtering is performed to filter out prediction results with local non-maximum probability;

[0038] Get the upper left corner coordinates, width and height threshold range of the target box, and filter and correct the prediction results according to the threshold range;

[0039] The position of the ship is obtained, and the false detection frame of non-black smoke is filtered out according to the detection value of the ship's position. The final black smoke frame and Ringelmann blackness are obtained by combining the filtering and correction results to realize the black smoke recognition of the ship.

[0040] A ship black smoke recognition method based on deep learning provided by the present invention also includes:

[0041] Estimate the bias, drift and other error parameters of the sensor according to the extended Kalman filter technology to obtain an estimation result;

[0042] Acquire motion observation values ​​of the vessel based on different sensors according to the estimation result, and fuse the motion observation values;

[0043] Perform nonlinear optimization on the fused observations to obtain and solve the ship's attitude;

[0044] Establishing a motion model of the vessel according to the posture and motion observation values ​​of the vessel;

[0045] The real-time position of the vessel is tracked based on the minimization of the loss function according to the results of the motion model and attitude solution.

[0046] A ship black smoke recognition system based on deep learning provided by the present invention includes:

[0047] Preprocessing module: obtaining high-definition video of the ship according to the high-definition equipment, obtaining key frames based on the target detection algorithm, and preprocessing the key frames;

[0048] Training module: Design a black smoke ship recognition network based on deep separable convolution and optimize the training according to the Adam algorithm;

[0049] Forward reasoning module: input the preprocessing results into the trained network model for forward reasoning operations, obtain the positions of all prediction boxes and obtain preliminary prediction results;

[0050] Identification module: Perform non-maximum suppression filtering on the prediction results, and obtain the final black smoke frame and Ringelmann blackness based on the threshold limit and the ship position to realize the black smoke identification of the ship.

[0051] Compared with the prior art, the present invention has the following beneficial effects:

[0052] By constructing a lightweight network dedicated to ship black smoke detection as the backbone convolutional layer to extract features, using the yolov4 neck structure to design a black smoke recognition network, and adding ship position detection, and then combining the classification loss function and regression loss function of each layer, the detection effect and speed can reach the state-of-art level, which is helpful to improve the accuracy of black smoke coordinates and directly give the Ringelmann blackness of black smoke without the need to calculate the blackness value twice, which can reduce the probability of false detection, alleviate the gradient diffusion problem of deep networks, and accelerate the network convergence process. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0054] Figure 1 It is a flowchart of a method for identifying black smoke from a ship based on deep learning provided by an embodiment of the present invention;

[0055] Figure 2 It is a structural schematic diagram of a ship black smoke recognition system based on deep learning provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0056] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0057] Embodiment 1:

[0058] The embodiment of the present invention provides a method for identifying ship black smoke based on deep learning, such as Figure 1 As shown, the method mainly includes the following steps:

[0059] Step 1: Obtain a high-definition video of the ship using a high-definition device, obtain key frames of the high-definition video of the ship based on a target detection algorithm, and preprocess the key frames to obtain a preprocessing result;

[0060] Step 2: Design a black smoke ship recognition network model based on deep separable convolution, and optimize and train the black smoke ship recognition network model according to the Adam algorithm;

[0061] Step 3: Input the preprocessing results into the trained network model for forward reasoning, obtain the positions of all prediction boxes, and obtain preliminary prediction results;

[0062] Step 4: Perform non-maximum suppression filtering on the prediction results, and obtain the final black smoke frame and Ringelmann blackness based on the threshold limit and the ship position to achieve black smoke recognition of the ship.

[0063] In this embodiment, the high-definition device refers to a device that displays or records video with a high-definition resolution, and the high-definition resolution generally refers to 720p (1280x720 pixels) or 1080p (1920x1080 pixels).

[0064] In this embodiment, the high-definition video of the ship refers to a ship video shot or recorded with a high-definition resolution (at least 720p or 1080p).

[0065] In this embodiment, the target detection algorithm may be: a moving object tracking algorithm.

[0066] In this embodiment, the key frame of the high-definition video refers to the most representative image frame in the high-definition video that best represents the video content and contains the most representative image frame in the video. These key frames are usually selected according to a certain time interval or event sequence during the video shooting process.

[0067] In this embodiment, the black smoke ship identification network model is mainly used to identify and track the black smoke from the ship.

[0068] In this embodiment, the Adam algorithm is an adaptive learning rate optimization algorithm, which is commonly used to train deep neural networks and other machine learning models.

[0069] In this embodiment, the forward reasoning operation refers to gradually transforming the input data (usually a two-dimensional tensor) into the final output result through a series of layers (convolutional layer, activation function, pooling layer).

[0070] In this embodiment, the non-maximum suppression filtering is to find the best target boundary area based on all predicted target boxes and their respective confidence calculations, and eliminate other redundant boundary boxes.

[0071] The beneficial effects of the above technical solution are: by constructing a lightweight network dedicated to ship black smoke detection as the backbone convolutional layer to extract features, using the yolov4 neck structure to design a black smoke recognition network, and adding ship position detection, and then combining the classification loss function and regression loss function of each layer, the detection effect and speed can reach the state-of-art level, which is helpful to improve the accuracy of black smoke coordinates and directly give the Ringelmann blackness of black smoke without the need to calculate the blackness value twice, which can reduce the probability of false detection, alleviate the gradient diffusion problem of deep networks, and accelerate the network convergence process.

[0072] Embodiment 2:

[0073] Based on Example 1, the embodiment of the present invention obtains a high-definition video of a ship according to a high-definition device, obtains a key frame based on a target detection algorithm, and preprocesses the key frame to obtain a preprocessing result, including:

[0074] Obtaining a high-definition camera with corresponding resolution and frame rate according to multiple weather conditions, and capturing a video stream of the ship based on the high-definition video data obtained by the high-definition camera;

[0075] The video stream is screened and cut according to the target detection algorithm to obtain a key frame containing the vessel;

[0076] Classifying the key frames with and without black smoke at different Ringelmann blackness levels, obtaining black smoke samples and samples without black smoke, and annotating the classified data frame by frame;

[0077] The black smoke samples are expanded according to the annotation results, and the expanded images are preprocessed and adjusted to obtain adjusted image data, and a training data set is determined according to the image data.

[0078] In this embodiment, weather conditions refer to a group of elements that describe the atmospheric conditions in a certain area or during a certain period of time, such as temperature, precipitation, and wind speed.

[0079] In this embodiment, the ship video stream refers to a real-time captured video of the ship's travel process, which can be used to monitor and track the ship's position, speed, heading and other information.

[0080] In this embodiment, the target detection algorithm may be: a moving object tracking algorithm.

[0081] In this embodiment, the Ringelmann blackness classification is a method for classifying ground objects based on color differences in remote sensing images. The classification is achieved by comparing the reflection or radiation spectral characteristics of different types of ground objects. Ships without black smoke and ships with black smoke are two different types of ships, and have different characteristics in remote sensing images. Ships without black smoke usually refer to those that do not emit exhaust gas or have less exhaust gas emissions and do not cause obvious pollution. The color characteristics presented in the remote sensing image are mainly black or gray.

[0082] In this embodiment, the frame-by-frame marking refers to determining the position of the black smoke, the Ringelmann blackness corresponding to the black smoke, and the position of the ship.

[0083] In this embodiment, preprocessing refers to the brightness and sharpness of the image, cropping, deformation, scaling, and adding noise to the image.

[0084] The beneficial effects of the above technical solution are: according to the high-definition camera capturing the video stream of the ship and cutting it, obtaining key frames, classifying different Ringelmann blackness and annotating frame by frame, which can greatly improve the understanding and analysis value of the video content. At the same time, the black smoke samples are expanded and preprocessed to ensure the quality and reliability of the samples.

[0085] Embodiment 3:

[0086] Based on the second embodiment, after obtaining the key frame containing the ship, the embodiment of the present invention further includes:

[0087] Obtaining the hull identification features of the ship, and judging whether the key frame contains a moving object other than the hull according to the hull identification features and the pixel features of the key frame;

[0088] If the key frame contains a moving object other than the hull, extract the motion vector in the key frame, and determine the motion rule of the moving object other than the hull according to the motion vector;

[0089] Determine the motion state of the moving object other than the hull based on the motion rule, and judge whether the moving object is black smoke according to the motion state;

[0090] If yes, determine the spatial characteristics of the black smoke according to the motion vector, and determine the flow gradient position direction of the black smoke based on the spatial characteristics;

[0091] Obtain the key frame pixel area in the direction of the flow gradient position, and extract the local fractal features of each key frame pixel area;

[0092] Determine the recognition loss weight of the black smoke in the key frame according to the local fractal features, and determine the tracking zoom factor for the key frame based on the recognition loss weight;

[0093] Determine an image processing method for the key frame based on the tracking zoom ratio of the key frame, and obtain a feature classification attribute of the image processing method;

[0094] Determine a feature capture form of the image processing method according to the feature classification attribute, and determine a black smoke concentration variation sequence under the feature capture form;

[0095] The pixel measurement accuracy of the key frame is determined according to the black smoke concentration change sequence, and the key frame is classified into different Ringelmann blackness categories of no black smoke and with black smoke based on the pixel measurement accuracy.

[0096] In this embodiment, ship hull recognition refers to extracting and identifying features of various types of ships from remote sensing images, such as texture features, color features, and shape features.

[0097] In this embodiment, a key frame refers to a certain moment in a certain segment of a video or image sequence, and the corresponding moment is called the "key frame" of the segment. The pixel feature of the key frame refers to the information of the pixel points in the image or video frame corresponding to the key frame.

[0098] In this embodiment, the motion vector in the key frame refers to the information of the motion direction and speed of the image corresponding to the time interval between the key frames.

[0099] In this embodiment, the motion rules of a moving object refer to the physical laws and constraints followed by the object during the motion process, such as collision and contact, friction and viscosity, and aerodynamic effects.

[0100] In this embodiment, the motion state of the moving object refers to information such as the position, velocity, and acceleration of the object at a certain moment.

[0101] In this embodiment, the airspace characteristics of black smoke refer to the phenomenon of black smoke appearing during the operation of the ship. When the black smoke emitted by the ship engine is discharged into the atmosphere, a black or gray smoke is formed. The airspace characteristics of black smoke include:

[0102] Color: Black smoke usually appears dark black or gray, which is different from the clear or light blue smoke produced by normal exhaust.

[0103] Density: Black smoke has a high density and can quickly gather in the air and cover large areas.

[0104] Odor: Black smoke usually has a strong, pungent odor.

[0105] Temperature: The temperature of the black smoke airspace is usually higher than the surrounding air because the black smoke itself has higher energy and heat.

[0106] In this embodiment, the flow gradient position direction of the black smoke can be: upward propagation, downwind propagation, and against wind propagation.

[0107] In this embodiment, the local fractal feature is a feature used to describe the local particle shape and structure, which can distinguish different objects and identify specific targets in the image.

[0108] In this embodiment, the black smoke recognition loss weight refers to the cost of accurately identifying the black smoke object in the image segmentation and target detection tasks, and represents the cost of target positioning error or classification error that may result if the black smoke is not correctly identified when training the model.

[0109] In this embodiment, the tracking zoom ratio is used to describe the magnification degree of the camera in the horizontal and vertical directions when tracking an object.

[0110] In this embodiment, the feature classification attributes of the image processing method refer to the attributes and characteristics used to describe the image processing method.

[0111] In this embodiment, the feature capture form of the image processing method refers to the process of extracting image features and classifying or identifying them using techniques used in computer vision and machine learning, such as: local binary pattern matching, oriented gradient histogram, and oriented gradient statistics.

[0112] In this embodiment, the black smoke concentration variation sequence refers to sequence data recording the variation of black smoke concentration over time.

[0113] In this embodiment, pixel measurement accuracy refers to the number of pixels and their sizes in image or video processing.

[0114] The beneficial effects of the above technical solution are: determining the pixel measurement accuracy for key frames according to the black smoke concentration change sequence, and classifying the key frames with and without black smoke according to different Ringelmann blackness values, which can improve the classification accuracy. At the same time, it can effectively screen out areas containing valuable information, thereby improving the utilization of data.

[0115] Embodiment 4:

[0116] Based on Example 3, the embodiment of the present invention designs a black smoke ship recognition network model based on deep separable convolution, and optimizes and trains the black smoke ship recognition network model according to the Adam algorithm, including:

[0117] Using the depthwise separable convolution in MobileNet as a basic module, and determining a backbone network based on the basic module;

[0118] Adding a spatial attention mechanism and function replacement to the backbone network, and performing nonlinear operations;

[0119] The operated backbone network is artificially divided into three parts, and the features of the image data are extracted according to the three parts. The outputs of the three parts are connected to the back-end neck part of yolov4 and connected to the yolo Head layer to predict the features.

[0120] According to the k-means algorithm, the prior frames of black smoke and ships are obtained in the data set, and the black smoke ship recognition network is constructed based on the loss function according to the prior frames and feature values;

[0121] The black smoke ship recognition network is optimized and trained according to the Adam algorithm to obtain the trained black smoke ship recognition network model.

[0122] In this embodiment, the depthwise separable convolution is a convolution operation in a deep neural network, which can be used to extract local features in data such as images or speech. The depthwise separable convolution can be divided into two steps: Dephwise convolution and Pointwise convolution. The former step can significantly reduce the amount of computation of the network, but there is no connection between channels. The 1x1 ordinary convolution used in the latter step has the ability to fuse channels, which can be used to fuse the channel data obtained in the first step to make up for the loss in the first step, and can also be used to change the number of channels.

[0123] In this embodiment, the spatial attention mechanism is an attention mechanism that can help the neural network automatically focus on the most important area when processing multi-dimensional data. It is used to focus on a specific area of ​​the ship's smoke exhaust port and strengthen the recognition of this area to increase the model's recognition rate of ship black smoke.

[0124] In this embodiment, the activation function in the depthwise separable convolution part of the backbone network is replaced with SiLU, and the activation function introduces nonlinear operations in all hidden layers and output layers, thereby enhancing the expressive power of the neural network.

[0125] In this embodiment, feature prediction means that if the input image data is 3x512x512, the last three parts of the features are 64x64x3x11, 32x32x3x11 and 16x16x3x11 respectively, where 11 is the 4-dimensional detection box position, 1-dimensional confidence and 6-dimensional number of categories (5 categories of Ringelmann blackness and 1 category of ships), and 3 is the number of anchors, that is, the number of prior boxes corresponding to each feature layer.

[0126] In this embodiment, the k-means algorithm is a clustering algorithm that defines similarity based on a distance metric and groups the data points into a group. The algorithm divides the data points into k clusters, and the distance between the points in each cluster is minimized as much as possible.

[0127] In this embodiment, the loss function includes: position loss, confidence loss and category loss.

[0128] In this embodiment, during training, EIoU Loss is used for bounding box regression, i.e., position regression, and the overlap loss, center distance loss, and width and height loss of the target box and the predicted box are comprehensively considered, so that the model can learn the characteristics of the target box more comprehensively. Through multiple iterations and back-propagation training, the gap between the predicted box and the real box is continuously reduced, and the network converges faster.

[0129] The beneficial effects of the above technical solution are: by constructing a black smoke ship recognition network based on deep separable convolution and optimizing the training according to the Adam algorithm, the overlap loss, center distance loss and width and height loss of the target box and the prediction box are fully considered, so that the model can learn the characteristics of the target box more comprehensively. Through multiple iterations and back-propagation training, the gap between the predicted box and the real box is continuously reduced, the network converges faster, and the recognition effect of the network is improved.

[0130] Embodiment 5:

[0131] Based on Example 4, the embodiment of the present invention inputs the preprocessing results into the trained network model to perform forward reasoning operations, obtains the positions of all prediction boxes, and obtains preliminary prediction results, including:

[0132] Convert the preprocessed key video frame data into an appropriate tensor shape and input it into the trained network model for forward reasoning to obtain the operation result;

[0133] According to the operation results, features are extracted based on the backbone network and input into each layer of the pyramid for prediction to obtain the prediction results of each layer;

[0134] According to the prediction results of each layer and the feature value decoding of the current layer, the positions of all prediction boxes are obtained to obtain preliminary prediction results.

[0135] In this embodiment, the tensor shape refers to the relationship between the dimensions of the tensor.

[0136] In this embodiment, the forward reasoning operation refers to gradually transforming the input data (usually a two-dimensional tensor) into the final output result through a series of layers (convolutional layer, activation function, pooling layer).

[0137] In this embodiment, the backbone network refers to a network that first subjects the image to some basic preprocessing and then divides the image into many small regions, each of which is composed of alternating convolutional layers and pooling layers, and finally obtains the final feature representation through a fully connected layer.

[0138] In this embodiment, each layer of the pyramid means that each backbone network is composed of multiple stacked convolutional layers, and the output of each convolutional layer is replaced by the input of another convolutional layer to form a multi-level pyramid structure. In this structure, each convolutional layer has a receptive field for extracting feature information of different scales. Therefore, each convolutional layer is performed on the basis of the previous layer and has a different receptive field, which can capture a wider range of features. The matrix size of each layer is (batchsize, 33, 16, 16), (batchsize, 33, 32, 32) and (batchsize, 33, 64, 64).

[0139] In this embodiment, feature value decoding refers to the process of mapping features to a higher-dimensional space to obtain a better representation.

[0140] The beneficial effects of the above technical solution are: by performing forward reasoning operations on the preprocessed key video frames, the computing efficiency and speed are greatly improved, end-to-end black smoke detection can be achieved, and real-time black smoke detection and classification become possible. At the same time, the forward reasoning operation can better capture the complex features and patterns in the video, thereby achieving more accurate classification and prediction.

[0141] Embodiment 6:

[0142] Based on Example 5, the embodiment of the present invention performs non-maximum suppression filtering on the prediction result, and obtains the final black smoke frame and Ringelmann blackness based on the threshold limit and the ship position to realize the black smoke recognition of the ship, including:

[0143] The prediction results are sorted according to probability, and non-maximum suppression filtering is performed to filter out prediction results with local non-maximum probability;

[0144] Get the upper left corner coordinates, width and height threshold range of the target box, and filter and correct the prediction results according to the threshold range;

[0145] The position of the ship is obtained, and the false detection frame of non-black smoke is filtered out according to the detection value of the ship's position. The final black smoke frame and Ringelmann blackness are obtained by combining the filtering and correction results to realize the black smoke recognition of the ship.

[0146] In this embodiment, the non-maximum suppression filtering is to find the best target boundary area based on all predicted target boxes and their respective confidence calculations, and eliminate other redundant boundary boxes.

[0147] In this embodiment, filtering and correction means that the prediction results are the upper left corner coordinates (x1, y1) of the target frame and the width and height (w, h) of the target frame, and the prediction results that exceed the image range are filtered; if w is greater than the image width, w is modified to the image width; if h is greater than the image height h is modified to the image height; if x1, y1 are less than 0, x1, y1 are corrected to 0;

[0148] In this embodiment, the non-black smoke false detection frame refers to a region composed of objects that are mistakenly judged as non-black smoke categories in the target detection task.

[0149] In this embodiment, Ringelmann blackness is a measure that describes the intensity of black body radiation.

[0150] The beneficial effect of the above technical solution is: by performing non-maximum suppression filtering on the prediction results and filtering and correcting them, and combining filtering out the false detection frames of non-black smoke to obtain the final black smoke frame and Ringelmann blackness, the false detection frames caused by noise or other factors can be removed, the specific coordinates of the black smoke can be obtained, and the accuracy of black smoke detection can be improved.

[0151] Embodiment 7:

[0152] Based on Example 6, the embodiment of the present invention further includes:

[0153] Estimate the bias, drift and other error parameters of the sensor according to the extended Kalman filter technology to obtain an estimation result;

[0154] Acquire motion observation values ​​of the vessel based on different sensors according to the estimation result, and fuse the motion observation values;

[0155] Perform nonlinear optimization on the fused observations to obtain and solve the ship's attitude;

[0156] Establishing a motion model of the vessel according to the posture and motion observation values ​​of the vessel;

[0157] The real-time position of the vessel is tracked based on the minimization of the loss function according to the results of the motion model and attitude solution.

[0158] In this embodiment, the extended Kalman filter is to process nonlinear or non-Gaussian processes by introducing additional model parameters so that the Kalman filter can be applied to a wider range of situations. For example, the Kalman filter can be extended to a Gaussian process random field framework to process nonlinear functions and uncertainties.

[0159] In this embodiment, the sensor error or bias refers to the difference between the value measured by the sensor and the true value. This difference may be caused by various factors, such as the inherent error of the sensor itself, environmental changes, and improper operation.

[0160] In this embodiment, sensor drift refers to the systematic change of the value of the sensor output over time, and this change may be caused by various factors, such as temperature change, material aging, and electronic component failure.

[0161] In this embodiment, other error parameters of the sensor include: sensor zero drift, sensor sensitivity drift, sensor gain drift, and sensor environment drift.

[0162] In this embodiment, the sensors include: a global positioning system, an inertial measurement unit and a lidar.

[0163] In this embodiment, the motion observation values ​​of the ship include: position information of the ship, speed information of the ship, and heading information of the ship.

[0164] In this embodiment, nonlinear optimization refers to an algorithm for finding an optimal solution or an optimal point when both the objective function and the constraint conditions are nonlinear.

[0165] In this embodiment, the posture includes: pitch, roll, and yaw.

[0166] In this embodiment, the motion model may be linear, such as uniform linear motion, or nonlinear, such as spiral curve motion.

[0167] In this embodiment, posture calculation refers to the process of using motion data acquired by sensors to calculate the posture information of an object, including position, velocity, and acceleration, through a certain algorithm.

[0168] In this embodiment, the minimization loss function includes: mean square error and mutual information.

[0169] The beneficial effects of the above technical solution are: by fusing the motion observation values ​​of the ship and performing nonlinear optimization, the penetration-value posture solution is obtained, and combined with the constructed penetration-value motion model, the position of the ship is tracked, the position of the ship can be determined quickly and accurately, and the real-time and accuracy of the tracking results are guaranteed.

[0170] Embodiment 8:

[0171] The present invention provides a ship black smoke recognition system based on deep learning, such as Figure 2 As shown, including:

[0172] Preprocessing module: obtaining high-definition video of the ship according to the high-definition equipment, obtaining key frames based on the target detection algorithm, and preprocessing the key frames;

[0173] Training module: Design a black smoke ship recognition network based on deep separable convolution and optimize the training according to the Adam algorithm;

[0174] Forward reasoning module: input the preprocessing results into the trained network model for forward reasoning operations, obtain the positions of all prediction boxes and obtain preliminary prediction results;

[0175] Identification module: Perform non-maximum suppression filtering on the prediction results, and obtain the final black smoke frame and Ringelmann blackness based on the threshold limit and the ship position to realize the black smoke identification of the ship.

[0176] The beneficial effects of the above technical solution are: by constructing a lightweight network dedicated to ship black smoke detection as the backbone convolutional layer to extract features, using the yolov4 neck structure to design a black smoke recognition network, and adding ship position detection, and then combining the classification loss function and regression loss function of each layer, the detection effect and speed can reach the state-of-art level, which is helpful to improve the accuracy of black smoke coordinates and directly give the Ringelmann blackness of black smoke without the need to calculate the blackness value twice, which can reduce the probability of false detection, alleviate the gradient diffusion problem of deep networks, and accelerate the network convergence process.

[0177] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0178] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A ship black smoke recognition method based on deep learning, characterized in that: include: Step 1: Obtain a high-definition video of the ship using a high-definition device, obtain key frames of the high-definition video of the ship based on a target detection algorithm, and preprocess the key frames to obtain a preprocessing result; Step 2: Design a black smoke ship recognition network model based on deep separable convolution, and optimize and train the black smoke ship recognition network model according to the Adam algorithm; Step 3: Input the preprocessing results into the trained network model for forward reasoning, obtain the positions of all prediction boxes, and obtain preliminary prediction results; Step 4: Perform non-maximum suppression filtering on the prediction results, and obtain the final black smoke frame and Ringelmann blackness based on the threshold limit and the ship position to achieve black smoke recognition of the ship.

2. The method for identifying ship black smoke based on deep learning according to claim 1 is characterized in that: Obtain a high-definition video of the ship according to the high-definition device, obtain key frames based on the target detection algorithm, and pre-process the key frames to obtain pre-processing results, including: Obtaining a high-definition camera with corresponding resolution and frame rate according to multiple weather conditions, and capturing a video stream of the ship based on the high-definition video data obtained by the high-definition camera; The video stream is screened and cut according to the target detection algorithm to obtain a key frame containing the vessel; Classifying the key frames with and without black smoke at different Ringelmann blackness levels, obtaining black smoke samples and samples without black smoke, and annotating the classified data frame by frame; The black smoke samples are expanded according to the annotation results, and the expanded images are preprocessed and adjusted to obtain adjusted image data, and a training data set is determined according to the image data.

3. The ship black smoke recognition method based on deep learning according to claim 2 is characterized in that: After getting the keyframe containing the ship, also include: Obtaining the hull identification features of the ship, and judging whether the key frame contains a moving object other than the hull according to the hull identification features and the pixel features of the key frame; If the key frame contains a moving object other than the hull, extract the motion vector in the key frame, and determine the motion rule of the moving object other than the hull according to the motion vector; Determine the motion state of the moving object other than the hull based on the motion rule, and judge whether the moving object is black smoke according to the motion state; If yes, determine the spatial characteristics of the black smoke according to the motion vector, and determine the flow gradient position direction of the black smoke based on the spatial characteristics; Obtain the key frame pixel area in the direction of the flow gradient position, and extract the local fractal features of each key frame pixel area; Determine the recognition loss weight of the black smoke in the key frame according to the local fractal features, and determine the tracking zoom factor for the key frame based on the recognition loss weight; Determine an image processing method for the key frame based on the tracking zoom ratio of the key frame, and obtain a feature classification attribute of the image processing method; Determine a feature capture form of the image processing method according to the feature classification attribute, and determine a black smoke concentration variation sequence under the feature capture form; The pixel measurement accuracy of the key frame is determined according to the black smoke concentration change sequence, and the key frame is classified into different Ringelmann blackness categories of no black smoke and with black smoke based on the pixel measurement accuracy.

4. The ship black smoke recognition method based on deep learning according to claim 1 is characterized in that: Design a black smoke ship recognition network model based on deep separable convolution, and optimize and train the black smoke ship recognition network model according to the Adam algorithm, including: Using the depthwise separable convolution in MobileNet as a basic module, and determining a backbone network based on the basic module; Adding a spatial attention mechanism and function replacement to the backbone network, and performing nonlinear operations; The operated backbone network is artificially divided into three parts, and the features of the image data are extracted according to the three parts. The outputs of the three parts are connected to the back-end neck part of yolov4 and connected to the yolo Head layer to predict the features. According to the k-means algorithm, the prior frames of black smoke and ships are obtained in the data set, and the black smoke ship recognition network is constructed based on the loss function according to the prior frames and feature values; The black smoke ship recognition network is optimized and trained according to the Adam algorithm to obtain the trained black smoke ship recognition network model.

5. The method for identifying ship black smoke based on deep learning according to claim 1, characterized in that: The preprocessing results are input into the trained network model for forward reasoning operations to obtain the positions of all prediction boxes and obtain preliminary prediction results, including: Convert the preprocessed key video frame data into an appropriate tensor shape and input it into the trained network model for forward reasoning to obtain the operation result; According to the operation results, features are extracted based on the backbone network and input into each layer of the pyramid for prediction to obtain the prediction results of each layer; According to the prediction results of each layer and the feature value decoding of the current layer, the positions of all prediction boxes are obtained to obtain preliminary prediction results.

6. The method for identifying ship black smoke based on deep learning according to claim 1, characterized in that: The prediction results are filtered by non-maximum suppression, and the final black smoke frame and Ringelmann blackness are obtained based on the threshold limit and the ship position to realize the black smoke recognition of the ship, including: The prediction results are sorted according to probability, and non-maximum suppression filtering is performed to filter out prediction results with local non-maximum probability; Get the upper left corner coordinates, width and height threshold range of the target box, and filter and correct the prediction results according to the threshold range; The position of the ship is obtained, and the false detection frame of non-black smoke is filtered out according to the detection value of the ship's position. The final black smoke frame and Ringelmann blackness are obtained by combining the filtering and correction results to realize the black smoke recognition of the ship.

7. The method for identifying ship black smoke based on deep learning according to claim 1, characterized in that: Also includes: Estimate the bias, drift and other error parameters of the sensor according to the extended Kalman filter technology to obtain an estimation result; Acquire motion observation values ​​of the vessel based on different sensors according to the estimation result, and fuse the motion observation values; Perform nonlinear optimization on the fused observations to obtain and solve the ship's attitude; Establishing a motion model of the vessel according to the posture and motion observation values ​​of the vessel; The real-time position of the vessel is tracked based on the minimization of the loss function according to the results of the motion model and attitude solution.

8. A ship black smoke recognition system based on deep learning, characterized in that: include: Preprocessing module: obtaining high-definition video of the ship according to the high-definition equipment, obtaining key frames based on the target detection algorithm, and preprocessing the key frames; Training module: Design a black smoke ship recognition network based on deep separable convolution and optimize the training according to the Adam algorithm; Forward reasoning module: input the preprocessing results into the trained network model for forward reasoning operations, obtain the positions of all prediction boxes and obtain preliminary prediction results; Identification module: Perform non-maximum suppression filtering on the prediction results, and obtain the final black smoke frame and Ringelmann blackness based on the threshold limit and the ship position to realize the black smoke identification of the ship.

Images