Sea floating garbage intelligent identification method based on video monitoring

Abstract

The invention discloses an intelligent sea floating garbage identification method based on video monitoring, and the method comprises the following steps: S1, data collection and preprocessing: obtaining image data from monitoring images of video monitoring equipment to form an image database, and carrying out the preprocessing of the image data; s2, constructing a model sample library: based on historical observation data of a research sea area and records of related literatures, determining categories of image classification, formulating a labeling scheme, and then performing image labeling according to the labeling scheme to obtain a model training sample library; s3, model construction and optimization: aiming at sea floating garbage, determining a target object and white waves to construct a corresponding recognition algorithm model, and optimizing the recognition algorithm model by adding samples and transfer learning; and S4, generating a visual information product: performing intelligent identification on new image data by adopting the optimized identification algorithm model, deleting an identification result of non-sea floating garbage, and marking and displaying the identified sea floating garbage in a rectangular frame manner.

Description

Technical Field

[0001] The present invention relates to the technical field of marine debris identification, and in particular to a marine debris intelligent identification method based on video monitoring. Background Art

[0002] Against the background of rapid development of the marine economy, marine garbage pollution has become one of the major challenges facing the realization of high-quality development of the marine economy. Marine garbage pollution not only causes water pollution and visual pollution, but may also threaten navigation safety, threaten the survival of marine life, and even directly threaten human health through the food chain. In addition, marine garbage will also affect the development of offshore fisheries, which will in turn have a negative effect on the marine economy. The means of monitoring marine floating garbage based on video surveillance technology can obtain dynamic change data of marine floating garbage in specific sea areas for a long time and continuously. Therefore, monitoring and managing marine garbage based on video surveillance technology will not only help reduce the negative impact of marine garbage on the marine ecological environment, but also contribute to the comprehensive management of the bay and the construction of a beautiful bay, which has important practical significance.

[0003] However, due to the complex and changeable marine environment, as well as the wide variety of marine debris and its different sources, there is an urgent need for an effective intelligent identification method to intelligently identify it. With the significant improvement of computer computing power and the continuous development of image recognition algorithm technology, deep learning models have made many important breakthroughs in the field of image classification. Among them, convolutional neural network (CNN) is one of the key technologies in the field of deep learning technology. CNN can automatically learn and extract features from data, and its generalization ability in image processing far exceeds that of traditional methods. CNN can not only effectively reduce the dimension of large-scale images into small-scale data, but also effectively retain the features of the image, which conforms to the basic principles of image processing. Therefore, the method based on convolutional neural network has high feasibility in the intelligent identification of marine debris.

[0004] In the field of image recognition, the main tasks include image classification, target detection and semantic segmentation. Target detection is mainly used to identify specific objects in an image and determine their location and category. Semantic segmentation can provide a more accurate understanding of objects in an image with pixel-level accuracy. However, due to the complex and diverse types of marine debris and its varied forms, it is difficult to accurately distinguish mixed garbage in a large area using semantic segmentation methods alone. It is difficult to obtain all the attribute characteristics of marine debris using target detection methods alone. In addition, white waves caused by ships, buoys, and ship navigation or sea wind are also easily considered to be marine debris. The current marine debris identification methods have poor recognition accuracy, are prone to misjudgment and missed judgments, and are difficult to meet actual usage needs. Summary of the invention

[0005] In order to solve the above problems, the present invention proposes an intelligent identification method of marine debris based on video monitoring.

[0006] To achieve the above object, the present invention adopts the following technical solutions:

[0007] A method for intelligently identifying marine debris based on video surveillance comprises the following steps:

[0008] S1. Data collection and preprocessing: acquiring image data from surveillance images of video surveillance equipment to form an image database, and preprocessing the image data;

[0009] S2. Construct a model sample library: Based on the historical observation data of the research sea area and the records of relevant literature, determine the image classification category, formulate a labeling scheme, and then label the images according to the labeling scheme to obtain a model training sample library;

[0010] S3, model construction and optimization: Build corresponding recognition algorithm models for marine debris, clear target objects and white waves, and optimize the recognition algorithm models by adding samples and transfer learning;

[0011] S4. Generate visual information products: Use the optimized recognition algorithm model to intelligently identify new image data, delete the identification results of non-marine debris, and mark and display the identified marine debris in the form of rectangular frames.

[0012] Preferably, the specific process of the pretreatment in step S1 is:

[0013] S11. Perform denoising, contrast enhancement, image segmentation and image transformation on image data to improve image quality and usability;

[0014] S12, expand the size of the image dataset by rotating and flipping the image;

[0015] S13. Classify and statistically analyze the types of garbage and background objects in the image data to provide a classification basis for subsequent image annotation and recognition processes.

[0016] Preferably, the specific process of image annotation in step S2 is:

[0017] S21. Determine the image classification category and formulate a labeling scheme based on the historical observation data of the research area and the records of relevant literature;

[0018] S22. Open the Labelme software and import the image folder that needs to be labeled;

[0019] S23, using the create polygons function box to select the category to be labeled, and label the image data in the image folder;

[0020] S24, after the annotation is completed, the file is saved and an annotation file in JASON format is generated;

[0021] S25. Convert the generated JASON-formatted annotation files into TXT-formatted annotation files and PNG-formatted annotation images, respectively. The TXT-formatted annotation files are used for target detection tasks, and the PNG-formatted annotation images are used for semantic segmentation tasks. The TXT-formatted annotation files, the PNG-formatted annotation images, and the original image data are jointly constructed into a model training sample library.

[0022] Preferably, the specific construction process of the recognition algorithm model in step S3 is:

[0023] S31. Use semantic segmentation model to build marine debris recognition algorithm model;

[0024] S32, using the target detection model to construct a recognition algorithm model for a clear target object, wherein the clear target objects include ships and buoys;

[0025] S33. Use machine vision methods to build a recognition algorithm model for Bailang.

[0026] Preferably, the semantic segmentation model in step S31 adopts a SegNet model, which includes an encoder network, a decoder network and a pixel-level classification layer; in the process of training the SegNet model, the original image data is resampled, and the image resolution is adjusted to 416×416 pixels to reduce the computational complexity of the convolution operation, and then the preprocessed image data is input into the network structure of the SegNet model, and the training hyperparameters are set, and the stochastic gradient descent algorithm is selected for training; the hyperparameter settings are: the impulse is set to 0.5, the learning rate is set to 0.001, the batch size is set to 40, the number of iterations is set according to the size of the training sample set, and other parameters use the default values ​​of the model;

[0027] The target detection model in step S32 adopts the YOLO model. The YOLO model replaces the Inception module in the GoogLeNet network by using 1×1 convolutional layers and 3×3 convolutional layers to construct a network architecture consisting of 24 convolutional layers and 2 fully connected layers. In the network architecture, the convolutional layer is used to extract features from the input image, and the fully connected layer is used to predict the position and category probability of the target in the image. In training the YOLO model, the YAML file required for training is configured. The YAML file includes setting the image training path and the classification information of the types of marine debris, adjusting the batch size according to the input image data set, adjusting the number of iterations according to the number of training sample sets, and other parameters adopt the default values ​​of the model.

[0028] The machine vision method in step S33 adopts a threshold method, and takes the pixel value of the R channel of the image as the basis for identifying white waves. Since the white waves are in the R channel of the image, and the pixel value is above 210, the pixel value 210 is selected as the threshold, and the pixel value of the R channel higher than 210 is considered to be white waves.

[0029] Preferably, after the recognition algorithm model in step S4 recognizes the marine debris, the image is thresholded or the Canny edge detection algorithm is used to convert the image into a binary image, and then the outline of the identified garbage patches is determined, and the garbage patches are framed with a circumscribed rectangle.

[0030] Preferably, the process of determining the outline of the garbage patch is specifically as follows: for a binary image with a black background and a white target, if the 8-neighborhood or 4-neighborhood of a white point in the image are all white, the white point is an internal point of the target, and the white point is set to black, visually presenting an effect of the interior being hollowed out; if there is a black point in the neighborhood of the white point, the white color is kept unchanged, and the white point is the outline point of the target.

[0031] After adopting the above technical scheme, the present invention has the following beneficial effects: in view of the diversity and complexity of marine debris, the present invention adopts a hierarchical and multi-model recognition strategy, and uses semantic segmentation models such as the SegNet model of the convolutional neural network CNN to perform semantic segmentation of marine debris, so as to accurately identify and segment the garbage area; uses target detection models such as the YOLO model to identify clear targets such as ships and buoys; and uses machine vision methods such as the threshold method to identify and process white wave phenomena that may be confused with marine debris. In the above manner, marine debris can be intelligently identified by combining a variety of deep learning methods for different target features and constructing corresponding technical processes, which can not only significantly improve the accuracy and universality of marine debris identification, but also has important significance for improving the overall performance of the marine debris identification system. This multi-level model combination effectively improves the recognition accuracy of marine debris, reduces misjudgment and missed judgment, and can provide strong support for marine environmental pollution monitoring and governance, and also provides scientific reference and decision-making basis for marine sanitation operations. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 is a flow chart of the present invention;

[0033] Figure 2 It is a flowchart of the present invention;

[0034] Figure 3 A flowchart of constructing a model training sample library for the present invention;

[0035] Figure 4 It is the original image without image annotation in the present invention;

[0036] Figure 5 An image that has been annotated by the present invention;

[0037] Figure 6 It is a network structure diagram of the SegNet model of the present invention;

[0038] Figure 7 It is a network structure diagram of the YOLO model of the present invention;

[0039] Figure 8 This is an example of misidentifying white waves caused by ships sailing or sea wind as marine debris;

[0040] Fig. 9 This is an example diagram of pixel value distribution in some misjudged areas;

[0041] Fig.10 This is an example diagram of the white wave identification threshold method of the present invention;

[0042] Fig.11 This is an example diagram of the marine debris visualization product of the present invention. DETAILED DESCRIPTION

[0043] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0044] like Figures 1 to 11 As shown, a method for intelligently identifying marine debris based on video surveillance comprises the following steps:

[0045] S1. Data collection and preprocessing: acquiring image data from surveillance images of video surveillance equipment to form an image database, and preprocessing the image data;

[0046] The specific process of the pretreatment in step S1 is:

[0047] S11. Perform denoising, contrast enhancement, image segmentation and image transformation on image data to improve image quality and usability;

[0048] S12, expand the size of the image dataset by rotating and flipping the image;

[0049] S13, classifying and statistically analyzing the types of garbage and background objects in the image data to provide a classification basis for subsequent image annotation and recognition processes;

[0050] S2. Constructing a model sample library: Based on the historical observation data of the research sea area and the records of relevant literature, determine the image classification category, formulate a labeling scheme, and then label the images according to the labeling scheme to obtain a model training sample library;

[0051] The specific process of image annotation in step S2 is as follows:

[0052] S21. Determine the image classification category and formulate a labeling scheme based on the historical observation data of the research area and the records of relevant literature;

[0053] S22. Open the Labelme software and import the image folder that needs to be labeled;

[0054] S23, using the create polygons function box to select the category to be labeled, and label the image data in the image folder;

[0055] S24, after the annotation is completed, the file is saved and an annotation file in JASON format is generated;

[0056] S25, converting the generated JASON format annotation file into a TXT format annotation file and a PNG format annotation image, respectively, wherein the TXT format annotation file is used for the target detection task, and the PNG format annotation image is used for the semantic segmentation task; the TXT format annotation file, the PNG format annotation image and the original image data are jointly constructed into a model training sample library;

[0057] S3, model construction and optimization: Build corresponding recognition algorithm models for marine debris, clear target objects and white waves, and optimize the recognition algorithm models by adding samples and transfer learning;

[0058] The specific construction process of the recognition algorithm model in step S3 is:

[0059] S31. Use semantic segmentation model to build marine debris recognition algorithm model;

[0060] The semantic segmentation model in step S31 adopts the SegNet model, which includes an encoder network, a decoder network and a pixel-level classification layer; in the process of training the SegNet model, the original image data is resampled, and the image resolution is adjusted to 416×416 pixels to reduce the computational complexity of the convolution operation, and then the preprocessed image data is input into the network structure of the SegNet model, the training hyperparameters are set, and the stochastic gradient descent algorithm is selected for training; the hyperparameter settings are as follows: the impulse is set to 0.5, the learning rate is set to 0.001, the batch size is set to 40, the number of iterations is set according to the size of the training sample set, and other parameters use the default values ​​of the model;

[0061] S32, using the target detection model to construct a recognition algorithm model for a clear target object, wherein the clear target objects include ships and buoys;

[0062] The target detection model in step S32 adopts the YOLO model. The YOLO model replaces the Inception module in the GoogLeNet network by using 1×1 convolutional layers and 3×3 convolutional layers to construct a network architecture consisting of 24 convolutional layers and 2 fully connected layers. In the network architecture, the convolutional layer is used to extract features from the input image, and the fully connected layer is used to predict the position and category probability of the target in the image. In training the YOLO model, the YAML file required for training is configured. The YAML file includes setting the image training path and the classification information of the types of marine debris, adjusting the batch size according to the input image data set, adjusting the number of iterations according to the number of training sample sets, and other parameters adopt the default values ​​of the model.

[0063] S33, using machine vision method to build a recognition algorithm model of Bailang;

[0064] The machine vision method in step S33 adopts a threshold method, and uses the pixel value of the R channel of the image as the basis for identifying white waves. Since the white waves are in the R channel of the image, and the pixel value is above 210, the pixel value 210 is selected as the threshold, and the pixel value of the R channel higher than 210 is considered to be white waves;

[0065] Through the analysis and statistics of multiple misjudgment sample images, it is finally found that white waves have significant characteristics in the R channel (red channel) of the image, and its pixel value is usually higher than about 210, such as Fig. 9 The pixel value distribution of some misjudged areas is shown in the figure. The x-axis is the misjudged area, i.e., misjudged areas 1 to 10; the y-axis is the pixel value in the area. The light-colored columns are the pixel value distribution of the white waves, and the dark-colored columns are the pixel value distribution of areas other than the white waves. Fig. 9 It can be found that the pixel values ​​of Bailang are distributed above 200, and some are above 210. After taking the intersection, the pixel value 210 is selected as the threshold. Fig.10 As shown in the figure, the boxes in the figure mark the misjudged areas of the surveillance image after the semantic segmentation model is recognized, and these areas are magnified. In the enlarged image, it can be seen that the pixel points marked by circles have pixel values ​​of R channels higher than 210, and these points are concentrated in the white wave area. This shows that the high pixel value of the R channel can be used as an important basis for white wave identification;

[0066] S4. Generate visual information products: Use the optimized recognition algorithm model to intelligently identify new image data, delete the identification results of non-marine debris, and mark and display the identified marine debris in the form of rectangular frames;

[0067] After the recognition algorithm model in step S4 recognizes the marine debris, the image is thresholded or the Canny edge detection algorithm is used to convert the image into a binary image, and then the outline of the identified debris patch is determined, and the debris patch is framed with a circumscribed rectangle;

[0068] The specific process of determining the outline of garbage patches is as follows: for a binary image with a black background and a white target, if the 8-neighborhood or 4-neighborhood of a white point in the image is all white, the white point belongs to the interior point of the target, and the white point is set to black, visually presenting the effect of the interior being hollowed out; if there is a black point in the neighborhood of the white point, the white color remains unchanged, and the white point is the outline point of the target.

[0069] The above is only a preferred specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed by the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A method for intelligent identification of marine debris based on video surveillance, characterized in that: The following steps are involved: S1. Data collection and preprocessing: acquiring image data from surveillance images of video surveillance equipment to form an image database, and preprocessing the image data; S2. Construct a model sample library: Based on the historical observation data of the research sea area and the records of relevant literature, determine the image classification category, formulate a labeling scheme, and then label the images according to the labeling scheme to obtain a model training sample library; S3, model construction and optimization: Build corresponding recognition algorithm models for marine debris, clear target objects and white waves, and optimize the recognition algorithm models by adding samples and transfer learning; S4. Generate visual information products: Use the optimized recognition algorithm model to intelligently identify new image data, delete the identification results of non-marine debris, and mark and display the identified marine debris in the form of rectangular frames.

2. The method for intelligently identifying marine debris based on video surveillance according to claim 1, characterized in that: The specific process of the pretreatment in step S1 is: S11. Perform denoising, contrast enhancement, image segmentation and image transformation on image data to improve image quality and usability; S12, expand the size of the image dataset by rotating and flipping the image; S13. Classify and statistically analyze the types of garbage and background objects in the image data to provide a classification basis for subsequent image annotation and recognition processes.

3. The method for intelligently identifying marine debris based on video surveillance as claimed in claim 1, characterized in that: The specific process of step S2 is: S21. Determine the image classification category and formulate a labeling scheme based on the historical observation data of the research area and the records of relevant literature; S22. Open the Labelme software and import the image folder that needs to be labeled; S23, using the create polygons function box to select the category to be labeled, and label the image data in the image folder; S24, after the annotation is completed, the file is saved and an annotation file in JASON format is generated; S25. Convert the generated JASON-formatted annotation files into TXT-formatted annotation files and PNG-formatted annotation images, respectively. The TXT-formatted annotation files are used for target detection tasks, and the PNG-formatted annotation images are used for semantic segmentation tasks. The TXT-formatted annotation files, the PNG-formatted annotation images, and the original image data are jointly constructed into a model training sample library.

4. The method for intelligently identifying marine debris based on video surveillance according to claim 1, characterized in that: The specific construction process of the recognition algorithm model in step S3 is: S31. Use semantic segmentation model to build marine debris recognition algorithm model; S32, using the target detection model to construct a recognition algorithm model for a clear target object, wherein the clear target objects include ships and buoys; S33. Use machine vision methods to build a recognition algorithm model for Bailang.

5. The method for intelligently identifying marine debris based on video surveillance as claimed in claim 4, characterized in that: The semantic segmentation model in step S31 adopts the SegNet model, which includes an encoder network, a decoder network and a pixel-level classification layer; in the process of training the SegNet model, the original image data is resampled, and the image resolution is adjusted to 416×416 pixels to reduce the computational complexity of the convolution operation, and then the preprocessed image data is input into the network structure of the SegNet model, the training hyperparameters are set, and the stochastic gradient descent algorithm is selected for training; the hyperparameter settings are as follows: the impulse is set to 0.5, the learning rate is set to 0.001, the batch size is set to 40, the number of iterations is set according to the size of the training sample set, and other parameters use the default values ​​of the model; The target detection model in step S32 adopts the YOLO model. The YOLO model replaces the Inception module in the GoogLeNet network by using 1×1 convolutional layers and 3×3 convolutional layers to construct a network architecture consisting of 24 convolutional layers and 2 fully connected layers. In the network architecture, the convolutional layer is used to extract features from the input image, and the fully connected layer is used to predict the position and category probability of the target in the image. In training the YOLO model, the YAML file required for training is configured. The YAML file includes setting the image training path and the classification information of the types of marine debris, adjusting the batch size according to the input image data set, adjusting the number of iterations according to the number of training sample sets, and other parameters adopt the default values ​​of the model. The machine vision method in step S33 adopts a threshold method, and takes the pixel value of the R channel of the image as the basis for identifying white waves. Since the white waves are in the R channel of the image, and the pixel value is above 210, the pixel value 210 is selected as the threshold, and the pixel value of the R channel higher than 210 is considered to be white waves.

6. The method for intelligently identifying marine debris based on video surveillance as claimed in claim 1, characterized in that: After the recognition algorithm model in step S4 recognizes the marine debris, the image is thresholded or the Canny edge detection algorithm is used to convert the image into a binary image, and then the outline of the identified garbage patch is determined, and the garbage patch is framed with an external rectangle.

7. The method for intelligently identifying marine debris based on video surveillance as claimed in claim 6, characterized in that: The specific process of determining the outline of garbage patches is as follows: for a binary image with a black background and a white target, if the 8-neighborhood or 4-neighborhood of a white point in the image is all white, the white point belongs to the interior point of the target, and the white point is set to black, visually presenting the effect of the interior being hollowed out; if there is a black point in the neighborhood of the white point, the white color remains unchanged, and the white point is the outline point of the target.

Images