Multi-scale collaborative algal bloom assessment method fusing underwater video

Abstract

The invention relates to a multi-scale collaborative algal bloom assessment method fused with an underwater video, and belongs to the field of image recognition. The method comprises the following steps: S1, acquiring underwater, water surface and high-altitude images; s2, collecting water quality data and meteorological data; s3, aligning the data in time and space; s4, performing binary mask segmentation on the image by using an image segmentation technology; s5, vectorizing the binary mask of the image, and performing concat splicing on the vectorized binary mask, the water quality data and the meteorological data to serve as input; s6, an improved Transform model is used for prediction; and S7, fusing the binary masks of the images by using a weighted summation method to obtain a binary mask of the algal bloom area of the monitoring area and a fused algal bloom coverage ratio. The method provided by the invention can realize calculation and prediction of the binary mask of the algal bloom area and the fusion algal bloom coverage ratio of the monitoring area at multiple moments, and is beneficial to dynamic and accurate evaluation of the algal bloom phenomenon.

Description

Technical Field

[0001] The present invention relates to a multi-scale collaborative algae bloom assessment method integrating underwater videos, belongs to the field of image recognition, and in particular to a multi-scale collaborative algae bloom assessment method integrating underwater videos. Background Art

[0002] Algal bloom, or water bloom, refers to the large-scale aggregation of algae in water bodies due to the abnormal reproduction of certain algae, which usually leads to the deterioration of water quality, the deterioration of the aquatic ecological environment, and even the death of a large number of aquatic organisms such as fish, greatly affecting the beauty and use function of the water body. With the impact of global climate change and human activities, algal blooms have become more frequent and have become a global environmental problem. Therefore, accurately and quickly assessing the distribution and severity of algal blooms is of great significance for water management and emergency response to algal blooms.

[0003] Traditional algal bloom assessments rely mainly on field sampling and satellite remote sensing technology. Although field sampling can obtain accurate water quality data, including algae species and quantity, this method is time-consuming and labor-intensive, and cannot achieve large-scale continuous monitoring. Satellite remote sensing technology can cover a wider area and infer the distribution of algal blooms by analyzing reflectance spectra, but this method has a certain time delay, and its data update frequency and cloud cover and other factors limit its real-time monitoring capabilities. In addition, these traditional technologies often have difficulty in meeting the requirements of real-time monitoring and rapid response, and cannot meet the urgent needs of environmental protection and water management.

[0004] To overcome these limitations, researchers have begun to explore new methods for real-time algal bloom assessment that integrate multiple data sources. Specifically, image processing technology is used to identify and measure the density and distribution of algal blooms. Image data is combined with data from other sensors (such as dissolved oxygen, temperature, light intensity sensors, etc.) and can be analyzed through machine learning models to predict the development trend and outbreak point of algal blooms. The implementation of these methods requires a powerful data processing platform and efficient algorithm optimization to ensure real-time processing and analysis of data.

[0005] However, there are still many challenges in integrating these methods for real-time algal bloom assessment. The effective integration and synchronization of different data sources requires the development of new data fusion technologies and models to improve the accuracy and efficiency of assessment. In addition, underwater video technology can more comprehensively reflect the growth status of algal blooms and their environmental factors by capturing direct images of underwater algae, and can be well utilized to achieve accurate assessment of algal blooms. Summary of the invention

[0006] The purpose of the present invention is to overcome the shortcomings of the above-mentioned prior art and to provide a multi-scale collaborative algal bloom assessment method that integrates underwater videos, with the intention of aligning and integrating multi-scale images and monitoring data of underwater, water surface and high altitude in time and space, and using interpolation to complete the missing data, making a preliminary judgment through image segmentation method, and then constructing time series features, establishing a Transformer model to correct the segmented image, so as to achieve accurate identification of algal bloom phenomenon and its development.

[0007] In order to achieve the above object, the present invention provides the following technical solutions:

[0008] A multi-scale collaborative algal bloom assessment method integrating underwater video includes the following steps:

[0009] S1: Take underwater images, surface images, and aerial images of the monitoring area according to the sampling time to obtain an image set;

[0010] S2: Collect water quality data and meteorological data in the monitoring area according to the sampling time;

[0011] S3: Align the image collection, water quality data, and meteorological data of the monitoring area in time and space;

[0012] S4: using image segmentation technology to perform binary mask segmentation of the algal bloom area on the aligned image set one by one to obtain a binary mask for each image;

[0013] S5: The binary mask of each image is vectorized and concatenated with the water quality data and meteorological data as the input of the Transformer model at the corresponding time;

[0014] S6: Use the Transformer model to predict the binary mask of each image at the next moment;

[0015] S7: The binary masks of each image are fused using a weighted summation method to obtain the binary mask of the algal bloom area and the fused algal bloom coverage ratio of the monitored area at the corresponding moment.

[0016] Furthermore, the underwater image described in step S1 is a high-definition image parallel to the water surface taken by an underwater infrared camera, and the surface image is taken by a fixed high-resolution camera or a radar installed on a drone; the high-altitude image is a satellite optical remote sensing image.

[0017] Preferably, for the high-altitude image, an interpolation method can be used to complete the data of areas blocked by clouds.

[0018] Furthermore, the water quality data described in step S2 are data measured by a temperature sensor, a dissolved oxygen sensor, a pH sensor, and a nitrogen and phosphorus concentration sensor, including: water temperature (°C), pH (dimensionless), dissolved oxygen (mg / L), conductivity (μS / cm), turbidity (NTU), permanganate index (mg / L), ammonia nitrogen (mg / L), total phosphorus (mg / L), total nitrogen (mg / L), chlorophyll a (ug / L); the meteorological data include: temperature (°C), precipitation (mm), wind speed (m / s), relative humidity (%), and air pressure (Pa).

[0019] Further, the step S3 is specifically as follows:

[0020] S301: using a timestamp filter to select all images and data of the monitoring area corresponding to time T from the image set, water quality data and meteorological data as images and data to be analyzed;

[0021] S302: Mapping the image to be analyzed and the data to be analyzed to a unified coordinate system using a geographic coordinate system, and adjusting the image to be analyzed to the same resolution using a bilinear interpolation method or a nearest neighbor interpolation method.

[0022] Further, the step S4 is specifically as follows:

[0023] S401: Build a MobileNet or EfficientNet model;

[0024] S402: Input the images to be analyzed one by one, extract the regions of interest, and output a binary mask of the algal bloom, where 1 represents the algal bloom area and 0 represents the background;

[0025] S403: Calculate the ratio of the algal bloom area to the entire image to be analyzed, which is the corresponding algal bloom coverage ratio.

[0026] Furthermore, before the processing in step S4, the image to be analyzed needs to be enhanced.

[0027] Preferably, the enhancement process for the underwater image is as follows: (1) decomposing the red, green and blue channels of the underwater image;

[0028] Adopt Multi-Scale Retinex with Color Restoration (MSRCR) algorithm,

[0029] Enhance the contrast of the image; (2) For the enhanced image, use fully convolutional GAN ​​to achieve image detail restoration and color correction.

[0030] Furthermore, the Transformer model described in step S5 is composed of a Transformer network whose output layer is a softmax layer and a binary mask segmentation function connected in series with it, and the output value is 0 or 1; wherein the binary mask segmentation function Among them, w is the weight coefficient to be trained, and x is the output of the softmax layer.

[0031] Preferably, the binary mask of the image described in step S5 is vectorized by sampling the binary mask of the image in a sliding window manner, stretching each sliding window into a vector, and concatenating the vectors of the sliding windows in sequence.

[0032] Further, the step S7 is specifically as follows:

[0033] S701: Use the weighted summation method to fuse the binary masks of each image at time t to obtain the binary mask of the algal bloom area in the monitoring area at time t. Among them, ω i is the binary mask X of the i-th image i (t) the corresponding weight;

[0034] S702: Count the number of pixels with a value of 1 in the binary mask X(t) of the algal bloom area in the monitoring area at time t, and fuse the algal bloom coverage ratio as the ratio of the number of pixels with a value of 1 to the total number of pixels.

[0035] Preferably, the weight ω described in step S701 is i Set by experience from an expert knowledge base.

[0036] The beneficial effects of the present invention are: providing a multi-scale collaborative algal bloom assessment method that integrates underwater videos, using image segmentation technology to perform binary segmentation on underwater images, surface images, and aerial images of the monitoring area, and then aligning them with water quality data and meteorological data in time and space dimensions, and using the Transformer model to realize the calculation and prediction of the binary mask of the algal bloom area and the fused algal bloom coverage ratio in the monitoring area at multiple times, which is conducive to the dynamic and accurate assessment of the algal bloom phenomenon. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to make the purpose and technical solution of the present invention more clear, the present invention provides the following drawings for explanation:

[0038] Figure 1 This is a flow chart of a multi-scale collaborative algal bloom assessment method integrating underwater video according to the present invention;

[0039] Figure 2 This is a diagram of the Transformer model architecture in Example 1 of the present invention. DETAILED DESCRIPTION

[0040] In order to make the purpose and technical solution of the present invention more clearly understood, the present invention is described in detail below with reference to the accompanying drawings and embodiments.

[0041] Embodiment 1: In order to realize real-time monitoring and dynamic assessment and early warning of the blue algae bloom phenomenon in the waters of Dianchi Lake, there are high-definition high-altitude images taken by existing satellite optical remote sensing, multiple water surface images taken by radar installed by high-resolution cameras and drones flying in the air covering paths, and underwater images parallel to the water surface taken by infrared cameras by unmanned boats traveling on the water surface covering paths; for this scenario, the present invention provides "a multi-scale collaborative algae bloom assessment method integrating underwater videos".

[0042] Combination Figure 1 , including the following steps:

[0043] S1: Take underwater images, surface images, and aerial images of the monitoring area according to the sampling time to obtain an image set.

[0044] S2: Collect water quality data and meteorological data in the monitoring area according to the sampling time.

[0045] The water quality data are data measured by temperature sensors, dissolved oxygen sensors, pH sensors, and nitrogen and phosphorus concentration sensors, including: water temperature (°C), pH (dimensionless), dissolved oxygen (mg / L), conductivity (μS / cm), turbidity (NTU), permanganate index (mg / L), ammonia nitrogen (mg / L), total phosphorus (mg / L), total nitrogen (mg / L), chlorophyll a (ug / L); the meteorological data include: air temperature (°C), precipitation (mm), wind speed (m / s), relative humidity (%), and air pressure (Pa).

[0046] S3: Align the image collection, water quality data and meteorological data of the monitoring area in time and space, and perform image enhancement processing.

[0047] Specifically:

[0048] S301: using a timestamp filter to select all images and data of the monitoring area corresponding to time T from the image set, water quality data and meteorological data as images and data to be analyzed;

[0049] S302: Mapping the image to be analyzed and the data to be analyzed to a unified coordinate system using a geographic coordinate system, and adjusting the image to be analyzed to the same resolution using a bilinear interpolation method or a nearest neighbor interpolation method.

[0050] The specific enhancement processing for underwater images is as follows: (1) the red, green and blue channels of the underwater image are decomposed, and a multi-scale retinal enhancement algorithm with color restoration is used to enhance the image contrast; (2) for the enhanced image, a fully convolutional GAN ​​is used to restore image details and correct color.

[0051] S4: Use image segmentation technology to perform binary mask segmentation of the algae bloom area on the aligned image set one by one to obtain a binary mask for each image. Specifically:

[0052] S401: Build a MobileNet or EfficientNet model;

[0053] S402: Input the images to be analyzed one by one, extract the regions of interest, and output a binary mask of the algal bloom, where 1 represents the algal bloom area and 0 represents the background;

[0054] S403: Calculate the ratio of the algal bloom area to the entire image to be analyzed, which is the corresponding algal bloom coverage ratio.

[0055] S5: The binary mask of each image is vectorized and concatenated with the water quality data and meteorological data as the input of the Transformer model at the corresponding time.

[0056] Combination Figure 2 The Transformer model is composed of an embedding layer, a multi-head attention mechanism, a linear layer, an Add&Norm layer, a feedforward neural network layer, an Add&Norm layer, a pooling layer, a Softmax layer, and a binary mask segmentation function in sequence; the input is an image binary mask vector, which is concatenated with water quality data and meteorological data after being embedded in the embedding layer; the output value of the model is 0 or 1; among them, the binary mask segmentation function Among them, w is the weight coefficient to be trained, and x is the output of the softmax layer.

[0057] The binary mask of the image is vectorized by sampling the binary mask of the image with a 4*4 sliding window, stretching each sliding window into a 16-dimensional vector, and concatenating the vectors of the sliding windows one by one in order from left to right and from top to bottom.

[0058] S6: Use the Transformer model to predict the binary mask of each image at the next moment.

[0059] S7: Use the weighted summation method to fuse the binary masks of each image to obtain the binary mask of the algal bloom area and the fused algal bloom coverage ratio of the monitoring area at the corresponding moment. Specifically:

[0060] S701: Use the weighted summation method to fuse the binary masks of each image at time t to obtain the binary mask of the algal bloom area in the monitoring area at time t. Among them, ω i is the binary mask X of the i-th image i (t) the corresponding weight;

[0061] S702: Count the number of pixels with a value of 1 in the binary mask X(t) of the algae bloom area in the monitoring area at time t, and the fused algae bloom coverage ratio is the ratio N of the number of pixels with a value of 1 to the total number of pixels. algae .

[0062] The weight ω i Set by experience from an expert knowledge base.

[0063] S8: Calculate the coverage area A of the algal bloom based on the resolution R of the image algae :

[0064] A algae =N algae ×R

[0065] Where R is the actual area represented by each pixel.

[0066] Finally, it should be noted that the above preferred embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail through the above preferred embodiments, those skilled in the art should understand that various changes can be made in form and details without departing from the scope defined by the claims of the present invention.

Claims

1. A multi-scale collaborative algal bloom assessment method integrating underwater video, characterized in that: The steps include: S1: Take underwater images, surface images, and aerial images of the monitoring area according to the sampling time to obtain an image set; S2: Collect water quality data and meteorological data in the monitoring area according to the sampling time; S3: Align the image collection, water quality data, and meteorological data of the monitoring area in time and space; S4: using image segmentation technology to perform binary mask segmentation of the algal bloom area on the aligned image set one by one to obtain a binary mask for each image; S5: The binary mask of each image is vectorized and concatenated with the water quality data and meteorological data as the input of the Transformer model at the corresponding time; S6: Use the Transformer model to predict the binary mask of each image at the next moment; S7: using a weighted summation method to fuse the binary masks of each image, and obtaining the binary mask of the algal bloom area and the fused algal bloom coverage ratio of the monitoring area at the corresponding moment; The underwater image in step S1 is a high-definition image parallel to the water surface taken by an underwater infrared camera, and the surface image is taken by a fixed high-resolution camera or a radar installed on a drone; the high-altitude image is a satellite optical remote sensing image; The water quality data described in step S2 are data measured by a temperature sensor, a dissolved oxygen sensor, a pH sensor, and a nitrogen and phosphorus concentration sensor, including: water temperature (°C), pH (dimensionless), dissolved oxygen (mg / L), conductivity (μS / cm), turbidity (NTU), permanganate index (mg / L), ammonia nitrogen (mg / L), total phosphorus (mg / L), total nitrogen (mg / L), chlorophyll a (ug / L); the meteorological data include: temperature (°C), precipitation (mm), wind speed (m / s), relative humidity (%), and air pressure (Pa); The step S3 is specifically as follows: S301: using a timestamp filter to select all images and data of the monitoring area corresponding to time T from the image set, water quality data and meteorological data as images and data to be analyzed; S302: Mapping the image to be analyzed and the data to be analyzed to a unified coordinate system using a geographic coordinate system, and adjusting the image to be analyzed to the same resolution using a bilinear interpolation method or a nearest neighbor interpolation method; The step S4 is specifically as follows: S401: Build a MobileNet or EfficientNet model; S402: Input the images to be analyzed one by one, extract the regions of interest, and output a binary mask of the algal bloom, where 1 represents the algal bloom area and 0 represents the background; S403: Calculate the ratio of the algal bloom area to the entire image to be analyzed, which is the corresponding algal bloom coverage ratio; The Transformer model described in step S5 is composed of a Transformer network whose output layer is a softmax layer and a binary mask segmentation function connected in series with it, and the output value is 0 or 1; wherein the binary mask segmentation function Among them, w is the weight coefficient to be trained, and x is the output of the softmax layer; The step S7 is specifically as follows: S701: Use the weighted summation method to fuse the binary masks of each image at time t to obtain the binary mask of the algal bloom area in the monitoring area at time t. Among them, ω i is the binary mask X of the i-th image i (t) the corresponding weight; S702: Count the number of pixels with a value of 1 in the binary mask X(t) of the algal bloom area in the monitoring area at time t, and fuse the algal bloom coverage ratio as the ratio of the number of pixels with a value of 1 to the total number of pixels.

2. The multi-scale collaborative algal bloom assessment method integrating underwater video according to claim 1 is characterized in that: For the high-altitude images, interpolation method can be used to complete the data in areas blocked by clouds.

3. The multi-scale collaborative algal bloom assessment method integrating underwater video according to claim 1 is characterized in that: Before the processing in step S4, the image to be analyzed needs to be enhanced.

4. The multi-scale collaborative algal bloom assessment method integrating underwater video according to claim 3 is characterized in that: The specific enhancement processing for underwater images is as follows: (1) The red, green and blue channels of the underwater image are decomposed, and the Multi-Scale Retinex with Color Restoration (MSRCR) algorithm is used to enhance the image contrast; (2) For the enhanced image, a fully convolutional GAN ​​is used to restore image details and correct color.

5. The multi-scale collaborative algal bloom assessment method integrating underwater video according to claim 1 is characterized in that: The binary mask vectorization of the image in step S5 is to sample the binary mask of the image in a sliding window manner, stretch each sliding window into a vector, and concat the vectors of the sliding windows in sequence.

6. The multi-scale collaborative algal bloom assessment method integrating underwater video according to claim 1 is characterized in that: The weight ω described in step S701 i Set by experience from an expert knowledge base.

Images