A method and system for ocean dynamic monitoring based on fused images

Abstract

The application provides a kind of ocean dynamic monitoring method and system based on fusion image, comprising: collecting the RGB image and short wave infrared SWIR image of target marine area, image registration is carried out to RGB image and SWIR image, and multivariate data set is constructed;Improved BEMD is used to carry out spatial decomposition to two-dimensional image in multivariate data set, improved MEMD is used to carry out joint decomposition to multichannel image in multivariate data set, and a plurality of eigenmode function images are obtained;A plurality of eigenmode function images are calculated, and the wave direction, wave height and ocean current velocity of sea wave are obtained.The application combines the principle of ocean dynamics, uses improved BEMD method and MEMD to decompose ocean image and infrared image, accurately calculates the wave and ocean current characteristics of ocean surface, and provides reliable data support for marine environment monitoring, shipping safety, ocean resource development and other fields.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and in particular to a marine dynamic monitoring method and system based on fused images. BACKGROUND

[0002] In marine environment monitoring, real-time and accurate acquisition of sea surface dynamic characteristics such as waves, ocean currents, wind speed, etc. is of great significance to marine resource development, environmental protection, shipping safety, etc. However, due to the influence of complex weather and sea conditions, traditional single sensor monitoring methods often face problems such as limited coverage and insufficient accuracy, making it difficult to fully capture sea surface dynamic information, thereby affecting the safety and efficiency of marine activities.

[0003] Visible light image sensors can provide high-resolution sea surface images under good weather conditions, helping to accurately obtain visual information of marine dynamics such as waves and swells. However, in adverse weather conditions (such as fog, dust, etc.), the performance of visible light images declines significantly, resulting in blurred images that cannot accurately reflect the sea surface state. At the same time, short-wave infrared (SWIR) images have good fog-penetrating ability and can penetrate fog or other atmospheric interference to obtain relatively clear sea surface images in low-visibility environments. However, the resolution of short-wave infrared images is lower than that of visible light images, and the image details are not as good, which may not provide enough information to accurately assess the sea conditions.

[0004] Therefore, using visible light images or short-wave infrared images alone has certain limitations. In adverse weather conditions, a single sensor cannot provide complete and accurate marine information, and traditional sensor data fusion often faces problems such as information redundancy and noise interference. SUMMARY

[0005] The present application provides a marine dynamic monitoring method and system based on fused images to solve the defects of using visible light images or short-wave infrared images alone in marine environment monitoring.

[0006] In a first aspect, the present application provides a marine dynamic monitoring method based on fused images, comprising:

[0007] Collecting RGB images and short-wave infrared (SWIR) images of a target marine area, performing image registration on the RGB images and the SWIR images, and constructing a multivariate data set;

[0008] Performing spatial decomposition on two-dimensional images in the multivariate data set using an improved BEMD, and performing joint decomposition on multi-channel images in the multivariate data set using an improved MEMD to obtain a plurality of intrinsic mode function images;

[0009] The wave direction, wave height and ocean current velocity of the sea wave are obtained by calculating the several intrinsic modal function images.

[0010] According to the ocean dynamic monitoring method based on fused images provided by the application, the RGB image and the SWIR image of the target ocean area are collected, the RGB image and the SWIR image are image-registered, and before a multivariate data set is constructed, the method further comprises the following steps of:

[0011] The high-definition visible light and infrared camera is installed at the belly position of the seaplane, and the collected images are transmitted to the inside of the plane in real time through the data transmission system.

[0012] According to the ocean dynamic monitoring method based on fused images provided by the application, the RGB image and the SWIR image of the target ocean area are collected, the RGB image and the SWIR image are image-registered, and before a multivariate data set is constructed, the method further comprises the following steps of:

[0013] The RGB image and the SWIR image are synchronously collected based on a fixed time interval;

[0014] The RGB image and the SWIR image are image-registered by using inertial measurement unit data and global positioning system data, and an aircraft attitude matrix and an aircraft position vector are established;

[0015] Image space registration coordinates are obtained according to the aircraft attitude matrix, the aircraft position vector and pixel coordinates of all images;

[0016] The size of the registered image is adjusted, so that all channel images have the same resolution and size;

[0017] The three channels of the adjusted RGB image and the SWIR image are combined to form a four-channel two-dimensional multivariate data set.

[0018] According to the ocean dynamic monitoring method based on fused images provided by the application, the two-dimensional images in the multivariate data set are spatially decomposed by using an improved BEMD, and the multichannel images in the multivariate data set are jointly decomposed by using an improved MEMD, and several intrinsic modal function images are obtained, comprising the following steps of:

[0019] Step 1, local maximum points and local minimum points are determined in the two-dimensional images of each channel in the multivariate data set, and a smoothing filter is used for pretreatment;

[0020] Step 2, a multi-dimensional direction vector set is generated by uniformly sampling on a high-dimensional spherical surface;

[0021] Step 3, projecting each direction vector in the set of multi-dimensional direction vectors in any channel and the pixel value of each pixel position in any channel to obtain a projected one-dimensional signal;

[0022] Step 4, using a two-dimensional interpolation method to interpolate the local maximum points and the local minimum points in the projected one-dimensional signal to obtain an upper envelope and a lower envelope;

[0023] Step 5, based on the upper envelope, the lower envelope and the total number of direction vectors, averaging the envelopes of all directions at each pixel position to obtain an average envelope with the same spatial dimension as the original signal;

[0024] Step 6, calculating the residual of each channel from the average envelope of each channel, the average value of any component of the direction vector and the pixel value of each pixel position in any channel, and if the normalized mean square error of each channel residual is less than a preset threshold, the first eigenmode function image is extracted, and the residual is updated from the first eigenmode function image and the pixel value of each pixel position in any channel;

[0025] Step 7: updating the remaining signal from the updated residual and returning to step 1 for iteration to extract the next eigenmode function image;

[0026] Step 8: repeating steps 1 to 7 until the stopping condition is met to obtain the plurality of eigenmode function images, wherein the plurality of eigenmode function images include high-frequency eigenmode function images, low-frequency eigenmode function images and residual components;

[0027] The stopping condition includes that the updated residual does not include local maximum points and local minimum points, the preset number of eigenmode function decomposition layers is reached, and the residual signal energy is lower than a preset energy threshold.

[0028] According to the ocean dynamic monitoring method based on the fusion image provided by the application, the plurality of eigenmode function images are calculated to obtain the wave direction and the wave height of the sea wave, comprising:

[0029] Performing two-dimensional fast Fourier transform on any determined eigenmode function image to obtain a frequency spectrum signal;

[0030] Calculating the wave number and the wave direction of the sea wave from the amplitude and the phase of the frequency spectrum signal;

[0031] According to the linear wave theory, the wavelength is obtained from the wave number, and the wave height is estimated by the image brightness change or the ocean dynamics model.

[0032] According to the ocean dynamic monitoring method based on fusion images provided by the application, the ocean current velocity of the sea wave is obtained by calculating the plurality of intrinsic mode function images, and the method comprises the following steps:

[0033] The flight height of the unmanned aerial vehicle, the size of the sea surface area, the horizontal resolution and the vertical resolution of the plurality of intrinsic mode function images are obtained, and the field of view angle of the camera is determined;

[0034] The pixel size is obtained from the horizontal resolution, the vertical resolution, the flight height of the unmanned aerial vehicle, the size of the sea surface area and the field of view angle of the camera;

[0035] The time interval of each frame of image is determined, the x-direction velocity and the y-direction velocity are calculated by the optical flow method, the x-direction actual velocity is obtained according to the x-direction velocity, the pixel size and the time interval of each frame of image, and the y-direction actual velocity is obtained according to the y-direction velocity, the pixel size and the time interval of each frame of image;

[0036] The x-direction actual velocity and the y-direction actual velocity are integrated to obtain the ocean current velocity.

[0037] In the second aspect, the application further provides an ocean dynamic monitoring system based on fusion images, which comprises:

[0038] The registration module is used for collecting the RGB image and the SWIR image of the target ocean area, performing image registration on the RGB image and the SWIR image, and constructing a multivariate data set;

[0039] The decomposition module is used for performing spatial decomposition on the two-dimensional images in the multivariate data set by using the improved BEMD, and performing joint decomposition on the multi-channel images in the multivariate data set by using the improved MEMD, to obtain a plurality of intrinsic mode function images;

[0040] The calculation module is used for calculating the plurality of intrinsic mode function images to obtain the wave direction, the wave height and the ocean current velocity of the sea wave.

[0041] In the third aspect, the application further provides an electronic device, which comprises a memory, a processor and a computer program stored in the memory and capable of running on the processor, and the processor implements the ocean dynamic monitoring method based on fusion images according to any one of the above aspects when executing the program.

[0042] In the fourth aspect, the application further provides a non-transitory computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the ocean dynamic monitoring method based on fusion images according to any one of the above aspects.

[0043] Compared with the prior art, the application has the following technical effects:

[0044] (1) By combining the principles of ocean dynamics, the improved BEMD and MEMD combination method is used to decompose the image of the ocean surface and extract the characteristics of waves and ocean currents of different scales. BEMD can effectively capture the high and low frequency components of ocean waves through the decomposition of two-dimensional images, and identify the wave characteristics of different scales such as small ripples and large waves. MEMD can make multiple image channels (such as RGB and SWIR images) be processed in time and space, further improving the accuracy of wave and current monitoring. This method provides more accurate and reliable data support for real-time monitoring of ocean dynamics, which can effectively predict sea surface fluctuations and flow conditions, support ocean environment monitoring, shipping safety, marine resource development and other fields;

[0045] (2) Through the multi-channel data fusion of RGB images and short-wave infrared images, the invention effectively overcomes the limitations of single image channel in ocean dynamic monitoring. Using the improved BEMD and MEMD algorithm, the invention can jointly decompose multi-channel image data, extract IMF components containing different information, and further calculate the wave direction, wave height and current speed. This multi-channel image processing technology not only improves the recognition ability of ocean surface details, but also enhances the monitoring ability of complex marine environment through accurate data fusion, especially in the changing sea conditions, it can provide more comprehensive and reliable information;

[0046] (3) The optical flow method is used in combination with the height and image resolution of the unmanned aerial vehicle to accurately estimate the flow rate of the ocean surface current. By analyzing the pixel displacement between image sequences and combining the IMU and GPS data of the unmanned aerial vehicle, the background interference caused by the aircraft motion to the image is successfully removed, ensuring the accurate calculation of the current speed. This method can estimate the real-time flow rate according to the dynamic changes of the sea surface, providing efficient and dynamic adaptive ability for ocean dynamic monitoring, which is suitable for complex marine environment monitoring such as shipping navigation and marine resource exploration. BRIEF DESCRIPTION OF DRAWINGS

[0047] In order to more clearly illustrate the technical solutions in the present application or prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings described below are some embodiments of the present application, and those skilled in the art can obtain other drawings according to these drawings without creative labor.

[0048] Figure 1 is the flowchart of the ocean dynamic monitoring method based on fused images provided by the present application;

[0049] Figure 2It is an improved BEMD and MEMD ocean current image decomposition schematic diagram provided by the application;

[0050] Figure 3 It is a structural schematic diagram of the ocean dynamic monitoring system based on fused images provided by the application;

[0051] Figure 4 It is a structural schematic diagram of the electronic device provided by the application. DETAILED DESCRIPTION

[0052] To make the purpose, technical scheme and advantages of the application clearer, the technical scheme in the application will be described clearly and completely below in combination with the drawings in the application. Obviously, the described embodiments are part of the embodiments of the application, rather than all the embodiments. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor belong to the protection scope of the application.

[0053] In view of the many limitations of the prior art, the application fuses short-wave infrared and visible light images in the field of ocean monitoring, combines the respective advantages of the two, and can effectively improve the accuracy and reliability of ocean dynamic monitoring. Through multi-modal fusion processing of these two kinds of image data, not only the stability of monitoring can be ensured under different climate conditions, but also the recognition ability of complex sea conditions can be improved, thereby providing more accurate sea condition evaluation and decision support for the fields of ocean environment monitoring, shipping safety, ocean resource development, etc. In addition, sea surface infrared images have wide innovative applications in ocean environment monitoring. For example, in the field of ocean pollution monitoring, infrared images can identify sea surface temperature anomalies to help detect environmental problems such as oil spills. In the management of fishery resources, sea surface temperature distribution is closely related to fish activity, and infrared images can assist in the investigation and management of fishery resources. Through the fusion of infrared and visible light images, the accuracy of marine weather forecasting can also be improved to support emergency response activities such as maritime rescue.

[0054] Figure 1 It is a flowchart of the ocean dynamic monitoring method based on fused images provided by the embodiment of the application, as shown in Figure 1 , comprising:

[0055] Step 100: acquiring an RGB image and a short-wave infrared SWIR image of a target ocean area, performing image registration on the RGB image and the SWIR image, and constructing a multivariate data set;

[0056] Step 200: performing spatial decomposition on the two-dimensional images in the multivariate data set by using an improved BEMD, and performing joint decomposition on the multichannel images in the multivariate data set by using an improved MEMD, to obtain a plurality of intrinsic mode function images;

[0057] Step 300: Calculate the number of intrinsic mode function images to obtain the wave direction, wave height and ocean current speed of the sea wave.

[0058] It should be noted that in the present application, the use of sea surface infrared images plays a key role in improving the accuracy and reliability of ocean dynamic monitoring. The principle of infrared imaging is based on the thermal radiation emitted by objects due to their temperature, and the ocean surface emits infrared radiation according to its temperature and emissivity. According to the Stefan-Boltzmann law, the total radiant exitance of a black body is: where, is the sea surface emissivity (about 0.98 for seawater), is the Stefan-Boltzmann constant, is the absolute temperature (in Kelvin). By capturing sea surface infrared images, changes in sea surface temperature (SST) can be detected, which is of great significance for monitoring ocean phenomena such as ocean currents, upwelling and temperature fronts. These thermal anomalies are crucial for ocean dynamics monitoring, as they affect wave formation, ocean current patterns, and can be related to events such as oil spills or pollutant emissions. In addition, infrared images, due to their ability to detect longer wavelength radiation, are less affected by fog, haze or low light conditions, thus enhancing the monitoring system's ability to operate effectively in various environmental conditions.

[0059] Specifically, the embodiments of the present application first install equipment, install high-definition visible light cameras and infrared cameras at the belly position of the sea plane, ensuring that the fields of view of both cover the ocean surface below the plane. The specific installation method is as follows:

[0060] High-resolution RGB cameras and short-wave infrared (SWIR) cameras are selected to obtain image data of the sea surface under normal visibility and low visibility (such as fog, haze, sandstorm weather, etc.). When installing the cameras, their optical axes need to be perpendicular or close to perpendicular to the flight direction of the plane to ensure that the sea surface images can be captured in real time and stably.

[0061] To combine an RGB sea wave image (containing red, green and blue channels) and a short-wave infrared (SWIR) image of the same area, as shown in Figure 2 , an improved decomposition method using bidimensional empirical mode decomposition (BEMD) and multivariate empirical mode decomposition (MEMD) is used to obtain a set of intrinsic mode function (IMF) images representing high-frequency, low-frequency and residual components in the sea wave. The following detailed steps are required:​

[0062] I. Data Image Registration

[0063] The camera is fixed under the belly of the plane and is securely mounted through a directional support to avoid being affected by plane vibrations or air flow during flight. The positions and angles of the two cameras are precisely adjusted to ensure that the image acquisition areas overlap and can completely cover the sea surface below the plane.

[0064] A high-speed data transmission system is installed to transmit image data captured by the camera to the data processing unit inside the plane in real time.

[0065] During the cruise of the plane, a set of infrared and visible light images are acquired every 0.5 seconds. To ensure data consistency, the images of the camera are synchronized in time and space using Global Positioning System (GPS) and Inertial Measurement Unit (IMU) data during each data acquisition period, ensuring complete alignment of infrared and visible light images in time. Every 0.5 seconds, the plane camera synchronously acquires a set of images with a resolution of 1920x1080 pixels. This resolution ensures that the details of the image are fully presented, especially suitable for accurately monitoring sea surface fluctuations and ocean current characteristics.

[0066] Spatial alignment: Ensure that the RGB image and the SWIR image are completely aligned in space. If there is a displacement, rotation, or scale difference, image registration needs to be performed using the Inertial Measurement Unit (IMU) data and Global Positioning System (GPS) data of the plane. Establish the attitude matrix and position vector of the plane to spatially register the images.

[0067] where, is the pixel coordinate.

[0068] Size uniformity: Adjust the image size so that all channels have the same resolution and size.

[0069] Multivariate data construction: Combine the three channels of the RGB image and the SWIR image into a four-channel multivariate two-dimensional data set.

[0070]

[0071] II. Improved BEMD Decomposition Method and MEMD Decomposition Method

[0072] By using improved BEMD and MEMD decomposition method, infrared image and visible light image are fused, and the advantages of the two kinds of images are fully utilized. The data fusion makes it possible to extract detailed spatial features and accurate thermal information, so as to more comprehensively understand the sea surface dynamics.

[0073] Among them, the goal of BEMD is to decompose two-dimensional images into several IMF, extract spatial features of different scales, and the method usually used is to find local extreme points in two-dimensional space, and construct envelope by two-dimensional interpolation; The goal of MEMD is to jointly decompose multivariate signals, and ensure that the IMFs of different channels are aligned in time or space, and the method usually used is to use multidimensional direction vector to project multivariate data to one-dimensional space and jointly decompose. Specifically, it includes the following steps:

[0074] Step 1: Detection of local extreme points

[0075] Objective: Find local maximum and minimum points in two-dimensional images of each channel.

[0076] For each channel Local maximum satisfies , and local minimum satisfies , . Because noise may cause too many extreme points, smoothing filtering (such as Gaussian filtering) can be used for pretreatment.

[0077] Step 2: Generate multidimensional direction vector

[0078] Objective: Generate a set of direction vectors uniformly distributed in high-dimensional space (corresponding to the number of channels) for data projection. Direction vector generation method: uniformly sample on a four-dimensional sphere to get a set of direction vectors , where , satisfies , and Hammersley point set or Sobol sequence can be used for uniform sampling.

[0079] Step 3: Data projection

[0080] Objective: Project multivariate two-dimensional data along each direction vector to get one-dimensional signal.

[0081] Projection formula: for each pixel position and each direction vector :

[0082] Where, is the component of direction vector in the first channel. is the first pixel value of the channel.

[0083] Step 4: Envelope estimation

[0084] Objective: Estimate the envelope of the projected data , and get the upper envelope and lower envelope.

[0085] Method: Two-dimensional interpolation: For the local maximum and minimum points of the projected data, use two-dimensional interpolation method to construct the upper envelope and lower envelope . The interpolation method can be selected from bicubic interpolation, thin plate spline (TPS), radial basis function (RBF), etc.

[0086] Step 5: Average envelope calculation

[0087] Objective: Average the envelopes of all directions to get the average envelope of each channel.

[0088] Average envelope calculation: , is the total number of direction vectors. Since the projection is performed in a multi-dimensional space, the average envelope should maintain the same spatial dimension as the original signal.

[0089] Step 6: Extract IMF from original data

[0090] Residual calculation: For each channel , there is , is the average value of the th component of the direction vector.

[0091] Stop criterion check: If the normalized mean square error SD is less than a preset threshold (such as 0.2), it is considered that the IMF extraction is completed. The first IMF is . Update the residual: .

[0092] Step 7: Residual calculation

[0093] Objective: Take the remaining signal as new data and continue to extract the next IMF.

[0094] Update data: , return to step 1 for loop, and continue to extract the next IMF using the updated .

[0095] Step 8: Iterative screening, repeat the above steps until the stop condition is met:

[0096] Residual is monotonic function: when residual signal contains no local extremum point anymore. The preset number of IMF decomposition is reached. The energy of residual signal is lower than a certain energy threshold.

[0097] The high-frequency IMF of a frame of image is finally obtained: representing the details and high-frequency changes in the sea waves (such as small ripples).

[0098] Low-frequency IMF: representing larger structures and low-frequency changes (such as large waves, swells).

[0099] Residual component: representing the overall trend or background brightness of the image.

[0100] III. Wave direction and wave height calculation for images decomposed at a single time

[0101] Perform two-dimensional FFT on the selected IMF component to obtain the frequency spectrum .

[0102]

[0103] By analyzing the amplitude and phase information of the frequency spectrum, the wave number and direction of the sea waves are calculated.

[0104] According to the linear wave theory, the relationship between the wavelength and the wave number , the wave height can be estimated by the image brightness changes or combined with ocean dynamics models.

[0105] IV. Ocean current velocity calculation

[0106] For the modal decomposition image with high-frequency part removed, the optical flow method is used to calculate the ocean current velocity on the sea surface. When combined with the height of the unmanned aerial vehicle to accurately estimate the ocean current velocity, the following factors need to be considered: the flight height of the unmanned aerial vehicle, the image resolution, the scale of the optical flow calculation, and the flow characteristics on the sea surface.

[0107] (1) Relationship between the height of the unmanned aerial vehicle and the size of the image

[0108] The flight height of the unmanned aerial vehicle determines the size of the ocean surface area in the captured image. When the flight height of the unmanned aerial vehicle is high, the size of the captured sea surface area is large, and when the flight height is low, the size of the captured sea surface area is small. Therefore, the height of the unmanned aerial vehicle directly affects the spatial resolution of the image and the scale of the optical flow calculation result. Assuming that the flight height of the unmanned aerial vehicle is , the horizontal and vertical resolutions of the image are and Then the size of the sea surface area represented by each pixel in the image can be calculated as: where the camera field of view (FOV) is the horizontal or vertical field of view of the camera, which affects the field of view of the image.

[0109] The optical flow method calculates the displacement of pixels in the image, in units of pixels per frame (px / frame). To convert this displacement into actual physical units (e.g., meters per second), the pixel displacement of the image needs to be related to the height of the drone, the spatial resolution of the image, and the time interval between frames. Assuming the time interval between frames is then the velocity components calculated by optical flow and can be converted to actual velocity (units: meters / second) by the following formula: , .

[0110] (2) Calculate the ocean current

[0111] The Farneback optical flow algorithm is used to calculate the motion between image sequences. This algorithm is a multi-scale pyramid-based algorithm that can estimate the displacement vector of each pixel and is suitable for processing motion estimation between consecutive frames. The formula is expressed as: where: is the pixel value of the image at time , and are the horizontal and vertical components of the optical flow, is the time interval.

[0112] To estimate the actual flow rate of the ocean current, the calculation results of the optical flow need to be considered in combination with the motion of the aircraft itself (pitch, yaw, roll). The motion of the aircraft will cause the background motion in the image, so first, the part of the sea surface flow needs to be separated from the optical flow to remove the background displacement caused by the motion of the drone itself. The IMU (Inertial Measurement Unit) and GPS data on the drone can be used to estimate the actual speed and position change of the drone, so that the displacement caused by the motion of the drone can be removed from the optical flow. According to the speed, azimuth angle, and field of view angle of the drone, motion compensation is performed to remove the pseudo-optical flow caused by the motion of the drone. The calculated optical flow velocity vectors and represent the relative motion of objects on the sea surface after removing the background motion. Using the height of the drone and the camera field of view, these pixel-level optical flow velocities can be converted to the flow rate of the ocean current on the sea surface. Finally, the size of the flow rate can be calculated by the following formula: .

[0113] The application provides a marine dynamic monitoring method based on short-wave infrared and visible light image fusion, combines marine dynamics principles, uses an improved BEMD method and MEMD (decomposes marine images and infrared images, and accurately calculates wave and ocean current characteristics of a marine surface, thereby providing reliable data support for marine environment monitoring, shipping safety, marine resource development and the like.

[0114] In one embodiment, detection is performed for marine oil spillage.

[0115] An oil tanker leakage accident occurs in a certain sea area, a large amount of crude oil is leaked into the sea surface, and a large area of oil pollution is formed. The rapid spread of oil pollution poses a serious threat to the marine ecological environment, marine life and coastal economy. In order to timely control pollution and reduce losses, it is urgent to quickly and accurately detect and locate the spread range of oil pollution and provide a scientific basis for emergency disposal. The implementation process includes:

[0116] (1) Data acquisition

[0117] Visible light image acquisition: a high-resolution visible light camera (such as a multispectral camera) is used to take aerial photographs of the accident sea area, and real-time RGB visible light images are obtained. The visible light image can provide information on the apparent characteristics of the sea surface, but it is difficult to accurately distinguish the oil pollution area using the visible light image alone because the reflectivity of oil pollution and seawater in the visible light spectrum may be close;

[0118] Infrared image acquisition: a short-wave infrared (SWIR) or medium-wave infrared (MWIR) sensor is used to obtain infrared images of the accident sea area. Since there are differences in the radiation characteristics of oil pollution and seawater in the infrared band, the oil pollution area will exhibit temperature anomalies (usually as a lower temperature area) in the infrared image, which is conducive to identifying oil pollution.

[0119] (2) Data preprocessing

[0120] Image registration: a feature point matching algorithm (such as SIFT, SURF) or the IMU and GPS data of the unmanned aerial vehicle / satellite platform is used to perform spatial registration of the visible light image and the infrared image, so as to ensure that the two images are aligned in the same coordinate system.

[0121] Registration formula: , wherein, is the original image coordinate, is the registered image coordinate, is the transformation matrix.

[0122] Image correction: the infrared image is subjected to radiation correction to eliminate the influence of the atmosphere and obtain the true sea surface temperature distribution. The visible light image is subjected to color correction and brightness adjustment to enhance the image contrast.

[0123] (3) Multi-source data fusion

[0124] Multivariate data construction: Combine the red, green, and blue channels of the visible light image with the infrared image to construct a four-channel multivariate two-dimensional data set: ;

[0125] Improved BEMD and MEMD decomposition: IMF extraction: Apply improved BEMD and MEMD methods to extract intrinsic mode function (IMF) components of different scales. These IMF components contain high-frequency details (such as texture, edge) and low-frequency trends (such as brightness, temperature distribution) of the image. Through joint decomposition, the spatial consistency of each channel IMF component is ensured, facilitating the fusion and comparison of multi-source information.

[0126] (4) Oil spill feature extraction

[0127] High-frequency feature analysis: Oil spill areas exhibit different texture characteristics from surrounding seawater in high-frequency IMF components. Use texture analysis methods (such as gray level co-occurrence matrix, local binary pattern) to extract texture features in high-frequency IMF components to identify the detailed structure of oil spills;

[0128] Low-frequency feature analysis: In the low-frequency IMF component, the temperature distribution of the oil spill area is different from that of the sea water. Analyze the low-frequency IMF component of the infrared channel to extract temperature anomaly features and further confirm the location of the oil spill.

[0129] (5) Oil spill detection and identification

[0130] Feature fusion: Fuse the features of high-frequency and low-frequency IMF components to form a comprehensive feature vector.

[0131] Fusion formula: , where is the weight coefficient, is the th IMF component.

[0132] Classification and segmentation: Use machine learning algorithms such as support vector machine (SVM), random forest (RF), or convolutional neural network (CNN) to establish an oil spill detection model. Input the fusion feature into the model to output a probability map of the oil spill area;

[0133] Threshold segmentation: Set a threshold to convert the probability map into a binary mask and extract the oil spill area.

[0134] (6) Result generation and application

[0135] ​Oil spill distribution map generation: Visualize the detected oil spill areas in a Geographic Information System (GIS) to generate an oil spill distribution map. Label the area, location, concentration, and other information of the oil spill;

[0136] Emergency decision support: Combine marine environmental data (such as sea currents, wind speed) to predict the trend of oil spill diffusion. Provide scientific basis for emergency response departments to develop oil spill cleanup strategies and resource allocation plans.

[0137] (7) Effect evaluation and optimization

[0138] Accuracy verification: Verify the detection results through field sampling and visual observation to evaluate the accuracy and reliability of the model.

[0139] Model optimization: Adjust model parameters and algorithms based on verification results to continuously improve detection performance.

[0140] Through the scheme of this embodiment, it can be seen that this embodiment has the following characteristics:

[0141] (1) Deep fusion of multi-source data: Fusion of visible light and infrared images, using improved BEMD and MEMD methods to realize multi-scale and multi-dimensional extraction of oil spill features.

[0142] (2) High-precision detection: Through joint decomposition and feature fusion, the limitations of single sensor are overcome, significantly improving the accuracy and robustness of oil spill detection.

[0143] (3) Rapid response capability: Automated data processing and analysis process greatly shortens the time from data acquisition to result output, gaining valuable time for emergency disposal.

[0144] In one embodiment, red tide is monitored.

[0145] Red tide is a marine ecological disaster caused by the abnormal proliferation of phytoplankton (such as algae). The outbreak of red tide will cause seawater discoloration, release harmful substances, cause the death of fish and large marine organisms, and seriously affect the marine ecological environment and aquaculture industry. In order to reduce the harm of red tide, timely and accurate monitoring and early warning of red tide are needed. The implementation process includes:

[0146] (1) Data acquisition

[0147] Infrared image acquisition: Use mid-wave infrared (MWIR) or long-wave infrared (LWIR) sensors to acquire sea surface temperature (SST) data of the target sea area. The high density of algae in the red tide area may cause abnormal sea surface temperature, showing temperature rise or drop;

[0148] Visible light image acquisition: High-resolution visible light cameras are used to capture RGB images of the sea area, capturing changes in water color. Red tides can cause the water to appear red, brown, or other abnormal colors, and visible light images can visually reflect these changes.

[0149] (2) Data preprocessing

[0150] Image registration: Spatial registration of infrared and visible light images ensures spatial consistency of data from different sensors.

[0151] Image correction: Radiometric correction of infrared images to obtain accurate sea surface temperature. Atmospheric correction and color balance of visible light images to enhance water color information.

[0152] (3) Multi-source data fusion

[0153] Multivariate data construction: Combine the three channels of visible light images with temperature data from infrared images to form a four-dimensional data set:

[0154] where, is the sea surface temperature.

[0155] Improved BEMD and MEMD decomposition:

[0156] IMF extraction: Extract IMF components for each channel using the improved decomposition method.

[0157] Feature enhancement: In the high-frequency IMF components of the visible light channel, the water color anomaly in the red tide area is highlighted. In the low-frequency IMF components of the infrared channel, the spatial distribution characteristics of the sea surface temperature are more obvious.

[0158] (4) Red tide feature extraction

[0159] Water color anomaly detection: Calculate the normalized difference water index (NDWI) or other water color indices of visible light IMF components to identify water color anomaly areas

[0160] Temperature anomaly detection: Analyze infrared IMF components to detect abnormal changes in sea surface temperature.

[0161] Spatial superposition analysis: Superimpose water color anomaly areas and temperature anomaly areas to determine possible red tide occurrence areas.

[0162] (5) Red tide detection and early warning

[0163] Classification model establishment: Use machine learning algorithms (such as SVM, decision tree) to establish a red tide detection model, input the fused feature data, and output a red tide probability map.

[0164] ​Threshold judgment: According to the model output, set a threshold to extract the red tide area.

[0165] Red tide distribution map making: Generate a red tide distribution map, mark the range and severity of red tide, etc.

[0166] (6) Result application and feedback

[0167] Early warning information release: Release red tide early warning information to relevant departments and the public, guide fishery production and marine environmental protection work.

[0168] Trend analysis and prediction: Combine meteorological data and ocean dynamic model to analyze the development trend of red tide and predict the possible spread range.

[0169] Through the scheme of this embodiment, it can be seen that this embodiment has the following characteristics:

[0170] (1) Multi-dimensional feature detection: Fusion of temperature and water color information, using improved decomposition method, fully capturing the characteristics of red tide, improving the sensitivity of detection.

[0171] (2) Automation and real-time: Realize the automation of red tide monitoring, can quickly obtain and analyze data, provide real-time warning.

[0172] (3) High reliability: Through multi-source data fusion and advanced algorithm, reduce the false alarm and missed alarm rate, enhance the reliability of early warning information.

[0173] The fusion image-based marine dynamic monitoring system provided by the present application is described below. The fusion image-based marine dynamic monitoring system described below can be mutually corresponding and referred to the fusion image-based marine dynamic monitoring method described above.

[0174] Figure 3 is a structural diagram of the fusion image-based marine dynamic monitoring system provided by the embodiment of the present application, as Figure 3 shown, comprising: a registration module 31, a decomposition module 32 and a calculation module 33, wherein:

[0175] The registration module 31 is used for collecting the RGB image and the SWIR image of the target marine area, performing image registration on the RGB image and the SWIR image, and constructing a multivariate data set; the decomposition module 32 is used for adopting the improved BEMD to perform spatial decomposition on the two-dimensional image in the multivariate data set, and adopting the improved MEMD to perform joint decomposition on the multichannel image in the multivariate data set, to obtain a plurality of intrinsic mode function images; the calculation module 33 is used for calculating the plurality of intrinsic mode function images to obtain the wave direction, wave height and ocean current speed of the sea wave.

[0176] Figure 4An example of a schematic diagram of a physical structure of an electronic device is shown in Figure 4 As shown, the electronic device can include a processor 410, a communications interface 420, a memory 430, and a communications bus 440, wherein the processor 410, the communications interface 420, and the memory 430 complete communications with each other through the communications bus 440. The processor 410 can invoke a logic instruction in the memory 430 to execute a fusion image-based ocean dynamic monitoring method, which includes: collecting an RGB image and a short-wave infrared (SWIR) image of a target ocean area, performing image registration on the RGB image and the SWIR image, and constructing a multivariate data set; performing spatial decomposition on a two-dimensional image in the multivariate data set using an improved BEMD (Bivariate Empirical Mode Decomposition), and performing joint decomposition on a multichannel image in the multivariate data set using an improved MEMD (Multivariate Empirical Mode Decomposition) to obtain a plurality of intrinsic mode function images; and calculating the plurality of intrinsic mode function images to obtain a wave direction, a wave height, and an ocean current flow rate of the sea wave.

[0177] In addition, the logic instruction in the memory 430 described above can be implemented in the form of a software functional unit and sold or used as an independent product, and can be stored in a computer-readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or part of the technical solutions can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a plurality of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various program code storage media.

[0178] In another aspect, the present application also provides a computer program product comprising a computer program, which can be stored on a non-transitory computer readable storage medium, and the computer program, when executed by a processor, enables a computer to perform the ocean dynamic monitoring method based on fused images provided by the above-mentioned methods, which comprises: collecting an RGB image and a short-wave infrared (SWIR) image of a target ocean area, performing image registration on the RGB image and the SWIR image, and constructing a multivariate data set; performing spatial decomposition on two-dimensional images in the multivariate data set by using an improved BEMD, and performing joint decomposition on multichannel images in the multivariate data set by using an improved MEMD to obtain a plurality of intrinsic mode function images; and calculating the plurality of intrinsic mode function images to obtain a wave direction, a wave height and a current velocity of the sea wave.

[0179] In another aspect, the present application also provides a non-transitory computer readable storage medium having a computer program stored thereon, and the computer program, when executed by a processor, enables a computer to perform the ocean dynamic monitoring method based on fused images provided by the above-mentioned methods, which comprises: collecting an RGB image and a short-wave infrared (SWIR) image of a target ocean area, performing image registration on the RGB image and the SWIR image, and constructing a multivariate data set; performing spatial decomposition on two-dimensional images in the multivariate data set by using an improved BEMD, and performing joint decomposition on multichannel images in the multivariate data set by using an improved MEMD to obtain a plurality of intrinsic mode function images; and calculating the plurality of intrinsic mode function images to obtain a wave direction, a wave height and a current velocity of the sea wave.

[0180] The device embodiments described above are merely illustrative, wherein the units described as separate components can or can not be physically separate, and the components displayed as units can or can not be physical units, i.e., can be located in one place or distributed on multiple network units. Part or all of the modules can be selected to achieve the purpose of the embodiment scheme according to actual needs. Those skilled in the art can understand and implement without creative labor.

[0181] From the above description of the embodiments, those skilled in the art can clearly understand that the embodiments can be implemented by means of software plus necessary general hardware platforms, and of course can also be implemented by hardware. Based on such understanding, the above technical solutions, essentially or in terms of the contribution to the prior art, can be embodied in the form of a software product. The computer software product can be stored in a computer readable storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, etc., and includes a plurality of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute the methods described in the various embodiments or some parts of the embodiments.

[0182] Finally, it should be noted that the above examples are only used to illustrate the technical solutions of the present application, and are not intended to limit the same; although the present application has been described in detail with reference to the foregoing examples, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing examples, or make equivalent replacements for some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for ocean dynamic monitoring based on fusion images, characterized in that: include: Collecting RGB images and shortwave infrared (SWIR) images of a target ocean area, performing image registration on the RGB images and the SWIR images, and constructing a multivariate dataset; Using an improved bilateral empirical mode decomposition (BEMD) to spatially decompose the two-dimensional image in the multivariate data set, and using an improved multivariate empirical mode decomposition (MEMD) to jointly decompose the multi-channel image in the multivariate data set to obtain a plurality of intrinsic mode function images; Calculating the plurality of intrinsic mode function images to obtain the wave direction, wave height and ocean current velocity; The improved BEMD is used to perform spatial decomposition on the two-dimensional image in the multivariate data set, and the improved MEMD is used to perform joint decomposition on the multi-channel image in the multivariate data set to obtain several intrinsic mode function images, including: Step 1, determining local maximum points and local minimum points in the two-dimensional image of each channel in the multivariate data set, and performing preprocessing using smoothing filtering; Step 2: uniformly sample on the high-dimensional sphere to generate a set of multi-dimensional direction vectors; Step 3, projecting the component of each direction vector in any channel and the pixel value of each pixel position in any channel in the multidimensional direction vector set to obtain a projected one-dimensional signal; Step 4, using a two-dimensional interpolation method to interpolate the local maximum points and local minimum points in the projected one-dimensional signal to obtain an upper envelope and a lower envelope; Step 5: Based on the upper envelope, the lower envelope, and the total number of directional vectors, the envelopes of all directions are averaged at each pixel position to obtain an average envelope of each channel with the same spatial dimension as the original signal; Step 6: Calculate the residual of each channel by the average envelope of each channel, the average value of any component of the directional vector, and the pixel value of each pixel position in any channel. If it is determined that the normalized mean square error of the residual of each channel is less than a preset threshold, the first intrinsic mode function image is extracted, and the residual is updated by the first intrinsic mode function image and the pixel value of each pixel position in any channel. Step 7: Update the remaining signal with the updated residual, return to step 1 to loop, and extract the next intrinsic mode function image; Step 8: performing iterative screening, repeating steps 1 to 7 until a stop condition is met, and obtaining the plurality of intrinsic mode function images, wherein the plurality of intrinsic mode function images include a high-frequency intrinsic mode function image, a low-frequency intrinsic mode function image, and a residual component; The stopping conditions include that the updated residual does not include local maximum points and local minimum points, reaches a preset intrinsic mode function decomposition layer number, and the residual signal energy is lower than a preset energy threshold.

2. The method for ocean dynamic monitoring based on fusion images according to claim 1, characterized in that: Acquiring an RGB image and a SWIR image of the target ocean area, performing image registration on the RGB image and the SWIR image, and constructing a multivariate dataset also includes: High-definition visible light and infrared cameras are installed on the belly of the seaplane, and the collected images are transmitted to the interior of the aircraft in real time through the data transmission system.

3. The method for ocean dynamic monitoring based on fusion images according to claim 2, characterized in that: Collecting RGB images and SWIR images of the target ocean area, performing image registration on the RGB images and the SWIR images, and constructing a multivariate dataset, including: Synchronously collect RGB and SWIR images based on fixed time intervals; Performing image registration on the RGB image and the SWIR image using inertial measurement unit data and global positioning system data to establish an aircraft attitude matrix and an aircraft position vector; Obtaining image space registration coordinates according to the aircraft attitude matrix, the aircraft position vector, and pixel coordinates of all images; Adjust the size of the registered image so that the images of all channels have the same resolution and size; The adjusted three channels of RGB image and SWIR image are combined to form a four-channel two-dimensional multivariate dataset.

4. The method for ocean dynamic monitoring based on fusion images according to claim 1, characterized in that: Calculating the plurality of intrinsic mode function images to obtain the wave direction and wave height includes: Perform two-dimensional fast Fourier transform on any determined intrinsic mode function image to obtain a spectrum signal; Calculating the wave number and wave direction of the ocean waves based on the amplitude and phase of the spectrum signal; According to linear wave theory, the wavelength is obtained from the wave number, and the wave height is obtained by estimating the wavelength through image brightness changes or ocean dynamics models.

5. The method for ocean dynamic monitoring based on fusion images according to claim 1, characterized in that: Calculating the plurality of intrinsic mode function images to obtain the ocean current velocity of the waves includes: Obtaining the flight altitude of the UAV, the size of the sea surface area, the horizontal resolution and the vertical resolution of the plurality of intrinsic mode function images, and determining the camera field of view angle; Obtaining a pixel size based on the horizontal resolution, the vertical resolution, the flight altitude of the UAV, the size of the sea surface area, and the camera field of view; Determine the time interval between each image frame, calculate the x-direction speed and the y-direction speed by optical flow method, obtain the actual x-direction speed based on the x-direction speed, the pixel size and the time interval between each image frame, and obtain the actual y-direction speed based on the y-direction speed, the pixel size and the time interval between each image frame; The ocean current velocity is obtained by combining the actual velocity in the x direction and the actual velocity in the y direction.

6. A fusion image-based ocean dynamic monitoring system, based on the fusion image-based ocean dynamic monitoring method according to any one of claims 1 to 5, characterized in that: include: a registration module, configured to collect RGB images and SWIR images of a target ocean area, perform image registration on the RGB images and the SWIR images, and construct a multivariate dataset; a decomposition module, configured to perform spatial decomposition on the two-dimensional image in the multivariate data set using an improved BEMD, and to perform joint decomposition on the multi-channel image in the multivariate data set using an improved MEMD, to obtain a plurality of intrinsic mode function images; The calculation module is used to calculate the plurality of intrinsic mode function images to obtain the wave direction, wave height and ocean current velocity of the waves.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the ocean dynamic monitoring method based on fused images as described in any one of claims 1 to 5 is implemented.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for ocean dynamic monitoring based on fused images as described in any one of claims 1 to 5 is implemented.

9. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the method for ocean dynamic monitoring based on fused images as described in any one of claims 1 to 5 is implemented.

Images