Marine dynamic monitoring method and system based on fused image

Abstract

The invention provides an ocean dynamic monitoring method and system based on a fused image, and the method comprises the steps: collecting an RGB image and a short-wave infrared SWIR image of a target ocean region, carrying out the image registration of the RGB image and the SWIR image, and constructing a multivariable data set; performing spatial decomposition on a two-dimensional image in the multivariable data set by adopting an improved BEMD, and performing joint decomposition on a multi-channel image in the multivariable data set by adopting an improved MEMD to obtain a plurality of intrinsic mode function images; and calculating a plurality of intrinsic mode function images to obtain the wave direction, the wave height and the ocean current flow velocity of the sea wave. According to the method, the ocean dynamics principle is combined, the improved BEMD method and the MEMD are used for decomposing the ocean image and the infrared image, the wave and ocean current characteristics of the ocean surface are accurately calculated, and reliable data support is provided for the fields of ocean environment monitoring, shipping safety, ocean resource development and the like.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a method and system for ocean dynamic monitoring based on fused images. Background Art

[0002] In marine environmental monitoring, real-time and accurate acquisition of sea surface dynamic characteristics, such as waves, currents, wind speed, etc., is of great significance to marine resource development, environmental protection, shipping safety, etc. However, affected by complex weather and sea conditions, traditional single sensor monitoring methods often face problems such as limited coverage and insufficient accuracy, making it difficult to fully capture sea surface dynamic information, which in turn affects the safety and efficiency of marine activities.

[0003] Visible light imaging sensors can provide high-resolution images of the sea surface under good weather conditions, which helps to accurately obtain visual information on ocean dynamics such as waves and swells. However, under severe weather conditions (such as haze, dust, etc.), the performance of visible light images is greatly reduced, resulting in blurred images and an inability to accurately reflect the state of the sea surface. At the same time, short-wave infrared (SWIR) images have better fog-penetrating capabilities and can penetrate haze or other atmospheric interference in low-visibility environments to obtain relatively clear images of the sea surface. However, short-wave infrared images have a lower resolution and are not as good as visible light images in terms of image detail, and may not provide enough information to accurately assess sea conditions.

[0004] Therefore, there are certain limitations in using only visible light images or short-wave infrared images. In bad weather, a single sensor cannot provide complete and accurate ocean information, and the fusion of traditional sensor data often faces problems such as information redundancy and noise interference. Summary of the invention

[0005] The present invention provides a method and system for ocean dynamic monitoring based on fused images, which are used to solve the defects existing in the prior art when single visible light images or short-wave infrared images are used in ocean environment monitoring.

[0006] In a first aspect, the present invention provides a method for monitoring ocean dynamics based on fused images, comprising: Collecting RGB images and shortwave infrared (SWIR) images of the target ocean area, performing image registration on the RGB images and the SWIR images, and constructing a multivariate data set; Using an improved BEMD to spatially decompose the two-dimensional image in the multivariate data set, and using an improved MEMD to jointly decompose the multi-channel image in the multivariate data set to obtain a plurality of intrinsic mode function images; The plurality of intrinsic mode function images are calculated to obtain the wave direction, wave height and ocean current velocity of the ocean waves.

[0007] According to a method for ocean dynamic monitoring based on fusion images provided by the present invention, an RGB image and a SWIR image of a target ocean area are collected, and image registration is performed on the RGB image and the SWIR image. Before constructing a multivariate data set, the method further includes: High-definition visible light and infrared cameras are installed on the belly of the seaplane, and the collected images are transmitted to the interior of the aircraft in real time through the data transmission system.

[0008] According to a method for ocean dynamic monitoring based on fused images provided by the present invention, an RGB image and a SWIR image of a target ocean area are collected, image registration is performed on the RGB image and the SWIR image, and a multivariate data set is constructed, including: Synchronously collect RGB images and SWIR images based on fixed time intervals; Performing image registration on the RGB image and the SWIR image using inertial measurement unit data and global positioning system data to establish an aircraft attitude matrix and an aircraft position vector; Obtaining image space registration coordinates according to the aircraft attitude matrix, the aircraft position vector and pixel coordinates of all images; Adjust the size of the registered image so that the images of all channels have the same resolution and size; The adjusted three channels of RGB image and SWIR image are combined to form a four-channel two-dimensional multivariate dataset.

[0009] According to a method for monitoring ocean dynamics based on fused images provided by the present invention, an improved BEMD is used to perform spatial decomposition on the two-dimensional image in the multivariate data set, and an improved MEMD is used to perform joint decomposition on the multi-channel image in the multivariate data set to obtain a plurality of intrinsic mode function images, including: Step 1, determining local maximum points and local minimum points in the two-dimensional image of each channel in the multivariate data set, and preprocessing by smoothing filtering; Step 2, uniformly sampling on the high-dimensional sphere to generate a set of multi-dimensional direction vectors; Step 3, projecting the component of each direction vector in any channel and the pixel value of each pixel position in any channel in the multidimensional direction vector set to obtain a projected one-dimensional signal; Step 4, using a two-dimensional interpolation method to interpolate the local maximum points and the local minimum points in the projected one-dimensional signal to obtain an upper envelope and a lower envelope; Step 5, based on the upper envelope, the lower envelope and the total number of directional vectors, averaging the envelopes of all directions at each pixel position to obtain an average envelope with the same spatial dimension as the original signal; Step 6, calculating the residual of each channel by the average envelope of each channel, the average value of any component of the directional vector and the pixel value of each pixel position in any channel, if it is determined that the standardized mean square error of the residual of each channel is less than a preset threshold, the first intrinsic mode function image extraction is completed, and the residual is updated by the first intrinsic mode function image and the pixel value of each pixel position in any channel; Step 7: Update the remaining signal with the updated residual, return to step 1 to loop, and extract the next intrinsic mode function image; Step 8: performing iterative screening, repeating steps 1 to 7 until a stop condition is met, and obtaining the plurality of intrinsic mode function images, wherein the plurality of intrinsic mode function images include a high-frequency intrinsic mode function image, a low-frequency intrinsic mode function image, and a residual component; The stopping condition includes that the updated residual does not include local maximum points and local minimum points, reaches a preset intrinsic mode function decomposition layer number, and the residual signal energy is lower than a preset energy threshold.

[0010] According to a method for monitoring ocean dynamics based on fused images provided by the present invention, the plurality of intrinsic mode function images are calculated to obtain the wave direction and wave height of the waves, including: Perform two-dimensional fast Fourier transform on any determined intrinsic mode function image to obtain a spectrum signal; The wave number and wave direction of the ocean waves are calculated based on the amplitude and phase of the frequency spectrum signal; According to linear wave theory, the wavelength is obtained from the wave number, and the wave height is obtained by estimating the wavelength through image brightness changes or ocean dynamics models.

[0011] According to a method for monitoring ocean dynamics based on fused images provided by the present invention, the plurality of intrinsic mode function images are calculated to obtain the ocean current velocity of the waves, including: Obtaining the flight altitude of the UAV, the size of the sea surface area, the horizontal resolution and the vertical resolution of the plurality of intrinsic mode function images, and determining the camera field of view angle; Obtaining a pixel size from the horizontal resolution, the vertical resolution, the flight altitude of the UAV, the size of the sea surface area, and the camera field of view; Determine the time interval of each image frame, calculate the x-direction speed and the y-direction speed by the optical flow method, obtain the x-direction actual speed according to the x-direction speed, the pixel size and the time interval of each image frame, and obtain the y-direction actual speed according to the y-direction speed, the pixel size and the time interval of each image frame; The actual speed in the x direction and the actual speed in the y direction are combined to obtain the current speed.

[0012] In a second aspect, the present invention further provides an ocean dynamic monitoring system based on fused images, comprising: A registration module, used for collecting RGB images and SWIR images of a target ocean area, performing image registration on the RGB images and the SWIR images, and constructing a multivariate data set; A decomposition module, used for spatially decomposing the two-dimensional image in the multivariate data set by using an improved BEMD, and jointly decomposing the multi-channel image in the multivariate data set by using an improved MEMD, to obtain a plurality of intrinsic mode function images; The calculation module is used to calculate the plurality of intrinsic mode function images to obtain the wave direction, wave height and ocean current velocity of the sea waves.

[0013] In a third aspect, the present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, it implements any of the above-mentioned methods for dynamic ocean monitoring based on fused images.

[0014] In a fourth aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the above-described methods for dynamic ocean monitoring based on fused images.

[0015] Compared with the prior art, the present invention has the following technical effects: (1) By combining the principles of ocean dynamics, an improved BEMD and MEMD combination method is used to deeply decompose the ocean surface image and extract wave and current characteristics of different scales. BEMD can effectively capture the high-frequency and low-frequency components of ocean fluctuations by decomposing two-dimensional images, and identify fluctuation characteristics of different scales such as small ripples and large waves. MEMD uses multivariate decomposition to enable multiple image channels (such as RGB and SWIR images) to be processed collaboratively in time and space, further improving the accuracy of wave and current monitoring. This method provides more accurate and reliable data support for real-time monitoring of ocean dynamics, can effectively predict sea surface fluctuations and flow conditions, and support applications in the fields of marine environmental monitoring, shipping safety, and marine resource development; (2) By fusing multi-channel data of RGB images and short-wave infrared images, the present invention effectively overcomes the limitations of a single image channel in dynamic ocean monitoring. Using improved BEMD and MEMD algorithms, the present invention can jointly decompose multi-channel image data, extract IMF components containing different information, and further calculate wave direction, wave height, and ocean current velocity. This multi-channel image processing technology not only improves the ability to identify details of the ocean surface, but also enhances the monitoring capability of complex ocean environments through precise data fusion, especially in changing sea conditions, and can provide more comprehensive and reliable information; (3) The optical flow method is combined with the altitude and image resolution of the drone to accurately estimate the velocity of ocean surface currents. By analyzing the pixel displacement between image sequences and combining the IMU and GPS data of the drone, the background interference caused by the aircraft movement on the image is successfully removed to ensure the accurate calculation of the ocean current velocity. This method can estimate the flow velocity in real time according to the dynamic changes of the sea surface, providing efficient and dynamically adaptable capabilities for dynamic ocean monitoring, and is suitable for complex marine environment monitoring, such as shipping navigation, marine resource exploration and other fields. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0017] Figure 1 It is a schematic diagram of the flow of the ocean dynamic monitoring method based on fusion images provided by the present invention; Figure 2 It is a schematic diagram of the improved BEMD and MEMD ocean current image decomposition provided by the present invention; Figure 3 It is a structural schematic diagram of the marine dynamic monitoring system based on fusion images provided by the present invention; Figure 4 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0018] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0019] In view of the many limitations of the prior art, the present invention fuses short-wave infrared and visible light images in the field of ocean monitoring, combining the advantages of both, which can effectively improve the accuracy and reliability of dynamic ocean monitoring. Through multimodal fusion processing of these two types of image data, not only can the stability of monitoring be guaranteed under different climatic conditions, but also the ability to identify complex sea conditions can be improved, thereby providing more accurate sea condition assessment and decision support for fields such as marine environmental monitoring, shipping safety, and marine resource development. In addition, sea surface infrared images have a wide range of innovative applications in marine environmental monitoring. For example, in terms of marine pollution monitoring, infrared images can identify abnormal sea surface temperatures and help detect environmental problems such as oil spills. In fishery resource management, the distribution of sea surface temperature is closely related to fish school activities, and infrared images can assist in the investigation and management of fishery resources. By fusing infrared and visible light images, the accuracy of marine meteorological forecasts can also be improved to support emergency response activities such as maritime rescue.

[0020] Figure 1 is a flow chart of a method for ocean dynamic monitoring based on fusion images provided by an embodiment of the present invention, such as Figure 1 As shown, including: Step 100: collecting an RGB image and a shortwave infrared SWIR image of a target ocean area, performing image registration on the RGB image and the SWIR image, and constructing a multivariate data set; Step 200: using an improved BEMD to spatially decompose the two-dimensional image in the multivariate data set, and using an improved MEMD to jointly decompose the multi-channel image in the multivariate data set to obtain a plurality of intrinsic mode function images; Step 300: Calculate the plurality of intrinsic mode function images to obtain the wave direction, wave height and ocean current velocity of the ocean waves.

[0021] It should be noted that, in the present invention, the use of sea surface infrared images plays a key role in improving the accuracy and reliability of ocean dynamic monitoring. The principle of infrared imaging is based on the thermal radiation emitted by an object due to its temperature. The ocean surface emits infrared radiation according to its temperature and emissivity. According to the Stefan-Boltzmann law, the total radiation emittance of a black body is for: ,in, is the sea surface emissivity (about 0.98 for seawater), is the Stefan-Boltzmann constant, is the absolute temperature (Kelvin). By capturing infrared images of the ocean surface, changes in sea surface temperature (SST) can be detected, which is important for monitoring ocean phenomena such as currents, upwelling, and temperature fronts. These thermal anomalies are critical for monitoring ocean dynamics as they affect wave formation, ocean current patterns, and can be associated with events such as oil spills or pollutant discharges. In addition, infrared imagery is less affected by fog, haze, or low light conditions due to its ability to detect longer wavelength radiation, thereby enhancing the ability of the monitoring system to operate effectively in a variety of environmental conditions.

[0022] Specifically, the embodiment of the present invention first installs the equipment, and installs a high-definition visible light camera and an infrared camera on the belly of the seaplane to ensure that the field of view of both cameras covers the ocean surface below the aircraft. The specific installation method is as follows: High-resolution RGB cameras and short-wave infrared (SWIR) cameras are used to obtain image data of the sea surface in normal visibility and low visibility (such as haze, sandstorms, etc.) conditions. When installing the cameras, ensure that their optical axes are perpendicular or nearly perpendicular to the flight direction of the aircraft to ensure that the sea surface images can be captured in real time and stably.

[0023] To combine an RGB ocean wave image (containing red, green, and blue channels) with a shortwave infrared (SWIR) image of the same area, Figure 2 As shown in the figure, using the improved decomposition method of Bidimensional Empirical Mode Decomposition (BEMD) and Multivariate Empirical Mode Decomposition (MEMD), a set of Intrinsic Mode Function (IMF) images representing the high-frequency, low-frequency and residual components of the waves are obtained. The following detailed steps are required: 1. Data Image Registration The camera is fixed under the belly of the aircraft and is firmly installed by a directional bracket to avoid being affected by aircraft vibration or air flow during flight. The position and angle of the two cameras are precisely adjusted to ensure that the image acquisition area overlaps and can fully cover the sea surface below the aircraft.

[0024] A high-speed data transmission system is installed to transmit the image data captured by the camera to the data processing unit inside the aircraft in real time.

[0025] During the aircraft's cruising, a set of infrared images and visible light images are acquired simultaneously every 0.5 seconds. To ensure data consistency, the global positioning system (GPS) and inertial measurement unit (IMU) data are used to synchronize the camera images in time and space during each data acquisition cycle to ensure that the infrared and visible light images are fully aligned in time. Every 0.5 seconds, the aircraft camera synchronously acquires a set of images with an image resolution of 1920×1080 pixels. This resolution ensures that the details of the image are fully presented, which is particularly suitable for accurately monitoring sea surface fluctuations and ocean current characteristics.

[0026] Spatial alignment: Ensure that the RGB image and SWIR image are perfectly aligned in space. If there is a displacement, rotation, or scale difference, the aircraft's inertial measurement unit (IMU) data and global positioning system (GPS) data need to be used for image registration. Establish the aircraft's attitude matrix and the position vector , perform spatial registration of images.

[0027] in, is the pixel coordinate.

[0028] Resize: Resize images so that all channels have the same resolution and size.

[0029] Multivariate data construction: The three channels of the RGB image and the SWIR image are combined into a four-channel multivariate two-dimensional dataset.

[0030]

[0031] 2. Using improved BEMD decomposition method and MEMD decomposition method By adopting the improved BEMD and MEMD decomposition methods, the infrared imagery is fused with the visible light imagery, making full use of the advantages of both types of imagery. The fusion of data enables the extraction of detailed spatial features and accurate thermal information, thus providing a more comprehensive understanding of the sea surface dynamics.

[0032] Among them, the goal of BEMD is to decompose a two-dimensional image into several IMFs and extract spatial features of different scales. The method usually adopted is to find local extreme points in two-dimensional space and perform two-dimensional interpolation to construct an envelope; the goal of MEMD is to jointly decompose multivariate signals to ensure that the IMFs of different channels are aligned in time or space. The method usually adopted is to use multi-dimensional direction vectors to project multivariate data into one-dimensional space for joint decomposition. It includes the following steps: Step 1: Detection of local extreme points Goal: Find local maxima and minima in a 2D image for each channel.

[0033] For each channel The local maximum satisfies , the local minimum satisfies , Since noise may lead to too many extreme points, a smoothing filter (such as Gaussian filter) can be used for preprocessing.

[0034] Step 2: Generate multi-dimensional direction vector Objective: Generate a set of direction vectors uniformly distributed in a high-dimensional space (corresponding to the number of channels) for data projection. Direction vector generation method: Uniform sampling on a four-dimensional sphere to obtain a set of direction vectors ,in ,satisfy , you can use the Hammersley point set or Sobol sequence for uniform sampling.

[0035] Step 3: Data Projection Objective: Project multivariate two-dimensional data along each direction vector to obtain a one-dimensional signal.

[0036] Projection formula: For each pixel position and each direction vector :

[0037] in, is the direction vector In the The components of a channel. It is The pixel value of each channel.

[0038] Step 4: Envelope estimation Objective: To project the data Perform envelope estimation to obtain the upper envelope and the lower envelope.

[0039] Method: Two-dimensional interpolation: For the local maximum and minimum points of the projection data, use the two-dimensional interpolation method to construct the upper envelope and lower envelope The interpolation method may include bicubic interpolation, thin plate spline function (TPS), radial basis function (RBF), etc.

[0040] Step 5: Average Envelope Calculation Goal: Average the envelopes in all directions to get the average envelope of each channel.

[0041] Average envelope calculation: , is the total number of direction vectors. Since the projection is done in multivariate space, the average envelope The same spatial dimensions as the original signal should be maintained.

[0042] Step 6: Extract IMF from raw data Residual calculation: For each channel Both , is the direction vector The average value of the components.

[0043] To stop criteria checking: If the standardized mean square error SD is less than the preset threshold (such as 0.2), the IMF extraction is considered complete. The first IMF is . Update the residual: .

[0044] Step 7: Residual Calculation Goal: Use the remaining signal as new data and continue to extract the next IMF.

[0045] Update data: , return to step 1 to loop, and use the updated Continue to extract the next IMF.

[0046] Step 8: Iterate the screening and repeat the above steps until the stop condition is met: The residual is a monotonic function: when the residual signal There are no more local extreme points. The preset number of IMF decomposition layers is reached. The energy of the residual signal is lower than a certain energy threshold.

[0047] Finally, we get the high-frequency IMF of a frame of image, which represents the details and high-frequency changes (such as small ripples) in the waves.

[0048] Low-frequency IMF: represents larger structures and low-frequency changes (such as large waves and swells).

[0049] Residual component: represents the overall trend or background brightness of the image.

[0050] 3. Calculate the wave direction and wave height of the image decomposed at a single moment For the selected IMF components , perform two-dimensional FFT and get the spectrum .

[0051]

[0052] Calculate the wave number of the ocean waves by analyzing the amplitude and phase information of the spectrum and direction .

[0053] According to linear wave theory, the wavelength With wave number Relationship , wave height It can be estimated by changes in image brightness or combined with ocean dynamics models.

[0054] 4. Calculation of ocean current velocity The optical flow method is used to calculate the current velocity on the ocean surface for the modal decomposition image with the high frequency removed. When accurately estimating the current velocity based on the altitude of the drone, the following factors need to be considered: the flight altitude of the drone, the image resolution, the scale of the optical flow calculation, and the flow characteristics on the sea surface.

[0055] (1) Relationship between drone altitude and image scale The flying altitude of the drone determines the size of the ocean surface area in the captured image. When the drone is at a higher altitude, the captured ocean surface area is larger, while a lower flying altitude corresponds to a smaller area. Therefore, the altitude of the drone directly affects the spatial resolution of the image and the scale of the optical flow calculation results. Assume that the drone is flying at an altitude of , the horizontal and vertical resolutions of the image are and , then the size of the sea surface area represented by each pixel in the image can be calculated as: , where the camera field of view (FOV) is the horizontal or vertical field of view of the camera, which affects the field of view of the image.

[0056] The optical flow method calculates the displacement of pixels in an image in pixels per frame (px / frame). In order to convert this displacement into actual physical units (such as meters per second), it is necessary to relate the pixel displacement of the image to the height of the drone, the spatial resolution of the image, and the time interval. Assume that the time interval between each frame is , then the velocity component calculated by optical flow is and It can be converted to actual speed (unit: m / s) by the following formula: , .

[0057] (2) Calculation of ocean currents The Farneback optical flow algorithm is used to calculate the motion between image sequences. This algorithm is a multi-scale pyramid-based algorithm that can estimate the displacement vector of each pixel and is suitable for processing motion estimation between consecutive frames. The formula is expressed as: ,in: is the image at time The pixel value at and are the optical flow components in the horizontal and vertical directions, respectively. is the time interval.

[0058] In order to estimate the actual flow rate of the ocean current, it is necessary to comprehensively consider the calculation results of the optical flow and the movement of the aircraft itself (dive, yaw, roll). The movement of the aircraft will cause background movement in the image. Therefore, it is first necessary to separate the part of the sea surface flow from the optical flow and remove the background displacement caused by the movement of the drone itself. The IMU (inertial measurement unit) and GPS data on the drone can be used to estimate the actual speed and position change of the drone, thereby removing the displacement caused by its movement from the optical flow. Motion compensation is performed based on the speed, azimuth and field of view of the drone to remove the pseudo optical flow caused by the movement of the drone. The calculated optical flow velocity vector and After removing the background motion, it represents the relative motion of objects on the sea surface. Using the altitude of the drone and the camera field of view, these pixel-level optical flow velocities can be converted into the current velocity on the sea surface. Finally, the velocity can be calculated using the following formula: .

[0059] The ocean dynamic monitoring method based on the fusion of short-wave infrared and visible light images provided by the present invention combines the principles of ocean dynamics, uses the improved BEMD method and MEMD (to decompose ocean images and infrared images, accurately calculate the characteristics of waves and currents on the ocean surface, and provide reliable data support for the fields of marine environment monitoring, shipping safety, and marine resource development.

[0060] In one embodiment, detection is performed for marine oil spills.

[0061] An oil tanker leak occurred in a certain sea area, and a large amount of crude oil leaked into the sea, forming a large area of ​​oil pollution. The rapid spread of oil pollution has posed a serious threat to the marine ecological environment, marine life and coastal economy. In order to control pollution and reduce losses in a timely manner, it is urgent to quickly and accurately detect and locate the spread of oil pollution to provide a scientific basis for emergency response. The implementation process includes: (1) Data collection Visible light image acquisition: Use a high-resolution visible light camera (such as a multispectral camera) to take aerial photos of the accident area and obtain real-time RGB visible light images. Visible light images can provide information on the surface characteristics of the sea surface, but because the reflectivity of oil pollution and seawater in the visible spectrum may be similar, it is difficult to accurately distinguish the oil pollution area using visible light images alone; Infrared image acquisition: Short-wave infrared (SWIR) or medium-wave infrared (MWIR) sensors are used simultaneously to obtain infrared images of the accident sea area. Due to the difference in the radiation characteristics of oil pollution and seawater in the infrared band, the oil pollution area will show abnormal temperature (usually a lower temperature area) on the infrared image, which is conducive to identifying oil pollution.

[0062] (2) Data preprocessing Image registration: Use feature point matching algorithms (such as SIFT, SURF) or use the IMU and GPS data of the drone / satellite platform to spatially register the visible light image and the infrared image to ensure that the two images are aligned in the same coordinate system.

[0063] Registration formula: ,in, is the original image coordinate, is the image coordinate after registration, is the transformation matrix.

[0064] Image correction: Perform radiation correction on infrared images to eliminate atmospheric effects and obtain the true sea surface temperature distribution. Perform color correction and brightness adjustment on visible light images to enhance image contrast.

[0065] (3) Multi-source data fusion Multivariate data construction: The red, green, and blue channels of the visible light image are combined with the infrared image to construct a four-channel multivariate two-dimensional data set: ; Improved BEMD and MEMD decomposition: IMF extraction: for multivariate datasets The improved BEMD and MEMD methods are applied to extract intrinsic mode function (IMF) components of different scales. These IMF components contain high-frequency details (such as texture and edge) and low-frequency trends (such as brightness and temperature distribution) of the image. Through joint decomposition, the spatial consistency of the IMF components of each channel is ensured, which facilitates the fusion and comparison of multi-source information.

[0066] (4) Oil pollution feature extraction High-frequency feature analysis: The oil pollution area shows different texture characteristics from the surrounding seawater in the high-frequency IMF component. Use texture analysis methods (such as gray-level co-occurrence matrix and local binary pattern) to extract texture features in the high-frequency IMF component and identify the detailed structure of the oil pollution; Low-frequency feature analysis: In the low-frequency IMF component, the temperature distribution of the oil pollution area is different from that of the seawater. Analyze the low-frequency IMF component of the infrared channel, extract the temperature anomaly characteristics, and further confirm the location of the oil pollution.

[0067] (5) Oil pollution detection and identification Feature fusion: The features of high-frequency and low-frequency IMF components are fused to form a comprehensive feature vector.

[0068] Fusion formula: ,in, is the weight coefficient, For the IMF components.

[0069] Classification and segmentation: Use machine learning algorithms such as support vector machine (SVM), random forest (RF) or convolutional neural network (CNN) to build an oil pollution detection model. Input the model and output the probability map of the oil pollution area; Threshold segmentation: Set the threshold, convert the probability map into a binary mask, and extract the oil pollution area.

[0070] (6) Results generation and application Oil pollution distribution map production: Visualize the detected oil pollution area in the Geographic Information System (GIS) to generate an oil pollution distribution map. Mark the area, location, concentration and other information of the oil pollution; Emergency decision support: Combined with marine environmental data (such as ocean currents and wind speed), predict the spread of oil pollution. Provide scientific basis for emergency response departments to formulate pollution cleanup strategies and resource allocation plans.

[0071] (7) Effect evaluation and optimization Accuracy verification: Through on-site sampling and visual observation, the test results are verified to evaluate the accuracy and reliability of the model.

[0072] Model optimization: Based on the verification results, adjust the model parameters and algorithms to continuously improve the detection performance.

[0073] Through the scheme of this embodiment, it can be seen that this embodiment has the following characteristics: (1) Deep fusion of multi-source data: By fusing visible light and infrared images and using the improved BEMD and MEMD methods, multi-scale and multi-dimensional extraction of oil pollution characteristics is achieved.

[0074] (2) High-precision detection: Through joint decomposition and feature fusion, the limitations of a single sensor are overcome, significantly improving the accuracy and robustness of oil pollution detection.

[0075] (3) Rapid response capability: Automated data processing and analysis processes greatly shorten the time from data collection to result output, thus gaining valuable time for emergency response.

[0076] In one embodiment, monitoring is performed for red tide.

[0077] Red tide is a marine ecological disaster caused by the abnormal proliferation of phytoplankton (such as algae). The outbreak of red tide will cause the sea water to change color, release harmful substances, cause the death of fish and large marine organisms, and seriously affect the marine ecological environment and aquaculture. In order to reduce the harm of red tide, timely and accurate monitoring and early warning of red tide are required. The implementation process includes: (1) Data collection Infrared image acquisition: Use medium-wave infrared (MWIR) or long-wave infrared (LWIR) sensors to obtain sea surface temperature (SST) data of the target sea area. Due to the high density of algae in red tide areas, the sea surface temperature may be abnormal, which is manifested as an increase or decrease in temperature; Visible light image acquisition: Use high-resolution visible light cameras to acquire RGB images of the sea area and capture changes in the color of the seawater. Red tides can cause the seawater to appear red, brown or other abnormal colors, and visible light images can directly reflect this change.

[0078] (2) Data preprocessing Image registration: spatially register infrared images and visible light images to ensure spatial consistency of data from different sensors; Image correction: Perform radiation correction on infrared images to obtain accurate sea surface temperature. Perform atmospheric correction and color balance on visible light images to enhance water color information.

[0079] (3) Multi-source data fusion Multivariate data construction: The three channels of visible light images are combined with the temperature data of infrared images to form a four-dimensional data set:

[0080] in, is the sea surface temperature.

[0081] Improved BEMD and MEMD decomposition: IMF extraction: The improved decomposition method is applied to obtain the IMF components of each channel.

[0082] Feature enhancement: In the high-frequency IMF component of the visible light channel, the water color anomaly in the red tide area is highlighted. In the low-frequency IMF component of the infrared channel, the spatial distribution characteristics of the sea surface temperature are more obvious.

[0083] (4) Red tide feature extraction Water color anomaly detection: Calculate the normalized difference water index (NDWI) or other water color index of the visible light IMF component to identify abnormal water color areas

[0084] Temperature anomaly detection: Analyze the infrared IMF component to detect abnormal changes in sea surface temperature.

[0085] Spatial overlay analysis: Overlay the abnormal water color areas with the abnormal temperature areas to determine the possible red tide occurrence areas.

[0086] (5) Red tide detection and early warning Classification model establishment: Use machine learning algorithms (such as SVM and decision tree) to establish a red tide detection model, input fused feature data, and output a red tide probability map.

[0087] Threshold judgment: According to the model output, set the threshold and extract the red tide area.

[0088] Red tide distribution map production: Generate a red tide distribution map, marking the scope, severity and other information of the red tide.

[0089] (6) Application of results and feedback Warning information release: Red tide warning information is released to relevant departments and the public to guide fishery production and marine environmental protection work.

[0090] Trend analysis and prediction: Combine meteorological data and ocean dynamic models to analyze the development trend of red tides and predict the possible spread range.

[0091] Through the scheme of this embodiment, it can be seen that this embodiment has the following characteristics: (1) Multi-dimensional feature detection: By integrating temperature and water color information and using an improved decomposition method, the characteristics of red tides can be fully captured, thereby improving the sensitivity of detection.

[0092] (2) Automation and real-time: The system realizes the automated processing of red tide monitoring, can quickly acquire and analyze data, and provide real-time warnings.

[0093] (3) High reliability: Through multi-source data fusion and advanced algorithms, the false alarm and missed alarm rates are reduced, and the reliability of early warning information is enhanced.

[0094] The following is a description of the ocean dynamic monitoring system based on fused images provided by the present invention. The ocean dynamic monitoring system based on fused images described below and the ocean dynamic monitoring method based on fused images described above can be referred to each other.

[0095] Figure 3 is a schematic diagram of the structure of an ocean dynamic monitoring system based on fusion images provided by an embodiment of the present invention, such as Figure 3 As shown, it includes: a registration module 31, a decomposition module 32 and a calculation module 33, wherein: The registration module 31 is used to collect RGB images and SWIR images of the target ocean area, perform image registration on the RGB images and the SWIR images, and construct a multivariate data set; the decomposition module 32 is used to use an improved BEMD to spatially decompose the two-dimensional image in the multivariate data set, and use an improved MEMD to jointly decompose the multi-channel images in the multivariate data set to obtain a number of intrinsic mode function images; the calculation module 33 is used to calculate the several intrinsic mode function images to obtain the wave direction, wave height and ocean current velocity of the waves.

[0096] Figure 4 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 4 As shown, the electronic device may include: a processor 410, a communication interface 420, a memory 430 and a communication bus 440, wherein the processor 410, the communication interface 420 and the memory 430 communicate with each other through the communication bus 440. The processor 410 may call the logic instructions in the memory 430 to execute the ocean dynamic monitoring method based on fusion images, the method comprising: collecting RGB images and short-wave infrared SWIR images of the target ocean area, performing image registration on the RGB images and the SWIR images, and constructing a multivariate data set; using an improved BEMD to perform spatial decomposition on the two-dimensional image in the multivariate data set, and using an improved MEMD to perform joint decomposition on the multi-channel images in the multivariate data set to obtain a number of intrinsic mode function images; calculating the several intrinsic mode function images to obtain the wave direction, wave height and ocean current velocity of the waves.

[0097] In addition, the logic instructions in the above-mentioned memory 430 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0098] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the ocean dynamic monitoring method based on fused images provided by the above methods, and the method includes: collecting RGB images and short-wave infrared SWIR images of the target ocean area, performing image registration on the RGB images and the SWIR images, and constructing a multivariate data set; using an improved BEMD to spatially decompose the two-dimensional image in the multivariate data set, and using an improved MEMD to jointly decompose the multi-channel images in the multivariate data set to obtain a number of intrinsic mode function images; calculating the several intrinsic mode function images to obtain the wave direction, wave height and ocean current velocity of the waves.

[0099] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the ocean dynamic monitoring method based on fused images provided by the above-mentioned methods, the method comprising: collecting RGB images and short-wave infrared SWIR images of the target ocean area, performing image registration on the RGB images and the SWIR images, and constructing a multivariate data set; using an improved BEMD to spatially decompose the two-dimensional image in the multivariate data set, and using an improved MEMD to jointly decompose the multi-channel images in the multivariate data set to obtain a number of intrinsic mode function images; calculating the several intrinsic mode function images to obtain the wave direction, wave height and ocean current velocity of the waves.

[0100] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0101] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0102] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for ocean dynamic monitoring based on fusion images, characterized in that: include: Collecting RGB images and shortwave infrared (SWIR) images of the target ocean area, performing image registration on the RGB images and the SWIR images, and constructing a multivariate data set; Using an improved bilateral empirical mode decomposition (BEMD) to spatially decompose the two-dimensional image in the multivariate data set, and using an improved multivariate empirical mode decomposition (MEMD) to jointly decompose the multi-channel image in the multivariate data set to obtain a plurality of intrinsic mode function images; The plurality of intrinsic mode function images are calculated to obtain the wave direction, wave height and ocean current velocity of the ocean waves.

2. The method for ocean dynamic monitoring based on fusion images according to claim 1, characterized in that: Collecting an RGB image and a SWIR image of the target ocean area, performing image registration on the RGB image and the SWIR image, and before constructing a multivariate data set, further comprising: High-definition visible light and infrared cameras are installed on the belly of the seaplane, and the collected images are transmitted to the interior of the aircraft in real time through the data transmission system.

3. The method for ocean dynamic monitoring based on fusion images according to claim 2 is characterized in that: Collecting RGB images and SWIR images of the target ocean area, performing image registration on the RGB images and the SWIR images, and constructing a multivariate data set, including: Synchronously collect RGB images and SWIR images based on fixed time intervals; Performing image registration on the RGB image and the SWIR image using inertial measurement unit data and global positioning system data to establish an aircraft attitude matrix and an aircraft position vector; Obtaining image space registration coordinates according to the aircraft attitude matrix, the aircraft position vector and pixel coordinates of all images; Adjust the size of the registered image so that the images of all channels have the same resolution and size; The adjusted three channels of RGB image and SWIR image are combined to form a four-channel two-dimensional multivariate dataset.

4. The method for ocean dynamic monitoring based on fusion images according to claim 1, characterized in that: The improved BEMD is used to perform spatial decomposition on the two-dimensional image in the multivariate data set, and the improved MEMD is used to perform joint decomposition on the multi-channel image in the multivariate data set to obtain several intrinsic mode function images, including: Step 1, determining local maximum points and local minimum points in the two-dimensional image of each channel in the multivariate data set, and preprocessing by smoothing filtering; Step 2, uniformly sampling on the high-dimensional sphere to generate a set of multi-dimensional direction vectors; Step 3, projecting the component of each direction vector in any channel and the pixel value of each pixel position in any channel in the multidimensional direction vector set to obtain a projected one-dimensional signal; Step 4, using a two-dimensional interpolation method to interpolate the local maximum points and the local minimum points in the projected one-dimensional signal to obtain an upper envelope and a lower envelope; Step 5, based on the upper envelope, the lower envelope and the total number of directional vectors, averaging the envelopes of all directions at each pixel position to obtain an average envelope of each channel with the same spatial dimension as the original signal; Step 6, calculating the residual of each channel by the average envelope of each channel, the average value of any component of the directional vector and the pixel value of each pixel position in any channel, if it is determined that the standardized mean square error of the residual of each channel is less than a preset threshold, the first intrinsic mode function image extraction is completed, and the residual is updated by the first intrinsic mode function image and the pixel value of each pixel position in any channel; Step 7: Update the remaining signal with the updated residual, return to step 1 to loop, and extract the next intrinsic mode function image; Step 8: performing iterative screening, repeating steps 1 to 7 until a stop condition is met, and obtaining the plurality of intrinsic mode function images, wherein the plurality of intrinsic mode function images include a high-frequency intrinsic mode function image, a low-frequency intrinsic mode function image, and a residual component; The stopping condition includes that the updated residual does not include local maximum points and local minimum points, reaches a preset intrinsic mode function decomposition layer number, and the residual signal energy is lower than a preset energy threshold.

5. The method for ocean dynamic monitoring based on fusion images according to claim 1, characterized in that: Calculating the plurality of intrinsic mode function images to obtain the wave direction and wave height of the sea wave includes: Perform two-dimensional fast Fourier transform on any determined intrinsic mode function image to obtain a spectrum signal; The wave number and wave direction of the ocean waves are calculated based on the amplitude and phase of the frequency spectrum signal; According to linear wave theory, the wavelength is obtained from the wave number, and the wave height is obtained by estimating the wavelength through image brightness changes or ocean dynamics models.

6. The method for ocean dynamic monitoring based on fusion images according to claim 1 is characterized in that: Calculating the plurality of intrinsic mode function images to obtain the ocean current velocity of the waves includes: Obtaining the flight altitude of the UAV, the size of the sea surface area, the horizontal resolution and the vertical resolution of the plurality of intrinsic mode function images, and determining the camera field of view angle; Obtaining a pixel size from the horizontal resolution, the vertical resolution, the flight altitude of the UAV, the size of the sea surface area, and the camera field of view; Determine the time interval of each image frame, calculate the x-direction speed and the y-direction speed by the optical flow method, obtain the x-direction actual speed according to the x-direction speed, the pixel size and the time interval of each image frame, and obtain the y-direction actual speed according to the y-direction speed, the pixel size and the time interval of each image frame; The actual speed in the x direction and the actual speed in the y direction are combined to obtain the current speed.

7. An ocean dynamic monitoring system based on fusion images, characterized in that: include: A registration module, used for collecting RGB images and SWIR images of a target ocean area, performing image registration on the RGB images and the SWIR images, and constructing a multivariate data set; A decomposition module, used for spatially decomposing the two-dimensional image in the multivariate data set by using an improved BEMD, and jointly decomposing the multi-channel image in the multivariate data set by using an improved MEMD, to obtain a plurality of intrinsic mode function images; The calculation module is used to calculate the plurality of intrinsic mode function images to obtain the wave direction, wave height and ocean current velocity of the sea waves.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the ocean dynamic monitoring method based on fused images as described in any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for ocean dynamic monitoring based on fused images as described in any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method for ocean dynamic monitoring based on fused images as described in any one of claims 1 to 6 is implemented.

Images