An unmanned aerial vehicle-borne sea surface oil spill detection system, method, equipment, medium and product

Abstract

This application discloses a drone-mounted sea surface oil spill detection system, method, equipment, medium, and product, relating to the field of sea surface oil spill detection. The system comprises: a polarization characteristic detection module, an image fusion module, and a data processing module; the polarization characteristic detection module includes a drone and an image capture unit; the image capture unit is connected to the image fusion module; the image fusion module is connected to the data processing module; the drone hovers at a set position according to a set navigation route; the image capture unit captures multimodal images of the sea surface to be detected while hovering, including a visible light intensity image, a visible light polarization image, an infrared intensity image, and an infrared polarization image; the image fusion module performs image registration, radiation correction, and image fusion processing on the multimodal images to obtain a fused image; and the data processing module determines the polarization degree of the fused image, thereby determining the type of oil spill based on the polarization degree. This application improves the accuracy of sea surface oil spill detection and reduces the false alarm rate.

Description

Technical Field

[0001] The present application relates to the field of sea surface oil spill detection, and in particular to an unmanned aerial vehicle (UAV)-borne sea surface oil spill detection system, method, equipment, medium and product. Background Art

[0002] With the rapid development of modern industry, oil spills from tankers, the development of coastal and estuarine oil deposits, and the discharge of industrial wastewater from refineries have all led to oil pollution of water bodies, especially in estuaries and offshore waters. Oil pollution is particularly severe, not only causing serious environmental pollution on the sea surface and severe damage to the marine ecosystem, but also affecting human health and causing huge economic losses. Therefore, detecting oil spills on the sea surface and improving detection accuracy have always been research hotspots in marine environmental ecological security.

[0003] The polarization bidirectional reflectance distribution function (pBRDF) is used to describe the polarization reflectance characteristics of a material. It can not only quantify the magnitude of scattering in different directions, but also provide the polarization characteristics of the scattering. Current observation equipment used for pBRDF characteristics usually uses track sliding to control the observation angle of the spectrometer to achieve multi-angle spectral reflectance measurement of ground objects. This type of observation equipment usually has the following shortcomings: the equipment is large in size and weight, making it inconvenient to transport, and requires on-site assembly, making it unsuitable for sites with poor accessibility; the equipment needs to be directly set up on the ground during observation, which to a certain extent damages the observed target and its surrounding environment, affecting the natural characteristics of the measured target; due to the size of the observation track, the distance between the instrument and the target is limited, making it impossible to perform effective pBRDF observations on larger-scale targets and mixed pixels; spectral measurements require manual operation, which is labor-intensive, has low measurement repeatability, and large random errors. Summary of the Invention

[0004] The purpose of this application is to provide an unmanned aerial vehicle (UAV)-mounted sea surface oil spill detection system, method, equipment, medium and product to accurately distinguish the types of oil spills and reduce the false alarm rate.

[0005] To achieve the above objectives, this application provides the following solutions:

[0006] In a first aspect, the present application provides an unmanned aerial vehicle (UAV)-mounted sea surface oil spill detection system, comprising: a polarization characteristic detection module, an image fusion module, and a data processing module; the polarization characteristic detection module comprises a pod, an UAV, and an image capture unit; the pod is suspended directly below the UAV; the image capture unit is placed inside the pod; the image capture unit is connected to the image fusion module; and the image fusion module is connected to the data processing module;

[0007] The UAV is configured to hover at a set position according to a set navigation route; the image capturing unit is configured to capture multimodal images of the sea surface to be detected while hovering; the multimodal images include a visible light intensity image, a visible light polarization image, an infrared intensity image, and an infrared polarization image;

[0008] The image fusion module is used to perform image registration processing, radiation correction processing and image fusion processing on the multimodal image to obtain a fused image;

[0009] The data processing module is used to determine the polarization degree of the fused image, thereby determining the type of the oil spill according to the polarization degree.

[0010] Optionally, the image capturing unit includes: a visible light intensity camera, an infrared intensity camera, a visible light polarization camera, and an infrared polarization camera;

[0011] The visible light intensity camera is used to capture a visible light intensity image of the sea surface to be detected;

[0012] The infrared intensity camera is used to capture infrared intensity images of the sea surface to be detected;

[0013] The visible light polarization camera is used to capture a visible light polarization image of the sea surface to be detected;

[0014] The infrared polarization camera is used to capture infrared polarization images of the sea surface to be detected.

[0015] Optionally, the image fusion module includes:

[0016] an image registration unit, configured to register the visible light intensity image, the visible light polarization image, the infrared intensity image, and the infrared polarization image to obtain a registered visible light intensity image, a registered visible light polarization image, a registered infrared intensity image, and a registered infrared polarization image;

[0017] an image preprocessing unit, configured to perform radiation correction processing on the registered visible light intensity image, the registered visible light polarization image, the registered infrared intensity image, and the registered infrared polarization image to obtain a corrected visible light intensity image, a corrected visible light polarization image, a corrected infrared intensity image, and a corrected infrared polarization image;

[0018] An image fusion unit is used to fuse the corrected visible light intensity image, the corrected visible light polarization image, the corrected infrared intensity image and the corrected infrared polarization image using a residual network to obtain a fused image.

[0019] In a second aspect, the present application provides a method for detecting oil spills on sea surfaces carried by drones, which is applied to the above-mentioned drone-mounted oil spill detection system. The method comprises:

[0020] Acquire a multimodal image of the sea surface to be detected; the multimodal image includes a visible light intensity image, a visible light polarization image, an infrared intensity image, and an infrared polarization image;

[0021] performing image registration processing, radiation correction processing, and image fusion processing on the multimodal images to obtain a fused image;

[0022] The polarization degree of the fused image is determined, thereby determining the type of the oil spill according to the polarization degree.

[0023] Optionally, performing image registration processing, radiation correction processing, and image fusion processing on the multimodal images to obtain a fused image specifically includes:

[0024] registering the visible light intensity image, the visible light polarization image, the infrared intensity image, and the infrared polarization image to obtain a registered visible light intensity image, a registered visible light polarization image, a registered infrared intensity image, and a registered infrared polarization image;

[0025] performing radiation correction processing on the registered visible light intensity image, the registered visible light polarization image, the registered infrared intensity image, and the registered infrared polarization image to obtain a corrected visible light intensity image, a corrected visible light polarization image, a corrected infrared intensity image, and a corrected infrared polarization image;

[0026] The corrected visible light intensity image, the corrected visible light polarization image, the corrected infrared intensity image and the corrected infrared polarization image are fused using a residual network to obtain a fused image.

[0027] Optionally, registering the visible light polarization image and the infrared polarization image to obtain a registered visible light polarization image and a registered infrared polarization image specifically includes:

[0028] extracting a first feature point set of the visible light polarization image and a second feature point set of the infrared polarization image using a ray profile feature point extraction method;

[0029] Calculating a first mapping transformation matrix from a visible light polarization image to an infrared polarization image based on the first feature point set and the second feature point set;

[0030] Determining whether the number of first polarization image pairs is greater than a set number of groups, wherein the first polarization image pairs include a visible light polarization image and an infrared polarization image collected at the same time;

[0031] If not, continue to acquire multimodal images of the sea surface to be detected;

[0032] If yes, then take the mean of the first mapping transformation matrices of the set number of groups to obtain a mapping transformation mean matrix;

[0033] determining a root mean square error between a first mapping transformation matrix of the remaining first polarization image pair and the mapping transformation mean matrix;

[0034] If the root mean square error is less than the set error value, the mapping transformation mean matrix is ​​used as the image registration matrix;

[0035] Based on the image registration matrix, the visible light polarization image and the infrared polarization image are registered to obtain a registered visible light polarization image and a registered infrared polarization image.

[0036] Optionally, determining the polarization degree of the fused image specifically includes:

[0037] Using the formula Determine the polarization degree of the fused image; where, DOLP is the polarization degree of the fused image; is the coefficient of the specular reflection component; ; ; is the diffuse reflectance; is the diffuse reflection component; ; is the coefficient of the scattered component; is the scattered component; ;

[0038] M ij s is the Mueller matrix element; q is a generalized parameter used to determine wide-angle scattering; θ i is the zenith angle of the incident direction; θ r is the zenith angle of the reflection direction; α is the angle between the microfacet normal and the macronormal of the rough surface; β is the angle between the incident light and the microfacet normal; σ is the surface roughness constant of the object; τ is the parameter that characterizes the shadow of the rough surface; Ω is the parameter that characterizes the occlusion effect, is the coefficient of the multiple scattering component; is the coefficient of the single scattering component; M 10 、M 00 is an element in the Mueller matrix; v is the incident angle of the scattering surface; v0 is the exit angle of the scattering surface.

[0039] In a third aspect, the present application provides a computer device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement any of the above-described methods for detecting sea surface oil spills by unmanned aerial vehicles.

[0040] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the above-mentioned drone-borne sea surface oil spill detection methods.

[0041] In a fifth aspect, the present application provides a computer program product, including a computer program, which, when executed by a processor, implements any of the above-mentioned drone-borne sea surface oil spill detection methods.

[0042] According to the specific embodiments provided in this application, this application has the following technical effects:

[0043] The present application provides an unmanned aerial vehicle (UAV)-based sea surface oil spill detection system, method, device, medium, and product. The system comprises: a polarization characteristic detection module, an image fusion module, and a data processing module. The polarization characteristic detection module comprises a pod, a UAV, and an image capture unit. The pod is suspended directly below the UAV. The image capture unit is placed inside the pod. The image capture unit is connected to the image fusion module. The image fusion module is connected to the data processing module. The UAV is configured to hover at a set position according to a set navigation route. The image capture unit is configured to capture multimodal images of the sea surface to be detected while hovering. The multimodal images include a visible light intensity image, a visible light polarization image, an infrared intensity image, and an infrared polarization image. The image fusion module is configured to perform image registration, radiation correction, and image fusion on the multimodal images to obtain a fused image. The data processing module is configured to determine the polarization degree of the fused image, thereby determining the type of oil spill based on the polarization degree. The present application combines visible light detection with polarization infrared detection to compare and analyze the polarization degrees of common marine oil spills, thereby achieving oil spill classification. This application combines fused images of infrared and visible light information to better utilize the indicative characteristics and texture details of the source image, making it easier to effectively distinguish between oil spills and other marine species such as algae, biofilms, foam, and different types of oil spills during detection, thereby improving detection accuracy and achieving all-weather detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0045] Figure 1 A schematic structural diagram of an unmanned aerial vehicle (UAV)-mounted sea surface oil spill detection system provided in one embodiment of the present application;

[0046] Figure 2 This is the structural block diagram of the UAV-mounted sea surface oil spill detection system;

[0047] Figure 3 Schematic diagram of the drone pod device for the drone-mounted sea surface oil spill detection system;

[0048] Figure 4 The flight trajectory diagram of the UAV carrying sea oil spill detection system;

[0049] Figure 5 This is a flow chart of the registration method for four types of images in the UAV-borne sea oil spill detection system;

[0050] Figure 6 This is the image fusion block diagram of four types of images in the UAV-mounted sea oil spill detection system;

[0051] Figure 7 A schematic diagram of the structure of a computer device provided in one embodiment of the present application.

[0052] Reference numerals: 1 - drone; 2 - pod; 3 - rotating wheel; 4 - visible light intensity camera; 5 - visible light polarization camera; 6 - infrared intensity camera; 7 - infrared polarization camera. DETAILED DESCRIPTION

[0053] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0054] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0055] The pBRDF measurement system accurately measures the reflective properties of an object's surface, including the intensity and polarization state of reflected light at different angles and wavelengths. Drones equipped with such systems can perform high-precision aerial measurements and inspections of ground targets. Drone-mounted pBRDF measurement systems have broad application prospects. In agriculture, drone-mounted pBRDF measurement systems can monitor crop growth, pest and disease conditions, and soil moisture, providing data support for precision agriculture. In the military, drone-mounted pBRDF measurement systems can be used for target reconnaissance and camouflage identification, enhancing the stealth and accuracy of military operations. Drone-mounted pBRDF measurement systems can transmit measurement data in real time, providing timely and accurate information support to decision-makers. Drones offer flexible deployment capabilities, allowing them to quickly reach designated areas for measurement and inspection, unrestricted by ground transportation and terrain. These advantages make drone-mounted polarization BRDF systems promising broad application prospects and significant practical value in a variety of fields.

[0056] Spectral polarization imaging technology organically combines spectral imaging and polarization imaging technologies, rendering the data acquired by the instrument multidimensional, greatly enriching the target's information content. This multidimensional information can enhance the accuracy of measurement results. Not only does it provide more information, but it can also simultaneously acquire multidimensional data and images, including intensity, polarization, and spectrum, enabling faster and more accurate identification of oil spill types and oil film thickness. The advantageous properties of polarization can also reduce the impact of environmental factors, improving the detector's detection capabilities and the stability of information acquisition. Combining spectral imaging and polarization imaging technologies allows for the differentiation of oil spill types through multidimensional polarization images, while enabling multidimensional wide-area search and precise identification of surface oil spills, providing decision-making information for rapid and accurate monitoring and source tracing of surface oil spills.

[0057] This application fuses infrared and visible light polarization information, which can better utilize the texture details of the source image, facilitate effective differentiation of oil spill types during detection, reduce false alarm rates, and give full play to the advantages of multi-source information fusion.

[0058] In an exemplary embodiment, Figure 1-Figure 3 As shown, a UAV-mounted sea surface oil spill detection system is provided, comprising: a polarization characteristic detection module (i.e. Figure 2 Polarization characteristic detection system in the image fusion module (i.e. Figure 2 Image fusion system in the Figure 2The polarization characteristic detection module includes a pod 2, an unmanned aerial vehicle (UAV) 1, and an image capture unit. The pod 2 is suspended directly below the UAV 1. The image capture unit is placed inside the pod 2. The image capture unit is connected to the image fusion module. The image fusion module is connected to the data processing module. This system can measure the pBRDF of oil spills on the sea surface within a hemispherical surface.

[0059] In actual application, the polarization characteristic detection module includes a drone 1 with a pod 2 suspended. Drone 1 is an A660 multi-rotor drone. Multiple cameras are placed inside pod 2: a visible light intensity camera 4, an infrared intensity camera 6, a visible light polarization camera 5, and an infrared polarization camera 7. Pod 2 is located directly below drone 1 and connected to drone 1 via a wheel 3. Remote control is used from the ground to control the flight of drone 1 and adjust the flight direction of drone 1 and the shooting directions of the multiple cameras. The multiple cameras include a Daheng visible light intensity camera, a Beifang Guangwei infrared intensity camera, a Daheng visible light polarization camera, and a Beifang Guangwei infrared polarization camera.

[0060] The visible light intensity camera 4 is used to capture a visible light intensity image of the sea surface to be detected.

[0061] The infrared intensity camera 6 is used to capture infrared intensity images of the sea surface to be detected.

[0062] The visible light polarization camera 5 is used to capture visible light polarization images of the sea surface to be detected.

[0063] The infrared polarization camera 7 is used to capture infrared polarization images of the sea surface to be detected.

[0064] The drone 1 is used to hover at a set position according to a set navigation route; the image capture unit is used to capture multimodal images of the sea surface to be detected while hovering; the multimodal images include visible light intensity images, visible light polarization images, infrared intensity images, and infrared polarization images.

[0065] In this embodiment, an experiment is conducted by taking the sea surface oil spill conditions simulated in a large outdoor pool as an example to illustrate the UAV-borne sea surface oil spill detection system of the present application.

[0066] The environmental conditions are outdoor and the light source is natural light.

[0067] Aircraft trajectory diagram Figure 4 As shown, UAV 1 first flies to the highest point in the aircraft trajectory diagram, that is, at a zenith angle of 0°, and the camera observes the surface target vertically downward; the set navigation route of UAV 1 is expressed by the following formula:

[0068] The observation point of UAV 1 at a specific zenith angle and azimuth angle can be expressed by the following formula:

[0069] .

[0070] Where, f is the function describing the relationship between the position of UAV 1 and the camera angle, θ is the zenith angle, is the azimuth angle, H is the initial height of UAV 1, and R is the initial flight radius of UAV 1.

[0071] The altitude and flight radius of UAV 1 follow the following formula:

[0072] H′=H-kθ.

[0073] Where H′ is the height of UAV 1 at the corresponding zenith angle; k is the zenith angle adjustment coefficient.

[0074] .

[0075] Where R′ is the flight radius of UAV 1 at the corresponding azimuth angle; d is the azimuth adjustment coefficient.

[0076] Zenith angle θ, azimuth angle The following relationship is satisfied:

[0077] .

[0078] .

[0079] in, represents the rounding symbol, and θ0 is the initial zenith angle.

[0080] The zenith angle θ ranges from 0° to 60°, and the azimuth angle The observation range is 0°-360°, and the azimuth observation interval is 30°. The pitch angle and flight radius of the camera on the stable platform are changed to complete hovering observation at different azimuths. The camera lens is always facing the oil spill surface.

[0081] Then, the visible light intensity camera 4 , the visible light polarization camera 5 , the infrared intensity camera 6 and the infrared polarization camera 7 respectively collect visible light intensity images, visible light polarization images, infrared intensity images and infrared polarization images of the oil spill surface at each observation point.

[0082] The image fusion module is used to perform image registration processing, radiation correction processing and image fusion processing on the multimodal images to obtain a fused image.

[0083] As an optional implementation manner, the image fusion module includes:

[0084] An image registration unit is used to register the visible light intensity image, the visible light polarization image, the infrared intensity image and the infrared polarization image to obtain a registered visible light intensity image, a registered visible light polarization image, a registered infrared intensity image and a registered infrared polarization image.

[0085] An image preprocessing unit is used to perform radiation correction processing on the registered visible light intensity image, the registered visible light polarization image, the registered infrared intensity image and the registered infrared polarization image to obtain a corrected visible light intensity image, a corrected visible light polarization image, a corrected infrared intensity image and a corrected infrared polarization image.

[0086] An image fusion unit is used to fuse the corrected visible light intensity image, the corrected visible light polarization image, the corrected infrared intensity image and the corrected infrared polarization image using a residual network to obtain a fused image.

[0087] In practical applications, during the image registration process of visible light intensity images, visible light polarization images, infrared intensity images, and infrared polarization images, the ray profile feature point extraction method is used to extract the first feature point set K and the second feature point set I of the visible light polarization image and the infrared polarization image.

[0088] The corresponding points of the visible light polarization image and the infrared polarization image to be registered are mapped using the transformation matrix H to achieve registration between the image pairs.

[0089] Furthermore, the mapping transformation matrix H is the homography matrix, which is expressed by the following formula:

[0090] .

[0091] The conversion relationship between the feature points of infrared polarization image and visible light polarization image can be written as follows:

[0092] .

[0093] Where (x, y) is the coordinate of the feature point of the visible light polarization image; (x', y') is the coordinate of the feature point of the infrared polarization image; h 00 -h 33 is the parameter of the homography matrix, σ H is the scale parameter.

[0094] Taking visible light polarization image and infrared polarization image as examples, the registration process is explained as follows: Figure 5 As shown, the specific process is as follows:

[0095] 1) Determine whether the number of image pairs (first polarization image pairs) of the collected visible light polarization image and infrared polarization image is greater than a set number of groups. In this embodiment, the set number of groups is 30. If not, proceed to step 2); if so, proceed to step 3).

[0096] 2) Select four pairs of feature points from the second feature point set I of the infrared polarization image and the first feature point set K of the visible light polarization image, and calculate the H matrix from the visible light polarization image to the infrared polarization image plane; continue with step 1).

[0097] 3) Take the mean M (mapping transformation mean matrix) of 30 groups of H matrices and save it.

[0098] 4) Determine the remaining groups, calculate the H matrix, and output the calibration results: Calculate the root mean square error between the H matrix of the remaining group and the mean M. If the error value is less than 30%, the mean M of the H matrix is ​​the image registration matrix for the visible light polarization image and the infrared polarization image. The visible light polarization image and the infrared polarization image are registered based on this image registration matrix.

[0099] like Figure 6 As shown in FIG, the registered visible light intensity image, the registered visible light polarization image, the registered infrared intensity image and the registered infrared polarization image are subjected to radiation correction preprocessing. First, bilinear interpolation is used to resample the images with mismatched resolution or pixel coordinates so that the images of all modalities maintain consistent resolution on the same plane. Then, histogram matching or other radiation correction methods are used to further balance the differences in grayscale value distribution or contrast between the visible light image and the infrared image, so as to obtain more unified and comparable multimodal input data. Then, the registered visible light intensity image and the registered visible light polarization image are combined in the network structure. The images are merged into two channels. The registered infrared intensity image and the registered infrared polarization image are also merged into two channels. These are fed into two improved residual network branches for multi-scale deep feature extraction. At the input layer, the original three-channel convolution is replaced with a two-channel-adapted convolution kernel, and some shallow convolution kernels are frozen to optimize shallow feature degradation during the initial training phase. The extracted multi-scale deep features are compressed to 512 channels using 1×1 convolutions and then fused element-by-element. The ECA-Net channel attention and CBAM spatial attention modules are introduced to adaptively assign feature weights in the channel and spatial dimensions, while residual connections are used to preserve the original information. Layer-by-layer transposed convolutions are then used to upsample to the target resolution. Low-level features from each stage of the residual network are concatenated to enhance texture and edge information. Finally, a 3×3 convolution is performed to output the fused image.

[0100] The data processing module is used to determine the polarization degree of the fused image, thereby determining the type of the oil spill according to the polarization degree.

[0101] The data processing module calculates the pBRDF data of the fused image, and then obtains polarization characteristic parameters such as the polarization degree of the fused image. By comparing the polarization characteristic parameters of different oil spill types and analyzing their polarization characteristic differences, the oil spill types can be effectively distinguished. Oil spill types include crude oil, diesel, gasoline, kerosene, etc.

[0102] Use pBRDF to describe the polarization degree of an object. The expression of the pBRDF model is as follows:

[0103] .

[0104] Among them, k s 、k m 、k scat are the coefficients of the specular reflection component, the diffuse reflection component, and the scattered component respectively; f s is the specular reflection component; f m is the diffuse reflection component; f scat is the scattered component; M ij s is the Mueller matrix element; q is a generalized parameter used to determine wide-angle scattering; θ i is the zenith angle of the incident direction; θ r is the zenith angle of the reflection direction; α is the angle between the microfacet normal and the macronormal of the rough surface; β is the angle between the incident light and the microfacet normal; σ is the surface roughness constant of the object; is the parameter that characterizes the shadow of the rough surface; Ω is the parameter that characterizes the occlusion effect, is the coefficient of the multiple scattering component; is the coefficient of the single scattering component; v is the incident angle of the scattering surface; v0 is the exit angle of the scattering surface.

[0105] Under passive lighting conditions of non-polarized light such as natural light, the Stokes vector of the incident light is expressed as , then the expression of the Stokes vector of the reflected light is as follows:

[0106] .

[0107] Where I is the total light intensity; Q is the linear polarization component in the horizontal and vertical directions; U is the linear polarization component in the ±45° direction; V is the circular polarization component; f 00 -f 33 is the Mueller matrix Elements in; E0, E1, E2, E3 are Stokes vectors Parameters in .

[0108] Calculate the polarization degree under natural light illumination, the expression is as follows:

[0109] .

[0110] Among them, f s10 、f s20 、f s30 is the specular reflection component in the Mueller matrix Elements in .

[0111] Since the circular polarization component in natural light is small, it can be ignored in reflected light. The incident light and pBRDF matrix can be reduced in dimension. , then the Stokes vector expression of the reflected light is as follows:

[0112] .

[0113] Then, after ignoring the circular polarization component, the polarization degree that satisfies the coplanar relationship can be further simplified, and the expression is as follows:

[0114]

[0115] Among them, M 10 、M 00 represents the Mueller matrix Elements in .

[0116] The drone-mounted sea oil spill detection system proposed in this application combines visible light detection with polarized infrared detection to compare and analyze the visible light / infrared polarization characteristics of common marine oil spills, enabling differentiation of spilled oil species. The proposed image fusion solution can reduce computational cost and complexity. The fused image, combining infrared and visible light information, can better utilize the indicative properties and texture details of the source image, facilitating effective differentiation between oil spills and other marine species such as algae, biofilms, and foam, as well as between different types of oil spills, improving detection accuracy and enabling all-weather detection.

[0117] Based on the same inventive concept, an embodiment of the present application also provides a drone-borne sea surface oil spill detection method implemented based on the above-mentioned drone-borne sea surface oil spill detection system.

[0118] In an exemplary embodiment, a method for detecting oil spills on a sea surface using an unmanned aerial vehicle is provided, comprising:

[0119] Acquire a multimodal image of the sea surface to be detected; the multimodal image includes a visible light intensity image, a visible light polarization image, an infrared intensity image, and an infrared polarization image.

[0120] The multimodal images are subjected to image registration processing, radiation correction processing and image fusion processing to obtain a fused image.

[0121] As an optional implementation manner, performing image registration processing, radiation correction processing, and image fusion processing on the multimodal images to obtain a fused image specifically includes:

[0122] The visible light intensity image, the visible light polarization image, the infrared intensity image, and the infrared polarization image are registered to obtain a registered visible light intensity image, a registered visible light polarization image, a registered infrared intensity image, and a registered infrared polarization image.

[0123] Radiation correction processing is performed on the registered visible light intensity image, the registered visible light polarization image, the registered infrared intensity image, and the registered infrared polarization image to obtain a corrected visible light intensity image, a corrected visible light polarization image, a corrected infrared intensity image, and a corrected infrared polarization image.

[0124] The corrected visible light intensity image, the corrected visible light polarization image, the corrected infrared intensity image and the corrected infrared polarization image are fused using a residual network to obtain a fused image.

[0125] As an optional implementation manner, registering the visible light polarization image and the infrared polarization image to obtain a registered visible light polarization image and a registered infrared polarization image specifically includes:

[0126] A ray profile feature point extraction method is used to extract a first feature point set of the visible light polarization image and a second feature point set of the infrared polarization image.

[0127] A first mapping transformation matrix from the visible light polarization image to the infrared polarization image is calculated according to the first feature point set and the second feature point set.

[0128] It is determined whether the number of first polarization image pairs is greater than a set number of groups; the first polarization image pairs include a visible light polarization image and an infrared polarization image collected at the same time.

[0129] If not, continue to acquire multimodal images of the sea surface to be detected.

[0130] If so, the mean of the first mapping transformation matrices of the set number of groups is taken to obtain a mapping transformation mean matrix.

[0131] A root mean square error between the first mapping transformation matrix and the mapping transformation mean matrix of the remaining first polarization image pair is determined.

[0132] If the root mean square error is less than the set error value, the mapping transformation mean matrix is ​​used as the image registration matrix.

[0133] Based on the image registration matrix, the visible light polarization image and the infrared polarization image are registered to obtain a registered visible light polarization image and a registered infrared polarization image.

[0134] The polarization degree of the fused image is determined, thereby determining the type of the oil spill according to the polarization degree.

[0135] As an optional implementation manner, determining the polarization degree of the fused image specifically includes:

[0136] Using the formula Determine the polarization degree of the fused image; where, DOLP is the polarization degree of the fused image; is the coefficient of the specular reflection component; ; ; is the diffuse reflectance; is the diffuse reflection component; ; is the coefficient of the scattered component; is the scattered component; .

[0137] In an exemplary embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor implements the above-mentioned drone-borne sea surface oil spill detection method when executing the computer program.

[0138] In an exemplary embodiment, a computer-readable storage medium is provided, storing a computer program, which, when executed by a processor, implements the above-mentioned drone-borne sea surface oil spill detection method.

[0139] In an exemplary embodiment, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the above-mentioned drone-borne sea surface oil spill detection method.

[0140] In an exemplary embodiment, a computer device is provided. The computer device may be a server or a terminal. The internal structure diagram thereof may be as follows: Figure 7As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, abbreviated as I / O) and a communication interface. The processor, memory and input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a method for detecting oil spills on the sea surface on an unmanned aerial vehicle is implemented.

[0141] Those skilled in the art will understand that Figure 7 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0142] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0143] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the above-mentioned embodiments. In particular, any reference to memory, database, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM may be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0144] The databases involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, distributed databases based on blockchains. The processors involved in the various embodiments provided herein may include, but are not limited to, general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic units, data processing logic units based on quantum computing, and the like.

[0145] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0146] This document uses specific examples to illustrate the principles and implementation methods of this application. The description of the above examples is only intended to help understand the method and core concept of this application. At the same time, for those skilled in the art, based on the concept of this application, there may be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. An unmanned aerial vehicle (UAV) sea surface oil spill detection system, characterized in that: include: A polarization characteristic detection module, an image fusion module, and a data processing module; the polarization characteristic detection module includes a pod, a drone, and an image capture unit; the pod is suspended directly below the drone; the image capture unit is placed inside the pod; the image capture unit is connected to the image fusion module; and the image fusion module is connected to the data processing module. The UAV is configured to hover at a set position according to a set navigation route; the image capturing unit is configured to capture multimodal images of the sea surface to be detected while hovering; the multimodal images include a visible light intensity image, a visible light polarization image, an infrared intensity image, and an infrared polarization image; The image fusion module is used to perform image registration processing, radiation correction processing and image fusion processing on the multimodal image to obtain a fused image; The image fusion module includes: an image registration unit, configured to register the visible light intensity image, the visible light polarization image, the infrared intensity image, and the infrared polarization image to obtain a registered visible light intensity image, a registered visible light polarization image, a registered infrared intensity image, and a registered infrared polarization image; an image preprocessing unit, configured to perform radiation correction processing on the registered visible light intensity image, the registered visible light polarization image, the registered infrared intensity image, and the registered infrared polarization image to obtain a corrected visible light intensity image, a corrected visible light polarization image, a corrected infrared intensity image, and a corrected infrared polarization image; an image fusion unit, configured to fuse the corrected visible light intensity image, the corrected visible light polarization image, the corrected infrared intensity image, and the corrected infrared polarization image using a residual network to obtain a fused image; The corrected visible light intensity image and the corrected visible light polarization image are merged into 2 channels, and the corrected infrared intensity image and the corrected infrared polarization image are also merged into 2 channels. They are respectively sent to two improved residual network branches for multi-scale deep feature extraction, and the original 3-channel convolution is replaced by the 2-channel convolution kernel in the input layer and the shallow part of the convolution kernel is frozen; the extracted multi-scale deep features are compressed to 512 channels by 1×1 convolution and then fused element by element. The ECA-Net channel attention and CBAM spatial attention modules are introduced to adaptively allocate feature weights in the channel dimension and spatial dimension, and the residual connection is used to retain the original information; then, layer-by-layer transposed convolution is used to upsample to the target resolution and the low-level features of each stage of the residual network are spliced ​​to enhance texture and edge information. Finally, the fused image is output through 3×3 convolution; The data processing module is used to determine the polarization degree of the fused image, thereby determining the type of oil spill according to the polarization degree; Determining the polarization degree of the fused image specifically includes: Using the formula Determine the polarization degree of the fused image; where, DOLP is the polarization degree of the fused image; is the coefficient of the specular reflection component; ; ; is the diffuse reflectance; is the diffuse reflection component; ; is the coefficient of the scattered component; is the scattered component; ; M ij s is the Mueller matrix element; q is a generalized parameter used to determine wide-angle scattering; θ i is the zenith angle of the incident direction; θ r is the zenith angle of the reflection direction; α is the angle between the microfacet normal and the macronormal of the rough surface; β is the angle between the incident light and the microfacet normal; σ is the surface roughness constant of the object; τ is the parameter that characterizes the shadow of the rough surface; Ω is the parameter that characterizes the occlusion effect, is the coefficient of the multiple scattering component; is the coefficient of the single scattering component; M 10 、M 00 is an element in the Mueller matrix; v is the incident angle of the scattering surface; v0 is the exit angle of the scattering surface.

2. The UAV-mounted sea surface oil spill detection system according to claim 1, characterized in that: The image capturing unit includes: a visible light intensity camera, an infrared intensity camera, a visible light polarization camera and an infrared polarization camera; The visible light intensity camera is used to capture a visible light intensity image of the sea surface to be detected; The infrared intensity camera is used to capture infrared intensity images of the sea surface to be detected; The visible light polarization camera is used to capture a visible light polarization image of the sea surface to be detected; The infrared polarization camera is used to capture infrared polarization images of the sea surface to be detected.

3. A method for detecting oil spills on sea surfaces using an unmanned aerial vehicle, characterized in that: The unmanned aerial vehicle (UAV)-borne sea surface oil spill detection method is applied to the unmanned aerial vehicle (UAV)-borne sea surface oil spill detection system according to any one of claims 1 to 2, and the unmanned aerial vehicle (UAV)-borne sea surface oil spill detection method comprises: Acquire a multimodal image of the sea surface to be detected; the multimodal image includes a visible light intensity image, a visible light polarization image, an infrared intensity image, and an infrared polarization image; performing image registration processing, radiation correction processing, and image fusion processing on the multimodal images to obtain a fused image; Performing image registration processing, radiation correction processing, and image fusion processing on the multimodal images to obtain a fused image, specifically including: registering the visible light intensity image, the visible light polarization image, the infrared intensity image, and the infrared polarization image to obtain a registered visible light intensity image, a registered visible light polarization image, a registered infrared intensity image, and a registered infrared polarization image; performing radiation correction processing on the registered visible light intensity image, the registered visible light polarization image, the registered infrared intensity image, and the registered infrared polarization image to obtain a corrected visible light intensity image, a corrected visible light polarization image, a corrected infrared intensity image, and a corrected infrared polarization image; Using a residual network, fusing the corrected visible light intensity image, the corrected visible light polarization image, the corrected infrared intensity image, and the corrected infrared polarization image to obtain a fused image; The corrected visible light intensity image and the corrected visible light polarization image are merged into 2 channels, and the corrected infrared intensity image and the corrected infrared polarization image are also merged into 2 channels. They are respectively sent to two improved residual network branches for multi-scale deep feature extraction, and the original 3-channel convolution is replaced by the 2-channel convolution kernel in the input layer and the shallow part of the convolution kernel is frozen; the extracted multi-scale deep features are compressed to 512 channels by 1×1 convolution and then fused element by element. The ECA-Net channel attention and CBAM spatial attention modules are introduced to adaptively allocate feature weights in the channel dimension and spatial dimension, and the residual connection is used to retain the original information; then, layer-by-layer transposed convolution is used to upsample to the target resolution and the low-level features of each stage of the residual network are spliced ​​to enhance texture and edge information. Finally, the fused image is output through 3×3 convolution; determining a degree of polarization of the fused image, thereby determining the type of the oil spill according to the degree of polarization; Determining the polarization degree of the fused image specifically includes: Using the formula Determine the polarization degree of the fused image; where, DOLP is the polarization degree of the fused image; is the coefficient of the specular reflection component; ; ; is the diffuse reflectance; is the diffuse reflection component; ; is the coefficient of the scattered component; is the scattered component; ; M ij s is the Mueller matrix element; q is a generalized parameter used to determine wide-angle scattering; θ i is the zenith angle of the incident direction; θ r is the zenith angle of the reflection direction; α is the angle between the microfacet normal and the macronormal of the rough surface; β is the angle between the incident light and the microfacet normal; σ is the surface roughness constant of the object; τ is the parameter that characterizes the shadow of the rough surface; Ω is the parameter that characterizes the occlusion effect, is the coefficient of the multiple scattering component; is the coefficient of the single scattering component; M 10 、M 00 is an element in the Mueller matrix; v is the incident angle of the scattering surface; v0 is the exit angle of the scattering surface.

4. The method for detecting sea surface oil spills by drone according to claim 3, characterized in that: Registering the visible light polarization image and the infrared polarization image to obtain a registered visible light polarization image and a registered infrared polarization image specifically includes: extracting a first feature point set of the visible light polarization image and a second feature point set of the infrared polarization image using a ray profile feature point extraction method; Calculating a first mapping transformation matrix from a visible light polarization image to an infrared polarization image based on the first feature point set and the second feature point set; Determining whether the number of first polarization image pairs is greater than a set number of groups, wherein the first polarization image pairs include a visible light polarization image and an infrared polarization image collected at the same time; If not, continue to acquire multimodal images of the sea surface to be detected; If yes, then take the mean of the first mapping transformation matrices of the set number of groups to obtain a mapping transformation mean matrix; determining a root mean square error between a first mapping transformation matrix of the remaining first polarization image pair and the mapping transformation mean matrix; If the root mean square error is less than the set error value, the mapping transformation mean matrix is ​​used as the image registration matrix; Based on the image registration matrix, the visible light polarization image and the infrared polarization image are registered to obtain a registered visible light polarization image and a registered infrared polarization image.

5. A computer device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method for detecting oil spills on sea surfaces carried out by an unmanned aerial vehicle according to any one of claims 3 to 4.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for detecting sea surface oil spills carried out by an unmanned aerial vehicle according to any one of claims 3 to 4 is implemented.

7. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the method for detecting sea surface oil spills carried out by an unmanned aerial vehicle according to any one of claims 3 to 4 is implemented.

Images