Marine ecological environmental protection image surveying method and device

Abstract

The invention discloses a marine ecological environmental protection image surveying method and device, which organically integrates topographic surveying and mapping and ecological recognition, obtains high-precision three-dimensional topography and rich ecological semantic information at the same time in one-time surveying, and remarkably improves the surveying efficiency. By introducing a water refraction correction algorithm, the problem of distortion of underwater imaging is effectively solved, and the positioning precision of an underwater target is improved from a meter level to a centimeter level; according to the invention, air, water surface and underwater multi-platform data and visible light, multispectral, sonar and other multi-modal data are fused, the limitation of a single sensor is overcome, and omnibearing marine environment perception is realized; the generated semantic point cloud data structure not only contains geometric information, but also contains ecological semantic information, and provides direct data support for subsequent intelligent analysis.

Description

Technical Field

[0001] This invention belongs to the field of marine surveying technology, specifically relating to a method and apparatus for marine ecological and environmental protection image surveying. Background Technology

[0002] As an important part of the Earth's surface, the ocean contains abundant natural resources. Accurately obtaining marine topographic information and ecological environment data is of great significance for marine resource development, ecological environment protection, coastal zone management, marine engineering construction, and scientific research. Traditional marine surveying methods mainly rely on large survey vessels equipped with sonar equipment for underwater topographic surveying, or on divers for manual observation and sampling. These methods suffer from problems such as high cost, low efficiency, adverse impacts on the marine ecosystem, and difficulty in obtaining high-resolution data.

[0003] In recent years, with the rapid development of unmanned aerial vehicle (UAV) technology, underwater robot technology, remote sensing technology, and artificial intelligence technology, marine surveying methods based on multi-platform collaboration and intelligent processing have gradually become a research hotspot. Current marine surveying methods generally suffer from the following core technical problems: Separation of surveying and identification: It is impossible to obtain high-precision terrain data and ecological environment data in a single operation; Low underwater positioning accuracy: There is a lack of effective water refraction correction mechanism, resulting in large deviations in the position of underwater targets; Missing semantic information: Traditional point cloud data only contains geometric information and lacks ecological semantic information, making it difficult to use directly for intelligent analysis. Summary of the Invention

[0004] To address the problems of separation between surveying and identification, low underwater positioning accuracy, and lack of semantic information in the existing technologies, the purpose of this invention is to provide a digital and intelligent marine ecological and environmental protection image surveying method, comprising the following steps: S1. Multi-source heterogeneous survey data of the target water area are collected by a multi-source data acquisition terminal in a collaborative operation; the multi-source heterogeneous survey data includes at least: aerial image data above the water surface and underwater image data below the water surface; S2. The multi-source heterogeneous survey data is preprocessed and fused across media to generate a three-dimensional point cloud model of the target water area; the preprocessing includes correcting water body refraction distortion in the underwater image data; S3. Use a pre-trained target recognition model to identify ecological targets in the aerial and underwater image data, and obtain the recognition results of ecological targets in the two-dimensional images; S4. Establish a mapping relationship between two-dimensional images and three-dimensional point cloud models, and map the recognition results in the two-dimensional images to the three-dimensional point cloud models according to the mapping relationship; S5. Based on the mapped results, generate a semantic 3D point cloud containing semantic information of ecological targets, and output the survey results.

[0005] Furthermore, the multi-source data acquisition terminal includes at least one of the following: an aerial drone equipped with imaging equipment, a surface unmanned vessel equipped with underwater acoustic detection equipment or water quality monitoring equipment, or an underwater robot equipped with underwater imaging equipment.

[0006] Furthermore, the underwater image data refraction distortion correction includes: calculating the deviation vector of the light propagation path based on the refraction model of light at the air-water interface, and compensating and correcting the three-dimensional coordinates corresponding to the underwater image according to the deviation vector.

[0007] Furthermore, the multi-source heterogeneous survey data also includes multispectral image data; The ecological target identification includes: extracting spectral features from multispectral image data, color features from visible light image data, and texture features; fusing the spectral features, color features, and texture features to construct a multimodal feature vector; and inputting the multimodal feature vector into the target identification model for classification and localization.

[0008] Furthermore, establishing the mapping relationship between the two-dimensional image and the three-dimensional point cloud model includes: Based on the camera imaging model, a forward projection relationship from 3D point cloud coordinates to 2D image pixel coordinates is established, as well as a reverse index table from 2D image pixel coordinates to 3D point cloud coordinates is established. When the target recognition model outputs a two-dimensional bounding box of the target region, the three-dimensional point cloud set corresponding to the pixels in that region is queried through the inverse index table.

[0009] Furthermore, the generation of the three-dimensional point cloud model of the target water area includes: Motion reconstruction structure calculations are performed on image data from different sources to generate sparse point clouds; The point cloud generated from lidar data or sonar data is registered and fused with the sparse point cloud; the registration and fusion includes coarse registration based on feature descriptors and fine registration based on iterative nearest points.

[0010] Furthermore, the spectral features include at least one of the normalized vegetation index, turbidity index, and normalized water index. The target recognition model is a deep learning model based on convolutional neural networks, and the ecological targets include corals, algae, aquatic organisms, and water pollutants.

[0011] Furthermore, the output survey results include: Statistical semantic analysis of the quantity, volume, and spatial distribution of various ecological targets in 3D point clouds; The semantic tags are visualized and rendered onto a 3D terrain model to generate a comprehensive electronic map containing terrain information and ecological elements.

[0012] Furthermore, the method employs an edge-cloud collaborative processing architecture: Real-time data stitching and preliminary target identification are performed on the data acquisition device or edge computing node; Global area network adjustment, high-precision point cloud fusion, and semantic mapping are performed on cloud servers.

[0013] A marine ecological and environmental protection image surveying device, comprising: The data acquisition module is used to acquire multi-source heterogeneous survey data of the target water area; The 3D reconstruction module is used to establish a 3D control network based on high-precision positioning control points and perform regional network adjustment. It uses the structure-of-motion algorithm to generate image point clouds, performs feature extraction, iterative nearest point registration and fusion processing on multi-source point cloud data, and generates fused point cloud data containing 3D geometric coordinates and image texture information. The intelligent recognition module is used to input multimodal feature vectors and visible light image data into a deep learning recognition model to identify corals, algae, fish and pollutants, and output two-dimensional bounding box coordinates, category labels and recognition confidence scores. The semantic mapping module is used to convert the two-dimensional recognition results into three-dimensional point cloud coordinates according to the mapping relationship, perform semantic annotation on the three-dimensional point cloud, and generate a semantic point cloud.

[0014] The beneficial effects of this invention are as follows: Compared with the prior art, this invention organically integrates topographic mapping and ecological identification, simultaneously acquiring high-precision three-dimensional topographic data and rich ecological semantic information in a single survey, significantly improving survey efficiency; by introducing a water refraction correction algorithm, it effectively solves the distortion problem of underwater imaging, achieving high-precision positioning of underwater targets; it integrates data from multiple platforms including air, surface, and underwater, as well as multimodal data such as visible light, multispectral, and sonar, overcoming the limitations of a single sensor and achieving comprehensive marine environmental perception; the generated semantic point cloud data structure not only contains geometric information but also ecological semantic information, providing direct data support for subsequent intelligent analysis, such as biomass estimation and pollution source tracing; and it adopts an "edge-cloud" collaborative processing architecture, which can achieve real-time data stitching and preliminary verification through edge computing at the data collection site, avoiding data omissions; and can also utilize cloud computing power for high-precision global adjustment and model reconstruction, effectively balancing the timeliness requirements of on-site operations with the accuracy requirements of surveying results. Attached Figure Description

[0015] Figure 1 This is the overall flowchart of the present invention; Figure 2This is a schematic diagram of a scenario for the multi-platform collaborative data acquisition system of the present invention; Figure 3 This is a schematic diagram illustrating the principle of the mapping relationship between two-dimensional images and three-dimensional point clouds in this invention; Figure 4 This is a schematic diagram of the structure of the multimodal feature fusion and target recognition model of the present invention; Figure 5 This is a structural block diagram of the digital intelligent marine ecological and environmental protection image survey device of the present invention. Detailed Implementation

[0016] To make the implementation objectives, technical solutions, and features of this invention application clearer, the technical solutions of this invention application will be clearly and completely described below with reference to the accompanying drawings of the embodiments. Obviously, the described embodiments are only a part of the examples of this invention application, and not all of the implementation cases. The embodiments of this invention application described and shown in the accompanying drawings can generally be arranged and designed in different configurations. Example 1

[0017] This embodiment provides a digital and intelligent marine ecological and environmental protection image survey method, such as Figure 1 As shown, the method mainly includes the following steps: Step S1: Collect multi-source heterogeneous survey data of the target water area using a multi-source data acquisition terminal for collaborative operations.

[0018] In this step, in order to fully understand the marine environment, a three-dimensional collaborative operation mode is adopted to acquire multi-source heterogeneous data.

[0019] The multi-source heterogeneous data includes: Aerial imagery data: acquired by aerial drones operating above the water surface. Data types include high-resolution visible light RGB imagery for texture mapping and general target identification, and multispectral imagery for vegetation index analysis and water quality retrieval.

[0020] Underwater imagery data: acquired by underwater robots operating below the water surface. Data types include high-resolution underwater video streams and image sequences. Due to severe light attenuation underwater, acquisition equipment is typically equipped with LED supplemental lighting.

[0021] Other auxiliary data includes multibeam echo sounder data carried by unmanned surface vessels to obtain seabed topographic contours, water quality monitoring sensor data such as pH value, dissolved oxygen, and turbidity, as well as high-precision positioning data from various platforms and RTK GPS data.

[0022] Collaborative data collection includes the following steps: Collaborative control: Through a collaborative control system at a shore-based command center or on a mother ship, the routes of drones, unmanned surface vessels (USVs), and underwater robots are planned in a unified manner. For example, the USV acts as a relay node, with drones flying above the USV and underwater robots lurking below it, maintaining relative position synchronization.

[0023] Spatiotemporal synchronization: In terms of time, all acquisition devices are synchronized via NTP. Spatially, RTK is used to ensure that the position coordinates of all platforms are unified under the WGS coordinate system.

[0024] Step S2: Preprocess and fuse the multi-source heterogeneous survey data across media to generate a three-dimensional point cloud model of the target water area, including the following steps: S2.1 Data Preprocessing: Denoising and Enhancement: For aerial images, the dark channel prior algorithm is used to remove sea surface reflections and fog; for underwater images, the color restoration algorithm is used to remove the blue-green color cast caused by water, and contrast-limited adaptive histogram equalization is used to enhance image details.

[0025] S2.2 Water Refraction Distortion Correction: When light travels from an underwater object to a camera lens in the air, it is refracted at the water-air interface. Without correction, directly using a pinhole camera model in the air will cause the depth of the underwater object to be underestimated, making it appear shallower than it actually is.

[0026] A dual-medium imaging model is established based on Snell's law.

[0027] Let the coordinates of the camera's optical center be C, and the point on the imaging plane be p, with the corresponding incident ray vector being... .

[0028] Calculate the intersection point I of the light ray and the water surface. This requires knowledge of the water surface height (which can be obtained through the sonar or wave sensor of the unmanned surface vessel, based on an instantaneous wave-fitted plane).

[0029] According to refractive index and Calculate the refracted ray vector The formula is: ,in This is the water surface normal vector.

[0030] According to the refracted light The three-dimensional spatial coordinates corresponding to the original image are compensated and corrected to restore the true depth and position of the object.

[0031] S2.3 3D point cloud model generation: Aerial component: Using aerial imagery data, SIFT feature points are extracted through the Structure for Motion Recovery (SfM) algorithm, feature matching is performed, camera pose is calculated, and sparse point clouds are generated; then, dense point clouds are generated through the Multi-View Stereo Vision (MVS) algorithm.

[0032] Underwater portion: SfM-MVS reconstruction was also performed using the corrected underwater images.

[0033] S2.4 Cross-media fusion: Coarse registration: Large-scale features of seabed topography obtained by unmanned surface vessel sonar, such as the outline of reef groups, are used as common features, and FPFH feature descriptors are used for initial alignment.

[0034] Precise registration: An improved ICP algorithm is employed. Considering the density difference between underwater and sonar point clouds, a point-to-surface error metric function is used, and normal vector constraints are introduced. The transformation matrix is ​​iteratively optimized until the error converges. Finally, the aerial and underwater point clouds are seamlessly stitched together to form a complete 3D point cloud model.

[0035] Step S3: Use a pre-trained target recognition model to identify ecological targets in the aerial and underwater image data, obtaining the recognition results of ecological targets in the two-dimensional images. The steps include: S3.1 Target Recognition Model Construction: Model architecture: A convolutional neural network (CNN) is used, specifically the YOLO architecture. To balance accuracy and speed, the backbone network uses CSPDarknet, and FPN is introduced to enhance the detection capability of targets at different scales, such as huge coral reefs and tiny starfish.

[0036] S3.2 Multimodal Feature Fusion: This invention incorporates multispectral data to enhance robustness and also includes a feature extraction step: Spectral characteristics: Normalized vegetation index, normalized water index and turbidity index were extracted from multispectral images.

[0037] Color characteristics: Extract hue and saturation components from the HSV color space.

[0038] Texture features: Texture features are extracted using LBP local binary mode.

[0039] Feature fusion: The above features are concatenated along the channel dimension to construct a multimodal feature vector. .

[0040] Input and output: The fused feature vector is input into the detection head, and the output includes the target category (e.g., coral, seaweed, fish, plastic waste), confidence score, and two-dimensional bounding box.

[0041] S3.3 model training: A marine ecology dataset containing tens of thousands of labeled images was constructed. CIoU Loss was used as the bounding box regression loss and Focal Loss was used as the classification loss to address the imbalance between positive and negative samples.

[0042] Step S4: Establish a mapping relationship between the 2D image and the 3D point cloud model, and map the recognition results in the 2D image to the 3D point cloud model according to the mapping relationship. This includes the following steps: S4.1 Forward Projection and Backward Indexing: Using the camera intrinsic parameter matrix K calculated in step S2 and the extrinsic parameter matrix [R|t] (rotation and translation) of each frame image, establish the projection relationship from the three-dimensional world coordinates (X,Y,Z) to the two-dimensional pixel coordinates (u,v): .

[0043] S4.2 Reverse Index Table Construction: Iterate through each point P_i in the 3D point cloud, project it onto all visible image frames I_j, and record the coordinates of the projected point (u{ij}, v {ij}).

[0044] Construct the index entries: {Frame_ID: {Pixel_Region: [Point_ID_List]}}. That is, for each pixel region of each frame of image, record the IDs of all 3D points falling into that region.

[0045] S4.3 Semantic Mapping Execution: When the target recognition model in step S3 outputs a target region, the system immediately queries the inverse index table.

[0046] Find the set of all 3D points whose projections fall within the target region, {P_k}.

[0047] A Bayesian voting mechanism is used to address the problem that a 3D point may be captured by multiple images, and the recognition results of different images may be inconsistent.

[0048] For point P, count the recognition results and confidence scores of all image frames covering that point.

[0049] Calculate the scores for each category: The formula for calculating Weight_i, along with the weights of the observation angle and distance, is as follows: in θ It is the observation angle. D It's distance.

[0050] The category with the highest score is used as the final semantic label for that 3D point.

[0051] Step S5: Based on the mapped results, generate a semantic 3D point cloud containing semantic information of the ecological targets, and output the survey results. This includes the following steps: S5.1 Semantic Point Cloud Generation: The generated semantic point clouds can be directly used for statistical analysis. For example, the volume of the point cloud classified as "coral" can be used to estimate the biomass of coral reefs; the distribution range of the point cloud classified as "plastic waste" can be used to assess the degree of pollution.

[0052] It can be used for visualization rendering: semantic labels are mapped to specific colors (such as red for coral and green for seaweed) and overlaid on a 3D terrain model to generate a "marine ecology thematic electronic map".

[0053] It can be used for change detection: by comparing semantic point clouds from different periods, it can automatically detect changes in the ecological environment (such as the expansion of coral bleaching areas and the increase of garbage accumulation) and generate change monitoring reports.

[0054] The proposed method employs a three-layer architecture for edge-cloud collaborative processing: Data collection devices include drones, unmanned boats, and underwater robots. Edge computing nodes: deployed on unmanned vessels; Cloud server; This embodiment also provides a marine ecological and environmental protection image surveying device, the device comprising: The data acquisition module is used to acquire multi-source heterogeneous survey data of the target water area; The 3D reconstruction module is used to establish a 3D control network based on high-precision positioning control points and perform regional network adjustment. It uses the structure-of-motion algorithm to generate image point clouds, performs feature extraction, iterative nearest point registration and fusion processing on multi-source point cloud data, and generates fused point cloud data containing 3D geometric coordinates and image texture information. The intelligent recognition module is used to input multimodal feature vectors and visible light image data into a deep learning recognition model to identify corals, algae, fish and pollutants, and output two-dimensional bounding box coordinates, category labels and recognition confidence scores. The semantic mapping module is used to convert the two-dimensional recognition results into three-dimensional point cloud coordinates according to the mapping relationship, perform semantic annotation on the three-dimensional point cloud, and generate a semantic point cloud.

[0055] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for marine ecological and environmental protection image surveying, characterized in that, Includes the following steps: S1. Multi-source heterogeneous survey data of the target water area are collected by a multi-source data acquisition terminal in a collaborative operation; The multi-source heterogeneous survey data includes at least: aerial imagery data above the water surface and underwater imagery data below the water surface; S2. The multi-source heterogeneous survey data is preprocessed and fused across media to generate a three-dimensional point cloud model of the target water area; the preprocessing includes correcting water body refraction distortion in the underwater image data; S3. Use a pre-trained target recognition model to identify ecological targets in the aerial and underwater image data, and obtain the recognition results of ecological targets in the two-dimensional images; S4. Establish a mapping relationship between two-dimensional images and three-dimensional point cloud models, and map the recognition results in the two-dimensional images to the three-dimensional point cloud models according to the mapping relationship; S5. Based on the mapped results, generate a semantic 3D point cloud containing semantic information of ecological targets, and output the survey results.

2. The marine ecological and environmental protection image survey method according to claim 1, characterized in that, The multi-source data acquisition terminal includes at least one of the following: an aerial drone equipped with imaging equipment, a surface unmanned vessel equipped with underwater acoustic detection equipment or water quality monitoring equipment, or an underwater robot equipped with underwater imaging equipment.

3. The marine ecological and environmental protection image survey method according to claim 1, characterized in that, The underwater image data refraction distortion correction includes: calculating the deviation vector of the light propagation path based on the refraction model of light at the air-water interface, and compensating and correcting the three-dimensional coordinates corresponding to the underwater image according to the deviation vector.

4. The marine ecological and environmental protection image survey method according to claim 1, characterized in that, The multi-source heterogeneous survey data also includes multispectral image data; The ecological target identification includes: extracting spectral features from multispectral image data, color features from visible light image data, and texture features; fusing the spectral features, color features, and texture features to construct a multimodal feature vector; and inputting the multimodal feature vector into the target identification model for classification and localization.

5. The marine ecological and environmental protection image survey method according to claim 1, characterized in that, The establishment of the mapping relationship between the two-dimensional image and the three-dimensional point cloud model includes: Based on the camera imaging model, a forward projection relationship from 3D point cloud coordinates to 2D image pixel coordinates is established, as well as a reverse index table from 2D image pixel coordinates to 3D point cloud coordinates is established. When the target recognition model outputs a two-dimensional bounding box of the target region, the three-dimensional point cloud set corresponding to the pixels in that region is queried through the inverse index table.

6. The marine ecological and environmental protection image survey method according to claim 1, characterized in that, The generated 3D point cloud model of the target water area includes: Motion reconstruction structure calculations are performed on image data from different sources to generate sparse point clouds; The point cloud generated from lidar data or sonar data is registered and fused with the sparse point cloud; the registration and fusion includes coarse registration based on feature descriptors and fine registration based on iterative nearest points.

7. The marine ecological and environmental protection image survey method according to claim 4, characterized in that, The spectral features include at least one of the normalized vegetation index, turbidity index, and normalized water index; The target recognition model is a deep learning model based on convolutional neural networks, and the ecological targets include corals, algae, aquatic organisms, and water pollutants.

8. The marine ecological and environmental protection image survey method according to claim 1, characterized in that, The output survey results include: Statistical semantic analysis of the quantity, volume, and spatial distribution of various ecological targets in 3D point clouds; The semantic tags are visualized and rendered onto a 3D terrain model to generate a comprehensive electronic map containing terrain information and ecological elements.

9. A marine ecological and environmental protection image survey method according to claim 1, characterized in that, The method employs an edge-cloud collaborative processing architecture: Real-time data stitching and preliminary target identification are performed on the data acquisition device or edge computing node; Global area network adjustment, high-precision point cloud fusion, and semantic mapping are performed on cloud servers.

10. A marine ecological and environmental protection image surveying device, performing the steps of the method as described in any one of claims 1 to 9, characterized in that: It also includes a data acquisition module for acquiring multi-source heterogeneous survey data of the target water area; The 3D reconstruction module is used to establish a 3D control network based on high-precision positioning control points and perform regional network adjustment. It uses the structure-of-motion algorithm to generate image point clouds, performs feature extraction, iterative nearest point registration and fusion processing on multi-source point cloud data, and generates fused point cloud data containing 3D geometric coordinates and image texture information. The intelligent recognition module is used to input multimodal feature vectors and visible light image data into a deep learning recognition model to identify corals, algae, fish and pollutants, and output two-dimensional bounding box coordinates, category labels and recognition confidence scores. The semantic mapping module is used to convert the two-dimensional recognition results into three-dimensional point cloud coordinates according to the mapping relationship, perform semantic annotation on the three-dimensional point cloud, and generate a semantic point cloud.

Images