Method, device and method for generating a model for identifying defects of underwater structures

Abstract

The application provides a method and device for generating an underwater structure disease identification model and a disease identification method. The generating method comprises the following steps: aligning pixels of an optical image with sonar point cloud data, then performing time synchronization, and performing disease labeling on the optical image and the sonar point cloud data; based on environmental parameters, performing weighted fusion on the optical image and the sonar point cloud data that have completed disease labeling to obtain a multi-source fusion data set; based on a GAN network, extracting optical texture feature vectors, sonar point cloud spatial feature vectors and environmental parameter feature vectors in the multi-source fusion data set, and fusing the feature vectors to obtain a simulation sample data set; based on the simulation sample data set, training an underwater structure surface crack segmentation network to obtain a disease identification model for identifying diseases of underwater structures; the segmentation network comprises an encoder, a decoder and a parallel attention module. The application can solve the problems of low identification accuracy and lack of engineering applicability in underwater structure disease identification.

Description

Technical Field

[0001] This invention relates to the field of underwater engineering structure health monitoring technology, specifically to a method, device, and method for generating an underwater structure defect identification model. Background Technology

[0002] Underwater structures are widely used in bridges, ports, waterways, reservoirs, and marine engineering platforms, forming a crucial component of transportation and water conservancy infrastructure. With the continuous expansion of transportation and water conservancy engineering construction in my country, the demand for underwater inspection is rapidly increasing. Statistics show that the number of underwater infrastructure structures in China has exceeded 300,000, with over 10,000 underwater inspection operations conducted annually, resulting in a market size of 5-8 billion yuan for related inspection services. In practical engineering projects, manual underwater inspection is not only costly per operation (approximately 8,000-50,000 yuan), but also carries significant risks, high safety requirements, and stringent demands on personnel expertise and environmental conditions. Therefore, the industry urgently needs to replace the high-cost, high-risk manual underwater inspection model with intelligent and automated technologies. Digital management and intelligent identification technologies for underwater structural defects have become essential capabilities for operation and maintenance in the fields of water conservancy, transportation, and marine engineering.

[0003] However, the underwater environment is complex and variable. Structures are constantly subjected to water erosion, biological adhesion, corrosion, and turbid media, making them highly susceptible to defects such as cracks, spalling, and exposed rebar. Failure to identify these defects promptly and accurately will lead to reduced structural durability, decreased load-bearing capacity, and even serious safety risks. Therefore, various engineering projects urgently require high-precision, automated, and quantifiable underwater defect detection technologies.

[0004] At present, the mainstream methods in the industry still rely on manual divers for detection or single-source sensor collection, which has obvious limitations: (1) Insufficient data: Most existing underwater disease datasets are based on a single data source, such as using only optical images or relying only on sonar point clouds, lacking multi-source collaborative information; at the same time, key environmental parameters such as flow velocity, turbidity, and water depth are not collected or integrated, resulting in single data dimensions and scarce samples, which cannot support the training and generalization of intelligent models in complex scenarios; (2) Limited model recognition performance: Existing recognition models mostly use general visual network structures and have not been optimized for underwater environmental light scattering, turbidity changes, dynamic particle interference and other features; the model has low segmentation accuracy for small target features such as cracks and corrosion edges, and it is difficult to maintain stable performance under different water quality and equipment conditions; (3) Insufficient engineering applicability: Existing models are not connected with the multi-source data management system of real engineering scenarios, and cannot form a reusable, scalable and engineering-applicable disease recognition capability, which is difficult to meet the requirements of long-term operation and maintenance scenarios for reliability, stability and full-scenario applicability.

[0005] Against this backdrop, the industry urgently needs an intelligent underwater structure defect identification technology that can integrate multi-source data, improve identification accuracy, and has engineering applicability. Summary of the Invention

[0006] In view of this, it is necessary to provide a method, device and method for generating underwater structure defect identification models, in order to solve the technical problems of low accuracy and lack of engineering applicability in underwater structure defect identification.

[0007] To address the aforementioned problems, in a first aspect, the present invention provides a method for generating an underwater structure defect identification model, comprising: Acquire optical images, sonar point cloud data, and environmental parameters of the target water area; After aligning the pixels of the optical image with the sonar point cloud data, the optical image, the sonar point cloud data, and the environmental parameters are synchronized in time. After the time synchronization is completed, the optical image and the sonar point cloud data are annotated with defects. Based on the environmental parameters, the optical image and the sonar point cloud data with disease annotation are weighted and fused to obtain a multi-source fusion dataset; Based on the GAN network, optical texture feature vectors, sonar point cloud spatial feature vectors, and environmental parameter feature vectors are extracted from the multi-source fusion dataset. The optical texture feature vectors, sonar point cloud spatial feature vectors, and environmental parameter feature vectors are then fused to obtain the simulation sample dataset. A surface crack segmentation network for underwater structures is trained based on the simulated sample dataset to obtain a defect identification model for identifying defects in underwater structures. The underwater structure surface crack segmentation network includes an encoder, a decoder, and a parallel attention module.

[0008] In one possible implementation, aligning the pixels of the optical image with the sonar point cloud data includes: Based on the calibration results between the camera used to capture optical images and the sonar used to collect sonar point cloud data, the three-dimensional spatial coordinates of the sonar point cloud data are mapped to the pixel coordinates of the optical image, thereby aligning the pixels of the optical image with the sonar point cloud data. The calibration result is determined based on a projection matrix constructed from the camera's intrinsic and extrinsic parameters and the sonar's extrinsic parameters.

[0009] In one possible implementation, the method for generating the underwater structure defect identification model also includes: After mapping the three-dimensional spatial coordinates of the sonar point cloud data to the pixel coordinates of the optical image, the registration result of the sonar point cloud data and the optical image is optimized by using an iterative nearest-point algorithm or a feature-based registration method.

[0010] In one possible implementation, the optical image and the sonar point cloud data are annotated with defects, including: The optical image is labeled with defects using pixel-based segmentation masks or polygonal multi-segment coordinate representation, and the sonar point cloud data is labeled with defects using point cloud indexes or three-dimensional bounding boxes.

[0011] In one possible implementation, the environmental parameters include: the flow velocity and turbidity of the target water body; Based on the environmental parameters, the optical image and sonar point cloud data with completed lesion annotation are weighted and fused to obtain a multi-source fusion dataset, including: If the turbidity of the target water area is higher than the first preset threshold or the flow velocity of the target water area is higher than the second preset threshold, the optical image and the sonar point cloud data that have completed the disease annotation are weighted and fused based on the first weight corresponding to the optical image to obtain a multi-source fusion dataset. When the turbidity of the target water area is lower than the first preset threshold and the flow velocity of the target water area is lower than the second preset threshold, the optical image and the sonar point cloud data that have completed the disease annotation are weighted and fused based on the second weight corresponding to the optical image to obtain a multi-source fusion dataset. Wherein, the first weight is less than the second weight.

[0012] In one possible implementation, the GAN network includes: a generator and a discriminator; The generator includes a fully connected mapping layer and six deconvolutional layers, and the output of the generator is equipped with a cross-modal feature fusion module. The discriminator comprises six convolutional layers and one fully connected layer, and introduces a dual-branch authenticity discrimination module in the penultimate layer.

[0013] In one possible implementation, a multi-scale attention weighting mechanism for sonar and optical features is set in each deconvolution layer of the generator.

[0014] In one possible implementation, the encoder is used to extract multi-scale feature information of cracks from the simulation sample dataset through a residual network to obtain multiple feature maps; The decoder is used to upsample and concatenate the multiple feature maps to obtain multiple new feature maps. The parallel attention module is used to upsample the third new feature map among the multiple new feature maps, concatenate it with the second new feature map to obtain a concatenated feature map, and then concatenate the concatenated feature map with the first new feature map to obtain a feature map that is then passed through the attention module and the convolutional layer in sequence before being output.

[0015] Secondly, the present invention also provides a method for identifying underwater structural defects, comprising: Acquire optical images, sonar point cloud data, and environmental parameters of the target water area to be identified; After aligning the pixels of the optical image to be identified with the sonar point cloud data, the optical image to be identified, the sonar point cloud data, and the environmental parameters are synchronized in time. Based on the environmental parameters that have completed time synchronization, the time-synchronized optical images and sonar point cloud data are weighted and fused to obtain the multi-source fusion dataset to be identified. Based on the GAN network, optical texture feature vectors, sonar point cloud spatial feature vectors, and environmental parameter feature vectors are extracted from the multi-source fusion dataset to be identified. The optical texture feature vectors, sonar point cloud spatial feature vectors, and environmental parameter feature vectors are then fused to obtain the simulation sample dataset to be identified. Input the simulation sample dataset to be identified into the disease identification model obtained by any of the methods described above to obtain the underwater structure disease identification results.

[0016] Thirdly, the present invention also provides an apparatus for generating an underwater structural defect identification model, comprising: The acquisition module is used to acquire optical images, sonar point cloud data, and environmental parameters of the target water area; The annotation module is used to align the pixels of the optical image with the sonar point cloud data, then synchronize the optical image, the sonar point cloud data and the environmental parameters in time, and annotate the optical image and the sonar point cloud data with defects after the time synchronization is completed. The first fusion module is used to perform weighted fusion of the optical image and the sonar point cloud data with disease annotation completed based on the environmental parameters to obtain a multi-source fusion dataset. The second fusion module is used to extract optical texture feature vectors, sonar point cloud spatial feature vectors, and environmental parameter feature vectors from the multi-source fusion dataset based on the GAN network, and to fuse the optical texture feature vectors, sonar point cloud spatial feature vectors, and environmental parameter feature vectors to obtain a simulation sample dataset. The training module is used to train an underwater structure surface crack segmentation network based on the simulation sample dataset to obtain a defect identification model for identifying underwater structure defects. The underwater structure surface crack segmentation network includes an encoder, a decoder, and a parallel attention module.

[0017] The beneficial effects of adopting the above implementation method are as follows: The underwater structure defect identification model generation method, device and defect identification method provided by the present invention provide high-quality, multi-modal input for intelligent identification model by acquiring and fusing multi-source data such as optical images, sonar point cloud data and environmental parameters of the target water area, which significantly improves the segmentation accuracy of the target defect and solves the problems of low accuracy and poor stability of existing models in complex underwater environments.

[0018] GAN networks can generate simulation data based on optical, sonar, and environmental parameters, solving the problem of scarce real underwater defect samples. This significantly improves the model's generalization ability under different water conditions, equipment parameters, and scenarios, enabling the defect identification model to adapt to various engineering environments such as harbors, bridge piers, reservoirs, and navigation structures. Therefore, this invention can solve the technical problems of low accuracy and lack of engineering applicability in underwater structural defect identification. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 A flowchart of an embodiment of the method for generating an underwater structure defect identification model provided by the present invention; Figure 2 A schematic diagram of the structure of the intelligent recognition model provided by the present invention; Figure 3 A flowchart of another embodiment of the method for generating an underwater structure defect identification model provided by the present invention; Figure 4 A schematic diagram illustrating multi-source data acquisition provided by the present invention; Figure 5 A flowchart for multi-source data fusion processing provided by the present invention; Figure 6 A schematic diagram of the simulation augmentation dataset provided by this invention; Figure 7 A flowchart of an embodiment of the device for generating an underwater structure defect identification model provided by the present invention; Figure 8 A flowchart of an embodiment of the underwater structure defect identification method provided by the present invention; Figure 9A schematic diagram of an embodiment of the electronic device provided by the present invention. Detailed Implementation

[0021] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0022] In the description of the embodiments of this application, unless otherwise stated, "a plurality of" means two or more.

[0023] In this embodiment of the invention, the terms "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, apparatus, product or device that includes a series of steps or modules is not necessarily limited to those steps or modules that are explicitly listed, but may include other steps or modules that are not explicitly listed or that are inherent to such process, method, product or device.

[0024] The naming or numbering of steps in the embodiments of the present invention does not mean that the steps in the method flow must be executed in the time / logical order indicated by the naming or numbering. The execution order of the named or numbered process steps can be changed according to the technical purpose to be achieved, as long as the same or similar technical effect can be achieved.

[0025] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a mutually exclusive, independent, or alternative embodiment. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0026] Existing underwater structure defect detection technologies have revealed significant shortcomings in engineering applications, mainly in the following two aspects: (1) There is a lack of high-quality, multi-source collaborative datasets that can be used in engineering at the data level.

[0027] Currently, the industry commonly uses single optical images or sonar point clouds for disease detection. This data lacks integration of key environmental parameters such as flow velocity, turbidity, and water depth, failing to reflect real-world operating conditions. Furthermore, the limited data scale and insufficient coverage of disease types make it difficult to support large-scale model training and generalization requirements in complex scenarios. This problem directly prevents the industry from establishing a stable and reliable underwater disease identification system, limiting the engineering application value of intelligent detection technology.

[0028] (2) The identification model is difficult to adapt to complex underwater environments, and the accuracy and stability of disease extraction are insufficient.

[0029] Most existing intelligent recognition models are based on general vision algorithms and have not been optimized for light scattering, dynamic interference and small target defects in underwater scenes. The recognition accuracy of small defects such as cracks and peeling edges is low. The performance of the models fluctuates greatly under different water quality, equipment and environmental conditions, making it difficult to meet the continuous and stable requirements of engineering sites and unable to support large-scale underwater structure operation and maintenance management.

[0030] Based on this, the present invention aims to construct a multi-source collaborative data system for underwater structural defects that can be used in engineering scenarios, and proposes a high-precision intelligent identification model that can adapt to complex underwater environments to solve the problems of insufficient data support and low identification accuracy, thereby improving the reliability, accuracy and engineering usability of underwater structural defect detection.

[0031] This invention provides a method, apparatus, and method for generating underwater structure defect identification models, which are described below.

[0032] This invention provides a method for generating an underwater structure defect identification model. This method can be implemented by executing an application on a server, which can be a cloud server or an edge server, such as... Figure 1 As shown, the method includes: S101. Acquire optical images, sonar point cloud data, and environmental parameters of the target water area.

[0033] Understandably, optical images can be acquired using high-resolution industrial cameras mounted on underwater inspection platforms. Before acquisition, the exposure time, gain, and white balance parameters are automatically adjusted by the control system based on the lighting conditions, flow rate, and turbidity of the inspected water area to ensure that the image brightness and contrast are within a recognizable range.

[0034] Sonar point cloud data can be acquired using multibeam or imaging sonar equipment, with the sonar emission direction kept perpendicular to the structure surface by a platform mounting bracket. During acquisition, the sonar equipment scans at preset angle steps, and the control unit records the echo intensity and depth information obtained from each scan, generating corresponding two-dimensional sonar images or three-dimensional point cloud data.

[0035] Environmental parameters can be collected using flow meters, turbidity sensors, and water depth and pressure sensors.

[0036] S102. After aligning the pixels of the optical image with the sonar point cloud data, synchronize the optical image, the sonar point cloud data and the environmental parameters in time. After completing the time synchronization, mark the defects in the optical image and the sonar point cloud data.

[0037] Understandably, the system adopts a unified data annotation standard and file format in the disease annotation process. All annotations record the environmental parameters (flow velocity, turbidity, water depth) and sensor ID at the time of collection, forming a complete {image-point cloud-environment} correlation record, which facilitates traceability and quality control.

[0038] S103. Based on the environmental parameters, the optical image and the sonar point cloud data with disease annotation are weighted and fused to obtain a multi-source fusion dataset.

[0039] Understandably, based on the association labeling and processing strategies of environmental parameters, the system defines priority rules for the use of multi-source data to achieve adaptive environmental identification. It dynamically adjusts the input sources under different flow rates, turbidity, and water depths to improve identification stability and accuracy, while providing clear operational guidelines for engineering sites.

[0040] S104. Based on the GAN network, extract the optical texture feature vector, sonar point cloud spatial feature vector, and environmental parameter feature vector from the multi-source fusion dataset, and fuse the optical texture feature vector, the sonar point cloud spatial feature vector, and the environmental parameter feature vector to obtain the simulation sample dataset.

[0041] Understandably, in terms of data expansion and generalization capabilities, the improved multi-source conditional generative adversarial network (GAN) can generate simulation data based on optical, sonar, and environmental parameters (flow velocity, turbidity, water depth), solving the problem of scarce real underwater disease samples and significantly improving the model's generalization ability under different water conditions, different equipment parameters, and different scenarios, enabling the model to adapt to various engineering environments such as seaports, bridge piers, reservoirs, and navigation structures.

[0042] S105. Train the underwater structure surface crack segmentation network based on the simulation sample dataset to obtain a defect identification model for identifying underwater structure defects. Among them, the underwater structure surface crack segmentation network, such as Figure 2 As shown, it includes: an encoder, a decoder, and a parallel attention module.

[0043] It is understandable that, such as Figure 3 As shown, by constructing a multi-source collaborative dataset of underwater structural defects and designing a segmentation network for surface cracks in underwater structures, a data acquisition (e.g., Figure 4 As shown), data fusion (such as) Figure 5 As shown), dataset construction (such as...) Figure 6 As shown, this is a complete system for model training and result output.

[0044] In some embodiments, aligning the pixels of the optical image with sonar point cloud data includes: Based on the calibration results between the camera used to capture optical images and the sonar used to collect sonar point cloud data, the three-dimensional spatial coordinates of the sonar point cloud data are mapped to the pixel coordinates of the optical image, thereby aligning the pixels of the optical image with the sonar point cloud data. The calibration result is determined based on a projection matrix constructed from the camera's intrinsic and extrinsic parameters and the sonar's extrinsic parameters.

[0045] Understandably, based on the calibration results of the camera and sonar, the three-dimensional spatial coordinates of the sonar point cloud are mapped to the optical image pixel coordinates through an engineered coordinate transformation and registration process, achieving precise alignment between pixels and the point cloud. In specific implementation, a projection matrix is ​​first constructed based on the camera's intrinsic parameters (focal length, distortion coefficients) and extrinsic parameters (pose transformation) as well as the sonar's extrinsic parameters, and then a rigid transformation is used to transform the point cloud coordinates to the camera coordinate system.

[0046] In some embodiments, the method for generating an underwater structure defect identification model further includes: After mapping the three-dimensional spatial coordinates of the sonar point cloud data to the pixel coordinates of the optical image, the registration result of the sonar point cloud data and the optical image is optimized by using an iterative nearest-point algorithm or a feature-based registration method.

[0047] Understandably, the registration results can be further optimized using iterative nearest neighbor (ICP) or feature-based registration methods when necessary. After registration, the reprojection error is calculated and compared with a preset threshold (e.g., 3–5 pixels). If the error exceeds the limit, automatic recalibration is triggered or the process reverts to manual verification to ensure that the registration accuracy meets engineering requirements. This registration and mapping process is executed by automated software tools, outputting a standardized mapping table (including frame number, point cloud index, pixel coordinates, reprojection error, etc.) to reduce manual intervention and improve the consistency and repeatability of large-scale engineering data processing.

[0048] In some embodiments, the process of annotating defects in the optical image and the sonar point cloud data includes: The optical image is labeled with defects using pixel-based segmentation masks or polygonal multi-segment coordinate representation, and the sonar point cloud data is labeled with defects using point cloud indexes or three-dimensional bounding boxes.

[0049] Understandably, optical image annotation uses pixel-based segmentation masks or polygonal multi-segment coordinates, outputting as standardized JSON containing the fields {image_id, polygon, bbox, category_id, confidence, timestamp}. Sonar point cloud annotation uses point cloud indexes or 3D bounding boxes, outputting as JSON or PLY / PCD annotation files with point cloud indexes, containing the fields {scan_id, point_indices, bbox_3d, category_id, depth_stats, timestamp}. To ensure consistency between optical and sonar annotations, the system introduces an automatic verification mechanism: performing reprojection consistency checks on each annotation item (projecting the point cloud annotation onto the image plane, calculating the reprojection intersection ratio, and labeling as inconsistent if the intersection is below a set threshold); cross-validating categories and locations; and calculating statistical differences in depth / area / boundaries for multi-source annotations of the same lesion, automatically marking as requiring manual review when differences exceed limits.

[0050] In some embodiments, the environmental parameters include: the flow rate and turbidity of the target water body; Based on the environmental parameters, the optical image and sonar point cloud data with completed lesion annotation are weighted and fused to obtain a multi-source fusion dataset, including: If the turbidity of the target water area is higher than the first preset threshold or the flow velocity of the target water area is higher than the second preset threshold, the optical image and the sonar point cloud data that have completed the disease annotation are weighted and fused based on the first weight corresponding to the optical image to obtain a multi-source fusion dataset. When the turbidity of the target water area is lower than the first preset threshold and the flow velocity of the target water area is lower than the second preset threshold, the optical image and the sonar point cloud data that have completed the disease annotation are weighted and fused based on the second weight corresponding to the optical image to obtain a multi-source fusion dataset. Wherein, the first weight is less than the second weight.

[0051] Understandably, when real-time turbidity exceeds a set threshold (example threshold: turbidity ≥ 40 NTU) or flow velocity exceeds a set threshold (example threshold: flow velocity ≥ 1.2 m / s), the identification process will reduce the weight of optical data and prioritize the use of sonar point clouds as the primary identification source. When both flow velocity and turbidity are in the low to medium range and optical quality is excellent, optical images will be prioritized, with sonar used as an auxiliary source. In case of anomalies or sensor failures, the system will select available data according to a predefined fallback strategy and trigger manual review. This priority logic ensures that the model dynamically adjusts the input source under different flow velocities, turbidities, and water depths, improving identification stability and accuracy, while providing clear operational guidance for engineering sites.

[0052] In some embodiments, the GAN network includes: a generator and a discriminator; The generator includes a fully connected mapping layer and six deconvolutional layers, and the output of the generator is equipped with a cross-modal feature fusion module. The discriminator comprises six convolutional layers and one fully connected layer, and introduces a dual-branch authenticity discrimination module in the penultimate layer.

[0053] The generator incorporates a multi-scale attention weighting mechanism that combines sonar and optical features in each deconvolution layer.

[0054] Understandably, the generator consists of one fully connected mapping layer and six deconvolutional layers, with a new "Cross-Modal Fusion Block (CMFB)" added to the traditional architecture. This module introduces a multi-scale attention weighting mechanism for sonar and optical features into the output of each deconvolutional layer. The discriminator consists of six convolutional layers and one fully connected layer, with a "Dual-Branch Authenticity Discriminator (DBAD)" introduced in the penultimate layer.

[0055] In some embodiments, the encoder is used to extract multi-scale feature information of cracks from the simulation sample dataset through a residual network to obtain multiple feature maps; The decoder is used to upsample and concatenate the multiple feature maps to obtain multiple new feature maps. The parallel attention module is used to upsample the third new feature map among the multiple new feature maps, concatenate it with the second new feature map to obtain a concatenated feature map, and then concatenate the concatenated feature map with the first new feature map to obtain a feature map that is then passed through the attention module and the convolutional layer in sequence before being output.

[0056] Understandably, the encoder uses ResNet50 instead of the traditional convolutional structure to improve feature extraction capabilities, and embeds a self-designed attention mechanism in the back end of the decoder to suppress various interferences from the background and noise, thereby improving the accuracy of underwater crack feature extraction.

[0057] During model training, the optical images from the multi-source dataset, the sonar point cloud information aligned with the optical image space by depth mapping, and the environmental parameter feature vectors are first fused and input into the encoder. The multi-scale feature information of the crack is extracted by the residual network ResNet50 to obtain feature maps F1, F2, F3, F4, and F5.

[0058] Among them, optical images provide texture and color information, while sonar point clouds provide geometric structure information. The two are jointly encoded in the channel dimension to achieve complementary perception of geometry and texture. The environmental feature vector is applied to the multi-scale feature map output by the encoder through the attention module, which guides the dynamic adjustment of channel and spatial feature weights, enabling the network to perform adaptive feature extraction under different underwater environmental conditions.

[0059] The obtained feature maps are upsampled and concatenated in the decoder to obtain new feature maps L1, L2, L3, L4, and L5. Simultaneously, in the decoder, feature maps L1, L2, and L3 are fed as inputs into a parallel attention module. After passing through the attention module, L3 is upsampled (Up-conv, 2×2) and concatenated with L2, then input into the attention module corresponding to L2. Its output is then concatenated with L1 and passed through the corresponding attention module. Finally, a 1×1 convolutional layer generates a predicted output of the same size as the original image. This achieves environment-adaptive crack segmentation driven by multi-source information.

[0060] In some embodiments, the solution provided by the present invention includes the following steps: (1) Multi-source data acquisition a) Optical image acquisition The optical image acquisition module employs a high-resolution industrial camera mounted on an underwater inspection platform. Before acquisition, the control system automatically adjusts the exposure time, gain, and white balance parameters based on the lighting conditions, flow velocity, and turbidity of the inspected water area to ensure that the image brightness and contrast are within a recognizable range. The camera performs a scanning motion along the underwater structure surface at a fixed speed and continuously acquires image data at a preset frame rate, ensuring that the acquisition area covers locations on the structure surface where cracks, peeling, or other defects may occur. The acquired images are filtered and distortion corrected before being stored in a data buffer for subsequent defect identification input.

[0061] b) Sonar data acquisition The sonar data acquisition module employs a multi-beam or imaging sonar device, with the sonar emission direction maintained perpendicular to the structural surface via a platform support. During acquisition, the sonar device scans at preset angular steps, and the control unit records the echo intensity and depth information obtained from each scan, generating corresponding two-dimensional sonar images or three-dimensional point cloud data. In environments with low optical visibility, the control system automatically increases the sonar pulse repetition frequency and gain to maintain the sonar sampling density within a identifiable range, ensuring that the location of the defect can be accurately represented in the point cloud.

[0062] c) Environmental parameter collection The environmental parameter acquisition module includes a flow meter, a turbidity sensor, and a water depth / pressure sensor. Each sensor connects to the main control unit via a serial port or industrial Ethernet, recording water flow velocity, turbidity, and water depth at a fixed sampling frequency. Each set of environmental parameters is accompanied by a timestamp in a standardized format and associated with the corresponding optical image frame number and sonar scan number for that time period, forming a synchronized data entry of {image – point cloud – environmental parameter}. This synchronization structure facilitates subsequent data fusion processing and the invocation of environmental adaptive recognition models.

[0063] (2) Multi-source data fusion processing Multi-source data fusion processing, based on camera and sonar calibration results, maps the 3D spatial coordinates of the sonar point cloud to the pixel coordinates of the optical image through an engineered coordinate transformation and registration process, achieving precise alignment between pixels and the point cloud. Specifically, a projection matrix is ​​first constructed based on the camera's intrinsic parameters (focal length, distortion coefficients) and extrinsic parameters (pose transformation), as well as the sonar's extrinsic parameters. A rigid transformation is then used to transform the point cloud coordinates to the camera coordinate system. If necessary, the registration results are further optimized using iterative nearest point (ICP) or feature-based registration methods. After registration, the reprojection error is calculated and compared with a preset threshold (e.g., 3–5 pixels). If the error exceeds the limit, automatic recalibration is triggered or a manual verification process is initiated to ensure that the registration accuracy meets engineering requirements. This registration and mapping process is executed by automated software tools, outputting a standardized mapping table (including frame number, point cloud index, pixel coordinates, reprojection error, etc.) to reduce manual intervention and improve the consistency and repeatability of large-scale engineering data processing.

[0064] In terms of time synchronization, the system generates a unified timestamp (using millisecond-level UTC time or ISO8601 format) for each frame of optical image, each sonar scan, and each set of environmental parameter samples, and matches multi-source data according to time windows through an automated alignment mechanism. The synchronization mechanism supports anomaly detection—when time alignment fails or data is lost, the system records anomaly logs and processes them according to a priority strategy to avoid incorrect pairing. Through the above spatiotemporal alignment strategy, the system can ensure reliable spatiotemporal matching of optical images, sonar point clouds, and environmental parameters under different flow rates, water depths, and turbidity conditions, providing a solid data foundation for subsequent data fusion and intelligent identification.

[0065] In the disease annotation stage, the system adopts a unified data annotation standard and file format. Optical image annotation uses pixel-based segmentation masks or polygon multi-segment coordinates, and the output is standardized JSON, containing the fields {image_id, polygon, bbox, category_id, confidence, timestamp}. Sonar point cloud annotation uses point cloud indexes or 3D bounding boxes, and the output is JSON or PLY / PCD annotation files with point cloud indexes, containing the fields {scan_id, point_indices, bbox_3d, category_id, depth_stats, timestamp}. To ensure the consistency between optical and sonar annotations, the system introduces an automatic verification mechanism: performing reprojection consistency checks on each annotation item (reprojecting the point cloud annotation onto the image plane, calculating the reprojection intersection ratio, and labeling it as inconsistent if the intersection is lower than a set threshold); cross-validating the category and location; calculating the statistical differences in depth / area / boundary for multi-source annotations of the same disease, and automatically marking it as requiring manual review when the difference exceeds the limit. All annotations record the environmental parameters (flow rate, turbidity, water depth) and sensor ID at the time of acquisition, forming a complete {image-point cloud-environment} association record, which facilitates traceability and quality control.

[0066] Based on environmental parameter-based association labeling and processing strategies, the system defines priority rules for multi-source data to achieve environmentally adaptive identification: when real-time turbidity exceeds a set threshold (example threshold: turbidity ≥ 40 NTU) or flow velocity exceeds a set threshold (example threshold: flow velocity ≥ 1.2 m / s), the identification process will reduce the weight of optical data and prioritize the use of sonar point clouds as the primary identification source; when flow velocity and turbidity are both in the low to medium range and optical quality is excellent, optical images are prioritized and sonar is used as an auxiliary source; in case of anomalies or sensor failures, the system selects available data according to a predefined fallback strategy and triggers manual review. This priority logic ensures that the model dynamically adjusts the input sources under different flow velocities, turbidities, and water depths, improving identification stability and accuracy, while providing clear operational guidance for engineering sites.

[0067] (3) Construction of simulation extended dataset To expand the data scale and enhance the diversity of samples under different environmental conditions, this invention proposes a multi-source conditional generative adversarial network based on environment adaptation. This model improves upon the traditional GAN ​​network framework in the following three aspects: Multi-source conditional input mechanism: Traditional GAN ​​networks only accept random noise as input. This approach fails to reflect the differences in actual data collection environments, resulting in a homogeneous distribution of generated samples. This invention introduces three types of conditional vectors at the input end: Optical texture feature vector ; Sonar point cloud spatial feature vector ; Environmental parameter feature vector (Including water flow velocity, turbidity, water depth, etc.); These vectors are concatenated by the feature fusion module to form a composite conditional input. This guides the generator to produce more representative simulation samples in different environmental scenarios.

[0068] The generated samples are environmentally adaptive, and the simulation data can reflect the disease characteristics under various underwater lighting, flow, and turbidity conditions, thereby significantly improving the diversity and generalization ability of the dataset.

[0069] Improved generator architecture: cross-modal feature fusion module; The generator consists of one fully connected mapping layer and six deconvolutional layers, and adds a "Cross-Modal Fusion Block (CMFB)" to the traditional architecture.

[0070] This module introduces a multi-scale attention weighting mechanism for sonar and optical features into the output of each deconvolution layer. Its core calculation is as follows:

[0071] Among them, weight parameters Dynamically adjust based on environmental parameter characteristics.

[0072] In scenarios with high turbidity and poor optical quality, the model automatically increases the weight of sonar features; Enhanced optical feature contribution in environments with clear water; This enables adaptive modality selection and feature fusion of simulation samples under different environments.

[0073] Discriminator structure improvement: dual-branch authenticity discrimination module The discriminator consists of six convolutional layers and one fully connected layer, with a "Dual-Branch Authenticity Discriminator (DBAD)" introduced in the penultimate layer.

[0074] This module outputs the following: Sample authenticity probability ; Environmental consistency probability This is used to determine whether the generated sample meets the input environment parameters.

[0075] The final loss function is defined as:

[0076] in For traditional combat losses, This is an environmental consistency constraint.

[0077] This design ensures that the generated samples are optimal in terms of both visual realism and environmental consistency, avoiding distortion in the generated disease images that do not match the environmental parameters.

[0078] (4) Intelligent recognition model To achieve intelligent identification of underwater structural defects, this invention constructs a multi-source collaborative dataset for underwater structural defects and designs an underwater structural surface crack segmentation network, forming a complete system from data acquisition, data fusion, dataset construction, model training, and result output. The multi-source dataset provides multimodal, high-quality input samples for the crack segmentation network, including: Optical image data: the main input to the segmentation network, used for segmentation training of the model.

[0079] Sonar point cloud data: under conditions of high turbidity or insufficient illumination, it is converted into depth information and stitched with optical images in the channel dimension and input into the network to achieve complementarity of geometric and texture information.

[0080] Environmental parameter data: used as environmental feature vectors during the training phase. The input attention module guides the model to dynamically adjust the channel and spatial feature weights in different environments, thereby achieving environment-adaptive segmentation.

[0081] The underwater structure surface crack segmentation network mainly consists of three parts: encoder, decoder, and parallel attention module.

[0082] The encoder uses ResNet50 instead of the traditional convolutional structure to improve feature extraction capabilities, and embeds a self-designed attention mechanism in the back end of the decoder to suppress various interferences from the background and noise, thereby improving the accuracy of underwater crack feature extraction.

[0083] During model training, the optical images from the multi-source dataset, the sonar point cloud information aligned with the optical image space by depth mapping, and the environmental parameter feature vectors are first fused and input into the encoder. The multi-scale feature information of the crack is extracted by the residual network ResNet50 to obtain feature maps F1, F2, F3, F4, and F5.

[0084] Among them, optical images provide texture and color information, while sonar point clouds provide geometric structure information. The two are jointly encoded in the channel dimension to achieve complementary perception of geometry and texture. The environmental feature vector is applied to the multi-scale feature map output by the encoder through the attention module, which guides the dynamic adjustment of channel and spatial feature weights, enabling the network to perform adaptive feature extraction under different underwater environmental conditions.

[0085] The obtained feature maps are upsampled and concatenated in the decoder to obtain new feature maps L1, L2, L3, L4, and L5. Simultaneously, in the decoder, feature maps L1, L2, and L3 are fed as inputs into a parallel attention module. After passing through the attention module, L3 is upsampled (Up-conv, 2×2) and concatenated with L2, then input into the attention module corresponding to L2. Its output is then concatenated with L1 and passed through the corresponding attention module. Finally, a 1×1 convolutional layer generates a predicted output of the same size as the original image. This achieves environment-adaptive crack segmentation driven by multi-source information.

[0086] The present invention has the following technical effects: Compared with existing technologies, this invention has clear technical advantages and beneficial effects, and possesses good engineering feasibility and industrial application potential.

[0087] In terms of data support and accuracy improvement, the invention provides high-quality, multi-modal input to the intelligent identification model through multi-source data acquisition and fusion of optical, sonar, and environmental parameters. This significantly improves the segmentation accuracy of small target defects and solves the problems of low accuracy and poor stability of existing models in complex underwater environments. Experiments show that the crack segmentation accuracy of this invention is about 20–35% higher than that of traditional methods. It can still maintain stable identification performance under visibility-limited conditions with turbidity of 20–60 NTU, and can maintain identification accuracy fluctuations of no more than 5–10% at water flow velocities of 1–1.5 m / s.

[0088] In terms of data expansion and generalization capabilities, the improved multi-source conditional generative adversarial network (GAN) can generate simulation data based on optical, sonar, and environmental parameters (flow velocity, turbidity, water depth), solving the problem of scarce real underwater disease samples and significantly improving the model's generalization ability under different water conditions, different equipment parameters, and different scenarios, enabling the model to adapt to various engineering environments such as seaports, bridge piers, reservoirs, and navigation structures.

[0089] In terms of efficiency improvement and engineering applicability, this invention constructs a complete engineering system from data acquisition, data fusion, dataset construction to model training and recognition output, realizing the automation and standardization of the disease identification process, significantly reducing the need for manual underwater inspection, reducing the number of operations and risks, and improving inspection efficiency. Compared with traditional manual methods, it can reduce overall maintenance and inspection costs by 30-50%, greatly improve the efficiency of the inspection cycle, and meet the reliability and rapid response requirements of long-term maintenance systems.

[0090] In terms of cost reduction and commercial value, this invention reduces reliance on divers, thereby reducing high-cost and high-risk human resource investment; through multi-source data-driven and intelligent identification technology, it improves the accuracy and consistency of disease detection, while shortening the detection time and reducing equipment deployment and operation costs, thereby enhancing the economic benefits and market competitiveness of underwater detection services.

[0091] In terms of market prospects, considering the industry demand of over 300,000 underwater infrastructure facilities in China and an annual inspection market size of 5-8 billion yuan, the intelligent identification and automated inspection technology provided by this invention can effectively meet the urgent needs of large-scale water conservancy, transportation, and marine engineering projects for high-precision, digital, and automated underwater inspection. The technology has a high degree of engineering sophistication, significant commercial potential, and good value for industrial promotion.

[0092] like Figure 7 As shown, the present invention also provides a method for identifying underwater structural defects, comprising: S701. Acquire optical images, sonar point cloud data, and environmental parameters of the target water area to be identified; S702. After aligning the pixels of the optical image to be identified with the sonar point cloud data, synchronize the optical image to be identified, the sonar point cloud data, and the environmental parameters in time. S703. Based on the environmental parameters of the completed time synchronization, the optical image and sonar point cloud data of the completed time synchronization are weighted and fused to obtain the multi-source fusion dataset to be identified. S704. Based on the GAN network, extract the optical texture feature vector, sonar point cloud spatial feature vector, and environmental parameter feature vector from the multi-source fusion dataset to be identified, and fuse the optical texture feature vector, the sonar point cloud spatial feature vector, and the environmental parameter feature vector to obtain the simulation sample dataset to be identified. S705. Input the simulation sample dataset to be identified into the defect identification model obtained by any of the above methods to obtain the underwater structure defect identification results.

[0093] like Figure 8 As shown, the present invention also provides a device 800 for generating an underwater structure defect identification model, comprising: The acquisition module 801 is used to acquire optical images, sonar point cloud data and environmental parameters of the target water area; The annotation module 802 is used to align the pixels of the optical image with the sonar point cloud data, and then synchronize the optical image, the sonar point cloud data and the environmental parameters in time. After the time synchronization is completed, the optical image and the sonar point cloud data are annotated with defects. The first fusion module 803 is used to perform weighted fusion of the optical image and the sonar point cloud data with disease annotation completed based on the environmental parameters to obtain a multi-source fusion dataset. The second fusion module 804 is used to extract optical texture feature vectors, sonar point cloud spatial feature vectors and environmental parameter feature vectors from the multi-source fusion dataset based on the GAN network, and to fuse the optical texture feature vectors, sonar point cloud spatial feature vectors and environmental parameter feature vectors to obtain a simulation sample dataset. Training module 805 is used to train an underwater structure surface crack segmentation network based on the simulation sample dataset to obtain a defect identification model for identifying underwater structure defects. The underwater structure surface crack segmentation network includes an encoder, a decoder, and a parallel attention module.

[0094] The underwater structure defect identification model generation device provided in the above embodiments can realize the technical solutions described in the above underwater structure defect identification model generation method embodiments. The specific implementation principles of each module or unit can be found in the corresponding content in the above underwater structure defect identification model generation method embodiments, and will not be repeated here.

[0095] like Figure 9 As shown, the present invention also provides an electronic device 900. The electronic device 900 includes a processor 901, a memory 902, and a display 903. Figure 9 Only some components of the electronic device 900 are shown, but it should be understood that it is not required to implement all of the components shown, and more or fewer components may be implemented instead.

[0096] In some embodiments, memory 902 may be an internal storage unit of electronic device 900, such as a hard disk or memory of electronic device 900. In other embodiments, memory 902 may also be an external storage device of electronic device 900, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc. equipped on electronic device 900.

[0097] Furthermore, the memory 902 may include both internal storage units of the electronic device 900 and external storage devices. The memory 902 is used to store application software and various types of data installed on the electronic device 900.

[0098] In some embodiments, processor 901 may be a central processing unit (CPU), microprocessor, or other data processing chip, used to run program code stored in memory 902 or process data, such as the underwater structure defect identification model generation method / underwater structure defect identification method in this invention.

[0099] In some embodiments, display 903 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen. Display 903 is used to display information from electronic device 900 and to display a visual user interface. Components 901-903 of electronic device 900 communicate with each other via a system bus.

[0100] In some embodiments of the present invention, when the processor 901 executes the program for generating the underwater structure defect identification model in the memory 902, the following steps can be implemented: Acquire optical images, sonar point cloud data, and environmental parameters of the target water area; After aligning the pixels of the optical image with the sonar point cloud data, the optical image, the sonar point cloud data, and the environmental parameters are synchronized in time. After the time synchronization is completed, the optical image and the sonar point cloud data are annotated with defects. Based on the environmental parameters, the optical image and the sonar point cloud data with disease annotation are weighted and fused to obtain a multi-source fusion dataset; Based on the GAN network, optical texture feature vectors, sonar point cloud spatial feature vectors, and environmental parameter feature vectors are extracted from the multi-source fusion dataset. The optical texture feature vectors, sonar point cloud spatial feature vectors, and environmental parameter feature vectors are then fused to obtain the simulation sample dataset. A surface crack segmentation network for underwater structures is trained based on the simulated sample dataset to obtain a defect identification model for identifying defects in underwater structures. The underwater structure surface crack segmentation network includes an encoder, a decoder, and a parallel attention module.

[0101] When processor 901 executes the underwater structure defect identification program in memory 902, the following steps can be performed: Acquire optical images, sonar point cloud data, and environmental parameters of the target water area to be identified; After aligning the pixels of the optical image to be identified with the sonar point cloud data, the optical image to be identified, the sonar point cloud data, and the environmental parameters are synchronized in time. Based on the environmental parameters that have completed time synchronization, the time-synchronized optical images and sonar point cloud data are weighted and fused to obtain the multi-source fusion dataset to be identified. Based on the GAN network, optical texture feature vectors, sonar point cloud spatial feature vectors, and environmental parameter feature vectors are extracted from the multi-source fusion dataset to be identified. The optical texture feature vectors, sonar point cloud spatial feature vectors, and environmental parameter feature vectors are then fused to obtain the simulation sample dataset to be identified. The simulation sample dataset to be identified is input into the disease identification model obtained by the method described in any one of claims 1-8 to obtain the underwater structure disease identification result.

[0102] It should be understood that when the processor 901 executes the underwater structure defect identification model generation program / underwater structure defect identification program in the memory 902, in addition to the functions mentioned above, it can also perform other functions, as can be found in the description of the corresponding method embodiments above.

[0103] Furthermore, the embodiments of the present invention do not specifically limit the type of electronic device 900 mentioned. Electronic device 900 can be a mobile phone, tablet computer, personal digital assistant (PDA), wearable device, laptop computer, or other portable electronic device. Exemplary embodiments of portable electronic devices include, but are not limited to, portable electronic devices running iOS, Android, Microsoft, or other operating systems. The aforementioned portable electronic device can also be other portable electronic devices, such as a laptop computer with a touch-sensitive surface (e.g., a touch panel). It should also be understood that in some other embodiments of the present invention, electronic device 900 may not be a portable electronic device, but rather a desktop computer with a touch-sensitive surface (e.g., a touch panel).

[0104] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a method for generating an underwater structure defect identification model provided by the methods described above, the method comprising: Acquire optical images, sonar point cloud data, and environmental parameters of the target water area; After aligning the pixels of the optical image with the sonar point cloud data, the optical image, the sonar point cloud data, and the environmental parameters are synchronized in time. After the time synchronization is completed, the optical image and the sonar point cloud data are annotated with defects. Based on the environmental parameters, the optical image and the sonar point cloud data with disease annotation are weighted and fused to obtain a multi-source fusion dataset; Based on the GAN network, optical texture feature vectors, sonar point cloud spatial feature vectors, and environmental parameter feature vectors are extracted from the multi-source fusion dataset. The optical texture feature vectors, sonar point cloud spatial feature vectors, and environmental parameter feature vectors are then fused to obtain the simulation sample dataset. A surface crack segmentation network for underwater structures is trained based on the simulated sample dataset to obtain a defect identification model for identifying defects in underwater structures. The underwater structure surface crack segmentation network includes an encoder, a decoder, and a parallel attention module.

[0105] When executed by a processor, the computer program is implemented to perform the underwater structure defect identification methods provided by the above methods, which include: Acquire optical images, sonar point cloud data, and environmental parameters of the target water area to be identified; After aligning the pixels of the optical image to be identified with the sonar point cloud data, the optical image to be identified, the sonar point cloud data, and the environmental parameters are synchronized in time. Based on the environmental parameters that have completed time synchronization, the time-synchronized optical images and sonar point cloud data are weighted and fused to obtain the multi-source fusion dataset to be identified. Based on the GAN network, optical texture feature vectors, sonar point cloud spatial feature vectors, and environmental parameter feature vectors are extracted from the multi-source fusion dataset to be identified. The optical texture feature vectors, sonar point cloud spatial feature vectors, and environmental parameter feature vectors are then fused to obtain the simulation sample dataset to be identified. The simulation sample dataset to be identified is input into the disease identification model obtained by the method described in any one of claims 1-8 to obtain the underwater structure disease identification result.

[0106] Those skilled in the art will understand that all or part of the processes of the methods described in the above embodiments can be implemented by a computer program instructing related hardware, and the program can be stored in a computer-readable storage medium. The computer-readable storage medium may be a disk, optical disk, read-only memory, or random access memory, etc.

[0107] The above provides a detailed description of the method, apparatus, and method for generating underwater structure defect identification models provided by this invention. Specific examples have been used to illustrate the principles and implementation methods of this invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of this invention. At the same time, those skilled in the art will recognize that there will be changes in the specific implementation methods and application scope based on the ideas of this invention. Therefore, the content of this specification should not be construed as a limitation of this invention.

Claims

1. A method for generating an underwater structure defect identification model, characterized in that, include: Acquire optical images, sonar point cloud data, and environmental parameters of the target water area; After aligning the pixels of the optical image with the sonar point cloud data, the optical image, the sonar point cloud data, and the environmental parameters are synchronized in time. After the time synchronization is completed, the optical image and the sonar point cloud data are annotated with defects. Based on the environmental parameters, the optical image and the sonar point cloud data with disease annotation are weighted and fused to obtain a multi-source fusion dataset; Based on the GAN network, optical texture feature vectors, sonar point cloud spatial feature vectors, and environmental parameter feature vectors are extracted from the multi-source fusion dataset. The optical texture feature vectors, sonar point cloud spatial feature vectors, and environmental parameter feature vectors are then fused to obtain the simulation sample dataset. A surface crack segmentation network for underwater structures is trained based on the simulated sample dataset to obtain a defect identification model for identifying defects in underwater structures. The underwater structure surface crack segmentation network includes an encoder, a decoder, and a parallel attention module.

2. The method for generating an underwater structure defect identification model according to claim 1, characterized in that, Aligning the pixels of the optical image with the sonar point cloud data includes: Based on the calibration results between the camera used to capture optical images and the sonar used to collect sonar point cloud data, the three-dimensional spatial coordinates of the sonar point cloud data are mapped to the pixel coordinates of the optical image, thereby aligning the pixels of the optical image with the sonar point cloud data. The calibration result is determined based on a projection matrix constructed from the camera's intrinsic and extrinsic parameters and the sonar's extrinsic parameters.

3. The method for generating an underwater structure defect identification model according to claim 2, characterized in that, Also includes: After mapping the three-dimensional spatial coordinates of the sonar point cloud data to the pixel coordinates of the optical image, the registration result of the sonar point cloud data and the optical image is optimized by using an iterative nearest-point algorithm or a feature-based registration method.

4. The method for generating an underwater structure defect identification model according to claim 1, characterized in that, Disease annotation is performed on the optical image and the sonar point cloud data, including: The optical image is labeled with defects using pixel-based segmentation masks or polygonal multi-segment coordinate representation, and the sonar point cloud data is labeled with defects using point cloud indexes or three-dimensional bounding boxes.

5. The method for generating an underwater structure defect identification model according to claim 1, characterized in that, The environmental parameters include: the flow velocity and turbidity of the target water area; Based on the environmental parameters, the optical image and sonar point cloud data with completed lesion annotation are weighted and fused to obtain a multi-source fusion dataset, including: If the turbidity of the target water area is higher than the first preset threshold or the flow velocity of the target water area is higher than the second preset threshold, the optical image and the sonar point cloud data that have completed the disease annotation are weighted and fused based on the first weight corresponding to the optical image to obtain a multi-source fusion dataset. When the turbidity of the target water area is lower than the first preset threshold and the flow velocity of the target water area is lower than the second preset threshold, the optical image and the sonar point cloud data that have completed the disease annotation are weighted and fused based on the second weight corresponding to the optical image to obtain a multi-source fusion dataset. Wherein, the first weight is less than the second weight.

6. The method for generating an underwater structure defect identification model according to claim 1, characterized in that, The GAN network includes: a generator and a discriminator; The generator includes a fully connected mapping layer and six deconvolutional layers, and the output of the generator is equipped with a cross-modal feature fusion module. The discriminator comprises six convolutional layers and one fully connected layer, and introduces a dual-branch authenticity discrimination module in the penultimate layer.

7. The method for generating an underwater structure defect identification model according to claim 6, characterized in that, The generator incorporates a multi-scale attention weighting mechanism that combines sonar and optical features in each deconvolution layer.

8. The method for generating an underwater structure defect identification model according to any one of claims 1-7, characterized in that, The encoder is used to extract multi-scale feature information of cracks from the simulation sample dataset through a residual network to obtain multiple feature maps. The decoder is used to upsample and concatenate the multiple feature maps to obtain multiple new feature maps. The parallel attention module is used to upsample the third new feature map among the multiple new feature maps, concatenate it with the second new feature map to obtain a concatenated feature map, and then concatenate the concatenated feature map with the first new feature map to obtain a feature map that is then passed through the attention module and the convolutional layer in sequence before being output.

9. A method for identifying defects in underwater structures, characterized in that, include: Acquire optical images, sonar point cloud data, and environmental parameters of the target water area to be identified; After aligning the pixels of the optical image to be identified with the sonar point cloud data, the optical image to be identified, the sonar point cloud data, and the environmental parameters are synchronized in time. Based on the environmental parameters that have completed time synchronization, the time-synchronized optical images and sonar point cloud data are weighted and fused to obtain the multi-source fusion dataset to be identified. Based on the GAN network, optical texture feature vectors, sonar point cloud spatial feature vectors, and environmental parameter feature vectors are extracted from the multi-source fusion dataset to be identified. The optical texture feature vectors, sonar point cloud spatial feature vectors, and environmental parameter feature vectors are then fused to obtain the simulation sample dataset to be identified. The simulation sample dataset to be identified is input into the disease identification model obtained by the method described in any one of claims 1-8 to obtain the underwater structure disease identification result.

10. A device for generating an underwater structure defect identification model, characterized in that, include: The acquisition module is used to acquire optical images, sonar point cloud data, and environmental parameters of the target water area; The annotation module is used to align the pixels of the optical image with the sonar point cloud data, then synchronize the optical image, the sonar point cloud data and the environmental parameters in time, and annotate the optical image and the sonar point cloud data with defects after the time synchronization is completed. The first fusion module is used to perform weighted fusion of the optical image and the sonar point cloud data with disease annotation completed based on the environmental parameters to obtain a multi-source fusion dataset. The second fusion module is used to extract optical texture feature vectors, sonar point cloud spatial feature vectors, and environmental parameter feature vectors from the multi-source fusion dataset based on the GAN network, and to fuse the optical texture feature vectors, sonar point cloud spatial feature vectors, and environmental parameter feature vectors to obtain a simulation sample dataset. The training module is used to train an underwater structure surface crack segmentation network based on the simulation sample dataset to obtain a defect identification model for identifying underwater structure defects. The underwater structure surface crack segmentation network includes an encoder, a decoder, and a parallel attention module.

Images