Optical and acoustic image registration method for underwater structure crack detection

By constructing a dual-path multi-scale feature extraction network and an iterative optimization mechanism, the problem of inconsistency between multimodal images from optical and acoustic imaging devices in underwater structural crack detection was solved, achieving high-precision structural crack detection and improving the stability and applicability of the registration algorithm.

CN121482115APending Publication Date: 2026-02-06GUANGZHOU MARITIME INST
View PDF 0 Cites 3 Cited by

Patent Information

Application Number
CN202511659073.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2025-11-13
Publication Date
2026-02-06

AI Technical Summary

Technical Problem

In existing underwater structural crack detection, the multimodal images from optical and acoustic imaging devices are inconsistent in scale, resolution, and semantic representation. Traditional methods lose accuracy in low-texture, high-noise, or distorted scenes, and underwater environmental interference makes it difficult for static registration parameters to be applied in the long term, affecting the stability of cross-modal fusion and its practical application effect.

Method used

By simultaneously acquiring optical and acoustic image data, a dual-path multi-scale feature extraction network is constructed. Features are extracted by combining dilated convolution and frequency domain sensing modules. A continuous local scale field is generated through local scale response maps and manifold regularization optimization. An iterative optimization mechanism is designed to achieve non-rigid transformation of optical and acoustic images and accurate alignment of key structural regions.

Benefits of technology

It significantly improves the alignment accuracy of multimodal features between optical and acoustic images, enhances the stability and robustness of geometric transformation calculations, improves the spatial alignment effect of key structural regions, reduces the dependence on manual annotation and manual parameter tuning, and expands the applicability of the registration algorithm in heterogeneous imaging device environments.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN121482115A_ABST
    Figure CN121482115A_ABST
Patent Text Reader

Abstract

The invention provides an optical and acoustic image registration method for underwater structure crack detection, and the method comprises the steps: collecting a multi-mode image through the vision field and time synchronization, building a one-to-one correspondence relation according to the pose and time metadata, and improving the image quality through the denoising, distortion correction and other means; optical and acoustic multi-scale features are respectively extracted through a dual-channel network, and cross-modal semantic alignment is realized in combination with a shared weight and a channel attention mechanism; generating a high-resolution scale offset field by using local cross-correlation and micro-upsampling, and introducing manifold regularization constraint to ensure that a vector field is smooth and continuous; according to the method, the local scale field and the affine parameters are combined, non-rigid space mapping is achieved through thin-plate spline interpolation, feedback iterative optimization based on edge structure consistency is assisted, the registration precision of a key structure is enhanced, the automatic registration effect of the underwater multi-modal image is improved, and the method has high robustness and practical application value.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of underwater multimodal image registration and processing technology, and in particular to an optical and acoustic image registration method for detecting cracks in underwater structures. Background Technology

[0002] Currently, in the fields of underwater structure crack detection, photoacoustic image registration, and geometric transformation calculation, mainstream technologies are mostly centered on multimodal image processing and registration, primarily relying on optical and acoustic imaging equipment to acquire information about the structure's surface and interior. With the increasing complexity of underwater inspection tasks, especially the growing demand for structural health monitoring in harsh environments, multimodal data fusion is increasingly becoming an industry trend. Common publicly available technical solutions mainly include point or region feature matching methods, geometric registration methods based on affine or non-rigid transformations, and deep learning-driven cross-modal mapping models. These methods typically establish a matching relationship by extracting texture features from optical images and echo structure features from acoustic images, and then align the multimodal data using spatial transformation parameters, thereby achieving multidimensional information fusion for underwater structure characterization.

[0003] However, the technology currently faces the following significant bottlenecks: (1) Due to the fundamental differences in the imaging mechanisms of optics and acoustics, optical images focus on material texture and surface reflection, with high resolution but are easily affected by disturbances, while acoustic images focus on internal echoes of structures, with strong anti-interference ability but low spatial resolution. This results in inconsistencies in scale, resolution, and semantic expression among multimodal images of the same structure. Traditional matching methods based on unified feature descriptors have decreased accuracy in low-texture, high-noise, or distorted scenes, and cannot dynamically handle scale drift caused by environmental changes. (2) Existing deformation models usually assume that the registration process is globally continuous or locally rigid, which makes it difficult to effectively adapt to the heterogeneous deformation characteristics of key areas such as cracks and faults on the surface of the structure, and is prone to edge misalignment or deformation fracture. (3) Due to the instability of the underwater environment, dynamic factors such as medium disturbance, platform jitter and illumination change significantly affect image quality, making it difficult for static registration parameters to be applicable in the long term, and the algorithm is prone to getting stuck in the convergence bottleneck, affecting the stability of cross-modal fusion and the actual application effect. Summary of the Invention

[0004] In order to solve the above-mentioned technical problems, the present invention provides an optical and acoustic image registration method for detecting cracks in underwater structures.

[0005] The technical solution of this invention is implemented as follows: an optical and acoustic image registration method for detecting cracks in underwater structures, comprising: S1: Simultaneously acquire optical and acoustic image data pairs of underwater structures under the same field of view, and use them as input samples for multimodal image registration processing; S2: Denoising and geometric distortion correction preprocessing are performed on the acquired optical and acoustic images respectively to eliminate the influence of medium disturbance and sensor error in the underwater environment on image quality. S3: Construct a dual-path multi-scale feature extraction network, in which the optical branch uses dilated convolution to enhance the receptive field to preserve texture details, and the acoustic branch introduces a frequency domain sensing module to highlight echo structure features, and promotes semantic alignment between modalities by sharing part of the network weights. S4: Generate a local scale response map at each scale layer, calculate a cross-modal similarity heatmap through cross-correlation operation, and regress it into a pixel-level scale offset vector field using differentiable upsampling; S5: Introducing a manifold regularization term to smooth the scale offset vector field ensures the continuity of the transformation gradient in adjacent regions, avoids breakage or distortion, and thus generates a continuous and stable local scale field. S6: The local scale field and affine transformation parameters are jointly encoded into a non-rigid deformable mesh, and a thin plate spline interpolation model is used to perform spatial coordinate mapping on optical and acoustic images; S7: Design an iterative optimization mechanism based on structural consistency feedback, dynamically adjust the scale field weight distribution by utilizing the edge alignment degree of the registered image, and focus on enhancing the alignment accuracy of key structural regions such as cracks; S8: Performs image resampling operation through differentiable bilinear sampling, outputs registered optical and acoustic image pairs, and calculates the weighted values ​​of structural similarity loss, gradient consistency loss and manifold smoothness loss for model training and parameter optimization.

[0006] The optical and acoustic image registration method provided by this invention for detecting cracks in underwater structures has the following advantages: (1) By constructing a dual-path multi-scale feature extraction network and combining dynamic scale alignment and manifold regularization optimization, this invention can significantly improve the multimodal feature alignment accuracy when the scales of optical and acoustic images are inconsistent. In particular, in complex underwater environments with low texture and high distortion, it can effectively enhance the stability and robustness of geometric transformation calculation. (2) This invention adopts local scale response modeling and thin plate spline interpolation non-rigid mesh mapping, which realizes adaptive correction of cross-modal scale differences without relying on point feature matching, greatly improves the spatial alignment effect of key structural areas such as cracks, and significantly expands the applicability of the registration algorithm in heterogeneous imaging equipment environment; (3) This invention combines an iterative optimization mechanism based on structural consistency feedback with multiple loss trade-off strategies to achieve structural detail protection, transformation smoothness constraints and accurate registration of edge structures in the overall process, effectively improving registration quality and training convergence efficiency, and significantly reducing the dependence on manual annotation and manual parameter tuning. Attached Figure Description

[0007] Figure 1 This is a flowchart of an optical and acoustic image registration method for detecting cracks in underwater structures according to the present invention; Figure 2 This is a sub-flowchart of an optical and acoustic image registration method for detecting cracks in underwater structures according to the present invention. Figure 3 This is another sub-flowchart of an optical and acoustic image registration method for detecting cracks in underwater structures according to the present invention. Detailed Implementation

[0008] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0009] The following disclosure provides many different embodiments or examples for implementing different structures of the invention. To simplify the disclosure, specific examples of components and arrangements are described below. Of course, these are merely examples and are not intended to limit the invention. Furthermore, reference numerals and / or letters may be repeated in different examples; such repetition is for simplification and clarity and does not in itself indicate a relationship between the various embodiments and / or arrangements discussed.

[0010] like Figure 1 As shown, this invention provides an optical and acoustic image registration method for detecting cracks in underwater structures, specifically including: S1: Simultaneously acquire optical and acoustic image data pairs of underwater structures under the same field of view, and use them as input samples for multimodal image registration processing; S2: Denoising and geometric distortion correction preprocessing are performed on the acquired optical and acoustic images respectively to eliminate the influence of medium disturbance and sensor error in the underwater environment on image quality. S3: Construct a dual-path multi-scale feature extraction network, in which the optical branch uses dilated convolution to enhance the receptive field to preserve texture details, and the acoustic branch introduces a frequency domain sensing module to highlight echo structure features, and promotes semantic alignment between modalities by sharing part of the network weights. S4: Generate a local scale response map at each scale layer, calculate a cross-modal similarity heatmap through cross-correlation operation, and regress it into a pixel-level scale offset vector field using differentiable upsampling; S5: Introducing a manifold regularization term to smooth the scale offset vector field ensures the continuity of the transformation gradient in adjacent regions, avoids breakage or distortion, and thus generates a continuous and stable local scale field. S6: The local scale field and affine transformation parameters are jointly encoded into a non-rigid deformable mesh, and a thin plate spline interpolation model is used to perform spatial coordinate mapping on optical and acoustic images; S7: Design an iterative optimization mechanism based on structural consistency feedback, dynamically adjust the scale field weight distribution by utilizing the edge alignment degree of the registered image, and focus on enhancing the alignment accuracy of key structural regions such as cracks; S8: Performs image resampling operation through differentiable bilinear sampling, outputs registered optical and acoustic image pairs, and calculates the weighted values ​​of structural similarity loss, gradient consistency loss and manifold smoothness loss for model training and parameter optimization.

[0011] Step S1: Simultaneously acquire optical and acoustic image data pairs of the underwater structure from the same field of view, using them as input samples for multimodal image registration processing. Specifically, this includes: S1.1: Based on the deployment location and field of view parameters of the underwater imaging system, plan the spatial layout of the optical imaging equipment and the acoustic imaging equipment to ensure that the imaging fields of the two modal images have spatial consistency and obtain the initial aligned imaging geometric relationship. The input conditions include the physical installation location parameters of the underwater optical imaging equipment and acoustic imaging equipment, the field of view (FOV) parameters, and the spatial coordinate system definition of the target detection area; A three-dimensional spatial coordinate calibration method (parameters: target coordinate system origin position, device coordinate system transformation matrix) is used to realize the installation position analysis of optical imaging equipment and acoustic imaging equipment in the same global / local coordinate system, and to establish the spatial relative pose relationship between the two. Furthermore, by using the field-of-view angle matching calculation method (parameters: horizontal / vertical field of view of optical equipment, beam divergence angle of acoustic equipment), the spatial overlap region analysis of the two-modal imaging areas is realized, and the set of device pointing vectors under the condition of maximum overlap rate is obtained; Furthermore, by using ray tracing modeling (parameters: underwater medium refractive index distribution, equipment installation depth, target detection area boundary), the propagation path of the optical line of sight and the acoustic beam in the underwater medium is simulated, and the coordinate data of the propagation path endpoints are generated; Furthermore, by using the optimal field of view coverage algorithm (parameters: calculation formula for overlap rate of two-modal fields of view, variable range constraint of installation position), the optimal selection of equipment installation position is achieved, and an initial alignment layout scheme under geometric conditions is generated; By constructing a geometric relation matrix, the results of the previous step are transformed into an initial alignment transformation matrix between the optical and acoustic coordinate systems, thereby achieving the spatial consistency effect of the two-modal imaging field of view. For example, in the scenario of detecting cracks in underwater bridge piers, the field of view of the optical imaging device is set to 40° horizontally and 30° vertically, and the beam divergence angle of the acoustic imaging device is set to 35° horizontally and 25° vertically. The two devices are mounted on fixed supports 1.5 meters apart. The relative installation angles of the two devices are obtained through three-dimensional spatial coordinate calibration, showing a horizontal deflection of 5° and a vertical deflection of 3°. During ray tracing modeling, the water refractive index is set to 1.34, the installation depth to 5 meters, and the target detection area radius to 20 meters. The calculated spatial intersection area between the optical line of sight and the acoustic beam in the target area reaches 92%. Using the optimal field of view coverage algorithm, with an overlap rate target of no less than 90% and an adjustable installation position range of ±0.2 meters, the final installation position coordinates satisfy the condition of maximizing the overlap area. The initial alignment transformation matrix between the optical and acoustic coordinate systems is calculated as follows: The verification results show that under the set conditions, the consistency of the field of view of the two-modal imaging is significantly improved, which meets the input requirements of high-precision geometric transformation calculation. S1.2: Perform time synchronization calibration on optical imaging equipment and acoustic imaging equipment, and achieve millisecond-level synchronization of image acquisition time based on external trigger signal or timestamp marking mechanism to eliminate dynamic motion error caused by differences in equipment response delay; The acquisition and control signal input of optical imaging equipment and acoustic imaging equipment are processed to establish a time reference. A high-precision clock synchronization algorithm (parameters: crystal oscillator frequency 10MHz, temperature compensation coefficient 1ppm) is adopted to realize the periodic correction function of the internal clock of the two devices. Furthermore, the acquisition control lines of the two devices are triggered in parallel by an external trigger signal distribution module (parameters: TTL level, pulse width 10μs, trigger interval 100ms) to achieve physical-level synchronization at the start of acquisition and obtain trigger response time data. Furthermore, a timestamp marking mechanism (parameters: timestamp resolution 1μs, reference clock source is GPS timing module) is adopted to implement time tag appending processing when each frame of image acquisition is completed, and to generate a timestamp dataset; Furthermore, by using a delay difference calculation method (based on the mean difference and variance difference between two timestamp sets), a quantitative assessment of the device acquisition response delay is achieved, and a delay error vector set is generated. Furthermore, a minimum delay error calibration algorithm is adopted (parameters: the acquisition period is fixed at 0.1s, and the delay difference threshold is ±0.5ms) to realize the dynamic adjustment of the internal cache read waiting time of the device and generate a trigger control configuration after millisecond-level synchronization; Through the above algorithm processing method, the spatial layout result of the previous step is transformed into acquisition and control data with millisecond-level time consistency, so as to achieve the expected technical effect of eliminating dynamic motion error caused by differences in device response delay. For example, in an underwater crack detection task, the optical imaging equipment and the acoustic imaging equipment are configured as a digital camera with a resolution of 1920×1080 and a multi-beam sonar with a 2MHz resolution, respectively. The system uses a 10MHz high-stability temperature-compensated crystal oscillator as a unified clock source. A 10μs pulse width trigger pulse is distributed to the acquisition control port of the two devices through an external TTL signal synchronization module. The GPS timing module provides a timestamp with a precision of 1μs. The acquisition period is set to 100ms. The delay calculation is performed on the acquired timestamp data, and the mean of the initial delay error vector set is 1.2ms and the variance is 0.3ms. Through delay difference calculation, under the condition of acquisition frame number n=100, the mean square of the delay deviation is 1.44 / (99)≈0.01455. Based on this result, the internal buffer waiting time of the two devices is adjusted, and the final delay difference is reduced to 0.3ms, which significantly improves the time consistency of images of different modalities and provides a synchronization guarantee for subsequent cross-modal matching of crack areas. S1.3: During the underwater structure inspection process, control the optical imaging equipment to acquire high-resolution texture images, and simultaneously control the acoustic imaging equipment to acquire echo intensity images, so as to obtain multimodal image pairs of the same structural region and form the original image dataset. In underwater structure inspection tasks, optical imaging equipment and acoustic imaging equipment, after spatial layout planning and time synchronization calibration, are used as the execution objects to collect multimodal raw image pairs of the same structural area; By employing a precise exposure control method (parameters: exposure time is automatically adjusted to the range of 1~10 ms according to the underwater light intensity, and ISO sensitivity is in the range of 200~800), the optical imaging device can achieve high-resolution texture acquisition under low-light conditions in water, and ensure that texture details are clearly distinguishable under underwater environmental scattering conditions; Furthermore, by using a pulse-modulated acoustic beam emission method (parameters: center frequency 1 MHz, pulse width 50 μs, pulse repetition frequency 200 Hz), the echo intensity acquisition function of the acoustic imaging device is realized, and high-density echo response data of the structural surface position reflection is obtained. Furthermore, a synchronous signal triggering mechanism (parameter: microsecond-level delay gating) is adopted to achieve time synchronization between the start of optical image exposure and acoustic pulse emission, and to generate time-consistent data results for cross-modal image acquisition; Furthermore, by using a multi-frame sliding window sampling mechanism (parameter: 5~10 frames per window), local temporal averaging of the original optical and acoustic images is achieved, and a noise-suppressed multimodal frame sequence is generated, thereby improving the stability of the original dataset. Through the above-mentioned multimodal synchronous acquisition and signal control processing method, the acquisition results under the conditions of spatial and temporal consistency in the previous step are transformed into a raw image dataset containing high-resolution texture information and echo intensity information, thereby optimizing the quality of the initial input data for multimodal image registration. For example, in an underwater reinforced concrete bridge pier crack monitoring scenario, the optical imaging device is equipped with a 5-megapixel CMOS sensor, a lens field of view of 70°, an exposure time set to 5 ms, and an ISO sensitivity of 400; the acoustic imaging device is equipped with a 1 MHz transducer array, a single pulse width of 50 μs, a pulse repetition frequency of 200 Hz, and a receiving window length of 2 ms. Both devices are linked via an external TTL trigger signal, with the trigger delay controlled within 2 μs. During the detection process, the optical device outputs high-resolution texture frames with a resolution of 2560×1920 pixels, and the acoustic device outputs echo intensity matrix frames with a resolution of 512×512. A sliding window averaging of 5 frames is used to perform short-time steady-state processing on the optical and acoustic data, and a unified acquisition index table is generated using the timestamps of synchronous acquisition. Verification showed that the original optical and acoustic image pairs acquired under this configuration exhibited stable spatial correspondence and temporal consistency, and the matching degree between texture boundaries and sound beam reflection positions was significantly improved, providing high-quality, multimodal input data for subsequent multi-scale feature extraction and geometric transformation calculations. S1.4: Metadata annotation is performed on the acquired optical and acoustic images, and an image pairing index table is generated based on the pose information and timestamp information of the imaging device to establish the correspondence between multimodal images; S1.5: Based on the image pairing index table, the optical and acoustic images are preprocessed for spatial resolution matching. Bilinear interpolation or super-resolution reconstruction algorithms are used to unify the image resolution in order to improve the initial alignment accuracy of subsequent image registration.

[0012] Step S2: The acquired optical and acoustic images are preprocessed with denoising and geometric distortion correction to eliminate the impact of underwater disturbances and sensor errors on image quality. Specifically, this includes: S2.1: Perform denoising processing based on nonlocal mean filtering on the original optical image and acoustic image respectively to suppress image noise introduced by scattering medium and sensor thermal noise during underwater imaging, and obtain denoised optical image and acoustic image; S2.2: Based on the underwater optical imaging model, the denoised optical image is subjected to refractive distortion correction processing to compensate for the image deformation caused by the water-air interface and the non-uniformity of the refractive index of the water body, and the refractive corrected optical image is obtained. S2.3: Perform beam offset correction on the denoised acoustic image using acoustic propagation path modeling to eliminate the geometric distortion of the image caused by the deflection of the propagation path of the sound wave in the non-uniform medium, and obtain the beam-corrected acoustic image. S2.4: Based on the device attitude sensor data, perform motion blur compensation processing on the optical image after refraction correction and the acoustic image after sound beam correction, respectively, to correct the image blur caused by underwater platform shaking or mechanical vibration, and obtain attitude-compensated optical and acoustic images. S2.5: Perform gamma correction and dynamic range compression on the pose-compensated optical and acoustic images to enhance image contrast and improve the visibility of image details under low light or high dynamic range conditions, and obtain high-quality image pairs after final preprocessing, which will serve as input for the subsequent multi-scale feature extraction module. For the optical and acoustic images after attitude compensation, a gamma correction method based on grayscale mapping is adopted (parameters: the initial gamma coefficient γ0 is preset according to the contrast characteristics of the optical mode histogram, and the acoustic mode γ0 is adjusted according to the echo intensity distribution characteristics) to achieve nonlinear mapping of the pixel grayscale range, so as to improve the visibility of details in the low and medium brightness areas. Furthermore, an adaptive gamma coefficient adjustment algorithm (parameter: local brightness mean) is used. luminance standard deviation This allows for dynamic gamma fine-tuning of different regions and the generation of an enhanced image brightness distribution matrix. Furthermore, a dynamic range compression method based on histogram equalization is adopted (parameter: compression coefficient β is set according to the maximum and minimum ratio of brightness distribution) to achieve brightness compression in high dynamic range areas and generate a compressed pixel brightness matrix to control the overflow of bright areas and the loss of dark details. Furthermore, a multi-scale brightness fusion algorithm is adopted (parameter: fusion weight is adaptively adjusted according to the goals of detail enhancement and contrast improvement) to achieve fusion processing of different brightness levels and generate a globally optimized image matrix; By combining gamma correction and dynamic range compression, the pose compensation result from the previous step is transformed into a high-quality image pair with enhanced contrast and improved detail visibility, thereby achieving input quality optimization under multimodal and non-uniform lighting conditions. For example, in the scenario of deep-sea bridge pier inspection, the optical image resolution is 1920×1080, the acoustic image resolution is 1024×1024, and the initial gamma coefficient of the optical mode is set to... The initial gamma coefficients of the acoustic modes are set to The local brightness average threshold is set to The standard deviation threshold for brightness is set to In gamma correction, the formula for adjusting the local gamma coefficient is: in The adjusted local gamma coefficients. The local gamma coefficients are as shown before adjustment. The dynamic range compression factor β is used when the maximum / minimum luminance ratio exceeds... Time setting After histogram equalization, the discernibility of details in dark areas is significantly improved. After multi-scale brightness fusion processing, the grayscale of crack areas in the dark parts of the optical image is enhanced to the original level. The acoustic image highlights and echo edges are eliminated, resulting in high-quality images that show significantly improved texture fidelity and structural clarity in subsequent multi-scale feature extraction networks.

[0013] like Figure 2 As shown, step S3 involves constructing a dual-path, multi-scale feature extraction network. The optical branch employs dilated convolutions to enhance the receptive field and preserve texture details, while the acoustic branch introduces a frequency-domain sensing module to highlight echo structure features. Furthermore, it promotes inter-modal semantic alignment by sharing some network weights. Specifically, this includes: S3.1: Perform dilated convolution on the input optical image data to expand the receptive field and preserve the texture details in the image, and obtain multi-scale dilated convolution feature maps as high-level semantic representations of the optical branch; For high-quality optical image data preprocessed in step S2, a dilated convolution algorithm (parameters: dilation r=2, 3, 5, kernel size k=3×3, stride s=1, padding method is zero padding) is used to expand the receptive field of the feature extraction layer and preserve fine-grained texture features without increasing the number of parameters. By inserting dilated displacements between elements of the convolution kernel through the expanded interval of dilated convolution, both global structure and local details are simultaneously covered, resulting in an initial feature mapping matrix containing multiple receptive scales. Furthermore, by stacking convolutional operations with different porosity parameters layer by layer (the porosity increases from 2 to 3 to 5), the fusion of features from multiple receptive domains is achieved, resulting in a multi-scale dilated convolutional feature map. This map enhances the response to linear structures such as cracks in the spatial domain, ensuring that fine cracks are not distorted in high-level semantic representation. This stacking process involves weighted fusion of feature maps layer by layer, with the weight coefficients adaptively adjusted according to the energy distribution of the feature channels to preserve low-frequency structural information without losing high-frequency texture details. Furthermore, batch normalization is applied to the feature maps to achieve a balanced distribution of cross-channel features, eliminate statistical biases in the mean and variance of convolutional outputs with different dilation rates, and obtain a numerically stable multi-scale dilated convolutional feature set. The batch normalization formula is expressed as: in, The features are normalized. To output feature values ​​for convolution, The characteristic mean, The characteristic standard deviation, and These are learnable scaling and translation parameters; Furthermore, the normalized features are processed using the nonlinear activation function ReLU. Element-wise activation is performed to enhance nonlinear representation capabilities, improve the separability of crack edges in feature space, and output the final multi-scale dilated convolutional feature map, which serves as the input for the high-level semantic representation of the optical branch to the subsequent feature pyramid structure construction in S3.2. By using the above-mentioned dilated convolution processing method, the original pixel distribution of the preprocessed image is transformed into a multi-scale, multi-receptive field feature representation, thereby achieving the optical feature robustness and scale adaptability required for non-rigid registration. For example, in a submarine pipeline inspection task, a refractively corrected optical image with a resolution of 1920×1080 is used as input. The dilated convolution parameters are set to three stacked layers: (r=2, k=3×3), (r=3, k=3×3), and (r=5, k=3×3). The convolution kernel weights are initialized using a He normal distribution and batch normalized. and Initialize to 1 and 0. During normalization, calculate the channel mean of the first layer output feature map. =12.35, standard deviation =3.27, which is substituted into the batch normalization formula to obtain the adjusted feature value. After ReLU processing, the channel energy is concentrated at the crack edge. Quantitative evaluation shows that the feature map enhanced by the hole convolution has a significantly improved response value in the crack region. In the subsequent registration process, this high-level feature expression effectively enhances the alignment accuracy of optical and acoustic modes in the critical defect region and reduces the matching error in non-critical regions. S3.2: Based on dilated convolutional feature maps, a multi-scale pyramid structure is constructed to extract and fuse optical image features at different scales in layers, generating multi-level optical feature representations with scale invariance, which is used to enhance the scale robustness of image registration. S3.3: Perform a Fast Fourier Transform (FFT) operation on the input acoustic image data to obtain its frequency domain features, and design a frequency domain sensing module based on the frequency domain energy distribution to extract the significant features of the echo structure in the acoustic image and generate a frequency domain enhanced feature map. For the acoustic image data preprocessed by S2.5, the Fast Fourier Transform algorithm (parameters: window function type is selected as rectangular window, window length is set to 256 sampling points, FFT length is 512 points) is used to convert the spatial domain pixel intensity distribution of the acoustic image into the corresponding frequency domain amplitude and phase spectrum data. Furthermore, by using the frequency domain amplitude spectrum analysis method (parameters: frequency resolution is determined by the FFT length, and the analysis range is 0 to half the sampling frequency), the energy distribution features of the acoustic image at each frequency component are extracted, and a complete frequency domain energy spectrum matrix is ​​obtained; Furthermore, by using the frequency domain filter bank analysis method (parameters: the filter center frequency is divided into equal intervals, and the bandwidth is set to narrowband mode according to the underwater acoustic characteristics), the frequency domain energy spectrum matrix is ​​analyzed by band, and the mean, variance and peak position of energy in each sub-band are extracted to form a band energy feature vector. Furthermore, by using a frequency domain pattern recognition algorithm (parameter: cosine similarity threshold of 0.85 for feature similarity measurement), the echo structure features based on the peak position and distribution pattern of the energy spectrum are determined, and a primary feature map highlighting the echo information is obtained. Furthermore, through the frequency domain enhancement module (parameter: enhancement coefficient is set according to feature importance weight, and the weight is calculated from the correlation coefficient of the feature to the target structure), the frequency components related to crack edges or structural contours in the primary feature map are weighted and enhanced to generate a frequency domain enhanced feature map, which is used for the construction of the multi-scale frequency domain pyramid in the subsequent S3.4 step; Through the above frequency domain feature extraction and enhancement processing, the acoustic image data of the previous step is transformed into a frequency domain enhanced feature map with echo structure saliency and smooth noise suppression effect, so as to preserve and enhance the acoustic modal structure information during cross-modal feature extraction. For example, in the scenario of detecting cracks in a subsea pipeline, an FFT transformation is performed on the acoustic imaging results after denoising and distortion correction. The sampling frequency is set to 200kHz, the window length is 256 points, and the FFT length is 512 points, obtaining the amplitude and phase spectra with a frequency resolution of approximately 390Hz. Using a filter bank with a center frequency ranging from 1kHz to 20kHz and a bandwidth of 200Hz, a band-wise energy analysis is performed on the frequency domain spectral matrix. The mean energy in each frequency band is found to be between 0.005 and 0.15, and the variance is distributed between 0.0001 and 0.008. The peak frequency positions are mainly concentrated around 3kHz and 7kHz. Using the cosine similarity pattern recognition method, under the condition of a threshold of 0.85, frequency components that match the historical crack echo patterns are extracted and weighted with an enhancement coefficient of 1.5. The resulting frequency domain enhanced feature map increases the energy peak value in the crack region to about 1.45 times the original value, and reduces the noise component amplitude in the background region to about 0.6 times the original value. It has been verified that inputting this feature map into the subsequent multi-scale frequency domain pyramid construction module can significantly improve the matching accuracy and structural stability of crack edges in cross-modal registration. S3.4: Based on the frequency domain enhanced feature map, a multi-scale frequency domain pyramid structure is constructed to perform layer-by-layer modeling and fusion of acoustic image features of different frequency components, generating a multi-level acoustic feature representation with spectral stability, which is used to improve the structural consistency of cross-modal matching; For the acoustic frequency domain enhanced feature map processed by the frequency domain sensing module, a multi-resolution spectral decomposition method (parameters: number of decomposition layers L=4, frequency division method is logarithmic scale) is used to realize the hierarchical decomposition and representation of acoustic image features of different frequency components. Furthermore, by using a multi-scale sub-band energy normalization algorithm (parameter: normalization factor is dynamically calculated based on the average energy of each sub-band), the contrast and energy amplitude of each frequency component are unified, and the normalized frequency domain sub-band feature matrix at each scale is obtained. Furthermore, a cross-scale correlation modeling method (parameter: correlation threshold τ=0.85, calculated based on Pearson correlation coefficient) is adopted to evaluate the structural correlation between different frequency components and generate a frequency correlation matrix to guide the weight allocation during subsequent feature fusion. Furthermore, a weighted fusion algorithm is applied (parameter: the fusion weight is the row normalized value of the frequency correlation matrix) to realize the feature fusion of each frequency component in the multi-scale space and generate a preliminary multi-level frequency domain feature representation; By using convolutional smoothing filtering (parameters: kernel size k=3, boundary processing method is mirror filling), the fusion result of the previous step is transformed into a multi-level acoustic feature representation with spectral stability and spatial continuity, thereby improving the structural consistency of cross-modal matching. For example, in the scenario of detecting cracks at the bottom of a bridge, the FFT result of the preprocessed acoustic image acquired by the acoustic imaging device is a 512-point spectrum. This spectrum is divided into four layers on a logarithmic scale, with frequency ranges corresponding to four sub-bands: [0, 100Hz], [100Hz, 500Hz], [500Hz, 2kHz], and [2kHz, 10kHz]. The average energy is calculated for each sub-band; for example, the average energy is 0.12 for the first sub-band, 0.35 for the second, 0.28 for the third, and 0.15 for the fourth. Normalization is performed on the feature matrix of each sub-band, and the normalized amplitude is calculated using the following formula. : in, This represents the current spectral amplitude. Let be the average energy of the i-th subband. Calculate the Pearson correlation coefficient between the feature matrices at each scale. When the correlation coefficient is higher than 0.85, the frequency component is given a higher weight during fusion. For example, the correlation coefficient between the first and second subbands reaches 0.92 in the crack edge region, so the weights for fusing features in this region are set to 0.52 and 0.48 respectively. A 3×3 convolution kernel is used to smooth the fusion result, ensuring the continuity and structural consistency of the frequency domain features in spatial distribution. Finally, a multi-level acoustic frequency domain feature representation that can be used for cross-modal feature fusion is generated. In this scenario, the generated multi-level acoustic features show significantly improved spectral stability in the crack detection region, and the structural consistency of the crack contour during cross-modal registration is significantly improved compared to before processing. S3.5: Based on the multi-level feature representation of optical and acoustic images, a cross-modal feature fusion layer with shared weights is constructed to jointly optimize the high-level features of the optical and acoustic branches, generate a unified cross-modal semantic feature space, and improve the semantic alignment capability between dual-modal images. Based on multi-level feature representation of optical and acoustic images, a shared weight convolution fusion method (parameter: sharing ratio range 0.25~0.5) is adopted to realize the unified mapping construction of cross-modal feature space. The high-level features of the two branches are input into a unified three-dimensional convolution kernel set to maintain the consistency of different modal features in spatial and semantic dimensions. Furthermore, by using a cross-modal feature regularization method (parameter: L2 norm constraint coefficient 0.001), the output of the fusion layer is unified on the numerical scale, and a cross-modal high-level feature tensor with norm normalization is obtained to reduce the distribution shift effect caused by the difference between optical and acoustic imaging. Furthermore, by using the feature correlation maximization method (parameter: mutual information threshold set to 0.6), the correlation between modalities in the fused features is enhanced, and a mutual information matrix is ​​generated to quantify the semantic consistency index of the fused features. Furthermore, a redundancy feature suppression method based on matrix singular value decomposition (parameter: singular value truncation ratio 0.7) is adopted to achieve linear redundancy compression of cross-modal fusion features and obtain a deredundant unified feature space representation, so as to improve the effective information ratio of subsequent registration calculations. Furthermore, by utilizing the cross-modal space mapping continuity constraint method (parameter: constraint weight 0.05), the fused features are mapped to a unified cross-modal semantic space, and a tensor structure with smooth and consistent feature distribution is output, thereby significantly improving the semantic alignment capability of bimodal images. By using shared weighted convolutional fusion and regularization optimization, the previous step of optical and acoustic multi-level feature representation is transformed into a unified cross-modal semantic feature space, achieving the expected technical effect of significantly enhanced cross-modal alignment accuracy and semantic consistency. For example, in the scenario of detecting cracks in seabed steel structures, the optical branch inputs a 1024×1024-resolution dilated convolutional pyramid feature map, and the acoustic branch inputs a 512×512-resolution frequency domain pyramid feature map. After expanding the acoustic features to the same spatial size through nearest neighbor interpolation, cross-modal fusion is performed using a 3D convolutional kernel with a shared ratio of 0.4. During the fusion process, L2 norm normalization is performed on each channel of the optical and acoustic features, with the coefficient set to 0.001, so that the modulus lengths of the feature vectors of the two branches remain within the same numerical scale range. The mutual information matrix is ​​obtained through mutual information calculation, and a subset of features with a threshold greater than 0.6 is selected to participate in subsequent fusion optimization. For redundancy suppression, singular value decomposition is first performed on the fused feature matrix, with a truncation ratio set to 0.7, retaining the feature vector group corresponding to the principal singular value to achieve effective feature compression. In the continuity constraint stage, a gradient constraint with a smoothing weight of 0.05 is applied to the mapped feature space, so that the feature changes in adjacent regions are in a continuous transition, the cross-modal feature alignment at the crack boundary is greatly improved, and the final output cross-modal semantic feature space shows obvious stability advantages in actual registration algorithm calls. S3.6: Perform a channel attention mechanism on the cross-modal semantic feature space, calculate the importance weight of each channel feature, and perform weighted fusion of the feature maps to generate a joint feature representation with channel enhancement characteristics, which is used to improve the generation accuracy of subsequent local scale response maps; S3.7: Based on channel-enhanced joint feature representation, a multi-scale feature pyramid fusion module is constructed to perform cross-scale fusion and upsampling operations on features at different levels, generating the final multi-scale dual-path feature output as the input basis for generating local scale response maps.

[0014] like Figure 3As shown, step S4 involves generating a local scale response map at each scale level, calculating a cross-modal similarity heatmap through cross-correlation operations, and regressing it into a pixel-level scale offset vector field using differentiable upsampling. Specifically, this includes: S4.1: The multi-scale feature maps output by the dual-path feature extraction network are processed in layers to construct optical and acoustic modal feature representations at each scale level, providing input for subsequent cross-modal matching; The feature map data output by the multi-scale dual-path feature extraction network is analyzed to clarify the hierarchical index and spatial resolution information of the optical modal feature map and the acoustic modal feature map in the multi-scale pyramid structure, which serves as the parameter basis for hierarchical processing. A scale separation algorithm (parameters: pyramid level index, sampling coefficients) is used to perform modal splitting on the joint feature output, dividing the optical modal features and acoustic modal features into independent feature subsets at each scale level, thereby achieving structural isolation of cross-modal features; Furthermore, the modal feature maps of each scale level are normalized by a hierarchical normalization method (parameters: global mean μ, variance σ) to eliminate the statistical differences in the distribution of feature amplitudes at different scales, and to obtain the normalized hierarchical optical feature set and hierarchical acoustic feature set. Furthermore, by using the channel-preserving convolutional filtering algorithm (parameters: kernel size 3×3, stride 1), local context filtering is applied to the hierarchical optical feature set and hierarchical acoustic feature set while maintaining the feature space resolution, so as to enhance the detail continuity of local regions and thus improve the discrimination ability of subsequent cross-modal matching. Furthermore, matrix dimension standardization is employed to uniformly map the optical modal features and acoustic modal features at different scale levels after hierarchical processing to the same tensor dimension structure, thereby constructing optical modal feature representation matrices and acoustic modal feature representation matrices at each scale level, providing a consistent input structure for cross-modal similarity calculation; By using a hierarchical processing algorithm, the multi-scale dual-path feature output is transformed into an optical and acoustic modal feature representation matrix with spatial and scale consistency, thereby achieving the input preparation technology effect for cross-modal similarity heatmap calculation. For example, for a dual-path feature map whose output is a five-layer pyramid structure, the pyramid level index is set to L1-L5. Modal splitting is performed on the optical and acoustic modal features of each level, forming five sets of optical feature subsets and five sets of acoustic feature subsets. For the optical feature map of layer L1, the global mean is calculated. =0.42 and variance =0.15, perform normalization to obtain a standardized feature map with a mean of 0 and a variance of 1; process the acoustic feature map of layer L1 in the same way to obtain the acoustic feature representation of the matching distribution. Apply channel-preserving convolution with a kernel size of 3×3 and a stride of 1 to the normalized feature maps of all levels to enhance the continuity of local texture or echo structure. Normalize the processed feature maps of different levels to a tensor dimension of 256×256 through zero-padding or cropping operations to form five sets of optical modal feature representation matrices and five sets of acoustic modal feature representation matrices. In practical verification, this hierarchical processing can significantly improve the stability of matching peaks in cross-correlation operations, reduce matching bias caused by scale differences, and ultimately improve the accuracy and robustness of cross-modal geometric transformation estimation. S4.2: Based on the optical and acoustic feature maps at each scale level, perform local cross-correlation operations to calculate cross-modal similarity heatmaps to reflect the degree of matching between different local regions in different modes; Based on optical and acoustic feature maps at various scale levels, a local cross-correlation calculation method (parameters: sliding window size w, step size s, correlation coefficient calculation radius r) is used to measure the degree of matching of cross-modal features in the local spatial range. Furthermore, the matching robustness in low-texture and high-noise underwater environments is enhanced by using the Zero-Normalized Cross-Correlation (ZNCC) algorithm (parameters: mean pixel gray level μ and standard deviation σ within the window), and the original cross-modal similarity matrix R is obtained. The ZNCC calculation process is defined using mathematical formulas, which calculate the similarity value at the center position u of the window. Furthermore, a multi-scale weighted fusion method (parameter: scale weight factor) is used. The weighted summation of similarity matrices at each scale is used to generate a cross-modal similarity heatmap H and obtain the local matching intensity distribution across the entire map. Smoothing filtering is used (parameters: Gaussian kernel standard deviation). This suppresses high-frequency noise responses in the similarity heatmap and improves the spatial continuity of local matching estimation. Through the above-mentioned cross-correlation calculation and fusion processing, the results of the previous step are transformed into a cross-modal similarity heatmap that can characterize the matching degree of optical and acoustic modes in each local region, thus achieving the expected technical effect of providing a high-precision matching metric for the scale response map generation step. For example, in the underwater bridge pier crack detection scenario, the input optical and acoustic feature maps are 512×512 pixels in size. A sliding window size of w=15 pixels, a step size of s=5 pixels, and a correlation coefficient calculation radius of r=7 pixels are selected. ZNCC is applied to calculate the similarity value at the center position of each window. The standard deviation of the Gaussian smoothing filter is then used. The value was set to 2.0 to suppress noise. Calculations were performed for three scale levels (512×512, 256×256, 128×128), and scale weight factors were set for each. =0.5, =0.3, =0.2. The cross-modal similarity heatmap generated after weighted fusion shows a significantly high response value in the crack region. The high-value region has good continuity and is highly consistent with the actual spatial distribution of the crack, indicating that the cross-correlation calculation in this step effectively improves the local sensitivity and overall spatial consistency of cross-modal matching. S4.3: Perform peak detection and normalization on the cross-modal similarity heatmap to extract the local optimal matching point pairs and generate an initial scale response map as the basis for scale offset vector estimation; S4.4: The initial scale response map is upgraded based on a differentiable upsampling algorithm to restore the pixel-level scale offset vector field and ensure high spatial resolution and continuity of scale estimation. S4.5: Normalize and transform the pixel-level scale offset vector field to generate local scale field parameters that can be used for subsequent non-rigid deformation modeling, thus realizing spatial alignment preparation for cross-modal images. For the pixel-level scale offset vector field recovered by differentiable upsampling, a vector normalization algorithm is used (parameter: normalization norm is set to L2 norm) to unify the length of each pixel-level scale offset vector to a standard value, thereby eliminating the scale inconsistency between different regions caused by resolution upsampling operations. Furthermore, by using a coordinate system transformation algorithm (parameters: rotation matrix R and translation vector T are solved based on the imaging device's extrinsic parameters), the scale offset vector field is mapped from the acquisition device's coordinate system to a unified world coordinate system, and scale distribution data under the unified coordinate system is obtained. Furthermore, by using a scale field parameter normalization model (parameters: reference resolution is the resolution of the preprocessed optical image, and reference scale factor is the peak value of the acoustic image feature response), dynamic matching of scale parameters between different modalities is achieved, and cross-modal equivalent local scale parameter matrices are generated. Furthermore, by using an interpolation reconstruction method (parameters: spline interpolation order is third, and the interpolation kernel function is selected as bicubic B-spline), continuous sampling of the scale field under a unified coordinate system is achieved, and a local scale parameter distribution map without discontinuities is obtained. By converting data types and quantizing parameters, the continuous local scale parameter matrix is ​​transformed into rasterized scale field data that can be directly called for non-rigid deformation modeling, thus fulfilling the preparatory conditions for cross-modal image spatial alignment. For example, in a subsea pipeline crack detection task, 640×480 resolution optical-acoustic registration samples are processed, and the scale offset vector field recovered by differentiable upsampling is set to the L2 norm during the normalization stage. Ensure all vectors have the same length. The rotation matrix R used in the coordinate transformation is... The translation vector T is This parameter is obtained from the device's extrinsic parameter calibration. During the scale field parameter standardization process, the reference resolution is set to 640×480, and the reference scale factor is taken as the peak value of the acoustic modal similarity. The resulting local scale parameter matrix, after reconstruction using bicubic B-spline interpolation, shows a significant improvement in continuity and a substantial increase in breakpoint elimination rate. The output rasterized scale field data is directly used in the manifold regularization step, resulting in a significant improvement in spatial alignment accuracy in crack regions during subsequent non-rigid deformable mesh generation.

[0015] Step S5: Introducing a manifold regularization term to smooth the scale offset vector field ensures the continuity of the transformation gradient in adjacent regions, avoiding breaks or distortions, thereby generating a continuous and stable local scale field. Specifically, this includes: S5.1: The pixel-scale offset vector field generated by cross-modal cross-correlation operation is divided into local neighborhoods, and the neighborhood vector distribution of each pixel is extracted based on the sliding window mechanism to build the basis for the calculation of local transformation gradient; S5.2: Calculate the scale offset gradient difference between adjacent pixels based on the local neighborhood vector distribution, quantify the gradient discontinuity using the Euclidean distance metric, and generate a gradient inconsistency matrix to identify local regions with the risk of breakage or distortion. S5.3: Construct a manifold regularization term, which is designed based on the local gradient inconsistency matrix. The Laplace-Beltrami operator is used to model the spatial continuity constraint, and a manifold smoothing constraint is applied to the scale offset vector field to suppress discontinuous perturbations in non-rigid deformation. S5.4: Embed the manifold regularization term into the optimization objective function of the scale offset vector field, and iteratively optimize the vector field through the conjugate gradient descent algorithm so that the optimized scale offset vector field can satisfy the local continuity requirement of spatial transformation while maintaining cross-modal matching capability. For constraints defined by the manifold regularization term, an objective function construction method (parameters: scale offset vector field (V), regularization parameter λ, matching error term weight α) is used to jointly model the smoothness of the vector field and the cross-modal matching capability. Furthermore, by introducing a summation quadratic form representation, the matching error term is... With manifold smoothing term Combined into a global optimization objective function Its formal definition is: in, This is the sum of squared errors arising from the difference between the scale offset field and cross-modal feature matching. Let be the square integral of the second derivative of the vector field calculated by the Laplace-Beltrami operator; Furthermore, through discretization, the constraints in continuous space are transformed into discrete gradient calculation expressions within the pixel neighborhood, defining the local smoothing error. for: in, Represents the total number of pixel neighbors. This represents the scale offset gradient difference of the i-th neighborhood center pixel; Furthermore, the conjugate gradient descent algorithm is employed (parameter: initial value vector field). The algorithm iteratively optimizes the objective function J by setting the maximum number of iterations K and the convergence threshold ε, and obtains the optimized scale offset vector field that simultaneously satisfies the smoothness constraint and matching accuracy. ; Furthermore, by calculating the conjugate of the gradient direction of each iteration with the previous direction, we can ensure that the search path does not backtrack redundantly in the high-dimensional space, improve the optimization convergence rate, and generate iterative convergence curves as optimization monitoring data. Through the above conjugate gradient descent optimization, the manifold regularization term is effectively embedded into the objective function of the scale offset vector field, resulting in an optimized result that maintains both cross-modal matching capability and meets the requirement of local continuity of spatial transformation, thereby improving the stability of non-rigid geometric transformation in underwater optical and acoustic image registration. For example, in the task of detecting cracks in marine bridge piers, the initial pixel-scale offset vector field V is generated by differentiable cross-correlation and upsampling, with a resolution of 1024×1024, and the unit is the pixel scale ratio. The matching error term weight α is set to 0.7, the smoothing regularization parameter λ is set to 0.3, and the initial value vector field... Data is loaded directly from the S4.5 output. The objective function is defined using the J expression described above, with a maximum global iteration count K of 200 and a convergence threshold ε of 1 × 10⁻⁶. -5 During the calculation, the local smoothing error is calculated using a 3×3 sliding window neighborhood. The gradient difference vector of each neighborhood is evaluated using the Euclidean norm. Iterative optimization converges on the 143rd iteration, yielding the optimized vector field. The maximum value of the local gradient discontinuity matrix is ​​less than 0.002, while the cross-modal matching error decreases by about 35%. The optimized scale field output is used for the generation of non-rigid deformable meshes in S6, and shows significant edge contour fitting in the registration visualization of the crack edge region, greatly improving the multimodal fusion stability in complex underwater dynamic backgrounds; S5.5: Perform local consistency verification on the optimized scale offset vector field, evaluate the manifold constraint effect based on the edge structure alignment index, and output a local scale field that meets the requirements of continuity and smoothness as the input parameter for non-rigid deformable mesh encoding; For the scale offset vector field after iterative optimization by the manifold regularization term, a method based on local structure consistency verification is adopted (parameters: window size is set to 5×5 pixels, edge detection scale threshold is set to 0.15) to achieve quantitative evaluation of the smoothness of geometric transformation in the neighborhood of each pixel; Furthermore, using an improved Canny edge detection algorithm (parameters: Gaussian smoothing σ = 1.2, and dual thresholds of 0.1 and 0.3 respectively), edge pixel sets of cracks and structural contours are extracted from the optimized optical and acoustic images, and edge binary masks are obtained. Furthermore, by using the edge reprojection alignment calculation method (parameter: a hybrid index of Hausdorff distance and edge point coincidence rate), the spatial alignment degree of optical and acoustic edge sets is quantified, and a local consistency metric matrix is ​​obtained, where each element corresponds to the matching accuracy of a certain neighborhood in the image space. Furthermore, the global edge alignment E is calculated using the following formula: in, The distance function is the coordinates of the edge points. and These are the edge point coordinate sets of the optical and acoustic images, respectively. This represents the total number of edge points; Furthermore, by comparing the local consistency metric of each neighborhood with the global alignment E, low consistency regions are filtered out, and the scale field correction module is called to replace them with the highest consistency vector in the neighborhood, thereby further enhancing the local geometric smoothness. By verifying local consistency and analyzing edge alignment, the optimized scale offset vector field from the previous step is transformed into a local scale field that satisfies the continuity and smoothness indices, thus achieving stable input conditions for non-rigid deformable mesh encoding. For example, in a scenario involving monitoring cracks in a seabed steel structure, the optical image resolution is set to 1920×1080 pixels, and the acoustic image resolution is set to 1024×768 pixels. A local consistency validator with a window size of 5×5 pixels is used. The Gaussian kernel σ of the Canny algorithm is 1.2, the low threshold is set to 0.1, and the high threshold is set to 0.3. The number of edge pixels extracted is 15200 for the optical image and 14800 for the acoustic image. The global alignment E is obtained by mixing the Hausdorff distance (maximum value 12 pixels) with the edge point overlap rate (average value 0.82), and the result is calculated using the formula... Significantly lower than the unoptimized state. After local consistency screening, the proportion of low consistency regions decreased from 8.3% to 3.1%. The final output local scale field maintained a continuous and smooth transformation gradient in the crack boundary region, ensuring the accurate spatial mapping effect of the crack structure in the subsequent thin plate spline interpolation stage.

[0016] Step S6: The local scale field and affine transformation parameters are jointly encoded into a non-rigid deformable mesh, and a thin-plate spline interpolation model is used to perform spatial coordinate mapping on the optical and acoustic images. Specifically, this includes: S6.1: Based on the preprocessed local scale field and affine transformation parameters, a joint parameter encoding vector is constructed to characterize the non-rigid deformation characteristics in the image space and provide parameter input for subsequent deformation mesh generation; S6.2: Generate a set of non-rigid deformable mesh control points based on the joint parameter encoding vector. The control points are distributed on regular grid points in the image space and are used for spatial mapping calculation of the subsequent thin plate spline interpolation model. S6.3: Based on the thin plate spline interpolation model, spatial mapping modeling is performed on the set of control points of non-rigid deformable mesh to generate pixel-level spatial coordinate transformation functions, so as to realize displacement prediction of any point in image space; S6.4: Input the optical image and acoustic image into the spatial coordinate transformation function respectively, and perform non-rigid mapping transformation of pixel coordinates to obtain the preliminary aligned cross-modal image spatial coordinates; The input terminal of the spatial coordinate transformation function generated by the thin plate spline interpolation model receives the pixel coordinate matrix of the optical image and the pixel coordinate matrix of the acoustic image as input data for non-rigid mapping calculation. A spatial coordinate transformation function evaluation method (parameters: local scale field vector, affine transformation matrix, interpolation weight coefficients) is used to achieve non-rigid coordinate relocation of the spatial position of each pixel in the optical image and obtain the initial displacement vector field of the optical image in the cross-modal space. Furthermore, by using the same spatial coordinate transformation function evaluation method (parameters: local scale field vector, affine transformation matrix, interpolation weight coefficients), non-rigid coordinate relocation of the spatial position of each pixel in the acoustic image is achieved, and the initial displacement vector field of the acoustic image in the cross-modal space is obtained; Furthermore, the pixel coordinate difference is calculated using the coordinate relocation results. Based on the transformed position difference of corresponding points in the optical and acoustic images, a cross-modal spatial coordinate alignment deviation matrix is ​​generated to quantify the spatial deviation of the initial registration effect. Furthermore, a coordinate field continuity verification method (parameters: gradient threshold, adjacent pixel window size) is used to perform local gradient detection on the displacement vector field of optical and acoustic images, screen out abnormal abrupt regions and fill them in, to ensure that the initially aligned spatial coordinates meet the continuity requirements; By using a non-rigid mapping transformation process, the local scale field and affine transformation results generated in the previous step are transformed into preliminary cross-modal spatial coordinate data of optical and acoustic images, thereby achieving the preliminary geometric alignment effect of cross-modal images in dynamic underwater environments. For example, in the scenario of detecting cracks in underwater bridge piers, the optical imaging module acquires an image matrix with a resolution of 1920×1080, and the acoustic imaging module acquires an echo intensity matrix with a resolution of 640×480. The local scale field parameters output after step S5 are a vector field of 1920×1080×2, and the affine transformation parameter matrix is ​​a 3×3 homography matrix. The optical image pixel coordinate matrix is ​​input into the spatial coordinate transformation function generated by the thin-plate spline to obtain the displacement prediction result for each pixel. For example, the center point of a crack region is predicted by... Mapped to The same mapping is performed on the acoustic image, and the center point of the crack region is determined by... Mapped to The difference calculation of the coordinate field after the two-modal mapping showed that the average deviation near the crack edge was significantly reduced in space. The edge continuity detection results showed that the gradient change in the region was smooth, achieving the initial expected effect of spatial alignment, and providing high-quality initial coordinates for the bilinear sampling of S6.5 and subsequent structural consistency feedback optimization. S6.5: Based on the pixel-level spatial coordinate transformation results, a differentiable bilinear sampling method is used to resample the image to output a registered image pair with continuous spatial alignment characteristics, providing initial mapping results for subsequent structural consistency feedback optimization.

[0017] Step S7: Design an iterative optimization mechanism based on structural consistency feedback, dynamically adjusting the scale field weight distribution using the edge alignment degree of the registered image, focusing on enhancing the alignment accuracy of key structural regions such as cracks. Specifically, this includes: S7.1: Perform edge detection processing on the registered optical and acoustic image pairs to extract the geometric boundary information of the crack region in the image, which serves as the basic input for structural consistency feedback; S7.2: Based on the extracted crack geometric boundary information, calculate the local alignment error of the corresponding edges between the optical and acoustic images to generate a pixel-level error distribution map of the edge alignment degree; S7.3: Based on the error distribution map of edge alignment, a Gaussian kernel function is used to perform weighted enhancement processing on areas with large errors to generate a feedback weight distribution map of the local scale field, thereby strengthening the priority of key structural areas such as cracks. Based on the edge alignment error distribution map data output by S7.2, a two-dimensional Gaussian kernel weighting method (parameters: kernel standard deviation σ, kernel size k×k) is used to realize the spatial weighting enhancement function for areas with significant errors. Furthermore, a Gaussian kernel function is used to perform a convolution operation on the error distribution map (parameter: σ is adaptively set according to the crack edge width characteristics) to achieve smooth diffusion of error values ​​in the local neighborhood and obtain a local response matrix containing weighted coefficients; Furthermore, a normalization processing algorithm (parameter: minimum-maximum normalization range [0,1]) is used to perform amplitude mapping on the convolution results, realize the relative calibration of weighting coefficients in different regions, and generate a normalized weighting coefficient field for feedback calculation; Furthermore, based on the crack region edge detection mask, a mask product operation is performed on the normalized weighted coefficient field to achieve key enhancement of the weighted coefficients within the crack region and generate a crack priority coefficient field. By using the aforementioned local weighted enhancement processing method, areas with large edge alignment errors are transformed into high-weight feedback coefficient data, thereby achieving the technical effect of prioritizing the adjustment of cracks and their adjacent structural regions in the subsequent scale field optimization process. For example, in the underwater crack detection scenario of a marine bridge pier, the edge alignment error distribution map has a resolution of 1920×1080, and the crack mask is generated by Canny edge detection. A two-dimensional Gaussian kernel weighting process is applied to this error distribution map, with the kernel standard deviation σ set to 3.0 and the kernel size to 11×11. After convolution, a local response matrix is ​​obtained. Linear mapping is used for normalization to compress the response matrix amplitude to the [0,1] interval. After the crack mask multiplication operation, a crack priority coefficient field is obtained, where the average coefficient in the crack region is increased to 0.92, while the average coefficient in the non-crack region remains at 0.35. The feedback weight for each pixel is calculated using the following weighting formula. : in, Let be the Euclidean distance between the pixel and the location of the error peak. The standard deviation of the Gaussian kernel is used. The application of this coefficient field in the subsequent scale field optimization process significantly improves the registration accuracy of crack edges, and reduces the cross-modal alignment misalignment distance of cracks to within 1.5 pixels, meeting the engineering requirements of iterative optimization for structural consistency. S7.4: Multiply the feedback weight distribution map with the current scale field pixel by pixel to adjust the scale transformation parameters of the local region, thereby achieving dynamic optimization of the non-rigid deformation mesh; S7.5: Based on the optimized non-rigid deformable mesh, the spatial coordinate mapping and image resampling operations are re-executed to obtain updated registered image pairs, which are then fed back to the edge detection module to form an iterative closed-loop optimization mechanism driven by structural consistency. Based on the optimized non-rigid deformable mesh parameters, a spatial coordinate mapping function (parameter source: control point coordinates of thin plate spline interpolation model and local scale field correction vector) is used to realize the non-rigid coordinate transformation of pixel positions of optical and acoustic images, so as to generate an updated cross-modal spatial alignment coordinate matrix. Furthermore, by using a differentiable bilinear sampling algorithm (parameters: updated spatial coordinate matrix, original image pixel intensity matrix), the optical and acoustic images are resampled, and a dataset of registered image pairs with refined matching edges is obtained. Furthermore, the geometric boundary line element information of the crack region is re-extracted from the updated registered image pair using the edge detection operator (parameter: Sobel operator kernel size is 3×3), and a new round of structural consistency measurement input data is generated. Furthermore, the re-extracted geometric boundary information is input into the edge alignment metric model (parameter: pixel-level Euclidean distance threshold set to 0.5 pixels) to calculate the matching error vector and generate an error distribution matrix for the feedback loop; By using a feedback loop to drive parameter update mechanism, the new edge matching error matrix is ​​fed back to the local scale field weight adjustment module to realize dynamic parameter correction of the crack priority region, thereby forming an iterative closed-loop optimization link driven by structural consistency. Through the above iterative optimization algorithm, the non-rigid deformable mesh result of the previous step is transformed into updated registration image data with continuous edge structure alignment characteristics, so as to achieve the effect of cross-modal high-precision spatial alignment technology for key structural regions such as cracks. For example, in the scenario of detecting cracks in a subsea pipeline, an optical imaging device acquires a texture image with a resolution of 1920×1080, and an acoustic imaging device acquires an echo intensity image with a resolution of 512×512. After processing in a previous step, an optimized non-rigid deformable mesh is generated, and iterative optimization is performed in this sub-step. The non-rigid coordinate mapping function is generated by thin-plate spline interpolation, the control point spacing is set to 50 pixels, and the local scale field weight enhancement coefficient in the crack region is 1.5. Through differentiable bilinear sampling, the difference in edge orientation between the resampled optical image and acoustic image within the crack region is controlled within a certain range. Within. Edge detection uses a third-order Sobel operator, with a matching error of [missing information]. The calculation uses the Euclidean distance formula: in, and These are the horizontal and vertical components of the coordinates of the edge points in the optical image, respectively. and These represent the horizontal and vertical components of the coordinates of the corresponding edge points in the acoustic image. After three rounds of iterative closed-loop execution, the edge error in the crack region converges significantly, and the updated registered image shows a significant improvement in the structural consistency evaluation index. The adaptability is stable, and it can be directly entered into the subsequent loss calculation module for training and optimization.

[0018] Step S8: Image resampling is performed using differentiable bilinear sampling, outputting registered optical and acoustic image pairs. Weighted values ​​of structural similarity loss, gradient consistency loss, and manifold smoothness loss are calculated for model training and parameter optimization. Specifically, this includes: S8.1: Based on the image registration transformation parameters, perform differentiable bilinear resampling processing on the original optical image and acoustic image to generate spatially aligned registered image pairs; S8.2: Perform pixel-level structural similarity measurement on the resampled optical and acoustic images. Calculate the structural similarity loss based on the mean, variance, and covariance of brightness within the local window to quantify the consistency deviation of the image texture structure. S8.3: Perform gradient magnitude and direction consistency analysis on the registered image pairs, calculate the gradient consistency loss based on the Euclidean distance in the image gradient domain, and evaluate the spatial alignment of image edge structures. S8.4: Based on the manifold regularization constraint of the local scale field, calculate the second derivative continuity measure of the scale offset vector field to obtain the manifold smoothing loss and ensure the spatial continuity and physical rationality of the registration transformation. S8.5: The structural similarity loss, gradient consistency loss and manifold smoothing loss are weighted and fused to generate a joint optimization objective function. The registration network parameters are iteratively optimized based on the gradient backpropagation mechanism to improve the overall registration accuracy and algorithm convergence efficiency.

[0019] The technical solution of the present invention has been described above with reference to the preferred embodiments shown in the accompanying drawings. However, it will be readily understood by those skilled in the art that the scope of protection of the present invention is obviously not limited to these specific embodiments. Without departing from the principles of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will all fall within the scope of protection of the present invention.

[0020] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and rules of the present invention should be included within the scope of protection of the present invention.

Claims

1. An optical and acoustic image registration method for detecting cracks in underwater structures, characterized in that, Includes the following steps: S1: Simultaneously acquire optical and acoustic image data pairs of underwater structures under the same field of view, and use them as input samples for multimodal image registration processing; S2: Denoising and geometric distortion correction preprocessing are performed on the acquired optical and acoustic images, respectively; S3: Construct a dual-path multi-scale feature extraction network, in which the optical branch uses dilated convolution to enhance the receptive field to preserve texture details, and the acoustic branch introduces a frequency domain sensing module to highlight echo structure features, and promotes semantic alignment between modalities by sharing part of the network weights. S4: Generate a local scale response map at each scale layer, calculate a cross-modal similarity heatmap through cross-correlation operation, and regress it into a pixel-level scale offset vector field using differentiable upsampling; S5: Introduce a manifold regularization term to smooth the scale offset vector field and generate a local scale field; S6: The local scale field and affine transformation parameters are jointly encoded into a non-rigid deformable mesh, and a thin plate spline interpolation model is used to perform spatial coordinate mapping on the optical and acoustic images; S7: Design an iterative optimization mechanism based on structural consistency feedback, and dynamically adjust the scale field weight distribution by utilizing the edge alignment degree of the registered image; S8: Performs image resampling operation through differentiable bilinear sampling, outputs registered optical and acoustic image pairs, and calculates the weighted values ​​of structural similarity loss, gradient consistency loss and manifold smoothness loss for model training and parameter optimization.

2. The optical and acoustic image registration method for detecting cracks in underwater structures according to claim 1, characterized in that, Step S1 specifically includes: Based on the deployment location and field of view parameters of the underwater imaging system, the spatial layout of the optical imaging equipment and the acoustic imaging equipment is planned to obtain the initial aligned imaging geometry. The optical imaging device and the acoustic imaging device are time-synchronized and calibrated, and the image acquisition time is synchronized at the millisecond level based on an external trigger signal or a timestamp marking mechanism. During underwater structure inspection, the optical imaging device is controlled to acquire optical images, and the acoustic imaging device is controlled to acquire acoustic images to obtain multimodal image pairs of the same structural region, forming an original image dataset. Metadata annotation is performed on the optical image and the acoustic image, and an image pairing index table is generated based on the pose information and timestamp information of the imaging device; Based on the image pairing index table, the optical image and the acoustic image are preprocessed to match spatial resolution, thereby unifying the image resolution.

3. The optical and acoustic image registration method for detecting cracks in underwater structures according to claim 2, characterized in that, The optical image is a high-resolution texture image, and the acoustic image is an echo intensity image.

4. The optical and acoustic image registration method for detecting cracks in underwater structures according to claim 1, characterized in that, Step S2 specifically includes: The original optical image and the acoustic image are denoised by nonlocal mean filtering respectively to obtain the denoised optical image and the denoised acoustic image. Based on the underwater optical imaging model, the denoised optical image is subjected to refractive distortion correction processing to obtain a refractive corrected optical image. The denoised acoustic image is subjected to beam offset correction processing using acoustic propagation path modeling to obtain a beam-corrected acoustic image. Based on the device attitude sensor data, motion blur compensation processing is performed on the refraction-corrected optical image and the sound beam-corrected acoustic image respectively to obtain attitude-compensated optical and acoustic images; The optical and acoustic images after attitude compensation are subjected to gamma correction and dynamic range compression to obtain high-quality image pairs.

5. The optical and acoustic image registration method for detecting cracks in underwater structures according to claim 1, characterized in that, Step S3 specifically includes: Perform dilated convolution operation on the input optical image data to obtain multi-scale dilated convolution feature maps; Based on the multi-scale dilated convolutional feature map, a multi-scale pyramid structure is constructed to extract and fuse optical image features at different scales in a hierarchical manner, generating multi-level optical feature representations. The input acoustic image data is subjected to a fast Fourier transform operation to obtain its frequency domain features. Based on the frequency domain energy distribution, a frequency domain sensing module is designed to extract the significant features of the echo structure in the acoustic image and generate a frequency domain enhanced feature map. Based on the frequency domain enhanced feature map, a multi-scale frequency domain pyramid structure is constructed to perform layer-by-layer modeling and fusion of acoustic image features of different frequency components, generating multi-level acoustic feature representations. Based on the multi-level optical feature representation and the multi-level acoustic feature representation, a cross-modal feature fusion layer with shared weights is constructed to jointly optimize the high-level features of the optical and acoustic branches, generating a unified cross-modal semantic feature space. A channel attention mechanism is performed on the cross-modal semantic feature space to calculate the importance weight of each channel feature, and the feature maps are weighted and fused to generate a joint feature representation of the channel-enhanced features; Based on the joint feature representation with channel enhancement, a multi-scale feature pyramid fusion module is constructed to perform cross-scale fusion and upsampling operations on features at different levels, generating the final multi-scale dual-path feature output.

6. The optical and acoustic image registration method for detecting cracks in underwater structures according to claim 5, characterized in that, Step S3 further includes the optical image feature extraction branch using stacked dilated convolutions with dilation rates of 2, 3, and 5, as well as batch normalization and ReLU activation for extraction, and the acoustic branch using fast Fourier transform, multi-subband energy decomposition, and frequency domain enhancement for extraction. The two are then fused by shared weight convolution and mutual information enhancement to generate a cross-modal unified semantic feature space.

7. The optical and acoustic image registration method for detecting cracks in underwater structures according to claim 1, characterized in that, Step S4 specifically includes: The multi-scale feature maps output by the dual-path feature extraction network are processed in layers to construct optical and acoustic modal feature representations at each scale level; Based on the optical and acoustic feature maps at each scale level, local cross-correlation operations are performed to calculate cross-modal similarity heatmaps. Peak detection and normalization are performed on the cross-modal similarity heatmap to extract local optimal matching point pairs and generate an initial scale response map. The initial scale response map is subjected to resolution enhancement processing based on a differentiable upsampling algorithm to restore the pixel-level scale offset vector field. The pixel-level scale offset vector field is normalized and subjected to coordinate transformation to generate local scale field parameters.

8. The optical and acoustic image registration method for detecting cracks in underwater structures according to claim 7, characterized in that, The cross-modal similarity heatmap is calculated using a zero-mean normalized cross-correlation algorithm and then smoothed by Gaussian. The pixel-level scale offset vector field is increased in dimensionality through differentiable upsampling, and after vector normalization, it is combined with rotation and translation coordinate transformation to generate a cross-modal equivalent local scale parameter matrix.

9. The optical and acoustic image registration method for detecting cracks in underwater structures according to claim 1, characterized in that, Step S5 specifically includes: The pixel-scale offset vector field generated by cross-modal cross-correlation operation is divided into local neighborhoods, and the local neighborhood vector distribution of each pixel is extracted based on the sliding window mechanism. Based on the local neighborhood vector distribution, the scale offset gradient difference between adjacent pixels is calculated, and the gradient discontinuity is quantified by Euclidean distance metric to generate a gradient inconsistency matrix. A manifold regularization term is constructed, which is designed based on the local gradient inconsistency matrix. The Laplace-Beltrami operator is used to model the spatial continuity constraint, and a manifold smoothing constraint is applied to the scale offset vector field. The manifold regularization term is embedded into the optimization objective function of the scale offset vector field, and the vector field is iteratively optimized using the conjugate gradient descent algorithm. Local consistency verification is performed on the optimized scale offset vector field, the manifold constraint effect is evaluated based on the edge structure alignment index, and the output is a local scale field that meets the requirements of continuity and smoothness.

10. The optical and acoustic image registration method for detecting cracks in underwater structures according to claim 9, characterized in that, The local consistency verification supports a window size of 5×5 pixels, and the edge detection method is the improved Canny edge detection algorithm. The edge structure alignment is quantized using a hybrid approach of Hausdorff distance and overlap rate.

Citation Information

Cited By

  • Crack early warning method driven by multi-modal data in repairing and remanufacturing process

    CN122150417A

  • Image dynamic range adaptive correction method in new energy automobile visual perception

    CN122175844A

  • Image dynamic range adaptive correction method in new energy vehicle visual perception

    CN122175844B