A submarine cable image splicing method based on dual-domain decoupling enhancement and multi-level fault-tolerant matching

Abstract

The application provides a submarine cable image splicing method based on dual-domain decoupling enhancement and multi-level fault-tolerant matching. The method comprises the following steps: obtaining a pair of adjacent frame submarine cable images, extracting original color channel information of each frame image, and performing channel enhancement processing on each frame image to obtain a channel enhanced image; processing the channel enhanced image based on a dual-domain decoupling enhancement network to obtain a final enhanced image; performing feature point matching on the final enhanced image pair based on a multi-level fault-tolerant matching network to obtain a pair of effective feature points matched with each other; calculating a homography transformation matrix of the final enhanced image pair based on the pair of effective feature points, performing image splicing on the final enhanced image pair according to the homography transformation matrix, and performing color enhancement processing on the spliced image based on the original color channel information to obtain a final spliced image. The technical scheme of the application effectively suppresses color distortion while improving image contrast and detail clarity.

Description

Technical Field

[0001] This invention relates to the field of underwater target detection technology, specifically to a submarine cable image stitching method based on dual-domain decoupling enhancement and multi-level fault-tolerant matching. Background Technology

[0002] In the daily inspection and maintenance of submarine cables, marine resource surveys and underwater pipelines, due to the severe attenuation of underwater ambient light, the different absorption rates of water for different wavelengths of light and the interference of suspended particulate matter, underwater inspection images generally have problems such as color cast (e.g., bluish or greenish), significantly reduced contrast, and sparse texture.

[0003] Traditional image stitching and enhancement techniques face the following core technical challenges when processing such underwater cable images:

[0004] First, there is a contradiction between feature extraction and color fidelity. To improve the success rate of image registration, it is usually necessary to enhance the contrast of the image, but this will destroy the true physical color of the target surface (such as attachments, corrosion), leading to serious deviations in downstream defect classification;

[0005] Second, there is insufficient utilization of frequency information and a lack of inter-domain complementarity. The underwater environment has significantly different effects on the amplitude and phase components in the frequency domain. Existing methods lack explicit modeling of key information in both domains and effective cross-domain fusion mechanisms, making it difficult to balance color fidelity and detail realism.

[0006] Third, the matching algorithms lack engineering robustness. When faced with solid-color sand beds or weakly textured areas, traditional algorithms such as SIFT and ORB are prone to failure, while deep learning models based on Transformers (such as LoFTR) often fail to infer due to limited computing power or sudden ambiguity when deployed on edge devices, causing the matching pipeline to collapse. Summary of the Invention

[0007] In view of this, this application proposes a submarine cable image stitching method based on dual-domain decoupling enhancement and multi-level fault-tolerant matching, in order to at least solve the problems of insufficient utilization of frequency domain information, difficulty in balancing structural stability and detail restoration, and easy stitching failure under extreme working conditions.

[0008] Specifically, this application is implemented through the following technical solution:

[0009] According to a first aspect of the embodiments of this specification, a method for stitching submarine cable images based on dual-domain decoupling enhancement and multi-level fault-tolerant matching is provided, comprising the following steps:

[0010] Step S1: Obtain submarine cable image pairs of adjacent frames, extract the original color channel information of each frame in the submarine cable image pair, and perform channel enhancement processing on each frame to obtain a channel-enhanced image.

[0011] Step S2: The channel enhancement image is processed based on a preset dual-domain decoupling enhancement network to obtain the final enhanced image; the dual-domain decoupling enhancement network includes a feature extraction branch, a dual-domain feature decoupling branch, and a feature fusion branch; wherein:

[0012] The feature extraction branch is configured to perform multi-scale feature extraction on each frame of channel-enhanced image to obtain a high-resolution spatial feature map and a low-resolution spatial feature map.

[0013] The dual-domain feature decoupling branch is configured to extract the spatial domain enhancement features and frequency domain correlation features of the low-resolution spatial feature map, and fuse the spatial domain enhancement features and frequency domain correlation features to obtain joint representation features. The phase and amplitude components of the joint representation features and the phase and amplitude components of the high-resolution spatial feature map are fused accordingly. The fused phase components and fused amplitude components are recombined and transformed to obtain frequency domain enhancement features.

[0014] The feature fusion branch is configured to perform cross-domain fusion of the frequency domain enhancement features and the spatial domain enhancement features based on an attention strategy, and to perform multi-scale reconstruction based on the joint representation features and the cross-domain fused features to obtain the final enhanced image;

[0015] Step S3: Based on the multi-level fault-tolerant matching network, feature point matching is performed on the final enhanced image pair to obtain mutually matching effective feature point pairs;

[0016] Step S4: Calculate the homography transformation matrix of the final enhanced image pair based on the effective feature point pairs, perform image stitching on the final enhanced image pair according to the homography transformation matrix, and perform color enhancement processing on the stitched image based on the original color channel information to obtain the final stitched image.

[0017] According to a second aspect of the embodiments of this specification, an electronic device is provided, including a processor; and a computer-readable storage medium storing computer program instructions that, when executed by the processor, cause the processor to perform the method described in the first aspect.

[0018] The embodiments of this application have at least the following technical effects:

[0019] (1) In this embodiment, while extracting the original color channel information, the image is also processed by channel enhancement, and the stitched image is processed by color enhancement based on the original color channel information. Through the dual-stream decoupling design, the feature enhancement stream is dedicated to improving the image quality to support subsequent feature extraction and registration, while the original color stream retains the real physical color information throughout the process. When stitching, the color is restored based on the original color stream, which not only ensures the accuracy of geometric registration, but also avoids color distortion caused by image enhancement, and ensures the accuracy of downstream defect detection.

[0020] (2) The embodiments of this application use a specially designed dual-domain decoupling enhancement network to enhance the image. Since the phase component preserves the structural contour of the image and the amplitude component reflects the color and illumination information, by enhancing and fusing the two respectively, the structural integrity and color authenticity of the image can be effectively restored. Furthermore, the spatial domain features provide local texture details and the frequency domain features provide global structural information. Cross-domain attention fusion makes the two complementary, and the output image is both clear in detail and structurally stable.

[0021] (3) The robustness of image matching is improved by using a multi-level fault-tolerant matching network. For example, when the computing power is sufficient and the image quality is good, high-precision dense matching is provided, while when the computing power is limited, the precision is automatically reduced. Thus, this embodiment effectively restores the clarity and color fidelity of submarine cable images through dual-domain decoupling enhancement, ensures the success rate of splicing under extreme working conditions through multi-level fault-tolerant matching, and achieves the synergistic optimization of geometric registration and color fidelity through dual-stream decoupling design. Finally, a high-fidelity, high-quality underwater submarine cable splicing image is output, providing reliable technical support for applications such as submarine cable inspection and underwater target detection. Attached Figure Description

[0022] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the accompanying drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Some specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings in an exemplary and non-limiting manner. The same reference numerals in the drawings indicate the same or similar parts or components. Those skilled in the art should understand that these drawings are not necessarily drawn to scale. In the drawings:

[0023] Figure 1 This is a flowchart illustrating an exemplary embodiment of the present application of a submarine cable image stitching method based on dual-domain decoupling enhancement and multi-level fault-tolerant matching;

[0024] Figure 2 This is a schematic diagram of a system framework for submarine cable image stitching, as shown in an exemplary embodiment of this application;

[0025] Figure 3This is a schematic diagram illustrating a multi-level fault-tolerant matching process in an exemplary embodiment of this application;

[0026] Figure 4 This is a schematic diagram illustrating the feature point matching result of adjacent frame images according to an exemplary embodiment of this application;

[0027] Figure 5 This is a schematic diagram illustrating the stitching effect of a submarine cable image according to an exemplary embodiment of this application;

[0028] Figure 6 This is a structural block diagram of an electronic device illustrated in an exemplary embodiment of this application;

[0029] Figure 7 This is a structural block diagram of a submarine cable image stitching device based on dual-domain decoupling enhancement and multi-level fault-tolerant matching, as illustrated in an exemplary embodiment of this application. Detailed Implementation

[0030] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0031] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0032] This application provides a method for stitching submarine cable images based on dual-domain decoupling enhancement and multi-level fault-tolerant matching. Figure 1 This is a schematic flowchart illustrating an exemplary embodiment of a submarine cable image stitching method based on dual-domain decoupling enhancement and multi-level fault-tolerant matching. Figure 2 This is a schematic diagram of the system architecture of an embodiment of this application, as shown below. Figure 1 and Figure 2 As shown, the submarine cable image stitching method includes at least the following steps:

[0033] Step S1: Obtain submarine cable image pairs of adjacent frames, extract the original color channel information of each frame in the submarine cable image pair, and perform channel enhancement processing on each frame to obtain a channel-enhanced image.

[0034] In this embodiment, when obtaining pairs of submarine cable images in adjacent frames, the pairs of submarine cable images can be preprocessed, for example, scaled to a size of 640×640 pixels, to meet the fixed-size input requirements of the subsequent depth model, while also taking into account computational efficiency.

[0035] Step S2: The channel enhancement image is processed based on a preset dual-domain decoupling enhancement network to obtain the final enhancement image; the dual-domain decoupling enhancement network includes a feature extraction branch, a dual-domain feature decoupling branch, and a feature fusion branch.

[0036] The feature extraction branch is configured to perform multi-scale feature extraction on each frame of channel-enhanced image to obtain a high-resolution spatial feature map and a low-resolution spatial feature map.

[0037] The dual-domain feature decoupling branch is configured to extract the spatial domain enhancement features and frequency domain correlation features of the low-resolution spatial feature map, and fuse the spatial domain enhancement features and frequency domain correlation features to obtain joint representation features. The phase and amplitude components of the joint representation features and the phase and amplitude components of the high-resolution spatial feature map are fused accordingly. The fused phase components and fused amplitude components are recombined and transformed to obtain frequency domain enhancement features.

[0038] The feature fusion branch is configured to perform cross-domain fusion of the frequency domain enhancement features and the spatial domain enhancement features based on an attention strategy, and to perform multi-scale reconstruction based on the joint representation features and the cross-domain fused features to obtain the final enhanced image.

[0039] Step S3: Based on the multi-level fault-tolerant matching network, feature point matching is performed on the final enhanced image pair to obtain mutually matching effective feature point pairs.

[0040] Step S4: Calculate the homography transformation matrix of the final enhanced image pair based on the effective feature point pairs, perform image stitching on the final enhanced image pair according to the homography transformation matrix, and perform color enhancement processing on the stitched image based on the original color channel information to obtain the final stitched image.

[0041] based on Figure 1As shown in the submarine cable image stitching method, this embodiment of the application extracts the original color channel information and performs channel enhancement processing on the image. It also performs color enhancement processing on the stitched image based on the original color channel information. Through a dual-stream decoupling design, the feature enhancement stream is dedicated to improving image quality to support subsequent feature extraction and registration, while the original color stream retains the true physical color information throughout the process. Color restoration is performed based on the original color stream during final stitching, ensuring both the accuracy of geometric registration and avoiding color distortion caused by image enhancement, thus ensuring the accuracy of downstream defect detection. Image enhancement is performed through a specially designed dual-domain decoupling enhancement network. Since the phase component retains the structural contour of the image, and the amplitude component reflects color and illumination information, by enhancing and fusing the two separately, the structural integrity of the image is effectively restored. This embodiment effectively restores the clarity and color fidelity of submarine cable images through dual-domain decoupling enhancement, ensures a high success rate of stitching under extreme conditions through multi-level fault-tolerant matching, and achieves synergistic optimization of geometric registration and color fidelity through dual-stream decoupling design. The final output image is a high-fidelity, high-quality underwater submarine cable stitching image, providing reliable technical support for applications such as submarine cable inspection and underwater target detection. Furthermore, it enhances image matching robustness through a multi-level fault-tolerant matching network, providing high-precision dense matching when computing power is sufficient and image quality is good, while automatically performing precision-downgraded matching when computing power is limited.

[0042] In some embodiments, step S1 obtains the channel-enhanced image through the following steps:

[0043] Each frame of the image is converted to the LAB color space, and Contrast Limited Adaptive Histogram Equalization (CLAHE) is performed on the luminance channel L. The color channels a and b are linearly stretched. The processed image is then converted back to the original RGB color space to obtain the channel-enhanced image.

[0044] This embodiment enhances the channel processing of submarine cable images, including color space transformation and enhancement, to effectively suppress problems such as color cast and low contrast caused by underwater light attenuation and suspended particles. At the same time, it enhances the texture details and edge structure of the image, thereby providing high-quality feature input for subsequent steps such as dual-domain feature decoupling, cross-domain fusion enhancement and feature matching, and improving the accuracy and robustness of image registration and stitching.

[0045] In some embodiments, the dual-domain feature decoupling branch of step S2 is configured to obtain the spatial domain enhanced features and frequency domain correlated features through the following steps:

[0046] The low-resolution spatial feature map is split along the channel, and the split feature map is subjected to feature interaction through an affine learning layer and a semi-instance normalization block to obtain spatial domain enhanced features; the frequency domain components of the low-resolution spatial feature map are converted to the spatial domain after convolutional serialization operation to obtain the frequency domain related features.

[0047] The frequency domain components of the low-resolution spatial feature map include phase components and amplitude components. The convolutional serialization operation refers to processing the phase components and amplitude components respectively through the structure block of the convolutional layer + Leaky ReLU activation function, so as to suppress noise interference and enhance effective structural information.

[0048] In some embodiments, the dual-domain feature decoupling branch of step S2 is configured to obtain the joint representation features through the following steps:

[0049] The spatial domain enhanced features and frequency domain related features are concatenated by channels. The concatenated features are then integrated and information is exchanged through a 1×1 convolutional layer to obtain the joint representation features, which provide a feature foundation for subsequent fine-grained frequency domain processing and cross-domain fusion.

[0050] In some embodiments, the dual-domain feature decoupling branch of step S2 is configured to obtain the fused phase component and the fused amplitude component by means of the following:

[0051] After processing the phase component of the high-resolution spatial feature map with a convolution sequence, it is concatenated with the phase component of the joint representation feature to obtain the fused phase component; the amplitude component of the joint representation feature is fused with the amplitude component of the high-resolution spatial feature map through an affine learning strategy to obtain the fused amplitude component.

[0052] In some embodiments, the feature fusion branch of step S2 is configured to obtain the final enhanced image through the following steps:

[0053] The joint representation features are convolved to obtain the enhanced result of the joint representation features; the cross-domain fusion features are refined and rearranged to obtain the enhanced result of the cross-domain fusion features; the enhanced result of the joint representation features and the enhanced result of the cross-domain fusion features are concatenated and convolved to obtain the final enhanced image.

[0054] In some embodiments, the dual-domain decoupling enhancement network in step S2 is jointly optimized based on a multi-scale joint loss function, wherein the multi-scale joint loss function includes at least two of the following losses:

[0055] Pixel-level loss is configured to constrain the pixel differences between the final enhanced image and the submarine cable image;

[0056] The structural loss is configured as a structural similarity index between the final enhanced image and the submarine cable image;

[0057] Perceptual loss is configured to constrain the perceptual consistency between the final enhanced image and the submarine cable image.

[0058] In some embodiments, the multi-level fault-tolerant matching network in step S3 includes a backbone matching layer, a secondary matching layer, and a fallback matching layer, wherein:

[0059] The backbone matching layer is configured to perform feature matching on the final enhanced image pairs based on a pre-trained LoFTR model;

[0060] The secondary matching layer is configured to extract key points of each frame of the final enhanced image based on the scale-invariant feature transform SIFT detector. The key points include local gradient feature descriptors. The local gradient feature descriptors of the final enhanced image pair are searched for nearest neighbors based on the fast nearest neighbor search library matcher.

[0061] The catch-all matching layer is configured to extract corner points of each frame of the final enhanced image based on an OFRB (Oriented FAST and Rotated BRIEF) detector, wherein the corner points include BRIEF binary descriptors, and calculate the similarity of the final enhanced image pairs based on a brute-force matcher combined with Hamming distance.

[0062] In some embodiments, step S3 obtains mutually matching valid feature point pairs through the following steps:

[0063] When the number of feature point pairs in the feature point pair set output by the main matching layer is not less than a preset threshold, the effective feature point pair is obtained based on the output of the main matching layer; when the number of feature point pairs output by the main matching layer is less than the preset threshold, the nearest neighbor search result output by the secondary matching layer is determined; if the number of key point pairs in the nearest neighbor search result is not less than a preset threshold, the effective feature point pair is obtained based on the output of the secondary matching layer; when the number of key point pairs output by the secondary matching layer is less than the preset threshold, the effective feature point pair is obtained based on the corner point pairs of the similarity calculation result output by the fallback matching layer.

[0064] Next, combined Figures 3 to 5 This application describes in detail the submarine cable image stitching embodiments.

[0065] First, perform image preprocessing.

[0066] In this embodiment, submarine cable images of adjacent frames are acquired. , These represent the height and width of the submarine cable image, respectively, and 3 indicates the number of RGB color channels. The following preprocessing is performed on each frame of the submarine cable image:

[0067] Original Color Stream: The submarine cable image is scaled to 640×640 pixels, preserving the RGB color channel information of the original submarine cable image. The submarine cable image is scaled to a uniform size to meet the size input requirements of subsequent models, while also taking into account computational efficiency.

[0068] The images from the original color stream are not used for feature extraction; they are only used as reference material for subsequent fusion rendering to ensure the fidelity of physical colors.

[0069] Feature enhancement flow: The scaled submarine cable image is converted from RGB color space to LAB color space, separating the luminance channel L and color channels a and b. The luminance channel L is subjected to CLAHE processing, for example, with a threshold of 2.0 and a grid of 8×8. Color channels a and b are linearly stretched using the following formula (1) to recover the attenuated color signal caused by water absorption:

[0070] (1)

[0071] in, Represents pixels The pixel value of channel a or the pixel value of channel b at that location. Indicates the gain coefficient. Indicates the offset; for example, , .

[0072] After the above processing, the image is converted back to RGB space to obtain a channel-enhanced image. The channel-enhanced image As input to the dual-domain decoupling enhancement network.

[0073] Then, image enhancement processing is performed.

[0074] The image enhancement process in this embodiment first involves enhancing the image channels using the feature extraction branch. Pixel rearrangement is performed to obtain a high-resolution spatial feature map. Low-resolution spatial feature maps are generated through 3×3 convolutional layers and bilinear downsampling operations. .

[0075] The low-resolution spatial feature map The input is a dual-domain feature decoupling branch, which includes a spatial-frequency domain decoupling (SFD) block. The SFD block comprises a spatial sub-branch and a frequency domain sub-branch, which are processed in parallel. Wherein:

[0076] Spatial sub-branch: splits the input features along the channels into By interacting with features through an affine learning layer and a semi-instance normalization (HIN) block, spatial domain enhanced features are output. :

[0077] (2)

[0078] In the above formula (2), ,

[0079] CAT is the channel concatenation operator. This is a valid element-wise multiplication.

[0080] Frequency domain sub-branch: Perform a two-dimensional fast Fourier transform on the input features to obtain the amplitude components. With phase components Convolutional sequence After optimization, the system is transformed back to the spatial domain using an inverse fast Fourier transform, and the frequency domain correlation features are output. :

[0081] (3)

[0082] In formula (3), This represents the inverse fast Fourier transform. .

[0083] The dual-domain feature decoupling branch enhances spatial domain features. Frequency domain correlation characteristics Channel concatenation and 1×1 convolution are performed to obtain the decoupled joint representation features. .

[0084] Next, the dual-domain feature decoupling branch pairs the joint representation features. Applying two-dimensional Fourier transform to extract amplitude components With phase components and extraction of high-resolution spatial feature maps phase components With amplitude components The fused phase components are obtained using the following formula (4). The fused amplitude components are obtained using the following formula (5). :

[0085] (4)

[0086] (5)

[0087] The dual-domain feature decoupling branch applies to the fused phase components. With the fused amplitude component After recombination, frequency domain enhancement features are obtained through inverse fast Fourier transform. :

[0088] (6)

[0089] In obtaining frequency domain enhancement features Enhanced features of the spatial domain Subsequently, feature fusion enhancement is performed based on an attention strategy.

[0090] Specifically, the feature fusion branch enhances spatial domain features. As a spatial query feature Enhance frequency domain features As Value characteristics Cross-domain fusion is performed based on the following formula (7):

[0091] (7)

[0092] In formula (7), The feature dimension is used to normalize the attention weights. This indicates the result of attention fusion.

[0093] Integrating attention fusion results with spatial domain enhancement features Residual connections are made to preserve the original spatial structure information, which is then input into a feedforward network (FFN). Feature refinement is performed through a combination of layer normalization, depthwise convolution, and activation function blocks to obtain cross-domain fused features. This allows for the adaptive injection of frequency domain color and structural information into spatial textures.

[0094] Next, the feature fusion branch will fuse features across domains. The input is fed into two concatenated SFD blocks to enhance the detailed feature representation. Then, the feature resolution is increased to the original image size through pixel rearrangement to obtain the enhanced result of cross-domain fusion features, so as to preserve fine-grained texture information.

[0095] Furthermore, the feature fusion branch will jointly represent the features. Convolutional layers are applied to generate enhanced results that jointly represent features. To achieve this enhanced result Constrain global color consistency and structural stability to avoid global distortion caused by subsequent detail enhancements.

[0096] Finally, the feature fusion branch will jointly represent the enhanced results of the features. The enhanced image is obtained by concatenating channels with the cross-domain fusion feature enhancement results and then fusing features through three cascaded 3×3 convolutional layers. In this way, local color deviations can be corrected and texture clarity enhanced, achieving synergistic optimization of global structure and local details.

[0097] The dual-domain decoupling enhancement network in this embodiment adopts the multi-scale joint loss function shown in the following formula (8). Optimize:

[0098] (8)

[0099] in, For pixel-level loss, multi-scale L1 loss is used to constrain the enhancement image. With submarine cable images Pixel-level differences For low-resolution loss weights, This represents the enhanced result of cross-domain fusion features corresponding to training image i. The enhancement result represents the joint representation features corresponding to training image i. and corresponding scale images of submarine cables Used to calculate the true gap. The number of training images.

[0100] This is a structural loss used to ensure the structural integrity of the enhanced image. The weighting coefficients of the structural loss, SSIM, have a value range of [value missing]. The closer this value is to 1, the more similar the structure of the enhanced image is to the submarine cable image.

[0101] To mitigate perceptual loss, a pre-trained VGG network is used to extract features, which are then used to constrain the perceptual consistency between the augmented image and the submarine cable image, thereby improving visual naturalness. This represents the feature extraction operation of the intermediate layer of the VGG network.

[0102] Example, , Using the ADAM optimizer ( , ), initial learning rate The batch size is 2. Under the NVIDIA RTX 4090 environment, the single-frame inference time is 0.061s, and the computation amount is 57.52G Flops.

[0103] Next, feature matching processing is performed.

[0104] The image stitching process in this embodiment is as follows: Figure 3 As shown, the enhanced image output by the dual-domain decoupling enhancement network is first... Grayscale and normalized to Interval, constructed dimension is Feature tensor ( ).

[0105] First, feature matching is performed based on the backbone matching layer.

[0106] Specifically, the system prioritizes loading pre-trained LoFTR model weights from a specified local path; if not found locally, it automatically attempts to download them online from a specified target source. Unlike traditional algorithms that rely on local corner or edge detection, this network first uses a convolutional neural network to extract multi-scale feature maps. Then, through self-attention and cross-attention mechanisms, it establishes a contextual relationship between the feature sequences of two enhanced images within the global receptive field. The system skips independent keypoint detection steps and directly performs dense matching from coarse-grained to sub-pixel level fine-grained by calculating the inner product of the feature maps and the Softmax probability distribution. Finally, it outputs a set of feature matching point pairs. If the number of feature matching point pairs in this set is... Greater than or equal to a preset threshold, for example, the preset threshold is 4, that is, if If the feature matching point pairs in the set are used as valid feature point pairs for the two enhanced images, then if an abnormal operating environment occurs, such as a missing Kornia dependency library, a non-existent local model, an unreachable network, or CUDA device incompatibility, or If the system determines that the conditions for deep learning inference are not met, it will automatically trigger feature matching based on the secondary matching layer.

[0107] When feature matching based on the secondary matching layer is triggered, it indicates that the current device's computing power is insufficient or the deep network fails under extreme image conditions. At this time, the system instantiates the SIFT detector to extract features. The SIFT detector has strong robustness to image scaling, rotation, and illumination changes, and can be used as the preferred solution for high-precision degradation.

[0108] The SIFT detector first performs a feature extraction process, which involves detecting extreme points as keypoints in the Difference-of-Gaussian (DoG) scale space of the feature-enhanced stream image and calculating the principal direction of the feature points, generating a 128-dimensional local gradient feature descriptor for each keypoint. Then, it performs a nearest neighbor search. Since the 128-dimensional descriptor is a high-dimensional floating-point vector, the system uses a FLANN matcher based on the Kd-Tree data structure to improve search efficiency. For each descriptor in one enhanced image frame, a search is performed in another enhanced image frame. K-nearest neighbor search is used to find the first and second nearest neighbors that are closest in Euclidean distance.

[0109] Optionally, to eliminate mismatches caused by underwater suspended particles or similar sand bed textures, Lowe's scale test is applied for mismatch elimination. The mathematical expression of the judgment criteria for mismatch elimination is as follows:

[0110] (9)

[0111] In formula (9), The Euclidean distance from the key point to its first nearest neighbor is given. Let Euclidean distance be the distance from the key point to the second nearest neighbor. For example, a set ratio threshold. This ratio threshold ensures that a first nearest neighbor is retained only when its matching degree is significantly better than that of the second nearest neighbor, thus effectively filtering out noise interference from underwater repetitive textures.

[0112] Count the number of keypoint pairs that meet the above conditions. ,like If this meets the minimum mathematical requirement for calculating the homography matrix, the number of keypoint pairs obtained based on the secondary matching layer will be used as the effective feature point pairs for the two enhanced images; if If the image is severely motion-blurred, causing SIFT gradient calculation to fail completely, then feature matching based on the fallback matching layer will be automatically triggered.

[0113] When feature matching based on the fallback matching layer is triggered, it indicates that the image degradation is extremely severe, such as severe blurring leading to the absence of gradient features. In this case, the system instantiates an OFRB detector with extremely low computational complexity as a last resort to prevent pipeline collapse.

[0114] The OFRB detector first performs a feature extraction process, extracting rotation-invariant FAST corner points and calculating a BRIEF binary descriptor for each corner point. Then, it performs a distance metric and matching process. Since the BRIEF descriptor is a binary string of 0s and 1s, the system can use a brute-force matcher combined with Hamming distance to calculate similarity. The search operation on edge computing devices only requires a simple XOR operation, which greatly saves CPU computing power.

[0115] Optionally, a ratio test as shown in formula (9) above can also be applied to eliminate false matches during the matching process. Optionally, the ratio threshold can be set to a more lenient value, for example... Since this level is the last layer of the fallback, and binary descriptors have weaker discriminative power than floating-point descriptors, continuing to use a strict threshold of 0.7 would easily lead to the accidental deletion of the few remaining correct matches. Therefore, the threshold is relaxed to 0.75, which allows for the retention of as many valid matches as possible under harsh conditions where features are extremely scarce, thus achieving a high-availability fallback for the system.

[0116] Count the number of corner pairs that satisfy the above conditions. ,like Then, effective feature point pairs are obtained from the two enhanced images; if there are still... If the geometric transformation cannot be completed at either the mathematical or physical level, the system determines that the two enhanced images do not have a valid overlapping area and returns an abnormal status code.

[0117] Based on the above embodiments, priority feature point pairs between two enhanced images can be obtained, such as Figure 4 As shown in the feature point matching results, this embodiment can obtain relatively accurate and robust registration results based on multi-level fault-tolerant matching.

[0118] Finally, image stitching is performed.

[0119] The optimal homography transformation matrix is ​​obtained by using the effective feature point pairs output by the multi-level fault-tolerant matching network. :

[0120] (10)

[0121] For ease of explanation, one of the two enhanced images will be referred to as the reference image, and the other as the image to be registered. The homography transformation matrix will then be used. Pixels of the image to be registered Perform a perspective transformation and map it to the reference image:

[0122] (11)

[0123] Based on the above perspective transformation results, image stitching is performed to obtain an initial stitched image; for the initial stitched image, the original color channel information is smoothly transitioned and fused based on the following formula (12) to obtain the color channel information of the final stitched image:

[0124]

[0125] For the pixels in the final stitched image Pixel value at that location, For the pixel points in the reference image Pixel value at that location, The pixels in the image to be registered after perspective transformation The corresponding pixel value, The fusion weight is preferably 0.5. A is the set of all pixels in the reference image, and B is the set of mapped pixels formed by perspective transformation of all pixels in the image to be registered.

[0126] Based on the above embodiments of this application, by stitching together two frames of submarine cable images, the following can be obtained: Figure 5 The image shown is a high-fidelity underwater mosaic.

[0127] Figure 6 This is a schematic diagram of an electronic device illustrated in this specification according to an exemplary embodiment. Please refer to... Figure 6 At the hardware level, the device includes a processor 602, an internal bus 604, a network interface 606, memory 608, a hardware acceleration device 610, and non-volatile memory 612, and may also include other hardware required for its functions. One or more embodiments of this application can be implemented in software, for example, the processor 602 reads the corresponding computer program from the non-volatile memory 612 into memory 608 and then runs it. Of course, in addition to software implementation, one or more embodiments of this application do not exclude other implementation methods, such as logic devices or a combination of hardware and software, etc. That is to say, the execution subject of the above processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0128] Figure 7 This is a structural block diagram of a submarine cable image stitching device based on dual-domain decoupling enhancement and multi-level fault-tolerant matching, as illustrated in an exemplary embodiment of this application. The submarine cable image stitching device can be applied to, for example... Figure 6 The electronic device shown implements the technical solution of this application. The submarine cable image stitching device includes: a preprocessing unit 710, an image enhancement unit 720, a feature matching unit 730, and an image stitching unit 740, wherein:

[0129] The preprocessing unit 710 is used to acquire submarine cable image pairs of adjacent frames, extract the original color channel information of each frame image in the submarine cable image pair, and perform channel enhancement processing on each frame image to obtain a channel-enhanced image.

[0130] Image enhancement unit 720 is used to process the channel enhancement image based on a preset dual-domain decoupling enhancement network to obtain the final enhanced image; the dual-domain decoupling enhancement network includes a feature extraction branch, a dual-domain feature decoupling branch, and a feature fusion branch; wherein:

[0131] The feature extraction branch is configured to perform multi-scale feature extraction on each frame of channel-enhanced image to obtain a high-resolution spatial feature map and a low-resolution spatial feature map.

[0132] The dual-domain feature decoupling branch is configured to extract the spatial domain enhancement features and frequency domain correlation features of the low-resolution spatial feature map, and fuse the spatial domain enhancement features and frequency domain correlation features to obtain joint representation features. The phase and amplitude components of the joint representation features and the phase and amplitude components of the high-resolution spatial feature map are fused accordingly. The fused phase components and fused amplitude components are recombined and transformed to obtain frequency domain enhancement features.

[0133] The feature fusion branch is configured to perform cross-domain fusion of the frequency domain enhancement features and the spatial domain enhancement features based on an attention strategy, and to perform multi-scale reconstruction based on the joint representation features and the cross-domain fused features to obtain the final enhanced image;

[0134] The feature matching unit 730 is used to perform feature point matching on the final enhanced image pair based on a multi-level fault-tolerant matching network to obtain mutually matching effective feature point pairs.

[0135] The image stitching unit 740 is used to calculate the homography transformation matrix of the final enhanced image pair based on the effective feature point pairs, stitch the final enhanced image pair according to the homography transformation matrix, and perform color enhancement processing on the stitched image based on the original color channel information to obtain the final stitched image.

[0136] In some embodiments, the preprocessing unit 710 is configured to convert each frame of image to the LAB color space, perform limited contrast adaptive histogram equalization on the luminance channel, and perform linear stretching on the color channel; and convert the processed image back to the original RGB color space to obtain the channel-enhanced image.

[0137] In some embodiments, the feature matching unit 730 is configured to: obtain the effective feature point pair based on the output of the main matching layer when the number of feature point pairs in the feature point pair set output by the main matching layer is not less than a preset threshold; determine the nearest neighbor search result output by the secondary matching layer when the number of feature point pairs output by the main matching layer is less than the preset threshold; obtain the effective feature point pair based on the output of the secondary matching layer if the number of key point pairs in the nearest neighbor search result is not less than the preset threshold; and obtain the effective feature point pair based on the corner point pairs of the similarity calculation result output by the fallback matching layer when the number of key point pairs output by the secondary matching layer is less than the preset threshold.

[0138] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this application according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0139] Accordingly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in any of the above embodiments.

[0140] Accordingly, embodiments of this application also provide a computer program product configured to perform the methods described in any of the above embodiments.

[0141] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer, which can take the form of a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email sending and receiving device, game console, tablet computer, wearable device, or any combination of these devices.

[0142] In a typical configuration, a computer includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0143] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0144] Computer-readable media, including both permanent and non-permanent, removable and non-removable media, can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random-access memory (SRAM), dynamic random-access memory (DRAM), other types of random-access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, disk storage, quantum memory, graphene-based storage media or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0145] While this specification contains numerous specific implementation details, these should not be construed as limiting the scope of any invention or the scope of the claims, but rather are primarily intended to describe features of specific embodiments of a particular invention. Certain features described in the various embodiments herein may also be implemented in combination in a single embodiment. Conversely, various features described in a single embodiment may also be implemented separately in various embodiments or in any suitable sub-combination. Furthermore, while features may function in certain combinations as described above and even initially claimed in this way, one or more features from a claimed combination may be removed from that combination in some cases, and a claimed combination may refer to a sub-combination or a variation thereof.

[0146] Similarly, although the operations are depicted in a specific order in the accompanying drawings, this should not be construed as requiring these operations to be performed in the specific order shown or sequentially, or requiring all illustrated operations to be performed to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above embodiments should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0147] Thus, specific embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. In some cases, the actions recited in the claims may be performed in a different order and still achieve the desired result. Furthermore, the processes depicted in the drawings are not necessarily shown in a specific order or sequence to achieve the desired result. In some implementations, multitasking and parallel processing may be advantageous.

[0148] It should be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0149] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A method for stitching submarine cable images based on dual-domain decoupling enhancement and multi-level fault-tolerant matching, characterized in that, Includes the following steps: Step S1: Obtain submarine cable image pairs of adjacent frames, extract the original color channel information of each frame in the submarine cable image pair, and perform channel enhancement processing on each frame to obtain a channel-enhanced image. Step S2: The channel enhancement image is processed based on a preset dual-domain decoupling enhancement network to obtain the final enhanced image; the dual-domain decoupling enhancement network includes a feature extraction branch, a dual-domain feature decoupling branch, and a feature fusion branch; wherein: The feature extraction branch is configured to perform multi-scale feature extraction on each frame of channel-enhanced image to obtain a high-resolution spatial feature map and a low-resolution spatial feature map. The dual-domain feature decoupling branch is configured to extract spatial domain enhancement features and frequency domain correlation features from the low-resolution spatial feature map, fuse the spatial domain enhancement features and frequency domain correlation features to obtain joint representation features, and fuse the phase and amplitude components of the joint representation features and the phase and amplitude components of the high-resolution spatial feature map accordingly. The fused phase and amplitude components are then recombined and transformed to obtain frequency domain enhancement features. The dual-domain feature decoupling branch is configured to obtain the fused phase and amplitude components as follows: the phase component of the high-resolution spatial feature map is processed by a convolutional sequence, and then concatenated with the phase component of the joint representation features through channel concatenation to obtain the fused phase component; the amplitude component of the joint representation features is fused with the amplitude component of the high-resolution spatial feature map through an affine learning strategy to obtain the fused amplitude component. The feature fusion branch is configured to perform cross-domain fusion of the frequency domain enhancement features and the spatial domain enhancement features based on an attention strategy, and to perform multi-scale reconstruction based on the joint representation features and the cross-domain fused features to obtain the final enhanced image; Step S3: Based on the multi-level fault-tolerant matching network, feature point matching is performed on the final enhanced image pair to obtain mutually matching effective feature point pairs; Step S4: Calculate the homography transformation matrix of the final enhanced image pair based on the effective feature point pairs, perform image stitching on the final enhanced image pair according to the homography transformation matrix, and perform color enhancement processing on the stitched image based on the original color channel information to obtain the final stitched image.

2. The method according to claim 1, characterized in that, Step S1 obtains the channel-enhanced image through the following steps: Each frame of the image is converted to the LAB color space, and the brightness channel is subjected to contrast adaptive histogram equalization, while the color channels are linearly stretched. The processed image is converted back to the original RGB color space to obtain the channel-enhanced image.

3. The method according to claim 1, characterized in that, The dual-domain feature decoupling branch in step S2 is configured to obtain the spatial domain enhanced features and frequency domain correlated features through the following steps: The low-resolution spatial feature map is split along the channel, and the split feature map is interacted with by an affine learning layer and a semi-instance normalization block to obtain spatial domain enhanced features. The frequency domain components of the low-resolution spatial feature map are converted to the spatial domain by performing convolutional serialization operations, thereby obtaining the frequency domain related features.

4. The method according to claim 1, characterized in that, The feature fusion branch in step S2 is configured to obtain the final enhanced image through the following steps: The joint representation features are convolved to obtain the enhanced result of the joint representation features; After performing feature refinement and pixel rearrangement on the cross-domain fusion features, the enhanced result of the cross-domain fusion features is obtained; The enhanced result of the joint representation feature and the enhanced result of the cross-domain fusion feature are subjected to channel concatenation and convolution to obtain the final enhanced image.

5. The method according to claim 1, characterized in that, The dual-domain decoupling enhancement network in step S2 is jointly optimized based on a multi-scale joint loss function, wherein the multi-scale joint loss function includes at least two of the following losses: Pixel-level loss is configured to constrain the pixel differences between the final enhanced image and the submarine cable image; The structural loss is configured as a structural similarity index between the final enhanced image and the submarine cable image; Perceptual loss is configured to constrain the perceptual consistency between the final enhanced image and the submarine cable image.

6. The method according to claim 1, characterized in that, The multi-level fault-tolerant matching network in step S3 includes a backbone matching layer, a secondary matching layer, and a fallback matching layer, wherein: The backbone matching layer is configured to perform feature matching on the final enhanced image pairs based on a pre-trained LoFTR model; The secondary matching layer is configured to extract key points of each frame of the final enhanced image based on the scale-invariant feature transform SIFT detector. The key points include local gradient feature descriptors. The local gradient feature descriptors of the final enhanced image pair are searched for nearest neighbors based on the fast nearest neighbor search library matcher. The fallback matching layer is configured to extract corner points of each frame of the final enhanced image based on the OFRB detector, the corner points including BRIEF binary descriptors, and calculate the similarity of the final enhanced image pairs based on a brute-force matcher combined with Hamming distance.

7. The method according to claim 6, characterized in that, Step S3 obtains mutually matching valid feature point pairs through the following steps: When the number of feature point pairs in the feature point pair set output by the backbone matching layer is not less than a preset threshold, the effective feature point pair is obtained based on the output of the backbone matching layer. When the number of feature point pairs output by the main matching layer is less than a preset threshold, the nearest neighbor search result output by the secondary matching layer is determined. If the number of key point pairs in the nearest neighbor search result is not less than a preset threshold, the effective feature point pair is obtained based on the output of the secondary matching layer. When the number of key point pairs output by the secondary matching layer is less than a preset threshold, the effective feature point pairs are obtained based on the corner point pairs of the similarity calculation results output by the bottom matching layer.

8. An electronic device, characterized in that, include: processor; A computer-readable storage medium storing computer program instructions that, when executed by the processor, cause the processor to perform the method as described in any one of claims 1 to 7.

Images