Underwater robot image real-time repair and target recognition method based on gan

Abstract

The application relates to the technical field of underwater robot vision processing, and discloses a GAN-based underwater robot image real-time repairing and target identification method. The method acquires an original image sequence obtained by an optical sensor of an underwater robot, extracts degradation features of each frame of image, including scattering noise distribution, color distortion parameters and motion blur intensity. The original image sequence is input into a network, and based on the degradation features, a repaired image is generated frame by frame, and local structure consistency features and global semantic coherence features of the repaired image are extracted. A cascaded classifier is used for multi-scale target detection of the repaired image, local structure features and global semantic features are combined, and a target boundary box and a category label are output. According to the spatial distribution density of the target boundary box, the priority of the repaired area is dynamically adjusted, and the synchronous output of the repaired result and the target identification result is realized. The method can effectively improve underwater image quality and target identification precision.

Description

Technical Field

[0001] This invention relates to the field of underwater robot vision processing technology, specifically a method for real-time image restoration and target recognition of underwater robots based on GAN. Background Technology

[0002] Underwater robots have wide applications in marine resource exploration, pipeline inspection, and ecological monitoring. Due to the complexity of the underwater environment, images acquired by optical sensors often suffer from severe quality degradation. The absorption and scattering of light by water leads to color distortion, decreased contrast, and blurred details. Simultaneously, suspended particles underwater cause scattering noise, while the robot's own movement and water current disturbances introduce motion blur. These degradation phenomena severely affect image readability and the accuracy of subsequent target recognition tasks. Traditional underwater image enhancement methods are mostly based on physical models, such as using dark channel priors or fog line theory for dehazing. These methods rely on simplified assumptions about underwater optical propagation processes and have poor generalization ability in real-world complex environments. Furthermore, while histogram equalization or white balance adjustment methods can improve image contrast, they easily amplify noise and cause color oversaturation, destroying the naturalness of the image.

[0003] In recent years, deep learning technology has been introduced into the field of underwater image processing. Convolutional neural networks have been used to learn the mapping relationship from degraded images to sharp images, but these methods usually require a large amount of paired data (i.e., degraded images and their corresponding sharp images) for supervised training. However, obtaining truly sharp images in underwater environments is extremely difficult, limiting the practical application of these methods. Generative adversarial networks (GANs) can alleviate the dependence on paired data to some extent through adversarial training mechanisms, but existing dual-channel GAN ​​methods mostly focus on image style transfer or overall quality improvement, failing to fully consider the diversity and spatial distribution differences of underwater degradation features.

[0004] Underwater target recognition tasks are often performed directly on degraded images, or image inpainting and target recognition are processed as two separate stages. The former suffers from poor image quality, resulting in recognition accuracy that fails to meet practical requirements; the latter suffers from error accumulation, and the inpainting process does not consider the specific needs of the recognition task, leading to inefficient allocation of computational resources. Existing methods lack a mechanism that can collaboratively optimize image inpainting and target recognition, making it difficult to achieve efficient and accurate real-time processing.

[0005] Furthermore, underwater robots typically possess limited computing resources, placing high demands on the real-time performance of algorithms. Current deep learning-based repair and recognition methods suffer from high computational complexity, making real-time inference difficult to implement on embedded platforms. Therefore, there is an urgent need for an integrated solution that can adapt to complex underwater degradation conditions, balance repair quality and recognition efficiency, and meet real-time processing requirements. Summary of the Invention

[0006] The purpose of this invention is to provide a GAN-based method for real-time image restoration and target recognition of underwater robots, in order to solve the problems mentioned in the background art.

[0007] To achieve the above objectives, this invention provides a GAN-based method for real-time image restoration and target recognition of underwater robots, the method comprising:

[0008] The raw image sequence acquired by the underwater robot's optical sensor is collected, and the degradation features of each frame are extracted. The degradation features include scattering noise distribution, color distortion parameters, and motion blur intensity.

[0009] A dual-channel generative adversarial network (GAN) is constructed. The first channel is used as the conditional input for degraded features, and the second channel is used for adversarial training to generate and repair images. The gradient updates of the two channels are coordinated through a dynamic weight allocation mechanism.

[0010] The original image sequence is input into a dual-channel generative adversarial network, which generates repaired images frame by frame based on degradation features, and extracts local structural consistency features and global semantic coherence features of the repaired images.

[0011] A cascaded classifier is used to perform multi-scale target detection on the restored image. Combining local structural consistency features and global semantic coherence features, the target bounding box and category label are output.

[0012] Based on the spatial distribution density of the target bounding box, the priority of the repair region of the dual-channel generative adversarial network is dynamically adjusted to generate the final repair result and the target recognition result in a synchronous output.

[0013] Preferably, the extraction of degradation features for each frame of image includes:

[0014] The original image sequence is divided into blocks, the scattering noise energy value of each image block is calculated, and the scattering noise distribution matrix is ​​generated by combining the water body transmittance model.

[0015] The color bias vector of each pixel is extracted by color space conversion, and the cluster center of the color bias vector in the CIELAB color space is counted to generate a set of color distortion parameters.

[0016] Motion trajectories are estimated based on the optical flow field between adjacent frames, and quantitative indicators of motion blur intensity are calculated based on trajectory length and direction dispersion.

[0017] Preferably, the construction of the dual-channel generative adversarial network includes:

[0018] A degradation feature encoder is deployed in the first channel to convert the scattering noise distribution matrix, color distortion parameter set and motion blur intensity quantization index into a conditional vector;

[0019] In the second channel, an adversarial structure is configured between the generator and the discriminator. The generator adopts the U-Net architecture and embeds an attention gating mechanism for conditional vectors, while the discriminator adopts a multi-scale PatchGAN structure.

[0020] A dynamic weight allocation mechanism is designed to adjust the weight ratio of the dual-channel loss function based on the difference in peak signal-to-noise ratio between the generated image and the real image in each training round.

[0021] Preferably, the adversarial training for generating the repaired image using the second channel includes:

[0022] Conditional vectors are injected into the skip connection layer of the generator, and feature maps related to degenerate features are filtered through an attention gating mechanism.

[0023] The discriminator performs multi-scale judgment on the authenticity of the generated image and outputs authenticity scores at different resolutions;

[0024] Based on the matching degree between the authenticity score and the conditional vector, the adversarial loss and content loss of the generator are calculated to drive the network parameter update.

[0025] Preferably, the simultaneous extraction of local structural consistency features and global semantic coherence features of the repaired image includes:

[0026] The shallow texture features and deep semantic features of the restored image are extracted using a pre-trained convolutional neural network.

[0027] Calculate the similarity matrix of shallow texture features between adjacent image patches to generate local structural consistency features;

[0028] Spatial pyramid pooling is applied to deep semantic features, and multi-level features are fused to generate global semantic coherence features.

[0029] Preferably, the step of using a cascaded classifier to perform multi-scale target detection on the repaired image includes:

[0030] The first detector of the cascaded classifier is configured as an anchor-based region proposal network, and candidate box generation is optimized by combining local structural consistency features.

[0031] The secondary detector employs a feature pyramid structure, fusing global semantic coherence features to predict the target category;

[0032] The final detector performs bounding box regression and class confidence calibration, and outputs the spatial location and class label of the target.

[0033] Preferably, the dynamic adjustment of the repair region priority of the dual-channel generative adversarial network includes:

[0034] The distribution density of target bounding boxes in the image is statistically analyzed to generate a spatial attention heatmap;

[0035] Based on the spatial attention heatmap, high-priority repair areas and low-priority repair areas are divided;

[0036] Increase the number of feature channels of the generator in high-priority repair regions, while reducing the discriminator sampling rate in low-priority regions.

[0037] Preferably, the synchronous output of generating the final repair result and the target recognition result includes:

[0038] Align the final repair results with the target recognition results by timestamp;

[0039] Overlay a visual layer of the target bounding box and category labels onto the repaired image;

[0040] Establish a mapping table between repair quality assessment indicators and target identification accuracy for subsequent model iterations.

[0041] Preferably, the method further includes:

[0042] Real-time monitoring of the motion state parameters of the underwater robot, and prediction of degradation features of the next frame image based on attitude change data;

[0043] The predicted degradation features are pre-input into a dual-channel generative adversarial network (GAN) to pre-generate the repaired image features for the next frame.

[0044] Preferably, the method further includes: establishing a residual connection channel between the repaired image features and the original image sequence; and dynamically updating the parameter weights of the generator by feeding back the repair error through the residual connection channel.

[0045] Compared with the prior art, the beneficial effects of the present invention are:

[0046] This invention achieves synergistic optimization of underwater image restoration and target recognition tasks through a dual-channel generative adversarial network structure and a dynamic weight allocation mechanism. This method adaptively adjusts the restoration strategy based on differences in image degradation features, significantly improving the visual quality and structural integrity of the restored image. During the restoration process, local structural consistency features and global semantic coherence features are extracted simultaneously, providing rich and robust feature representations for subsequent target recognition.

[0047] Multi-scale target detection is achieved through a cascaded classifier, combining local and global features to enhance the recognition capability of targets at different scales, maintaining high recognition accuracy even under complex conditions such as target occlusion, deformation, or partial loss. The priority of the repair region is dynamically adjusted based on the spatial distribution density of the target bounding box, concentrating computational resources on the region of interest and avoiding over-processing of the background region, thus improving overall processing efficiency. This method overcomes the error propagation problem inherent in traditional staged processing, forming a closed-loop optimization between the repair process and the recognition task, with both mutually reinforcing each other. Image repair provides high-quality input for recognition, while the recognition results, in turn, guide the fine-tuning of the repair region, creating a virtuous cycle. This integrated design reduces redundant computation, accelerates processing speed, and meets the real-time requirements of underwater robots.

[0048] The conditional input mechanism of the dual-channel generative adversarial network (GAN) makes the repair process highly targeted, effectively handling the combined effects of different degradation types (such as scattering, color distortion, and motion blur). The dynamic weight allocation mechanism ensures the stability of network training, avoids mode collapse caused by dual-channel gradient conflicts, and improves the model's convergence speed and generalization ability. Attached Figure Description

[0049] Figure 1 This is a flowchart of the GAN-based underwater robot image real-time restoration and target recognition method described in this invention;

[0050] Figure 2 A flowchart for extracting degradation features from each frame of the image;

[0051] Figure 3 A flowchart for constructing a dual-channel generative adversarial network;

[0052] Figure 4 A model framework diagram for constructing a dual-channel generative adversarial network;

[0053] Figure 5 A flowchart for extracting local structural consistency features and global semantic coherence features from the restored image. Detailed Implementation

[0054] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0055] Terminology Explanation:

[0056] GAN: Two-channel generative adversarial network.

[0057] Please see Figure 1 This invention provides a GAN-based method for real-time image restoration and target recognition for underwater robots. The method is implemented as follows: The original image sequence acquired by the underwater robot's optical sensor is collected. Degradation features of each frame are extracted using an image processing module. These features include scattering noise distribution, color distortion parameters, and motion blur intensity. A dual-channel generative adversarial network (GAN) is constructed, where the first channel is used for conditional input of the degradation features, and the second channel is used for adversarial training to generate the restored image. A dynamic weight allocation mechanism coordinates the gradient updates of the two channels. The original image sequence is input into the dual-channel GAN, and restored images are generated frame by frame based on the degradation features. Simultaneously, local structural consistency features and global semantic coherence features of the restored images are extracted. A cascaded classifier is used to perform multi-scale target detection on the restored images. Combining local structural consistency features and global semantic coherence features, target bounding boxes and category labels are output. The restoration region priority of the dual-channel GAN ​​is dynamically adjusted according to the spatial distribution density of the target bounding boxes, generating a synchronized output of the final restoration result and the target recognition result. This method achieves efficient underwater image processing by integrating degradation feature extraction, dual-channel network restoration, and multi-scale target recognition.

[0058] Example 1: See Figure 2 After acquiring the raw image sequence from the underwater robot's optical sensor, the process initiates a block-based processing operation. Each image block is divided into a fixed-size pixel region, with the block size set to 16x16 pixels. This block processing decomposes the entire frame into multiple local regions for refined analysis. The scattering noise energy value of each image block is calculated. This calculation depends on the variance of pixel intensity and the local contrast index within the image block. The pixel intensity variance is obtained by statistically analyzing the standard deviation of all pixel grayscale values ​​within the image block. The local contrast index is derived by calculating the difference between the maximum and minimum pixel intensity within the image block. The specific formula for the pixel intensity variance is as follows:

[0059]

[0060] in, It is the variance of pixel intensity of the image patch. It is the image patch size. It is the grayscale value of the pixel in the i-th row and j-th column within the image block. It is the average gray value of all pixels within the image block. It is the standard deviation of pixel intensity;

[0061] The specific formula for the local contrast index is:

[0062]

[0063] in: It is the maximum grayscale value of all pixels within an image block. Its value ranges from 0 to 255. It is the minimum grayscale value of all pixels within an image block.

[0064] The specific formula for the energy value of scattered noise is:

[0065]

[0066] Here, α is a weighting coefficient in the calculation of the scattering noise energy value, used to balance the contributions of pixel intensity variance and local contrast index to the scattering noise energy value. Its value is a preset empirical parameter that can be adjusted through the configuration interface according to the actual underwater environment; usually, the range of values ​​for this type of weighting coefficient is determined based on experimental optimization (such as a reasonable value between 0.1 and 0.9) to adapt to the scattering noise feature extraction requirements under different water quality conditions.

[0067] A scattering noise distribution matrix is ​​generated by combining the water body transmittance model. The water body transmittance model is constructed based on the Beer-Lambert law. The model input parameters include preset water quality type parameters, underwater depth data, and light source intensity information. The water body transmittance model outputs the theoretical light intensity attenuation coefficient for each pixel position. Each element value of the scattering noise distribution matrix is ​​generated by weighted fusion of the scattering noise energy value of the corresponding image block and the theoretical light intensity attenuation coefficient.

[0068] The color shift vector of each pixel is extracted through chroma space transformation. This transformation linearly converts the image from the original RGB color space to the CIELAB color space, using a standard transformation matrix and assuming a standard light source D65 as the reference white point. In the CIELAB color space, the color shift vector of each pixel is calculated. This vector is a three-dimensional vector, with components representing the deviation of the pixel color from the reference white point in the L, a, and b* channels. The reference white point corresponds to the color value of a perfectly diffuse surface. The cluster centers of the color shift vectors in the CIELAB color space are statistically analyzed using the K-means algorithm. The number of clusters, K, is dynamically set based on the image's color complexity, typically ranging from 3 to 5. The coordinates of the cluster centers constitute a color distortion parameter set, which contains a quantitative description of the main color deviation patterns in the image. Motion trajectories are estimated based on the inter-frame optical flow field. The optical flow field is calculated using the Lucas-Kanade difference method, which uses the spatiotemporal gradient information of pixel intensity between two adjacent frames in the image sequence to solve for the motion velocity vector of each pixel. The motion blur intensity is quantified based on the motion trajectory length and directional dispersion. The motion trajectory length is calculated by integrating the displacement vector of the same pixel across multiple consecutive frames, while the directional dispersion is measured by the standard deviation of the motion vector direction angles. The specific formula for the motion trajectory length is as follows:

[0069]

[0070] Where: L is the length of the motion trajectory of the same pixel in n consecutive frames. It is the x-direction displacement vector from frame n to frame (n+1). It is the y-direction displacement vector from frame n to frame n+1, where n is the number of consecutive frames involved in the trajectory calculation.

[0071] The specific formula for the motion blur intensity quantification index is as follows:

[0072]

[0073] Where: I is the quantification index of motion fuzziness intensity. It is the normalized directional dispersion.

[0074] For example, the total parallel computation time was 0.08s. The three types of features were encapsulated into a unified data structure (feature type identifier, data array, timestamp) and transmitted via PCIe 4.0 bus (single frame data size 1.2MB, rate 8GB / s, time 0.00015s). Parameters (16×16 image blocks, K=3, 5×5 optical flow window) were stored in non-volatile memory. Integral images and lookup tables were used to optimize computation time, and nearly 10 frames of historical features were buffered for prediction.

[0075] The quantification index of motion blur intensity is ultimately expressed as the product of trajectory length and directional dispersion; a larger index value indicates a more significant motion blur effect. The entire degradation feature extraction process employs a parallel computing architecture, with subtasks such as image block processing, color space conversion, and optical flow calculation distributed across multiple processing units for simultaneous execution. The three types of features—scattering noise distribution matrix, color distortion parameter set, and motion blur intensity quantification index—are encapsulated into a unified data structure containing feature type identifiers, feature data arrays, and timestamp information. The data interface between the feature extraction module and the dual-channel generative adversarial network uses a high-speed bus transmission protocol, ensuring that degradation feature data can be transmitted to the first channel of the network in real time. The parameter configuration information of the degradation feature extraction algorithm is stored in non-volatile memory, including key parameters such as image block size, number of clusters K, and optical flow calculation window size. These parameters can be adjusted through the configuration interface according to the actual underwater environment. The computational complexity of the feature extraction process has been optimized, employing integral image technology to accelerate local statistical calculations and using lookup tables to accelerate color space conversion operations, thereby meeting the real-time requirements of underwater robot image processing. The extracted degradation features are not only used for image restoration in the current frame, but also serve as a historical data buffer for degradation feature prediction in subsequent frames, thus forming a continuous feature evolution trajectory.

[0076] Example 2: See Figure 3 and Figure 4 The process of constructing a dual-channel generative adversarial network (GAN) begins with deploying a degenerate feature encoder in the first channel. This encoder employs a multilayer perceptron structure consisting of three fully connected layers, each followed by a LeakyReLU activation function. The three types of degenerate features—scattering noise distribution matrix, color distortion parameter set, and motion blur intensity quantization index—are flattened into one-dimensional vectors and concatenated before being input into the multilayer perceptron. The output dimension of the multilayer perceptron is fixed at 128 dimensions; this 128-dimensional vector is the conditional vector. The dimension of the conditional vector perfectly matches the conditional embedding dimension of the generator input layer in the second channel, ensuring correct information injection.

[0077] In the second channel, an adversarial structure is configured between the generator and the discriminator. The generator adopts the U-Net architecture with an encoder-decoder symmetric structure. The encoder part of the U-Net contains eight convolutional layers, each followed by a batch normalization layer and a ReLU activation function, and then performs a downsampling operation with a stride of 2. The decoder part of the U-Net contains eight transposed convolutional layers, each followed by a batch normalization layer and a ReLU activation function, and then performs an upsampling operation with a stride of 2. Skip connections concatenate the feature maps of each layer of the encoder with the feature maps of the corresponding layers of the decoder along the channel dimension. The conditional vector is embedded into the skip connections through an attention gating mechanism. The attention gating mechanism is implemented by a gated linear unit. The gated linear unit receives the feature map and the conditional vector from the encoder as input, generates an attention weight map through a sigmoid function, and multiplies the attention weight map element-wise with the original feature map to filter the feature maps related to degenerate features.

[0078] The discriminator employs a multi-scale PatchGAN architecture, which comprises three independently operating discriminant branches, each corresponding to a different image resolution scale. The first branch processes the image at its original resolution, the second branch processes the image downsampled by one time, and the third branch processes the image downsampled by two times. Each branch consists of five convolutional layers with a 4x4 kernel size and a stride of 2. Each convolutional layer is followed by a batch normalization layer and a LeakyReLU activation function. Each branch ultimately outputs a two-dimensional matrix, where each element represents a probability score of the realism of a corresponding local region in the image.

[0079] A dynamic weight allocation mechanism is designed to automatically adjust the weight ratio of the dual-channel loss function based on the peak signal-to-noise ratio (PSNR) difference between the generated and real images in each training round. The PSNR difference is obtained by calculating the mean squared error between the generated and real images and then converting it to a decibel value. The loss function weight ratio is updated using an adaptive algorithm based on gradient statistics. The generator's total loss function consists of a weighted sum of adversarial loss and content loss. The initial weights of the adversarial loss and content loss are set to 1.0 and 100.0, respectively. During training, if the PSNR difference increases, the weight of the content loss will be increased accordingly to enhance the fidelity of the image content.

[0080] In the adversarial training for generating the restored image in the second channel, the conditional vector is injected into the skip connection layer of the generator, with the injection point located before the encoder and decoder feature maps are concatenated. An attention gating mechanism applies channel attention weights to the feature maps passed from the encoder; the weighting coefficients are generated by matrix multiplication of the conditional vector with the feature map after a fully connected layer transformation. The discriminator performs multi-scale judgment on the realism of the generated image. The realism score matrix output by each discriminator branch is reduced to a scalar value, and the average of the scalar values ​​from the three discriminator branches is used as the final realism score. The matching degree between the realism score and the conditional vector is measured by calculating cosine similarity, which requires performing a dot product operation between the conditional vector and a feature vector extracted from the discriminator's intermediate layer.

[0081] Based on the matching degree between the realism score and the conditional vector, the adversarial loss and content loss of the generator are calculated. The adversarial loss adopts the least squares loss function form, and the goal is to minimize the squared difference between the generated image and the real image in the discriminator output. The specific formula for the adversarial loss is:

[0082]

[0083] in: It represents the adversarial loss of the generator, where K is the number of discriminant branches in the multi-scale PatchGAN. The k-th branch of the discriminator generates the image. Authenticity rating This is the repaired image output by the generator. This is the realism score of the discriminator's k-th branch for the real, sharp image y. For example, the discriminator scores the generated image. , , Scoring of real images , , ,but:

[0084] Pick , Given that the mean absolute pixel difference is 8.2 and the L2 distance of the feature map is 12.6, then:

[0085] The content loss consists of two parts: a pixel-level L1 loss, which calculates the absolute difference in pixel intensity between the generated and real images; and a perceptual loss, which extracts feature maps using a pre-trained VGG network and calculates the L2 distance between them. Network parameter updates employ the Adam optimization algorithm with a learning rate of 0.0002, an exponential decay rate of 0.5 for first-moment estimation, and an exponential decay rate of 0.999 for second-moment estimation. The training process uses an alternating update strategy: first, the generator parameters are fixed while the discriminator parameters are updated, then the discriminator parameters are fixed while the generator parameters are updated. The number of images processed per batch is set to 16. The number of training iterations is dynamically determined based on the loss convergence on the validation set, typically requiring at least 100,000 iterations to reach a stable state. After training, the generator parameters are saved as a binary file for subsequent real-time inference. The inference phase only requires running the generator's forward propagation process, without running the discriminator, thus meeting the real-time computational requirements of the underwater robot platform.

[0086] Example 3: See Figure 5In the stage of extracting local structural consistency features and global semantic coherence features of the restored image, a VGG-16 convolutional neural network pre-trained on the ImageNet dataset is used to extract hierarchical features of the restored image. The input layer of the pre-trained VGG-16 network receives the restored image with a size adjusted to 224x224 pixels. Shallow texture features are obtained from the output of the activation layer after the second max pooling layer of the VGG-16 network. This layer outputs a feature map with a dimension of 56x56x128, which mainly contains low-level visual information such as edges and textures. Deep semantic features are obtained from the output of the activation layer after the last convolutional layer of the VGG-16 network. This layer outputs a feature map with a dimension of 7x7x512, which encodes higher-level semantic content, such as object parts and overall shape. The similarity matrix of shallow texture features between adjacent image patches is calculated. This process divides the 56x56x128 shallow feature map into an 8x8 grid in spatial dimension, with each grid cell corresponding to a 7x7 local region. For each local region, cosine similarity is calculated between its 128-dimensional feature vector and the feature vectors of its eight neighboring regions. The cosine similarity value is calculated by dividing the dot product of the two feature vectors by the product of their magnitudes. The similarity values ​​of all local regions and their respective neighboring regions together form an 8x8x9 three-dimensional similarity matrix. This three-dimensional matrix is ​​flattened and mapped to a 256-dimensional vector through a fully connected layer. This 256-dimensional vector is defined as the local structural consistency feature, which quantifies the structural continuity and regularity between local regions of the image.

[0087] Spatial pyramid pooling is applied to the deep semantic features. The spatial pyramid pooling structure contains three pooling layers of different scales, with output sizes of 4x4, 2x2, and 1x1, respectively. The 7x7x512 deep semantic features are first input to a global average pooling layer with an output size of 4x4, producing a 4x4x512 feature tensor, which is then flattened into an 8192-dimensional vector. Similarly, the feature map is input to a global average pooling layer with an output size of 2x2, producing a 2x2x512 feature tensor, which is flattened into a 2048-dimensional vector. Finally, the feature map is input to a global average pooling layer with an output size of 1x1, producing a 512-dimensional vector. The pooling result vectors of these three different scales are concatenated to form a composite feature vector of 10752 dimensions. This composite feature vector is then compressed to 1024 dimensions through a dimension reduction fully connected layer. The final 1024-dimensional vector is defined as the global semantic coherence feature, which integrates global semantic information at multiple scales.

[0088] When using a cascaded classifier for multi-scale object detection on the restored image, the cascaded classifier consists of three cascaded detection stages. The first detector of the cascaded classifier is configured as an anchor-based region proposal network. The region proposal network slides a 3x3 convolutional kernel on the feature map of the restored image, with each spatial location corresponding to k anchor boxes of different scales and aspect ratios. The region proposal network outputs each anchor box containing the target's confidence score and the bounding box's coordinate offset. This, combined with structural information extracted from local structural consistency features, optimizes the candidate box generation process. The optimization is achieved by adjusting the overlap threshold of the non-maximum suppression algorithm, which is adaptively adjusted based on the image structural complexity reflected in the local structural consistency features.

[0089] The secondary detector employs a feature pyramid structure, which fuses feature maps from different depths of the VGG-16 network through a top-down path and lateral connections. Specifically, feature maps from the conv3_3, conv4_3, and conv5_3 layers of the VGG-16 network are upsampled and channel-aligned before being fused to form a multi-scale feature pyramid. Global semantic coherence features are then fused for target category prediction. These global semantic coherence features are transformed through a fully connected layer and concatenated with the feature vectors at each position in the feature pyramid. The concatenated feature vector is then input into a softmax classifier to predict the probability that each candidate region belongs to each target category.

[0090] The final detector performs bounding box regression and class confidence calibration. Bounding box regression uses a regression model based on smoothed L1 loss to fine-tune the center coordinates, width, and height of candidate boxes. Class confidence calibration uses the Plattscaling method, employing a logistic regression model to calibrate the initially predicted class probabilities. The logistic regression model is trained on an independent validation set. The final output includes the spatial coordinates of the target and its class label. The spatial coordinates are represented as normalized bounding boxes, and the class label includes a calibrated confidence score. All three detection stages of the cascaded classifier share the computational results of the backbone feature extraction network. This cascaded structure and feature-sharing mechanism achieve a balance between detection accuracy and computational efficiency.

[0091] A feature interaction mechanism was designed to collaboratively apply local structural consistency features and global semantic coherence features. Local structural consistency features primarily guide the region proposal network to generate higher-quality candidate boxes, particularly improving recall in image regions with complex textures and dense structures. Global semantic coherence features are mainly used to improve the accuracy of category discrimination in the secondary detection stage, especially providing supplementary contextual information when the target is partially occluded or under poor lighting conditions. The two types of features leverage their respective advantages at different stages of the cascaded classifier: local features focus on spatial localization accuracy, while global features focus on semantic robustness. The cascaded classifier is trained using a phased strategy. First, the primary region proposal network is trained with randomly initialized weights until the loss function converges. Then, the parameters of the region proposal network are fixed, and the secondary feature pyramid classification network is trained. Finally, the parameters of the first two stages are fixed, and the final-stage bounding box regression and confidence calibration networks are trained. This phased training strategy facilitates stable optimization of network parameters and avoids gradient conflicts that may occur during simultaneous training in all three stages. The training data is enhanced through methods such as horizontal flipping, random cropping, and color jittering to improve the model's adaptability to different underwater scenarios. After training, the parameters of the cascaded classifier are deployed to the underwater robot's embedded computing platform, working in conjunction with a dual-channel generative adversarial network to achieve an end-to-end processing flow from image inpainting to target recognition.

[0092] Example 4: The priority of the repair regions in the dual-channel generative adversarial network is dynamically adjusted based on the spatial distribution density of the target bounding boxes. The distribution density of the target bounding boxes in the image is statistically analyzed using a kernel density estimation method. The kernel density estimation method uses a Gaussian kernel function, with the bandwidth parameter set to 5% of the image diagonal length. The calculation process is performed in a normalized image coordinate space. Each target bounding box contributes a probability density function. The density functions of all bounding boxes are superimposed to generate a spatial attention heatmap with the same resolution as the original image. Each pixel value in the heatmap represents the probability that the location belongs to an important target region. High-priority repair regions and low-priority repair regions are divided based on the spatial attention heatmap, with the adaptive median of the heatmap values ​​used as the division threshold. Regions with values ​​higher than the adaptive median in the heatmap are marked as high-priority repair regions, which typically contain dense target clusters. Regions with values ​​lower than the adaptive median in the heatmap are marked as low-priority repair regions, which are mainly background or isolated target areas. The region division results are stored in the form of binary masks, with each mask corresponding one-to-one with the pixels of the original image. The specific formula for kernel density estimation is as follows:

[0093]

[0094] Where: n is the number of target bounding boxes detected in the image, h is the kernel function bandwidth, and (x, y) are the pixel coordinates of the density to be calculated. These are the center coordinates of the i-th target bounding box. It is a Gaussian kernel function.

[0095] For example, an image resolution of 1280×720 pixels, with a diagonal length of... Pixels Pixel (normalized h=0.05), 3 target center coordinates (0.27, 0.28), (0.43, 0.56), (0.69, 0.39). Calculate the density of pixel (0.28, 0.29):

[0096]

[0097] Prioritization is based on density, with the number of channels in the high-priority region generator increasing from 64 to 96, and the sampling rate in the low-priority region discriminator increasing from 2 to 4.

[0098] To increase the number of feature channels in high-priority restoration regions, a dynamic convolution module is introduced into the decoder of the U-Net generator. This module adaptively adjusts the number of channels in the convolution kernel based on the spatial location of the input pixels, increasing the number of channels to 1.5 times the original design in the feature maps corresponding to high-priority regions. Simultaneously, the discriminator sampling rate in low-priority regions is reduced. When processing low-priority regions, the discriminator uses strided convolution instead of regular convolution, increasing the feature map downsampling rate from 2 to 4, thus reducing computation. When generating synchronized outputs of the final restoration and target recognition results, the final restoration and target recognition results are aligned by timestamps, extracted from the metadata of the image sequence. A double-buffering mechanism is used during alignment, with the restoration result buffer and recognition result buffer managed independently. Data synchronization occurs when the timestamp difference is less than a threshold. A visualization layer of the target bounding box and category label is overlaid on the restored image. The bounding box uses RGB color encoding, with different colors used for different target categories, and the category label is rendered using an anti-aliased font. The visualization layer and the repaired image are overlaid using an alpha blending method, with an alpha value set to 0.7 to maintain the visibility of the underlying repaired image. A mapping table is established between restoration quality assessment metrics and target recognition accuracy. Restoration quality assessment metrics include structural similarity index and peak signal-to-noise ratio (PSNR). The structural similarity index evaluates the restoration effect from three aspects: brightness, contrast, and structure. PNR is a key indicator of image quality, commonly used to assess the similarity between generated and real images. Its core is to quantify the quality difference by calculating the mean square error of pixel values ​​in two images and combining it with the maximum pixel value of the image. A higher PNR value indicates a smaller deviation between the generated and real images and better quality. In this method, the PNR difference is used to dynamically adjust the weight ratio of the loss function in the dual-channel generative adversarial network to optimize the image restoration effect during model training. Target recognition accuracy is comprehensively measured by precision, recall, and F1 score. The mapping table records the correspondence between different restoration quality levels and various recognition performance metrics (precision, recall, and parameter adjustment strategies). As shown in Table 1, the mapping table uses a two-dimensional structure for storage. The row index represents the restoration quality level, the column index represents the recognition accuracy range, and the cell stores the corresponding parameter adjustment strategy. Specific restoration quality is represented by levels.

[0099] Level 1: Excellent restoration effect. The generated image is highly consistent with the real image in terms of brightness, contrast and structure. The target recognition accuracy and recall are both at a high level, and no adjustment of model parameters is required.

[0100] Level 2: The repair effect is good but there are slight deviations. The target recognition performance is slightly reduced. The number of generator channels needs to be increased appropriately to improve the ability to repair details.

[0101] Level 3: The repair effect is moderate. The image has obvious structural or color deviations, and the target recognition performance is significantly reduced. It is necessary to significantly increase the number of generator channels to enhance feature extraction.

[0102] Level 4: The restoration effect is poor, the image deviates significantly from the true value, and the target recognition performance does not meet expectations. It is necessary to comprehensively adjust the network parameters (including generator structure, loss function weights, etc.) to optimize the restoration quality.

[0103] Each level of repair quality is quantified using a structural similarity index, and correlated with the precision and recall ranges of target recognition to guide dynamic adjustment strategies for model parameters.

[0104] Table 1: Mapping Table Between Repair Quality Level and Recognition Accuracy

[0105]

[0106] The spatial attention heatmap is updated every five frames processed, using a sliding window mechanism to smooth the current heatmap by incorporating historical distribution data. A direct data path is established between the heatmap generation module and the dual-channel generative adversarial network (GAN). Heatmap data is transmitted in floating-point array format with millisecond-level latency. Priority region segmentation results are fed back to the generator's forward propagation process in real time, and the generator's convolutional layers dynamically select convolutional kernel configurations based on the priority labels of the input pixels. The rendering of the visualization layer is independent of the core inpainting algorithm, implemented using a GPU-accelerated graphics pipeline, with the rendering frame rate synchronized with the image acquisition frame rate. The font size of the category labels is adaptively adjusted based on the size of the target bounding boxes to ensure readability at different resolutions. The line width of the bounding boxes is proportional to the length of the image diagonal, maintaining visual consistency across different resolutions. The update mechanism of the association mapping table is designed as an incremental learning mode; after each processing task, the system automatically adjusts the mapping relationship based on actual performance metrics. The mapping table is stored in non-volatile memory, supporting manual correction and batch import / export, facilitating rapid configuration switching in different operating environments. The repair quality assessment module employs a parallel computing architecture, with the calculation of structural similarity index and peak signal-to-noise ratio distributed across multiple processing units simultaneously. The confidence threshold for the target recognition result is dynamically adjusted based on the repair quality. When the repair quality is high, the confidence threshold is appropriately lowered to improve recall; when the repair quality is low, the confidence threshold is raised to ensure precision. The confidence threshold adjustment algorithm uses fuzzy logic control, with the repair quality score as the input variable and the confidence threshold adjustment amount as the output variable. The entire priority adjustment system forms a closed-loop control, optimizing the allocation of repair resources based on recognition performance feedback to achieve collaborative optimization between repair and recognition. The repair quality score is divided into three fuzzy linguistic variables—"low," "mid," and "high"—using a triangular membership function, as shown in the following formula:

[0107] (1) Membership function for low repair quality

[0108]

[0109] (2) Membership function of repair quality

[0110]

[0111] (3) High-quality repair membership function

[0112]

[0113] Where: s is the repair quality score.

[0114] For example, the initial parameters for the priority adjustment system's closed-loop control are set to the initial confidence threshold of the last stage of the cascaded classifier. The restoration quality score (s) of the current frame of the 1280×720 resolution image acquired by the underwater robot compared to the original clear image is s=0.87 (medium to low restoration quality). The initial recognition effect is based on... The target recognition precision was 81% and the recall was 76%. Then, fuzzy logic control calculations were performed. First, the input variable s=0.87 was fuzzified to obtain low-repair quality membership. Membership of medium-level repair quality High restoration quality membership Then, the fuzzy rules are triggered and the strength is calculated, triggering the strength of rule 1 (low repair quality → PB). ( The intensity of trigger rule 2 (medium to low repair quality → PS) ( Other rules have a trigger strength of 0, and then the centroid method is used to defuzzify and calculate... Then, closed-loop control and feedback optimization are performed, and the confidence threshold is updated to... After the update, the recognition performance improved to 88% precision and 83% recall. Simultaneously, based on the recognition performance feedback, the allocation of repair resources was optimized. Since the target bounding boxes in the current frame are concentrated on the left side of the image (coordinate range 200-600, 150-450) and the repair quality score s=0.87 in this region still needs optimization, the number of feature channels in the dual-channel GAN ​​generator in the left region was increased from 64 to 96 (high priority), while the number of feature channels in the right region (no target) remained at 32 (low priority), reducing redundant computation. In the next closed-loop iteration, the s of the newly repaired frame increased to 0.93 (medium-high repair quality). The repeated fuzzy logic calculations yielded... The threshold was updated to 0.81-0.04=0.77, resulting in a final recognition accuracy of 86% and a recall rate of 87%, achieving synergistic optimization of repair and recognition.

[0115] The synchronous output interface supports multiple data formats, including serialized data streams for subsequent analysis, compressed image data for real-time display, and metadata files for logging. Output data packets use timestamps as unique identifiers to ensure a strict correspondence between the repair and identification results. Data packet transmission employs a checksum mechanism to guarantee the integrity and accuracy of data transmission. The system operation status monitoring module tracks resource allocation in real time, automatically reducing the processing quality of low-priority areas when computing resources are strained, while ensuring the processing effectiveness of high-priority areas.

[0116] Example 5: Real-time monitoring of the motion parameters of an underwater robot involves the collaborative work of multiple sensors. These motion parameters include three-dimensional acceleration, three-axis angular velocity, and depth data, which are collected by an inertial measurement unit (IMU) and pressure sensors mounted on the robot's main body. The acceleration data is collected at a frequency of 100 Hz, with a measurement range of ±16g. The angular velocity data is also collected at 100 Hz, with a measurement range of ±2000 degrees / second. The depth sensor uses a piezoresistive principle, achieving a measurement accuracy of 0.1%FS, and outputs data at a frequency of 50 Hz. All sensor data is transmitted to the central processing unit via a CAN bus, where timestamp alignment and coordinate system unification are performed in the data fusion module.

[0117] The degradation features of the next frame image are predicted based on attitude change data. The attitude change data represents the robot's 3D attitude using quaternions, which are obtained by fusing accelerometer and gyroscope data through Kalman filtering. The prediction process uses a multiple linear regression model. The model's input variables include the current attitude quaternion, angular velocity vector, and acceleration vector. The output variables are the predicted values ​​of scattering noise distribution, color distortion, and motion blur for the next frame image. The coefficient matrix of the multiple linear regression model is obtained through offline training using historical data, which includes the correspondence between robot motion trajectories and image degradation features under different underwater environments. The predicted degradation features are pre-input into a dual-channel generative adversarial network to pre-generate the restored image features for the next frame. This preprocessing process is completed during the interval between image sensor exposure and readout. The predicted degradation features are converted into conditional vectors by an encoder. These conditional vectors are injected into the intermediate layer of the generator, which performs partial forward propagation calculations until high-level semantic feature maps are generated. These high-level semantic feature maps are temporarily stored in a cache and can be directly used for subsequent calculations when the next frame of the original image arrives, reducing overall processing latency.

[0118] A residual connection channel is established between the restored image features and the original image sequence. This residual connection channel consists of multiple parallel paths, each corresponding to a different feature scale. The input of the residual connection channel is connected to the feature extraction layer of the original image, and the output is connected to multiple decoding layers of the generator. The channel contains learnable weight parameters. Residual information is passed at the feature level, including low-level edge features, mid-level texture features, and high-level semantic features. Features at different levels are fused through skip connections. The restoration error is fed back through the residual connection channel, calculated by comparing the restored features output by the generator with the features of the real image. Feature difference calculation is performed in multiple intermediate layers of the VGG-16 network, extracting the activation values ​​of the generated and real features in ReLU1_2, ReLU2_2, and ReLU3_3 layers respectively, and calculating the Euclidean distance between these activation values. The generator's parameter weights are dynamically updated using a stochastic gradient descent algorithm with momentum. The momentum coefficient is set to 0.9, and the learning rate is adaptively adjusted according to the error magnitude. The prediction module maintains strict time synchronization with the image acquisition system. At the start of each frame's exposure, the prediction module receives the latest motion parameters and performs degradation feature prediction. The validity period of the prediction results is set to one image acquisition cycle; after the validity period, the prediction data automatically becomes invalid to prevent the use of outdated prediction information. A prediction confidence evaluation mechanism monitors the deviation between the prediction results and the actual values ​​in real time. When the prediction deviation exceeds a threshold for multiple consecutive frames, an online model calibration process is triggered.

[0119] The weights of the residual connection channels are initialized using the Xavier method with an initial learning rate of 0.001, dynamically adjusted during training based on the decrease in the loss function. Residual connections transmit not only feature difference information but also feature uncertainty information, quantified by feature variance. Regions with high uncertainty receive lower weights during feature fusion. The generator's parameter updates employ a grouping strategy, dividing network parameters into multiple groups based on sensitivity. High-sensitivity parameters are updated with a smaller learning rate, while low-sensitivity parameters are updated with a larger learning rate. The motion state prediction module uses a sliding window mechanism to maintain historical data, with the window size set to the motion parameters of the most recent 2 seconds. The oldest frame is discarded with each new frame. The data within the window is used for both short-term prediction and model adaptation; when a significant change in motion pattern is detected, the structural parameters of the prediction model are automatically adjusted. The prediction model supports online learning, continuously collecting new samples and periodically updating model parameters to adapt to environmental changes during actual operation.

[0120] The residual feedback system is designed with a multi-loop structure, including a fast loop and a slow loop. The fast loop directly transmits low-level feature differences, offering fast response but limited accuracy, primarily used for real-time adjustment of the generator's low-level parameters. The slow loop transmits high-level semantic differences, requiring longer computation time but offering higher accuracy, used for periodically adjusting the generator's deep parameters. This dual-loop design balances system response speed and control precision, optimizing generator performance across different time scales. The preprocessing result verification mechanism immediately compares the similarity between predicted and extracted features upon receiving real image data. Cosine similarity is used for similarity calculation; when the similarity falls below a set threshold, the predicted feature is discarded and recalculated, ensuring the system continues to function correctly even when predictions fail. The verification results are also used for online evaluation of the prediction model, providing data support for model improvement.

[0121] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0122] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for real-time image restoration and target recognition of underwater robots based on GAN, characterized in that, include: The raw image sequence acquired by the underwater robot's optical sensor is collected, and the degradation features of each frame are extracted. The degradation features include scattering noise distribution, color distortion parameters, and motion blur intensity. A dual-channel generative adversarial network is constructed. The first channel is used as the conditional input for degradation features, and the second channel is used for adversarial training to generate and restore the image. The gradient update of the two channels is coordinated through a dynamic weight allocation mechanism. The original image sequence is input into a dual-channel generative adversarial network, which generates repaired images frame by frame based on degradation features, and extracts local structural consistency features and global semantic coherence features of the repaired images. A cascaded classifier is used to perform multi-scale target detection on the restored image. Combining local structural consistency features and global semantic coherence features, the target bounding box and category label are output. Based on the spatial distribution density of the target bounding box, the priority of the repair region in the dual-channel generative adversarial network is dynamically adjusted to generate a synchronized output of the final repair result and the target recognition result. The dynamic adjustment of the repair region priority in the dual-channel generative adversarial network includes: The distribution density of target bounding boxes in the image is statistically analyzed to generate a spatial attention heatmap; Based on the spatial attention heatmap, high-priority repair areas and low-priority repair areas are divided; Increase the number of feature channels of the generator in high-priority repair regions, while reducing the discriminator sampling rate in low-priority regions.

2. The method for real-time image restoration and target recognition of underwater robots based on GAN according to claim 1, characterized in that, The extraction of degradation features for each frame of image includes: The original image sequence is divided into blocks, the scattering noise energy value of each image block is calculated, and the scattering noise distribution matrix is ​​generated by combining the water body transmittance model. The color bias vector of each pixel is extracted by color space conversion, and the cluster center of the color bias vector in the CIELAB color space is counted to generate a set of color distortion parameters. Motion trajectories are estimated based on the optical flow field between adjacent frames, and quantitative indicators of motion blur intensity are calculated based on trajectory length and direction dispersion.

3. The method for real-time image restoration and target recognition of underwater robots based on GAN according to claim 2, characterized in that, The construction of the dual-channel generative adversarial network includes: A degradation feature encoder is deployed in the first channel to convert the scattering noise distribution matrix, color distortion parameter set and motion blur intensity quantization index into a conditional vector; In the second channel, an adversarial structure is configured between the generator and the discriminator. The generator adopts the U-Net architecture and embeds an attention gating mechanism for conditional vectors, while the discriminator adopts a multi-scale PatchGAN structure. A dynamic weight allocation mechanism is designed to adjust the weight ratio of the dual-channel loss function based on the difference in peak signal-to-noise ratio between the generated image and the real image in each training round.

4. The method for real-time image restoration and target recognition of underwater robots based on GAN according to claim 3, characterized in that, The adversarial training for the second channel used to generate the repaired image includes: Conditional vectors are injected into the skip connection layer of the generator, and feature maps related to degenerate features are filtered through an attention gating mechanism. The discriminator performs multi-scale judgment on the authenticity of the generated image and outputs authenticity scores at different resolutions; Based on the matching degree between the authenticity score and the conditional vector, the adversarial loss and content loss of the generator are calculated to drive the network parameter update.

5. The method for real-time image restoration and target recognition of underwater robots based on GAN according to claim 4, characterized in that, The simultaneous extraction of local structural consistency features and global semantic coherence features of the repaired image includes: The shallow texture features and deep semantic features of the restored image are extracted using a pre-trained convolutional neural network. Calculate the similarity matrix of shallow texture features between adjacent image patches to generate local structural consistency features; Spatial pyramid pooling is applied to deep semantic features, and multi-level features are fused to generate global semantic coherence features.

6. The method for real-time image restoration and target recognition of underwater robots based on GAN according to claim 5, characterized in that, The step of using a cascaded classifier to perform multi-scale target detection on the repaired image includes: The first detector of the cascaded classifier is configured as an anchor-based region proposal network, and candidate box generation is optimized by combining local structural consistency features. The secondary detector employs a feature pyramid structure, fusing global semantic coherence features to predict the target category; The final detector performs bounding box regression and class confidence calibration, and outputs the spatial location and class label of the target.

7. The method for real-time image restoration and target recognition of underwater robots based on GAN according to claim 1, characterized in that, The synchronous output of the final repair result and the target recognition result includes: Align the final repair results with the target recognition results by timestamp; Overlay a visual layer of the target bounding box and category labels onto the repaired image; Establish a mapping table between repair quality assessment indicators and target identification accuracy for subsequent model iterations.

8. The method for real-time image restoration and target recognition of underwater robots based on GAN according to claim 7, characterized in that, Also includes: Real-time monitoring of the motion state parameters of the underwater robot, and prediction of degradation features of the next frame image based on attitude change data; The predicted degradation features are pre-input into a dual-channel generative adversarial network to pre-generate the repaired image features for the next frame.

9. The method for real-time image restoration and target recognition of underwater robots based on GAN according to claim 8, characterized in that, Also includes: Establish a residual connection channel between the repaired image features and the original image sequence; Errors are corrected by feedback through residual connection channels, and the parameter weights of the generator are dynamically updated.

Images