An unmanned ship vision image local deblurring method based on adaptive weight

Abstract

In order to solve the problem of motion blur caused by sea wave motion interference in the recovery task of offshore unmanned ship, an unmanned ship visual image local deblurring method based on adaptive weight is designed. First, on the basis of the motion blur restoration algorithm, the color clustering module is added to fuse the attention mechanism for the color and edge features of the recovery cabin, so as to distinguish the target and background areas. Secondly, in the training process, dynamic adaptive weight is used, the weight of the target area and the global area is adjusted according to the restoration quality of the target area during training, and the aggregation loss function is used in combination with the dynamic weight to improve the model training quality, so that the quality of the target area is significantly improved while the global quality is considered. Finally, the model is lightened by using distillation training. The application focuses on reducing the extraction quantity of non-target features, improving the extraction efficiency of the network for target features in the picture, and using the method of distillation training to lighten the model and improve the speed of the algorithm in processing blurred pictures.

Description

Technical Field

[0001] This invention relates to the field of intelligent sensing and control equipment for unmanned systems and robots, particularly to a method for restoring visual images of unmanned vessels, and especially to a method for local deblurring of visual images of unmanned vessels based on adaptive weights. Background Technology

[0002] Motion blur image restoration is a technique that uses algorithms and technologies to improve the quality of images blurred due to the relative motion between the camera and the subject. This technology is widely used in intelligent sensing and control equipment, represented by unmanned systems and robots, such as autonomous intelligent unmanned vehicles, unmanned surface vessels, underwater salvage equipment, intelligent unmanned aerial vehicles, industrial robots, special-purpose robots, automated guided vehicles (AGVs), stacker cranes, and intelligent control systems for unmanned storage yards. Unmanned surface vessels equipped with vision devices are already widely used in maritime search and rescue, patrol, and other missions. However, unmanned vessels have limited endurance and need to return to designated locations for resupply after completing their operations. On the sea surface, multiple environmental interferences severely affect the imaging capabilities of unmanned vessels' vision, easily leading to motion blur and resulting in low processing efficiency for subsequent machine vision tasks. Therefore, how to quickly restore blurred images to clear images that meet the requirements of subsequent tasks has become a new challenge for visual operations of unmanned surface vessels at sea.

[0003] To address the above issues, the paper "Multi-Stage Progressive Image Restoration" proposes a multi-stage progressive image restoration algorithm, which has the following problems:

[0004] (1) MPRNet's deblurring module restores the entire image. Although it uses an attention mechanism to improve the extraction of important features, it does not distinguish between the target region and the background region. It focuses more on the overall image blur restoration quality, which may lead to a significant waste of computational resources in tasks such as unmanned surface vessel recovery, greatly extending the model's image restoration time. In this invention, after acquiring the preprocessed image, based on the known target information, the model focuses on the target region and enhances it through color and attention mechanisms. At the same time, it uses a dilation operation on the target region to prevent target loss or damage. This method allows the model to pay more attention to the details of the target region during image restoration while reducing the computational load of the algorithm.

[0005] (2) During training, MPRNet uses edge loss and Charbonnier loss, focusing on the overall image quality and the restoration of edge details. However, in the task of recovering unmanned marine vessels, focusing too much on the quality of background blur restoration can easily reduce the model's restoration speed, making it difficult to quickly improve image quality for subsequent tasks. In this invention, restoration loss functions for global and local target regions are used, and dynamically adjusted weights are used during training to control their proportion. This allows the model to increase its focus on the target region when the restoration quality of local target regions is poor, thereby helping to restore the image quality of specific targets in the scenario of recovering unmanned marine vessels. This achieves the effect of focusing on the restoration of local target regions while taking into account the overall image quality.

[0006] (3) Although MPRNet's design and training did not involve model lightweighting, lightweighting was considered during its design. However, actual testing still revealed that using the official weights provided by the MPRNet authors to recover images from the GoPro dataset still required a relatively long time, which could not meet the image restoration speed requirements of unmanned surface vessels in operations. In this invention, the impact of processing time was considered while restoring local target images. Therefore, this invention uses distillation training to reduce model parameters and improve the model restoration speed. The student model was obtained by modifying the model parameters, and the weights with the best performance during the original model training were used as the teacher model for distillation training. This allows the student model to retain most of the image restoration capabilities of the teacher model while also having a high image restoration speed and reducing the pressure on hardware processing capabilities.

[0007] The invention "A Local Blur Restoration Method Using Image Detection and Region Extraction" proposes a method for local blur restoration, but it has the following problems:

[0008] (1) In this invention, a method of performing blur detection on the entire image is used to mark the approximate blurry areas, and a method of image matting is used to extract the blurry areas. The blurred areas are then restored and finally integrated to obtain a locally restored image. While global blur detection can indeed target blurry areas, if the target area has high clarity but the background area has low clarity, restoring the background area would waste computational resources. This invention, however, directly focuses on the blur restoration of the target area and does not perform detailed quality enhancement on the background area, thus saving resources and increasing speed.

[0009] (2) In this invention, blurred areas in an image are recovered through iterative estimation combined with Fast Fourier Transform. In contrast, this invention uses deep learning for image restoration, employing an attention mechanism during training to focus on the target region, thus more effectively restoring its details. A dynamic comprehensive loss function is used during training, achieving good restoration of the target region while also contributing to the restoration of the background region. Summary of the Invention

[0010] This invention provides a local deblurring method for unmanned surface vessel (USV) visual images based on adaptive weights. For image blurring caused by relative motion from sea surface undulations, this method addresses the drawback of extracting unnecessary information from the sea surface as background during feature extraction, increasing computational load and reducing real-time performance. To improve the speed and accuracy of motion blur restoration algorithms for local targets in sea surface undulation environments, this invention incorporates an attention mechanism. Simultaneously, an aggregate loss function is used for training, ensuring the model prioritizes target regions from the initial setup. If local blur restoration is poor during training, the weight of the target region is increased, ensuring that the overall image quality is considered while focusing on restoring local target areas. Finally, a distillation training method is used to train the student network, reducing model parameters and improving the model's image restoration speed.

[0011] To achieve the above objectives, this invention provides a method for local deblurring of unmanned surface vessel visual images based on adaptive weights, comprising the following steps:

[0012] S1. A dataset was created using two-dimensional harmonic waves to simulate the impact of sea surface motion on images. The images in the dataset were divided into training samples and test samples according to a certain ratio. The training samples included both blurred images and images with clear targets.

[0013] S11. Take a set of 1000 clear images with markers using a high-definition camera.

[0014] S12. Estimating the fuzzy kernel for simulating the impact of sea surface motion on unmanned surface vessels using two-dimensional harmonic waves. Two-dimensional harmonic waves are represented by the following formula:

[0015] z(x,y,t)=A·cos(k x x+k y y-ωt+φ) (1)

[0016] In the formula: A is the amplitude, k x and k y The wave number determines the frequency of the wave in the x and y directions, ω is the angular frequency, and φ is the initial phase. In the creation of this dataset, motion under sea state 2 was simulated, so A was set to 0.15, wave number k to 0.21, and ω to 1 rad / s.

[0017] Camera velocities in the x and y directions:

[0018] v x (t)=-A·k x ·sin(k x x+k y y-ωt+φ) (2)

[0019] v y (t)=-A·k y ·sin(k x x+k y y-ωt+φ) (3)

[0020] The camera's trajectory over time t can be obtained by integration:

[0021]

[0022]

[0023] The formula for estimating the fuzzy kernel H(i,j) is:

[0024]

[0025] In the formula: T is the camera shutter speed, and δ is the Dirac function.

[0026] S13. Degrade a sharp image into a blurry image using a blur kernel:

[0027]

[0028] In the formula: B is the generated blurred image, I is the acquired clear image, H is the blur kernel, and N is Gaussian noise.

[0029] S14. Divide the created dataset into a training set and a test set in a 7:3 ratio.

[0030] S2. Use the model to extract multi-layer features to obtain a clear restored image:

[0031] S21. Acquire images taken by the camera, preprocess the images, and adjust them to a uniform size.

[0032] S22. Use the preprocessed image to perform shallow feature extraction. At the same time, cluster the target area and background area in the image based on the color information of the prior target.

[0033] S23. Dilate the target region to expand its influence range and make edge information more prominent. Extract shallow features from the target region image and generate a feature mask using color and edge attention mechanisms. Use a spatial attention mechanism to generate an enhanced feature map from the preprocessed shallow features. Fuse this enhanced feature map with the shallow features of the target region, further enhancing the features using color and edge attention mechanisms. Encode the enhanced features to generate a new multi-scale feature map. Input the encoder's output along with the color and edge attention mask into the decoder to obtain the decoding result. Fuse the preprocessed shallow features with the enhanced features to obtain a comprehensive feature map.

[0034] S24. The combined feature map, the multi-scale feature map obtained by the encoder, and the high-resolution image obtained by the decoder are put into ORSNet. While maintaining the original resolution, the feature maps are processed and enhanced to obtain high-dimensional feature maps.

[0035] S25. Use a convolutional layer to reduce the number of channels in the high-dimensional feature map generated by ORSNet to 3, and output an RGB image.

[0036] S3. Train the model using an aggregated loss function for both local targets and the global image to optimize the model's deblurring effect on local targets and the overall image quality:

[0037] S31. The formula for local fuzziness loss is as follows:

[0038]

[0039] In the formula: L local For local fuzziness loss, N target I represents the number of pixels in the target region. i ′ is the value of the i-th pixel in the image restored by the model, I i To determine the value of the i-th pixel in the clear image.

[0040] S32. The global fuzzy loss function is as follows:

[0041]

[0042] In the formula: L glocal For global fuzzy loss, N image This represents the number of pixels in the entire image.

[0043] S33. The weights for local loss and global loss are assigned as follows:

[0044]

[0045] λ global =1-λlocal (11)

[0046] In the formula: λ local λ represents the weight of the local loss, with a value ranging from 0.6 to 1. global This is the global loss function.

[0047] S34. The comprehensive loss function is as follows:

[0048] L=λ local L local +λ global L blobal (12)

[0049] In the formula: L represents the overall loss.

[0050] S4. The model is lightweighted by using knowledge distillation, so that the lightweight model retains most of the image restoration quality of the original model while significantly speeding up the image restoration process.

[0051] The present invention has the following beneficial effects:

[0052] 1. By using a residual structure, feature maps from multiple stages are fused and enhanced. While maintaining the original resolution of the image, the restoration quality is improved by utilizing features from multiple stages and an attention mechanism. Finally, the fused and enhanced features are converted into an output image, ensuring that the image is not only refined and enhanced but also retains basic details.

[0053] 2. By utilizing prior information on color and edge attention mechanisms, target and background regions are separated and assigned different weights. Applying these weights, more focused blur restoration is performed on the target region. An aggregated loss function is used to increase the loss weight of the target region when the restoration quality is low, reducing the impact of background restoration loss and improving the restoration quality of the target region. In terms of restoration quality, MPRNet achieves an average PSNR of 32.8, while the blur restoration model in this invention achieves 29.1. Because the restored image in this invention focuses more on the target area, the restoration of non-target areas is poor. Therefore, the average PSNR measurement result after extracting the target region separately is 31.7.

[0054] 3. A knowledge distillation method is used to lightweight the model. By reducing parameters and feature enhancement modules, the computational load is reduced, resulting in a 77% reduction in parameters compared to the original model. While retaining most of the original model's image restoration capabilities, the restoration speed of motion-blurred images retrieved by the unmanned surface vessel is improved. Compared to the original model, the lightweight model is approximately 30 times faster. In multi-image restoration experiments, the MPRNet model takes about 75 seconds to restore one image, while the lightweight model of this invention takes only 1.3 seconds. This demonstrates that the lightweight method mentioned in this invention significantly improves speed while maintaining a certain level of restoration quality for the target region. Attached Figure Description

[0055] Other features, objects, and advantages of the invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0056] Figure 1 Flowchart of the comprehensive loss function used during model training;

[0057] Figure 2 A flowchart of a method for local deblurring of visual images from unmanned surface vessels based on adaptive weights;

[0058] Figure 3 Main components of the experimental site;

[0059] Figure 4 A blurred image from a laboratory simulation of unmanned vessel recovery.

[0060] Figure 5 A clear image restored by MPRNet;

[0061] Figure 6 A clear image restored by the fuzzy restoration model in this invention. Detailed Implementation

[0062] To more clearly explain the operation and advantages of this invention, this invention provides a method for local deblurring of unmanned surface vessel visual images based on adaptive weights, specifically including the following steps:

[0063] S1. Create a suitable dataset, dividing the images in the dataset into training samples and test samples. The training samples include both blurry images to be processed and sharp images of the target. This specifically includes the following sub-steps:

[0064] S11. Take a set of 1000 clear images with markers using a high-definition camera.

[0065] S12. Unmanned surface vessels (USVs) typically experience six different motion variations during navigation on the sea surface, defined as: sway, roll, heave, pitch, and bow roll. When considering images captured by a camera fixed to the USV, the focus is primarily on the x and y axes. Therefore, a two-dimensional harmonic wave model is used to simulate the impact of sea surface motion on the USV and estimate the ambiguity kernel. The two-dimensional harmonic wave is represented by the following formula:

[0066] z(x,y,t)=A·cos(k x x+k y y-ωt+φ) (1)

[0067] In the formula: A is the amplitude, k x and k y The wave number determines the frequency of the wave in the x and y directions, ω is the angular frequency, and φ is the initial phase. In the creation of this dataset, motion under sea state 2 was simulated, so A was set to 0.15, wave number k to 0.21, and ω to 1 rad / s.

[0068] Based on the two-dimensional simple harmonic wave function, the camera's velocities in the x and y directions can be calculated:

[0069] v x (t)=-A·k x ·sin(k x x+k y y-ωt+φ) (2)

[0070] v y (t)=-A·k y ·sin(k x x+k y y-ωt+φ) (3)

[0071] Therefore, the camera's trajectory during time t can be obtained by integration:

[0072]

[0073] The formula for estimating the fuzzy kernel H(i,j) is:

[0074]

[0075] In the formula: T is the camera shutter speed, and δ is the Dirac function.

[0076] S13. Degrade a sharp image into a blurry image using a blur kernel:

[0077]

[0078] In the formula: B is the generated blurred image, I is the acquired clear image, H is the blur kernel, and N is Gaussian noise.

[0079] S14. Divide the created dataset into a training set and a test set in a 7:3 ratio.

[0080] S2. Use a motion blur restoration network to output a clear image, such as... Figure 2 As shown, the motion blur restoration network includes the following sub-steps:

[0081] S21. Acquire images taken by the camera, preprocess the images, and adjust them to a uniform size.

[0082] S22. Use the preprocessed image to perform shallow feature extraction. At the same time, cluster the target region and background region in the image based on the color information of the prior target, so that the network pays more attention to the target region and reduces attention to the background.

[0083] S23. Dilate the target region to expand its influence range and make edge information more prominent. Extract shallow features from the target region image and generate a feature mask using color and edge attention mechanisms. Use a spatial attention mechanism to generate an enhanced feature map from the preprocessed shallow features. Fuse this enhanced feature map with the shallow features of the target region, further enhancing the features using color and edge attention mechanisms. Encode the enhanced features to generate a new multi-scale feature map. Input the encoder's output along with the color and edge attention mask into the decoder to obtain the decoding result. Fuse the preprocessed shallow features with the enhanced features to obtain a comprehensive feature map.

[0084] S24. The combined feature map, the multi-scale feature map obtained by the encoder, and the high-resolution image obtained by the decoder are put into ORSNet. While maintaining the original resolution, the feature maps are processed and enhanced to obtain high-dimensional feature maps and improve the image clarity.

[0085] S25. Use a convolutional layer to reduce the number of channels in the high-dimensional feature map generated by ORSNet to 3, and output an RGB image.

[0086] S3. Train the model using an aggregated loss function targeting both local targets and the global image to optimize the model's deblurring effect on local targets and the overall image quality. Figure 1 As shown, the comprehensive loss function includes the following specific steps:

[0087] S31. Define the local blur restoration loss. This loss is specifically used to determine the degree of restoration of the target region, enabling the model to better handle the target region during training. The formula for this loss is as follows:

[0088]

[0089] In the formula: L local For local fuzziness loss, N target I represents the number of pixels in the target region. i ′ is the value of the i-th pixel in the image restored by the model, I i To determine the value of the i-th pixel in the clear image.

[0090] S32. Define the global blur restoration loss. This loss function is used to ensure that the quality of non-target areas is not too low. The global blur loss function is as follows:

[0091]

[0092] In the formula: L glocal For global fuzzy loss, N image This represents the number of pixels in the entire image.

[0093] S33. Set weights for local and global losses to ensure that the target region receives some attention from the initial conditions. If the local loss value is too high, dynamic weights can be used to make the network pay more attention to the target region with poor restoration, thereby improving the restoration quality of the target region. The weights for local and global losses are allocated as follows:

[0094]

[0095] λ global =1-λ local (11)

[0096] In the formula: λ local λ represents the weight of the local loss, with a value ranging from 0.6 to 1. global This is the global loss function.

[0097] S34. The overall loss function initially assigns equal weights to the target region and the global region. As training progresses, if the loss for the target region gradually increases, its weight also increases, making the model focus more on image restoration of the target region. The overall loss function is as follows:

[0098] L=λ local L local +λ global L blobal (12)

[0099] In the formula: L represents the overall loss.

[0100] S4. The trained model is lightweighted. The best-performing model from training is used as the teacher network. The model complexity is reduced by decreasing the number of channels, feature enhancement modules, and increasing attention mechanisms to reduce factors, thus generating a student network for knowledge distillation. This allows the student network to retain most of the image restoration capabilities of the teacher network while significantly accelerating the image restoration process.

[0101] During training, the learning rate was set to 10. -4 And a warm-up method was used to reduce the learning rate from 10 -6 Gradually increase to 10 -4 To prevent gradient explosion during training, the epoch was set to 300, the batch size to 1, and the distillation temperature to 7. After modifying the model parameters and reducing some feature processing modules, the student network had 77.33% fewer parameters compared to the teacher network. Although the lightweight model's blur restoration quality decreased compared to the teacher model, it still met the requirements for restoring markers in unmanned surface vessel retrieval. Furthermore, the lightweight model significantly improved the speed of blurry image restoration. In multiple image restoration experiments, MPRNet took approximately 75 seconds to restore one image on the same equipment, while the lightweight model of this invention took only 1.2 seconds. Meanwhile, the... Figure 5 and Figure 6 A comparison was made between the restoration models of MPRNet and this invention. Compared to MPRNet's global quality improvement, this invention focuses on enhancing the quality of the target region and moderately neglects the improvement of background quality, thereby increasing the image restoration speed. In terms of restoration quality, MPRNet achieved an average PSNR of 32.8, while the blur restoration model in this invention achieved 29.1. Because the restored image in this invention focuses more on the target area, the restoration of non-target areas is poor. Therefore, the average PSNR measurement result after extracting the target region separately is 31.7. Although the restoration quality of the target region is somewhat inferior to MPRNet, the speed is significantly improved.

[0102] The unmanned vessel recovery scenario built in the laboratory, such as... Figure 3 As shown, a camera is placed on an unmanned surface vessel (USV) to capture images, and a wave generator simulates sea surface movement. The above scenario simulates the USV operating on the sea surface, capturing blurry images, which are then restored. Figures 4-6 The comparison shows that MPRNet has a significant effect on restoring global image quality. However, the images restored by the model in this invention show that... Figure 4-6The algorithm sets the target color to the area within the red box, resulting in a significant improvement in image quality within that area. However, the area within the green box, considered background, receives a smaller weight during training, leading to less than ideal restoration results. Nevertheless, the algorithm significantly improves restoration speed, with each image restoration taking approximately 1.2 seconds. Furthermore, the restoration quality for the target area already meets the requirements for subsequent tasks. In conclusion, this algorithm achieves both high restoration speed and significant quality improvement, laying a foundation for future tasks.

[0103] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Any person skilled in the art can make some modifications or alterations to the above-disclosed technical content to create equivalent embodiments without departing from the scope of the present invention. Any simple modifications, equivalent changes, and alterations made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the scope of the present invention.

Claims

1. A method for local deblurring of visual images from unmanned surface vessels based on adaptive weights, characterized in that, Includes the following steps: S1. Create a suitable dataset and divide the images in the dataset into training samples and test samples. The training samples contain the blurry images to be processed and the clear images that serve as the targets. S2. Use a motion blur restoration network to improve the quality of the input blurred image and output a clear image. The motion blur restoration network includes the following sub-steps: S21. Acquire images captured by the camera, preprocess the images, and adjust them to a uniform size; S22. Use the preprocessed image to perform shallow feature extraction. At the same time, cluster the target region and background region in the image according to the color information of the prior target, so that the network pays more attention to the target region and reduces attention to the background. S23. Dilate the target region to expand its influence range and make the edge information more obvious; extract the shallow features of the target region image, and generate a feature mask through color and edge attention mechanisms. The preprocessed shallow features are used to generate an enhanced feature map through a spatial attention mechanism. This enhanced feature map is then fused with the shallow features of the target region, and further enhanced by combining color and edge attention mechanisms. The enhanced features are then encoded to generate a new multi-scale feature map. The encoder's output, along with the color and edge attention masks, is fed into the decoder to obtain the decoding result. Finally, the preprocessed shallow features are fused with the enhanced features to obtain a comprehensive feature map. S24. The combined feature map, the multi-scale feature map obtained by the encoder, and the high-resolution image obtained by the decoder are put into ORSNet (Original Resolution Subnetwork). While maintaining the original resolution, the feature maps are processed and enhanced to obtain high-dimensional feature maps and improve the image clarity. S25. Use a convolutional layer to reduce the number of channels in the high-dimensional feature map generated by ORSNet to 3, and output an RGB image; S3. Train the model using an aggregated loss function for both local targets and the global image to optimize the model's deblurring effect on local targets and the overall image quality. The aggregated loss function includes the following specific steps: S31. Define the local blur restoration loss. This loss is specifically used to determine the degree of restoration of the target region, enabling the model to better handle the target region during training. The formula for this loss is as follows: In the formula: L local For local fuzziness loss, N target I represents the number of pixels in the target region. i ′ is the value of the i-th pixel in the image restored by the model, I i To determine the value of the i-th pixel in the clear image; S32. Define the global blur restoration loss. This loss function is used to ensure that the quality of non-target areas is not too low. The global blur loss function is as follows: In the formula: L glocal For global fuzzy loss, N image This represents the number of pixels in the entire image. S33. Set weights for local and global losses to ensure that the target region receives some attention from the initial conditions. If the local loss value is too high, dynamic weights can be used to make the network pay more attention to the target region with poor restoration performance, thereby improving the restoration quality of the target region. The weights for local and global losses are allocated as follows: l global =1-λ local (4) In the formula: λ local λ represents the weight of the local loss, with a value ranging from 0.6 to 1. global The global loss function; S34. The overall loss function initially sets the weights of the target region and the global region to be equal. As training progresses, if the loss of the target region gradually increases, the weight of the target region will also increase, making the model focus more on image restoration of the target region. The overall loss function is as follows: L=λ local L local +λ global L blobal (5) In the formula: L represents the overall loss; S4. Perform lightweight processing on the trained model.

Images