Method for enhancing underwater continuous video frames and keeping time consistency

Abstract

The invention relates to a method for enhancing underwater continuous video frames and keeping time consistency. The method comprises the following steps: acquiring an underwater continuous video frame sequence of a video to be enhanced; sequentially inputting each underwater continuous video frame in the underwater continuous video frame sequence of the video to be enhanced into the improved U-Net network unit of the trained underwater video frame enhancement model for processing to obtain a first underwater feature map; inputting underwater continuous video frames to be enhanced into a feature fusion unit of the trained underwater video frame enhancement model for processing to obtain a second underwater feature map; performing feature series connection on the first underwater feature map and the second underwater feature map to obtain an underwater feature map after series connection; and inputting the serially connected underwater feature maps into a feature enhancement network unit of the trained underwater video frame enhancement model for processing to obtain an enhanced underwater video frame with time consistency. The time instability in the frame sequence can be effectively solved, and the quality of the enhanced underwater video is improved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a method for enhancing underwater continuous video frames and maintaining temporal consistency. Background Art

[0002] Visual information, as an easily accessible information, plays an important role in underwater robot detection and perception of the environment. Due to the large number of uncertain factors in the underwater environment, the absorption and scattering of light by water has an adverse effect on underwater optical imaging, resulting in serious degradation of the quality of underwater video images directly collected. Low-quality video images are not conducive to the perception of the human eye, and will also have an adverse effect on subsequent computer vision tasks, such as video image segmentation, target detection, 3D reconstruction and other visual processing tasks.

[0003] In practical applications, low-quality underwater video images pose a huge challenge to underwater archaeology, underwater biological research, underwater collection and other projects. How to use existing technology to obtain high-quality underwater video images is very important. Improving the underwater imaging environment, optimizing the collection equipment and other ways to strengthen the video image acquisition have certain effects, but the cost is often too high. In comparison, using computer equipment to obtain high-quality images through digital image processing is more convenient and quick.

[0004] Underwater image enhancement is an important task with broad application prospects. Most existing methods use single image methods. Although they perform satisfactorily on static images, the enhanced underwater videos suffer from severe temporal instability, resulting in poor quality of the enhanced underwater videos. Summary of the invention

[0005] Based on this, it is necessary to provide a method for enhancing underwater continuous video frames and maintaining temporal consistency, which can solve the problem that the enhanced underwater video suffers from serious temporal instability, resulting in poor quality of the enhanced underwater video, in response to the above technical problems.

[0006] A method for enhancing underwater continuous video frames and maintaining temporal consistency, the method comprising:

[0007] Obtain an underwater continuous video frame sequence of a video to be enhanced;

[0008] Inputting each underwater continuous video frame in the underwater continuous video frame sequence of the video to be enhanced into the improved U-Net network unit of the trained underwater video frame enhancement model for processing in turn to obtain a first underwater feature map;

[0009] Inputting the underwater continuous video frames to be enhanced into the feature fusion unit of the trained underwater video frame enhancement model for processing to obtain a second underwater feature map;

[0010] Performing feature concatenation on the first underwater feature map and the second underwater feature map to obtain a concatenated underwater feature map;

[0011] The series-connected underwater feature map is input into the feature enhancement network unit of the trained underwater video frame enhancement model for processing to obtain an enhanced underwater video frame with time consistency.

[0012] In one embodiment, the improved U-Net network unit includes: a first convolution layer, a second convolution layer, a third convolution layer, a fourth convolution layer, a first downsampling, a second downsampling, a third downsampling, a first feature fusion module, a second feature fusion module, a third feature fusion module, a first deconvolution layer, a second deconvolution layer, a third deconvolution layer, a fourth deconvolution layer, a first upsampling, a second upsampling, a third upsampling, and a fourth upsampling;

[0013] The step of inputting the underwater continuous video frames to be enhanced into an improved U-Net network unit of a trained underwater video frame enhancement model for processing to obtain a first underwater feature map comprises:

[0014] The underwater continuous video frames to be enhanced are input into the first convolutional layer for processing to obtain a first feature map;

[0015] Processing the first feature map sequentially through the first down-sampling, the first feature fusion module and the second convolutional layer to obtain a second feature map;

[0016] Processing the second feature map sequentially through the second down-sampling, the second feature fusion module and the third convolutional layer to obtain a third feature map;

[0017] Processing the third feature map sequentially through the third down-sampling, the third feature fusion module and the fourth convolutional layer to obtain a fourth feature map;

[0018] Processing the fourth feature map through the first deconvolution layer and the first upsampling layer in sequence to obtain a fifth feature map;

[0019] The fifth feature map and the third feature map are concatenated and processed by the second deconvolution layer and the second upsampling to obtain a sixth feature map;

[0020] The sixth feature map and the second feature map are concatenated and processed by the third deconvolution layer and the third upsampling to obtain a seventh feature map;

[0021] The seventh feature map and the first feature map are spliced ​​and then processed by the fourth deconvolution layer and the fourth upsampling to obtain a first underwater feature map.

[0022] In one of the embodiments, the feature fusion unit includes: a fifth convolutional layer and a fourth feature fusion module;

[0023] The step of inputting the underwater continuous video frames to be enhanced into the feature fusion unit of the trained underwater video frame enhancement model for processing to obtain a second underwater feature map comprises:

[0024] The underwater continuous video frames to be enhanced are processed in sequence by the fifth convolutional layer and the fourth feature fusion module to obtain a second underwater feature map.

[0025] In one of the embodiments, the feature enhancement network unit includes: n sequentially connected feature fusion modules and a sixth convolutional layer;

[0026] The step of inputting the serially connected underwater feature maps into the feature enhancement network unit of the trained underwater video frame enhancement model for processing to obtain an enhanced underwater video frame with temporal consistency comprises:

[0027] The underwater feature map after the series connection is processed by n sequentially connected feature fusion modules, and then processed by the sixth convolution layer to obtain an eighth feature map;

[0028] The concatenated underwater feature map is added to the eighth feature map to obtain an enhanced underwater video frame with temporal consistency.

[0029] In one of the embodiments, the feature fusion module includes: global average pooling, global maximum pooling, a seventh convolutional layer, an eighth convolutional layer, a ninth convolutional layer, a tenth convolutional layer, an eleventh convolutional layer, a twelfth convolutional layer and a sigmoid activation function;

[0030] The feature f input to the feature fusion module in After passing through the global maximum pooling layer, the feature f is obtained max , the feature f input to the feature fusion module in After the global average pooling layer, the feature f is obtained avg , the feature f max After the seventh convolutional layer, the feature conv 1 (f max ), the feature conv 1 (f max ) and the feature f avg After addition, the eighth convolutional layer is used to obtain the feature conv 1 (favg ), the feature conv 1 (f max ) and the feature conv 1 (f avg ) after adding the features, the feature f1 is obtained through the ninth convolutional layer. 1 (f avg ) and the feature f1 are added through the tenth convolutional layer to obtain the feature f2, and then the feature f1 and the feature f2 are concatenated and passed through the eleventh convolutional layer and the sigmoid activation function to output the feature f in Multiply them to obtain feature f3, and add the feature f3 output after passing through the twelfth convolutional layer to the feature f in Add together to get the feature f out , the feature f out is the output of the feature fusion module.

[0031] In one embodiment, the underwater video frame enhancement model is trained as follows:

[0032] Step 1: obtaining an underwater video frame sample set, wherein each underwater video frame sample in the underwater video frame sample set includes: a clear underwater video frame, a degraded underwater video frame of the clear underwater video frame, and optical flow information of the clear underwater video frame;

[0033] Step 2: inputting the degraded underwater video frame of the clear underwater video frame into a first underwater video frame enhancement model to perform video frame enhancement to obtain a first enhancement result;

[0034] Step 3: using the optical flow information of the clear underwater video frame to distort the first enhancement result to obtain a distorted first enhancement result;

[0035] Step 4: after distorting the degraded underwater video frame using the optical flow information of the clear underwater video frame, inputting the second underwater video frame enhancement model to perform video frame enhancement to obtain a second enhancement result, wherein the network structures of the first underwater video frame enhancement model and the second underwater video frame enhancement model are the same and the training weights are shared;

[0036] Step 5: according to the distorted first enhancement result, the second enhancement result and the clear underwater video frame, the loss is calculated using an l1 loss function to determine a loss value;

[0037] Step 6: updating and optimizing the network parameters of the first underwater video frame enhancement model and the second underwater video frame enhancement model according to the loss value;

[0038] Step 7: Repeat steps 2 to 5 until the training termination condition is met to obtain a trained underwater video frame enhancement model.

[0039] In one embodiment, the l1 loss function is expressed as:

[0040]

[0041] in, is the loss value, To enhance the loss, is the consistency loss, λ is the weight, g(x1) is the first enhancement result, y1 is the clear underwater video frame, g(x2) is the second enhancement result, y2 is the video frame after the clear underwater video frame is distorted by the optical flow information of the clear underwater video frame, and W(g(x1),f) is the image after the first enhancement result is distorted by the optical flow information of the clear underwater video frame.

[0042] In one embodiment, a method of obtaining each underwater video frame sample in the underwater video frame sample set is:

[0043] Get clear underwater video frames;

[0044] Using CycleGAN to perform degradation processing on the clear underwater video frame to obtain a degraded underwater video frame of the clear underwater video frame;

[0045] Inputting the clear underwater video frame into an underwater mask model to perform instance segmentation to obtain target mask information of the segmented instance;

[0046] The target mask information and the clear underwater video frame are input into a conditional propagation model to obtain optical flow information of the clear underwater video frame.

[0047] The above method for enhancing underwater continuous video frames and maintaining temporal consistency is as follows: obtaining an underwater continuous video frame sequence of the video to be enhanced; sequentially inputting each underwater continuous video frame in the underwater continuous video frame sequence of the video to be enhanced into the improved U-Net network unit of the trained underwater video frame enhancement model for processing to obtain a first underwater feature map; inputting the underwater continuous video frame to be enhanced into the feature fusion unit of the trained underwater video frame enhancement model for processing to obtain a second underwater feature map; performing feature concatenation on the first underwater feature map and the second underwater feature map to obtain a concatenated underwater feature map; inputting the concatenated underwater feature map into the feature enhancement network unit of the trained underwater video frame enhancement model for processing to obtain an enhanced underwater video frame with temporal consistency. Thus, by improving the ability of the U-Net network unit to capture all useful multi-scale features when restoring underwater images, the temporal consistency of the video frame is enhanced, and the temporal instability in the frame sequence can be effectively resolved, thereby improving the quality of the enhanced underwater video. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 A schematic flow chart of a method for enhancing underwater continuous video frames and maintaining temporal consistency in one embodiment;

[0049] Figure 2 A schematic diagram of the structure of an underwater video frame enhancement model in one embodiment;

[0050] Figure 3 Schematic diagram of the structure of a feature fusion module of an underwater video frame enhancement model in one embodiment;

[0051] Figure 4 A schematic diagram of the structure of a feature enhancement network unit of an underwater video frame enhancement model in one embodiment;

[0052] Figure 5 (a) is a degraded underwater video frame in an underwater video frame sample in an embodiment;

[0053] Figure 5 (b) is a segmentation example of a clear underwater video frame in an underwater video frame sample in an embodiment;

[0054] Figure 5 (c) is an optical flow prediction image of a clear underwater video frame in an underwater video frame sample in an embodiment;

[0055] Figure 5 (d) is a clear underwater video frame among underwater video frame samples in one embodiment;

[0056] Figure 6 A schematic diagram of a process of optical flow prediction in one embodiment;

[0057] Figure 7 The figure is a schematic diagram of the training process of an underwater video frame enhancement model in one embodiment. DETAILED DESCRIPTION

[0058] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0059] In one embodiment, Figure 1 and Figure 2 As shown, a method for enhancing underwater continuous video frames and maintaining temporal consistency is provided, comprising the following steps:

[0060] Step S220, obtaining a sequence of underwater continuous video frames of the video to be enhanced.

[0061] Step S240, sequentially input each underwater continuous video frame in the underwater continuous video frame sequence of the video to be enhanced into the improved U-Net network unit of the trained underwater video frame enhancement model for processing to obtain a first underwater feature map.

[0062] Among them, the underwater video frame enhancement model is a two-stage network. The first stage uses the improved U-Net (i.e., the improved U-Net network unit). The network structure of the improved U-Net network unit is fully convolutional, does not use a fully connected layer, has fewer model parameters and faster processing speed. Taking into account the detail loss generated by the existing U-Net network during the encoding-decoding process, this application adds a feature fusion module to each downsampling process of the existing U-Ne t network. The feature fusion strategy in the feature fusion module can integrate information at different scales, so that the network can capture more local and global information, and can more effectively process the complex features in the underwater video frame, such as light refraction, color deviation and dynamic objects, thereby providing richer features for the reconstruction and enhancement of the video frame. In addition, through the refined processing of each layer and a deeper understanding of global information, the time series derived from a single frame is made more accurate and coherent. At the same time, the second stage is a feature fusion unit and a feature enhancement network unit. In the second stage, an enhancement network unit is introduced. The enhancement network unit is composed of n feature fusion modules and a convolution layer with a convolution kernel of 3 (i.e., the sixth convolution layer) in series, and the output of the first stage is used as the input of the second stage. The network refines the input features on the video frame of the original resolution size, so that the generated image has richer detail features.

[0063] Among them, the number of feature fusion modules in the enhanced network unit can be changed accordingly according to the complexity of the task. In one embodiment, the best effect is achieved by connecting three feature fusion modules in series.

[0064] like Figure 2 As shown, in one embodiment, the improved U-Net network unit includes: a first convolution layer, a second convolution layer, a third convolution layer, a fourth convolution layer, a first down-sampling, a second down-sampling, a third down-sampling, a first feature fusion module, a second feature fusion module, a third feature fusion module, a first deconvolution layer, a second deconvolution layer, a third deconvolution layer, a fourth deconvolution layer, a first up-sampling, a second up-sampling, a third up-sampling, and a fourth up-sampling.

[0065] In one embodiment, the underwater continuous video frames to be enhanced are input into the improved U-Net network unit of the trained underwater video frame enhancement model for processing to obtain a first underwater feature map, including:

[0066] The underwater continuous video frames to be enhanced are input into the first convolution layer for processing to obtain a first feature map; the first feature map is processed by the first downsampling, the first feature fusion module and the second convolution layer in sequence to obtain a second feature map; the second feature map is processed by the second downsampling, the second feature fusion module and the third convolution layer in sequence to obtain a third feature map; the third feature map is processed by the third downsampling, the third feature fusion module and the fourth convolution layer in sequence to obtain a fourth feature map; the fourth feature map is processed by the first deconvolution layer and the first upsampling in sequence to obtain a fifth feature map; the fifth feature map and the third feature map are spliced ​​and processed by the second deconvolution layer and the second upsampling to obtain a sixth feature map; the sixth feature map and the second feature map are spliced ​​and processed by the third deconvolution layer and the third upsampling to obtain a seventh feature map; the seventh feature map and the first feature map are spliced ​​and processed by the fourth deconvolution layer and the fourth upsampling to obtain a first underwater feature map.

[0067] Among them, the first feature fusion module, the second feature fusion module and the third feature fusion module are feature fusion modules with the same structure.

[0068] like Figure 3 As shown, in one embodiment, the feature fusion module includes: global average pooling, global maximum pooling, a seventh convolution layer, an eighth convolution layer, a ninth convolution layer, a tenth convolution layer, an eleventh convolution layer, a twelfth convolution layer and a sigmoid activation function;

[0069] The feature f input to the feature fusion module in After the global maximum pooling layer, the feature f is obtained max , the feature f input to the feature fusion module inAfter the global average pooling layer, the feature f is obtained avg , feature f max After the seventh convolutional layer, the feature conv 1 (f max ), feature conv 1 (f max ) and feature f avg After addition, the eighth convolutional layer is used to obtain the feature conv 1 (f avg ), feature conv 1 (f max ) and feature conv 1 (f avg ) After adding the features, the ninth convolutional layer is used to obtain the feature f1, conv 1 (f avg ) and feature f1 are added through the tenth convolutional layer to obtain feature f2, and then feature f1 and feature f2 are concatenated and passed through the eleventh convolutional layer and sigmoid activation function to output the same feature f in Multiply them to get feature f3, and add the feature f3 output after passing through the twelfth convolutional layer to feature f in Add together to get the feature f out , feature f out is the output of the feature fusion module.

[0070] Among them, the seventh convolutional layer is a convolutional layer with a convolution kernel of 1.

[0071] Among them, the eighth convolution layer is a convolution layer with a convolution kernel of 1.

[0072] Among them, the ninth convolution layer is a convolution layer with a convolution kernel of 3.

[0073] Among them, the tenth convolutional layer is a convolutional layer with a convolution kernel of 3.

[0074] Among them, the eleventh convolutional layer has a convolution kernel of 1.

[0075] Among them, the twelfth convolutional layer is a convolutional layer with a convolution kernel of 3.

[0076] Among them, the feature fusion module is used to fuse multiple features. The feature fusion module uses two pooling layers, global average pooling and global maximum pooling, to extract features. Different types of global pooling layers can extract rich features. After the feature information is pooled by global average pooling and global maximum pooling, the feature fusion is performed in the same way but with different paths. The calculation formula of feature f3 is as follows:

[0077] f3=f in ·sigmoid(Conv 1 (Cat(f1,f2)))

[0078] Among them, sigmoid represents the sigmoid activation function, Conv 1 represents a convolution layer with a convolution kernel of 1, Cat represents feature concatenation, and f in To input the features into the feature fusion module, the calculation process of feature f1 and feature f2 is as follows:

[0079]

[0080] Among them, conv 3 Indicates that the convolution kernel is 3 convolution layers, and the input feature f in After the global maximum pooling layer and the global average pooling layer, the feature f is obtained. max and f avg , f max After the convolution layer with a convolution kernel of 1, the feature conv is obtained 1 (f max ), conv 1 (f max ) and f avg After addition, the feature conv is obtained by passing through the convolution layer with a convolution kernel of 1. 1 (f avg ), conv 1 (f max ) and conv 1 (f avg ) After adding the features, the feature f1 is obtained through a convolution layer with a convolution kernel of 3. 1 (f avg ) is added to f1 and passed through a convolution layer with a convolution kernel of 3 to obtain feature f2, and then features f1 and f2 are concatenated. The advantage of this is that the features extracted by global average pooling and global maximum pooling are integrated, and different convolutional layers, pooling operations and feature fusion techniques are used to extract and process input features. Through gradually refined processing, the underwater video frame enhancement model's ability to understand and express input data is enhanced, and distorted information is better reconstructed. At the same time, the weight of global information features in the video frame is increased, which helps to improve the temporal consistency of the video frame sequence, thereby reducing flickering and jumping.

[0081] Step S260: input the underwater continuous video frames to be enhanced into the feature fusion unit of the trained underwater video frame enhancement model for processing to obtain a second underwater feature map.

[0082] like Figure 2As shown, in one embodiment, the feature fusion unit includes: a fifth convolutional layer and a fourth feature fusion module; the underwater continuous video frames to be enhanced are input into the feature fusion unit of the trained underwater video frame enhancement model for processing to obtain a second underwater feature map, including: the underwater continuous video frames to be enhanced are processed in turn by the fifth convolutional layer and the fourth feature fusion module to obtain the second underwater feature map.

[0083] Among them, the fourth feature fusion module is a feature fusion module with the same structure as the first feature fusion module, the second feature fusion module and the third feature fusion module.

[0084] Step S280, performing feature concatenation on the first underwater feature map and the second underwater feature map to obtain a concatenated underwater feature map.

[0085] Step S300, input the concatenated underwater feature map into the feature enhancement network unit of the trained underwater video frame enhancement model for processing to obtain an enhanced underwater video frame with time consistency.

[0086] like Figure 4 As shown, in one embodiment, the feature enhancement network unit includes: n sequentially connected feature fusion modules and a sixth convolutional layer; the series-connected underwater feature map is input into the feature enhancement network unit of the trained underwater video frame enhancement model for processing to obtain an enhanced underwater video frame with temporal consistency, including: the series-connected underwater feature map is processed by n sequentially connected feature fusion modules, and then processed by the sixth convolutional layer to obtain an eighth feature map; the series-connected underwater feature map is added to the eighth feature map to obtain an enhanced underwater video frame with temporal consistency.

[0087] Among them, the underwater continuous video frame sequence of the video to be enhanced is passed through an underwater video frame enhancement model with trained weights, and features are extracted and encoded and decoded to output enhanced underwater video frames arranged in time series while maintaining their time stability, thereby solving the problem of flickering or jumping in the enhanced image.

[0088] It should be understood that after an underwater continuous video frame is enhanced by the trained underwater video frame enhancement model, a corresponding enhanced underwater video frame with time consistency is obtained. After each underwater continuous video frame of the underwater continuous video frame sequence of the video to be enhanced is sequentially input into the trained underwater video frame enhancement model for enhancement processing according to the time sequence, a corresponding enhanced underwater video frame sequence with time consistency is obtained, thereby obtaining an enhanced underwater video.

[0089] It should be understood that the underwater video frame enhancement model uses different convolutional layers, pooling operations and feature fusion techniques to extract and process input features. Through gradually refined processing, the underwater video frame enhancement model enhances its ability to understand and express input data, and better reconstructs distorted color information. Among them, the improved U-Net network unit has the ability to capture all useful multi-scale features when restoring underwater images. A method of continuous video frames is used to enhance the temporal stability of underwater video enhancement, learn and infer the motion field (i.e., optical flow information) from a single image, and synthesize a short-range video sequence. By predicting motion optical flow, the temporal consistency of video frames is enhanced, which can effectively solve the temporal instability in frame sequences, and can also have good effects in other visual aspects, with excellent generalization ability.

[0090] The above method for enhancing underwater continuous video frames and maintaining temporal consistency is as follows: obtaining an underwater continuous video frame sequence of the video to be enhanced; sequentially inputting each underwater continuous video frame in the underwater continuous video frame sequence of the video to be enhanced into the improved U-Net network unit of the trained underwater video frame enhancement model for processing to obtain a first underwater feature map; inputting the underwater continuous video frame to be enhanced into the feature fusion unit of the trained underwater video frame enhancement model for processing to obtain a second underwater feature map; performing feature concatenation on the first underwater feature map and the second underwater feature map to obtain a concatenated underwater feature map; inputting the concatenated underwater feature map into the feature enhancement network unit of the trained underwater video frame enhancement model for processing to obtain an enhanced underwater video frame with temporal consistency. Thus, by improving the ability of the U-Net network unit to capture all useful multi-scale features when restoring underwater images, the temporal consistency of the video frame is enhanced, and the temporal instability in the frame sequence can be effectively resolved, thereby improving the quality of the enhanced underwater video.

[0091] In one embodiment, the underwater video frame enhancement model is trained as follows:

[0092] Step 1: Obtain an underwater video frame sample set, where each underwater video frame sample in the underwater video frame sample set includes: a clear underwater video frame, a degraded underwater video frame of the clear underwater video frame, and optical flow information of the clear underwater video frame;

[0093] Step 2: inputting the degraded underwater video frame of the clear underwater video frame into the first underwater video frame enhancement model to perform video frame enhancement to obtain a first enhancement result;

[0094] Step 3: Using the optical flow information of the clear underwater video frame to distort the first enhancement result, to obtain a distorted first enhancement result;

[0095] Step 4: after distorting the degraded underwater video frame using the optical flow information of the clear underwater video frame, inputting the second underwater video frame enhancement model to perform video frame enhancement to obtain a second enhancement result, wherein the network structure of the first underwater video frame enhancement model and the second underwater video frame enhancement model are the same and the training weights are shared;

[0096] Step 5: According to the distorted first enhancement result, the second enhancement result and the clear underwater video frame, the loss is calculated using the l1 loss function to determine the loss value;

[0097] Step 6: Update and optimize the network parameters of the first underwater video frame enhancement model and the second underwater video frame enhancement model according to the loss value;

[0098] Step 7: Repeat steps 2 to 5 until the training termination condition is met to obtain a trained underwater video frame enhancement model.

[0099] Among them, the clear underwater video frames are instance segmented through an underwater mask model (such as the WaterMask model), and the target mask information of the segmented instances can be directly obtained. The clear underwater video frames and the target mask information are input into the conditional propagation model to obtain the optical flow information of the clear underwater video frames. The optical flow information and the degraded underwater video frames are input into the underwater video frame enhancement model that needs to be trained for supervised training.

[0100] It should be understood that the first underwater video frame enhancement model and the second underwater video frame enhancement model have the same network structure, the training weights will be shared, and only the input is different, so as to perform supervised training. After the training is completed, the trained first underwater video frame enhancement model can be used as the trained underwater video frame enhancement model.

[0101] In one embodiment, the l1 loss function is expressed as:

[0102]

[0103] in, is the loss value, To enhance the loss, is the consistency loss, λ is the weight, g(x1) is the first enhancement result, y1 is the clear underwater video frame, g(x2) is the second enhancement result, y2 is the video frame after the clear underwater video frame is distorted by the optical flow information of the clear underwater video frame, and W(g(x1),f) is the image after the first enhancement result is distorted by the optical flow information of the clear underwater video frame.

[0104] In one embodiment, each underwater video frame sample in the underwater video frame sample set is obtained in the following manner:

[0105] Obtain clear underwater video frames; use CycleGAN to perform degradation processing on the clear underwater video frames to obtain degraded underwater video frames of the clear underwater video frames; input the clear underwater video frames into the underwater mask model for instance segmentation to obtain target mask information of the segmented instances; input the target mask information and the clear underwater video frames into the conditional propagation model to obtain optical flow information of the clear underwater video frames.

[0106] Among them, the existing paired underwater video frames (i.e., clear underwater video frames, degraded underwater video frames of clear underwater video frames, and optical flow information of clear underwater video frames) are difficult to obtain. Clear 4K underwater video frames can be extracted from videos publicly available on the Internet as clear underwater video frames. Then, the unsupervised CycleGAN network is used to perform degradation processing using unpaired clear underwater video frames of different styles to obtain degraded underwater video frames that meet the requirements of different styles and paired clear underwater video frames. Before training the underwater video frame enhancement model, a reasonable optical flow is first predicted based on the good quality ground truth, and the underwater mask model can be used for processing to obtain the optical flow information.

[0107] In one example, a carefully selected Figure 5 Some of the 40 underwater videos shown, more than 4,000 clear underwater video frames that can meet the natural changes of scenes, water types, lighting conditions, etc. in the data.

[0108] In one example, if Figure 6 As shown, the underwater mask model of the present application uses the WaterMask segmentation model to perform instance segmentation on clear underwater video frames. After being processed by the WaterMask segmentation model, the required mask data format is converted based on the visual image of the segmented instance. First, the clear underwater video frame is input into the WaterMask segmentation model to obtain the target mask information of the segmented instance. Secondly, the target mask information and the clear underwater video frame are input into the conditional propagation model to obtain the optical flow information of the clear underwater video frame. The optical flow information of the clear underwater video frame is used for the training of the underwater video frame enhancement model. The purpose is to add timing information when training the underwater video frame enhancement model, so that the network can learn the timing characteristics and obtain a temporally stable enhanced output.

[0109] It should be understood that the optical flow information is used to represent the motion occurring between video frames of dynamic scenes, and the images of adjacent frames are simulated by performing corresponding optical flow distortions on the images. Given the original image and the distorted image pair, the underwater video frame enhancement model is trained in a Siamese manner, and they are input into the underwater video frame enhancement model one by one. By imposing consistency between the output pairs, it can help the network remain stable in time.

[0110] In one embodiment, Figure 7 As shown in the figure, after the necessary optical flow information is prepared, the image-based underwater video frame enhancement model is trained in a conjoined manner. The upper part of the pipeline is the first pass of the underwater video frame enhancement model. Similar to the general training process, the degraded underwater video frame x1 in the training underwater video frame sample set is input into the underwater video frame enhancement model g(·), and the enhancement result g(x1) (i.e., the first enhancement result) is predicted. The network learns to output the enhanced underwater video frame with the help of the supervision of the corresponding clear underwater video frame, i.e., the enhancement result g(x1). In order to provide more time information, the input degraded underwater video frame x1 is optically distorted using the random optical flow information f predicted based on the clear underwater video frame. The distorted image x2 is used as the input of the second pass of the lower part of the pipeline. The output enhancement result g(x2) (i.e., the second enhancement result) is also compared with the corresponding clear underwater video frame distorted by the optical flow for supervised training. The training weights of the two branches are shared. Finally, the output g(x1) is distorted into W(g(x1),f) using the same optical flow information f and compared with the output g(x2). The calculation formula is as follows:

[0111] f=CMP(y,V)

[0112] x2=W(x1,f)

[0113] Wherein, CMP is the conditional propagation model, y and V represent the clear underwater video frame and the brake optical flow vector respectively, f represents the optical flow information of the clear underwater video frame, x1 and x2 are the degraded underwater video frame and the image of the degraded underwater video frame distorted by the optical flow information of the clear underwater video frame respectively;

[0114] Among them, all losses use the l1 loss function, and the loss used to train the underwater video frame enhancement model can be defined as the enhancement loss and consistency loss Combination of:

[0115]

[0116] Among them, λ is the weight that balances the constraints of the two loss parts. Specifically, the enhancement loss and consistency loss It can be expressed as:

[0117]

[0118] Among them, g(x1) is the first enhancement result, y1 is the clear underwater video frame, g(x2) is the second enhancement result, y2 is the video frame after the clear underwater video frame is distorted by using the optical flow information of the clear underwater video frame, and W(g(x1),f) is the image after the first enhancement result is distorted by using the optical flow information of the clear underwater video frame.

[0119] Beneficial effects: During use, an underwater video frame sample set is obtained, and each underwater video frame sample in the underwater video frame sample set includes: a clear underwater video frame, a degraded underwater video frame of the clear underwater video frame, and optical flow information of the clear underwater video frame. These data can meet the natural changes in the scene, water type, lighting conditions, etc. in the data; then a feature fusion module is designed, and the feature fusion module uses different convolutional layers, pooling operations and feature fusion technologies to extract and process input features, which can gradually and finely process the input data, thereby enhancing the underwater video frame enhancement model's ability to understand and express data; the feature fusion module is combined with the improved U-Net to form a new network model, namely the underwater video frame enhancement model, which can retain more important feature information during the downsampling process and more effectively process complex features in underwater video frames, such as light refraction, color deviation and dynamic objects. The underwater video frame enhancement model uses layer-by-layer enhancement processing to make the time series derived from a single frame more accurate and coherent, and further enhance the smooth transition between video frames, reducing flicker and time jitter; by predicting motion optical flow, the temporal consistency of video frames is enhanced, which can effectively solve the temporal instability in frame sequences. It also has good results in other video enhancement fields and has excellent generalization ability.

[0120] It should be understood that although Figure 1 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 1 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0121] The technical features of the above embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0122] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A method for enhancing underwater continuous video frames and maintaining temporal consistency, characterized in that: The method comprises: Obtain an underwater continuous video frame sequence of the video to be enhanced; Inputting each underwater continuous video frame in the underwater continuous video frame sequence of the video to be enhanced into the improved U-Net network unit of the trained underwater video frame enhancement model for processing in turn to obtain a first underwater feature map; Inputting the underwater continuous video frames to be enhanced into the feature fusion unit of the trained underwater video frame enhancement model for processing to obtain a second underwater feature map; Performing feature concatenation on the first underwater feature map and the second underwater feature map to obtain a concatenated underwater feature map; The series-connected underwater feature map is input into the feature enhancement network unit of the trained underwater video frame enhancement model for processing to obtain an enhanced underwater video frame with time consistency.

2. The method according to claim 1, characterized in that: The improved U-Net network unit includes: a first convolution layer, a second convolution layer, a third convolution layer, a fourth convolution layer, a first down-sampling, a second down-sampling, a third down-sampling, a first feature fusion module, a second feature fusion module, a third feature fusion module, a first deconvolution layer, a second deconvolution layer, a third deconvolution layer, a fourth deconvolution layer, a first up-sampling, a second up-sampling, a third up-sampling, and a fourth up-sampling; The step of inputting the underwater continuous video frames to be enhanced into an improved U-Net network unit of a trained underwater video frame enhancement model for processing to obtain a first underwater feature map comprises: The underwater continuous video frames to be enhanced are input into the first convolutional layer for processing to obtain a first feature map; Processing the first feature map sequentially through the first down-sampling, the first feature fusion module and the second convolutional layer to obtain a second feature map; Processing the second feature map sequentially through the second down-sampling, the second feature fusion module and the third convolutional layer to obtain a third feature map; Processing the third feature map sequentially through the third down-sampling, the third feature fusion module and the fourth convolutional layer to obtain a fourth feature map; Processing the fourth feature map through the first deconvolution layer and the first upsampling layer in sequence to obtain a fifth feature map; The fifth feature map and the third feature map are concatenated and processed by the second deconvolution layer and the second upsampling to obtain a sixth feature map; The sixth feature map and the second feature map are concatenated and processed by the third deconvolution layer and the third upsampling to obtain a seventh feature map; The seventh feature map and the first feature map are spliced ​​and then processed by the fourth deconvolution layer and the fourth upsampling to obtain a first underwater feature map.

3. The method according to claim 1, characterized in that The feature fusion unit includes: a fifth convolutional layer and a fourth feature fusion module; The step of inputting the underwater continuous video frames to be enhanced into the feature fusion unit of the trained underwater video frame enhancement model for processing to obtain a second underwater feature map comprises: The underwater continuous video frames to be enhanced are processed in sequence by the fifth convolutional layer and the fourth feature fusion module to obtain a second underwater feature map.

4. The method according to claim 1, characterized in that: The feature enhancement network unit includes: n sequentially connected feature fusion modules and a sixth convolutional layer; The step of inputting the serially connected underwater feature maps into the feature enhancement network unit of the trained underwater video frame enhancement model for processing to obtain an enhanced underwater video frame with temporal consistency comprises: The underwater feature map after the series connection is processed by n sequentially connected feature fusion modules, and then processed by the sixth convolution layer to obtain an eighth feature map; The concatenated underwater feature map is added to the eighth feature map to obtain an enhanced underwater video frame with temporal consistency.

5. The method according to claim 4, characterized in that The feature fusion module includes: global average pooling, global maximum pooling, a seventh convolution layer, an eighth convolution layer, a ninth convolution layer, a tenth convolution layer, an eleventh convolution layer, a twelfth convolution layer and a sigmoid activation function; The feature f input to the feature fusion module in After passing through the global maximum pooling layer, the feature f is obtained max , the feature f input to the feature fusion module in After the global average pooling layer, the feature f is obtained avg , the feature f max After the seventh convolutional layer, the feature conv 1 (f max ), the feature conv 1 (f max ) and the feature f avg After addition, the eighth convolutional layer is used to obtain the feature conv 1 (f avg ), the feature conv 1 (f max ) and the feature conv 1 (f avg ) after adding the features, the feature f1 is obtained through the ninth convolutional layer. 1 (f avg ) and the feature f1 are added through the tenth convolutional layer to obtain the feature f2, and then the feature f1 and the feature f2 are concatenated and passed through the eleventh convolutional layer and the sigmoid activation function to output the feature f in Multiply them to obtain feature f3, and add the feature f3 output after passing through the twelfth convolutional layer to the feature f in Add together to get the feature f out , the feature f out is the output of the feature fusion module.

6. The method according to claim 1, characterized in that The training method of the underwater video frame enhancement model is: Step 1: obtaining an underwater video frame sample set, wherein each underwater video frame sample in the underwater video frame sample set includes: a clear underwater video frame, a degraded underwater video frame of the clear underwater video frame, and optical flow information of the clear underwater video frame; Step 2: inputting the degraded underwater video frame of the clear underwater video frame into a first underwater video frame enhancement model to perform video frame enhancement to obtain a first enhancement result; Step 3: using the optical flow information of the clear underwater video frame to distort the first enhancement result to obtain a distorted first enhancement result; Step 4: after distorting the degraded underwater video frame using the optical flow information of the clear underwater video frame, inputting the second underwater video frame enhancement model to perform video frame enhancement to obtain a second enhancement result, wherein the network structures of the first underwater video frame enhancement model and the second underwater video frame enhancement model are the same and the training weights are shared; Step 5: according to the distorted first enhancement result, the second enhancement result and the clear underwater video frame, using the l1 loss function to calculate the loss and determine the loss value; Step 6: updating and optimizing the network parameters of the first underwater video frame enhancement model and the second underwater video frame enhancement model according to the loss value; Step 7: Repeat steps 2 to 5 until the training termination condition is met to obtain a trained underwater video frame enhancement model.

7. The method according to claim 6, characterized in that The l1 loss function is expressed as: in, is the loss value, To enhance the loss, is the consistency loss, λ is the weight, g(x1) is the first enhancement result, y1 is the clear underwater video frame, g(x2) is the second enhancement result, y2 is the video frame after the clear underwater video frame is distorted by the optical flow information of the clear underwater video frame, and W(g(x1),f) is the image after the first enhancement result is distorted by the optical flow information of the clear underwater video frame.

8. The method according to claim 6, characterized in that The method of obtaining each underwater video frame sample in the underwater video frame sample set is: Get clear underwater video frames; Using CycleGAN to perform degradation processing on the clear underwater video frame to obtain a degraded underwater video frame of the clear underwater video frame; Inputting the clear underwater video frame into an underwater mask model to perform instance segmentation to obtain target mask information of the segmented instance; The target mask information and the clear underwater video frame are input into a conditional propagation model to obtain optical flow information of the clear underwater video frame.

Images