Unmanned aerial vehicle video image rain removal method and device based on potential frequency domain representation

Abstract

The invention relates to the technical field of image processing, and discloses an unmanned aerial vehicle video image rain removal method and device based on potential frequency domain characterization, and the method comprises the steps: carrying out the image comparison regularization processing of a complex dynamic video image of an unmanned aerial vehicle in rainy days, and obtaining a contrast regularization image; according to the contrast regularized image, guiding a pre-constructed potential frequency domain representation extraction network to extract potential frequency domain representation of the complex dynamic video image of the unmanned aerial vehicle in rainy days; performing dual-frequency potential characterization enhancement on the potential frequency domain characterization to obtain enhanced characterization; and performing representation aggregation image restoration according to the enhanced representation to obtain a rain-removed image. According to the unmanned aerial vehicle video image rain removal method, the image definition of the unmanned aerial vehicle in a complex dynamic environment in rainy days can be effectively improved, the robust and efficient video image rain removal effect of the unmanned aerial vehicle in the complex dynamic scene in rainy days is achieved, the visual effect of the image is finally improved, and a more accurate rain removal image is obtained.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a method and device for removing rain from unmanned aerial vehicle video images based on potential frequency domain representation. Background Art

[0002] Drone video image deraining technology has important applications in many fields, such as video surveillance, autonomous driving, medical imaging, etc. It can improve image clarity and ensure the normal operation of the visual system. However, in severe weather conditions (such as rainy days), the video images taken by drones will be disturbed by raindrops, resulting in blurred images and reduced contrast, which seriously affects the performance of the visual system.

[0003] Current research on single image deraining focuses on solving the interference of dense rain streaks and their stacking effects on the recovery of background details, and uses a progressive elimination strategy to remove rain streaks while retaining image content. However, existing methods are generally limited to the learning paradigm of fixed rain streak features. When faced with actual scenes with uncertain rain density distribution, a unified feature extraction mechanism is usually adopted to achieve model universality, rather than dynamically adjusting the processing strategy according to the rain streak density. This one-size-fits-all approach is prone to excessive deraining in high-density rain areas, especially when there are fine-grained objects in the background. It is difficult to balance the contradiction between rain streak removal and detail retention, resulting in high-frequency information loss and texture distortion. Therefore, how to improve the effect of deraining UAV video images has become an urgent problem to be solved. Summary of the invention

[0004] The present invention provides a method and device for removing rain from drone video images based on potential frequency domain representation, the main purpose of which is to solve the problem of poor effect of removing rain from drone video images.

[0005] To achieve the above-mentioned purpose, the present invention provides a method for removing rain from drone video images based on latent frequency domain representation, comprising: obtaining a complex dynamic video image of a drone on a rainy day, performing image contrast regularization processing on the complex dynamic video image of the drone on a rainy day, and obtaining a contrast regularized image; extracting the latent frequency domain representation of the complex dynamic video image of the drone on a rainy day by guiding a pre-constructed latent frequency domain representation extraction network according to the contrast regularized image; performing dual-frequency latent representation enhancement on the latent frequency domain representation to obtain an enhanced representation; and performing representation aggregation image restoration based on the enhanced representation to obtain a derained image of the complex dynamic video image of the drone on a rainy day.

[0006] The present invention also provides a device for removing rain from drone video images based on potential frequency domain representation, comprising: an image contrast regularization processing module, used to obtain complex dynamic video images of drones on rainy days, and perform image contrast regularization processing on the complex dynamic video images of drones on rainy days to obtain contrast regularized images; a potential frequency domain representation extraction module, which guides a pre-constructed potential frequency domain representation extraction network according to the contrast regularized image to extract the potential frequency domain representation of the complex dynamic video images of drones on rainy days; a dual-frequency potential representation enhancement module, which is used to perform dual-frequency potential representation enhancement on the potential frequency domain representation to obtain an enhanced representation; and an image restoration module, which is used to perform representation aggregation image restoration according to the enhanced representation to obtain a derained image of the complex dynamic video images of drones on rainy days.

[0007] The embodiment of the present invention performs image contrast regularization processing on the complex dynamic video image of the drone on rainy days to obtain a contrast regularized image, which can highlight the image texture information and detail information, and ensure that the image maintains a reasonable contrast and visual structure during the processing process; according to the contrast regularized image, the pre-constructed potential frequency domain representation extraction network is guided to extract the potential frequency domain representation of the complex dynamic video image of the drone on rainy days, which can identify the high-frequency components related to the rain streak noise in the image, retain the low-frequency components related to the overall structure and background of the image, and effectively extract the robust potential frequency domain representation; the potential frequency domain representation is enhanced by dual-frequency potential representation to obtain an enhanced representation, which can enhance the potential frequency domain representation from both low-frequency and high-frequency aspects, thereby effectively focusing on the low-frequency local texture details and object contours in the complex dynamic image of the drone on rainy days, and improving the effect of subsequent image restoration; according to the enhanced representation, the representation aggregation image is restored to achieve a robust and efficient video image deraining effect in the complex dynamic scene of the drone on rainy days, and finally improve the visual effect of the image to obtain a more accurate derained image. Therefore, the method and device for deraining drone video images based on potential frequency domain representation proposed by the present invention can solve the problem of poor deraining effect of drone video images. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 A schematic diagram of a process of a method for removing rain from drone video images based on potential frequency domain representation provided by one embodiment of the present invention; Figure 2 A functional module diagram of residual wavelet transform convolution provided by an embodiment of the present invention; Figure 3 A functional module diagram of an image contrast regularization prior module provided by an embodiment of the present invention; Figure 4 A functional module diagram of a frequency domain state space module provided by an embodiment of the present invention; Figure 5 A functional module diagram of a frequency domain characterization mining module provided by an embodiment of the present invention; Figure 6 A functional module diagram of a contrast prior refinement module provided by an embodiment of the present invention; Figure 7 A functional module diagram of a priori gated feedforward module provided in one embodiment of the present invention; Figure 8 A functional module diagram of a deep frequency domain representation aggregated image restoration network provided by an embodiment of the present invention; Fig. 9 A network architecture diagram of a rain removal framework for complex dynamic video images of drones in rainy days based on potential frequency domain representation provided by an embodiment of the present invention; Fig.10 A functional module diagram of a drone video image deraining device based on potential frequency domain representation is provided for one embodiment of the invention.

[0009] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0010] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0011] The embodiment of the present application provides a method for removing rain from drone video images based on potential frequency domain representation. The execution subject of a method for removing rain from drone video images based on potential frequency domain representation includes but is not limited to at least one of the electronic devices such as a server and a terminal that can be configured to execute the method provided in the embodiment of the present application. In other words, the method for removing rain from drone video images based on potential frequency domain representation can be executed by software or hardware installed on a terminal device or a server device. The server includes but is not limited to: a single server, a server cluster, a cloud server or a cloud server cluster, etc. The server can be an independent server, or it can be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0012] Reference Figure 1 , which is a flow chart of a method for removing rain from drone video images based on potential frequency domain representation provided by one embodiment of the present invention.

[0013] S1. Obtain a complex dynamic video image of a drone on a rainy day, perform image contrast regularization processing on the complex dynamic video image of the drone on a rainy day, and obtain a contrast regularized image.

[0014] In an embodiment of the present invention, the complex dynamic video image of a drone on a rainy day is a dynamic video image taken by a drone on a rainy day. However, the rainy day images collected by the drone are often severely obscured by rain and have a blurred background. Therefore, image processing is required to obtain an image after rain removal.

[0015] In detail, image contrast regularization processing is performed on the complex dynamic video image of the drone in rainy days to obtain a contrast regularized image, including: using a pre-built image contrast regularization prior module to extract the contrast of the complex dynamic video image of the drone in rainy days; performing a contrast regularization operation on the complex dynamic video image of the drone in rainy days according to the contrast to obtain a regularized contrast; and performing a residual wavelet transform convolution on the regularized contrast to obtain a contrast regularized image.

[0016] In detail, the Image Contrast -Regularization Prior Module (ICPM) calculates the contrast by fully extracting the maximum and minimum channel values ​​in the complex dynamic video images of drones in rainy days, and highlights the image texture information and detail information through regularization and convolution operations, thereby providing image contrast prior information for the subsequent latent frequency domain representation extraction network, ensuring that the image maintains a reasonable contrast and visual structure during the processing process, avoiding excessive smoothing or information loss.

[0017] Specifically, the image contrast regularization prior module can be expressed as:

[0018]

[0019]

[0020]

[0021] in, Represents the complex dynamic video image of a drone in rainy weather. express function, Represents the minimum value of the image channel of the complex dynamic video image of the drone in rainy days, It represents the maximum value of the image channel of the complex dynamic video image of the drone in rainy days. represents the contrast regularization operation, represents 1×1 convolution, represents the residual wavelet transform convolution, Represents the contrast regularized image. It is worth mentioning that the undefined parameters in the above formula are all intermediate parameters used in the calculation process.

[0022] Furthermore, UAV platforms often have characteristics such as strong mobility, which makes the real-time performance requirements of algorithm deployment strict. In this regard, the method of the present invention has fully considered the unique characteristics of the deployment platform at the beginning of the design of the neural network model architecture, taking the lightweight design of the model as an important starting point, and reducing the number of model parameters as the main means to reduce the computational load at runtime as much as possible to adapt to the limited computing power resources of the UAV platform. In order to reduce the number of model parameters without affecting the performance of the model as much as possible, improve its operating efficiency, and thus adapt to the scenario where the computing resources of the UAV platform are limited, the method of the present invention adopts a residual wavelet transform convolution method, namely RWTConv, such as Figure 2 shown.

[0023] Figure 2 In the convolution method, WTConv (Wavelet Transform Convolution) is a convolution method that combines wavelet transform and traditional convolution operations. It can expand the receptive field of convolution and improve feature extraction capabilities. WTConv uses wavelet transform to perform multi-level decomposition of input data, decomposing the image into different frequency components (such as low frequency and high frequency), and then performs small-size convolution operations on each frequency layer. Finally, the results are recombined through inverse wavelet transform. In this way, it can effectively capture low-frequency information in the image and expand the receptive field of convolution without significantly increasing the number of parameters; Conv1x1 is a one-dimensional convolution. represents an element-wise multiplication operation, represents the tensor addition operation, and GhostConv represents the ghost convolution module.

[0024] In detail, the Residual Wavelet Transform Convolution (RWTConv) module is used as the basic operation for building the model. This operation uses wavelet transform to process local information in the frequency domain, and can provide a larger receptive field with the same number of parameters. In addition, it combines 1×1 convolution and identity mapping in a convolution layer to maintain the uniformity of parameters, and uses the residual structure to reduce the loss of low-dimensional potential frequency domain representation information. Finally, the potential frequency domain representation information is integrated through the convolution operation. This operation has great advantages in reducing the number of model parameters while maintaining model performance. Its specific operations are as follows: Figure 2 As shown, for any input tensor , which is mapped to the output tensor through residual wavelet transform convolution .

[0025] In the embodiment of the present invention, the residual wavelet transform convolution is expressed as:

[0026] in, represents the input tensor of the residual wavelet transform convolution, represents wavelet transform convolution, is a 1×1 convolution, represents phantom convolution, represents an element-wise multiplication operation, represents the tensor addition operation, Represents the output tensor of the residual wavelet transform convolution. It is worth mentioning that the undefined parameters in the above formula are the intermediate parameters used in the calculation process.

[0027] In detail, the image contrast regularization prior module can be Figure 3 As shown, Indicates the maximum value of the image channel, Represents the minimum value of the image channel, represents the Sigmoid function, represents the contrast regularization operation, represents 1×1 convolution, Represents residual wavelet transform convolution, in the figure represents the contrast regularization operation, represents an element-wise multiplication operation, Represents a tensor addition operation.

[0028] In the embodiment of the present invention, by performing image contrast regularization processing on the complex dynamic video image of the drone in rainy days, the image texture information and detail information can be highlighted, ensuring that the image maintains a reasonable contrast and visual structure during the processing process, thereby improving the image deraining effect.

[0029] S2. Based on the contrast regularized image, the pre-constructed latent frequency domain representation extraction network is guided to extract the latent frequency domain representation of the complex dynamic video image of the drone in rainy days.

[0030] In the embodiment of the present invention, in view of the generally low resolution of objects in the complex dynamic images of drones on rainy days, serious rain streak occlusion, blurred background and other problems, the present invention designs a contrast-aware frequency-domain latent-representation extraction network (CFRNet), which converts image information into the frequency domain, identifies high-frequency components related to rain streak noise in the image, and retains low-frequency components related to the overall structure and background of the image, thereby effectively extracting robust latent frequency domain representation. In addition, the network makes full use of the prior knowledge of image contrast to avoid the loss of key original image information and enhance the image details and texture information in the robust latent frequency domain representation. The latent frequency domain representation extraction network consists of a frequency-domain state space module (Frequency-Domain State Space Block, FSSB) and multiple repeating units, each of which consists of a frequency-domain representation mining module (Frequency-Domain Representation Mining Module, FRMM), a priori gated feed forward module (Prior-Gated Feed Forward Module, PFFM), and a contrast prior refinement module (Contrast Prior Refinement Module, CPRM).

[0031] In detail, the latent frequency domain representation of the complex dynamic video image of the drone in rainy days is extracted by guiding the pre-constructed latent frequency domain representation extraction network according to the contrast regularized image, including: utilizing the frequency domain state space module in the latent frequency domain representation extraction network to extract the image frequency domain representation in the complex dynamic video image of the drone in rainy days; utilizing the frequency domain representation mining module in the latent frequency domain representation extraction network to calculate the spectrally robust latent representation of the complex dynamic video image of the drone in rainy days; utilizing the contrast prior refinement module in the latent frequency domain representation extraction network to perform representation mining on the contrast regularized image to obtain the mined representation; utilizing the prior gated feedforward module in the latent frequency domain representation extraction network to perform gated learning on the spectrally robust latent representation and the mined representation to obtain the gated representation; and generating the latent frequency domain representation of the complex dynamic video image of the drone in rainy days according to the image frequency domain representation and the gated representation.

[0032] In detail, in order to improve the learning ability of the potential frequency domain representation of complex dynamic images of drones in rainy days, the present invention proposes a frequency domain state space module FSSB. Among them, FSSB is mainly composed of a linear embedding layer, a 2D selective scanning module SS2D and a potential frequency domain representation learning layer. The frequency domain state space module can effectively traverse the frequency domain information of the image, thereby modeling a robust potential frequency domain representation. In addition, the frequency domain state space module can achieve linear complexity without sacrificing the global receptive field, improve computational efficiency, and facilitate the edge deployment of the inventive method on drone platforms with limited computing resources and loads. The specific structure is as follows Figure 4 As shown. Among them, LayerNorm(·) represents layer normalization operation, Linear(·) represents linear mapping, FFT2D(·) represents two-dimensional fast Fourier transform, RWTConv(·) represents residual wavelet transform convolution, GeLU(·) represents GeLU activation function, InvFFT2D(·) represents two-dimensional inverse fast Fourier transform, SS2D(·) represents 2D selective scanning module, represents an element-wise multiplication operation, Represents a tensor addition operation.

[0033] Specifically, the frequency domain state space module can be expressed as:

[0034] in, Represents the complex dynamic video image of a drone in rainy weather. Representation layer normalization operation, represents a linear mapping, represents the two-dimensional fast Fourier transform, represents the residual wavelet transform convolution, represents the GeLU activation function, represents the two-dimensional inverse fast Fourier transform, Represents the 2D selective scanning module, It is worth mentioning that the undefined parameters in the above formula are all intermediate parameters used in the calculation process.

[0035] Furthermore, in order to effectively focus on the different rain trace representations in the complex dynamic video images of drones in rainy days and perform robust latent representation modeling in the image spectrum, the method of the present invention proposes a frequency domain representation mining module FRMM, which is used to perform multi-receptive field information mining on the potential frequency domain representation of the input regularized image. Specifically, the frequency domain representation mining module first uses residual wavelet transform convolution to perform preliminary representation modeling, and then uses asymmetric convolution to focus on rain noise in different directions, thereby achieving lightweight while improving the representation modeling performance. Afterwards, the image frequency domain information is effectively extracted through operations such as Fourier transform, so that the modeled potential frequency domain representation is more conducive to the subsequent rain removal process. The specific structure is as follows: Figure 5 As shown, represents the residual wavelet transform convolution, Indicates a separation operation. represents asymmetric convolution, represents the ReLU activation function, represents the connection operation on the channel dimension, represents 1×1 convolution, represents the two-dimensional fast Fourier transform, represents the GeLU activation function, Represents the two-dimensional inverse fast Fourier transform, in the figure Represents channel dimension connection and convolution dimension reduction operation, Represents a tensor addition operation.

[0036] In detail, the frequency domain representation mining module can be expressed as:

[0037] in, Represents the complex dynamic video image of a drone in rainy weather. represents the residual wavelet transform convolution, Indicates a separation operation. represents asymmetric convolution, represents the ReLU activation function, represents the connection operation on the channel dimension, represents 1×1 convolution, represents the two-dimensional fast Fourier transform, represents the GeLU activation function, represents the two-dimensional inverse fast Fourier transform, Represents channel dimension connection and convolution dimension reduction operation, It is worth mentioning that the undefined parameters in the above formula are all intermediate parameters used in the calculation process.

[0038] Furthermore, in order to continuously enrich the channel dimension information of the contrast of complex dynamic video images of drones in rainy days, so as to continuously supplement low-frequency details and local textures from shallow to deep in the process of iteratively extracting the potential frequency domain representation of the image, avoid excessive smoothing or information loss, and effectively retain the key original clear structure of the image, the present invention proposes a contrast prior refinement module CPRM. The contrast prior refinement module enhances local contrast information through residual wavelet transform convolution, and retains the original image contrast information through the residual structure, so as to prevent potential interference caused by too deep a network, and finally transmits the integrated image contrast prior knowledge to the corresponding prior gated feedforward module.

[0039] Specifically, the contrast prior refinement module can be expressed as:

[0040] in, represents the contrast normalized image, represents the residual wavelet transform convolution, represents asymmetric convolution, express function, represents 1×1 convolution, Indicates soft pooling operation. It is worth mentioning that the undefined parameters in the above formula are all intermediate parameters used in the calculation process.

[0041] The specific structure of the contrast prior refinement module can be found in Figure 6 .

[0042] Furthermore, in order to effectively utilize the prior knowledge of contrast of complex dynamic images of drones in rainy days, the method of the present invention designs a prior gated feedforward module PFFM. The prior gated feedforward module integrates the prior knowledge of image contrast into the modeled potential frequency domain representation in a gating manner to improve local details and structure recovery. Specifically, the prior gated feedforward module first enriches the channel dimension through residual wavelet transform convolution, and then uses asymmetric convolution to refine local details and object contours. At the same time, the prior branch introduces the enhanced image contrast information and uses this information as the gating weight, thereby significantly enhancing and reconstructing the image background information weakened by rain noise. The specific structure is as follows: Figure 7 As shown, represents the residual wavelet transform convolution, represents asymmetric convolution, represents the Tanh activation function, represents 1×1 convolution, represents the gate control unit, Figure 7 middle represents the gate control unit, represents an element-wise multiplication operation, Represents a tensor addition operation.

[0043] The a priori gated feedforward module can be expressed as:

[0044] in, represents the spectrally robust latent representation, represents mining representation, represents the residual wavelet transform convolution, represents asymmetric convolution, represents the Tanh activation function, represents 1×1 convolution, represents the gate control unit, It is worth mentioning that the undefined parameters in the above formula are all intermediate parameters used in the calculation process.

[0045] In an embodiment of the present invention, the number of units N of the frequency domain representation mining module, the prior gated feedforward module, and the contrast prior refinement module can be adaptively adjusted according to the rain pattern density and raindrop occlusion degree of the specific task environment of the UAV complex dynamic video image deraining in rainy days, thereby extracting a robust potential frequency domain representation from the UAV complex dynamic video image in rainy days for subsequent representation enhancement.

[0046] Each unit can be expressed as:

[0047] in, It is the frequency domain characterization mining module. is the contrast prior refinement module, A module for refining contrast priors.

[0048] In the embodiment of the present invention, the image frequency domain representation and the gated representation are multiplied element by element to obtain a potential representation in the frequency domain, i.e., a potential frequency domain representation. Through the potential frequency domain representation extraction network CFRNet, the image information can be converted to the frequency domain, the high-frequency components related to the rain streak noise in the image can be identified, the low-frequency components related to the overall structure and background of the image can be retained, and the robust potential frequency domain representation can be effectively extracted.

[0049] S3. Perform dual-frequency latent representation enhancement on the latent frequency domain representation to obtain an enhanced representation.

[0050] In an embodiment of the present invention, dual-frequency latent representation enhancement is divided into a low-frequency latent representation enhancement branch and a high-frequency latent representation enhancement branch. By separately enhancing different frequency representations, it effectively focuses on low-frequency local texture details and object contours in complex dynamic images of drones in rainy days, filters out high-frequency rain noise, and thus provides a robust latent frequency domain representation for the subsequent image representation deraining and restoration process.

[0051] Specifically, the low-frequency latent representation enhancement module targets information-rich low-frequency components, and uses the linear complexity and powerful information modeling capabilities of the frequency-domain state-space module to efficiently enhance low-frequency information, thereby balancing performance and computational cost, which is conducive to edge deployment. The high-frequency latent representation enhancement module uses a multi-head attention mechanism to focus on the high-frequency rain noise distribution area, ensuring that the subsequent image representation deraining and restoration process can accurately locate the area to be derained.

[0052] In an embodiment of the present invention, a dual-frequency potential representation enhancement is performed on a potential frequency domain representation to obtain an enhanced representation, including: performing low-frequency enhancement on the potential frequency domain representation using a low-frequency potential representation enhancement branch in a pre-constructed dual-frequency potential representation enhancement network to obtain a low-frequency enhanced representation; performing high-frequency enhancement on the potential frequency domain representation using a high-frequency potential representation enhancement branch in a dual-frequency potential representation enhancement network to obtain a high-frequency enhanced representation; and performing convolution connection on the low-frequency enhanced representation and the high-frequency enhanced representation to obtain an enhanced representation.

[0053] In detail, the dual-frequency latent representation enhancement network can be expressed as:

[0054] in, represents the potential frequency domain representation, Representation layer normalization operation, represents the frequency domain state space module, represents the residual wavelet transform convolution, represents 1×1 convolution, represents a linear mapping, represents multi-head self-attention, represents the connection operation on the channel dimension, It is worth mentioning that the undefined parameters in the above formula are all intermediate parameters used in the calculation process.

[0055] In an embodiment of the present invention, a dual-frequency latent representation enhancement network (DFRNet) can be used to enhance the latent frequency domain representation from both low-frequency and high-frequency aspects, thereby effectively focusing on the low-frequency local texture details and object contours in the complex dynamic images of drones in rainy days, thereby improving the effect of subsequent image restoration.

[0056] S4. Performing characterization aggregation image restoration based on the enhanced characterization to obtain a derained image of the complex dynamic video image of the drone in rainy days.

[0057] In an embodiment of the present invention, aggregate image restoration is to restore the image to its original size, thereby achieving a robust deraining effect for complex dynamic images of unmanned aerial vehicles in rainy days. The present invention can utilize a deep frequency domain representation aggregate image restoration network mainly composed of two parts: a parallel spatial potential frequency domain representation branch and a channel potential frequency domain representation branch. Specifically, the spatial potential frequency domain representation branch processes the frequency domain spatial information through the above-mentioned frequency domain state space module, maximum pooling, average pooling and other operations, effectively restoring the background information interfered by rain noise. At the same time, the channel potential frequency domain representation branch processes the frequency domain channel information through the above-mentioned frequency domain state space module, global maximum pooling, global average pooling and other operations, enhancing the expression of key original information in the image channel dimension, suppressing the expression of rain noise channel dimension information, and effectively fusing the two results. Finally, the image is restored to its original size through multi-layer residual wavelet transform convolution, thereby achieving a robust deraining effect for complex dynamic images of unmanned aerial vehicles in rainy days.

[0058] In detail, representation aggregation image restoration is performed based on the enhanced representation to obtain a derained image of the complex dynamic video image of the drone in rainy days, including: using the spatial latent frequency domain representation branch in the pre-constructed deep frequency domain representation aggregation image restoration network to perform spatial representation aggregation on the enhanced representation to obtain a spatial aggregated representation; using the channel latent frequency domain representation branch in the deep frequency domain representation aggregation image restoration network to perform frequency domain representation aggregation on the enhanced representation to obtain a frequency domain aggregated representation; performing multi-layer residual wavelet transform convolution on the spatial aggregated representation and the frequency domain aggregated representation to obtain a derained image corresponding to the complex dynamic video image of the drone in rainy days.

[0059] Specifically, the deep frequency domain representation aggregation image restoration network can be expressed as:

[0060] in, represents enhanced representation, represents the residual wavelet transform convolution, represents the frequency domain state space module, represents the maximum pooling operation, represents the average pooling operation, represents the connection operation on the channel dimension, represents 1×1 convolution, represents the global maximum pooling operation, represents the global average pooling operation, represents multi-layer residual wavelet transform convolution, It is worth mentioning that the undefined parameters in the above formula are all intermediate parameters used in the calculation process.

[0061] In detail, the deep frequency domain representation aggregation image restoration network can be Figure 8 As shown, in Figure 8 middle, represents the residual wavelet transform convolution, represents the frequency domain state space module, represents the maximum pooling operation, represents the average pooling operation, represents the connection operation on the channel dimension, represents 1×1 convolution, represents the global maximum pooling operation, represents the global average pooling operation, represents multi-layer residual wavelet transform convolution, represents an element-wise multiplication operation, Represents tensor addition operation, in the figure It represents the channel dimension connection and the convolution dimension reduction operation. The number of layers of the multi-layer residual wavelet transform convolution is adaptively determined according to the number of network units extracted based on the potential frequency domain representation of contrast perception.

[0062] In detail, the present invention combines an image contrast regularization prior module, a contrast-aware potential frequency domain representation extraction network, a dual-frequency potential representation enhancement network, and a deep frequency domain representation aggregation image restoration network through a rain removal framework for complex dynamic video images of drones in rainy days based on potential frequency domain representation. Fig. 9 shown.

[0063] Among them, the Image Contrast -Regularization Prior Module (ICPM) can make full use of the maximum and minimum values ​​of the channels of the complex dynamic video images of the drone in rainy days to calculate the image contrast, and extract the image detail texture prominent area through regularization operation and convolution operation, so as to provide image contrast prior information for the extraction of potential frequency domain representation, ensure that the image maintains a reasonable contrast and visual structure during the processing process, avoid over-smoothing or information loss, effectively retain the key original representation of the image, and achieve a clearer and more natural image restoration effect; Contrast-Aware Frequency-Domain Latent-Representation Extraction Network (CFRNet), which can fully extract the latent frequency domain representation of complex dynamic video images of drones in rainy days, and supplement the image contrast information. In the process of separating the high-frequency detail texture information and low-frequency background information of the image, it avoids the problems of excessive background smoothing or distortion caused by the high maneuverability of the drone, suppresses the generation of artifacts, and thus makes the outline of objects in the image clearer, maintains the naturalness and layering of the image, and significantly improves the visual effect; The Dual-Frequency Latent-Representation Enhancement Network (DFRNet) can fully focus on the high-frequency detail texture and low-frequency overall structure of the complex dynamic video images of drones in rainy days through high-frequency and low-frequency dual-frequency representation modeling, and perform adaptive fusion and enhancement in the frequency domain, thereby effectively filtering out high-frequency rain noise and retaining key image original information. In addition, it can accurately locate the area to be derained in the complex dynamic video images of drones in rainy days in response to challenges such as the high maneuverability of drones and the generally low resolution of objects in images. The Deep Frequency-Domain Representation Aggregation Image-Restoration Network (DRANet) can use parallel processing in the spatial frequency domain and the channel frequency domain to fully focus on the spatial structure and texture information of complex dynamic video images of drones in rainy days, strengthen or suppress the representation contribution of different channels in the frequency domain, thereby effectively removing high-frequency rain noise while enhancing the local representation and overall structure of the image, optimizing the network representation capability, and ultimately achieving robust deraining effect for complex dynamic video images of drones in rainy days.

[0064] Therefore, the present invention can effectively improve the image clarity of drones in complex dynamic environments on rainy days, solve problems such as raindrop occlusion, image blur, background blur, color distortion, etc., and use image frequency domain information to effectively remove rain noise in response to challenges such as high dynamics of drones and generally low resolution of objects in images, and form more effective detail texture information and object contour information in image representation, thereby achieving robust and efficient video image deraining effects in complex dynamic scenes of drones on rainy days, ultimately improving the visual effect of the image and obtaining a more accurate derained image.

[0065] Preferably, the method of the present invention can be used to be deployed in an industrial-grade UAV edge vision camera based on an embedded GPU platform to realize edge computing. Specifically, the edge computing realized by the method of the present invention refers to collecting complex dynamic image data in rainy scenes through an airborne edge vision camera on the UAV flight body, and independently realizing edge vision intelligence in the camera, that is, based on the above-mentioned UAV video image deraining method based on potential frequency domain representation, high-precision, strong robustness, and real-time image deraining results are obtained at the image data acquisition end. The method of the present invention can serve the obtained image deraining results for fast human-UAV interaction, communication data optimization, real-time response operation, intelligent analysis application, privacy protection and data security. Ensure that the number and amount of communications with the cloud platform are reduced as much as possible, thereby reducing waiting time and computing costs. In view of the fact that the complex dynamic image deraining task of the UAV in rainy days is deployed to the image data acquisition end, it can effectively reduce the congestion of the backbone network, alleviate bandwidth occupancy, achieve lower latency, improve processing efficiency, speed up response requests, and further improve the quality of complex dynamic image deraining of the UAV in rainy days. In particular, the speed of the method of the present invention after edge deployment can reach real-time processing speed, and its rain removal effect and image quality meet the requirements of industrial-grade applications.

[0066] like Fig.10 As shown, an embodiment of the present invention provides a functional module diagram of a drone video image deraining device based on potential frequency domain representation.

[0067] The device 100 for removing rain from drone video images based on potential frequency domain representation of the present invention can be installed in an electronic device. According to the functions to be implemented, the device 100 for removing rain from drone video images based on potential frequency domain representation can include an image contrast regularization processing module 101, a potential frequency domain representation extraction module 102, a dual-frequency potential representation enhancement module 103, and an image restoration module 104. The module of the present invention can also be referred to as a unit, which refers to a series of computer program segments that can be executed by an electronic device processor and can complete fixed functions, and is stored in the memory of the electronic device.

[0068] In this embodiment, the functions of each module / unit are as follows: an image contrast regularization processing module 101 is used to obtain a complex dynamic video image of a drone on a rainy day, and perform image contrast regularization processing on the complex dynamic video image of a drone on a rainy day to obtain a contrast regularized image; a potential frequency domain representation extraction module 102 is used to guide a pre-constructed potential frequency domain representation extraction network according to the contrast regularized image to extract the potential frequency domain representation of the complex dynamic video image of the drone on a rainy day; a dual-frequency potential representation enhancement module 103 is used to perform dual-frequency potential representation enhancement on the potential frequency domain representation to obtain an enhanced representation; an image restoration module 104 is used to perform representation aggregation image restoration based on the enhanced representation to obtain a derained image of the complex dynamic video image of the drone on a rainy day.

[0069] In detail, in one embodiment, each module in a drone video image deraining device 100 based on potential frequency domain representation adopts the same technical means as the drone video image deraining method based on potential frequency domain representation in the accompanying drawings when used, and can produce the same technical effects, which will not be repeated here.

[0070] For example, although not shown, the electronic device may also include a power source (such as a battery) for supplying power to various components. Preferably, the power source may be logically connected to at least one processor through a power management device, so that the power management device can realize functions such as charging management, discharging management, and power consumption management. The power source may also include any components such as one or more DC or AC power sources, recharging devices, power failure detection circuits, power converters or inverters, and power status indicators. The electronic device may also include a variety of sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be repeated here.

[0071] It should be understood that the embodiment is for illustration only and the scope of the patent application is not limited by this structure.

[0072] The present invention also provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it can implement a method for removing rain from drone video images based on potential frequency domain representation according to any of the above embodiments.

[0073] It should be noted that the computer-readable storage medium may be volatile or non-volatile. For example, the computer-readable medium may include: any entity or device capable of carrying computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM). In the several embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses and methods may be implemented in other ways. For example, the device embodiments described above are merely schematic, for example, the division of modules is only a logical function division, and there may be other division methods in actual implementation.

[0074] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0075] In addition, each functional module in each embodiment of the present invention may be integrated into one processing unit, each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional modules.

[0076] It is obvious to those skilled in the art that the present invention is not limited to the details of the above exemplary embodiments, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0077] Therefore, no matter from which point of view, the embodiments should be regarded as illustrative and non-restrictive, and the scope of the present invention is limited by the appended claims rather than the above description, so it is intended that all changes falling within the meaning and scope of the equivalent elements of the claims are included in the present invention. Any attached figure mark in the claims should not be regarded as limiting the claims involved.

[0078] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0079] In addition, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices stated in the system claim can also be implemented by one unit or device through software or hardware. The words first, second, etc. are used to indicate names, and do not indicate any particular order.

[0080] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the spirit and scope of the technical solution of the present invention.

Claims

1. A method for removing rain from drone video images based on latent frequency domain representation, characterized in that: The method comprises: Acquire a complex dynamic video image of a drone on a rainy day, and perform image contrast regularization processing on the complex dynamic video image of the drone on a rainy day to obtain a contrast regularized image; Extracting the latent frequency domain representation of the complex dynamic video image of the drone in rainy day by guiding the pre-constructed latent frequency domain representation extraction network according to the contrast regularized image; Performing dual-frequency potential representation enhancement on the potential frequency domain representation to obtain an enhanced representation; The representation aggregation image is restored according to the enhanced representation to obtain a rain-free image of the complex dynamic video image of the drone in rainy days.

2. The method for removing rain from drone video images based on potential frequency domain representation according to claim 1, characterized in that: The image contrast regularization processing is performed on the complex dynamic video image of the drone in rainy days to obtain a contrast regularized image, including: The contrast of the complex dynamic video image of the UAV in rainy day is extracted by using a pre-built image contrast regularization prior module; Performing a contrast regularization operation on the complex dynamic video image of the drone on a rainy day according to the contrast to obtain a regularized contrast; The regularized contrast is subjected to residual wavelet transform convolution to obtain a contrast regularized image.

3. The method for removing rain from drone video images based on potential frequency domain representation as claimed in claim 2, characterized in that: The image contrast regularization prior module can be expressed as: in, Represents the complex dynamic video image of a drone in rainy weather. express function, Represents the minimum value of the image channel of the complex dynamic video image of the drone in rainy days, It represents the maximum value of the image channel of the complex dynamic video image of the drone in rainy days. represents the contrast regularization operation, represents 1×1 convolution, represents the residual wavelet transform convolution, represents the contrast normalized image, , , These are all intermediate parameters used in the calculation process.

4. The method for removing rain from drone video images based on potential frequency domain representation as claimed in claim 2, characterized in that: The residual wavelet transform convolution is expressed as: in, represents the input tensor of the residual wavelet transform convolution, represents wavelet transform convolution, is a 1×1 convolution, represents phantom convolution, represents an element-wise multiplication operation, represents the tensor addition operation, represents the output tensor of the residual wavelet transform convolution, is the intermediate parameter used in the calculation process.

5. The method for removing rain from drone video images based on latent frequency domain representation according to claim 1, characterized in that: The method of extracting the potential frequency domain representation of the complex dynamic video image of the drone in rainy days by guiding the pre-constructed potential frequency domain representation extraction network according to the contrast regularized image comprises: The frequency domain state space module in the potential frequency domain representation extraction network is used to extract the image frequency domain representation in the complex dynamic video image of the drone in rainy days; Utilizing the frequency domain representation mining module in the potential frequency domain representation extraction network to calculate the spectrum robust potential representation of the complex dynamic video image of the drone in rainy days; Performing representation mining on the contrast regularized image using a contrast prior refinement module in the potential frequency domain representation extraction network to obtain a mined representation; Performing gated learning on the spectrum robust potential representation and the mining representation using a priori gated feedforward module in the potential frequency domain representation extraction network to obtain a gated representation; A potential frequency domain representation of the complex dynamic video image of the drone in rainy days is generated according to the image frequency domain representation and the gated representation.

6. The method for removing rain from drone video images based on potential frequency domain representation as claimed in claim 5, characterized in that: The frequency domain state space module can be expressed as: in, Represents the complex dynamic video image of a drone in rainy weather. Representation layer normalization operation, represents a linear mapping, represents the two-dimensional fast Fourier transform, represents the residual wavelet transform convolution, represents the GeLU activation function, represents the two-dimensional inverse fast Fourier transform, Represents the 2D selective scanning module, represents the frequency domain representation of the image, , , These are all intermediate parameters used in the calculation process.

7. The method for removing rain from drone video images based on latent frequency domain representation as claimed in claim 5, characterized in that: The frequency domain characterization mining module can be expressed as: in, Represents the complex dynamic video image of a drone in rainy weather. represents the residual wavelet transform convolution, Indicates a separation operation. represents asymmetric convolution, represents the ReLU activation function, represents the connection operation on the channel dimension, represents 1×1 convolution, represents the two-dimensional fast Fourier transform, represents the GeLU activation function, represents the two-dimensional inverse fast Fourier transform, Represents channel dimension connection and convolution dimension reduction operation, represents the spectrally robust latent representation, , , , , These are all intermediate parameters used in the calculation process.

8. The method for removing rain from drone video images based on latent frequency domain representation as claimed in claim 5, characterized in that: The contrast prior refinement module can be expressed as: in, represents the contrast normalized image, represents the residual wavelet transform convolution, represents asymmetric convolution, express function, represents 1×1 convolution, Indicates soft pooling operation. represents mining representation, , , These are all intermediate parameters used in the calculation process.

9. The method for removing rain from drone video images based on potential frequency domain representation as claimed in claim 5, characterized in that: The a priori gated feedforward module can be expressed by the formula: in, represents the spectrally robust latent representation, represents mining representation, represents the residual wavelet transform convolution, represents asymmetric convolution, represents the Tanh activation function, represents 1×1 convolution, represents the gate control unit, represents the gated representation, , , , These are all intermediate parameters used in the calculation process.

10. The method for removing rain from drone video images based on potential frequency domain representation according to claim 1, characterized in that: The step of performing dual-frequency potential representation enhancement on the potential frequency domain representation to obtain an enhanced representation includes: Performing low-frequency enhancement on the latent frequency domain representation using a low-frequency latent representation enhancement branch in a pre-constructed dual-frequency latent representation enhancement network to obtain a low-frequency enhanced representation; Performing high-frequency enhancement on the latent frequency domain representation using the high-frequency latent representation enhancement branch in the dual-frequency latent representation enhancement network to obtain a high-frequency enhanced representation; The low-frequency enhanced representation and the high-frequency enhanced representation are convolutionally connected to obtain an enhanced representation.

11. The method for removing rain from drone video images based on potential frequency domain representation according to claim 10, characterized in that: The dual-frequency potential representation enhancement network can be expressed as: in, represents the potential frequency domain representation, Representation layer normalization operation, represents the frequency domain state space module, represents the residual wavelet transform convolution, represents 1×1 convolution, represents a linear mapping, represents multi-head self-attention, represents the connection operation on the channel dimension, represents enhanced representation, , , These are all intermediate parameters used in the calculation process.

12. The method for removing rain from drone video images based on potential frequency domain representation according to claim 1, characterized in that: The method of restoring the representation aggregation image according to the enhanced representation to obtain the rain-free image of the complex dynamic video image of the drone in rainy days includes: Using the spatial latent frequency domain representation branch in the pre-constructed deep frequency domain representation aggregation image restoration network to perform spatial representation aggregation on the enhanced representation to obtain a spatial aggregated representation; Perform frequency domain representation aggregation on the enhanced representation using the channel potential frequency domain representation branch in the deep frequency domain representation aggregation image restoration network to obtain a frequency domain aggregated representation; Multi-layer residual wavelet transform convolution is performed on the spatial aggregation representation and the frequency domain aggregation representation to obtain a derained image corresponding to the complex dynamic video image of the drone on a rainy day.

13. The method for removing rain from drone video images based on latent frequency domain representation according to claim 1, characterized in that: The deep frequency domain representation aggregation image restoration network can be expressed as: in, represents enhanced representation, represents the residual wavelet transform convolution, represents the frequency domain state space module, represents the maximum pooling operation, represents the average pooling operation, represents the connection operation on the channel dimension, represents 1×1 convolution, represents the global maximum pooling operation, represents the global average pooling operation, represents multi-layer residual wavelet transform convolution, represents the derained image, , , , , , , These are all intermediate parameters used in the calculation process.

14. A UAV video image deraining device based on potential frequency domain representation, characterized in that: The device comprises: An image contrast regularization processing module is used to obtain a complex dynamic video image of a drone on a rainy day, and perform image contrast regularization processing on the complex dynamic video image of the drone on a rainy day to obtain a contrast regularized image; A potential frequency domain representation extraction module is used to guide a pre-constructed potential frequency domain representation extraction network to extract the potential frequency domain representation of the complex dynamic video image of the drone in rainy days according to the contrast regularized image; A dual-frequency potential representation enhancement module, used to perform dual-frequency potential representation enhancement on the potential frequency domain representation to obtain an enhanced representation; The image restoration module is used to restore the representation aggregation image according to the enhanced representation to obtain a rain-free image of the complex dynamic video image of the drone in rainy days.

Images