A video rain removal method based on adaptive tensor weighted kernel norm

Abstract

The application discloses a video rain removal method based on an adaptive tensor weighted kernel norm, belongs to the field of computer vision and image processing, and comprises the following steps: acquiring a rainy video to be removed from rain lines; analyzing prior information with distinguishability of a rain-free video and rain lines; constructing a rain removal model for removing the rain lines in the rainy video based on the prior information; and removing the rain lines from the rainy video based on the rain removal model. The application can make the video after rain removal retain more detailed features, avoid rain line blurring, effectively avoid loss of video detailed information, make the video after rain removal retain more abundant and fine detailed features, and realize high-fidelity video rain removal.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and image processing, and relates to a video deraining model based on adaptive tensor weighted nuclear norm. Background Technology

[0002] With the rapid development of computer technology, computer vision systems have been widely used in coastal monitoring, remote sensing, and target recognition. However, rainy weather can severely degrade video captured by outdoor vision systems, affecting subsequent processing. Furthermore, rainy weather can also impact ship navigation and traffic management. Therefore, processing contaminated video images is of significant importance. Currently, video deraining algorithms are mainly divided into three categories: time-domain based deraining methods, frequency-domain based deraining methods, and sparse-domain based deraining methods.

[0003] One type of rain removal method based on the time domain is [1] Rain removal methods based on temporal median filtering treat raindrops as random noise and apply median filtering to each pixel of the video frame. However, when moving objects appear in the video, they blur details, limiting their application to static scenes. To remove the influence of moving objects on rain removal, many rain removal algorithms have been proposed, such as... [2] The rain removal method based on the raindrop dynamics model and photometric model assumes that two consecutive frames will not be blocked by raindrops, which can reduce the impact of moving objects on rain removal. However, the rain removal effect of this method is not ideal in heavy rain or even rainstorm. [3] An initial rain line is obtained using a photometric model, followed by secondary processing using an edge-preserving smoothing filter to reduce interference from fine rain lines. Finally, inter-frame differencing is used to subtract moving objects to obtain the final rain-removed result. However, significant changes in background illumination can severely interfere with the motion detection process. Therefore... [4] A rain removal method based on multi-frame anisotropic filtering is proposed. It utilizes the local characteristics of rain lines to simultaneously detect and remove rain lines. It performs well in scenes with moving objects, but its ability to detect fine rain lines is poor.

[0004] One type of rain removal method based on the frequency domain is... [5] A fuzzy Gaussian model is used to approximate rain lines, and then a frequency domain filter is applied to eliminate them. However, this method is less effective when the rain line frequencies are highly variable. [6]A video rain removal method based on local phase consistency is proposed, utilizing phase consistency and optical flow to remove rain lines. Another type is... [7][8] Rain lines can be removed using wavelet transform and bilateral filtering, but this method is greatly affected by moving objects.

[0005] One type of rain removal method based on sparse domains is... [9] Image deraining methods based on morphological component analysis (MCA) have been developed. Some researchers have extended this to video deraining, but its performance in detecting rain lines in dynamic scenes is poor. One type of method is...

[10] A video deraining method based on temporal correlation and low-rank matrix filling is proposed. An initial rain map is obtained through optical flow estimation, refined using a support vector machine classifier, and rain lines are eliminated through low-rank matrix filling. However, the algorithm is relatively complex.

[0006] In response

[11] A video deraining method based on a tensor model is proposed. This method establishes a video deraining model by constraining the intrinsic prior information of rain lines and rainless backgrounds through regularization. Compared to single low-rank or sparse constraints, this method is more flexible and stable. However, this method uses the tensor kernel norm to ensure the global low-rank of rainless videos, treating the low-rank nature of different dimensions of rainless videos equally. This can lead to blurred and incomplete deraining of the video. Summary of the Invention

[0007] To address the issue of incomplete rain removal and resulting video blurring, this invention employs the following technical solution: a video rain removal method based on adaptive tensor weighted nuclear norm, comprising the following steps:

[0008] Obtain a video containing rain lines that need to be removed;

[0009] Analyze discriminative prior information between rainless videos and rain lines;

[0010] A rain removal model for removing rain streaks from rainy videos is constructed based on prior information.

[0011] Rain lines are removed from rainy videos based on the aforementioned rain removal model.

[0012] Furthermore: the prior information includes sparser rain lines compared to rainless videos, using regularization terms. To depict rain lines, and the rain lines have a stronger smoothness in the vertical direction, using... of Norms are used to characterize the smoothness of rain lines in the vertical direction;

[0013] Compared to rain lines, rainless videos exhibit stronger smoothness in both the horizontal and temporal directions, therefore, they are used respectively. of Norm and of Norms are used to characterize the smoothness of rainless videos in the horizontal and temporal directions, and... Different weights are assigned to different dimensions of low rank in rainless videos to ensure the global low rank of rainless videos.

[0014] Furthermore, the expression for the rain line removal model is as follows:

[0015]

[0016]

[0017] in: There is a rainy video. It's a video without rain. It's a rain line. These are total variation operators in a single direction, respectively, along the rain line direction and along the vertical direction. It is a time-difference operator, where {α1,α2,α3,α4} are experimental parameters;

[0018] It uses an adaptive tensor weighted nuclear norm to characterize the overall low-rank property of rainless videos.

[0019] Furthermore, the rain line removal model is solved using the Alternating Direction Multiplier Method (ADMM), as follows:

[0020] By introducing four intermediate variables The expression for the rain line removal model is represented by the following equivalent constraints:

[0021]

[0022]

[0023] The augmented Lagrangian function in the above equation is:

[0024]

[0025] Where: Λ=[Λ1,Λ2,Λ3,Λ4,Λ5] are Lagrange multipliers, and β=[β1,β2,β3,β4,β5] are nonnegative scalar parameters;

[0026] Within the framework of the alternating direction multiplier method, the problem to be solved is divided into subproblems. Solve for;

[0027] Alternating solutions;

[0028] Fix other parameter variables hour, Iterative updates are performed using the following methods:

[0029]

[0030] The sub-problems are as follows:

[0031]

[0032] The R subproblem is a least squares problem as shown below:

[0033]

[0034] Formula (8) has the following closed-form solution:

[0035]

[0036] in: and Let these represent the Fast Fourier Transform and its inverse transform, respectively, where:

[0037]

[0038]

[0039] Multiplier update: Based on the framework of the alternating direction multiplier method, the Lagrange multiplier Λ = [Λ1,Λ2,Λ3,Λ4,Λ5] is updated as follows.

[0040]

[0041] A video deraining device based on adaptive tensor weighted nuclear norm, comprising:

[0042] Acquisition module: Used to acquire rainy videos where rain lines need to be removed;

[0043] Analysis module: Used to analyze discriminative prior information between rainless videos and rain lines;

[0044] Module: Used to build a rain line removal model based on prior information for removing rain lines from rainy videos;

[0045] Rain removal module: used to remove rain lines from rainy videos based on the rain removal model.

[0046] This invention provides a video deraining method based on adaptive tensor weighted nuclear norm. By considering the strength of low-rank in different dimensions of the rainless video and assigning different weights according to the low-rank in different dimensions, it ensures the global low rank of the rainless video. At the same time, a sparse regularization term is introduced to promote rain line separation. Finally, the alternating direction multiplier method (ADMM) is used to solve the model. This application can make the derained video retain more detailed features, avoid rain line blurring, effectively avoid the loss of video detail information, and make the derained video retain richer and finer detail features, thus achieving high-fidelity video deraining. Attached Figure Description

[0047] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0048] Figure 1 Here is a flowchart of the application;

[0049] Figure 2 (a) is a histogram of the absolute values ​​of the vertical difference of the rain line, and (b) is a histogram of the absolute values ​​of the vertical difference of the no-rain video.

[0050] Figure 3 (a) is a histogram of the absolute values ​​of the differences in the horizontal direction of the rain line, and (b) is a histogram of the absolute values ​​of the differences in the horizontal direction of the no-rain video.

[0051] Figure 4 (a) is the histogram of the absolute value of the difference in the time direction of the rain line (Figure III), and (b) is the histogram of the absolute value of the difference in the time direction of the rainless video (Figure III).

[0052] Figure 5 (a) is a video image without rain; (b) is a distribution map of singular values ​​in the temporal and spatial dimensions of the video without rain.

[0053] Figure 6 (a) is the video frame I with rain, (b) is the video frame I without rain;

[0054] Figure 7 (a) is the video frame II with rain, (b) is the video frame II without rain. Detailed Implementation

[0055] It should be noted that, unless otherwise specified, the embodiments and features in the embodiments of the present invention can be combined with each other. The present invention will be described in detail below with reference to the accompanying drawings and embodiments.

[0056] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit the present invention or its application or use. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0057] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0058] Unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps described in these embodiments do not limit the scope of the invention. It should also be understood that, for ease of description, the dimensions of the various parts shown in the drawings are not drawn to actual scale. Techniques, methods, and devices known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and devices should be considered part of the specification. In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values. It should be noted that similar reference numerals and letters in the following figures denote similar items; therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0059] In the description of this invention, it should be understood that the orientation or positional relationship indicated by directional terms such as "front, back, up, down, left, right", "horizontal, vertical, horizontal" and "top, bottom" is generally based on the orientation or positional relationship shown in the accompanying drawings, and is only for the convenience of describing this invention and simplifying the description. Unless otherwise stated, these directional terms do not indicate or imply that the device or element referred to must have a specific orientation or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation on the scope of protection of this invention. The directional terms "inner" and "outer" refer to the inner and outer contours relative to the outline of each component itself.

[0060] For ease of description, spatial relative terms such as "above," "over," "on the upper surface of," "above," etc., are used herein to describe the spatial positional relationship of a device or feature as shown in the figures to other devices or features. It should be understood that spatial relative terms are intended to encompass different orientations in use or operation besides the orientation of the device as described in the figures. For example, if the device in the figures is inverted, a device described as "above" or "above" other devices or structures would subsequently be positioned as "below" or "under" other devices or structures. Thus, the exemplary term "above" can include both "above" and "below." The device may also be positioned in other different ways (rotated 90 degrees or in other orientations), and the spatial relative descriptions used herein will be interpreted accordingly.

[0061] Furthermore, it should be noted that the use of terms such as "first" and "second" to define components is merely for the purpose of distinguishing the corresponding components. Unless otherwise stated, the above terms have no special meaning and therefore should not be construed as limiting the scope of protection of this invention.

[0062] Figure 1 Here is a flowchart of the application;

[0063] A video deraining method based on adaptive tensor weighted nuclear norm includes the following steps:

[0064] S1: Obtain the rainy video containing the rain lines to be removed;

[0065] S2: Analyze discriminative prior information between rainless videos and rain lines;

[0066] S3: Construct a rain line removal model based on prior information to remove rain lines from rainy videos;

[0067] S4: Remove rain lines from rainy videos based on the aforementioned rain removal model.

[0068] Steps S1 / S2 / S3 / S4 are executed sequentially;

[0069] Furthermore, the prior information, including rain lines, is sparser compared to rainless videos; therefore, we use regularization terms. To depict the rain lines.

[0070] Figure 2 (a) is a histogram of the absolute values ​​of the vertical difference in the rain line, and (b) is a histogram of the absolute values ​​of the vertical difference in the no-rain video. It can be seen from the figures that compared to the no-rain video, the rain line is smoother and sparser in the vertical direction. Therefore, by using… of Norms are used to enhance the smoothness of rain lines in the vertical direction;

[0071] Figure 3 (a) is a histogram of the absolute values ​​of the differences in the horizontal direction for rain lines (Figure II), and (b) is a histogram of the absolute values ​​of the differences in the horizontal direction for rainless videos (Figure II). It can be seen from the figures that the differences in the horizontal direction between rain lines and rainless videos are smoother in rainless videos, therefore... of Norms are used to characterize the smoothness of rainless videos in the horizontal direction;

[0072] Figure 4 (a) is a histogram of the absolute values ​​of the temporal differences of the rain line (Figure III), and (b) is a histogram of the absolute values ​​of the temporal differences of the rainless video (Figure III). It can be seen from the figures that the temporal differences of the rain line and the clean video are smoother in the rainless video, therefore... The l1 norm is used to characterize the smoothness of rainless video in the temporal direction;

[0073] This application utilizes an adaptive tensor weighted nuclear norm to ensure the global low rank of rainless videos;

[0074] For the i-th expansion matrix X (i) We assume that their singular values ​​in ascending order are:

[0075]

[0076] Where: k i It is X (i) The singular values;

[0077] The adaptive weights are defined as follows:

[0078] First, define a parameter 0 < p < 1, which represents the information proportion of the first k singular values ​​among all singular values. Mathematically, this can be defined as:

[0079]

[0080] Then, a threshold T is defined, and the minimum k is found when P≥T. For different expansions, the minimum k is denoted as k1, k2, and k3, respectively. In this paper, the threshold T is set to 0.85 in all experiments.

[0081] Next, using the singular value number n i For k i Normalize to ensure that the values ​​are on the same scale:

[0082]

[0083] parameter The smaller the value of w, the stronger the low-rank property during the i-th expansion; therefore, a smaller weight w should be set in the formula. i This is done to maintain low rank and enforce better low-rank regularization. Conversely, if in the i-th expansion... A larger value indicates that the expansion has a weaker low-rank property, and a larger weight w should be set. i This reduces the contribution of low-rank regularization to the expansion.

[0084] Due to the decaying property of the singular value distribution, we can use an exponential function to calculate the normalized value, and finally set the adaptive weights as follows:

[0085]

[0086] Figure 5 (a) is a video image of a rainless video; (b) is a distribution of singular values ​​in different dimensions of the rainless video, showing that the time dimension exhibits strong low-rank property. Through the analysis of the above prior information and regularization terms,

[0087] Furthermore, the expression for the rain line removal model is as follows:

[0088]

[0089]

[0090] in: There is a rainy video. It's a video without rain. It's a rain line. These are total variation operators in a single direction, respectively, along the rain line direction and along the vertical direction. It is a time-difference operator, where {α1,α2,α3,α4} are experimental parameters.

[0091] It uses an adaptive tensor weighted nuclear norm to characterize the overall low-rank property of clean videos.

[0092] Furthermore, the rain line removal model is solved using the alternating direction multiplier method (ADMM) framework, as follows:

[0093] By introducing intermediate variables The expression for the rain line removal model is represented by the following equivalent constraints:

[0094]

[0095]

[0096] Where: Unfold(χ) (i) This refers to the operation of expanding a tensor along its i-th dimension;

[0097] The augmented Lagrange function in the above equation is:

[0098]

[0099] Where: Λ=[Λ1,Λ2,Λ3,Λ4,Λ5] are Lagrange multipliers, and β=[β1,β2,β3,β4,β5] are nonnegative scalar parameters;

[0100] Within the framework of the alternating direction multiplier method, the problem to be solved is divided into six simpler subproblems. The solution is obtained by alternating steps;

[0101] Fix other variables At that time, Y, S, X, T are iteratively updated in the following manner:

[0102]

[0103] The sub-problems are as follows:

[0104]

[0105] The subproblem is a least squares problem as shown below:

[0106]

[0107] Formula (8) has the following closed-form solution:

[0108]

[0109] in: and Let these represent the Fast Fourier Transform and its inverse transform, respectively, where:

[0110]

[0111]

[0112] Multiplier update: Based on the framework of the alternating direction multiplier method, the Lagrange multiplier Λ = [Λ1,Λ2,Λ3,Λ4,Λ5] is updated as follows.

[0113]

[0114] Figure 6 (a) is the video frame I with rain, (b) is the video frame I without rain;

[0115] Figure 7 (a) is the video frame II with rain, (b) is the video frame II without rain.

[0116] A video deraining device based on adaptive tensor weighted nuclear norm, characterized in that it includes:

[0117] Acquisition module: Used to acquire rainy videos where rain lines need to be removed;

[0118] Analysis module: Used to analyze discriminative prior information between rainless videos and rain lines;

[0119] Module: Used to build a rain line removal model based on prior information for removing rain lines from rainy videos;

[0120] Rain removal module: used to remove rain lines from rainy videos based on the rain removal model.

[0121] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

[0122] [1]HASEH,MIYAKEK,YONEDAM.Real-timesnowfallnoiseelimination[C] / / ProcofIEEEInternationalConferenceonImageProcessing.NewYork:IEEEPress,1999:406-409.

[0123] [2]GARGK,NAYARSK.Visionandrain[J].InternationalJournalofComputerVision,2007,75(1):3-27.

[0124] [3]ZHUQS,SHAOL,HENGPA,etal.Anovelraindetectionandremovalapproachusingguidedfilteringandformationmodeling[C] / / ProceedingsoftheIEEEInternationalConferenceonRoboticsandBiomimetics.NewYork:IEEEPress,2013:563-567.

[0125] [4]WANGC,SHENMM,YAOC.Rainstreakremovalbymulti-frame-basedanisotropicfiltering[J].MultimediaToolsandApplication,2017,76(2):2019-2038.

[0126] [5]BARNUMP,KANADET,NARASIMHANSG.Spatiotemporalfrequencyanalysisforremovingrainandsnowfromvideos[C] / / In:WorkshoponPhotometricAnalysisForComputerVision(PACV),inConjunctionwithInternationalConferenceonComputerVision.NewYork:IEEEPress,2007:1-8.

[0127] [6]SANTHASEELANV,ASARIVK.Utilizinglocalphaseinformationtoremoverainfromvideo[J].InternationalJournalofComputerVision,2015,112(1):71-89.

[0128] [7]XUEXW,ZHANGCY,GOTOSS.Motionrobustraindetectionandremovalfromvideos[C] / / ProceedingsofIEEEMMSP.NewYork:IEEEPress,2012:170-174.

[0129] [8]TOMASC,MANDUCHIR.Bilateralfilteringforgrayandcolorimages[C] / / ProceedingofIEEEInternationalConferenceonComputerVision.NewYork:IEEEPress,1998:839-846.

[0130] [9]KANGLW,LINCW,FUYH.Automaticsingle-image-basedrainstreaksremovalviaimagedecomposition[J].IEEETransactionsonImageProcessing,2011,21(4):1742-1755.

[0131]

[10] KIMJH,SIMJY,KIMCS.Videoderaininganddesnowingusingtemporalcorrelationandlow-rankmatrixcompletion[J].IEEETransactionsonImageProcessing,2015,24(9):2658-2670.

[0132]

[11] T.-X.Jiang,T.-Z.Huang,X.-L.Zhao,et al.A Novel Tensor-based VideoRainStreaks Removal Approach via Utilizing Discriminatively Intrinsic Priors[C].IEEEComputer Vision and Pattern Recognition(CVPR),2017,4057–4066。

Claims

1. A video deraining method based on adaptive tensor weighted nuclear norm, characterized in that: Includes the following steps: Obtain a video containing rain lines that need to be removed; Analyze discriminative prior information between rainless videos and rain lines; A rain removal model for removing rain streaks from rainy videos is constructed based on prior information. Rain lines are removed from rainy videos based on the aforementioned rain removal model; The expression for the rain line removal model is as follows: （5） （6） in: There is a rainy video. It's a video without rain. It's a rain line. , These are total variation operators in a single direction, respectively, along the rain line direction and along the vertical direction. It is a time-difference operator. These are experimental parameters; It uses an adaptive tensor weighted nuclear norm to characterize the overall low-rank property of rainless videos; The prior information includes the fact that rain lines are sparser compared to rainless videos, using regularization terms. To depict rain lines, and the rain lines have a stronger smoothness in the vertical direction, using... of Norms are used to characterize the smoothness of rain lines in the vertical direction; Compared to rain lines, rainless videos exhibit stronger smoothness in both the horizontal and temporal directions, therefore, they are used respectively. of Norm and of Norms are used to characterize the smoothness of rainless videos in the horizontal and temporal directions, and the strength of low-rank properties in different dimensions of the rainless video is considered. Different weights are assigned according to the low-rank properties in different dimensions to ensure the global low-rank of the rainless video, expressed as: .

2. The video deraining method based on adaptive tensor weighted nuclear norm according to claim 1, characterized in that: The rain line removal model uses the alternating direction multiplier method. The solution process is as follows: By introducing four intermediate variables The expression for the rainline removal model is represented by the following equivalent constraints: （7） （8） The augmented Lagrangian function in the above equation is: （9） in: It is a Lagrange multiplier. It is a non-negative scalar parameter; Within the framework of the alternating direction multiplier method, the problem to be solved is divided into subproblems. Solve for; Alternating solutions; Fix other parameter variables hour, Iterative updates are performed using the following methods: （10） The sub-problems are as follows: （11） The subproblem is a least squares problem as shown below: （12） Formula (8) has the following closed-form solution: （13） in: Let these represent the Fast Fourier Transform and its inverse transform, respectively, where: （14） （15） Multiplier Update: Based on the framework of the alternating direction multiplier method, Lagrange multipliers The update method is as follows. （16） 3. A video deraining device based on adaptive tensor weighted nuclear norm, characterized in that: include: Acquisition module: Used to acquire rainy videos where rain lines need to be removed; Analysis module: Used to analyze prior information that distinguishes between rainless videos and rain lines; Module: Used to build a rain line removal model based on prior information for removing rain lines from rainy videos; Rain removal module: used to remove rain streaks from rainy videos based on the rain removal model; The expression for the rain line removal model is as follows: （5） （6） in: There is a rainy video. It's a video without rain. It's a rain line. , These are total variation operators in a single direction, respectively, along the rain line direction and along the vertical direction. It is a time-difference operator. These are experimental parameters; It uses an adaptive tensor weighted nuclear norm to characterize the overall low-rank property of rainless videos; The prior information includes the fact that rain lines are sparser compared to rainless videos, using regularization terms. To depict rain lines, and the rain lines have a stronger smoothness in the vertical direction, using... of Norms are used to characterize the smoothness of rain lines in the vertical direction; Compared to rain lines, rainless videos exhibit stronger smoothness in both the horizontal and temporal directions, therefore, they are used respectively. of Norm and of Norms are used to characterize the smoothness of rainless videos in the horizontal and temporal directions, and the strength of low-rank properties in different dimensions of the rainless video is considered. Different weights are assigned according to the low-rank properties in different dimensions to ensure the global low-rank of the rainless video, expressed as: .

Images