Image sequence denoising method and device based on spatiotemporal domain total variation regularization

Abstract

The present disclosure relates to a method and device for image sequence denoising based on spatiotemporal total variation regularization, the method comprising: acquiring an image sequence and analyzing the image sequence to obtain prior information; determining a target function about a target image sequence based on the image sequence and the prior information, the target image sequence being used to represent a denoised image sequence, the target function comprising a data fidelity term and a regularization term, the data fidelity term being used to indicate a maximum likelihood estimation function based on additive white Gaussian noise, and the regularization term being used to indicate a spatiotemporal total variation regularization function; performing dualization processing on the target function to obtain a dual function about a dual variable, and performing projection gradient updating on the dual function until a preset condition is met, with the maximum of the dual function as the target; and generating the target image sequence based on the dual variable when the preset condition is met. The present disclosure can effectively denoise the image sequence.

Description

Technical Field

[0001] This disclosure relates to the field of digital image processing technology, and in particular to an image sequence denoising method and apparatus based on spatiotemporal domain total variational regularization. Background Technology

[0002] Noise is inevitably generated during image acquisition and transmission, leading to image quality degradation and information loss. Although the performance of detection devices has rapidly evolved in recent years, high-quality image acquisition and information gathering still face challenges in certain scenarios. For example, in dark environments, the light signal received by the detector is extremely weak, and the acquired digital signal is often submerged in noise. Furthermore, many applications, including mobile phone photography and security monitoring, have strict limitations on camera size; the photosensitive area and pixel size of detection devices are typically small, resulting in limited light sensitivity. Consequently, raw images acquired in low-light environments often contain high levels of noise. Therefore, image denoising algorithms—computer numerical processing methods for recovering realistic scene information from noisy images—have significant application value in many fields.

[0003] Based on their underlying principles, existing image denoising algorithms can be broadly categorized into spatial filtering (such as Wiener filtering, median filtering, and bilateral filtering), sparse and low-rank priors, statistical methods (such as expert field models), 3D block matching algorithms (Block-Matching and 3D Filtering, BM3D), and deep learning (such as convolutional neural networks and visual Transformers). These algorithms have been applied in numerous real-world image denoising scenarios. Currently, most mainstream image denoising methods are based on single-frame images. While these methods can achieve excellent results when images contain low levels of noise, in extremely low-light imaging scenarios, such as when the electrical signal converted from incident photons is comparable to the detector's dark field noise level, the signal-to-noise ratio of the acquired raw data is extremely low. Much of the information in the real scene is buried in noise, making it difficult to fully extract from a single frame. Furthermore, for moving scenes, increasing exposure time cannot directly improve image quality. Therefore, denoising algorithms based on video or image sequences have been proposed. Since image sequences carry richer information than single-frame images, they can fully utilize the differences in the statistical characteristics of signal and noise in the spatial and temporal dimensions to decouple them. Therefore, denoising algorithms based on video or image sequences can theoretically achieve better performance than single-frame image denoising algorithms. In 2012, M. Maggioni et al. proposed the V-BM4D video denoising algorithm based on the BM3D algorithm, which utilizes spatiotemporal three-dimensional data for block matching and collaborative filtering. In 2018, P. Arias et al. proposed a nonlocal Bayesian denoising method for video. In 2019, M. Tassano et al. proposed the FastDVDnet deep neural network, which uses 5 temporal images as input and can quickly denoise videos containing different levels of Gaussian noise. In 2022, K. Monakhova et al. also used deep learning methods to achieve 10... -4 Video denoising and enhancement in low-light environments at the lx level. However, current deep learning-based image sequence denoising methods are often only applicable to specific scenes, noise levels, or noise statistical distributions, while traditional methods such as collaborative filtering and Bayesian methods also face problems of high computational cost and long processing time. Therefore, existing image sequence denoising methods still need further improvement in terms of robustness and computational efficiency. Summary of the Invention

[0004] In view of this, this disclosure proposes an image sequence denoising method and apparatus based on spatiotemporal domain total variational regularization, which can effectively denoise image sequences and has robustness and efficiency.

[0005] According to one aspect of this disclosure, an image sequence denoising method based on spatiotemporal domain total variation regularization is provided, comprising:

[0006] Acquire a sequence of images and analyze the sequence to obtain prior information. The sequence of images is used to represent a sequence of images containing noise.

[0007] Based on the acquired image sequence and the prior information, an objective function is constructed for the target image sequence, wherein the target image sequence represents the image sequence after noise removal, and the objective function includes a data fidelity term and a regularization term. The data fidelity term indicates the maximum likelihood estimation function based on additive white Gaussian noise, and the regularization term indicates the spatiotemporal total variation regularization function.

[0008] The objective function is dualized to obtain a dual function with respect to the dual variable. The dual function is then updated by projecting gradients with the goal of maximizing the dual function until a preset condition is met.

[0009] The target image sequence is generated based on the dual variables that satisfy the preset conditions.

[0010] This disclosure, on the one hand, utilizes a spatiotemporal domain total variational regularization function to construct the objective function. This not only fully leverages the differences in statistical characteristics between signal and noise in the spatiotemporal dimensions to achieve effective decoupling, but also models the motion between frames in the image sequence, avoiding motion blur and achieving optimal dynamic scene denoising. On the other hand, it utilizes a maximum likelihood estimation function based on additive white Gaussian noise to construct the objective function, which can robustly suppress the noise distribution that may actually exist. Simultaneously, by employing a dual objective function, the problem of minimizing the objective function is transformed into a problem of maximizing the dual function, and the dual problem is solved using the projection gradient method. This allows for a more efficient solution to the target image sequence, thereby achieving effective denoising of the image sequence with robustness and efficiency.

[0011] In one possible implementation, the prior information includes at least a regularization coefficient, which indicates the weight relationship between the data fidelity term and the regularization term, and the regularization coefficient is determined based on the noise level.

[0012] The embodiments disclosed herein balance the weight relationship between the data fidelity term and the regularization term by adjusting the regularization coefficient. That is, by selecting an appropriate regularization coefficient, it can be applied to any level of noise, thereby suppressing different levels of noise and maintaining a balance between the denoising amplitude and scene fidelity, thus exhibiting generalization ability.

[0013] In one possible implementation, the prior information includes at least weighting factors, which include a width-direction weighting factor, a height-direction weighting factor, and a time-direction weighting factor, wherein the weighting factors are determined based on the spatial distribution and motion characteristics of the scene in which the acquired image sequence is located.

[0014] This embodiment of the present disclosure determines appropriate width, height, and time weighting factors by analyzing the acquired image sequence to be denoised, and applies corresponding sparsity constraints in the width, height, and time dimensions. This not only enables noise suppression by utilizing spatial smoothness and temporal coherence, but also avoids motion blur while restoring and improving image quality.

[0015] In one possible implementation, the prior information includes at least a set of sparse transform translations, which includes at least a set of sparse transform operators. Each set of sparse transform operators includes a first translation, a second translation, and a third translation. The first translation indicates the number of pixels that the target image sequence to be determined is translated along the width direction, the second translation indicates the number of pixels that the target image sequence to be determined is translated along the height direction, and the third translation indicates the number of pixels that the target image sequence to be determined is translated along the time direction.

[0016] This embodiment of the present disclosure denoises the width, height, and time dimensions of an image sequence by setting translation amounts along the width, height, and time directions, respectively. This achieves good denoising results while reducing computational complexity and improving computational efficiency.

[0017] In one possible implementation, constructing the objective function for the target image sequence based on the acquired image sequence and the prior information includes: constructing the data fidelity term based on the acquired image sequence and the target image sequence to be determined; and constructing the regularization term based on the prior information, the target image sequence to be determined, and a linear transformation; wherein the width, height, and number of frames of the images in the target image sequence to be determined are all in the real number domain, and the linear transformation is used to indicate translating the target image sequence to be determined by a first translation amount along the width direction, a second translation amount along the height direction, and a third translation amount along the time direction.

[0018] This disclosure transforms the image sequence denoising problem into its inverse problem for solution. It can be extended to data of any dimension, with no specific limitations on the width, height, or number of frames of the image sequence. Furthermore, it can utilize prior information to achieve a balance between denoising amplitude and scene fidelity, and has a relatively robust denoising effect for random noise with arbitrary statistical characteristics.

[0019] In one possible implementation, the dualization of the objective function to obtain a dual function with respect to the dual variable includes: determining multiple difference operators based on the identity matrix and the linear transformation, the number of difference operators being determined based on the number of sparse transformation operators in the prior information; concatenating all the multiple difference operators to obtain a difference matrix; and performing dualization on the objective function based on the difference matrix to obtain the dual function, the dual function being used to indicate the relationship between the acquired image sequence, the difference matrix, and the dual variable.

[0020] This embodiment of the disclosure utilizes dualization in convex optimization theory to transform the original problem into a dual problem that is easier to solve, that is, to transform the objective function into a dual function for solving, thus making it easier to obtain the target image sequence.

[0021] In one possible implementation, updating the dual function by projecting its gradient with the objective of maximizing the dual function until a preset condition is met includes: initializing the dual variable and the auxiliary variable, wherein the auxiliary variable is used to determine the direction of the projected gradient; updating the dual variable along the direction of the projected gradient with the objective of maximizing the dual function until the preset condition is met, wherein the projected gradient represents the projection of the auxiliary variable about a first set, the first set being the set of all dual variables that satisfy the condition that the maximum value of the elements in the dual variable is less than or equal to the regularization coefficient in the prior information, and the projected gradient is determined based on the auxiliary variable and the regularization coefficient in the prior information.

[0022] The embodiments disclosed herein use a projection gradient method accelerated by the Nesterov momentum method to solve the dual problem. When the number of iterations reaches a predetermined number or a specific convergence criterion is met, the target dual variables can be obtained, which is beneficial for subsequent determination of the target image sequence. In addition, the calculation method involved in the projection gradient method is conducive to parallel processing and has high efficiency.

[0023] In one possible implementation, generating the target image sequence based on the dual variables that satisfy the preset conditions includes: converting the dual variables that satisfy the preset conditions into the target image sequence according to a relation; wherein the relation is used to indicate the relationship between the target dual variables, the acquired image sequence, the difference matrix, and the target image sequence.

[0024] This embodiment utilizes the strong duality condition satisfied by the convex optimization problem to determine the relationship between the solution of the objective function and the solution of the dual function. After determining the objective dual variable, the target image sequence can be determined according to this relationship to achieve denoising of the acquired image sequence and simplify the computational complexity.

[0025] In one possible implementation, the preset condition is that the number of times the projection gradient is updated reaches a preset threshold, or the preset condition is that the projection gradient is updated to meet a preset convergence condition.

[0026] This disclosure provides different preset conditions, and the appropriate preset conditions can be selected according to actual needs, so as to generate the target image sequence by using the dual variables when the preset conditions are met, which is flexible.

[0027] According to another aspect of this disclosure, an image sequence denoising device based on spatiotemporal domain total variational regularization is provided, comprising: an acquisition and analysis module configured to acquire a sequence of images and analyze the sequence to obtain prior information, wherein the acquired image sequence represents a noisy image sequence; a function construction module configured to construct an objective function about a target image sequence based on the acquired image sequence and the prior information, wherein the target image sequence represents a noise-removed image sequence, the objective function including a data fidelity term and a regularization term, wherein the data fidelity term indicates a maximum likelihood estimation function based on additive white Gaussian noise, and the regularization term indicates a spatiotemporal domain total variational regularization function; a function processing module configured to perform dualization processing on the objective function to obtain a dual function about a dual variable, and to update the dual function by projecting gradients with the goal of maximizing the dual function until a preset condition is met; and a sequence generation module configured to generate the target image sequence based on the dual variable when the preset condition is met.

[0028] In one possible implementation, the prior information includes at least a regularization coefficient, which indicates the weight relationship between the data fidelity term and the regularization term, and the regularization coefficient is determined based on the noise level.

[0029] In one possible implementation, the prior information includes at least weighting factors, which include a width-direction weighting factor, a height-direction weighting factor, and a time-direction weighting factor, wherein the weighting factors are determined based on the spatial distribution and motion characteristics of the scene in which the acquired image sequence is located.

[0030] In one possible implementation, the prior information includes at least a set of sparse transform translations, which includes at least a set of sparse transform operators. Each set of sparse transform operators includes a first translation, a second translation, and a third translation. The first translation indicates the number of pixels that the target image sequence to be determined is translated along the width direction, the second translation indicates the number of pixels that the target image sequence to be determined is translated along the height direction, and the third translation indicates the number of pixels that the target image sequence to be determined is translated along the time direction.

[0031] In one possible implementation, constructing the objective function for the target image sequence based on the acquired image sequence and the prior information includes: constructing the data fidelity term based on the acquired image sequence and the target image sequence to be determined; and constructing the regularization term based on the prior information, the target image sequence to be determined, and a linear transformation; wherein the width, height, and number of frames of the images in the target image sequence to be determined are all in the real number domain, and the linear transformation is used to indicate translating the target image sequence to be determined by a first translation amount along the width direction, a second translation amount along the height direction, and a third translation amount along the time direction.

[0032] In one possible implementation, the dualization of the objective function to obtain a dual function with respect to the dual variable includes: determining multiple difference operators based on the identity matrix and the linear transformation, the number of difference operators being determined based on the number of sparse transformation operators in the prior information; concatenating all the multiple difference operators to obtain a difference matrix; and performing dualization on the objective function based on the difference matrix to obtain the dual function, the dual function being used to indicate the relationship between the acquired image sequence, the difference matrix, and the dual variable.

[0033] In one possible implementation, updating the dual function by projecting its gradient with the objective of maximizing the dual function until a preset condition is met includes: initializing the dual variable and the auxiliary variable, wherein the auxiliary variable is used to determine the direction of the projected gradient; updating the dual variable along the direction of the projected gradient with the objective of maximizing the dual function until the preset condition is met, wherein the projected gradient represents the projection of the auxiliary variable about a first set, the first set being the set of all dual variables that satisfy the condition that the maximum value of the elements in the dual variable is less than or equal to the regularization coefficient in the prior information, and the projected gradient is determined based on the auxiliary variable and the regularization coefficient in the prior information.

[0034] In one possible implementation, generating the target image sequence based on the dual variables that satisfy the preset conditions includes: converting the dual variables that satisfy the preset conditions into the target image sequence according to a relation; wherein the relation is used to indicate the relationship between the target dual variables, the acquired image sequence, the difference matrix, and the target image sequence.

[0035] In one possible implementation, the preset condition is that the number of times the projection gradient is updated reaches a preset threshold, or the preset condition is that the projection gradient is updated to meet a preset convergence condition.

[0036] According to another aspect of this disclosure, an image sequence denoising apparatus based on spatiotemporal domain total variational regularization is provided, comprising: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to implement the above-described image sequence denoising method when executing the instructions stored in the memory.

[0037] According to another aspect of this disclosure, a non-volatile computer-readable storage medium is provided, on which computer program instructions are stored, wherein the computer program instructions, when executed by a processor, implement the above-described image sequence denoising method.

[0038] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Other features and aspects of this disclosure will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0039] The accompanying drawings, which are included in and form part of this specification, illustrate exemplary embodiments, features, and aspects of this disclosure together with the specification and serve to explain the principles of this disclosure.

[0040] Figure 1 A flowchart illustrating an image sequence denoising method provided according to an embodiment of the present disclosure is shown.

[0041] Figure 2 A schematic diagram of an image sequence denoising method provided according to an embodiment of the present disclosure is shown.

[0042] Figure 3 This diagram illustrates the denoising result of image sequence 1 at a signal-to-noise ratio of 5dB, according to an embodiment of the present disclosure.

[0043] Figure 4 This diagram illustrates the denoising result of image sequence 2 at a signal-to-noise ratio of 5dB, according to an embodiment of the present disclosure.

[0044] Figure 5This diagram illustrates the denoising result of image sequence 1 at a signal-to-noise ratio of 0dB, according to an embodiment of the present disclosure.

[0045] Figure 6 This diagram illustrates the denoising result of image sequence 2 with a signal-to-noise ratio of 0dB, according to an embodiment of the present disclosure.

[0046] Figure 7 This diagram illustrates the denoising results of a noisy image sequence in a low-light clock scene according to an embodiment of the present disclosure.

[0047] Figure 8 This diagram illustrates the denoising results of a noisy image sequence in a low-light scene at a school gate, according to an embodiment of the present disclosure.

[0048] Figure 9 A block diagram of an image sequence denoising apparatus provided according to an embodiment of the present disclosure is shown.

[0049] Figure 10 A block diagram of an apparatus for performing an image sequence denoising method according to an embodiment of the present disclosure is shown. Detailed Implementation

[0050] Various exemplary embodiments, features, and aspects of this disclosure will now be described in detail with reference to the accompanying drawings. The same reference numerals in the drawings denote elements that have the same or similar functions. Although various aspects of the embodiments are shown in the drawings, they are not necessarily drawn to scale unless specifically indicated otherwise.

[0051] The term “exemplary” as used herein means “serving as an example, embodiment, or illustration.” Any embodiment illustrated herein as “exemplary” is not necessarily to be construed as superior to or better than other embodiments.

[0052] Furthermore, to better illustrate this disclosure, numerous specific details are set forth in the following detailed description. Those skilled in the art will understand that this disclosure can be practiced without certain specific details. In some instances, methods, means, components, and circuits well known to those skilled in the art have not been described in detail in order to highlight the main points of this disclosure.

[0053] To facilitate understanding of the technical solutions provided by the embodiments of this disclosure by those skilled in the art, the technical environment for implementing the technical solutions will be described below.

[0054] Noise is inevitably generated during image acquisition and transmission, leading to image quality degradation and information loss. Therefore, image denoising methods are needed to restore the true image as much as possible. Image denoising methods are computer numerical processing methods that recover the true scene information from noisy images. Although some image denoising methods are provided in related technologies, in certain scenarios, such as low-light imaging scenes, moving scenes, and imaging scenes with strict limitations on camera size (e.g., mobile phone photography, security monitoring), existing image denoising methods cannot achieve ideal denoising effects, and high-quality image acquisition and information acquisition still face challenges.

[0055] Most current mainstream image denoising methods are based on single-frame images. While these methods can achieve excellent results when the image contains low levels of noise, in low-light imaging scenarios, such as when the electrical signal converted from the incident photons is comparable to the noise level of the detector's dark field, the signal-to-noise ratio of the acquired raw data is extremely low. Most of the information in the real scene is buried in noise and cannot be fully extracted from a single frame image. Furthermore, for moving scenes, increasing the exposure time cannot directly improve image quality. In addition, current deep learning-based image sequence denoising methods are mostly designed for image sequences of fixed size or number of frames, and are often only applicable to specific scenes, noise levels, or noise statistical distributions. Traditional methods such as collaborative filtering and Bayesian methods also face problems of high computational cost and long processing time. Therefore, existing image denoising methods still need further improvement in terms of scalability, generalization, robustness, and computational efficiency.

[0056] The image sequence denoising method provided in this disclosure utilizes a spatiotemporal domain total variational regularization function to construct the objective function. This not only fully leverages the differences in statistical features between signal and noise in the spatiotemporal dimensions to achieve effective decoupling but also models the motion between sequence frames, avoiding motion blur and achieving optimal dynamic scene denoising. Furthermore, by constructing the objective function using a maximum likelihood estimation function based on additive white Gaussian noise, it can robustly suppress potentially existing noise distributions. Simultaneously, by employing a dual objective function, the problem of minimizing the objective function is transformed into maximizing the dual function, and the dual problem is solved using the projection gradient method. This allows for a more efficient solution to the target image sequence, thus achieving effective denoising of the image sequence. This method is robust and efficient, solving problems such as unsatisfactory denoising effects, limited application scenarios, and high computational overhead. Furthermore, the image sequence denoising method provided in this disclosure can solve the image sequence denoising problem under arbitrary noise levels based on appropriate regularization coefficients, maintaining a balance between denoising amplitude and scene fidelity, and has generalization ability; and it can be extended to data of arbitrary dimensions, with no specific limitations on the width, height and number of sequence frames of the image to be denoised, and has scalability.

[0057] When representing a dynamic natural scene in a computer, its spatial and temporal distribution needs to be discretized. A discretized image sequence can be represented as a three-dimensional tensor. Where, n ξ n represents the width of the images in the image sequence. υ n represents the height of the images in the image sequence. T This indicates the number of frames in the image sequence. Represents the real number field.

[0058] The actual acquired image sequence can be expressed as a superposition of the original image sequence and noise, that is, the imaging process can be expressed as the following forward model:

[0059] Formula 1: Y = X + N

[0060] In the formula, This refers to the acquired image sequence containing noise, hereinafter referred to as the acquired image sequence; This represents the true, noise-free image sequence, i.e., the original image sequence, which is the image sequence that we hope to restore through image sequence denoising methods. The closer the image sequence obtained by denoising the acquired image sequence is to the original image sequence, the better the denoising effect of this image sequence denoising method. This represents measurement noise. For ease of subsequent annotation and derivation, the entire image sequence is represented using a vectorized form:

[0061] Formula 2: y = x + n

[0062] In the formula, Represents the vectorized sequence of acquired images Represents the vectorized original image sequence This represents the measurement noise, and n = n ξ n v n τ It is worth noting that the purpose of vectorization is to simplify mathematical expression. In the actual implementation of denoising methods, there is no need to vectorize the data. For the sake of simplicity, x and y will be referred to as the original image sequence and the acquired image sequence, respectively. In addition, the width, height, number of sequence frames, and translation amount in the linear transformation below all refer to the image sequence.

[0063] Image sequence denoising refers to the process of restoring a real, noise-free image sequence given a noisy image sequence. Figure 1 A flowchart illustrating an image sequence denoising method provided according to embodiments of the present disclosure is shown. Figure 1 As shown, the method may include:

[0064] Step S101: Obtain the acquired image sequence and analyze the acquired image sequence to obtain prior information.

[0065] Step S102: Construct an objective function for the target image sequence based on the acquired image sequence and prior information.

[0066] Step S103: Perform dualization on the objective function to obtain the dual function with respect to the dual variable, and update the dual function by projecting gradient with the goal of maximizing the dual function until the preset conditions are met.

[0067] Step S104: Generate the target image sequence based on the dual variables that meet the preset conditions.

[0068] The acquired image sequence can be used to represent the acquired image sequence containing noise. The target image sequence can be used to represent the image sequence after noise removal. The target image sequence can be obtained through an objective function. The construction of the objective function represents the construction of the original noise-free scene. Therefore, the smaller the value of the objective function, the closer the obtained target image sequence is to the original image sequence, and the more it satisfies the goal of image sequence denoising. That is, we hope that the final generated target image sequence is as close as possible to the original image sequence.

[0069] Thus, on the one hand, constructing the objective function using the spatiotemporal domain total variational regularization function can not only fully utilize the differences in statistical characteristics between signal and noise in the spatiotemporal dimensions to achieve effective decoupling, but also model the motion between sequence frames, avoiding motion blur and achieving the best dynamic scene denoising effect. On the other hand, constructing the objective function using the maximum likelihood estimation function based on additive white Gaussian noise can have a relatively robust suppression effect on all possible noise distributions. At the same time, by adopting the dual objective function approach, the problem of minimizing the objective function is transformed into the problem of maximizing the dual function, and the dual problem is solved using the projection gradient method, which can more efficiently solve the target image sequence, thereby achieving effective denoising of the image sequence with robustness and efficiency.

[0070] In step S101, preliminary analysis of the acquired image sequence can obtain prior information about the original image sequence. In one example, the prior information may include at least one or more of the following: regularization coefficients, weighting factors, and sparse transform translation amounts.

[0071] The regularization coefficient λ can be determined based on the noise level. Generally, λ > 0. Typically, the higher the noise level, the greater the difference between the acquired image sequence and the real scene. Therefore, a larger value for λ is often needed to apply stronger sparsity constraints to achieve effective noise suppression. Conversely, when the noise level is low, a smaller value for λ can be used to make the reconstructed target image sequence have higher consistency with the acquired image sequence. The regularization coefficient λ can be used to indicate the weight relationship between the data fidelity term and the regularization term (see below) in the objective function. By adjusting the regularization coefficient to balance the weight relationship between the data fidelity term and the regularization term, that is, by selecting an appropriate regularization coefficient, it can be applied to any level of noise, suppress different levels of noise, and maintain a balance between denoising amplitude and scene fidelity, thus exhibiting generalization ability.

[0072] The weighting factors can be determined based on the spatial distribution and motion characteristics of the scene in which the acquired image sequence is located. The weighting factors include the width-direction weighting factor α. ξ Altitude-direction weighting factor α υ and time direction weighting factor α τ A larger value for the weighting factor indicates a stronger sparsity constraint imposed on the corresponding dimension. Generally, α... ξ ,α υ ,α τ ∈[0,1]. Generally, when the scene spatial distribution has a block-smooth characteristic, i.e., mainly composed of sharp edges and smooth regions, it exhibits strong sparsity in the spatial gradient domain, allowing for the selection of a larger α. ζ and α υ This allows for noise suppression by leveraging spatial smoothness; however, when the scene changes slowly over time or contains relatively static parts, it exhibits strong sparsity in the temporal gradient domain, allowing for the selection of a larger α. τ This allows for noise suppression by utilizing temporal coherence. In extreme cases, when α... τ When α = 0, no sparsity constraint is applied in the time domain, which is equivalent to performing single-frame denoising on each frame of the image sequence; while when α = 0... τ When T = 1 and λ → +∞, the time domain constraint is extremely strong, which is equivalent to performing mean denoising on the image sequence. When the value of T is between these two, the advantages of both basic methods can be combined to fully exploit the temporal prior of the signal, improve the restoration quality, and avoid motion blur. By analyzing the acquired image sequence to be denoised, appropriate width, height, and time direction weighting factors are determined. Appropriate sparsity constraints are applied in the width, height, and time dimensions. This not only utilizes spatial smoothness and temporal coherence to achieve noise suppression but also avoids motion blur while improving image quality.

[0073] Here, the sparse transform translation set P represents the set of all translations (i,j,k). The sparse transform translation set P can include at least one set of sparse transform operators (i,j,k), each set of sparse transform operators (i,j,k) can include a first translation i, a second translation j, and a third translation k. The first translation i can be used to indicate the number of pixels that the target image sequence to be determined is translated along the width direction. The second translation j can be used to indicate the number of pixels that the target image sequence to be determined is translated along the height direction. The third translation k can be used to indicate the number of pixels that the target image sequence to be determined is translated along the time direction. By setting translations along the width, height, and time directions respectively to denoise the width, height, and time dimensions of the image sequence, good denoising results can be achieved while reducing computational complexity and improving computational efficiency.

[0074] After analyzing and determining suitable prior information, such as regularization coefficients, weighting factors, and the set of sparse transform translation amounts, the objective function can be determined by combining the acquired image sequence. The objective function may include a data fidelity term and a regularization term. In one possible implementation, step S102 may include: constructing a data fidelity term based on the acquired image sequence and the target image sequence to be determined, and constructing a regularization term based on the prior information, the target image sequence to be determined, and a linear transform. Here, the width, height, and number of frames of the images in the target image sequence to be determined are all in the real number domain, and the linear transform is used to indicate shifting the target image sequence to be determined by a first translation amount along the width direction, a second translation amount along the height direction, and a third translation amount along the time direction.

[0075] The data fidelity term can be used to indicate the maximum likelihood estimation function based on additive white Gaussian noise. The data fidelity term can be constructed from the acquired image sequence y and the target image sequence x to be determined. The data fidelity term can be used to ensure that the final generated target image sequence x satisfies the measurement constraints.

[0076] The regularization term can be used to indicate the spatiotemporal domain total variation regularization function. The regularization term can be constructed based on the target image sequence x to be determined and prior information about the original image sequence (regularization coefficients, weighting factors, and a set of sparse transform translations). Thus, the spatiotemporal domain total variation regularization function can represent a total variation regularization function containing parameters of both the temporal and spatial dimensions. Determining the regularization term based on prior information about the original image sequence takes into account the ill-conditioned nature of the denoising problem, thus suppressing erroneous solutions. Furthermore, by utilizing the high sparsity of the image sequence in the gradient transform domains of the spatial and temporal dimensions and using the spatiotemporal domain total variation (TV) regularization function as part of the objective function, it can effectively decouple the signal and noise while modeling the motion between sequence frames, avoiding motion blur and achieving optimal dynamic scene denoising results.

[0077] The goal of image sequence denoising is to generate a final image sequence that is as close as possible to the original image sequence. When the image sequence denoising problem is transformed into an objective function problem, the objective of image sequence denoising is also transformed into a problem of minimizing the objective function.

[0078] In one example, the problem of denoising an image sequence can be transformed into an inverse problem that can be solved by the following objective function F(x):

[0079]

[0080] In the formula, F(x) represents the objective function for the target image sequence x to be determined; x represents the (vectorized) target image sequence to be determined. y This represents the (vectorized) sequence of acquired images. λ represents the regularization coefficient, λ > 0; α ξ α represents the width-direction weighting factor. v α represents the height-direction weighting factor. τ α represents the time-direction weighting factor. ξ ,α υ ,α τ ∈[0,1]; P represents the set of sparse transformation translations {(i,j,k)}, (i,j,k) represents the sparse transformation translation, i represents the first translation, j represents the second translation, and k represents the third translation; This represents a linear transformation that shifts the target image sequence to be determined by i pixels along the width direction. This represents a linear transformation that shifts the target image sequence to be determined by j pixels along the height direction. This represents a linear transformation that shifts the sequence of target images to be determined by k pixels along the time direction; n = nξ n v n τ n ξ n represents the width of the target image sequence to be determined. υ n represents the height of the target image sequence to be determined. τ This represents the number of frames in the target image sequence to be determined. Represents the real number field.

[0081] In this example, the data fidelity term can be expressed in the form of L2 squared, which corresponds to the maximum likelihood estimate of additive white Gaussian noise, i.e., in Equation 3. The regularization term can be a total variational regularization function in the spatiotemporal domain, i.e., the one in Formula 3. For the most basic total variational regularization function, P = {(1,0,0),(0,1,0),(0,0,1)}, which considers the gradient information of the image sequence along the three orthogonal directions of width, height, and time with a step size of 1 pixel. For more complex regularization functions such as bilateral total variational regularization, the set P can also include translations greater than 1 pixel and cross terms between the three directions. Generally, the more sparse transform operators P contains for filtering, the better the denoising effect, but the more complex the computation and the more time it takes. Therefore, an appropriate number of sparse transform operators should be selected based on a comprehensive consideration of computational efficiency and denoising effect.

[0082] This disclosure transforms the image sequence denoising problem into its inverse problem for solution. It can be extended to data of any dimension, with no specific limitations on the width, height, or number of frames of the image sequence. Furthermore, it can utilize prior information to achieve a balance between denoising amplitude and scene fidelity, and has a relatively robust denoising effect for random noise with arbitrary statistical characteristics.

[0083] Once the objective function F(x) is determined, the image sequence denoising problem becomes finding the minimum value. x The F(x) problem (i.e. the F(x) minimization problem) is used to obtain the target image sequence that satisfies the denoising effect.

[0084] The dualization process of the objective function in step S103 to obtain the dual function with respect to the dual variable may include: determining multiple difference operators based on the identity matrix and linear transformation, the number of difference operators being determined based on the number of sparse transformation operators in the prior information; connecting all multiple difference operators to obtain a difference matrix; and performing dualization on the objective function based on the difference matrix to obtain the dual function, which is used to indicate the relationship between the acquired image sequence, the difference matrix, and the dual variable.

[0085] In one example, we first simplify the mathematical notation. Assuming that the set P has p elements, we first define the difference operator. Where I represents the n-order identity matrix, and further, all p difference operators D i,j,k Connect them in a certain order to form a matrix. Let the translation along the width, height, and time directions corresponding to the q-th term (1≤q≤p) be i, ... q j q and k q Each pixel. Therefore, the original objective function problem shown in Equation 3 can be equivalently expressed in the following concise form:

[0086]

[0087] In the formula, F(x) represents the objective function for the target image sequence x to be determined; x represents the (vectorized) target image sequence to be determined. y This represents the (vectorized) sequence of acquired images. λ represents the regularization coefficient, λ > 0; D represents the difference matrix, i.e. The definitions of the other parameters involved in the formula are detailed above and will not be repeated here.

[0088] Since the norm term of the objective function in Formula 4 is not differentiable everywhere, directly minimizing this function is difficult. Therefore, the dual method in convex optimization theory is used to solve the problem of minimizing the original objective function (i.e., minimizing...). x F(x) is transformed into a more easily solvable dual problem. In this example, the minimization problem of the objective function shown in Equation 4 can be expressed as the maximization problem of the dual function:

[0089]

[0090] In the formula, H(z) represents the dual function with respect to the dual variable z; Let H(z) represent the problem of maximizing the dual function H(z); z represents the dual variable to be determined, which is the solution to the dual problem. A set C is defined as containing all sets of inequalities ||z|| ∞ ≤λ The set of (||z||) ∞ y represents the maximum value of the elements in vector z; y represents the (vectorized) sequence of acquired images. D represents the difference matrix, i.e. The definitions of the other parameters involved in the formula are detailed above and will not be repeated here.

[0091] This embodiment utilizes dualization in convex optimization theory to transform the original problem into a dual problem that is easier to solve, that is, to transform the objective function into a dual function for solving, which makes it easier to obtain the target image sequence.

[0092] Once the dual function H(z) corresponding to the objective function F(x) is determined, the image sequence denoising problem then becomes finding the minimum value. x F(x) becomes finding the maximum value. z∈C The H(z) problem (i.e., the H(z) maximization problem) is used to determine the target image sequence that satisfies the denoising effect.

[0093] Step S103, which aims to maximize the dual function and update the projection gradient of the dual function until a preset condition is met, may include: initializing the dual variables and auxiliary variables, where the auxiliary variables are used to determine the projection gradient direction; updating the dual variables along the projection gradient direction with the dual function as the goal, until the preset condition is met, wherein the projection gradient represents the projection of the auxiliary variables about a first set, and the first set is the set of all dual variables that satisfy the condition that the maximum value of the elements in the dual variables is less than or equal to the regularization coefficient in the prior information, and the projection gradient is determined based on the auxiliary variables and the regularization coefficient in the prior information.

[0094] In one example, the projected gradient method, accelerated by the Nesterov momentum method, can be used to solve the dual problem shown in Equation 5, i.e., the constrained smooth optimization problem. The solution process may include:

[0095] First, initialize the dual variable z. (0) ←0 and auxiliary variables

[0096] Secondly, iterative updates should be performed in the following manner:

[0097]

[0098] u (t+1) ←z (t+1) +t / (t+3)(z (t+1) -z (t) In formula seven, t = 0 , 1 , 2,... represents the number of iterations; The projection of u onto set C is calculated as follows: ⊙ represents element-wise multiplication, and sgn is the sign function; This represents the gradient of the objective function H(x). u represents the auxiliary variable, and D represents the difference matrix; the explanations of other parameters involved in the formula are detailed above and will not be repeated here.

[0099] Finally, when the preset conditions are met, the final objective dual variable z is obtained. ＊ The objective and dual variables z ＊ That is, the dual variable when the preset conditions are met, i.e., maxz∈C The solution to the H(z) problem. The preset condition can be that the number of projection gradient updates reaches a preset threshold, or it can be that the projection gradient updates satisfy a preset convergence condition. By providing different preset conditions, a suitable preset condition can be selected according to actual needs, so as to utilize the dual variables that satisfy the preset conditions to generate the target image sequence, thus providing flexibility.

[0100] Furthermore, during the iterative solution of the dual problem, when the step size γ for gradient updates, as shown in Equation 6, satisfies the following equation, it will converge to the global optimum:

[0101]

[0102] In the formula, γ represents the step size during gradient update; α ξ α represents the width-direction weighting factor. v α represents the height-direction weighting factor. τ α represents the time-direction weighting factor. ξ ,α υ ,α τ ∈[0,1];i q Describes p difference operators D i,j,k The translation along the width direction of the q-th term (1≤q≤p) in the equation, j q Describes p difference operators D i,j,k The translation along the height direction of the q-th term (1≤q≤p) in the equation, k q Describes p difference operators D i,j,k The q-th term (1≤q≤p) in the time direction is the translation amount along the time direction.

[0103] Therefore, given a weighting factor, the image sequence denoising method can be calculated with a constant step size, without the need for parameter tuning during the iteration process, which is more convenient.

[0104] The embodiments disclosed herein use a projection gradient method accelerated by the Nesterov momentum method to solve the dual problem. When the number of iterations reaches a predetermined number or a specific convergence criterion is met, the target dual variables can be obtained, which is beneficial for subsequent determination of the target image sequence. In addition, the calculation method involved in the projection gradient method is conducive to parallel processing and has high efficiency.

[0105] When the maximum is calculated z∈C When solving the H(z) problem, then according to min x F(x) problem and max z∈C The transformation between H(z) problems, the objective and dual variables z ＊ Convert to target image sequence x ★ That's all.

[0106] Step S104 may further include: converting the dual variables that meet the preset conditions into a target image sequence according to the relation; wherein the relation is used to indicate the relationship between the target dual variables, the acquired image sequence, the difference matrix and the target image sequence.

[0107] In one example, based on the strong duality condition satisfied by the convex optimization problem, it can be determined that the solution to the objective function problem shown in Equation 4 and the solution to the dual problem shown in Equation 5 satisfy the following relationship:

[0108] x ＊ =yD T z ＊ Formula 8

[0109] In the formula, x ＊ This represents the final generated target image sequence; y D represents the sequence of acquired images; D represents the difference matrix; z ★ The dual variable z represents the variable that meets the preset conditions, i.e., the target dual variable. ★ .

[0110] After determining the target dual variables, the final generated target image sequence can be determined through Formula 8, thereby achieving denoising of the acquired image sequence.

[0111] This embodiment utilizes the strong duality condition satisfied by the convex optimization problem to determine the relationship between the solution of the objective function and the solution of the dual function. After determining the objective dual variable, the target image sequence can be determined according to this relationship to achieve denoising of the acquired image sequence and simplify the computational complexity.

[0112] Figure 2 A schematic diagram is shown of an image sequence denoising method provided according to an embodiment of the present disclosure. For example... Figure 2 As shown, the method may include: acquiring a sequence of images containing noise; performing preliminary analysis on the acquired image sequence; determining prior information based on the noise level and the spatiotemporal characteristics of the acquired image sequence; determining the objective function problem corresponding to the image sequence denoising problem; converting the objective function problem into a dual problem; iteratively solving the dual problem using the projection gradient method, i.e., updating the projection gradient of the current dual variable; when a preset condition is met, determining the solution to the dual problem and converting the solution to the objective function problem into a solution, thereby determining the final target image sequence; when the preset condition is not met, continuing to update the projection gradient of the current dual variable until the preset condition is met.

[0113] In this example, two dynamic image sequences are used for simulation testing to quantitatively evaluate the effectiveness of the proposed denoising algorithm. Scene 1 shows a container ship slowly sailing across a river, and image sequence 1 is the image sequence under scene 1. Scene 2 shows the actions and expressions of a woman making a phone call, and image sequence 2 is the image sequence under scene 2. In addition, both scene 1 and scene 2 contain 100 frames of images.

[0114] To simulate measurement data under low-light conditions, additive white Gaussian noise with signal-to-noise ratios of 5dB and 0dB was added to the image sequences, respectively. The simulation used multi-frame averaging denoising, single-frame total variation regularization denoising, and spatiotemporal total variation regularization-based image sequence denoising (hereinafter referred to as spatiotemporal total variation regularization denoising) to process the noisy image sequences (i.e., the aforementioned acquired image sequences). All methods used P = {(1,0,0),(0,1,0),(0,0,1)}, which considers differential transformations in the width, height, and time directions with a step size of 1 pixel.

[0115] Figure 3 This diagram illustrates the denoising result of image sequence 1 at a signal-to-noise ratio of 5dB, according to an embodiment of the present disclosure. Figure 4 This diagram illustrates the denoising result of image sequence 2 at a signal-to-noise ratio of 5dB, according to an embodiment of the present disclosure. Figure 5 This diagram illustrates the denoising result of image sequence 1 at a signal-to-noise ratio of 0dB, according to an embodiment of the present disclosure. Figure 6 This diagram illustrates the denoising result of image sequence 2 at a signal-to-noise ratio of 0 dB according to an embodiment of this disclosure. Figures 3 to 6 The denoising results shown indicate that at high noise levels, most information in a single frame of the image sequence is submerged in noise. Simply relying on the spatial differences between signal and noise is insufficient to achieve effective decoupling, thus the single-frame total variation regularization denoising method fails to achieve the desired denoising effect. In contrast, multi-frame averaging effectively suppresses random noise, but blurs moving parts of the image. The spatiotemporal total variation regularization-based image sequence denoising method, however, effectively decouples signal and noise while modeling motion between frames, avoiding motion blur and achieving optimal dynamic scene denoising.

[0116] Table 1 shows the quantitative evaluation of the denoising results of image sequence 1 and image sequence 2 at a signal-to-noise ratio (SNR) of 5 dB. Table 2 shows the quantitative evaluation of the denoising results of image sequence 1 and image sequence 2 at a SNR of 0 dB. The denoising results are for three methods: multi-frame averaging denoising, single-frame total variation regularization denoising, and spatiotemporal total variation regularization denoising. The metrics include root-mean-square error (RMSE), peak signal-to-noise ratio (PSNR), and structural similarity (SSIM). The results show that, under both noise levels and the three evaluation metrics, the spatiotemporal total variation regularization-based image sequence denoising method achieves the best denoising performance.

[0117] Table 1. Quantitative evaluation of the denoising performance of image sequence 1 and image sequence 2 at a signal-to-noise ratio of 5 dB.

[0118]

[0119] Table 2. Quantitative evaluation of the denoising effect of image sequence 1 and image sequence 2 at a signal-to-noise ratio of 0 dB.

[0120]

[0121] To further verify the effectiveness of the image sequence denoising method based on spatiotemporal domain total variation regularization, tests were conducted using actual low-light image sequences.

[0122] Figure 7 This diagram illustrates the denoising results of a noisy image sequence in a low-light clock scene according to an embodiment of the present disclosure. Figure 8 This diagram illustrates the denoising results of a noisy image sequence in a low-light scene at a school gate, according to an embodiment of the present disclosure. Figure 7 and Figure 8 The denoising results shown are based on three denoising methods: multi-frame averaging denoising, single-frame total variation regularization denoising, and spatiotemporal total variation regularization denoising. The low-light clock scene and the low-light school gate scene each contain 100 noisy images. Figure 7 In the illustrated case, due to the high noise level, the Arabic numerals on the clock face are almost impossible to distinguish clearly from a single frame image, making it difficult for single-frame-based image denoising methods to accurately restore scene details. Furthermore, the clock's second hand experienced significant displacement during filming, resulting in motion blur when using multi-frame averaging, hindering the accurate reconstruction of moving objects in the scene. Employing a spatiotemporal total variational regularization method can achieve both clear reconstruction of the clock face numerals and analysis of the second hand's motion. Figure 8 In the example shown, the school gate model underwent significant rotation during the acquisition process. Similar to the previous example, the image sequence denoising method based on spatiotemporal domain total variation regularization can achieve the highest reconstruction quality and restore the most detailed information while avoiding motion blur.

[0123] Compared to existing image sequence denoising methods, the image sequence denoising algorithm based on spatiotemporal domain total variational regularization provided in this disclosure has the following advantages: Scalability: It is flexibly applicable to data of any dimension, and has no special requirements on the size (width, height) of the input image or the number of frames in the image sequence, provided that computational resources allow; Generalization: It is applicable to noise of any level, and by adjusting the regularization coefficient, it can suppress noise of different levels, achieving a balance between denoising amplitude and scene fidelity; Robustness: It has a relatively robust suppression effect on all possible noise distributions, and this robustness has been experimentally verified; High efficiency: The main calculations involved include element-wise operations on arrays and linear convolution and its transpose operations. These operations are highly conducive to parallel processing and can be combined with GPU programming to achieve real-time image denoising, significantly improving processing speed.

[0124] This disclosure also provides an image sequence denoising device based on spatiotemporal domain total variational regularization. Figure 9 A block diagram of an image sequence denoising apparatus provided according to an embodiment of the present disclosure is shown. Figure 9 As shown, the device 900 may include:

[0125] An acquisition and analysis module 901 is configured to acquire a sequence of acquired images and analyze the sequence of acquired images to obtain prior information. The sequence of acquired images is used to represent a sequence of acquired images containing noise.

[0126] The function construction module 902 is configured to construct an objective function about a target image sequence based on the acquired image sequence and the prior information. The target image sequence represents an image sequence after noise removal. The objective function includes a data fidelity term and a regularization term. The data fidelity term indicates the maximum likelihood estimation function based on additive white Gaussian noise, and the regularization term indicates the spatiotemporal total variation regularization function.

[0127] The function processing module 903 is configured to perform dualization processing on the objective function to obtain a dual function with respect to the dual variable, and to perform projection gradient update on the dual function with the goal of maximizing the dual function until a preset condition is met.

[0128] Sequence generation module 904 is configured to generate the target image sequence based on the dual variables when the preset conditions are met.

[0129] In one possible implementation, the prior information includes at least a regularization coefficient, which indicates the weight relationship between the data fidelity term and the regularization term, and the regularization coefficient is determined based on the noise level.

[0130] In one possible implementation, the prior information includes at least weighting factors, which include a width-direction weighting factor, a height-direction weighting factor, and a time-direction weighting factor, wherein the weighting factors are determined based on the spatial distribution and motion characteristics of the scene in which the acquired image sequence is located.

[0131] In one possible implementation, the prior information includes at least a set of sparse transform translations, which includes at least a set of sparse transform operators. Each set of sparse transform operators includes a first translation, a second translation, and a third translation. The first translation indicates the number of pixels that the target image sequence to be determined is translated along the width direction, the second translation indicates the number of pixels that the target image sequence to be determined is translated along the height direction, and the third translation indicates the number of pixels that the target image sequence to be determined is translated along the time direction.

[0132] In one possible implementation, constructing the objective function for the target image sequence based on the acquired image sequence and the prior information includes: constructing the data fidelity term based on the acquired image sequence and the target image sequence to be determined; and constructing the regularization term based on the prior information, the target image sequence to be determined, and a linear transformation; wherein the width, height, and number of frames of the images in the target image sequence to be determined are all in the real number domain, and the linear transformation is used to indicate shifting the target image sequence to be determined by a first shift along the width direction, a second shift along the height direction, and a third shift along the time direction.

[0133] In one possible implementation, the dualization of the objective function to obtain a dual function with respect to the dual variable includes: determining multiple difference operators based on the identity matrix and the linear transformation, the number of difference operators being determined based on the number of sparse transformation operators in the prior information; concatenating all the multiple difference operators to obtain a difference matrix; and performing dualization on the objective function based on the difference matrix to obtain the dual function, the dual function being used to indicate the relationship between the acquired image sequence, the difference matrix, and the dual variable.

[0134] In one possible implementation, updating the dual function by projecting its gradient with the objective of maximizing the dual function until a preset condition is met includes: initializing the dual variable and the auxiliary variable, wherein the auxiliary variable is used to determine the direction of the projected gradient; updating the dual variable along the direction of the projected gradient with the objective of maximizing the dual function until the preset condition is met, wherein the projected gradient represents the projection of the auxiliary variable about a first set, the first set being the set of all dual variables that satisfy the condition that the maximum value of the elements in the dual variable is less than or equal to the regularization coefficient in the prior information, and the projected gradient is determined based on the auxiliary variable and the regularization coefficient in the prior information.

[0135] In one possible implementation, generating the target image sequence based on the dual variables that satisfy the preset conditions includes: converting the dual variables that satisfy the preset conditions into the target image sequence according to a relation; wherein the relation is used to indicate the relationship between the target dual variables, the acquired image sequence, the difference matrix, and the target image sequence.

[0136] In one possible implementation, the preset condition is that the number of times the projection gradient is updated reaches a preset threshold, or the preset condition is that the projection gradient is updated to meet a preset convergence condition.

[0137] In some embodiments, the functions or modules of the image sequence denoising apparatus provided in this disclosure can be used to perform the methods described in the above method embodiments. The specific implementation can be referred to the description of the image sequence denoising method embodiments above, and for the sake of brevity, it will not be repeated here.

[0138] This disclosure also proposes an image sequence denoising device based on spatiotemporal domain total variational regularization, comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to implement the above-described image sequence denoising method when executing the instructions stored in the memory.

[0139] In some embodiments, the functions or modules of the image sequence denoising apparatus provided in this disclosure can be used to perform the methods described in the above method embodiments. The specific implementation can be referred to the description of the image sequence denoising method embodiments above, and for the sake of brevity, it will not be repeated here.

[0140] This disclosure also proposes a computer-readable storage medium storing computer program instructions, which, when executed by a processor, implement the aforementioned image sequence denoising method. The computer-readable storage medium can be volatile or non-volatile.

[0141] In some embodiments, the functions or modules of the computer-readable storage medium provided in this disclosure can be used to perform the methods described in the above method embodiments. The specific implementation can be referred to the description of the above image sequence denoising method embodiments, which will not be repeated here for the sake of brevity.

[0142] This disclosure also provides a computer program product, including computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code, wherein when the computer-readable code is run in a processor of an electronic device, the processor in the electronic device performs the above-described image sequence denoising method.

[0143] In some embodiments, the functions or modules of the computer-readable storage medium provided in this disclosure can be used to perform the methods described in the above method embodiments. The specific implementation can be referred to the description of the above image sequence denoising method embodiments, which will not be repeated here for the sake of brevity.

[0144] Figure 10 This is a block diagram illustrating an apparatus 1900 for performing an image sequence denoising method according to an exemplary embodiment. For example, apparatus 1900 may be provided as a server or terminal device. (Refer to...) Figure 10 The apparatus 1900 includes a processing component 1922, which further includes one or more processors, and memory resources represented by memory 1932 for storing instructions, such as application programs, that can be executed by the processing component 1922. The application programs stored in memory 1932 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processing component 1922 is configured to execute instructions to perform the methods described above.

[0145] Device 1900 may also include a power supply component 1926 configured to perform power management of device 1900, a wired or wireless network interface 1950 configured to connect device 1900 to a network, and an input / output interface 1958 (I / O interface). Device 1900 can operate on an operating system, such as Windows Server, stored in memory 1932. TM macOS X TM Unix TM Linux TM FreeBSD TM Or similar.

[0146] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions that can be executed by a processing component 1922 of the device 1900 to perform the above-described method.

[0147] This disclosure can be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of this disclosure.

[0148] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example—but not limited to—electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination thereof. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0149] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0150] Computer program instructions used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing the status information of the computer-readable program instructions to implement various aspects of this disclosure.

[0151] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0152] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0153] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0154] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0155] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or technical improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. An image sequence denoising method based on spatiotemporal domain total variation regularization, characterized in that, include: Acquire a sequence of images and analyze the sequence to obtain prior information. The sequence of images is used to represent a sequence of images containing noise. Based on the acquired image sequence and the prior information, an objective function is constructed for the target image sequence, wherein the target image sequence represents the image sequence after noise removal, and the objective function includes a data fidelity term and a regularization term. The data fidelity term indicates the maximum likelihood estimation function based on additive white Gaussian noise, and the regularization term indicates the spatiotemporal total variation regularization function. The objective function is dualized to obtain a dual function with respect to the dual variable. The dual function is then updated by projecting gradients with the goal of maximizing the dual function until a preset condition is met. The target image sequence is generated based on the dual variables that satisfy the preset conditions. The prior information includes at least a set of sparse transform translations, which includes at least a set of sparse transform operators. Each set of sparse transform operators includes a first translation, a second translation, and a third translation. The first translation indicates the number of pixels that the target image sequence to be determined is translated along the width direction, the second translation indicates the number of pixels that the target image sequence to be determined is translated along the height direction, and the third translation indicates the number of pixels that the target image sequence to be determined is translated along the time direction.

2. The method according to claim 1, characterized in that, The prior information includes at least a regularization coefficient, which is used to indicate the weight relationship between the data fidelity term and the regularization term, and the regularization coefficient is determined based on the noise level.

3. The method according to claim 1, characterized in that, The prior information includes at least a weighting factor, which includes a width-direction weighting factor, a height-direction weighting factor, and a time-direction weighting factor. The weighting factors are determined based on the spatial distribution and motion characteristics of the scene in which the acquired image sequence is located.

4. The method according to claim 1, characterized in that, The construction of the objective function for the target image sequence based on the acquired image sequence and the prior information includes: The data fidelity term is constructed based on the acquired image sequence and the target image sequence to be determined, and the regularization term is constructed based on the prior information, the target image sequence to be determined, and the linear transformation. In the target image sequence to be determined, the width, height, and number of frames of the images are all in the real number domain. The linear transformation is used to indicate that the target image sequence to be determined is translated by the first translation amount along the width direction, the second translation amount along the height direction, and the third translation amount along the time direction.

5. The method according to claim 4, characterized in that, The process of dualizing the objective function to obtain a dual function with respect to the dual variable includes: Multiple difference operators are determined based on the identity matrix and the linear transformation, and the number of difference operators is determined according to the number of sparse transformation operators in the prior information; Connect all the multiple difference operators to obtain the difference matrix; The objective function is dualized based on the difference matrix to obtain the dual function, which is used to indicate the relationship between the acquired image sequence, the difference matrix, and the dual variables.

6. The method according to claim 1, characterized in that, The step of updating the projection gradient of the dual function with the objective of maximizing the dual function until a preset condition is met includes: Initialize the dual variables and auxiliary variables, wherein the auxiliary variables are used to determine the direction of the projection gradient; With the goal of maximizing the dual function, the dual variable is updated along the direction of the projection gradient until the preset condition is met. The projection gradient represents the projection of the auxiliary variable onto a first set, which is the set of all dual variables that satisfy the condition that the maximum value of the elements in the dual variable is less than or equal to the regularization coefficient in the prior information. The projection gradient is determined based on the auxiliary variable and the regularization coefficient in the prior information.

7. The method according to any one of claims 1 to 6, characterized in that, The step of generating the target image sequence based on the dual variables that satisfy the preset conditions includes: Based on the relation, the dual variables that satisfy the preset conditions are converted into the target image sequence; The relational expression is used to indicate the relationship between the target dual variable, the acquired image sequence, the difference matrix, and the target image sequence.

8. The method according to any one of claims 1 to 6, characterized in that, The preset condition is that the number of times the projection gradient is updated reaches a preset threshold, or the preset condition is that the projection gradient is updated to meet a preset convergence condition.

9. An image sequence denoising device based on spatiotemporal domain total variation regularization, characterized in that, include: An acquisition and analysis module is configured to acquire a sequence of images and analyze the sequence of images to obtain prior information, wherein the sequence of images is used to represent a sequence of images containing noise. A function construction module is configured to construct an objective function about a target image sequence based on the acquired image sequence and the prior information. The target image sequence represents an image sequence after noise removal. The objective function includes a data fidelity term and a regularization term. The data fidelity term indicates the maximum likelihood estimation function based on additive white Gaussian noise, and the regularization term indicates the spatiotemporal total variation regularization function. The function processing module is configured to perform dualization on the objective function to obtain a dual function with respect to the dual variable, and to update the dual function by projecting gradient with the goal of maximizing the dual function until a preset condition is met. A sequence generation module, configured to generate the target image sequence based on dual variables that satisfy the preset conditions; The prior information includes at least a set of sparse transform translations, which includes at least a set of sparse transform operators. Each set of sparse transform operators includes a first translation, a second translation, and a third translation. The first translation indicates the number of pixels that the target image sequence to be determined is translated along the width direction, the second translation indicates the number of pixels that the target image sequence to be determined is translated along the height direction, and the third translation indicates the number of pixels that the target image sequence to be determined is translated along the time direction.

10. The apparatus according to claim 9, characterized in that, The prior information includes at least a regularization coefficient, which is used to indicate the weight relationship between the data fidelity term and the regularization term, and the regularization coefficient is determined based on the noise level.

11. The apparatus according to claim 9, characterized in that, The prior information includes at least a weighting factor, which includes a width-direction weighting factor, a height-direction weighting factor, and a time-direction weighting factor. The weighting factors are determined based on the spatial distribution and motion characteristics of the scene in which the acquired image sequence is located.

12. The apparatus according to claim 9, characterized in that, The construction of the objective function for the target image sequence based on the acquired image sequence and the prior information includes: The data fidelity term is constructed based on the acquired image sequence and the target image sequence to be determined, and the regularization term is constructed based on the prior information, the target image sequence to be determined, and the linear transformation. In the target image sequence to be determined, the width, height, and number of frames of the images are all in the real number domain. The linear transformation is used to indicate that the target image sequence to be determined is translated by the first translation amount along the width direction, the second translation amount along the height direction, and the third translation amount along the time direction.

13. The apparatus according to claim 12, characterized in that, The process of dualizing the objective function to obtain a dual function with respect to the dual variable includes: Multiple difference operators are determined based on the identity matrix and the linear transformation, and the number of difference operators is determined according to the number of sparse transformation operators in the prior information; Connect all the multiple difference operators to obtain the difference matrix; The objective function is dualized based on the difference matrix to obtain the dual function, which is used to indicate the relationship between the acquired image sequence, the difference matrix, and the dual variables.

14. The apparatus according to claim 9, characterized in that, The step of updating the projection gradient of the dual function with the objective of maximizing the dual function until a preset condition is met includes: Initialize the dual variables and auxiliary variables, wherein the auxiliary variables are used to determine the direction of the projection gradient; With the goal of maximizing the dual function, the dual variable is updated along the direction of the projection gradient until the preset condition is met. The projection gradient represents the projection of the auxiliary variable onto a first set, which is the set of all dual variables that satisfy the condition that the maximum value of the elements in the dual variable is less than or equal to the regularization coefficient in the prior information. The projection gradient is determined based on the auxiliary variable and the regularization coefficient in the prior information.

15. The apparatus according to any one of claims 9 to 14, characterized in that, The step of generating the target image sequence based on the dual variables that satisfy the preset conditions includes: Based on the relation, the dual variables that satisfy the preset conditions are converted into the target image sequence; The relational expression is used to indicate the relationship between the target dual variable, the acquired image sequence, the difference matrix, and the target image sequence.

16. The apparatus according to any one of claims 9 to 14, characterized in that, The preset condition is that the number of times the projection gradient is updated reaches a preset threshold, or the preset condition is that the projection gradient is updated to meet a preset convergence condition.

17. An image sequence denoising device based on spatiotemporal domain total variation regularization, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to implement the image sequence denoising method according to any one of claims 1 to 8 when executing instructions stored in the memory.

18. A non-volatile computer-readable storage medium storing computer program instructions thereon, characterized in that, When the computer program instructions are executed by the processor, they implement the image sequence denoising method according to any one of claims 1 to 8.

Images