An image processing apparatus and method

Abstract

An image processor forming an output image sequence from an input image sequence, characterized in that the image processor is configured to implement a trained artificial intelligence model, the model being trained to satisfy constraints on a recursive processing of the input image sequence, the constraints being selected to have to stabilize the recursive processing. Thus, the proposed approach can enable an efficient computation of the matrix product required to globally perform the recursive processing across images in a stable manner.

Description

TECHNICAL FIELD

[0001] The present invention relates to image processing, and in particular to recursively processing image sequences while preserving global stability. BACKGROUND

[0002] Low-level computer vision problems such as denoising, demosaicking, or super-resolution are usually processed on a single frame basis. Given a noisy image, a mosaicked image, or a low-resolution image as input, a convolutional neural network (CNN) trained in a supervised manner processes the image to generate a corresponding denoised, demosaicked, or high-resolution image as output. Multi-frame approaches for burst processing are gaining increasing interest, which can produce significantly better results by exploiting the temporal consistency inherent in sequential images.

[0003] There are two main approaches to burst processing. All frames in a burst can be processed simultaneously, or they can be processed one by one in a recursive fashion. In a recursive fashion, the processing of a frame is influenced by the processing of previous frames. This approach, called recursive processing, is particularly promising for three reasons.

[0004] The first reason is that it provides good performance at a reasonable computational cost. The second reason is that a single-frame architecture can be easily converted to a multi-frame architecture by simply adding recursive connections. The third reason is that a recursive model trained on bursts of images of length around 10 to 20 frames can be directly used to process video sequences of length hundreds, thousands, or more frames.

[0005] However, care needs to be taken to ensure their stability.

[0006] By construction, recursive architectures contain feedback loops: the processing of an input at time t is influenced by the processing of an input at time t-1. When the number of frames is large, recursive methods are more prone to instability through positive feedback. Small inconsistencies or errors in the processing process will increase by themselves at each stage of the processing until they become large errors or inconsistencies, thus damaging the parts of the image that are being correctly processed. In this way, a large output leads to an even larger output, eventually developing to infinity. In practice, it is observed that some recursive architectures become unstable after a few tens or hundreds of frames. This can be a key limitation of recursive architectures when applied to video data.

[0007] Figure 1(a) shows two examples of this phenomenon, plotting the performance of two recurrent denoising models in terms of Peak Signal-to-Noise Ratio (PSNR) to measure quality as a function of frame number for real video sequences. A sharp drop is observed around frames 20 and 300, corresponding to the onset of instability. The quality, initially good, is above 28 by PSNR measurement, but due to the instability of the recursions, the PSNR of the first model (dashed line 102) drops around frame 20 and the second model (solid line 104) drops around frame 300.

[0008] These instabilities lead to strong artefacts that eventually cover the whole output image. The effect in the denoised output image for a pair of model images 106 and 108 is shown in Figure 1(b), where the denoising process has occurred from left to right. Irregular white and black patches 110 are observed in the middle of the image for the first model (top image 106). A black region 112 is observed around the horizontal stripes on the right of the image for the second model (bottom image 108). These artefacts tend to grow locally from frame to frame until they cover the whole image.

[0009] There are two main recurrent architectures in the literature, as shown in Figure 2 .

[0010] The first architecture is frame recurrence (as described in Sajjadi et al., “Frame- wise video super-resolution”, CVPR, 2018), which includes using the previous output y t-1 to process the current input x t . The previous output frame is fed back as input to help process the current frame. This frame recurrence is represented by the dashed arrow 202 at the bottom of the figure.

[0011] The second architecture is feature recurrence (as described in Godard et al., “Deep bursty denoising”, ECCV, 2018), which includes using the previous features h 1 t-1 ,..., h l t-1 to process the current input x t . In feature recurrence of layers or blocks in the network, the previous output features are fed back as input to the layer or block to help process the current frame. These feature recurrence connections are shown using the three loops with dashed arrows 204a-c at the top of the figure.

[0012] For short bursts of 10 to 20 frames, feature recursion often outperforms frame-only recursion in video denoising. Similar observations on video super-resolution can be made in the following literature: "Efficient Video Super-Resolution Based on Recursive Latent Space Propagation" by Fuoli et al. (2019), arXiv. However, Godard et al. stated that their feature recursion architecture "is unstable on longer sequences." This instability is briefly mentioned, but no explanation or solution is provided.

[0013] There is a need to develop a method that can perform stable global multi-frame recursive processing on video sequences of arbitrary length. Summary of the Invention

[0014] According to one aspect, an image processor is provided that forms an output image sequence from an input image sequence. The image processor is used to implement a trained artificial intelligence model, the model being trained to satisfy constraints on recursive processing of the input image sequence, wherein the selected constraints must stably perform the recursive processing. The method enables stable recursive processing of the image sequence.

[0015] The recursive processing can be performed using an equivalent representation of the recursive matrix in the matrix space of convolutions between partially connected layers. By performing operations in the convolution space, the required computations can be performed efficiently.

[0016] The recursive processing of the input image sequence can occur on the inner convolutions of the trained artificial intelligence model. This can improve image quality.

[0017] The constraint may include a condition that limits the maximum singular value of each recursive inner convolution to less than 1. By imposing this constraint, the recursive processing can be ensured to be stable.

[0018] The equivalent of the recursive matrix representation of the convolution in convolution space and in matrix space can be a kernel tensor K. The kernel tensor K can be used to allow computations to be performed more efficiently and accurately in convolution space than in matrix space.

[0019] The constraints can be enforced at each step of the training of the AI ​​model by dividing the kernel tensor K of each recursive inner convolution by its corresponding maximum singular value. By performing this calculation, the constraints required for stable recursive processing of the image sequence can be applied efficiently at each step of training.

[0020] The maximum singular value can be calculated using a power-law iteration method. Using this method provides an efficient way to determine the value required to impose the stability constraint.

[0021] The power iteration method may include performing equivalent computations of K*u and K in the convolution space. T *v determines the matrix products v = Wu and u = W in the matrix space. T The multiple iterations of v, where u and v represent the random initial vector and the result vector, respectively, and W and W T Let K and K represent the biblock cyclic matrix representations of the convolution and its transpose, respectively. T These represent the operations of the four-dimensional kernel tensor and the four-dimensional tensor, respectively. By performing the equivalent computation in the convolution space, the constraints required for the recursive processing of the input image sequence can be imposed using a smaller and more efficient representation of the matrix W. This further improves process efficiency, as fewer computations are required to successfully process the image stably.

[0022] The four-dimensional tensor K T The operation may include transpose, invert, and shift K, such that: K T i,j,k,l =K mod(n-i,n),mod(n-j,n),l,k Where i, j, k, and l represent the tensor K T The dimensions i and j are in the range of 0 to n–1, where n is the height and width of each input image in the input image sequence, and k and l are in the range of 0 to m–1, where m is the number of input and output channels. The computed kernel tensor K successfully allows equivalent computations in matrix space to be performed in convolution space, which in turn minimizes the size of these computations and allows for global and efficient stability of the recursive processing on the image.

[0023] The trained AI model can be trained using sequences comprising multiple frames propagated backward through time. By using this training method, errors in the network's output images can be minimized.

[0024] The training may include an unfolded time with shared weights, and the backpropagation may be performed simultaneously on the multiple frames fed with corresponding inputs. By arranging the time-based backpropagation training in this way, the training can be performed more efficiently, thus requiring less time from start to finish.

[0025] According to another aspect, a method implemented at an image processor is provided for forming an output image sequence based on an input image sequence. The method includes implementing a trained artificial intelligence model to satisfy constraints on recursive processing of the input image sequence, wherein the selected constraints require stable recursive processing. The method enables stable recursive processing of the image sequence.

[0026] According to another aspect, a training method is provided for training an artificial intelligence model, the model being configured to stably perform recursive processing of an input image sequence to form an output image sequence. The training method includes training the model to satisfy constraints on the recursive processing of the input sequence, the selected constraints requiring stable recursive processing. The method is capable of stably performing recursive processing on image sequences.

[0027] The constraint can be that the maximum singular value of each inner convolution of the recursive process is less than 1, and can be enforced at each step of the training of the model by dividing the kernel tensor K of each inner convolution of the artificial intelligence model by its maximum singular value. By imposing this constraint, it can be ensured that the recursive process is stable and that the constraint is applied in an efficient manner at each step of training.

[0028] The maximum singular value can be computed using a power-law iteration method, which includes performing equivalent computations of K*u and K in the convolution space. T *v determines the matrix products v = Wu and u = W in the matrix space. T Multiple iterations are performed. Calculating the maximum singular value using the power iteration method provides an efficient way to determine the value required to impose the stability constraint. By performing the equivalent computation in convolutional space, a smaller and more efficient representation of the matrix W in convolutional space can be used to impose the constraints required for the recursive processing of the input image sequence. This further improves process efficiency, as fewer computations are required to successfully process the image stably.

[0029] The four-dimensional tensor K T The operation may include transpose, invert, and shift K, such that: K T i,j,k,l =K mod(n-i,n),mod(n-j,n),l,k Where i, j, k, and l represent the tensor K T The dimensions i and j are in the range of 0 to n–1, where n is the height and width of each input image in the input image sequence, and k and l are in the range of 0 to m–1, where m is the number of input and output channels. The computed kernel tensor K successfully allows equivalent computations in matrix space to be performed in convolution space, which in turn minimizes the size of these computations and allows for global and efficient stability of the recursive processing on the image. Attached Figure Description

[0030] The invention will now be described by way of example with reference to the accompanying drawings. In the drawings:

[0031] Figure 1(a) illustrates an example of peak signal-to-noise ratio instability during image processing;

[0032] Figure 1(b) illustrates the effect of instability in the output image after denoising a pair of images;

[0033] Figure 2 A schematic diagram illustrating the concepts of frame recursion and feature recursion is shown;

[0034] Figure 3 A schematic diagram of the three components of this method for maintaining stability in recursive image processing is shown;

[0035] Figure 4 The basic principle of the effect of operations on the first two dimensions of the four-dimensional kernel tensor K is shown;

[0036] Figure 5 shows examples of (a) a clean image, (b) a noisy image, and (c) a denoised image using this method;

[0037] Figure 6 A schematic diagram of an example video denoising architecture combining frame recursion and feature recursion is shown;

[0038] Figure 7 This diagram illustrates training a CNN using backpropagation.

[0039] Figure 8(a) shows the PSNR as a function of the number of frames on a long Vimeo sequence for each of the unconstrained model, the previously known model, and the currently stated model;

[0040] Figure 8(b) shows a series of test frames, all of which contain Gaussian noise except for frame 100, which is blank due to an applied impulse noise event.

[0041] Figure 8(c) shows a graph of PSNR as a function of the number of frames in the impulse response sequence for each of the unconstrained model, the previously known model, and the currently stated model;

[0042] Figure 9 An example of a camera used to implement the image processor and the method described herein is shown. Detailed Implementation

[0043] The method described above facilitates stable recursive processing of video sequences (i.e., stable performance for image sequences of arbitrary length).

[0044] The method is designed with three steps, as follows: Figure 3 As shown. The steps include: defining stability conditions (301), identifying the two-dimensional matrix representation of convolution (302), and applying global spectral normalization (303).

[0045] One contribution of this method is linking the instability problem to the stability of simpler non-convolutional recursive models. The following literature illustrates the relationship between the stability and convergence of recursive functions: Miller and Hardt, “Stable Recursive Models,” ICLR, 2019. For a linear layer with a ReLU activation function, Ф can be written as:

[0046] h t =Ф(x t ,h t-1 =ReLU(Ux) t +Wh t-1 )

[0047] Where Ux t It depends on the input x t Feedforward term, Wh t-1 It depends on the previous output h t-1 The recursive terms. Then Miller and Hardt proved that Ф is stable in the following case:

[0048] ||W||<1

[0049] In other words, Ф is stable as long as the maximum singular value of the recursive matrix is ​​less than 1. That is, the maximum singular value within the matrix is ​​less than 1.

[0050] For low-level computer vision CNNs, this stability condition presents two challenges: how should it be applied to convolutions, and how can the condition be enforced during training?

[0051] Convolution is a linear mapping that can be represented by a two-dimensional matrix W. However, W is usually inaccessible and is only implicitly known through a four-dimensional kernel tensor K.

[0052] The constraint ||W||<1 maintains the stability of the linear layer, but it is important to minimize the loss function subject to this constraint.

[0053] To enforce this stability constraint, a spectral normalization method described in the following literature is employed: Miyato et al., “Spectral Normalization of Generative Adversarial Networks,” ICLR, 2018. In spectral normalization, the convolutional layers are normalized at each training step using the maximum singular value of a two-dimensional integer of K, computed via power iteration. The problem with this approach is that it only applies a “local constraint” to K, not W, thus disallowing the preservation of global stability.

[0054] For the two-dimensional matrix representation of convolution, a biblock cyclic representation as described in the following literature is employed: Sedghi et al., “Singular Values ​​of Convolutional Layers”, ICLR, 2019. Singular value pruning is performed on the matrix W, defined as a biblock cyclic representation of K. The problem with this approach is the high computational cost of full SVD for W and the lack of guaranteed convergence during training.

[0055] Combining these two elements is important. This paper describes an algorithmic design called Global Spectral Normalization (GSN). In this approach, the exponential iteration is derived in the convolution space, eliminating the need to explicitly compute the doubly circular matrix representation of the convolution.

[0056] Matrix representation is a method of defining transformations applicable to a set of numbers in memory. Typically, this set of numbers will be in the form of a vector describing input conditions or a set of values, and the matrix representation defines the operations to be performed on that input data to obtain the desired meaningful output. Matrix representation is particularly useful for storing matrices with more than one dimension in memory.

[0057] Convolution is a mathematical operation that applies to two functions to produce a third function that describes how the shape of one function is modified by the other. In other words, convolution can be used to represent the amount of overlap between two functions by mixing them together in a specific way to generate a third function derived from them. The term convolution can be used to describe either the output of this process or the process itself.

[0058] A key difference is that the GSN algorithm applies the recursive matrix W, which is defined as a bi-block cyclic representation of the convolution kernel K, rather than applying it to a two-dimensional integer form of K.

[0059] For linear layers, stability is guaranteed when the spectral norm of the recursive matrix W is less than 1. The currently described global spectral normalization enforces this constraint at each training step by dividing W by its maximum singular value. This maximum singular value is computed using a power-law iteration method. For a randomly initialized vector u, the power-law iteration method comprises multiple iterations of computing the matrix product:

[0060] v = W u

[0061] v = v / ||v||2

[0062] u = W T v

[0063] u = u / ||u||2

[0064] By definition, multiplying the left side by matrix W is equivalent to convolving the four-dimensional tensor K. However, multiplying by matrix W...T This is very important. The method described above shows that it is equivalent to passing through kernel K. T Perform convolution:

[0065] Table 1: Explains the equivalent matrix product in matrix space as a convolution in convolution space using tensor kernel K.

[0066]

[0067] Where K T Defined as:

[0068] K T i,j,k,l =K mod(n–i,n),mod(n–j,n),l,k

[0069] i and j are in the range of 0 to n–1, where n is the width and height of each element in the input image, while k and l are in the range of 0 to m–1, where m is the number of input and output channels. It is important to note that this operation corresponds to the transpose in matrix space, but it is not itself the transpose in convolution space, and K… T The superscript T in the code is used for a slight misuse of the notation. The effect of T is to transpose the dimensions k and l of K, and to invert and shift the dimensions i and j. An illustrative example of this operation on K is as follows: Figure 4 As shown.

[0070] Figure 4 This illustrates the basic principle of the effect of the "transpose" operation on the first two dimensions of kernel K. It can be seen that a 4x4 kernel is equivalent to a 6x6 kernel padded with zeros. The transpose operation reverses dimensions i and j, and shifts the kernel one position to the right and one position down in the matrix.

[0071] In convolution space, the transpose operation of this operation or transformation can be used to efficiently compute the product of two matrices using the power iteration method. This guarantees global stability.

[0072] The following describes an exemplary implementation of the method in a video denoising scenario.

[0073] The example model was trained on the Vimeo-90K dataset, a large-scale, high-quality video dataset collected from vimeo.com, as described in the following paper: Xue et al., “Enhancing Videos with a Task-Oriented Flow.”, IJCV, 2019. Specifically, the Septuplet dataset was used, which consists of 91,701 bursts of 7 frames at a fixed resolution of 448x256 pixels. More precisely, the example was trained on 64x64 pixel data blocks randomly extracted from the Septuplet dataset. The model is fully convolutional, thus usable at inference time at any resolution. For simplicity, the example was trained on black and white images corrupted by first-order Gaussian noise (where std = 20 / 255).

[0074] A trained artificial intelligence model can be an algorithm or a computational structure, for example, depending on the weights that define its operations, which have been trained through a feedback process involving comparisons between the model's output and the desired output.

[0075] Figure 5 shows examples of (a) a clean image 502, (b) a noisy image 504, and (c) a denoised image 506 used for verification. Figure 5(a) shows a clean grayscale image 502, and Figure 5(b) shows a version 504 of that image with synthetically added noise. Figure 5(c) shows the denoising result 506 of the method currently described.

[0076] For this model, the example uses a CNN architecture consisting of 12 convolutions and 64 channels, each followed by a ReLU activation function, except that the last layer has only one channel and no ReLU function. This architecture is inspired by the following literature: Godard et al., “Deep Burst Denoising,” ECCV, 2018. The model of the method described above uses both frame recursion and feature recursion, such as... Figure 6 As shown.

[0077] Figure 6 An exemplary embodiment of a video denoising architecture is schematically illustrated. This method implements a recursive video denoising network that combines frame recursion and feature recursion.

[0078] The network ("Net") is trained on bursts of seven frames via temporal backpropagation. Specifically, the network is expanded seven times using shared weights and backpropagation is performed simultaneously on seven copies fed with the corresponding inputs. Figure 7 This illustration demonstrates the training method using backpropagation over time. The network uses seven copies with shared weights, which are then spread out over time. This allows it to be trained on bursts of seven frames.

[0079] Training a network can involve using frame bursts of any appropriate length (not just the seven-frame length in the example above), depending on the network's end use and tailoring the training accordingly. For example, when the network is known to be designed to process longer videos, longer frame bursts can be used to train it. That is, the number of frames within a burst used to train the network can be proportional to the length of the video the network will ultimately use to process.

[0080] Feature recursion is typically achieved through concatenation of feedforward input and recursive feature_prev, followed by convolutions in each block:

[0081] out=tf.concat(input,feat_prev)

[0082] out=tf.layers.conv2d(out,num_filters,kernel_size)

[0083] The problem with this formula is that it combines feedforward and recursive operations within a single convolution. These two operations can be separated by replacing the previous formula with the following equivalent:

[0084] out1=tf.layers.conv2d(input,num_filters,kernel_size)

[0085] out2=tf.layers.conv2d(feat_prev,num_filters,kernel_size)

[0086] out = out1 + out2.

[0087] Therefore, in this method, the stability of each block can be improved by performing gradient descent on a second convolution corresponding to a recursive operation, after Global Spectral Normalization (GSN), and defined as Algorithm 1, as described below.

[0088] Algorithm 1: Layer Normalization with ||·||2 Norm

[0089] Input: Number of iterations M, number of power method iterations N, initial values ​​K0, u0, learning rate η

[0090] Result: K M

[0091]

[0092] The algorithm (Algorithm 1) uses the PowerIt subroutine to implement exponential iteration in the convolution space, and is defined in Algorithm 2 as follows.

[0093] Algorithm 2

[0094] Modified power iteration for ||.||2-norm calculation

[0095] algorithm:

[0096] Input: Convolution kernel The initial vector u0 is considered constant, and the number of iterations N is...

[0097] Result: Norm vector Estimate

[0098]

[0099] After training, the model described above was compared with an unconstrained trained model and another model trained using local spectral normalization as defined by Miyato et al., using two sequences. The first sequence was a long Vimeo sequence of approximately 700 frames. The second sequence was an impulse response sequence specifically designed to trigger instabilities. The impulse response sequence was intended to simulate a sudden surge of errors in a video stream, a known phenomenon in the field of image enhancement. The second sequence consisted of uniform grayscale images perturbed by Gaussian noise (stdd = 20 / 255), except at frame 100, when the input and internal features were artificially amplified to a constant value of 1000.

[0100] Comparing the current method with the standard spectral normalization algorithm of Miyato et al., it was found that only the current method maintains stability on long test video sequences and challenging impulse response sequences specifically designed to test the stability of recursive video denoising models. The current method is also compared with singular value pruning described by Sedghi et al., but the latter leads to performance degradation on a small model and training diverges on a 10-layer network. As shown in Figure 8, in this implementation, the current method using the GSN algorithm maintains stability on both sequences.

[0101] Figure 8(a) shows PSNR as a function of the number of frames on a long Vimeo sequence for the three models described above. The unconstrained feature recursive model (dashed line 802) remains stable for approximately 300 frames until the PSNR drops sharply. The local spectral normalization method described by Miyato et al. (dotted line 804) also becomes unstable after only about 20 frames. The method currently described using the GSN method provides sustained stability (solid line 806) and is able to handle the entire length of the video sequence.

[0102] Figure 8(b) shows a series of frames, all of which contain Gaussian noise except for frame 100, which appears blank due to the replication of the applied impulse noise event.

[0103] Figure 8(c) shows the PSNR of each of the three models described above as a function of the number of frames in the impulse response sequence, intended to induce instability. It can be seen that only the method currently described using the GSN method (solid line 806) is able to recover stability after a strong impulse. The unconstrained model is shown by dashed line 802, the local spectral normalization method by dotted line 804, and the method currently described using the created global spectral normalization model by solid line 806.

[0104] Figure 9 An example camera is shown for implementing an image processor to process images captured by the image sensor 1102 in camera 1101. Such a camera 1101 typically has some onboard processing capabilities. This may be provided by processor 1104. The processor 1104 can also be used for basic device functions. The camera typically also includes memory 1103.

[0105] Transceiver 1105 is capable of communicating with other entities 1110 and 1111 via a network. These entities may be physically located away from camera 1101. The network may be a publicly accessible network, such as the Internet. Entities 1110 and 1111 may be cloud-based. In one example, entity 1110 is a computing entity, and entity 1111 is a command and control entity. These entities are all logical entities. In practice, each may be provided by one or more physical devices (e.g., servers and data storage), and the functionality of two or more entities may be provided by a single physical device. Each physical device implementing an entity includes a processor and memory. The device may also include a transceiver for sending data to and receiving data from transceiver 1105 of camera 1101. The memory stores in a non-transient manner code that can be executed by the processor to implement the respective entity as described herein.

[0106] The command and control entity 1111 can train an artificial intelligence model used in each module of the system. This is typically a computationally intensive task, even though the resulting model can be efficiently described; therefore, developing the algorithm in the cloud is likely to be efficient, where a large amount of energy and computing resources can be expected to be available. This can be expected to be more efficient than forming such a model on a typical camera.

[0107] In one implementation, once the deep learning algorithm is developed in the cloud, the command and control entity can automatically generate the corresponding model and transmit it to the relevant camera device. In this example, the system is implemented at camera 1101 by processor 1104.

[0108] In another possible implementation, images can be captured by camera sensor 1102, and the image data can be sent to the cloud by transceiver 1105 for processing in the system. The resulting target image can then be sent back to camera 1101, such as... Figure 9 As shown in 1112.

[0109] Therefore, the method can be deployed in various ways, such as in the cloud, on a device, or in dedicated hardware. As mentioned above, cloud facilities can perform training to develop new algorithms or improve existing ones. Depending on the computing power close to the data corpus, training can be performed near the source data or in the cloud, for example, using an inference engine. The system can also be implemented in a camera, dedicated hardware, or in the cloud.

[0110] The applicant hereby discloses individually each individual feature described herein, as well as any combination of two or more such features. With ordinary knowledge of those skilled in the art, such features or combinations can be implemented as a whole based on this specification, regardless of whether such features or combinations of features solve any of the problems disclosed herein; and without limiting the scope of the claims. This application demonstrates that aspects of the invention can be constituted by any such individual features or combinations of features. Various modifications that can be made within the scope of the invention, given the foregoing description, will be apparent to those skilled in the art.

Claims

1. An image processor (1104) forming an output sequence of images from an input sequence of images, characterized in that, The image processor is configured to implement a trained artificial intelligence model trained to satisfy constraints on recursive processing of the input image sequence, the constraints selected to stabilize the recursive processing; The constraints include a condition that limits a maximum singular value of each recursive internal convolution to be less than 1; the constraints are implemented by a global spectral normalization method, specifically including: enforcing the constraints at each step of the training of the artificial intelligence model by dividing a kernel tensor K of each recursive internal convolution by its corresponding maximum singular value; the maximum singular values are computed using a power iteration method; wherein the equivalent in convolution space of the recursive matrix representation of the convolution in matrix space is the kernel tensor K.

2. The image processor of claim 1, wherein, The recursive processing is performed using the equivalent in matrix space of the convolution between partially connected layers in convolution space of the recursive matrix representation of the convolution.

3. The image processor of claim 1 or 2, wherein, The recursive processing of the input image sequence occurs on internal convolutions of the trained artificial intelligence model.

4. The image processor of claim 1 or 2, wherein, The power iteration method includes performing equivalent computations of K*u and in the convolution space. *v determines the matrix products v=Wu and u= in the matrix space. Multiple iterations of v, where u and v represent the random initial vector and result vector, respectively, and W and Let K and K represent the biblock cyclic matrix representations of the convolution and its transpose, respectively. These represent the operations of the four-dimensional kernel tensor and the four-dimensional tensor of the convolution, respectively.

5. The image processor of claim 4, wherein, the four-dimensional tensor the operations include transposing, inverting, and shifting K, such that: = where i, j, k and l represent dimensions of the tensor i and j are in the range of 0 to n - 1, where n is the height and width of each input image of the input image sequence, and k and l are in the range of 0 to m - 1, where m is the number of input and output channels.

6. The image processor of claim 1 or 2, wherein, The trained artificial intelligence model is based on training using a sequence comprising multiple frames backpropagated through time.

7. The image processor of claim 6, wherein, The training includes unfolded time with shared weights and the backpropagation is performed simultaneously on the multiple frames fed with corresponding inputs.

8. A method implemented at an image processor (1104), characterized by, A method for forming an output image sequence from an input image sequence, the method comprising implementing a trained artificial intelligence model to satisfy constraints on recursive processing of the input image sequence, the constraints selected to stabilize the recursive processing; the constraints include a condition that limits a maximum singular value of each recursive internal convolution to be less than 1; the constraints are implemented by a global spectral normalization method, specifically including: enforcing the constraints at each step of the training of the artificial intelligence model by dividing a kernel tensor K of each recursive internal convolution by its corresponding maximum singular value; the maximum singular values are computed using a power iteration method; wherein the equivalent in convolution space of the recursive matrix representation of the convolution in matrix space is the kernel tensor K.

9. A training method for training an artificial intelligence model, the method comprising: receiving a plurality of training data sets; training the artificial intelligence model using the plurality of training data sets; and outputting a trained artificial intelligence model. The model is arranged to stabilize recursive processing of an input image sequence for forming an output image sequence, the training method comprising training the model to satisfy constraints on the recursive processing of the input image sequence, the constraints selected to stabilize the recursive processing; The constraints are a maximum singular value of each internal convolution of the recursive processing to be less than 1, the constraints are implemented by a global spectral normalization method, specifically including: enforcing the constraints at each step of the training of the model by dividing a kernel tensor K of each internal convolution of the artificial intelligence model by its maximum singular value; the maximum singular values are computed using a power iteration method; wherein the equivalent in convolution space of the recursive matrix representation of the convolution in matrix space is the kernel tensor K.

10. The training method of claim 9, wherein, The power iteration method includes determining the matrix product v = Wu and u = Wv in the matrix space by performing equivalent calculations K * u and v in the convolution space multiple iterations. of the matrix product v = Wu and u = Wv in the matrix space.

11. The training method of claim 10, wherein, Four-dimensional tensor Operations on the four-dimensional tensor T include transposition, inversion, and shifting K, such that: = where i, j, k and l represent dimensions of the tensor i and j are in the range of 0 to n - 1, where n is the height and width of each input image of the input image sequence, and k and l are in the range of 0 to m - 1, where m is the number of input and output channels.

Images