Multi-view convolutional neural network for video processing

Abstract

The invention relates to processing video data. Some aspects relate to segmenting input video data into two or more segments, each segment comprising a number T of frames, where each frame comprises a frame height, a frame width, and a frame channel dimension Cin. Encoding each segment into S encoded representations comprising a code height, a code width and a code channel dimension, where T and S are integers, where T > = Sgt; 1. Encoding each segment into S encoded representations may include concatenating all T frames of the segment into an input tensor along a frame channel dimension, and encoding the input tensor into S encoded representations using a convolutional neural network (CNN) encoder.

Description

[0001] Cross-references to related applications

[0002] This application claims the benefit of U.S. Patent Application No. 18 / 466,415, filed September 13, 2023, entitled “MULTI-VIEW CONVOLUTIONAL NEURALNETWORKS FOR VIDEO PROCESSING,” the entire contents of which are expressly incorporated herein by reference. Technical Field

[0003] This disclosure relates in general to video processing. For example, aspects of this disclosure relate to systems and techniques for performing video processing using one or more machine learning systems based on convolutional neural network (CNN) architectures. Background Technology

[0004] Many devices and systems allow a scene to be captured by generating images (or frames) and / or video data (including multiple frames). For example, a camera or a device that includes a camera can capture a sequence of frames of a scene (e.g., video of the scene). In some cases, the frame sequence can be processed to perform one or more functions, can be output for display, can be output for processing and / or consumption by other devices, and for other purposes.

[0005] Artificial neural networks attempt to replicate, using computer technology, the logical reasoning performed by the biological neural networks that make up the animal brain. Deep neural networks (such as convolutional neural networks) are widely used in many applications, such as object detection, object classification, object tracking, and big data analysis. For example, a convolutional neural network can extract high-level features (such as facial shape) from an input image and use these features to output, for example, the probability that the input image contains a specific object. Summary of the Invention

[0006] The following is a simplified summary of the invention relating to one or more aspects disclosed herein. Therefore, this summary should not be considered an exhaustive overview relating to all conceived aspects, nor should it be considered to identify key or decisive elements relating to all conceived aspects or to depict the scope associated with any particular aspect. Thus, the sole purpose of this summary is to present, in a concise form, certain concepts relating to one or more aspects involving the mechanisms disclosed herein, prior to the detailed description presented below.

[0007] Systems, methods, apparatuses, and computer-readable media for performing video processing using one or more convolutional neural networks are disclosed.

[0008] According to one example, a method for processing video data is provided. The method may include: segmenting input video data into two or more segments, each segment comprising a number of T frames, wherein each frame includes a frame height, a frame width, and a frame channel dimension C. in Each segment is encoded into S encoded representations, including code height, code width, and code channel dimension, where T and S are integers, and T ≥ S > 1. Encoding each segment into S encoded representations may include concatenating all T frames of the segment along the frame channel dimension into an input tensor; and using a convolutional neural network (CNN) encoder to encode the input tensor into S encoded representations.

[0009] In another example, an apparatus for processing video data is provided. The apparatus may include: at least one memory configured to store the video data; and at least one processor coupled to the at least one memory, the at least one processor configured to: segment the input video data into two or more segments, each segment comprising a number of frames T, wherein each frame includes a frame height, a frame width, and a frame channel dimension C. in The process involves encoding each segment into S encoded representations, each including a code height, a code width, and a code channel dimension, where T and S are integers, and T ≥ S > 1. Encoding each segment into S encoded representations may include: concatenating all T frames of the segment along the frame channel dimension into an input tensor; and using a convolutional neural network (CNN) encoder to encode the input tensor into S encoded representations.

[0010] In another example, an apparatus for processing video data is provided. The apparatus may include: a component for segmenting input video data into two or more segments, each segment comprising a number of frames T, wherein each frame includes a frame height, a frame width, and a frame channel dimension C. in The component used to encode each segment into S encoded representations, including code height, code width, and code channel dimension, where T and S are integers, and T ≥ S > 1. The component used to encode each segment into S encoded representations may include: a component for concatenating all T frames of the segment along the frame channel dimension into an input tensor; and a component for encoding the input tensor into S encoded representations using a convolutional neural network (CNN) encoder.

[0011] Systems, methods, apparatuses, and computer-readable media for training one or more convolutional neural networks (CNNs) are further disclosed.

[0012] Based on an illustrative example, a method is provided for training a convolutional neural network (CNN) to process input video data, which can be divided into two or more segments, each segment comprising a number of T frames, wherein each frame includes a frame height, a frame width, and a frame channel dimension C. in The CNN model may include: a CNN encoder for encoding each segment into S encoded representations, where T ≥ S > 1; and a CNN decoder for decoding each encoded representation of each segment into a decoded frame of output video data. The method may include: initializing the network parameters of the CNN model; and training the CNN model based on one of the following: supervised learning, unsupervised learning, or semi-supervised learning.

[0013] In another example, an apparatus is provided for training a convolutional neural network (CNN) to process input video data, which can be divided into two or more segments, each segment comprising a number of T frames, wherein each frame includes a frame height, a frame width, and a frame channel dimension C. in The CNN model may include: a CNN encoder for encoding each segment into S encoded representations, where T ≥ S > 1; and a CNN decoder for decoding each encoded representation of each segment into a decoded frame of output video data. The apparatus may include: at least one memory configured to store input video data; and at least one processor coupled to the at least one memory, configured to: initialize network parameters of the CNN model; and train the CNN model based on one of: supervised learning, unsupervised learning, or semi-supervised learning.

[0014] In another example, an apparatus is provided for training a convolutional neural network (CNN) to process input video data, which can be divided into two or more segments, each segment comprising a number of T frames, wherein each frame includes a frame height, a frame width, and a frame channel dimension C. in The CNN model may include: a CNN encoder for encoding each segment into S encoded representations, where T ≥ S > 1; and a CNN decoder for decoding each encoded representation of each segment into a decoded frame of output video data. The apparatus may include: components for initializing network parameters of the CNN model; and components for training the CNN model based on one of the following: supervised learning, unsupervised learning, or semi-supervised learning.

[0015] The features and technical advantages of the examples according to this disclosure have been summarized rather extensively above in order to better understand the detailed description below. Additional features and advantages will be described below. The disclosed concepts and specific examples can be readily used as the basis for modifying or designing other structures for achieving the same purpose as this disclosure. Such equivalent constructions do not depart from the scope of the appended claims. The characteristics of the concepts disclosed herein (both their organization and operation) and their associated advantages will be better understood from the following description when considered in conjunction with the accompanying drawings. Each figure in the drawings is provided for illustrative and descriptive purposes and not as a definition of limitation of the claims. The foregoing, as well as other features and aspects, will become more apparent upon reference to the following specification, claims, and appended drawings.

[0016] This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used in isolation to define the scope of the claimed subject matter. This subject matter should be understood with reference to the appropriate portions of the entire specification, any or all drawings, and each claim. Brief description of the attached diagram

[0018] The accompanying drawings are provided to aid in describing various aspects of this disclosure, and are provided for illustrative purposes only and not for limiting the scope of the aspects. For a more detailed understanding of the foregoing features of this disclosure, a more specific description of the invention, briefly summarized above, can be obtained by referring to the aspects, some of which are illustrated in the drawings. However, it should be noted that the drawings illustrate only certain typical aspects of this disclosure and are therefore not to be considered as limiting its scope, as other equally valid aspects are permissible in this description. The same reference numerals in different drawings may identify the same or similar elements.

[0019] Figure 1 An example of a recurrent neural network (RNN) for video processing is shown.

[0020] Figure 2 An example of a 3D convolutional neural network (CNN) for video processing is shown.

[0021] Figure 3 Multi-view CNNs are illustrated according to various aspects of this disclosure.

[0022] Figure 4 Examples are shown as follows Figure 2 The encoding of the frame set in the illustrated 3D CNN.

[0023] Figure 5 The encoding of a set of frames in a multi-view CNN according to various aspects of this disclosure is illustrated.

[0024] Figure 6 Examples of 2D CNNs according to various aspects of this disclosure are provided.

[0025] Figure 7 Examples of various aspects according to this disclosure are illustrated. Figure 2 D CNN.

[0026] Figure 8 Exemplary implementations based on various aspects of this disclosure are illustrated.

[0027] Figure 9 Another exemplary implementation according to various aspects of this disclosure is illustrated.

[0028] Figure 10a A graph illustrating the average measurement results of the processing according to various aspects of this disclosure is shown.

[0029] Figure 10b A graph illustrating the measurement results of static processing according to various aspects of this disclosure is shown.

[0030] Figure 10c A graph illustrating the measurement results of motion processing according to various aspects of this disclosure is provided.

[0031] Figure 11 A flowchart illustrating a method for processing video data according to various aspects of this disclosure is provided.

[0032] Figure 12 A flowchart illustrating a method for training a CNN according to various aspects of this disclosure is provided.

[0033] Figure 13 An exemplary computing system for implementing the neural network-based video processing techniques disclosed herein is illustrated. Detailed Implementation

[0034] Certain aspects of this disclosure are provided below for illustrative purposes. Alternative aspects may be devised without departing from the scope of this disclosure. Furthermore, well-known elements of this disclosure will not be described in detail or will be omitted so as not to obscure the relevant details of this disclosure. Some of the aspects described herein can be applied independently, and some of them can be combined, as will be apparent to those skilled in the art. In the following description, specific details are set forth for illustrative purposes to provide a thorough understanding of various aspects of this application. However, it will be apparent that various aspects can be practiced without these specific details. The accompanying drawings and descriptions are not intended to be limiting.

[0035] The following description provides only exemplary aspects and is not intended to limit the scope, applicability, or configuration of this disclosure. Rather, the following description of the exemplary aspects will provide those skilled in the art with a description that can be used to implement the exemplary aspects. It should be understood that various changes can be made to the function and arrangement of the elements without departing from the scope of this application as set forth in the appended claims.

[0036] The demand for and consumption of video data has increased significantly in both consumer and professional settings. As previously mentioned, devices and systems are typically equipped with the ability to capture and process video data. For example, cameras or computing devices that include cameras (e.g., mobile phones or smartphones that include one or more cameras) can capture video of scenes, people, objects, etc. This video can be captured, processed, and output (and / or stored) for consumption. The video can be further processed for effects such as compression, frame rate upconversion, sharpening, color space conversion, image enhancement, high dynamic range (HDR), noise reduction, low-light compensation, etc. The video can also be further processed for applications such as computer vision, extended reality (e.g., augmented reality, virtual reality, etc.), image recognition (e.g., facial recognition, object recognition, scene recognition, etc.), and autonomous driving. In some examples, one or more video artificial intelligence (AI) models can be used to process the video, and these models may include, but are not limited to, AI quality enhancement and AI augmentation models.

[0037] Video processing operations can be computationally intensive. In some cases, video processing operations can become increasingly computationally intensive as the resolution of the input frames of video data increases (e.g., as the number of pixels to be processed per input frame of video data increases). For example, a video data frame with 4K resolution may comprise approximately four times the number of individual pixels as a video data frame with full HD (e.g., 1080p) resolution. In some examples, video processing operations can be performed by processing each pixel individually. In some examples, one or more machine learning models can be used to perform video processing operations to derive a mapping from input video data (e.g., raw video data captured by one or more cameras) to the final output video.

[0038] For example, one or more machine learning models can be used to derive a mapping between raw video data, including color values ​​at each pixel location, and the final output video. The final output video may include processed video data derived from the raw video data (e.g., based on a mapping learned by one or more machine learning models). In some examples, one or more machine learning models may be implemented in an image signal processor (ISP) of a smartphone, camera, or other computing device and used to process captured frames of video data. For example, one or more machine learning models may be used to perform processing operations on the captured frames of video data, such as denoising, deblurring, high dynamic range (HDR) imaging, upscaling (e.g., super-resolution), compression, etc. In another example, one or more machine learning models may be additionally or alternatively used to perform processing operations, such as depth estimation, semantic segmentation, human pose estimation, etc., using captured frames of video data as input.

[0039] In some examples, one or more machine learning models may include neural networks trained to perform video processing tasks. For example, the neural network may be implemented using a convolutional neural network (CNN) architecture that includes one or more convolutional filters, and / or using a recurrent neural network (RNN) architecture that utilizes one or more recurrent connections and recurrent states. The video processing neural network may include an input layer, multiple hidden layers, and an output layer. The input layer may include raw video data from one or more sensors. The hidden layers may include convolutional filters that can be applied to the input data or to the output from previous hidden layers to generate feature maps. The filters in the hidden layers may include weights used to indicate the importance of the nodes in the filters. In some cases, the neural network may have a series of many hidden layers, where earlier layers determine simple and low-level characteristics of the raw video input data, and later layers build a hierarchical structure of more complex and abstract characteristics. The neural network may then generate a final output video (e.g., constituting the output layer) based on the determined high-level features.

[0040] As previously mentioned, video processing operations performed using machine learning networks can increase computational complexity as the number of pixels per input video frame increases. In some cases, the computational complexity of video processing operations performed using machine learning networks can increase with the number of hidden layers, the number of nodes or activation functions, and / or the number of connections between nodes or layers. For example, nodes (e.g., neurons) in a neural network apply activation functions to a set of weighted inputs and return an output generated by the activation functions. Increasing the number of hidden layers or nodes can lead to an increase in the computational complexity of video processing machine learning networks, based on a larger number of mathematical operations performed for each video frame being processed.

[0041] Increasing the number of hidden layers or nodes can also lead to an increase in the size of video processing machine learning networks. For example, the activation functions and weights associated with a neural network may each be associated with one or more numerical values ​​(e.g., the values ​​used to apply the activation functions or weights). As the number of hidden layers or nodes in a machine learning network increases, the amount (e.g., quantity) of numerical values ​​stored and applied in association with the machine learning network also increases.

[0042] As previously mentioned, the operations implemented to perform video processing operations can be computationally intensive and can place a significant burden on the device's hardware resources. For example, the computational complexity of video processing operations performed using machine learning networks (e.g., neural networks) can increase with the resolution of each frame of the video data. For instance, a frame of 4K video data includes approximately four times the number of individual pixels as a frame of full HD video data, and the computational complexity of the video processing operation can increase based on the increased number of pixels being processed. In some cases, the size and complexity of the machine learning network used to perform image and video processing operations can also increase with the frame rate of the input video data. For example, when real-time video processing operations are desired, each frame of video data included in a 4K 60fps video must be processed in half the time available for processing a 4K 30fps frame (e.g., 1 / 60s vs. 1 / 30s).

[0043] In some examples, one or more image processing machine learning models (e.g., neural networks) may be implemented by a smartphone, mobile computing device, or another edge computing device including a camera or another image capture device or otherwise associated with it. In some cases, the smartphone or other edge computing device implementing the video processing machine learning model may be limited by a combination of available computing resources and power supply and / or consumption limitations of the device. The smartphone or other edge computing device implementing the image processing machine learning model may be additionally or alternatively limited based on the maximum permissible inference time (e.g., the amount of time it takes for the machine learning model to generate the processed video output based on a given input of raw video data). For example, in order to perform real-time augmentation or enhancement operations on video data, in some cases, the video processing machine learning model may have a latency target of approximately 20 milliseconds (ms) or less per frame of video data.

[0044] In some examples, smartphones and other edge computing devices, which are associated with limited computing resources and power, can implement video processing machine learning models using models with reduced sizes. For instance, the size of a video processing machine learning model can be reduced by decreasing the total number of hidden layers, nodes, activation functions, weights, etc., applied by the machine learning model when generating the processed output video based on the raw video data input. In some cases, reducing the model size can affect the accuracy of the processed output video generated by the video processing machine learning model.

[0045] In some examples, the size and / or computational complexity of a video processing machine learning model (e.g., a neural network) can be reduced by decreasing the amount of recurrent connections and / or removing them, such that the hidden states and outputs associated with a given time step are independent of the hidden states and outputs determined in previous time steps. Reducing or eliminating recurrent connections can be associated with reduced accuracy of the processed output video frames generated by the machine learning model. For example, reducing or eliminating recurrent connections may affect the machine learning model's ability to utilize temporal information when generating processed output video frames (e.g., the use of recurrent connections and temporal information may be associated with higher visual quality in the processed output images).

[0046] For example, using recurrent connections and / or temporal information in video processing machine learning models can be used to recover high-frequency details of adjacent (e.g., temporally) frames across video data to achieve temporal consistency in the processed output video generated by the video processing machine learning network. Various recurrent architectures (e.g., RNNs) that include one or more recurrent units in the network's feedforward layers can be used to perform video processing operations to generate processed output videos with relatively high visual quality. For example, recurrent units can be implemented using vanilla-RNN architectures, Conv-GRU (Gated Recurrent Unit) architectures, Conv-LSTM (Long Short-Term Memory) architectures, and various other architectures.

[0047] There is a constant need to improve video processing operations in terms of both quality and speed. Since subsequent video frames often exhibit similar features—such as similar noise, blur, and objects—RNNs are sometimes used to leverage similarity to save computational resources and improve processing quality and speed. In such RNNs, information flows indirectly across frames via a state tensor, which can be the output of the neural network processing one frame and then used as the input parameters for the neural network processing the next frame.

[0048] However, such RNNs are difficult to optimize, which may result in suboptimal quality when processing video data. Furthermore, such RNNs may require significant memory overhead, which can increase runtime and energy consumption.

[0049] Alternatively, 3D convolutional neural networks (CNNs) are sometimes used, where an entire segment consisting of a fixed number of frames is encoded into a single representation, which is then decoded into the desired number of processed frames. However, decoding a single representation into multiple frames is challenging and can lead to suboptimal results. Furthermore, 3D CNNs typically require significantly more computational resources than 2D CNNs.

[0050] Figure 1 An example of a recurrent neural network (RNN) 100 for video processing is shown. According to... Figure 1 The original video frame 110 can be input into the encoder network 120. The frame data of frame t can be obtained from x. t This indicates that the state tensor s of the previous frame t-1 is also represented. t-1 This can be input into encoder network 120. The encoder network can then output a state tensor s. t This state tensor can be used as the encoder input for the subsequent frame t+1. Additionally, the encoder network can output a representation z of frame 110. t 125. This representation can then be fed into a decoder network 130, which can further process the representation z. t And output the processed frame 140.

[0051] By utilizing the state tensor s t This ensures consistency in processing across frames. However, such RNNs are difficult to optimize, which can lead to suboptimal quality in processing video data. Furthermore, these RNNs require significant memory overhead, which can increase runtime and energy consumption.

[0052] Figure 2 An example of a 3D convolutional neural network (CNN) 200 for video processing is shown.

[0053] according to Figure 2 The set of raw video frames 2101 to 210T can be input into encoder network 220. Frame data from frames 1 to T can be obtained from x 1:T The encoder network can generate representations z of all frames in the original video frame set 210 from the raw frame data. 1:T 225. Then, it represents z. 1:T 225 can be input into decoder network 230, which can then output the processed video frame set 240. This ensures that the CNN also considers frame-related features for subsequent frames. However, decoding a single representation into multiple frames is challenging and can lead to suboptimal results. Furthermore, 3D CNNs typically require significantly more computational resources than 2D CNNs.

[0054] Therefore, this disclosure provides a video processing neural network that also considers some frame features in subsequent frames, but avoids the drawbacks of high complexity, large memory overhead, and challenging network training.

[0055] Figure 3 Multi-view CNNs are illustrated according to various aspects of this disclosure.

[0056] Figure 3 Segment 310, comprising T frames, is shown. Figure 3 In the illustrated example, T equals 3, such that segment 310 comprises three frames 312, 314, and 316. However, it should be understood that different numbers of frames per segment are also possible, such as 2 frames per segment, 4 frames per segment, 8 frames per segment, or 16 frames per segment. Each frame 312, 314, and 316 of segment 310 may include a frame height, which may correspond to the number of pixels in the vertical direction. Each frame 312, 314, and 316 of segment 310 may also include a frame width, which may correspond to the number of pixels in the horizontal direction. Each frame 312, 314, and 316 may also include a frame channel dimension. The frame channel dimension may correspond to color information. For example, for a black and white frame, the frame channel dimension C... in It can be equal to one. In other examples, in frames with RGB (red, green, blue) color information, the frame channel dimension can be equal to three. In yet another example, in frames with CMYK (cyan, magenta, yellow, key / black) color information, the frame channel dimension can be equal to four. Other exemplary frame channel dimensions are also possible.

[0057] according to Figure 3 Segment 310 can be encoded into a representation S using a multi-view convolutional network 320. The multi-view convolutional network 320 may include a convolutional neural network. The number of representations S may be less than or equal to the number of frames T for each segment. Figure 3 In the illustrated example, the number of representations S is equal to the number of frames per segment, which is three. Therefore, the frames of a segment can be encoded as a first representation 322, a second representation 324, and a third representation 326.

[0058] Encoding a segment into the S representations may include: concatenating all T frames of the segment along the frame channel dimension into the input tensor; and using a convolutional neural network (CNN) encoder to encode the input tensor into S encoded representations.

[0059] For example, each frame could have a frame height of 100 pixels, a frame width of 100 pixels, and a frame channel dimension of 3. Then, each frame could be represented by a dimension tensor. Description. Concatenating three such frames along the frame channel dimensions will thus produce a dimension tensor. .

[0060] Cascading offers the following benefits: CNNs can more easily handle similar features that may be present throughout the entire segment, such as blur, noise, or similar elements, if multiple frames are processed simultaneously (i.e., cascaded). Thus, if all cascaded frames exhibit the corresponding features, blur or image noise that may be present in all frames can be suppressed more effectively. Furthermore, the network is easier to train compared to recurrent neural networks, for example, through training algorithms explained elsewhere in this paper.

[0061] according to Figure 3 For each representation, a separate decoder network can decode the corresponding representation. Therefore, the first representation 322 can be decoded by decoder network 332, which produces the processed video frame 342. The second representation 324 can be decoded by decoder network 334, which produces the processed video frame 344. The third representation 326 can be decoded by decoder network 336, which produces the processed video frame 346.

[0062] Figure 4 Examples are shown as follows Figure 2 The encoding of the frame set in the described 3D CNN. According to Figure 4 The four frames 410, 420, 430, and 440 can be encoded in the 3D CNN 450. Encoding can be accomplished using the CNN head 460, and a single representation 470 is produced that includes the features of all four frames 410, 420, 430, and 440.

[0063] In comparison, Figure 5 The encoding of a set of frames in a multi-view CNN according to various aspects of this disclosure is illustrated.

[0064] according to Figure 5 The four frames 510, 520, 530, and 540 can be concatenated before being encoded in the multi-view CNN 550. Then, the concatenated frames 510, 520, 530, and 540 can be encoded using four separate CNN heads 562, 564, 566, and 568, resulting in four representations of the frames. Therefore, according to... Figure 5The four concatenated frames 510, 520, 530, and 540 can be encoded via CNN header 562 to generate a first representation 572 of the concatenated frames. Furthermore, the four concatenated frames 510, 520, 530, and 540 can be encoded via CNN header 564 to generate a first representation 574 of the concatenated frames. Furthermore, the four concatenated frames 510, 520, 530, and 540 can be encoded via CNN header 566 to generate a first representation 576 of the concatenated frames. Furthermore, the four concatenated frames 510, 520, 530, and 540 can be encoded via CNN header 568 to generate a first representation 578 of the concatenated frames.

[0065] Therefore, each of 572, 574, 576, and 578 can include features from all frames in the concatenated frames 510, 520, 530, and 540. Thus, unlike conventional 3D CNNs, the multi-view CNN according to this disclosure does not fold all frames into a single representation for further processing by the decoder network, but instead encodes frames into multiple representations, e.g., one representation per frame. During further processing, each representation can be further processed individually using a 2D CNN to produce the processed frame, as described above for example regarding... Figure 3 The subject of discussion.

[0066] Figure 6 An example of a 2D CNN 600 according to various aspects of this disclosure is shown.

[0067] according to Figure 6 A single frame 610 can be processed in a multi-layer CNN with L layers, each layer having C in Each frame channel input dimension and C out Each frame channel output dimension. For example, frame 610 can be processed in the first layer 620. The first layer may have a certain number (e.g., quantity) of frame channel input dimensions corresponding to the frame channel dimensions of frame 610. The first layer 620 can also have a certain number (e.g., quantity) of frame channel output dimensions. In some examples, and They can be equal. In some examples, and They can be different.

[0068] The output data from the first layer can then be processed in the second layer (630). The second layer's channel input dimensions... This corresponds to the 620 channel output dimension of the first layer. After processing the last layer 640, the processed video frame 650 can be the output of the CNN.

[0069] Figure 7Examples of various aspects according to this disclosure are illustrated. Figure 2 D CNN.

[0070] according to Figure 7 Fragment 710 can be used for multi-view Figure 2 The input to D CNN 700. Segment 700 can include a number (e.g., quantity) of T frames. Figure 7 In the illustrated example, T equals 3, such that segment 710 comprises three frames 712, 714, and 716. However, it should be understood that different numbers of frames per segment are also possible, such as 2 frames per segment, 4 frames per segment, 8 frames per segment, or 16 frames per segment.

[0071] Frames 712, 714, and 716 can be concatenated and encoded using a multi-view CNN 720 to produce the S-representation. Figure 7 In the illustrated example, S equals T, such that the number of encoded representations equals the number of frames in the encoded segment. However, different architectures are also possible where T and S are different.

[0072] The decoder network can correspond to a multi-layer 2D CNN, such as... Figure 6 The described 2D CNN, a multi-layer 2D CNN, has a number (e.g., quantity) in the i-th layer respectively. and Input channels and / or output channels.

[0073] In some examples, the parameter "grouping" in at least one convolutional layer can be set to the number of frames per segment (i.e., set to T). Preferably, the parameter "grouping" can be set to the number of frames per segment across all convolutional layers. This allows for reduced memory usage without compromising processing accuracy. Furthermore, this provides permutation invariance with respect to frame position, meaning the order of frames within a segment does not alter the processing result.

[0074] exist Figure 7 In the example where S=T, a separate representation is generated for each input video frame. This can increase the memory footprint of the CNN. Therefore, in some examples, the input frame can be divided into two or more frame partitions, and convolution can be performed separately on each frame partition. The final processed video frame can then be obtained by assembling the frames from the separately processed frame partitions. In some examples, each input frame can be divided into 64 to 128 frame partitions. This partitioning allows the CNN to have the performance benefits discussed above while avoiding the undesirable increase in memory usage.

[0075] Figure 8 Exemplary implementations based on various aspects of this disclosure are illustrated.

[0076] according to Figure 8 The CNN 800 encodes segment 810 in a multi-view convolutional encoder 820. Segments can include frames 812, 814, and 816 of number T. Figure 8 In the illustrated example, T equals 3, such that segment 810 comprises three frames 812, 814, and 816. However, it should be understood that different numbers of frames per segment are also possible, such as 2 frames per segment, 4 frames per segment, 8 frames per segment, or 16 frames per segment.

[0077] In addition to frames 812, 814, and 816, the multi-view convolutional encoder also receives the state tensor S from previous segments. t-T:t 822 is taken as input. This state tensor may contain information about features from previous segments, such as noise, blur, etc. In addition to the representations of frames 812, 812, and 816 generated by encoder 820, the encoder may also generate a state tensor S that includes features of the currently processed segment. t:t+T 824, this state tensor can then be used as input when processing subsequent segments.

[0078] The generated representations of frames 812, 814, and 816 can then be decoded in decoder network 830 to produce an output comprising processed segment 840 including processed frames 842, 844, and 846. Decoder network 830 may correspond to, as per [reference to...] Figure 3 or Figure 7 The decoder network described.

[0079] like Figure 8 The circular architecture described in the exemplary implementation allows for cross-segment connections, and thus allows for the capture of temporal features on timescales beyond the segment length. This improves temporal consistency across individual segments while using fewer computational resources than required for segment expansion.

[0080] Figure 9 Another exemplary implementation according to various aspects of this disclosure is illustrated.

[0081] according to Figure 9 Fragment 910 can be encoded in the multi-view convolutional encoder 920. Figure 9 In the illustrated example, T equals 6, such that segment 910 comprises six frames 911, 912, 913, 914, 915, and 916. However, it should be understood that different numbers of frames per segment are also possible, such as 2 frames per segment, 4 frames per segment, 8 frames per segment, or 16 frames per segment.

[0082] according to Figure 9 The encoder 920 can generate S representations 930 for T frames. However, in Figure 9 In the illustrated example, S can be less than T (T > S > 1). Then S representations can be processed in the decoder network 940. The decoder network 940 can correspond to the decoder network as described with respect to Figure 3 or Figure 7 Thus, the decoder network can generate S < T processed frames 950 from T input frames. The upsampler 960 can then generate an additional T - S frames by upsampling the S processed frames 950 received from the decoder network 940. As an output, the upsampler 960 can thus generate a total processed segment 970, which consists of T processed frames 971, 972, 973, 974, 975, and 976, i.e., the same number of frames as those that have been fed into the encoder network 920.

[0083] Thus, in the Figure 9 example, the encoder network 920 can generate only S < T representations instead of generating T representations, i.e., one representation per frame. After processing, a smaller number of frames can be upsampled by the upsampler 960 to achieve the original number of frames. Thus, this exemplary embodiment provides a more efficient use of memory and computational resources because a neural network of lower complexity is sufficient to perform the required operations. In addition, compared to the case in a CNN with T representations, this computation takes less time.

[0084] Figures 10a to 10c Illustrates the measurement results obtained by analyzing the performance of several CNNs in video denoising.

[0085] For the Figures 10a to 10c measurements shown, a dataset of 414 4K (UHD) videos is pre - processed by applying synthetic noise according to a blind noise model to the video data and feeding the so - prepared video data into the corresponding CNNs for video denoising. The peak signal - to - noise ratio (PSNR) is analyzed as a quality metric and is plotted on the Figures 10a to 10c y - axis of. To compare the performance of the multi - view convolutional neural network according to aspects of the present disclosure with the performance of conventional convolutional neural networks, several such networks are compared.

[0086] Video sequences of different lengths, i.e., 8 frames, 16 frames, 24 frames, and 32 frames, have been analyzed. The length of the sequence is plotted on the Figures 10a to 10c x - axis of the graph in.

[0087] For comparison, a highly optimized UNet and a complex recurrent neural network (RNN) based on three UNets connected to Long Short-Term Memory (LSTM) units were used. Additionally, an optimized RNN based on the original UNet with hardware-optimized RNN units and an optimized GRU based on UNet with hardware-optimized gated recurrent unit (GRU) units were also used.

[0088] Figure 10a A graph illustrating the measurement results of static processing according to various aspects of this disclosure is shown.

[0089] It can be seen that the (frame-based) UNet exhibits relatively constant performance, while the complex RNN, optimized RNN, and optimized GRU show improved performance with larger sequence lengths. In the static case, this can be attributed to the repetitive nature of these architectures.

[0090] The graph further shows that, in a static case, the multi-view CNN outperforms all four alternative networks and exhibits constant performance quality over sequence length.

[0091] Similarly, Figure 10b A graph illustrating the measurement results of motion processing according to various aspects of this disclosure is provided.

[0092] For motion scenarios, all network architectures show performance degradation with increasing sequence lengths. However, multi-view CNNs still outperform all alternatives.

[0093] Figure 10c A graph illustrating the average measurement results of the processing according to various aspects of this disclosure is shown.

[0094] Similarly, for the "average" case, which is the average between static performance and motion performance, it can be seen that the multi-view CNN outperforms all other processing networks tested.

[0095] Figure 11 A flowchart illustrating a method for processing video data according to various aspects of this disclosure is provided.

[0096] according to Figure 11 The method 1100 for processing video data may include dividing the input video data 1110 into two or more segments, each segment comprising a number of frames T, wherein each frame includes a frame height, a frame width, and a frame channel dimension C. in .

[0097] Segmenting video data into segments with several frames allows for concurrent processing of multiple frames within a segment. Therefore, it improves the efficiency of video data processing performed by the network.

[0098] according to Figure 11, method 1100 may also include encoding 1120 each segment into S encoded representations including a code height, a code width, and a code channel dimension, where T and S are integers, with T≥S>1.

[0099] Encoding each segment into S encoded representations, where S>1, ensures increased efficiency in processing the image because multiple representations, for example, via multiple decoder networks, provide more efficient decoding. In some embodiments, S may be equal to T, i.e., the number of frames in each clip may be equal to the number of encoded representations. This provides optimal quality for video frame processing.

[0100] In other embodiments, S may be less than T and greater than 1 (T>S>1). In some aspects, if S<T, the CNN decoder may include an upsampling layer that uses pixel shuffle operations, transposed convolution operations, or similar techniques to increase the channel dimension during decoding to T times C in .

[0101] Such embodiments may provide reduced memory and computational resource usage because fewer representations are required. Additionally, processing time may be significantly reduced compared to the case where S=T. Although this slightly compromises the quality of video frame processing, as long as S>1, the method is still beneficial in terms of quality compared to RNNs or CNNs with only one representation.

[0102] According to Figure 11 , encoding 1120 may include concatenating 1122 all T frames of the segment along the frame channel dimension into an input tensor.

[0103] Concatenation allows for simultaneous processing of multiple frames at once. Simultaneous processing provides improved processing results because features (such as blur, noise, or the like) occurring in multiple consecutive frames can be more effectively identified and processed by the CNN.

[0104] According to Figure 11 , encoding 1120 may also include using a Convolutional Neural Network (CNN) encoder to encode 1124 the input tensor into S encoded representations.

[0105] The CNN provides a very suitable and simple method for processing the concatenated video frames.

[0106] In some aspects, the CNN encoder may include an encoder input layer associated with an input layer encoder convolution kernel including an encoder kernel height, an encoder kernel width, an encoder kernel input channel dimension, and an encoder kernel output channel dimension C out The encoder kernel input channel dimension may be equal to T times C in , and the encoder kernel output channel dimension C out may be equal to S times a positive integer N.

[0107] By setting the kernel input channel dimension to T multiplied by C in This ensures that a sufficient number of neurons are provided for each value of the cascaded input tensor.

[0108] In some aspects, method 1100 may further include generating output video data from encoded segments of the encoded input video data. Generating the output video data may include, for each segment, decoding each of S encoded representations via a CNN decoder to obtain T decoded frames of output video data for each segment. The CNN decoder may include a decoder input layer associated with a decoder convolutional kernel, the decoder convolutional kernel including a decoder kernel height, a decoder kernel width, a decoder kernel input channel dimension, and a decoder kernel output channel dimension. The decoder kernel input channel dimension may be equal to the code channel dimension.

[0109] By setting the decoder kernel input channel dimension to the code channel dimension, a sufficient number of neurons can be provided for each value of a tensor corresponding to S representations.

[0110] In some respects, the decoder core output channel dimension can be equal to the frame channel input dimension C. in This ensures that the processed video frames have the same channel dimensions as the original unprocessed video frames.

[0111] In some aspects, segmenting the input video data into two or more segments may include dividing each frame of the input video data into two or more frame partitions, such that each segment of the input video data includes T frame partitions. Generating output video data from the coded representation may include assembling frames of the output video data from the decoded frame partitions of each decoded segment.

[0112] This segmentation allows CNNs to have the performance benefits discussed above, while avoiding an undesirable increase in memory usage.

[0113] In some aspects, method 1100 may further include: generating an initial state tensor for the first segment, forwarding the initial state tensor as input to a CNN for the first segment, and generating an updated state tensor for subsequent segments based on the initial state tensor and the output of the CNN for the first segment.

[0114] As already regarding Figure 8 In more detail, this circular architecture allows for cross-segment connections, and thus allows for the capture of temporal features on timescales beyond the segment length. This improves temporal consistency across individual segments while using fewer computational resources than required for segment expansion.

[0115] Figure 12A flowchart illustrating a method for training a CNN according to aspects of the present disclosure.

[0116] A CNN can be used to process input video data, which can be segmented into two or more segments, each segment including a number T of frames, where each frame includes a frame height, a frame width, and a frame channel dimension C in .

[0117] The CNN model can include a CNN encoder for encoding each segment into S encoded representations, where T ≥ S > 1; and a CNN decoder for decoding each encoded representation of each segment into decoded frames of the output video data.

[0118] Encoding each segment into S encoded representations, where S > 1 ensures increased efficiency in processing images because multiple representations provide more efficient decoding, for example, by means of multiple decoder networks. In some embodiments, S can be equal to T, i.e., the number of frames in each clip can be equal to the number of encoded representations. This provides the best quality for video frame processing.

[0119] In other embodiments, S can be less than T and greater than 1 (T > S > 1). In some aspects, if S < T, the CNN decoder can include an upsampling layer that uses pixel shuffle operations, transposed convolution operations, or similar techniques to increase the channel dimension during decoding to T times C in .

[0120] Such embodiments can provide reduced memory and computational resource usage because fewer representations are required. Additionally, the processing time can be significantly reduced compared to the case where S = T. Although this slightly compromises the quality of video frame processing, as long as S > 1, the method is still beneficial in terms of quality compared to RNNs or CNNs with only one representation.

[0121] According to Figure 12 , a method 1200 for training a CNN can include initializing 1210 the network parameters of the CNN model and training 1230 the CNN model based on one of: supervised learning, unsupervised learning, semi - supervised learning.

[0122] According to Figure 12 , the method 1200 can further include obtaining 1220 labeled training video data.

[0123] The labeled training data can be used for, for example, supervised or semi - supervised training of neural networks.

[0124] Training may include modifying at least a subset of the initialized network parameters by processing labeled training video data using the initialized CNN model. This provides an updated CNN that is better able to handle previously unknown video frames, as discussed in embodiments of this disclosure.

[0125] In some aspects of method 1200, the CNN model can be used for denoising video data, and the labeled training video data may include synthetic noise. Training the CNN model based on the labeled training video data may include changing a subset of the initialized network parameters such that known video noise in the labeled training video data processed by the CNN model is suppressed.

[0126] In some aspects, method 1200 may also include generating labeled training video data by adding artificially generated video noise to multiple video data.

[0127] Providing known video data and adding artificial noise, blur, or similar effects to the data offers a simple and effective way for neural networks to learn how to suppress "natural" noise or blur from unprocessed video frames. Therefore, a fully automated learning algorithm can be implemented that only requires input video data.

[0128] Figure 13 An exemplary computing system for implementing the neural network-based video processing techniques disclosed herein is illustrated.

[0129] according to Figure 13 The computing system 1300 may include a memory 1310. The memory may include data storage media such as random access memory (RAM) (such as synchronous dynamic random access memory (SDRAM)), read-only memory (ROM), non-volatile random access memory (NVRAM), electrically erasable programmable read-only memory (EEPROM), flash memory, magnetic or optical data storage media, etc.

[0130] The computing system 1300 may also include one or more processors 1320. The one or more processors may be coupled to memory 1310. The one or more processors 1320 may be configured to perform actions such as those described above. Figure 11 and / or Figure 12 The described method. Therefore, the computing system 1300 may include methods for performing such... Figure 11 and / or Figure 12 The components of the described method.

[0131] Various aspects of this disclosure are applicable to any suitable electronic device (such as a security system, smartphone, tablet, laptop, vehicle, drone, or other device) that includes or is coupled to one or more active depth sensing systems. Although devices having or coupled to a light projector are described below, various aspects of this disclosure are applicable to devices having any number of light projectors and are therefore not limited to any particular device.

[0132] The term "device" is not limited to one or a specific number of physical objects (such as a smartphone, a controller, a processing system, etc.). As used herein, a device can be any electronic device having one or more parts that implement at least some parts of this disclosure. Although the following description and examples use the term "device" to describe various aspects of this disclosure, the term "device" is not limited to a specific configuration, type, or number of objects. Additionally, the term "system" is not limited to multiple components or specific aspects. For example, a system may be implemented on one or more printed circuit boards or other substrates and may have movable or static components. Although the following description and examples use the term "system" to describe various aspects of this disclosure, the term "system" is not limited to a specific configuration, type, or number of objects.

[0133] Specific details are provided in the foregoing description to provide a thorough understanding of the aspects and examples presented herein. However, those skilled in the art will understand that these aspects can be practiced without these specific details. For clarity, in some instances, the technology may be presented as comprising individual functional blocks, including functional blocks containing devices, device components, steps or routines in methods embodied in software or a combination of hardware and software. Additional components may be used in addition to those shown in the figures and / or described herein. For example, circuits, systems, networks, processes and other components may be shown as components in block diagram form to avoid obscuring these aspects in unnecessary detail. In other instances, well-known circuits, processes, algorithms, structures and techniques may be shown without unnecessary detail to avoid obscuring the aspects.

[0134] Various aspects described above can be presented as processes or methods, depicted as flowcharts, diagrams, data flow graphs, structure diagrams, or block diagrams. Although a flowchart can describe operations as a sequential process, many operations within an operation can be executed in parallel or concurrently. Furthermore, the order of operations can be rearranged. A process terminates when its operations are completed, but a process may have additional steps not included in the accompanying diagrams. A process can correspond to a method, function, procedure, subroutine, subroutine, etc. When a process corresponds to a function, the termination of the process may correspond to the function returning to the calling function or the main function.

[0135] The processes and methods described in the examples above can be implemented using stored computer-executable instructions or computer-executable instructions otherwise obtainable from a computer-readable medium. Such instructions may include, for example, instructions and data that configure, cause or otherwise configure, a general-purpose computer, special-purpose computer, or processing device to perform a function or group of functions. The portion of the computer resources used may be accessible via a network. Computer-executable instructions may be, for example, binary files, intermediate format instructions (such as assembly language), firmware, source code, etc.

[0136] The term "computer-readable medium" includes, but is not limited to, portable or non-portable storage devices, optical storage devices, and various other media capable of storing, containing, or carrying instructions and / or data. Computer-readable media may include non-transitory media in which data can be stored and which do not include carrier waves and / or transient electronic signals propagating wirelessly or over a wired connection. Examples of non-transitory media may include, but are not limited to, magnetic disks or magnetic tapes, optical storage media (such as flash memory), memory or memory devices, magnetic disks or optical discs, flash memory, USB devices provided with non-volatile memory, network storage devices, compressed optical discs (CDs) or digital versatile optical discs (DVDs), any suitable combinations thereof, etc. Computer-readable media may store code and / or machine-executable instructions thereon, which may represent processes, functions, subroutines, programs, routines, subroutines, modules, engines, software packages, classes, or any combination of instructions, data structures, or program statements. Code segments may be coupled to other code segments or hardware circuitry by passing and / or receiving information, data, arguments, parameters, or memory contents. Information, independent variables, parameters, data, etc., can be transmitted, forwarded, or sent through any suitable means, including memory sharing, message passing, token passing, network sending, etc.

[0137] In some respects, computer-readable storage devices, media, and memories may include cables or wireless signals containing bit streams, etc. However, when referred to, non-transitory computer-readable storage media explicitly exclude media such as energy, carrier signals, electromagnetic waves, and the signals themselves.

[0138] Devices implementing the processes and methods according to these disclosures may include hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof, and may take any of a variety of form factors. When implemented as software, firmware, middleware, or microcode, program code or code segments (e.g., computer program products) for performing necessary tasks may be stored in a computer-readable or machine-readable medium. A processor may perform the necessary tasks. Typical examples of form factors include laptop computers, smartphones, mobile phones, tablet devices, or other small form factor personal computers, personal digital assistants, rack-mounted devices, standalone devices, etc. The functionality described herein may also be embodied in peripheral devices or interlocking cards. By further example, such functionality may also be implemented on circuit boards of different chips or different processes executed on a single device.

[0139] Instructions, media for delivering such instructions, computing resources for executing them, and other structures for supporting such computing resources are example components for providing the functionality described in this disclosure.

[0140] In the foregoing description, aspects of this application have been described with reference to their specific aspects, but those skilled in the art will recognize that this application is not limited thereto. Therefore, although illustrative aspects of this application have been described in detail herein, it is to be understood that the inventive concepts may be embodied and employed in various other ways, and the appended claims are not intended to be construed as including these variations unless limited by prior art. The various features and aspects of the applications described above may be used individually or in combination. Furthermore, aspects may be utilized in any number of environments and applications beyond those described herein without departing from the broader spirit and scope of this specification. Therefore, the specification and drawings should be considered illustrative rather than restrictive. For illustrative purposes, the methods are described in a particular order. It should be understood that, in alternative aspects, the methods may be performed in a different order than described.

[0141] Those skilled in the art will understand that the less than and greater than symbols or terms used herein may be replaced with less than or equal to and greater than or equal to symbols without departing from the scope of this description.

[0142] When a component is described as being “configured” to perform certain operations, such configuration can be achieved, for example, by designing electronic circuits or other hardware to perform the operations, by programming programmable electronic circuits (e.g., microprocessors or other suitable electronic circuits) to perform the operations, or any combination thereof.

[0143] The phrase “coupled to” means any component that is physically connected directly or indirectly to another component, and / or any component that communicates directly or indirectly with another component (e.g., connected to another component via a wired or wireless connection and / or other suitable communication interface).

[0144] The claim language or other language that states "at least one of" and / or "one or more of" in a set indicates that one member of the set or multiple members of the set (in any combination) satisfies the claim. For example, the claim language that states "at least one of A and B" or "at least one of A or B" means A, B, or A and B. In another example, the claim language that states "at least one of A, B, and C" or "at least one of A, B, or C" means A, B, C, or A and B, or A and C, or B and C, or A and B and C. The language that states "at least one of" and / or "one or more of" in a set does not limit the set to the items listed in the set. For example, the claim language that states "at least one of A and B" or "at least one of A or B" may mean A, B, or A and B, and may additionally include items not listed in the set of A and B.

[0145] The various exemplary logic blocks, modules, engines, circuits, and algorithm steps described in conjunction with the aspects disclosed herein can be implemented as electronic hardware, computer software, firmware, or combinations thereof. To clearly illustrate this interchangeability between hardware and software, various exemplary components, blocks, modules, circuits, and steps have been described in general terms of their functionality. Whether such functionality is implemented as hardware or software depends on the specific application and the design constraints imposed on the overall system. Those skilled in the art may implement the described functionality in different ways for each specific application, but such implementation decisions should not be construed as departing from the scope of this application.

[0146] The techniques described herein can also be implemented in electronic hardware, computer software, firmware, or any combination thereof. Such techniques can be implemented in any of a variety of devices, such as general-purpose computers, wireless communication devices (mobile phones), or integrated circuit devices with multiple uses, including applications in wireless communication devices (mobile phones) and other devices. Any feature described as a module or component can be implemented together in an integrated logic device or separately as discrete but interoperable logic devices. If implemented in software, these techniques can be implemented at least in part by a computer-readable data storage medium comprising program code including instructions that, when executed, perform one or more of the methods described above. The computer-readable data storage medium can form part of a computer program product, which may include packaging materials. The computer-readable medium may include memory or data storage media, such as random access memory (RAM) (such as synchronous dynamic random access memory (SDRAM)), read-only memory (ROM), non-volatile random access memory (NVRAM), electrically erasable programmable read-only memory (EEPROM), flash memory, magnetic or optical data storage media, etc. Additionally or alternatively, the technology may be implemented at least in part by a computer-readable communication medium that carries or conveys program code in the form of instructions or data structures that can be accessed, read and / or executed by a computer, such as propagated signals or waves.

[0147] The program code can be executed by a processor, which may include one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field-programmable arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Such processors can be configured to perform any of the techniques described in this disclosure. A general-purpose processor may be a microprocessor; however, in alternatives, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors combined with a DSP core, or any other such configuration. Therefore, as used herein, the term "processor" may refer to any of the foregoing structures, any combination of the foregoing structures, or any other structure or means suitable for implementing the techniques described herein.

[0148] Further exemplary aspects of this disclosure include:

[0149] Aspect 1. A method for processing video data, the method comprising: dividing input video data into two or more segments, each segment comprising a number of frames T, wherein each frame includes a frame height, a frame width, and a frame channel dimension C. in; encoding each segment into S encoded representations including a code height, a code width, and a code channel dimension, where T and S are integers, where T ≥ S > 1, and where encoding each segment into the S encoded representations includes: concatenating all T frames of the segment along the frame channel dimension into an input tensor; and using a Convolutional Neural Network (CNN) encoder to encode the input tensor into the S encoded representations.

[0150] Aspect 2. The method according to aspect 1, wherein the CNN encoder includes an encoder input layer associated with an encoder input layer convolutional kernel, the encoder input layer convolutional kernel including an encoder kernel height, an encoder kernel width, an encoder kernel input channel dimension, and an encoder kernel output channel dimension C out ; wherein the encoder kernel input channel dimension is equal to T multiplied by C in , and wherein the encoder kernel output channel dimension C out is equal to S multiplied by a positive integer N.

[0151] Aspect 3. The method according to any one of aspects 1 and 2, the method further comprising: generating output video data from the encoded segments of the encoded input video data, wherein generating the output video data includes: for each segment, decoding each of the S encoded representations via a CNN decoder to obtain T decoded frames of the output video data for each segment; wherein the CNN decoder includes a decoder input layer associated with a decoder convolutional kernel, the decoder convolutional kernel including a decoder kernel height, a decoder kernel width, a decoder kernel input channel dimension, and a decoder kernel output channel dimension; wherein the decoder kernel input channel dimension is equal to the code channel dimension.

[0152] Aspect 4. The method according to aspect 3, wherein the decoder kernel output channel dimension is equal to the frame channel input dimension C in .

[0153] Aspect 5. The method according to any one of aspects 1 to 4, wherein if S < T, the CNN decoder includes an upsampling layer that uses pixel shuffle operations, transposed convolution operations, or similar techniques to increase the channel dimension during decoding to T multiplied by C in .

[0154] Aspect 6. The method according to any one of Aspects 1 to 5, wherein segmenting the input video data into the two or more segments comprises segmenting each frame of the input video data into two or more frame partitions, such that each segment of the input video data comprises T frame partitions; and wherein generating the output video data from the coded representation comprises assembling the frames of the output video data from the decoded frame partitions of each decoded segment.

[0155] Aspect 7. The method according to any one of Aspects 1 to 6, the method further comprising: generating an initial state tensor for the first segment; forwarding the initial state tensor as input to a CNN for the first segment; and generating an updated state tensor for subsequent segments based on the initial state tensor and the output of the CNN for the first segment.

[0156] Aspect 8. A method for training a convolutional neural network (CNN) for processing input video data, the input video data being segmentable into two or more segments, each segment comprising a number of T frames, wherein each frame includes a frame height, a frame width, and a frame channel dimension C. in The CNN model comprises: a CNN encoder for encoding each segment into S encoded representations, where T ≥ S > 1; and a CNN decoder for decoding each encoded representation of each segment into a decoded frame of output video data; the method comprises: initializing network parameters of the CNN model; obtaining labeled training video data; and training the CNN model based on the labeled training video data, wherein training comprises: modifying at least a subset of the initialized network parameters by processing the labeled training video data using the initialized CNN model.

[0157] Aspect 9. The method according to aspect 8, wherein the CNN model is used for denoising video data, and wherein the labeled training video data includes synthetic noise; and wherein training the CNN model based on the labeled training video data comprises: changing the subset of initialized network parameters such that the known video noise in the labeled training video data processed by the CNN model is suppressed.

[0158] Aspect 10. The method according to any one of Aspects 8 and 9, further comprising: generating the labeled training video data by adding artificially generated video noise to a plurality of video data.

[0159] Aspect 11. A method for denoising video data, the method comprising: obtaining the video data; and processing the video using a CNN model trained via the method according to any one of Aspects 8 to 10.

[0160] Aspect 12. An apparatus for processing video data, the apparatus comprising: at least one memory configured to store the video data; and at least one processor coupled to the at least one memory, the at least one processor being configured to: segment the input video data into two or more segments, each segment comprising a number of frames T, wherein each frame includes a frame height, a frame width, and a frame channel dimension C. in Each segment is encoded into S encoded representations, including code height, code width, and code channel dimension, where T and S are integers, where T ≥ S > 1, and encoding each segment into the S encoded representations includes: concatenating all T frames of the segment along the frame channel dimension into an input tensor; and using a convolutional neural network (CNN) encoder to encode the input tensor into the S encoded representations.

[0161] Aspect 13. The apparatus according to Aspect 12, wherein the CNN encoder includes an encoder input layer associated with an input layer encoder convolutional kernel, the input layer encoder convolutional kernel including encoder kernel height, encoder kernel width, encoder kernel input channel dimension, and encoder kernel output channel dimension C. out The encoder core input channel dimension is equal to T multiplied by C. in And wherein the encoder core output channel dimension C out It equals S multiplied by a positive integer N.

[0162] Aspect 14. The apparatus according to any one of Aspects 12 and 13, wherein the at least one processor is further configured to: generate output video data from encoded segments of encoded input video data, wherein generating the output video data comprises: for each segment, decoding each of the S encoded representations via a CNN decoder to obtain T decoded frames of the output video data for each segment; wherein the CNN decoder includes a decoder input layer associated with a decoder convolutional kernel, the decoder convolutional kernel including a decoder kernel height, a decoder kernel width, a decoder kernel input channel dimension, and a decoder kernel output channel dimension; wherein the decoder kernel input channel dimension is equal to the code channel dimension.

[0163] Aspect 15. The apparatus according to aspect 14, wherein the decoder core output channel dimension is equal to the frame channel input dimension C. in .

[0164] Aspect 16. The apparatus according to any one of aspects 12 to 15, wherein if S < T, the CNN decoder includes an upsampling layer that uses pixel shuffle operation, transposed convolution operation or similar techniques to increase the channel dimension during decoding to T times C in .

[0165] Aspect 17. The apparatus according to any one of aspects 12 to 16, wherein splitting the input video data into the two or more segments includes splitting each frame of the input video data into two or more frame partitions such that each segment of the input video data includes T frame partitions; and wherein generating the output video data from the encoded representation includes: assembling the frames of the output video data from the decoded frame partitions of each decoded segment.

[0166] Aspect 18. The apparatus according to any one of aspects 12 to 17, the at least one processor is further configured to: generate an initial state tensor for the first segment; forward the initial state tensor as an input to the CNN for the first segment; and generate an updated state tensor for subsequent segments based on the initial state tensor and the output of the CNN for the first segment.

[0167] Aspect 19. An apparatus for training a convolutional neural network (CNN) for processing input video data that can be split into two or more segments, each segment including a number T of frames, where each frame includes a frame height, a frame width, and a frame channel dimension C in ; wherein the CNN model includes: a CNN encoder for encoding each segment into S encoded representations, where T ≥ S > 1; and a CNN decoder for decoding each encoded representation of each segment into decoded frames of the output video data; the apparatus includes: at least one memory configured to store the input video data; and at least one processor coupled to the at least one memory, the at least one processor being configured to: initialize network parameters of the CNN model; obtain labeled training video data; and train the CNN model based on the labeled training video data, where training includes: changing at least a subset of the initialized network parameters based on processing the labeled training video data using the initialized CNN model.

[0168] Aspect 20. The apparatus according to aspect 19, wherein the CNN model is used for denoising video data, and wherein the labeled training video data includes synthetic noise; and wherein training the CNN model based on the labeled training video data comprises: changing the subset of initialized network parameters such that the known video noise in the labeled training video data processed by the CNN model is suppressed.

[0169] Aspect 21. The apparatus according to any one of Aspects 19 and 20, wherein the at least one processor is further configured to generate the labeled training video data by adding artificially generated video noise to a plurality of video data.

[0170] Aspect 22. An apparatus for denoising video data, the apparatus comprising: at least one memory configured to store the video data; and at least one processor coupled to the at least one memory, the at least one processor configured to: acquire the video data; and process the video using a CNN model trained via the method according to any one of Aspects 8 to 10.

[0171] Aspect 23. A computer-readable storage medium having computer-executable code stored thereon, the computer-executable code, when executed by one or more processors, causing the one or more processors to: segment input video data into two or more segments, each segment comprising a number of frames T, wherein each frame includes a frame height, a frame width, and a frame channel dimension C. in Each segment is encoded into S encoded representations, including code height, code width, and code channel dimension, where T and S are integers, where T ≥ S > 1, and encoding each segment into the S encoded representations includes: concatenating all T frames of the segment along the frame channel dimension into an input tensor; and using a convolutional neural network (CNN) encoder to encode the input tensor into the S encoded representations.

[0172] Aspect 24. The computer-readable storage medium according to aspect 23, wherein the CNN encoder includes an encoder input layer associated with an input layer encoder convolutional kernel, the input layer encoder convolutional kernel including encoder kernel height, encoder kernel width, encoder kernel input channel dimension, and encoder kernel output channel dimension C. out The encoder core input channel dimension is equal to T multiplied by C. in And wherein the encoder core output channel dimension C out It equals S multiplied by a positive integer N.

[0173] Aspect 25. The computer-readable storage medium according to any one of Aspects 23 and 24, wherein the computer-executable code further causes the one or more processors, when executed by the one or more processors: to generate output video data from the encoded segments of the encoded input video data, wherein generating the output video data includes: for each segment, decoding each of the S encoded representations via a CNN decoder to obtain T decoded frames of the output video data for each segment; wherein the CNN decoder includes a decoder input layer, the decoder input layer being associated with a decoder convolutional kernel, the decoder convolutional kernel including a decoder kernel height, a decoder kernel width, a decoder kernel input channel dimension, and a decoder kernel output channel dimension; and wherein the decoder kernel input channel dimension is equal to the code channel dimension.

[0174] Aspect 26. The computer-readable storage medium according to Aspect 25, wherein the decoder kernel output channel dimension is equal to the frame channel input dimension C in .

[0175] Aspect 27. The computer-readable storage medium according to any one of Aspects 23 to 26, wherein if S < T, the CNN decoder includes an upsampling layer that uses pixel shuffle operations, transposed convolution operations, or similar techniques to increase the channel dimension during decoding to T times C in .

[0176] Aspect 28. The computer-readable storage medium according to any one of Aspects 23 to 27, wherein splitting the input video data into the two or more segments includes splitting each frame of the input video data into two or more frame partitions such that each segment of the input video data includes T frame partitions; and wherein generating the output video data from the encoded representations includes: assembling the frames of the output video data from the decoded frame partitions of each decoded segment.

[0177] Aspect 29. The computer-readable storage medium according to any one of Aspects 23 to 28, wherein the computer-executable code further causes the one or more processors, when executed by the one or more processors: to generate an initial state tensor for the first segment; forward the initial state tensor as an input to the CNN for the first segment; and generate an updated state tensor for subsequent segments based on the initial state tensor and the output of the CNN for the first segment.

[0178] Aspect 30. A computer-readable storage medium having computer-executable code stored thereon for training a convolutional neural network (CNN) for processing input video data capable of being segmented into two or more segments, each segment comprising a number of T frames, wherein each frame includes a frame height, a frame width, and a frame channel dimension C. in The CNN model comprises: a CNN encoder for encoding each segment into S encoded representations, where T ≥ S > 1; and a CNN decoder for decoding each encoded representation of each segment into a decoded frame of output video data; the computer-executable code, when executed by one or more processors, causes the one or more processors to: initialize the network parameters of the CNN model; obtain labeled training video data; and train the CNN model based on the labeled training video data, wherein training includes: modifying at least a subset of the initialized network parameters based on processing the labeled training video data using the initialized CNN model.

[0179] Aspect 31. The computer-readable storage medium according to aspect 30, wherein the CNN model is used for denoising video data, and wherein the labeled training video data includes synthetic noise; and wherein training the CNN model based on the labeled training video data comprises: changing the subset of initialized network parameters such that the known video noise in the labeled training video data processed by the CNN model is suppressed.

[0180] Aspect 32. The computer-readable storage medium according to any one of Aspects 30 and 31, wherein the computer-executable code, when executed by one or more processors, further causes the one or more processors to generate the labeled training video data by adding artificially generated video noise to a plurality of video data.

[0181] Aspect 33. A computer-readable storage medium having computer-executable code stored thereon, the computer-executable code, when executed by one or more processors, causing the one or more processors to: obtain the video data; and process the video using a CNN model trained via any one of aspects 8 to 10.

[0182] Aspect 34: An apparatus comprising components for performing the method according to any one of aspects 1 to 7.

[0183] Aspect 34: An apparatus comprising components for performing the method according to any one of aspects 8 to 10.

[0184] Aspect 35: An apparatus comprising components for performing the method according to aspect 11.

[0185] Aspect 36: A computer program comprising instructions that, when executed by a computer, cause the computer to perform the method according to any one of aspects 1 to 7.

[0186] Aspect 37: A computer program comprising instructions that, when executed by a computer, cause the computer to perform the method according to any one of aspects 8 to 10.

[0187] Aspect 38: A computer program comprising instructions that, when executed by a computer, cause the computer to perform the method according to aspect 11.

Claims

1. A method for processing video data, the method comprising: The input video data is divided into two or more segments, each segment consisting of T frames, where each frame includes a frame height, a frame width, and a frame channel input dimension C. in ; Each segment is encoded into S encoded representations, each including a code height, a code width, and a code channel dimension, where T and S are integers, where T ≥ S > 1, and encoding each segment into the S encoded representations includes: Concatenate all T frames of the segment along the frame channel dimension into an input tensor; and A convolutional neural network (CNN) encoder is used to encode the input tensor into the S encoded representations.

2. The method according to claim 1, wherein the CNN encoder includes an encoder input layer, the encoder input layer being associated with an input layer encoder convolutional kernel, and the input layer encoder convolutional kernel including encoder kernel height, encoder kernel width, encoder kernel input channel dimension, and encoder kernel output channel dimension C. out ;and The encoder core input channel dimension is equal to T multiplied by C. in And wherein the encoder core output channel dimension C out It equals S multiplied by a positive integer N.

3. The method according to claim 1, further comprising: Generating output video data from the encoded segments of the input video data, wherein generating the output video data includes: For each segment, each of the S encoded representations is decoded via a CNN decoder to obtain T decoded frames of the output video data for each segment; The CNN decoder includes a decoder input layer, which is associated with a decoder convolutional kernel. The decoder convolutional kernel includes kernel height, kernel width, input channel dimension, and output channel dimension. The dimension of the decoder core input channel is equal to the dimension of the code channel.

4. The method according to claim 3, wherein the decoder core output channel dimension is equal to the frame channel input dimension C. in .

5. The method according to claim 3, wherein if S < T, the CNN decoder includes an upsampling layer that uses pixel shuffle operations, transposed convolution operations, or similar techniques to increase the channel dimension during decoding to T multiplied by C in .

6. The method of claim 3, wherein segmenting the input video data into the two or more segments comprises segmenting each frame of the input video data into two or more frame partitions, such that each segment of the input video data comprises T frame partitions; and Generating the output video data from the encoded representation includes: The output video data is assembled from the decoded frame partitions of each decoded segment.

7. The method according to claim 1, further comprising: Generate an initial state tensor for the first segment of the two or more segments; The initial state tensor is forwarded as input to the CNN used for the first segment; as well as Based on the initial state tensor and the output of the CNN used for the first segment, an updated state tensor is generated for subsequent segments.

8. A method for training a convolutional neural network (CNN) model, the CNN model being used to process input video data, the input video data being segmentable into two or more segments, each segment comprising a number of T frames, wherein each frame includes a frame height, a frame width, and a frame channel dimension C. in The CNN model mentioned above includes: A CNN encoder is used to encode each segment into S encoded representations, where T ≥ S > 1; and A CNN decoder, wherein the CNN decoder is used to decode each coded representation of each segment into a decoded frame of output video data; the method includes: Initialize the network parameters of the CNN model; as well as The CNN model is trained based on one of the following: supervised learning, unsupervised learning, or semi-supervised learning.

9. The method according to claim 8, further comprising: Obtain labeled training video data; and The training includes modifying at least a subset of the initialized network parameters by processing the labeled training video data using the CNN model.

10. The method of claim 9, wherein the CNN model is used for denoising video data, and wherein the labeled training video data includes known synthetic noise; and wherein training the CNN model based on the labeled training video data comprises: The subset of initialized network parameters is modified such that the known synthetic noise in the labeled training video data processed by the CNN model is suppressed.

11. A method for denoising video data, the method comprising: Obtain the video data; as well as The video data is processed using a CNN model trained via the method of claim 9.

12. An apparatus for processing video data, the apparatus comprising: At least one memory, the at least one memory being configured to store the video data; and At least one processor, coupled to the at least one memory, is configured to: The input video data is divided into two or more segments, each segment consisting of T frames, where each frame includes a frame height, a frame width, and a frame channel input dimension C. in ; Each segment is encoded into S encoded representations, each including a code height, a code width, and a code channel dimension, where T and S are integers, where T ≥ S > 1, and encoding each segment into the S encoded representations includes: Concatenate all T frames of the segment along the frame channel dimension into an input tensor; as well as A convolutional neural network (CNN) encoder is used to encode the input tensor into the S encoded representations.

13. The apparatus of claim 12, wherein the at least one processor is further configured to: Generating output video data from the encoded segments of the input video data, wherein generating the output video data includes: For each segment, each of the S encoded representations is decoded via a CNN decoder to obtain T decoded frames of the output video data for each segment; The CNN decoder includes a decoder input layer, which is associated with a decoder convolutional kernel. The decoder convolutional kernel includes a decoder kernel height, a decoder kernel width, a decoder kernel input channel dimension, and a decoder kernel output channel dimension. and The dimension of the decoder core input channel is equal to the dimension of the code channel.

14. The apparatus of claim 13, wherein the decoder core output channel dimension is equal to the frame channel input dimension C. in .

15. The apparatus according to claim 13, wherein if S < T, the CNN decoder includes an upsampling layer that uses pixel shuffling operations, transposed convolution operations, or similar techniques to increase the channel dimension during decoding to T times C in .

16. The apparatus of claim 13, wherein segmenting the input video data into the two or more segments comprises segmenting each frame of the input video data into two or more frame partitions, such that each segment of the input video data comprises T frame partitions; and Generating the output video data from the encoded representation includes: The output video data is assembled from the decoded frame partitions of each decoded segment.

17. The apparatus of claim 12, wherein the at least one processor is further configured to: Generate an initial state tensor for the first segment of the two or more segments; The initial state tensor is forwarded as input to the CNN used for the first segment; and Based on the initial state tensor and the output of the CNN used for the first segment, an updated state tensor is generated for subsequent segments.

18. An apparatus for training a convolutional neural network (CNN) model, the CNN model being used to process input video data, the input video data being capable of being segmented into two or more segments, each segment comprising a number of T frames, wherein each frame includes a frame height, a frame width, and a frame channel dimension C. in The CNN model mentioned above includes: A CNN encoder is used to encode each segment into S encoded representations, where T ≥ S > 1; and A CNN decoder is used to decode each coded representation of each segment into a decoded frame of output video data; The device includes: At least one memory, the at least one memory being configured to store the input video data; and At least one processor, coupled to the at least one memory, is configured to: Initialize the network parameters of the CNN model; and The CNN model is trained based on one of the following: supervised learning, unsupervised learning, or semi-supervised learning.

19. The apparatus of claim 18, wherein the at least one processor is further configured to: Obtain labeled training video data; and the training includes: Based on processing the labeled training video data using the CNN model to change at least a subset of the initialized network parameters.

20. The apparatus of claim 19, wherein the CNN model is used for denoising video data, and wherein the labeled training video data includes known synthetic noise; and wherein training the CNN model based on the labeled training video data comprises: The subset of initialized network parameters is modified such that the known synthetic noise in the labeled training video data processed by the CNN model is suppressed.

21. An apparatus for denoising video data, the apparatus comprising: At least one memory, the at least one memory being configured to store the video data; and At least one processor, coupled to the at least one memory, is configured to: Obtain the video data; and The video is processed using a CNN model trained by the method according to claim 9.

22. A computer-readable storage medium having computer-executable code stored thereon, the computer-executable code causing the one or more processors, when executed by one or more processors: The input video data is divided into two or more segments, each segment consisting of T frames, where each frame includes a frame height, a frame width, and a frame channel dimension C. in ; Each segment is encoded into S encoded representations, each including a code height, a code width, and a code channel dimension, where T and S are integers, where T ≥ S > 1, and encoding each segment into the S encoded representations includes: Concatenate all T frames of the segment along the frame channel dimension into an input tensor; as well as A convolutional neural network (CNN) encoder is used to encode the input tensor into the S encoded representations.

23. A computer-readable storage medium having computer-executable code stored thereon for training a convolutional neural network (CNN) model for processing input video data capable of being segmented into two or more segments, each segment comprising a number of T frames, wherein each frame includes a frame height, a frame width, and a frame channel dimension C. in The CNN model mentioned above includes: A CNN encoder is used to encode each segment into S encoded representations, where T ≥ S > 1; and A CNN decoder is used to decode each coded representation of each segment into a decoded frame of output video data; The computer-executable code, when executed by one or more processors, causes the one or more processors to: Initialize the network parameters of the CNN model; Obtain labeled training video data; as well as The CNN model is trained based on the labeled training video data, wherein the training includes: Based on processing the labeled training video data using the CNN model to change at least a subset of the initialized network parameters.

Images