On-device reasoning method for multi-frame processing in neural network

Abstract

A method for optimizing a multi-frame processing model of a neural network includes: receiving, by a processing engine configured to execute the multi-frame processing model, a plurality of input frames, the multi-frame processing model including a plurality of convolutional layers; selecting a predetermined number of frames from the received plurality of frames to be processed by a plurality of convolutional layers; determining at least a previous frame and a plurality of subsequent frames in the selected predetermined number of frames as a frame sequence; removing previous frames by processing the sequence of frames using a plurality of filters in the multi-frame processing model; and concatenating the plurality of subsequent frames in order to the plurality of input frames for subsequent reception by the multi-frame processing model.

Description

Technical Field

[0001] The disclosed embodiments relate generally to image processing, and more particularly to a method for optimizing a multi-frame processing model of a neural network to provide fast on-device inference for a multi-frame architecture. Background Art

[0002] Multi-frame architecture refers to the design and structure of computational models or algorithms that utilize information from multiple frames or data sequences. Recurrent neural networks (RNNs), convolutional neural networks (CNNs), and multi-frame deep neural network (DNN) reasoning architectures are some examples of multi-frame architectures. Methods implemented in multi-frame architectures can be referred to as multi-frame models.

[0003] Multi-frame models are designed to process and utilize temporal and spatial information across multiple frames for various tasks, such as video processing, image processing, and / or sequential data analysis. In addition, multi-frame models rely on utilizing multiple frames to extract useful information, such as temporal and spatial information, to solve complex problems. Due to the inherent dependency of multiple frames and the corresponding useful information obtained from multiple frames, multi-frame processing is more efficient than single-frame processing techniques. Therefore, multi-frame models have been widely adopted in applications such as high dynamic range (HDR), noise removal (NR), super-resolution (SR), and low-light photography. In addition, multi-frame architectures are also implemented in various electronic devices (such as cameras, video capture devices, and / or surveillance devices).

[0004] Traditionally, multi-frame architectures maintain a queue in a buffer based on the number of frames required for inference. Thus, by maintaining a queue, the required number of frames are extracted from the input frame sequence and stored in a buffer. The multi-frame architecture processes all the required number of frames in the queue to obtain the result. However, this approach requires the use of repeated computations in higher dimensions to process all the frames in the queue. For example, multi-frame deep neural network (DNN) inference architectures deploy buffer queuing and suboptimal operations in the graph (i.e., model). Figure 1 FIG. 1 shows a queuing mechanism in a conventional multi-frame DNN inference architecture according to conventional techniques. According to the multi-frame DNN inference architecture, frames from an input sequence need to be buffered in a queue before they can be processed. The queue allows frames to be sequentially extracted and processed in the correct order. For example, in Figure 1In the exemplary queuing mechanism 100 shown, frames 1 to n are considered to be inserted into the queue 101. In addition, consider that during the first processing cycle, frames 1 to 5 are considered to be processed in the queue 101. Therefore, frames 1 to 5 are extracted from the queue 101 and given to the multi-frame DNN inference model 103 for processing. Thereafter, during the next processing cycle, frames 2 to 6 are considered for processing and given to the multi-frame DNN model 103. However, during these two processing cycles, frames 2 to 5 are common, so frames 2 to 5 are processed again. This results in redundant processing of frames between the two processing cycles. This redundant operation further introduces delays in the entire process. Therefore, the buffering process and the extraction process consume unnecessary computing and device cycles. Since the buffering process and the extraction process involve a data bus, it results in the consumption of additional power cycles in the electronic device. Therefore, in power and resource-constrained electronic devices (e.g., cameras), suboptimal inference architectures may result in poor performance and higher power usage. This further affects the overall user experience of the electronic device. Therefore, a solution that helps avoid redundant calculations is needed.

[0005] Figure 2 The first layer of processing in a higher dimension in a conventional multi-frame DNN inference architecture according to conventional techniques is shown. Multi-frame DNN inference typically involves processing higher dimensional data, such as processing spatiotemporal information. For example, in the first layer of processing 211, a concatenation (concat) 201 operation is performed in a higher dimension width (w) and height (h), which is large for high-resolution applications such as high definition (HD), full high definition (FHD), ultra high definition (4K), and applications with a resolution of 2048×1080 (2K). The output is then processed by successive layers, which also process outputs of higher dimensions of width (w) and height (h). For example, as Figure 2 As shown, output_1 from concat 201 is processed by convolution (conv) layer 203. The first operation in the multi-frame DNN inference model 103 is performed in higher dimensions (height / width). However, this is suboptimal for on-device computing elements (such as digital signal processing (DSP) / neural processing units (NPU)), which are mainly optimized to process in higher channels than higher dimensions. Therefore, this architecture is suboptimal for on-device computing elements. This greatly affects the performance of on-device computing elements. Therefore, it is necessary to process combined information in lower dimensions and higher channels to obtain better performance of on-device computing elements.

[0006] Figure 3Preprocessing and postprocessing in a multi-frame DNN inference architecture with pipeline execution according to conventional techniques are shown. The data used in multi-frame DNN inference may need to be preprocessed or postprocessed to ensure that it is in the expected format of the neural network. The preprocessing and postprocessing steps may involve data conversion, resizing, normalization, quantization, dequantization, or other transformations to prepare frames or sequences for input into the multi-frame DNN inference model. These additional operations increase computational overhead and may affect the overall efficiency of the inference process. For example, Figure 3 As shown, at step 301, input frames, i.e., frames 1 to n, are pre-processed by quantization. The quantized frames are then processed at step 303 by performing a series of operations involving concatenation and convolution operations. Thereafter, at step 305, the output of the further dequantization process is converted into the original form of the frames. Therefore, these pre-processing and post-processing operations take extra computing cycles, which results in high power and poor performance of electronic devices. Therefore, it is necessary to eliminate the above-mentioned pre-processing and post-processing steps.

[0007] It would therefore be desirable to at least address the above-mentioned disadvantages or other disadvantages, or at least provide a useful alternative for optimizing multi-frame processing in a multi-frame architecture. Summary of the invention

[0008] Technical Solution

[0009] The disclosed embodiments are provided to introduce a selection of concepts in a simplified format, which are further described in the Detailed Description. This Summary is neither intended to identify key or essential concepts nor to determine the scope of the disclosed embodiments.

[0010] A method for optimizing a multi-frame processing model of a neural network may include: receiving multiple input frames by a processing engine configured to execute the multi-frame processing model, the multi-frame processing model including multiple convolutional layers; selecting a predetermined number of frames from the received multiple frames to be processed by the multiple convolutional layers; determining at least a previous frame and multiple subsequent frames of the selected predetermined number of frames as a frame sequence; removing the previous frame by processing the frame sequence using multiple filters in the multi-frame processing model; and sequentially concatenating the multiple subsequent frames to the multiple input frames to be subsequently received by the multi-frame processing model.

[0011] The processing engine may be a component of a neural network that is at least one of a recurrent neural network (RNN), a convolutional neural network (CNN), or a deep neural network (DNN).

[0012] A method for optimizing a multi-frame processing model of a neural network may include: receiving, by a processing engine configured to execute the multi-frame processing model, a first set of cascaded frames and a plurality of previously processed input frames, the first set of cascaded frames including a current input frame; generating a second set of cascaded frames by discarding the oldest previously processed input frame among the plurality of previously processed input frames cascaded within the first set of cascaded frames; and providing the second set of cascaded frames to the multi-frame processing model to cascade the second set of cascaded frames with the current input frame.

[0013] The multi-frame processing model may include a plurality of predetermined filters. Generating the second set of cascaded frames may include: performing one or more convolution operations on the first set of cascaded frames using one or more filters of the plurality of predetermined filters; and discarding the oldest previously processed input frame based on the result of performing the one or more convolution operations.

[0014] One or more convolution operations may be performed using all of a plurality of predetermined filters.

[0015] The filter weight of each of the plurality of predetermined filters may correspond to a virtual weight. The virtual weight may be associated with each of the plurality of predetermined filters via a virtual weight interleaving method.

[0016] The method may further include: downsampling the current input frame and each concatenated frame in the second concatenated frame set to a specific resolution; and generating a third concatenated frame set by concatenating the downsampled current input frame and the downsampled second concatenated frame set.

[0017] Downsampling the current input frame to the particular resolution may include performing a convolution on the current input frame using filter weights of one or more filters.

[0018] The multi-frame processing model may also include multiple convolutional layers. The method may also include adjusting the weights of the convolutional layers in the multiple convolutional layers receiving the first set of cascaded frames based on the change of the specific resolution and the change of the input channel.

[0019] The multi-frame processing model may include a cropping layer. The method may also include: determining a cropping range of the cropping layer based on a bit depth and an input range associated with one or more input frames within the second cascade frame set; and limiting one or more outputs of the processing engine to a predetermined range based on the determined cropping range of the cropping layer.

[0020] The processing engine may be a component of a neural network that is at least one of a recurrent neural network (RNN), a convolutional neural network (CNN), or a deep neural network (DNN).

[0021] An apparatus for optimizing a multi-frame processing model of a neural network may include: a processing engine configured to execute the multi-frame processing model. The processing engine is further configured to: receive a first set of cascaded frames and a plurality of previously processed input frames, the first set of cascaded frames including a current input frame; generate a second set of cascaded frames by discarding the oldest previously processed input frame among the plurality of previously processed input frames cascaded within the first set of cascaded frames; and provide the second set of cascaded frames to the multi-frame processing model to cascade the second set of cascaded frames with the current input frame.

[0022] The multi-frame processing model may include a plurality of predetermined filters. The processing engine is configured to generate the second set of cascaded frames including being configured to: perform one or more convolution operations on the first set of cascaded frames using one or more filters of the plurality of predetermined filters; and discard the oldest previously processed input frame based on the results of the performed one or more convolution operations.

[0023] One or more convolution operations may be performed using all of the plurality of predetermined filters.

[0024] The filter weight of each of the plurality of predetermined filters may correspond to a virtual weight. The virtual weight may be associated with each of the plurality of predetermined filters via a virtual weight interleaving method.

[0025] The processing engine may also be configured to: downsample the current input frame and each of the second concatenated frame sets to a specific resolution; and generate a third concatenated frame set by concatenating the downsampled current input frame and the downsampled second concatenated frame set.

[0026] The processing engine is configured to downsample the current input frame to a particular resolution including being configured to perform convolution on the current input frame using filter weights of one or more filters.

[0027] The multi-frame processing model may also include multiple convolutional layers. The processing engine may also be configured to adjust the weights of the convolutional layers in the multiple convolutional layers receiving the first set of cascaded frames based on the change in the specific resolution and the change in the input channel.

[0028] The multi-frame processing model may include a cropping layer. The processing engine may also be configured to: determine a cropping range of the cropping layer based on a bit depth and an input range associated with one or more input frames within the second set of cascaded frames; and limit one or more outputs of the processing engine to a predetermined range based on the determined cropping range of the cropping layer.

[0029] The processing engine may be a component of a neural network that is at least one of a recurrent neural network (RNN), a convolutional neural network (CNN), or a deep neural network (DNN).

[0030] In order to further illustrate the advantages and features of the disclosed embodiments, a more specific description will be presented by reference to specific embodiments thereof shown in the accompanying drawings. It should be understood that these drawings depict only typical embodiments and therefore should not be considered as limiting the scope thereof. The disclosed embodiments will be described and explained in the accompanying drawings with additional features and details. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] These and other features, aspects, and advantages of the disclosed embodiments will be better understood when the following detailed description is read with reference to the accompanying drawings, in which like characters refer to like parts throughout the several views, and in which:

[0032] Figure 1 The queuing mechanism in a conventional multi-frame DNN inference architecture according to conventional techniques is shown;

[0033] Figure 2 shows a first processing layer of a higher dimension in a conventional multi-frame DNN inference architecture according to conventional techniques;

[0034] Figure 3 Pre-processing and post-processing in a multi-frame DNN inference architecture with pipeline execution according to conventional techniques are shown;

[0035] Figure 4 A block diagram of an electronic device 401 for optimizing a multi-frame processing model of a neural network according to an embodiment is shown;

[0036] Figure 5 shows a detailed architecture of a multi-frame processing model according to an embodiment;

[0037] Figure 6 A flow chart for optimizing a multi-frame processing model according to an embodiment is shown;

[0038] Figure 7 An operational flow for optimizing a multi-frame processing model according to an embodiment is shown;

[0039] Figure 8 An example scenario depicting a convolution operation using a plurality of predetermined filters based on a virtual weight interleaving method according to an embodiment is shown;

[0040] Fig. 9 shows a flow chart for setting filter weights and generating custom weights according to an embodiment;

[0041] Fig.10 shows custom kernel generation for downsampling concatenated frames and current input frame according to an embodiment;

[0042] Fig.11 A flowchart for optimizing a multi-frame processing model according to an embodiment is shown; and

[0043] Fig.12 The use of the embodiment is shown Figure 6 The original multi-frame processing model is converted into a device optimized model.

[0044] Furthermore, those skilled in the art will appreciate that those elements in the drawings are illustrated for simplicity and may not necessarily be drawn to scale. For example, a flow chart illustrates the method in terms of the most prominent steps involved to help improve understanding of the various aspects of the disclosed embodiments. Furthermore, with respect to the construction of the device, one or more components of the device may have been represented in the drawings by conventional symbols, and the drawings may show only those specific details relevant to understanding the disclosed embodiments so as not to obscure the drawings with details that would be clear to a person of ordinary skill in the art having the benefit of the description herein. DETAILED DESCRIPTION

[0045] To promote an understanding of the principles of the disclosed embodiments, reference will now be made to the embodiments illustrated in the drawings and specific language will be used to describe the embodiments. It will be understood, however, that no limitation of the scope of the disclosed embodiments is intended thereby and that such changes and further modifications in the illustrated systems and such further applications of the principles of the disclosed embodiments as illustrated therein are what would normally occur to one skilled in the art to which the disclosed embodiments pertain.

[0046] Those skilled in the art will understand that both the foregoing general description and the following detailed description are illustrative of the disclosed embodiments and are not intended to be restrictive thereof.

[0047] References throughout this specification to "on one hand," "on the other hand," or similar language mean that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment. Thus, the appearances of the phrases "in an embodiment," "in another embodiment," and similar language throughout this specification may, but do not necessarily, all refer to the same embodiment.

[0048] The terms "comprise", "comprising" or any other variation thereof are intended to cover a non-exclusive inclusion, such that a process or method that includes a list of steps includes not only those steps but may also include other steps not expressly listed or inherent to such process or method. Similarly, one or more devices or subsystems or elements or structures or parts preceded by "comprises..." do not exclude the presence of other devices or other subsystems or other elements or other structures or other parts or additional devices or additional subsystems or additional elements or additional structures or additional parts, without more constraints.

[0049] The embodiments of this article and its various features and advantageous details are explained more fully with reference to the non-limiting embodiments shown in the drawings and described in detail in the following description. The description of known components and processing techniques is omitted so as not to unnecessarily obscure the embodiments of this article. In addition, the various embodiments described herein are not necessarily mutually exclusive, because some embodiments can be combined with one or more other embodiments to form new embodiments. Unless otherwise stated, the term "or" as used herein refers to non-exclusive. The examples used herein are intended only to facilitate understanding of the manner in which the embodiments of this article can be practiced, and further enable those skilled in the art to practice the embodiments of this article. Therefore, the examples should not be interpreted as limiting the scope of the embodiments of this article.

[0050] As conventional in the art, embodiments can be described and illustrated according to the blocks that perform the described one or more functions. These blocks (which may be referred to herein as, for example, units or modules) are physically implemented by analog or digital circuits (such as logic gates, integrated circuits, microprocessors, microcontrollers, memory circuits, passive electronic components, active electronic components, optical components and / or hard-wired circuits), and may be optionally driven by firmware and software. The circuit may be, for example, embodied in one or more semiconductor chips, or embodied on a substrate support such as a printed circuit board. The circuit constituting the block may be implemented by dedicated hardware, by a processor (for example, one or more programmed microprocessors and associated circuits) or by a combination of dedicated hardware that performs some functions of the block and a processor that performs other functions of the block. Without departing from the scope of the disclosed embodiments, each block of the embodiment may be physically divided into two or more interactive and discrete blocks. Similarly, without departing from the scope of the disclosed embodiments, the blocks of the embodiment may be physically combined into more complex blocks.

[0051] The accompanying drawings are used to help easily understand various technical features, and it should be understood that the embodiments presented herein are not limited by the accompanying drawings. Therefore, the embodiments of the present disclosure should be interpreted as extending to any changes, equivalents and substitutes other than those specifically set forth in the accompanying drawings. Although the terms first, second, etc. can be used herein to describe various elements, these elements should not be limited by these terms. These terms are usually only used to distinguish one element from another element.

[0052] The terms "neural networks" and "neural network layers" and "neural network" are used interchangeably. The terms "multi-frame model", "multi-frame DNN model", "multi-frame model", "multi-frame processing model", and "multi-frame processing engine" are used interchangeably.

[0053] Referring now to the drawings, and more particularly to Figures 4 to 12 , wherein like reference numerals indicate corresponding features throughout the several views, there is shown a preferred embodiment.

[0054] Figure 4 A block diagram of an electronic device 401 for optimizing a multi-frame processing model of a neural network according to an embodiment as disclosed herein is shown. Examples of electronic device 401 may include, but are not limited to, a smart phone, a tablet computer, a personal digital assistant (PDA), a camera, a monitoring device, an image capture device, a video recording device, or any other electronic device capable of processing image or video data.

[0055] The electronic device 401 may include an input unit 403, a processor 405, an output unit 409, and a memory 415 coupled to each other. The processor 405 is coupled to a processing engine 407. The processing engine 407 includes a multi-frame processing engine 411. A detailed explanation of each component as described above will be explained in the following paragraphs.

[0056] The input unit 403 may be configured to receive a plurality of frames. In a non-limiting example, the plurality of frames 413 may correspond to a plurality of frames from image or video data. The plurality of frames may be provided as input to a multi-frame processing engine 411 of the processing engine 407. Without departing from the scope of the disclosed embodiments, the plurality of frames may alternatively be referred to as frames.

[0057] The processor 405 operates with the memory 415, the processing engine 407, and the output unit 409. The processor 405 can be a single processing unit or multiple units, all of which can include multiple computing units. The processor 405 can be implemented as one or more microprocessors, microcomputers, microcontrollers, digital signal processors, central processing units, logic processors, virtual processors, state machines, logic circuits, and / or any device that manipulates signals based on operating instructions. In addition to other capabilities, the processor 405 is configured to obtain and execute computer-readable instructions and data stored in the memory 415. The processor 120 may include one or more processors, which may be a general-purpose processor (such as a central processing unit (CPU)), an application processor (AP), a graphics processing unit (such as a graphics processing unit (GPU)), a visual processing unit (VPU), and / or an artificial intelligence (AI) dedicated processor (such as a neural processing unit (NPU)).

[0058] The processing engine 407 is implemented by processing circuits such as logic gates, integrated circuits, microprocessors, microcontrollers, memory circuits, passive electronic components, active electronic components, optical components, and / or hard-wired circuits, and may optionally be driven by firmware. The circuits may, for example, be embodied in one or more semiconductor chips, or on a substrate support such as a printed circuit board.

[0059] The memory 415 may store instructions to be executed by the processor 405 for optimizing the multi-frame processing engine 411. The memory 415 may include a non-volatile storage element. Examples of such non-volatile storage elements may include a magnetic hard disk, an optical disk, a floppy disk, a flash memory, or an electrically programmable memory (EPROM) or an electrically erasable programmable (EEPROM) memory. In addition, in some examples, the memory 415 may be considered to be a non-transitory storage medium. The term "non-transitory" may indicate that the non-transitory storage medium is not embodied in a carrier or propagation signal. However, the term "non-transitory" should not be interpreted as the memory 415 being non-removable. In some examples, the memory 415 may be configured to store a larger amount of information. In some instances, the non-transitory storage medium may store data that may change over time (e.g., in a random access memory (RAM) or a cache memory). The memory 415 may be an internal storage unit, or it may be an external storage unit of the electronic device 401, a cloud storage device, or any other type of external storage device.

[0060] The multi-frame processing engine 411 may be implemented in a neural network such as a CNN, a DNN, and / or a RNN. Figure 5 Detailed architecture of the multi-frame processing engine 411 is shown. The multi-frame processing engine 411 may include a plurality of predetermined filters 501, a plurality of convolutional layers 505, a cascade layer 503, and custom weight convolutional layers 507 and 509.

[0061] For multiple predetermined filters 501, the filter weight of each of the multiple predetermined filters 501 may correspond to a virtual weight. The virtual weight may alternatively be referred to as a custom weight or filter weight. In addition, without departing from the scope of the disclosed embodiment, the multiple predetermined filters 501 may alternatively be referred to as predetermined filters or filters. A pseudo weight may be assigned to each of the multiple predetermined filters 501 using a pseudo weight interleaving method to retain features in any frame. For example, consider the case where the first feature in the 2×2 block matrix of the input frame needs to be retained. Therefore, the virtual weight of the predetermined filter 501 is assigned to "1, 0, 0, 0". That is, the corresponding filter weight of the corresponding filter whose feature requires to be retained is assigned to "1", and the remaining filters may be assigned to zero. Therefore, when the value of the first feature is multiplied by the assigned virtual weight (ie, "1"), the corresponding first feature is retained. In addition, for the remaining features, when multiplied by the assigned virtual weight (ie, 000), the corresponding features of the remaining features in the 2×2 block matrix become zero and are therefore not retained.

[0062] The concatenation layer 503 may be configured to concatenate one or more frames. A frame may be a processed frame or an unprocessed frame. For example, a frame (such as an original frame or an initial frame) that has not undergone any processing (such as a convolution or concatenation operation) may be referred to as an unprocessed frame. In addition, a frame that has undergone any processing (such as a convolution) or any other operation may be referred to as a processed frame. The concatenation operation performed by the concatenation layer 503 may alternatively be referred to as "concat" or a concat operation.

[0063] Multiple convolutional layers 505 may be configured to perform convolution operations on multiple input frames received. The convolution operations performed by the convolutional layer 505 may be alternatively referred to as "conv" or conv operations. As described above, in order to retain features, convolution operations are performed using filter weights. In a non-limiting example, the convolution operation may be performed by, for example, stride convolution, dilated convolution, transposed convolution, and / or grouped convolution. The explanation of the convolution operation is performed with reference to stride convolution. However, the disclosed method may be performed by any convolution operation. Typically, stride convolution is a type of convolution operation in which the kernel moves on the input feature map with a step size greater than 1. This results in a downsampled output feature map compared to the input. The stride value determines how many pixels the kernel shifts in each direction during the convolution process. A stride convolution with a pseudo-weight interleaving method may be performed on the input frame. The operation retains width and height while retaining the original information.

[0064] Additional convolutional layers, referred to herein as custom weight convolutional layers 507 , may be implemented in the multi-frame processing engine 411 .

[0065] The custom weight convolution layer 507 can be configured to determine the custom weights of the predetermined filters 501 and the number of filters required in the custom weight convolution layer 507. In addition, an additional convolution layer referred to herein as a cropping layer 509 is implemented in the multi-frame processing engine 411. The cropping layer 509 can be implemented to eliminate conventional pre-processing and post-processing steps. Therefore, the cropping layer 509 is configured to limit one or more outputs of the processing engine 407 to a predetermined range. The detailed operation of determining the custom weight and the operation of the cropping layer are explained in the following paragraphs.

[0066] Return to reference Figure 4 , the output unit 409 can be configured to provide output processed by the processing engine 407 .

[0067] A framework for converting a multi-frame processing model into a device-optimal multi-frame inference model through an auxiliary feedback loop may be provided. Thus, the output from the auxiliary feedback loop is provided as the input of the original multi-frame processing model, which reduces computation time and power consumption. The detailed implementation and working will be explained in the following paragraphs.

[0068] Figure 64. A flowchart depicting a method 600 for optimizing a multi-frame processing model according to an embodiment is shown. The method 600 may be implemented at an electronic device 401. Figure 4-Figure 11 The method 600 is explained. The method 600 can be implemented in any multi-frame processing model during offline mode to obtain a device optimal model for inference. Figure 7 FIG. 2 shows an operation flow for optimizing a multi-frame processing model according to an embodiment. Figure 7 , the auxiliary feedback loop 700A is added to the original loop 700 of the multi-frame processing engine 411. For the sake of brevity, the following descriptions are made in conjunction with each other. Figure 6 and Figure 7 .

[0069] Initially, during processing of an image or video stream, processing engine 407 may be configured to receive a plurality of frames 413, such as Figure 4 and Figure 5 As explained above in Figure 5 As explained in , generally, in a multi-frame reasoning model, for each processing cycle, a predetermined number of frames are selected for processing. Subsequently, after the predetermined number of frames are selected, the selected frames are processed in the original loop 700 of the multi-frame processing engine 411. For example, according to Figure 7 In the example embodiment shown, the predetermined number of frames is considered to be three. Thus, the selected frame undergoes a first processing cycle through the primitive loop 700.

[0070] At step 601, the processing engine 407 may be configured to receive a first set of cascaded frames including a current input frame and a plurality of previously processed input frames. The plurality of previously processed input frames may be frames modified and received from a previous execution or a previous processing cycle or a first processing cycle after being processed by the original loop 700. As an example, the first set of cascaded frames may be frames processed from a predetermined number of frames. In addition, the current input frame may be an incoming frame received by the processing engine 407 during each processing cycle of an image or video. The current input frame may alternatively be referred to as a current frame.

[0071] You can refer to Figure 7 To explain step 601. Figure 7As shown, at block 703, the cascade layer 503 of the processing engine 407 may be configured to receive a first set of cascaded frames including a current input frame. In the example scenario, frame 1 and frame 2 are depicted as F1 and F2, respectively. Frames F1 and F2 may be a plurality of previously processed input frames 717 received by the cascade layer 503 at block 703. In addition, frame 3, depicted as F3, is a current frame at block 705, which is received by the cascade layer 503 at block 703. According to an example embodiment, the frames (i.e., the previously processed input frames and the current input frame) are cascaded to an output cascaded frame 719, i.e., F1, F2, and F3 are cascaded at block 703. Thus, according to the given example, the first set of cascaded frames 719 is frames F1, F2, and F3 arranged as a sequence.

[0072] After generating the first set of cascaded frames, at step 603, the multi-frame processing engine 411 of the processing engine 407 can be configured to generate a second set of cascaded frames by discarding the oldest previously processed input frame among the plurality of previously processed input frames and cascaded within the first set of cascaded frames. Referring to the same example scenario as described above, in step 603, a second set of cascaded frames 721 is generated by discarding frame F1, which is the oldest of the plurality of previously processed input frames 717 and cascaded within the first set of cascaded frames 719. When the frames are arranged in a sequence, the oldest frame in the sequence can be identified and discarded. Therefore, the second set of cascaded frames can include frames F2 and F3 in the sequence. In the example embodiment, only a single frame is shown as being removed or discarded. However, according to alternative embodiments, more than one frame can be removed.

[0073] The generation of the second cascade frame set may include steps 603-1 and 603-2. Therefore, in order to generate the second cascade frame set, a convolution operation may be performed by multiple convolution layers 505. In a non-limiting example, a stride convolution with a pseudo-weight interleaving method may be used to generate the second cascade frame set. In the above paragraphs and with reference to Figure 8 The explanation of the virtual weight interleaving method using predetermined filters and filter weights to preserve features is explained. Therefore, for the sake of brevity, the detailed description is omitted here.

[0074] Therefore, at step 603-1, the plurality of convolutional layers 505 of the multi-frame processing engine 411 in the processing engine 407 are configured to perform one or more convolution operations on the first set of cascaded frames using one or more filters of the plurality of predetermined filters 501. Thereafter, at step 603-2, the plurality of convolutional layers 505 of the multi-frame processing engine 411 in the processing engine 407 are configured to discard the oldest previously processed input frame among the plurality of previously processed input frames based on the one or more convolution operations performed. Figure 8 The example scenario depicted in explains a convolution operation using a plurality of predetermined filters 501.

[0075] Figure 8 An example scenario depicting a convolution operation using multiple predetermined filters based on a virtual weight interleaving method according to an embodiment is shown. In the example, the output of the cascade operation 703 of the input frame is represented as a matrix 801. A stride convolution with a pseudo weight interleaving method is performed on the matrix 801. The convolution operation is a multiplication of the input with the filter weight. Therefore, for example, if the input is "F1, F2, F3", and the filter is "0, 1, 0", the output of the filtering process is given by Equation 1.

[0076] (F1*0)+(F2*1)+(F3*0)=F2*---(1)

[0077] Therefore, if the filter is "0, 0, 1", the output is given by Equation 2.

[0078] (F1*0)+(F2*0)+(F3*1)=F3·-----(2)

[0079] According to the above equation, the filter weights are required to be set so as to preserve the features corresponding to the corresponding frames. Figure 7 In the example scenario of , to remove or discard features corresponding to frame F1, the corresponding filter weights of F1 are set to zero and the corresponding filter weights of frames F2 and F3 are set to 1. Therefore, the filter weights of filter 1 and filter 2 correspond to (0, 1, 0) and (0, 0, 1).

[0080] Fig. 9 A flow chart depicting a method 900 for setting filter weights and generating custom weights according to an embodiment is shown. The method 900 may be an operational flow for setting filter weights to retain features of a corresponding frame and generating custom weights for a new output. The new output may refer to an output generated after discarding or removing the oldest frame in the predetermined frames in the auxiliary feedback loop 700A. The method 900 may be performed in Figure 4 401. Initially, at step 901, the processing engine 407 may be configured to determine a predetermined number of frames for processing by the plurality of convolutional layers 505. The predetermined number of frames may be received as input for inference. Thereafter, at step 903, the processing engine 407 is configured to determine the number of filters required in the custom weight convolutional layer 507. The determination of the number of filters required in the custom weight convolutional layer 507 is given by Equation 3.

[0081] The number of filters required in the custom weighted convolutional layer = the number of previous frames required for processing --- (3)

[0082] At step 905, the processing engine may be configured to create a matrix of size 1*1*number of frames required-1. Figure 8The matrix thus created is depicted at box 801 in the example of . Thereafter, at step 907, the processing engine 407 can be configured to initialize the matrix with zeros. Then, at step 909, the processing engine 407 can be configured to determine whether the number of filters remaining is greater than zero. Based on the determination of "no", that is, the number of filters remaining is not greater than zero, at step 911, one or more weights of the custom weight convolution layer 507 for the new output are generated. The new output is a second set of cascaded frames, which will be used for the next processing in conjunction with the next current input frame. Then, in Figure 7 At block 711 of FIG. 4 , the generated custom weights of the new output are processed. In addition, based on the determination of “yes”, i.e., the number of remaining filters is greater than zero, at step 915, the processing engine 407 is configured to set the corresponding filter to 1. Then, at step 913, the filter count is increased, and the processing engine 407 performs step 909 again. Therefore, the custom weights are assigned to multiple predetermined filters using the pseudo-weight interleaving method.

[0083] The generated second set of concatenated frames and the current input frame may be downsampled. Fig.10 A method 1000 for generating a custom kernel for downsampling a generated second set of cascaded frames and a current input frame according to an embodiment is shown. The method 1000 may be an operational flow for generating a custom kernel for downsampling a generated second set of cascaded frames and a current input frame. Downsampling may be a method of reducing the resolution of a frame to a lower resolution. The method 1000 may be a method of generating a custom kernel for downsampling a generated second set of cascaded frames and a current input frame. Figure 4 1001, the processing engine 407 may be configured to determine the required reduction of the dimensions of the second set of cascaded frames and the current frame to a specific resolution. As an example, the reduction requirement is based on model complexity. Based on this determination, at step 1003, the processing engine 407 may be configured to calculate the kernel shape. In a non-limiting example, the kernel shape, the number of filters, and the stride required for the convolution are determined. The equations for the kernel shape, the number of filters, and the stride are provided below:

[0084] Core shape = reduction ratio * reduction ratio -- (4)

[0085] Number of filters = reduction ratio × reduction ratio – (5)

[0086] Step = decrease ratio -- (6)

[0087] In a non-limiting example, consider that the required reduction in dimensionality is 2. Then, based on equations 4, 5, and 6, the kernel shape, number of filters, and stride are found as follows:

[0088] Kernel shape = 2*2 (read as 2 times 2)

[0089] Number of filters = 2 × 2 = 4

[0090] Stride = 2

[0091] Thereafter, at step 1005, the processing engine 407 may be configured to initialize the filter weights with zeros. Since few positions will have values, the remaining positions will be dummies interleaved with zeros, so the initial values ​​are initialized to zero. At step 1007, the processing engine 407 may be configured to determine whether the number of remaining filters is greater than zero. The determination of whether the number of remaining filters is greater than zero is performed based on Equation 3. Therefore, based on the determination of "yes", that the number of remaining filters is greater than zero, the filter weights are set at step 1009. Therefore, when the number of remaining filters is not greater than zero, the shaped kernel weights are provided to the custom weight convolution layer 507 for the convolution operation at step 1010. The shaped kernel weights are provided to Figure 7 The filter count is incremented at step 1111.

[0092] The processing engine 407 may be configured to generate a third set of concatenated frames by concatenating the downsampled current input frame and the downsampled second set of concatenated frames at block 703. The third set of concatenated frames is further processed by the original loop 700 to provide an output.

[0093] Return to reference Figure 6 , after performing various operations at step 603, method 600 also includes, at step 605, providing the second set of cascaded frames to the multi-frame processing model 411 for cascading the second set of cascaded frames with the next received input frame. Therefore, the new output generated as the second set of cascaded frames is provided as input for the next processing cycle. Therefore, the new output acts as an auxiliary feedback output that preserves data while discarding dimensions. This auxiliary feedback output eliminates redundant processing present in traditional solutions. Reference Figure 7 , the second cascaded frame set F2 and F3 are provided to block 701 for use in the next processing cycle. In addition, the new input frame (e.g., frame F4 (not shown)) then becomes the current frame. The rest of the process remains the same as described above.

[0094] The processing engine 407 may be configured to adjust the weights of the convolutional layers in the plurality of convolutional layers 505 that receive the first set of concatenated frames based on the changes in the specific resolution and input channels. Figure 7 , at box 709, the convolution layer with the adjusted original weights is provided to the next convolution layer 713. When the previous frame and the current frame are downsampled at box 705, the weights of the multiple convolution layers 505 are required to be adjusted in subsequent operations. Therefore, based on the specific resolution and input channels required, the weights are adjusted for consecutive operations.

[0095] A clipping layer 509 may be provided at the output stage to eliminate the need for pre-processing and post-processing steps. Figure 7 , providing cropping layer 1 715 at the output of the original loop 700, and providing cropping layer 2 501 at the auxiliary feedback loop 700A. Therefore, the processing engine 407 can be configured to determine the cropping range of the cropping layer based on the bit depth and input range associated with one or more input frames within the second cascade frame set. In particular, when determining the cropping layer range, the value and bit depth of the input or output are considered. In the case where the value range and bit depth are different, a custom quantization value can be used to adjust the bias and scaling. In addition, the processing engine 407 can also be configured to limit one or more outputs of the processing engine to a predetermined range based on the determined cropping range of the cropping layer. For example, if the input range is from -10 to 200 and the input frame bit depth is 8 bits, that is, the maximum number of levels is 256, then the clip minimum and maximum ranges are 0 and 255, respectively.

[0096] Fig.11 A flow chart for optimizing a multi-frame processing model according to an embodiment is shown. Fig.11 Depicted is a method 1111 implemented at an electronic device 401 according to some embodiments. The method 1111 is similar to Figure 6 Therefore, for the sake of brevity, a detailed explanation thereof is omitted here. Figure 5-10 The explanation of the method depicted in also applies to method 1111. At step 1101, the processing engine 404 can be configured to receive multiple input frames. Thereafter, at step 1103, the processing engine 407 can be configured to select a predetermined number of frames from the received multiple frames for processing by multiple convolutional layers 505. Thereafter, at step 1105, the processing engine 407 can be configured to determine at least a previous frame and multiple subsequent frames of the selected predetermined number of frames as a frame sequence. Here, the previous frame is the oldest frame. Thereafter, at step 1107, the processing engine 407 can be configured to remove or discard the previous frame by processing the frame sequence using multiple filters in the multi-frame processing model 411. Thereafter, at step 1109, the processing engine 407 can be configured to sequentially cascade multiple subsequent frames other than the previous frame to multiple input frames so as to be subsequently received into the multi-frame processing model 411.

[0097] Fig.12 The use of the embodiment is shown Figure 6 The method converts the original multi-frame processing model into a device optimized model. Initially, the pre-trained original multi-frame processing model is considered as the starting point of the multi-frame processing. The frames can be cascaded at block 1201, and the weights can be adjusted based on the pseudo weight interleaving method to preserve the output at block 1203. The processes at blocks 1201 and 1203 are as follows Figure 7 1205, the output quality of the original loop 700 is checked against the expected output quality. If the output has the expected output quality, the same weights and parameters are provided to Figure 7 The auxiliary feedback loop 700A is added to the multi-frame processing engine 411 for further processing. That is, at block 1209, the multi-frame processing engine 411 adds an auxiliary feedback loop 700A for customizing the output, which has the same weights and parameters to be utilized in the next processing cycle. If the output does not have the desired output quality, the multi-frame processing engine is then trained with new operations for further processing through the auxiliary feedback loop 700A to generate a device optimization model.

[0098] The disclosed method eliminates redundant processing of input frames based on the application of discarding the oldest frame. In addition, post-processing calculations are further eliminated by adding a cropping layer at the output.

[0099] Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art.The systems, methods, and examples provided herein are illustrative only and are not intended to be limiting.

[0100] Although specific language has been used to describe this subject, any limitation resulting therefrom is not intended. It is apparent to those skilled in the art that various work modifications may be made to the method to realize the inventive concept taught herein. The accompanying drawings and the foregoing description provide examples of embodiments. It will be appreciated by those skilled in the art that one or more of the elements described may be well combined into a single functional element. Alternatively, some elements may be divided into multiple functional elements. Elements from one embodiment may be added to another embodiment.

[0101] The embodiments disclosed herein may be implemented using at least one hardware device and performing a network management function to control elements.

[0102] The foregoing description of the specific embodiments will so fully reveal the general nature of the embodiments herein that others can easily modify and / or adapt these specific embodiments for various applications by applying current knowledge without departing from the general concepts, and therefore, these adaptations and modifications should and are intended to be understood to be within the meaning and scope of equivalents of the disclosed embodiments. It should be understood that the wording or terminology employed herein is for descriptive and not limiting purposes. Therefore, although the embodiments herein have been described according to preferred embodiments, those skilled in the art will recognize that the embodiments herein can be practiced by modification within the scope of the embodiments as described herein.

Claims

1. A method for optimizing a multi-frame processing model of a neural network, the method comprising: receiving a plurality of input frames by a processing engine, the processing engine being configured to execute a multi-frame processing model, the multi-frame processing model comprising a plurality of convolutional layers; selecting a predetermined number of frames from the received plurality of frames to be processed by the plurality of convolutional layers; determining at least a previous frame and a plurality of subsequent frames among the selected predetermined number of frames as a frame sequence; removing previous frames by processing the frame sequence using multiple filters in a multi-frame processing model; as well as The multiple subsequent frames are sequentially concatenated to the multiple input frames to be subsequently received by the multi-frame processing model.

2. A method for optimizing a multi-frame processing model of a neural network, the method comprising: receiving, by a processing engine, a first set of cascaded frames and a plurality of previously processed input frames, the processing engine being configured to execute a multi-frame processing model, the first set of cascaded frames including a current input frame; generating a second set of concatenated frames by discarding an oldest previously processed input frame among a plurality of previously processed input frames concatenated within the first set of concatenated frames; as well as The second set of concatenated frames is provided to the multi-frame processing model to concatenate the second set of concatenated frames with the current input frame.

3. The method according to claim 2, in, The multi-frame processing model includes a plurality of predetermined filters, and The step of generating the second cascade frame set includes: performing one or more convolution operations on the first set of concatenated frames using one or more filters of a plurality of predetermined filters; and The oldest previously processed input frame is discarded based on the results of performing one or more convolution operations.

4. The method according to claim 3, wherein: The one or more convolution operations are performed using all of a plurality of predetermined filters.

5. The method according to claim 3, in, The filter weight of each predetermined filter of the plurality of predetermined filters corresponds to a virtual weight, and Wherein, the virtual weight is associated with each predetermined filter of the plurality of predetermined filters via a virtual weight interleaving method.

6. The method according to claim 3, further comprising: downsampling the current input frame and each cascade frame in the second cascade frame set to a specific resolution; as well as A third set of concatenated frames is generated by concatenating the downsampled current input frame with the downsampled second set of concatenated frames.

7. The method according to claim 6, wherein: Downsampling the current input frame to a specific resolution includes performing a convolution on the current input frame using filter weights of one or more filters.

8. The method according to claim 6, in, The multi-frame processing model also includes multiple convolutional layers, and The method further includes adjusting the weight of a convolutional layer among the multiple convolutional layers that receives the first set of cascaded frames based on changes in the specific resolution and changes in the input channel.

9. The method according to claim 2, in, The multi-frame processing model includes a clipping layer, and Wherein, the method further comprises: determining a cropping extent of the cropping layer based on a bit depth and an input extent associated with one or more input frames within the second set of cascaded frames; and Based on the determined clipping range of the clipping layer, one or more outputs of the processing engine are restricted to a predetermined range.

10. A device for optimizing a multi-frame processing model of a neural network, the device comprising: A processing engine configured to execute a multi-frame processing model, the processing engine further configured to: receiving a first set of concatenated frames and a plurality of previously processed input frames, the first set of concatenated frames including a current input frame; generating a second set of concatenated frames by discarding an oldest previously processed input frame among a plurality of previously processed input frames concatenated within the first set of concatenated frames; as well as The second set of concatenated frames is provided to the multi-frame processing model to concatenate the second set of concatenated frames with the current input frame.

11. The device according to claim 10, in, The multi-frame processing model includes a plurality of predetermined filters, and The processing engine being configured to generate the second cascade frame set includes being configured to: performing one or more convolution operations on the first set of concatenated frames using one or more filters of a plurality of predetermined filters; and Based on the results of the one or more convolution operations performed, the oldest previously processed input frame is discarded.

12. The device according to claim 11, wherein The one or more convolution operations are performed using all of a plurality of predetermined filters.

13. The device according to claim 11, in, The filter weight of each predetermined filter of the plurality of predetermined filters corresponds to a virtual weight, and Wherein, the virtual weight is associated with each predetermined filter of the plurality of predetermined filters via a virtual weight interleaving method.

14. The device according to claim 11, wherein: The processing engine is further configured to: downsampling the current input frame and each cascade frame in the second cascade frame set to a specific resolution; as well as A third set of concatenated frames is generated by concatenating the downsampled current input frame with the downsampled second set of concatenated frames.

15. The device according to claim 14, wherein: The processing engine being configured to downsample the current input frame to a particular resolution includes being configured to: perform convolution on the current input frame using filter weights of one or more filters.

Images