Method and system for reducing the complexity of a processing pipeline using feature expansion training - Patents.com

Abstract

A method and electronic device are provided for low complexity in-loop filter inference using feature augmentation training, including combining spatial and spectral domain features, using spectral domain features for global feature extraction and signaling to spatial streams during training, using separable spectral domain streams for differential complexity of training vs. inference, and combining unique loss sets resulting from multi-stream and multi-feature approaches to obtain optimal output.

Description

[Technical field]

[0001] The present invention relates to artificial intelligence (AI), and more particularly to a method and system for reducing the complexity of a processing pipeline using feature expansion training. [Background technology]

[0002] Generally, video compression is the process of reducing the amount of data required to represent a particular image or video clip / content. For the video content, such reduction in data amount means lower storage requirements and lower transmission bandwidth requirements. Video compression typically involves the removal of information that is not necessary to view the video content. Currently, in compression systems, there are many existing methods for video compression, including intraframe compression (or spatial compression) such as JPEG (M-JPEG: motion joint photographic experts group) and interframe compression such as H.264. However, video content compressed using existing methods has lower quality (e.g., pixelated edges, haze, rings, etc.) and requires more computing power.

[0003] Additionally, artificial intelligence (AI)-based video compression methods are frequently used to compress video content to maintain video quality. AI-based video compression refers to the use of artificial intelligence algorithms to analyze and optimize the compression of video content. The process utilizes machine learning techniques to efficiently reduce the size of video files while maintaining or even improving the quality of the video files. This type of compression is becoming increasingly popular because it can provide better compression results compared to existing compression methods.

[0004] However, AI-based video compression methods have certain limitations due to memory and time requirements. The deployment of AI-based compression solutions on embedded systems or electronic devices such as smartphones is becoming increasingly impractical. As a result, it is important to develop new solutions for improving the performance of electronic devices while maintaining the feasibility of using AI-based compression solutions in such embedded systems.

[0005] Complexity is also a key factor in video compression codecs / methods. Existing AI-based video compression methods add more residual layers to the model, making the model deeper and improving the performance of video compression / visual processing. This creates a gradient problem, hindering real-time solutions, especially in embedded systems such as smartphones. Furthermore, in many use cases, the increased amount of data in the weighted aspects contributes to metadata, requiring more memory.

[0006] It would therefore be desirable to address the shortcomings noted above, or other shortcomings of existing compression systems, or at least provide a useful alternative that reduces the complexity of the processing pipeline and provides lightweight on-device inference, e.g., for video compression.

[0007] The above information is presented solely as background information to aid in the understanding of the present disclosure. It has not been determined or asserted that any of the foregoing is prior art to the present disclosure. Summary of the Invention [Problem to be solved by the invention]

[0008] This Summary is provided to introduce a selection of concepts in a simplified format that are further described in the Detailed Description of the Disclosure. This Summary is not intended to identify key or essential inventive concepts of the disclosure, nor is it intended to delineate the scope of the disclosure.

[0009] Aspects of the present disclosure address at least the problems and / or shortcomings noted above and provide at least the advantages described below. Accordingly, one aspect of the present disclosure is to provide a method and system for reducing the complexity of a processing pipeline using feature augmentation training.

[0010] Additional aspects will be set forth in part in the description which follows, and in part will be obvious from the detailed description, or may be learned by practice of the embodiments presented. [Means for solving the problem]

[0011] According to one aspect of the present disclosure, a method for global view feature expansion is provided. The method includes receiving an input signal, the input signal being a spatial domain signal, by converting the received input signal into a spectral domain signal to obtain a spectral domain signal. The method also includes extracting a first predefined deep learning feature set from the corresponding spectral domain signal and extracting a second predefined deep learning feature set from the spatial domain signal. The method also includes converting the extracted first spectral domain feature set into a spatial domain. The method also includes concatenating the second spatial domain feature set with the first feature set converted from the spectral domain to the spatial domain.

[0012] According to one aspect of the present disclosure, there is provided a method for training a neural network using feature expansion. The method includes receiving an input image signal, the input image signal being a spatial domain signal. The method also includes converting the received input image signal to a corresponding spectral domain signal using a neural network. The method also includes extracting a first predefined training feature set from the spectral domain signal and extracting a second predefined training feature set from the spatial domain signal. The method also includes converting the first predefined training feature set extracted from the spectral domain signal to a spatial domain signal using a neural network. The method also includes concatenating the extracted second predefined training feature set and the converted first predefined training feature set. The method also includes calculating a final loss caused during processing of the input image signal using the first loss, the second loss, and the third loss. The method also includes providing feature expansion training to the neural network using the calculated final loss.

[0013] According to one embodiment of the present disclosure, the first loss is computed by performing a first operation, the second loss is computed by performing a second operation, and the third loss is computed by performing a third operation.

[0014] According to one embodiment of the present disclosure, a first operation includes comparing the transformed first predefined training feature set with a predefined spectral domain ground truth associated with the input image signal, and calculating a first loss based on the comparison result.

[0015] According to one embodiment of the present disclosure, the second operation includes extracting a set of reconstructed features from the extracted second predefined training feature set by rescaling pixel values ​​to be independent model outputs, comparing the reconstructed features with the received input image signal and an associated predefined ground truth, and calculating a second loss based on the comparison result.

[0016] According to one embodiment of the present disclosure, the third operation includes blending the set of concatenated learning features, comparing the blended set of learning features with a predefined ground truth associated with the received input image signal, and calculating a third loss based on the comparison result.

[0017] According to one embodiment of the present disclosure, the method includes assigning a first weight value, a second weight value, and a third weight value to the first loss, the second loss, and the third loss, respectively, and further includes calculating a final loss by combining the first loss, the second loss, and the third loss based on the first weight value, the second weight value, and the third weight value, respectively.

[0018] According to one embodiment of the present disclosure, the method includes performing at least one of testing a neural network trained using only the extracted first predefined training feature set or distributing a neural network trained using only the extracted first predefined training feature set.

[0019] According to one embodiment of the present disclosure, the trained neural network is configured to perform video data compression.

[0020] According to one aspect of the present disclosure, a system for global view feature enhancement is provided. The system also includes a memory and at least one processor. The system also includes an image processing engine operatively coupled to the memory. The at least one processor is configured to receive an input signal, the input signal being a spatial domain signal. The at least one processor obtains a spectral domain signal by converting the received input signal into a spectral domain signal. The at least one processor extracts a first predefined deep learning feature set from the corresponding spectral domain signal and extracts a second predefined deep learning feature set from the spatial domain signal. The at least one processor converts the extracted first spectral domain feature set into a spatial domain. The at least one processor concatenates the second spatial domain feature set and the first feature set converted from the spectral domain to the spatial domain.

[0021] According to one aspect of the disclosure, a system for training a neural network using feature expansion is provided. The system also includes a memory and at least one processor. The system also includes an image processing engine operatively coupled to the memory. The at least one processor is configured to receive an input image signal, the input image signal being a spatial domain signal. The at least one processor converts the received input image signal into a corresponding spectral domain signal. The at least one processor extracts a first predefined training feature set from the spectral domain signal and extracts a second predefined training feature set from the spatial domain signal. The at least one processor converts the first predefined training feature set extracted from the spectral domain signal into a spatial domain signal. The at least one processor concatenates the extracted second predefined training feature set and the converted first predefined training feature set. The at least one processor uses the first loss, the second loss, and the third loss to calculate a final loss caused during processing of the input image signal. The at least one processor uses the calculated final loss to provide feature expansion training to the neural network.

[0022] Other aspects, advantages and salient features of the present disclosure will become apparent to those of ordinary skill in the art from the following detailed description, taken in conjunction with the annexed drawings, which disclose various embodiments of the present disclosure. [Brief description of the drawings]

[0023] The foregoing and other features, aspects, and advantages of certain embodiments of the present disclosure will become more apparent from the following description taken in conjunction with the accompanying drawings, in which: [Figure 1] 1 is a block diagram illustration of an electronic device for compressing video content using feature expansion training and low-complexity in-loop filter inference, according to one embodiment of the present disclosure. [Figure 2A]1 is a flowchart illustrating a method for compressing video content using feature expansion training and low-complexity in-loop filter inference, according to one embodiment of the present disclosure. [Figure 2B] 1 is a flowchart illustrating a method for compressing video content using feature expansion training and low-complexity in-loop filter inference, according to one embodiment of the present disclosure. [Diagram 3] FIG. 1 illustrates a system for feature expansion training for compressing video content, according to one embodiment of the present disclosure. [Figure 4] FIG. 2 illustrates a low-complexity in-loop filter inference for compressing video content, according to one embodiment of the present disclosure.

[0024] The same reference numbers are used throughout the drawings to denote the same elements. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0025] The following description with reference to the accompanying drawings is provided to aid in the comprehensive understanding of the various embodiments of the present disclosure as defined by the claims and their equivalents. Although various specific details are included to aid in the understanding, these details should be considered as merely exemplary. Thus, a person of ordinary skill in the art will recognize that various changes and modifications of the various embodiments described in the present disclosure can be made without departing from the scope and spirit of the present disclosure. Furthermore, descriptions of commonly known functions and configurations may be omitted for clarity and conciseness.

[0026] The terms and words used in the following description and claims are not limited to their bibliographical meanings, but are merely used by the inventor to ensure a clear and consistent understanding of the present disclosure. Therefore, it should be apparent to one of ordinary skill in the art that the following description of various embodiments of the present disclosure is provided for illustrative purposes only, and is not intended to limit the present disclosure, as defined by the appended claims and their equivalents.

[0027] The singular forms applicable to the use of "a," "an," and "the" should be understood to include plural references unless expressly stated to the contrary. Thus, for example, reference to a "component surface" includes reference to one or more of such surfaces.

[0028] References throughout this specification to "in one aspect," "in another aspect," or similar language mean that a particular feature, structure, or characteristic described in an embodiment is included in at least one embodiment of the disclosure. Thus, the appearances of the phrases "in one embodiment," "in another embodiment," and similar language throughout this specification may, but do not necessarily, refer to the same embodiment.

[0029] The terms "comprising," "comprising," or any other variations thereof are intended to cover a non-exclusive inclusion, and a process or method including a list of steps does not include only those steps, but also other steps explicitly listed or inherent in such process or method. Similarly, a reference to one or more devices, subsystems, elements, structures, or components described as "comprising..." does not, without further constraints, exclude the presence of other devices, subsystems, elements, structures, or components, or further devices, subsystems, elements, structures, or components.

[0030] The embodiments of the present disclosure, as well as various features and advantageous details thereof, will be more fully described with reference to the non-limiting embodiments illustrated in the accompanying drawings and detailed in the following description. Descriptions of widely known components and processing techniques are omitted so as not to unnecessarily obscure the embodiments of the present disclosure. Furthermore, the various embodiments described in the present disclosure are not necessarily mutually exclusive, but because some embodiments may be combined with one or more other embodiments to form new embodiments. The term "or" as used in the present disclosure means non-exclusive unless otherwise indicated. The examples used in the present disclosure are merely intended to facilitate understanding of how the embodiments of the present disclosure may be implemented and to further enable those of ordinary skill in the art to implement the embodiments of the present disclosure. Thus, the examples are not to be construed as limiting the scope of the embodiments of the present disclosure.

[0031] As is common in the art, the present embodiment may be described and illustrated in terms of described functions or blocks that perform the functions. The blocks, which may be referred to in the present disclosure as units, modules, etc., may be physically embodied by analog or digital circuits, such as logic gates, integrated circuits, microprocessors, microcontrollers, memory circuits, passive electronic components, active electronic components, optical components, hardwired circuits, and may be optionally driven by firmware and software. The circuits may be implemented, for example, in one or more semiconductor chips or on a substrate support, such as a printed circuit board. The circuits constituting the blocks may be embodied by dedicated hardware, by a processor (e.g., one or more programmed microprocessors and associated circuits), or by a combination of dedicated hardware performing some functions of the block and a processor performing other functions of the block. Each block of the present embodiment may be physically separated into two or more interacting and separate blocks without departing from the scope of the present disclosure. Similarly, the blocks of the present embodiment may be physically combined into more complex blocks without departing from the scope of the present disclosure.

[0032] The accompanying drawings are used to help easily understand various technical features, and it should be understood that the embodiments presented in the present disclosure are not limited by the accompanying drawings. As such, the present disclosure should be interpreted to extend to any modifications, equivalents, and replacements other than those specifically presented in the accompanying drawings. Even if terms such as first and second may be used in the present disclosure to describe various elements, the elements are not limited by these terms. These terms are generally used only to distinguish one element from another element.

[0033] Throughout this disclosure, the terms "first predefined deep learning feature set" and "first predefined training feature set" are used interchangeably and mean the same thing. The terms "second predefined deep learning feature set" and "second predefined training feature set" are used interchangeably and mean the same thing.

[0034] Referring now to the drawings, and more particularly to Figures 1, 2A, 2B, 3 and 4, in which like reference characters indicate corresponding features consistently throughout the drawings, there are shown preferred embodiments.

[0035] FIG. 1 illustrates a block diagram of an electronic device 100 for compressing video content using feature expansion training and low-complexity in-loop filter inference, according to one embodiment of the present disclosure.

[0036] Referring to FIG. 1, examples of electronic device 100 include, but are not limited to, a smartphone, a tablet computer, a personal digital assistant (PDA), an Internet of things (IoT) device, a wearable device, and the like.

[0037] In one embodiment of the present disclosure, the electronic device 100 includes a memory 110, a processor 120, a communication unit 130, a display 140, a camera 150, and an image processing engine 160. According to one embodiment of the present disclosure, the electronic device 100 does not include at least one of the communication unit 130, the display 140, the camera 150, and the image processing engine 160, and may include other additional components.

[0038] In one embodiment of the present disclosure, the memory 110 stores a plurality of input image signals, a first predefined deep learning feature set, a second predefined deep learning feature set, a first loss, a second loss, a third loss, a final loss, a first weight value, a second weight value, and a third weight value. The memory 110 stores instructions executed by the processor 120. The memory 110 may also include a non-volatile storage element. Examples of such non-volatile storage elements include magnetic hard disks, optical disks, floppy disks, flash memory, or electrically programmable read only memories (ROMs) (EPROMs) or electrically erasable and programmable ROMs (EEPROMs). Furthermore, the memory 110 may be considered as a non-transitory recording medium, as some examples. The term "non-transitory" may indicate that the recording medium is not embodied by a carrier wave or a propagated signal. However, the term "non-transitory" is not to be construed as meaning that the memory 110 is non-movable. In some examples, memory 110 may be configured to store more information than a memory. In certain examples, the non-transitory storage medium may store data that may change over time (e.g., in a random access memory (RAM) or cache). Memory 110 may be an internal storage unit or may be an external storage unit of electronic device 100, a cloud storage, or any other type of external storage.

[0039] The processor 120 is configured to communicate with the memory 110, the communication unit 130, the display 140, the camera 150, and the image processing engine 160. The processor 120 is configured to execute instructions stored in the memory 110 to perform various processes. The processor 120 may include one or more processors, such as a general-purpose processor, i.e., a central processing unit (CPU), an application processor (AP), a dedicated graphics processing unit, i.e., a graphics processing unit (GPU), a visual processing unit (VPU), and / or a dedicated artificial intelligence (AI) processor, i.e., a neural processing unit (NPU). The processor 120 may include processing circuits such as logic gates, integrated circuits, microprocessors, microcontrollers, memory circuits, passive electronic components, active electronic components, optical components, hardwired circuits, and may be optionally driven by firmware. The circuitry may be embodied, for example, in one or more semiconductor chips, or on a substrate support such as a printed circuit board.

[0040] The communication unit 130 is configured to communicate between internal hardware components, internally, and with external devices (e.g., servers) via one or more networks (e.g., radio technologies). The communication unit 130 includes standard-specific electronic circuitry that enables wired or wireless communication. The display 140 receives user input and may be a liquid crystal display (LCD), light emitting diode (LED), organic light emitting diode (OLED), or other type of display. The user input may include, but is not limited to, touch, swipe, drag, gesture, voice command, etc. The camera 150 includes one or more image sensors (e.g., charged coupled device (CCD), complementary metal-oxide semiconductor (CMOS)) for capturing one or more images / image frames.

[0041] The image processing engine 160 is operatively coupled to the memory 110 and the processor 120 and configured to perform various operations for compressing video content using feature enhancement training and low-complexity in-loop filter inference. Referring to FIG. 1, the image processing engine 160 is illustrated as an independent component according to an embodiment of the present disclosure, but is not limited thereto. According to an embodiment of the present disclosure, the image processing engine 160 may be configured as a module stored in the memory 110. The processor 120 may execute the image processing engine 160 stored in the memory 110 to perform operations of the methods described below.

[0042] In one embodiment of the present disclosure, the image processing engine 160 includes an input module 161, a transformation unit 162, a spatial domain feature learning engine 163, a spectral domain feature learning engine 164, a concatenation module 165, an intelligent ensemble component (IEC) 166, an inference control unit 167, and an artificial intelligence (AI) engine 168.

[0043] The input module 161 receives an input image signal, where the input image signal is a spatial domain signal. The input module 161 transmits the received input image signal to the transform unit 162 for further processing. The input image signal is represented by a first set of dimensions having a particular height and width per input image pixel and one or more first channels (e.g., HXWXC1), where H denotes the tensor height, W denotes the tensor width, and C denotes the number of channels in the tensor.

[0044] The converter 162 converts the received input image signal into a corresponding spectral domain signal using a neural network / AI engine 168. The converted spectral domain signal is represented by a set of second dimensions with specific heights and weights for each image pixel of the input image signal and one or more second channels (e.g., HXWXC2).

[0045] The spatial domain feature learning engine 163 utilizes an AI engine 168 (e.g., a residual neural network (ResNet)) to extract a second predefined training feature set (e.g., a content-specific deep learning feature map, which is a feature map that leads to a desired transformation based on an input typology, such as smooth content versus texture content) from the spatial domain signals and transmits the extracted second predefined training feature set to the concatenation module 165 for further processing. Furthermore, the spatial domain feature learning engine 163 extracts a set of reconstructed features from the extracted second predefined training feature set. The spectral domain feature learning engine 164 utilizes an AI engine 168 (e.g., a ResNet) to extract a first predefined training feature set (e.g., a content-specific deep learning feature map) from the spectral domain signals.

[0046] Then, the transformer 162 transforms the first set of predefined training features extracted from the spectral domain signal into a spatial domain signal. The concatenation module 165 concatenates the extracted second set of predefined training features and the transformed first set of predefined training features. The concatenation process can be, but is not limited to, a simple method arranged along multiple channels or as complex as using a deep learning block (e.g., AI engine 168). Here, the concatenated signal is represented by the image pixels of the input image signal and a third set of dimensions with specific heights and weights for one or more third channels (e.g., HXWXC3). Furthermore, the concatenation module 165 transmits the concatenated learning features to the intelligent ensemble component (IEC) 166 for further processing.

[0047] When the Intelligent Ensemble Component (IEC) 166 receives the concatenated learning features from the concatenation module 165, it performs a blending operation to blend the concatenated learning features to generate a blended learning feature / blended signal. The blending process can be as simple as blending using classical methods at the pixel level, or as complex as using deep learning blocks for blending, but is not limited thereto. The blended signal is represented by a third dimension set with specific height and weight values ​​for the image pixels of the input image signal and one or more third channels (e.g., HXWXC4).

[0048] Furthermore, the intelligent ensemble component (IEC) 166 uses the first loss, the second loss, and the third loss to calculate a final loss caused during processing of the input image signal, where the first loss is computed by performing a first operation, the second loss is computed by performing a second operation, and the third loss is computed by performing a third operation.

[0049] The first operation includes comparing the transformed first predetermined training feature set with a predefined spectral domain ground truth associated with the input image signal, and calculating a first loss based on the comparison result, where the predefined spectral domain ground truth is a first reference signal for comparison.

[0050] The second operation includes extracting a set of reconstructed features from the extracted second set of predefined training features by rescaling pixel values ​​to be an independent model output used in the inference stage, comparing the extracted set of reconstructed features with a predefined ground truth associated with the received input image signal, and calculating a second loss based on the comparison result. The predefined ground truth is a second reference signal for comparison.

[0051] A third operation includes blending the set of concatenated training features, comparing the blended set of training features with a predefined ground truth associated with the received input image signal, and calculating a third loss based on a comparison result.

[0052] Further, the intelligent ensemble component (IEC) 166 assigns a first weight, a second weight, and a third weight to the first loss, the second loss, and the third loss, respectively, and then the intelligent ensemble component (IEC) 166 calculates a final loss by combining the first loss, the second loss, and the third loss based on the first weight, the second weight, and the third weight, respectively.

[0053] The inference controller 167 performs backpropagation during model training so that the learning from the spectral domain feature learning engine 164 and the learning from the intelligent ensemble components (IEC) 166 also influence the performance of the spatial domain feature learning engine 163, allowing the spatial domain feature learning engine 163 to learn beyond its own capabilities. The inference controller 167 performs testing of the neural network trained using only the extracted first predefined training feature set (referred to as, but not limited to, separating the spectral streams and ensemble components) and / or distributing the neural network trained using only the extracted first predefined training feature set.

[0054] In one embodiment of the present disclosure, the inference controller 167 uses the final loss obtained from the intelligent ensemble component (IEC) 166 and low-complexity in-loop filter inference to train the model by separating the spectral stream and ensemble components of the system, greatly reducing the solution complexity during inference (and reducing memory put-print since the system does not need to consider the feature weights of the spectral stream and ensemble components). The inference controller 167 then distributes the trained model (e.g., deep neural network / AI engine 168) by using only the first predefined training feature set extracted from the corresponding first spatial domain signal.

[0055] Various hardware components and associated functions of the electronic device 100 may be performed via the non-volatile memory, the volatile memory, and the processor 120. The one or more processors control the processing of input data by predefined operating rules or AI models (i.e., the AI ​​engine 168) stored in the non-volatile memory and the volatile memory. The predefined operating rules or AI models are provided via training or learning. Providing via learning means that a predefined operating rule or AI model with desired characteristics is created by applying a learning algorithm to a plurality of learning data. The learning may be performed in the device itself in which the AI ​​according to the present embodiment is performed, and / or may be embodied via a separate server / system. A learning algorithm is a method of training a predetermined target device (e.g., a robot) using a plurality of learning data to induce, allow, or control the target device to make decisions or predictions. Examples of the learning algorithm include, but are not limited to, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning.

[0056] The AI ​​model may be composed of multiple neural network layers. Each layer has multiple weights, and the layer is calculated through the calculation of the previous layer and the calculation of the multiple weights. Examples of neural networks include, but are not limited to, convolutional neural networks (CNNs), deep neural networks (DNNs), recurrent neural networks (RNNs), restricted Boltzmann machines (RBMs), deep belief networks (DBNs), bidirectional recurrent deep neural networks (BRDNNs), generative adversarial networks (GANs), and deep Q-networks.

[0057] 1 illustrates exemplary hardware components of an electronic device 100 according to one embodiment, but is not limited thereto. According to one embodiment of the present disclosure, the electronic device 100 may include fewer or more components. Moreover, the labels or names of the components are used for illustrative purposes only and are not intended to limit the scope of the present disclosure. One or more components may be combined to perform the same or substantially similar functions of compressing video content using feature enhancement training and low-complexity in-loop filter inference.

[0058] FIG. 2A is a flow chart illustrating a method (200A) for compressing video content using feature expansion training and low-complexity in-loop filter inference, according to one embodiment of the present disclosure.

[0059] Referring to FIG. 2A, operations 201A through 205A are performed by electronic device 100 to compress video content using feature expansion training and low-complexity in-loop filter inference.

[0060] In operation 201A, the method (200A) includes receiving an input signal, the input signal being a spatial domain signal. In operation 202A, the method (200A) includes obtaining a spectral domain signal by converting the received input signal into a spectral domain signal. In operation 203A, the method (200A) includes extracting a first predefined deep learning feature set from the corresponding spectral domain signal and extracting a second predefined deep learning feature set from the spatial domain signal. In operation 204A, the method (200A) includes converting the extracted first spectral domain feature set into a spatial domain. In operation 205A, the method (200A) includes concatenating the second spatial domain feature set with the first feature set converted from the spectral domain to the spatial domain.

[0061] FIG. 2B is a flow chart illustrating a method (200B) for compressing video content using feature expansion training and low-complexity in-loop filter inference, according to one embodiment of the present disclosure.

[0062] Referring to FIG. 2B, operations 201B through 207B are performed by electronic device 100 to compress video content using feature expansion training and low-complexity in-loop filter inference.

[0063] In act 201B, the method (200B) includes acquiring an input image signal, where the input image signal is a spatial domain signal, which is related to act 201A of FIG. 2A. For example, the input module 161 receives an input image signal, where the input image signal is a spatial domain signal. In act 202B, the method (200B) includes converting the acquired input image signal into a corresponding spectral domain signal, which is related to act 202A of FIG. 2A. For example, the transform unit 162 converts the received input image signal into the corresponding spectral domain signal using the neural network / AI engine 168. In act 203B, the method (200B) includes extracting a first set of predefined training features from the corresponding spectral domain signal and extracting a second set of predefined training features from the spatial domain signal, which is related to act 203A of FIG. 2A. For example, the spatial domain feature learning engine 163 utilizes the AI ​​engine 168 to extract a second set of predefined training features from the spatial domain signals, and the spectral domain feature learning engine 164 utilizes the AI ​​engine 168 to extract a first set of predefined training features from the spectral domain signals. In act 204B, the method (200B) includes converting the first set of predefined training features extracted from the spectral domain signals to spatial domain signals. For example, the transform unit 162 then converts the first set of predefined training features extracted from the spectral domain signals to spatial domain signals, which is involved in act 204A of FIG. 2A.

[0064] In operation 205B, the method (200B) includes concatenating the extracted second predefined training feature set and the transformed first predefined training feature set. For example, the concatenation module 165 concatenates the extracted second predefined training feature set and the transformed first predefined training feature set, which corresponds to operation 205A of FIG. 2A. In operation 206B, the method (200B) includes calculating a final loss caused during processing of the input image signal using the first loss, the second loss, and the third loss, which corresponds to operation 205A of FIG. 2A. For example, the intelligent ensemble component (IEC) 166 calculates a final loss caused during processing of the input image signal using the first loss, the second loss, and the third loss. In operation 207B, the method (200B) includes providing feature expansion training to the neural network using the calculated final loss, which corresponds to operation 205A of FIG. 2A. For example, the inference control unit 167 may perform testing of a neural network trained using only the extracted first default training feature set and / or distributing a neural network trained using only the extracted first default training feature set.

[0065] The various actions, acts, blocks, operations, etc. in the methods may be performed in the order presented, in a different order, or simultaneously. In accordance with an embodiment of the present disclosure, some of the actions, acts, blocks, operations, etc. may be omitted, added, modified, skipped, etc. without departing from the scope of the present disclosure.

[0066] The disclosed method (200A and 200B) includes various stages for compressing video content using feature expansion training and low-complexity in-loop filter inference as illustrated in FIG. 3. Stage-1 is for feature extraction and expansion using multi-stream (e.g., spatial stream, spectral stream, etc.) and multi-domain networks (e.g., spatial domain, spectral domain, etc.) leading to simultaneous local and global input representations. The local input representations include, but are not limited to, local convolutions of the input image signal. The global input representations include, but are not limited to, a transformation leading to each pixel containing information related to the global image, such that a convolution on such transformed input provides information related to the global image. Stage-2 is for blending and merging the original features (e.g., first set of predefined training features, second set of predefined training features, etc.) and the expanded features (e.g., concatenated training features). Stage-3 is for a multi-loss approach where various stages contribute different losses (e.g. loss-1, loss-2, loss-3), resulting in feature augmentation training. Stage-4 is for reducing complexity by isolating parts of the overall pipeline (grey part) during inference.

[0067] FIG. 3 illustrates a system 300 for feature expansion training for compressing video content, according to one embodiment of the present disclosure.

[0068] Referring to FIG. 3, operations 301 through 317 are performed by electronic device 100 to compress video content using feature expansion training and low-complexity in-loop filter inference.

[0069] In operation 301-302, the system 300 includes receiving an input image signal (e.g., image frame 301) by the input module 161, where the input image signal is a spatial domain signal. The input image signal is represented by a first set of dimensions having a particular height and width for input image pixels and one or more first channels (e.g., HXWXC1). The converter 162 (not shown in FIG. 3) then converts the received input image signal into a corresponding spectral domain signal. Furthermore, the converter 162 transmits the received input image signal to the spatial domain feature learning engine 163 for further processing.

[0070] In operations 303-304, the system 300 includes extracting, using the AI ​​engine 168, a second set of predefined training features from the spatial domain signal by the spatial domain feature learning engine 163. The spatial domain feature learning engine 163 then transmits the extracted second set of predefined training features to the concatenation module 165 for further processing. In operation 305, the system 300 includes extracting, by the spatial domain feature learning engine 163, a set of reconstructed features from the extracted second set of predefined training features.

[0071] In operation 306, the system 300 includes converting the received input image signal into a corresponding spectral domain signal by the converter 162. The converted spectral domain signal is represented by a second set of dimensions with specific heights and weights for the image pixels of the input image signal and one or more second channels (e.g., HXWXC2). The converter 162 then transmits the converted spectral domain signal to the spectral domain feature learning engine 164 for further processing. In operation 307-308, the system 300 includes extracting a first set of predefined training features from the spectral domain signal by the spectral domain feature learning engine 164 using the AI ​​engine 168. Furthermore, the converter 162 then converts the first set of predefined training features extracted from the spectral domain signal into a spatial domain signal, whereby the extracted spectral domain signal is transmitted to the concatenation module 165 for further processing.

[0072] In operation 309, the system 300 includes concatenating the extracted second set of predefined training features with the transformed first set of predefined training features by the concatenation module 165. The concatenation module 165 then generates a concatenated signal (concatenated training features) represented by a third set of dimensions having specific heights and weights for the image pixels of the input image signal and one or more third channels (e.g., HXWXC3). The concatenation module 165 further transmits the concatenated training features to an intelligent ensemble component (IEC) 166 for further processing. In operation 310, the method includes performing a blending operation by an ensemble information combiner of the intelligent ensemble component (IEC) 166 when receiving the concatenated learning features from the concatenation module 165, and generating a blended learning feature (e.g., HXWXC4) by the ensemble information combiner by blending the concatenated learning features.

[0073] In operations 311-312, the system 300 includes determining a first loss (i.e., Loss-1) by the intelligent ensemble component (IEC) 166. The first loss is computed by performing a first operation, which includes comparing the transformed first predefined training feature set with a predefined spectral domain ground truth associated with the input image signal, and calculating the first loss based on the comparison result. In operations 313-314, the system 300 includes determining a second loss (i.e., Loss-2) by the intelligent ensemble component (IEC) 166. The second loss is computed by performing a second operation, which includes comparing the extracted reconstructed feature set with a predefined ground truth associated with the received input image signal, and calculating the second loss based on the comparison result. In operation 315, the system 300 includes determining, by the intelligent ensemble component (IEC) 166, a third loss (i.e., Loss-3), which is computed by performing a third operation including blending the set of concatenated learning features, comparing the blended set of learning features with a predefined ground truth associated with the received input image signal, and calculating a third loss based on the comparison result.

[0074] In operation 316, the system 300 includes determining, by the intelligent ensemble component (IEC) 166, a final loss caused during processing of the input image signal using the first loss, the second loss, and the third loss. In one embodiment of the present disclosure, the final loss is determined by performing a number of computational operations (e.g., summing, assigning specific weights, etc.). In one embodiment of the present disclosure, the intelligent ensemble component (IEC) 166 assigns a first weight, a second weight, and a third weight to the first loss, the second loss, and the third loss, respectively. The intelligent ensemble component (IEC) 166 determines the final loss by combining the first loss, the second loss, and the third loss based on the first weight, the second weight, and the third weight, respectively. In operation 317, the system 300 includes storing, by the intelligent ensemble component (IEC) 166, the determined final loss for backpropagation that may be used in the inference stage. All losses are designed in a way that they are based on both streams and allow separable stream functions and backpropagation.

[0075] FIG. 4 illustrates a low-complexity in-loop filter inference for compressing video content, according to one embodiment of the present disclosure.

[0076] Referring to FIG. 4, consider a scenario in which the electronic device 100 detects the inference stage / mode opening when it receives codec data (e.g., compressed or decompressed media files such as songs or videos) from its codec inference (401). The codec inference (401) communicates the codec data to the inference control unit 167. The inference control unit 167 then decouples the spectral streams 404 and 405 represented by the inactive clocks / inactive paths and the ensemble components 406 while keeping only the spatial streams 402 and 403 represented by the active blocks / active paths. The inference control unit 167 improves the quality of the codec data based on a final loss calculated using only the spatial streams 402 and 403, also known as the in-loop filter. As a result, the disclosed method / system reduces the complexity during the inference stage / mode (and also reduces memory put print since the system 300 no longer needs to consider the feature weights of the spectral streams and the ensemble components).

[0077] Unlike existing methods and systems, the disclosed method (200A and 200B) and / or system 300 provides feature expansion training to a neural network (NN) (e.g., AI engine 168) based on the determined final loss. The final loss is determined by processing a plurality of input image signals by performing a plurality of computational operations based on a first predefined training feature set, a second predefined training feature set, and a concatenated training feature blending operation. Furthermore, the disclosed method (200A and 200B) and / or system 300 uses the disclosed / novel technique for in-loop filter inference using a neural network (NN) that reduces the complexity through feature expansion training and includes novel features such as an amalgamation module (i.e., ensemble information merger) and a separable model (i.e., inference control unit 167) for improving the video codec output, resulting in better performance of embedded systems such as smartphones. The feature expansion training enabled by combining multiple losses in a novel manner is useful for reducing model complexity during inference, and thus helps to avoid making the neural network (NN) model deeper to achieve better performance. As a result, the disclosed method (200A and 200B) and / or system 300 continue to extract features quickly during in-loop filter inference, achieving better image quality enhancement with lower complexity, without using deeper architectures of neural network (NN) models that would otherwise be required for convolution to have a larger receptive field. Furthermore, encoding the above selection method in the beat stream will assist the decoder in adopting the same method as the encoder.

[0078] Unlike existing methods and systems, the disclosed methods (200A and 200B) provide various technical advantages. The disclosed system 300 and / or electronic device 100 may, for example, not employ deeper architectures of neural networks (NNs) that would otherwise be required for convolution to have a larger receptive field, and may extract features faster and achieve better image quality enhancement with reduced complexity of the processing pipeline for video compression. The disclosed system 300 and / or electronic device 100 may reduce memory requirements as well as processing time while producing better results, resulting in better performance in embedded systems or electronic devices 100. The disclosed system 300 and / or electronic device 100 may produce better results not only in terms of metrics, but also in terms of subjective quality.

[0079] According to an embodiment of the present disclosure, there is provided a method for training a neural network using feature expansion. The method includes obtaining an input image signal. The input image signal is a spatial domain signal. The method also includes converting the received input image signal to a corresponding spectral domain signal. The method also includes extracting a first predefined training feature set from the spectral domain signal and extracting a second predefined training feature set from the spatial domain signal. The method also includes converting the first predefined training feature set extracted from the spectral domain signal to a spatial domain signal. The method also includes concatenating the extracted second predefined training feature set and the converted first predefined training feature set. The method also includes calculating a final loss based on the first loss, the second loss, and the third loss. The method also includes training the neural network using the calculated final loss.

[0080] According to one embodiment of the present disclosure, the method also includes calculating a first loss based on the transformed first predefined training feature set and a predefined spectral domain ground truth associated with the input image signal.

[0081] According to one embodiment of the present disclosure, the method also includes extracting a set of reconstructed features from the extracted second set of predefined training features. The method also includes calculating a third loss based on the blended set of training features and a predefined ground truth associated with the received input image signal.

[0082] According to one embodiment of the present disclosure, the method also includes blending the set of concatenated training features, and calculating a third loss based on the blended set of training features and a predefined ground truth associated with the received input image signal.

[0083] According to one embodiment of the present disclosure, the final loss may be determined based on a weighted sum of the first weight value, the second weight value, and the third weight value.

[0084] According to one embodiment of the present disclosure, the method includes performing at least one of testing a neural network trained using only the extracted first predefined training feature set or distributing a neural network trained using only the extracted first predefined training feature set.

[0085] According to one embodiment of the present disclosure, the trained neural network is configured to perform video data compression.

[0086] According to one embodiment of the present disclosure, there is provided a computer readable recording medium having a program executable by a process recorded thereon to perform any one of the methods described above.

[0087] According to an embodiment of the present disclosure, an electronic device for training a neural network using feature expansion is provided. The electronic device also includes a memory and at least one processor. The at least one processor is configured to acquire an input image signal, the input image signal being a spatial domain signal. The at least one processor may be configured to convert the received input image signal into a corresponding spectral domain signal. The at least one processor may be configured to extract a first predefined training feature set from the spectral domain signal and extract a second predefined training feature set from the spatial domain signal. The at least one processor may be configured to convert the first predefined training feature set extracted from the spectral domain signal into a spatial domain signal. The at least one processor may be configured to concatenate the extracted second predefined training feature set and the converted first predefined training feature set. The at least one processor may be configured to calculate a final loss based on the first loss, the second loss, and the third loss. The at least one processor may be configured to train the neural network using the calculated final loss.

[0088] According to one embodiment of the present disclosure, the at least one processor may be configured to calculate a first loss based on the transformed first predefined training feature set and a predefined spectral domain ground truth associated with the input image signal.

[0089] According to an embodiment of the present disclosure, the at least one processor may be configured to extract a set of reconstructed features from the extracted second set of predefined training features, and the at least one processor may be configured to calculate a second loss based on the extracted set of reconstructed features and a predefined ground truth associated with the received input image signal.

[0090] According to an embodiment of the present disclosure, the at least one processor may be configured to blend the set of concatenated training features, and the at least one processor may be configured to calculate a third loss based on the blended set of training features and a predefined ground truth associated with the received input image signal.

[0091] According to one embodiment of the present disclosure, the final loss is determined based on a weighted sum of the first weight value, the second weight value and the third weight value.

[0092] According to one embodiment of the present disclosure, the at least one processor may be configured to perform at least one of testing the neural network trained using only the extracted first predefined training feature set or distributing the neural network trained using only the extracted first predefined training feature set.

[0093] According to one embodiment of the present disclosure, the trained neural network is configured to perform video data compression.

[0094] Unless otherwise defined, all terms (including technical and scientific terms) used in this disclosure have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs. The systems, methods, and examples provided in this disclosure are illustrative and are not intended to be limiting.

[0095] Although specific language expressions have been used to describe the subject matter of the present invention, any limitations arising therefrom are not intended. As would be apparent to one of ordinary skill in the art, various operational modifications may be made to the method to embody the inventive concepts as taught in this disclosure. The drawings and the foregoing description provide examples of one or more embodiments. One of ordinary skill in the art would understand that one or more of the described elements may be preferably combined into a single functional element. Alternatively, a particular element may be divided into multiple functional elements. An element from one embodiment may be added to another embodiment.

[0096] One or more embodiments disclosed in this disclosure may be implemented using at least one hardware device to perform network management functions and control elements.

[0097] The foregoing description of one or more specific embodiments fully illustrates the general nature of one or more embodiments in the present disclosure, such that others may, by applying current knowledge, easily modify and / or adapt such one or more specific embodiments for various applications without departing from the general concept, and therefore, such adaptations and modifications must and are intended to be understood within the meaning and range of equivalents of the one or more disclosed embodiments. It should be understood that the phraseology or terminology employed in this disclosure is for purposes of description and not of limitation.

[0098] Although the present disclosure has been shown and described with reference to one or more embodiments, it will be understood by those of ordinary skill in the art that various changes in form and detail can be made therein without departing from the spirit and scope of the disclosure, as defined by the appended claims and their equivalents.

Claims

1. A method (200B) performed by the electronic device 100 for training a neural network using feature augmentation, comprising: acquiring an input image signal, the input image signal being a spatial domain signal (201B); converting (202B) the acquired input image signal into a corresponding spectral domain signal; Extracting (203B) a first set of predefined training features from the spectral domain signal and a second set of predefined training features from the spatial domain signal; Transforming (204B) the first set of predefined training features extracted from the spectral domain signal into the spatial domain signal; Concatenating the extracted second set of predefined training features with the transformed first set of predefined training features (205B); calculating (206B) a final loss based on the first loss, the second loss, and the third loss; and training (207B) the neural network using the calculated final loss.

2. 2. The method of claim 1, further comprising: calculating the first loss based on the transformed first predefined training feature set and a predefined spectral domain ground truth associated with the input image signal.

3. extracting a set of reconstructed features from the extracted second set of predefined training features; 2. The method of claim 1, further comprising: calculating the second loss based on the set of extracted reconstructed features and a predefined ground truth associated with the acquired input image signal.

4. blending the set of concatenated learning features; 2. The method of claim 1, further comprising: calculating the third loss based on the blended set of learning features and a predefined ground truth associated with the acquired input image signal.

5. 2. The method of claim 1, wherein the final loss is determined based on a weighted sum of the first weight value, the second weight value, and the third weight value.

6. testing the trained neural network using only the extracted first predefined training feature set; or 2. The method of claim 1, further comprising: performing at least one of: distributing the trained neural network using only the extracted first predefined learning feature set.

7. 2. The method of claim 1, wherein the trained neural network is configured to perform video data compression.

8. 1. An electronic device 100 for training a neural network using feature augmentation, A memory 110 and at least one processor 120, The at least one processor 120 is obtaining an input image signal, the input image signal being a spatial domain signal; Converting the acquired input image signal into a corresponding spectral domain signal; Extracting a first set of predefined training features from the spectral domain signal and extracting a second set of predefined training features from the spatial domain signal; Transforming the first set of predefined training features extracted from the spectral domain signal into the spatial domain signal; Concatenating the extracted second set of predefined training features with the transformed first set of predefined training features; Calculate the final loss based on the first loss, the second loss, and the third loss; The electronic device 100 is configured to train the neural network using the calculated final loss.

9. The at least one processor 120 is 9. The electronic device 100 of claim 8, configured to calculate the first loss based on the transformed first predefined training feature set and a predefined spectral domain ground truth associated with the input image signal.

10. The at least one processor 120 is Extracting a set of reconstructed features from the extracted second set of predetermined training features; 10. The electronic device 100 of claim 8, configured to calculate the second loss based on the set of extracted reconstructed features and a predefined ground truth associated with the acquired input image signal.

11. The at least one processor 120 is Blending the set of concatenated learning features; 10. The electronic device 100 of claim 8, configured to calculate the third loss based on the blended set of learning features and a predefined ground truth associated with the acquired input image signal.

12. The electronic device of claim 8 , wherein the final loss is determined based on a weighted sum of the first weighted value, the second weighted value, and the third weighted value.

13. The electronic device of claim 8 , wherein the trained neural network is configured to perform video data compression.

14. The at least one processor 120 is testing the trained neural network using only the extracted first set of predefined training features; or 14. The electronic device 100 of claim 8, configured to perform at least one of the following: distributing the trained neural network using only the extracted first predefined learning feature set.

15. A computer readable recording medium having recorded thereon a program executable by a process to perform the method according to any one of claims 1 to 7.

Images