Distributed computing system and method of artificial neural network

Abstract

According to an example of the present disclosure, an NPU (Neural Processing Unit) capable of performing encoding is provided. The NPU includes at least one processing element (PE) that performs calculations for a plurality of layers within an artificial neural network to generate a plurality of output feature maps; and an encoder for encoding at least one arbitrary output feature map from the plurality of output feature maps in a bit stream and then transmitting the encoded output feature map.

Description

Technical Field

[0001] The present disclosure relates to a distributed computing system and method for artificial neural networks. Background Art

[0002] With the continuous development of the information and communication industry, high-definition (HD) resolution broadcasting services have become popular around the world.

[0003] For this reason, many users have become accustomed to high-resolution images and / or videos. Therefore, the demand for higher image quality, i.e., high-resolution and high-quality videos / videos such as ultra-high definition (UHD) images / videos above 4K or 8K, is increasing in many fields.

[0004] To this end, the amount of image / video data transmission has increased rapidly, and technology that can reduce the amount of data transmission is needed.

[0005] The technology for encoding these video data is accomplished through the 2013 standard technology High Efficiency Video Coding (HEVC). HEVC is the next generation video compression technology, which has higher compression rate and lower complexity than the previous H.264 / AVC technology. HEVC is a technology that can effectively compress the huge data of HD and UHD videos.

[0006] HEVC performs block-unit encoding like previous compression standards. However, HEVC differs from H.264 / AVC in that it has only one profile. The core encoding technologies included in the only profile of HEVC are a total of 8 areas. Specifically, the encoding technologies are hierarchical encoding structure technology, transformation technology, quantization technology, intra-picture prediction encoding technology, inter-picture motion prediction technology, entropy encoding technology, loop filtering technology, and other technologies.

[0007] Since the HEVC video codec was established in 2013, realistic video and virtual reality services using 4K and 8K video images have continued to expand. As a result, the Versatile Video Coding (VVC) standard was developed as a next-generation video codec that improves performance more than twice that of HEVC. VVC is called H.266.

[0008] The development goal of H.266 (VVC) is to improve the efficiency of the previous generation codec H.265 (HEVC) by more than two times. VVC was originally developed with 4K or higher resolution in mind, but due to the expansion of the virtual reality (VR) market, 16K-level ultra-high-resolution video processing was developed for the purpose of corresponding to 360-degree video. In addition, due to the development of display technology, the HDR market is expanding. To cope with this, the display supports 10-bit color depth of course, but also 16-bit color depth, and supports brightness performance of 1000 nits, 4000 nits, and 10,000 nits. In addition, since it was developed considering the VR market and the 360-degree video market, some frame rates in the range of 0 to 120FPS are supported.

[0009] <Development of Artificial Intelligence> Artificial intelligence (AI) is also developing gradually. AI is human intelligence, which means artificial imitation of intelligence that can perform recognition, classification, inference, prediction, control / decision making, etc.

[0010] Due to the development of artificial intelligence technology and the increase in IoT devices, the traffic between machines is exploding. In particular, the traffic of AI for video analysis performed by machines is exploding. In particular, the business execution of machines can make human intervention unnecessary or minimized. Therefore, using artificial intelligence technology, various repetitive tasks can be handled efficiently with minimal personnel.

[0011] In AI video analysis technology, object recognition technology and image classification technology both use images as input. In other words, the development of AI video analysis technology is becoming the main reason for the explosive growth of data transmission between electronic devices.

[0012] As a result, the number of images that need to be analyzed by machines is growing exponentially, and the problem of increased load and power consumption on communication network channels and servers is growing. Therefore, a lot of research is being done on image coding techniques to enable machines to perform reasoning.

[0013] On the one hand, traditional encoding technology, as a technology for optimizing image compression, focuses on improving the compression rate while maintaining the quality of the video based on the audience's visual perception.

[0014] Therefore, the Moving Picture Experts Group (MPEG) of the International Organization for Standardization of Multimedia Coding is discussing the Video Coding for Machine (VCM) technology. VCM is an encoding and decoding technology for images or feature maps that considers the reasoning performance and compression rate of the machine rather than the human visual standard. Summary of the invention

[0015] Technical issues The inventors of the present disclosure recognize that an artificial neural network model has structural features including multiple layers and branch points. Each layer of the artificial neural network model receives an input feature map and a weight and applies convolution, matrix multiplication, or an activation function to output an output feature map. The artificial neural network model has the characteristic of using the output value corresponding to one layer as the input value of the next layer. Here, each layer of the artificial neural network model may include a feature map containing information corresponding to a specific machine task. Therefore, the feature map of a specific layer may contain more useful information for a specific machine task than the feature maps of other layers.

[0016] The inventors of the present invention have noticed that the calculation of the artificial neural network model is performed layer by layer. In particular, the inventors of the present invention have recognized that the size of the parameters and the amount of calculation of the first layer and the second layer of a specific artificial neural network model have an extreme difference. In addition, the inventors of the present invention have also recognized that in the case of a specific layer, even if the parameter size of the input feature map and the parameter size of the weight are large, the parameter size of the output feature map may be very small.

[0017] Also, the inventors of the present invention have recognized that the output of the first layer of the artificial neural network model processed by the first processor is passed to the second processor so that the second processor can process the calculation of the second layer. In other words, the inventors of the present invention have recognized the fact that the calculation of each layer of the artificial neural network model having multiple layers can be processed by multiple processors in a distributed manner. In addition, the inventors of the present invention have recognized the fact that when the output feature map of a specific layer processed by the first processor is sent to the second processor, the second processor can continue to perform the calculation.

[0018] In particular, in the case of a particular artificial neural network model, the size of the output feature map of a particular layer may be very small. In these cases, the inventors of the present disclosure recognize that when a small output feature map is transmitted, the amount of data transmission can be minimized.

[0019] In addition, the feature map of a specific layer of a specific artificial neural network model may include intermediate information required for the specific artificial neural network inference calculation. Therefore, the inventors of the present disclosure recognize the fact that in order for the feature map of a specific layer to be inferred and calculated by the specific artificial neural network model, it can be sent to the assigned processor.

[0020] Therefore, the present disclosure is directed to providing a processor that can efficiently perform image analysis by a machine and a system including the processor.

[0021] Technical Solution In order to achieve the above object, according to an example of the present disclosure, a neural processing unit (NPU) capable of performing encoding is provided. The NPU includes at least one processing element (PE) that performs calculations for multiple layers in an artificial neural network to generate multiple output feature maps; and an encoder that encodes at least one arbitrary output feature map from the multiple output feature maps in a bit stream for transmission.

[0022] The encoder may select at least one arbitrary output feature map from the plurality of output feature maps based on arbitrary information.

[0023] The arbitrary information may be received from a server or other NPU that receives the bitstream.

[0024] The arbitrary information may include identification information of the at least one arbitrary output feature map or identification information of a layer corresponding to the arbitrary output feature map.

[0025] The arbitrary information is generated based on at least one of size information of the output feature map, information of a machine task performed by other neural processing units receiving the bit stream, and information of an artificial neural network model calculated by other neural processing units receiving the bit stream.

[0026] The bitstream is transmitted in order for other NPUs to perform calculations for subsequent layers within the artificial neural network.

[0027] The bitstream may include an identifier of the arbitrary output feature map. Also, the bitstream may be divided into a header and a payload.

[0028] In order to achieve the above-mentioned purpose, according to an example of the present disclosure, a neural processing unit (NPU) capable of performing decoding is provided. The NPU includes: a decoder that decodes the received bit stream and restores it to at least one arbitrary feature map; and at least one processing element (PE) that performs calculations for an artificial neural network model. The restored at least one arbitrary feature map can be an output feature map of any layer among multiple layers within the artificial neural network. The at least one processing element uses the restored at least one arbitrary feature map as an input feature map of a subsequent layer of the arbitrary layer.

[0029] The NPU may transmit any information that the arbitrary layer or the at least one feature map can be selected by other neural processing units to which the bitstream is transmitted.

[0030] The arbitrary information may include identification information of the at least one arbitrary feature map or identification information of a layer corresponding to the at least one arbitrary feature map.

[0031] The arbitrary information is generated based on at least one of size information of a feature map, information of a machine task executed in the neural processing unit, and information of the artificial neural network model.

[0032] The bitstream may include an identifier of the arbitrary output feature map.The bitstream may be distinguished into a header and a payload.

[0033] In addition, in order to achieve the above-mentioned purpose, according to an example of the present disclosure, a system is provided. The system may include: a first neural processing unit configured to process a first layer interval of an artificial neural network model including multiple layers; and a second neural processing unit configured to process a second layer interval of the artificial neural network model. The first neural processing unit may be configured to input an output feature map of the last layer of the first layer interval to the second neural processing unit.

[0034] The magnitude of the sum of the weights of the layers in the first layer interval may be relatively smaller than the magnitude of the sum of the weights of the layers in the second layer interval.

[0035] There are multiple first neural processing units, and the second neural processing unit can be configured to process each of the output feature maps output from the first neural processing unit in sequence.

[0036] The second neural processing unit can be configured at least to reuse weights corresponding to the number of the first neural processing units.

[0037] The size of the first memory of the first neural processing unit may be relatively smaller than the second memory of the second neural processing unit.

[0038] The first layer interval and the second layer interval of the artificial neural network model can be determined based on the size of the output feature map of a specific layer.

[0039] The first layer interval and the second layer interval of the artificial neural network model may be determined based on the size of the sum of weights of the respective layer intervals.

[0040] The magnitude of the first weight of the first layer interval of the artificial neural network model may be relatively smaller than the magnitude of the second weight of the second layer interval.

[0041] Beneficial Effects According to the present disclosure, the artificial neural network model is based on the size of the output feature map of a specific layer of the artificial neural network model, and can be distributedly processed by heterogeneous processors. At this time, the amount of data transmission between heterogeneous processors can be minimized.

[0042] According to the present disclosure, the artificial neural network model is based on the size of the input feature map of a specific layer of the artificial neural network model, and can be distributedly processed by heterogeneous processors. At this time, the amount of data transmission between heterogeneous processors can be minimized.

[0043] According to the NPU proposed in the present disclosure, through the decentralized processing of artificial neural networks, the edge-type NPU can reduce manufacturing costs and power consumption, and the server-type NPU can reduce computing load. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 is a diagram schematically illustrating an example of a video / image encoding system.

[0045] Figure 2 is a diagram schematically showing the configuration of a video / image encoding device.

[0046] Figure 3 is a diagram schematically showing the configuration of a video / image decoding device.

[0047] Figure 4 is a conceptual diagram schematically illustrating a neural processing unit according to the present disclosure.

[0048] Figure 5a is a conceptual diagram schematically illustrating one processing element among a plurality of processing elements applicable to the present disclosure.

[0049] Figure 5b is a conceptual diagram schematically illustrating an SFU applicable to the present disclosure.

[0050] Figure 6 It is shown Figure 4 An example diagram of a variation of neural processing unit 100 is shown in FIG.

[0051] Figure 7 is a conceptual diagram schematically illustrating an exemplary artificial neural network model.

[0052] Figure 8a It is a diagram illustrating the basic structure of a convolutional neural network.

[0053] Figure 8b is a comprehensive diagram showing the operation of a convolutional neural network for ease of understanding.

[0054] Figures 9a to 9d An example diagram of an NPU including a VCM encoder and an NPU including a VCM decoder.

[0055] Fig.10a and 10b is an example diagram showing the location of a bit stream in an artificial neural network model.

[0056] Fig.11a 2 is a diagram showing a VCM encoder for compressing a feature map and transmitting a bit stream, and a VCM decoder for restoring the received bit stream.

[0057] Fig.11b It is shown Fig.11a An example of generating feature maps P2 to P5 shown in .

[0058] Fig.11c It shows the detailed display Fig.11a An example diagram of an MSFF execution unit is shown in FIG.

[0059] Fig.12 It is shown Fig.11a An example diagram of a modification of .

[0060] Fig.13a is a conceptual diagram illustrating an example of distributed processing between an edge device and a cloud server according to an example of the present disclosure based on the data size of each layer in an exemplary artificial neural network model.

[0061] Fig.13b It is shown Fig.13a An exemplary table of data size, etc. for each layer in the exemplary artificial neural network model shown in .

[0062] Fig.14a yes Fig.13a Example diagram of feature maps of the 9th to 13th layers of the artificial neural network model shown.

[0063] Fig.14b It shows that according to Fig.14aAn example diagram of the feature maps P9 to P12 shown in FIG. 1 , which are compressed, transmitted to a VCM encoder of a bit stream, and a VCM decoder that restores the received bit stream.

[0064] Fig.15a is a diagram showing an example from the perspective of operating an NPU according to the present disclosure.

[0065] Fig.15b is a flowchart showing the signal flow between the transmission-side NPU and the reception-side NPU.

[0066] Fig.15c yes Fig.15b A variation of .

[0067] Fig.15d yes Fig.15b Another variation of .

[0068] Fig.16 is an exemplary diagram showing an exemplary first scenario to which an example of the present disclosure is applied.

[0069] Fig.17 is an example diagram illustrating an exemplary second example scenario to which an example of the present disclosure is applied.

[0070] Fig.18 is an example diagram showing an example of a bit stream according to an example of the present disclosure. DETAILED DESCRIPTION

[0071] The specific structure or step-by-step description of the examples according to the present disclosure is only for the purpose of explaining the examples of the concepts according to the present disclosure. The examples of the concepts according to the present invention can be implemented in various forms and should not be interpreted as limited to the examples described in the present disclosure.

[0072] Examples according to the contents of the present disclosure may have various deformations and may take various forms. Therefore, specific examples are shown in the accompanying drawings and described in detail in the present disclosure. However, this does not mean that the examples according to the concepts of the present disclosure are limited to specific public forms, but should be understood to include all deformations, equivalents or substitutes included in the spirit and technical scope of the present disclosure.

[0073] The first and / or second terms may be used to describe various constituent elements, but the constituent elements should not be limited by these terms. The naming of the terms is intended only to distinguish one constituent element from another constituent element. For example, without departing from the scope of rights of the conception according to the present disclosure, the first constituent element may be named as the second constituent element, and similarly, the second constituent element may also be named as the first constituent element.

[0074] When a component is referred to as being "connected" or "accessed" to another component, it may be directly connected or accessed to the other component. However, it should be understood that there may be other components between the components. On the other hand, when a component is "directly connected" or "directly accessed" to another component, it should be understood that there are no other components in between.

[0075] Other expressions describing the relationship between constituent elements, such as “between” and “directly between” or “adjacent” and “directly adjacent”, should be interpreted similarly.

[0076] The terms used in this disclosure are only used to describe specific examples and are not intended to limit the disclosure. Unless the context clearly indicates otherwise, singular expressions include plural expressions. In this disclosure, the terms "including" or "having" are intended to specify the presence of the features, numbers, steps, operations, constituent elements, parts, or combinations thereof. That is, terms such as "including" or "having" do not preclude the presence or addition of one or more other features, numbers, steps, operations, constituent elements, parts, or combinations thereof described.

[0077] Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as those generally understood by ordinary technicians in the field to which the present disclosure belongs. Terms defined in commonly used dictionaries should be interpreted as having a meaning consistent with the meaning in the context of the relevant technology, and should not be interpreted in an idealized or overly formal sense unless clearly defined in the present disclosure.

[0078] When illustrating the examples, descriptions of technical contents that are well known in the technical field to which the present invention belongs and are not directly related to the present invention are omitted. This is to convey the gist of the present disclosure more clearly without obscuring it by omitting unnecessary explanations.

[0079] The present disclosure may relate to video / image coding. For example, the method / example according to the present disclosure may be associated with the Versatile Video Coding (VVC) standard (ITU-T Rec. H.266), the next generation video / image coding standard after VVC, or other video coding related standards (e.g., High Efficiency Video Coding (HEVC) standard (ITU-T Rec. H.265), the essential video coding (EVC) standard, the AVS2 standard, etc.).

[0080] The present disclosure suggests various examples regarding video / image encoding, and unless otherwise stated, these examples may be performed in combination with each other.

[0081] According to the present disclosure, a video may refer to a collection of a series of images according to the flow of time. An image / picture generally refers to a unit that displays an image within a specific time period. A slice / tile is a unit that constitutes a part of a picture in encoding. A slice / tile may contain at least one coding tree unit (CTU). A picture may consist of at least one slice / tile. A picture may consist of at least one tile group. A tile group may include at least one tile.

[0082] A pixel or a pixel (pel) may refer to the smallest unit constituting a picture (or video). In addition, a "sample" may be used as a term corresponding to a pixel. A sample may generally represent a pixel or a pixel value, and may represent only a pixel / pixel value of a luminance (luma) component, or only a pixel / pixel value of a chrominance (chroma) component. A sample may mean a pixel value in the spatial domain, and when these pixel values ​​are transformed to the frequency domain, they may represent a transform coefficient in the frequency domain.

[0083] A unit may represent a basic unit of image processing. A unit may include at least one of a specific area of ​​a picture and information related to the corresponding area. A unit may include a luminance block and two chrominance (e.g., cb, cr) blocks. Depending on the situation, a unit may be mixed with terms such as block or area. In general, an M×N block may include a set (or array) of samples (or sample arrays) or transform coefficients consisting of M columns and N rows.

[0084] <Definition of terms> In order to help understand the present disclosure, the terms used in the present disclosure are briefly summarized below.

[0085] NPU: As an abbreviation of Neural Processing Unit, it is different from the Central Processing Unit (CPU) and can mean a specialized processor for the calculation of artificial neural network models. NPU can be distinguished as an NPU dedicated to reasoning calculations or an NPU that can perform reasoning and learning. NPU can include multiple processing elements suitable for processing convolution operations or matrix multiplication.

[0086] AI accelerator: As an AI computing accelerator, it can refer to NPU.

[0087] ANN: It is the abbreviation of artificial neural network. ANN can refer to a network that connects nodes in a layer structure, imitating the way neurons in the human brain are connected through synapses, in order to imitate human intelligence.

[0088] Information about the structure of the artificial neural network: including information about the number of layers, information about feature maps of each layer, information about the number of channels, information about weights, information about activation functions, etc. For example, the structure of the artificial neural network can be expressed in the open neural network exchange (ONNX) format, a directed acyclic graph (DAG), etc.

[0089] Information on the locality of artificial neural network data: including information on the order of memory access requests of neural processing units corresponding to the calculation of the artificial neural network model. In the inference stage, the data locality of the artificial neural network is actually fixed.

[0090] Therefore, when the same artificial neural network model is reused, pre-fetching of data can be made easier. In addition, latency hiding is made easier because the next data access request can be predicted. Information about the locality of artificial neural network data can be generated by a compiler and can be information included in the machine code or binary code generated to drive the ANN. The machine code or binary code may be compiled taking into account the hardware resources of the NPU and the size of the NPU memory, and is a dedicated code generated to process a specific ANN in a specific NPU.

[0091] DNN: As the abbreviation of Deep Neural Network, it means the increase in the number of hidden layers of artificial neural networks in order to achieve higher artificial intelligence.

[0092] CNN: Convolutional Neural Network is a neural network similar to the one that processes images in the visual cortex of the human brain. It is well known that convolutional neural networks are suitable for image processing and are easy to extract features from input data and grasp the pattern of features.

[0093] Transformer: Transformer is an ANN widely used in natural language processing operations such as machine translation, text classification, and sentiment analysis. Transformer is an ANN based on self-attention. Self-attention can evaluate the importance of other parts of the input data when predicting the model. Transformer has been widely studied because it shows better reasoning performance than CNN.

[0094] Visual Transformer: Visual Transformer is a specialized Transformer neural network for object recognition or image classification. Like Transformer, Visual Transformer can use self-attention to evaluate the importance of other parts of the input image. Visual Transformer can handle input images of various sizes. Visual Transformer is suitable for image classification operations because it can learn complex spatial relationships between other parts of the image.

[0095] Kernel: A weight matrix for CNN, Transformer, etc. The value of the kernel can be determined through machine learning.

[0096] The present disclosure will be described in detail below by describing preferred examples of the present disclosure with reference to the accompanying drawings. The embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings.

[0097] Figure 1 is a diagram schematically illustrating an example of a video / image encoding system.

[0098] Reference Figure 1 The video / image coding system may include a source device and a receiving device. The source device may send the coded video / image information or data in the form of a file or stream to the receiving device via a digital storage medium or a network.

[0099] The source device may include a video source, an encoding device, and a transmitting unit. The receiving device may include a receiving unit, a decoding device, and a renderer. The encoding device may be referred to as a video / image encoding device. The decoding device may be referred to as a video / image decoding device. The transmitter may be included in the encoding device. The receiver may be included in the decoding device. The renderer may also include a display unit. The display unit may consist of a separate device or an external component.

[0100] The video source may acquire the video / image by a process of capturing, synthesizing or generating the video / image. The video source may include a video / image capturing device and / or a video / image generating device. The video / image capturing device may include, for example, at least one camera, a video / image archive containing previously captured videos / images, etc. The video / image generating device may include, for example, a computer, a tablet computer, a smart phone, etc., and may (electronically) generate the video / image. For example, a virtual video / image may be generated by a computer, etc. In this case, the video / image capturing process may be replaced by a process of generating the relevant data.

[0101] The encoding device can encode the input video / image. For compression and encoding efficiency, the encoding device can perform a series of processes such as prediction, transformation and quantization. The encoded data (encoded video / image information) can be output in the form of a bitstream.

[0102] The sending unit can send the encoded video / image information or data output in the form of a bit stream to the receiving unit of the receiving device in the form of a file or stream through a digital storage medium or a network. The digital storage medium may include various storage media, such as USB, SD, CD, DVD, Blu-ray, HDD and SSD. The sending unit may include an element for generating a media file in a predetermined file format. The sending unit may include an element for transmitting through a broadcast / communication network.

[0103] The receiving unit may receive / extract the bitstream and send it to the decoding device.

[0104] The decoding device may decode the video / image by performing a series of processes corresponding to the operations of the encoding device, such as dequantization, inverse transformation, and prediction.

[0105] The renderer may render the decoded video / image. The rendered video / image may be displayed by a display unit.

[0106] Figure 2 is a diagram schematically showing the configuration of a video / image encoding device.

[0107] Hereinafter, a video encoding device may include a video encoding device.

[0108] Reference Figure 2, the encoding device 10a may include an image partitioner (image partitioner, 10a-10), a prediction unit (predictor, 10a-20), a residual processing unit (residual processor, 10a-30) and an entropy encoder (entropy encoder, 10a-40), an adder (adder, 10a-50), a filter unit (filter, 10a-60) and a memory (memory, 10a-70). The prediction unit 10a-20 may include an inter-frame prediction unit 10a-21 and an intra-frame prediction unit 10a-22. The residual processing unit 10a-30 may include a transformer (transformer, 10a-32), a quantizer (quantizer, 10a-33), a dequantizer (dequantizer, 10a-34) and an inverse transformer (inverse transformer, 10a-35). The residual processing unit 10a-30 may also include a subtractor (10a-31). The addition unit 10a-50 may be referred to as a reconstructor or a reconstructed block generator. The above-mentioned image segmenter 10a-10, prediction unit 10a-20, residual processing unit 10a-30, entropy encoder 10a-40, addition unit 10a-50 and filtering unit 10a-60 may be composed of at least one hardware constituent element (e.g., an encoder chipset or a processor) according to the example. In addition, the memory 10a-70 may include a decoded picture buffer (DPB) and may be configured by a digital storage medium. The hardware constituent element may also include the memory 10a-70 as an internal / external constituent element.

[0109] The image segmenter 10a-10 can divide the input image (or picture, frame) input to the encoding device 10a into at least one processing unit (processing unit). As an example, the processing unit can be referred to as an encoder (coding unit, CU). In this case, the encoder will be recursively divided from the coding tree unit (coding tree unit, CTU) or the largest coding unit (largest coding unit, LCU) according to the quadtree, binary tree, ternary tree (Quad-tree binary-treeternary-tree, QTBTTT) structure. For example, an encoder can be divided into multiple encoders of deeper depth based on a quadtree structure, a binary tree structure and / or a ternary structure. In this case, for example, a quadtree structure can be applied first, and a binary tree structure and / or a ternary tree structure can be applied subsequently. Alternatively, a binary tree structure can be applied first. The encoding process according to this document can be performed based on the final encoder that is no longer divided. At this time, according to the coding efficiency of the image characteristics, etc., the maximum encoder can be directly used as the final encoder, or the encoder can be recursively divided into encoders of a deeper depth as needed, so that the encoder of the optimal size can be used as the final encoder. Among them, the encoding process may include processes such as prediction, transformation and restoration described later. As another example, the processing unit may also include a prediction unit (PU) or a transform unit (TU). In this case, both the prediction unit and the transform unit can be divided or partitioned from the above-mentioned final encoder. The prediction unit can be a unit of sample prediction, and the transform unit can be a unit for deriving a transform coefficient and / or a unit for deriving a residual signal from a transform coefficient.

[0110] Depending on the situation, the unit can be mixed with terms such as block or area. In general, an M×N block can represent a set of samples or transform coefficients consisting of M columns and N rows. Samples can generally represent pixels or pixel values, and can represent only the pixel / pixel value of the brightness (luma) component, or only the pixel / pixel value of the chrominance (chroma) component. Sample can be used as a term corresponding to a pixel (pixel) or pixel (pel) of a picture (or image).

[0111] The subtractor 10a-31 subtracts the prediction signal (prediction block, prediction sample or prediction sample array) output from the prediction unit 10a-20 from the input image signal (original block, original sample or original sample array) to generate a residual signal (residual block, residual sample or residual sample array), and the generated residual signal is sent to the transform unit 10a-32. The prediction unit 10a-20 can perform prediction on the processing object block (hereinafter referred to as the current block) and can generate a predicted block including the prediction sample of the current block. The prediction unit 10a-20 can determine whether intra-frame prediction or inter-frame prediction is applicable based on the current block or CU unit. As described later, in the description of each prediction mode, the prediction unit can generate various information related to the prediction (such as prediction mode information) and send it to the entropy encoder 10a-40. Information about the prediction can be encoded in the entropy encoder 10a-40 and output in the form of a bit stream.

[0112] The intra-frame prediction unit 10a-22 can predict the current block by referring to the samples in the current picture. Depending on the prediction mode, the reference sample can be located near the current block (neighbor) or can be located far away from the current block. In intra-frame prediction, the prediction mode may include multiple non-directional modes and multiple directional modes. The non-directional mode may include, for example, a DC mode and a planar mode. Depending on the fineness of the prediction direction, the directional mode may include, for example, 33 directional prediction modes or 65 directional prediction modes. However, this is an example and more or less directional prediction modes can be used according to the settings. The intra-frame prediction unit 10a-22 can use the prediction mode applied to the surrounding blocks to determine the prediction mode applied to the current block.

[0113] The inter prediction unit 10a-21 can derive a block of prediction for the current block based on a reference block (reference sample array) specified by a motion vector in a reference picture. At this time, in order to reduce the amount of motion information sent in the inter prediction mode, the motion information can be predicted in blocks, sub-blocks or sample units based on the correlation of the motion information between the surrounding blocks and the current block. The motion information may include a motion vector and a reference image index. The motion information may also include inter prediction direction (L0 prediction, L1 prediction, Bi prediction, etc.) information. In the case of inter prediction, the surrounding blocks may include spatial neighboring blocks present in the current picture and temporal neighboring blocks present in the reference image. The reference picture including the reference block and the reference picture including the temporal neighboring block may be the same or different. The temporal neighboring block may be referred to as a collocated reference block, a collocated CU (colCU), etc., and the reference image including the temporal neighboring block may be referred to as a collocated picture (collocated Picture, colPic). For example, the inter prediction unit 10a-21 constructs a motion information candidate list based on the surrounding blocks, and in order to derive the motion vector and / or reference image index of the current block, information indicating which candidate to use may be generated. Inter prediction may be performed based on various prediction modes. For example, in the case of skip mode and merge mode, the inter prediction unit 10a-21 may use the motion information of the surrounding blocks as the motion information of the current block. In the case of skip mode, unlike the merge mode, residual information may not be sent. In the case of motion information prediction (motion vector Prediction, MVP) mode, the motion vector of the surrounding blocks is used as a motion vector predictor, and the motion vector difference is signaled, so that the motion vector of the current block can be indicated.

[0114] The prediction units 10a-20 can generate prediction signals based on various prediction methods described later. For example, the prediction unit can not only apply intra-frame prediction or inter-frame prediction to predict a block, but also apply intra-frame prediction and inter-frame prediction at the same time. This can be called combined inter-frame and intra-frame prediction (CIIP). In addition, the prediction unit can also perform intra-frame block copy (IBC) for the prediction of the block. Intra-frame block copy can be used for game content video / video coding such as screen content coding (SCC). IBC basically performs prediction within the current picture, but it can also be performed similarly to inter-frame prediction in terms of deriving reference blocks within the current picture. That is, IBC can use at least one inter-frame prediction technique described in this document.

[0115] The prediction signal generated by the inter-frame prediction unit 10a-21 and / or the intra-frame prediction unit 10a-22 can be used to generate a restored signal or a residual signal. The transform unit 10a-32 can generate transform coefficients by applying a transform technique to the residual signal. For example, the transform technique may include a discrete cosine transform (DCT), a discrete sine transform (DST), a graph-based transform (GBT), or a conditional nonlinear transform (CNT). Among them, GBT refers to a transform obtained from a graph when the relationship information between pixels is expressed as a graph. CNT refers to a transform obtained by generating a prediction signal using all previously restored pixels (allPreviously reconstructedPixel) and based on this. In addition, the transform process can be applied to pixel blocks of the same size having a square, and can also be applied to pixel blocks of variable size that are not square.

[0116] The quantization unit 10a-33 quantizes the transform coefficients and sends them to the entropy encoder 10a-40, and the entropy encoder 10a-40 can encode the quantized signal (information about the quantized transform coefficients) and output it as a bit stream. The information about the quantized transform coefficients can be called residual information. The quantization unit 10a-33 can rearrange the quantized transform coefficients in block form into a one-dimensional vector form based on the coefficient scanning order (scan order), and generate information about the quantized transform coefficients based on the quantized transform coefficients in the one-dimensional vector form. The entropy encoder 10a-40 can perform various encoding methods, such as exponential Golomb, context-adaptive variable length coding (context-adaptivevariablelengthcoding, CAVLC) and context-adaptivebinaryarithmeticcoding (context-adaptivebinaryarithmeticcoding, CABAC).

[0117] In addition to the quantized transform coefficients, the entropy encoder 10a-40 can also encode the information required for video / image restoration (e.g., the values ​​of syntax elements, etc.) together or separately. The encoded information (e.g., the encoded video / image information) can be sent or stored in a network abstraction layer (NAL) unit unit in the form of a bitstream. The video / image information can also include information about various parameter sets, such as an adaptive parameter set (APS), a picture parameter set (PPS), a sequence parameter set (SPS), or a video parameter set (VPS). In addition, the video / image information can also include general constraint information. The signaling / transmission information and / or syntax elements described later in this document can be encoded and included in the bitstream through the above-mentioned encoding process. The bitstream can be transmitted through a network or stored in a digital storage medium. Among them, the network can include a broadcast network and / or a communication network, etc., and the digital storage medium can include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD and SSD. The signal output from the entropy encoder 10a-40 can be configured as a transmission unit (not shown) for transmission and / or a storage unit (not shown) for storage, or the transmission unit can also be included in the entropy encoder 10a-40.

[0118] The quantized transform coefficients output from the quantization unit 10a-33 can be used to generate a prediction signal. For example, dequantization and inverse transformation can be applied to the quantized transform coefficients by the dequantization unit 10a-34 and the inverse transformation unit 10a-35, thereby restoring the residual signal (residual block or residual sample). The addition unit 10a-50 adds the restored residual signal to the prediction signal output from the prediction unit 10a-20, thereby generating a restored (reconstructed) signal (restored picture, restored block, restored sample or restored sample array). If there is no residual for the processing object block in the case of the skip mode, the predicted block can be used as a restored block. The generated restored signal can be used for intra-frame prediction of the next processing object block in the current picture, and can also be used for inter-frame prediction of the next picture of the filtered wave as described later.

[0119] Meanwhile, during the image encoding and / or restoration process, lumamapping with chroma scaling (LMCS) may also be applied.

[0120] The filtering unit 10a-60 can improve the subjective / objective image quality by applying filtering to the restored signal. For example, the filtering unit 10a-60 can generate a modified restored image by applying various filtering methods to the restored image, and store the modified restored image in the memory 10a-70, specifically in the DPB of the memory 10a-70. Various filtering methods may include, for example, deblocking filtering, sample adaptive offset (sample adaptive offset, SAO), adaptive loop filter (adaptive loop filter), bilateral filter (bilateral filter), etc. The filtering unit 10a-60 can generate various information about filtering and send it to the entropy encoder 10a-90, as described later for each filtering method. Information about filtering can be encoded in the entropy encoder 10a-90 and output in the form of a bit stream.

[0121] The modified restored picture sent to the memory 10a-70 can be used as a reference picture in the inter-frame prediction unit 10a-80. By this, when inter-frame prediction is applied, the encoding device can avoid prediction mismatch in the encoding device 10a and the decoding device, and can also improve encoding efficiency.

[0122] The DPB of the memory 10a-70 can store the modified restored picture to use it as a reference picture in the inter-frame prediction unit 10a-21. The memory 10a-70 can store the motion information of the block from which the motion information in the current picture is derived (or encoded) and / or the motion information of the block in the restored picture. The stored motion information can be sent to the inter-frame prediction unit 10a-21 to be used as the motion information of the spatial surrounding blocks or the motion information of the temporal surrounding blocks. The memory 10a-70 can store the restored samples of the restored blocks in the current image and send them to the intra-frame prediction unit 10a-22.

[0123] Figure 3 is a diagram schematically showing the configuration of a video / image decoding device.

[0124] Reference Figure 3 , the decoding device 10b may include an entropy decoder (entropy decoder, 10b-10), a residual processing unit (residualProcessor, 10b-20), a prediction unit (predictor, 10b-30), an addition unit (adder, 10b-40). A filtering unit (filter, 10b-50) and a memory (memory, 10b-60). The prediction unit 10b-30 may include an inter-frame prediction unit 10b-31 and an intra-frame prediction unit 10b-32. The residual processing unit 10b-20 may include a dequantizer (dequantizer, 10b-21) and an inverse transformer (inverse transformer, 10b-21). According to an example, the above-mentioned entropy decoding unit 10b-10, residual processing unit 10b-20, prediction unit 10b-30, addition unit 10b-40 and filtering unit 10b-50 may be configured by one hardware component (e.g., a decoder chipset or processor). In addition, the memory 10b-60 may include a decoded picture buffer (DPB) and may be configured by a digital storage medium. The hardware component may also include the memory 10b-60 as an internal / external component.

[0125] When a bit stream including video / image information is input, the decoding device 10b may respond to the Figure 2The image is restored by the process of processing video / image information in the encoding device of the decoding device. For example, the decoding device 10b can derive the unit / block based on the block division related information obtained from the bit stream. The decoding device 10b can perform decoding using the processing unit applied in the encoding device. Accordingly, the processing unit of decoding can be, for example, an encoder, and the encoder can be divided from the coding tree unit or the maximum encoder along the quadtree structure, the binary tree structure and / or the ternary tree structure. At least one transformation unit can be derived from the encoder. And, the restored image signal decoded and output by the decoding device 10b can be played by a playback device.

[0126] The decoding device 10b may receive the Figure 2 The received signal may be decoded by the entropy decoding unit 10b-10. For example, the entropy decoding unit 10b-10 may parse the bitstream to derive information (e.g., video / image information) required for image restoration (or picture restoration). The video / image information may also include information about various parameter sets, such as an adaptive parameter set (APS), a picture parameter set (PPS), a sequence parameter set (SPS), or a video parameter set (VPS). In addition, the video / image information may also include general constraint information.

[0127] The decoding device can also decode the image based on information about the parameter set and / or general constraint information. The signaling / received information and / or syntax elements described later in this document can be decoded by a decoding process and obtained from the bitstream. For example, the entropy decoding unit 10b-10 decodes the information in the bitstream based on a coding method such as exponential Golomb coding, CAVLC or CABAC, and can output the value of the syntax element required for image restoration and the quantized value of the transform coefficient associated with the residual. In more detail, the CABAC entropy decoding method receives the binary bits corresponding to each syntax element from the bitstream, and uses the decoding object syntax element information, the decoding information of the decoding object block, or the information of the symbol / binary bit decoded in the previous step to determine the context model, predict the occurrence probability of the binary bit (bin) according to the determined context model and perform arithmetic decoding of the binary bit (arithmetic decoding), and generate symbols belonging to each syntax element value. At this time, the CABAC entropy decoding method can use the decoded symbol / binary bit information to update the context model for the context model of the next symbol / binary bit after determining the context model. Among the information decoded in the entropy decoding unit 10b-10, information about prediction is provided to the prediction unit 10b-30, and information about the residual of entropy decoding performed in the entropy decoding unit 10b-10, that is, quantized transformation coefficients and related parameter information can be input to the dequantization unit 10b-21.

[0128] In addition, information about filtering in the information decoded by the entropy decoding unit 10b-10 can be provided to the filtering unit 10b-50. At the same time, a receiving unit (not shown) that receives a signal output from the encoding device can also be configured as an internal / external element of the decoding device 10b, or the receiving unit can be a constituent element of the entropy decoding unit 10b-10. At the same time, the decoding device according to the present document can be referred to as a video / image / picture decoding device, and the decoding device can be divided into an information decoder (video / image / picture information decoder) and a sample decoder (video / image / picture sample decoder). The information decoder may include an entropy decoding unit 10b-10, and the sample decoder may include at least one of a dequantization unit 10b-21, an inverse transform unit 10b-22, a prediction unit 10b-30, an addition unit 10b-40, a filtering unit 10b-50, and a memory 10b-60.

[0129] The dequantization unit 10b-21 may dequantize the quantized transform coefficient and output the transform coefficient. The dequantization unit 10b-21 may rearrange the quantized transform coefficient into a two-dimensional block form. In this case, the rearrangement may be performed based on a coefficient scanning order performed in the encoding device. The dequantization unit 10b-21 may dequantize the quantized transform coefficient using a quantization parameter (e.g., quantization step size information) and obtain a transform coefficient.

[0130] The inverse transform unit 10b-22 inversely transforms the transform coefficients to obtain a residual signal (residual block, residual sample array).

[0131] The prediction unit may perform prediction on the current block and generate a predicted block including prediction samples of the current block. The prediction unit may determine whether to apply intra prediction or inter prediction to the current block based on the information about the prediction output from the entropy decoding unit 10b-10, and may determine a specific intra / inter prediction mode.

[0132] The prediction unit can generate a prediction signal based on various prediction methods described later. For example, in order to predict a block, the prediction unit can not only apply intra prediction or inter prediction, but also apply intra prediction and inter prediction at the same time. This can be called combined inter and intraPrediction (CIIP). In addition, the prediction unit can perform intra block copy (IBC) to predict the block. Intra block copy can be used for game content video / video coding such as screen content coding (SCC). IBC basically performs prediction within the current picture, but it can also be performed similarly to inter prediction in terms of deriving reference blocks within the current picture. That is, IBC can use at least one inter prediction technique described in this document.

[0133] The intra prediction unit 10b-32 can predict the current block by referring to samples in the current picture. Depending on the prediction mode, the reference sample can be located near the current block (neighbor) or can also be located far away from the current block. In intra prediction, the prediction mode can include multiple non-directional modes and multiple directional modes. The intra prediction unit 10b-32 can use the prediction mode applied to the surrounding blocks to determine the prediction mode applied to the current block.

[0134] The inter-frame prediction unit 10b-31 can derive a prediction block for the current block based on a reference block (reference sample array) that is specified by a motion vector in a reference picture. At this time, in order to reduce the amount of motion information sent in the inter-frame prediction mode, the motion information can be predicted in units of blocks, sub-blocks or samples based on the correlation of motion information between the surrounding blocks and the current block. The motion information may include a motion vector and a reference picture index. The motion information may also include inter-frame prediction direction (L0 prediction, L1 prediction, Bi prediction, etc.) information.

[0135] In the case of inter-frame prediction, the neighboring blocks may include spatial neighboring blocks present in the current picture and temporal neighboring blocks present in the reference image. For example, the inter-frame prediction unit 10b-31 may construct a motion information candidate list based on the neighboring blocks, and derive a motion vector and / or a reference picture index of the current block based on the received candidate selection information. Inter-frame prediction may be performed based on various prediction modes, and information about the prediction may include information indicating a mode of inter-frame prediction for the current block.

[0136] The adding unit 10b-40 adds the obtained residual signal to the prediction signal (prediction block, prediction sample array) output from the prediction unit 10b-30 to generate a restored signal (restored picture, restored block, restored sample array). If there is no residual for the processing object block when the skip mode is applied, the prediction block can be used as the restored block.

[0137] The adding unit 10b-40 may be referred to as a restoration unit or a restoration block generating unit. The generated restoration signal may be used for intra-frame prediction of the next processing target block in the current picture, and may be output through filtering as described later, or may be used for inter-frame prediction of the next picture.

[0138] On the one hand, luma mapping with chroma scaling (LMCS) can be applied during the image decoding process.

[0139] The filtering unit 10b-50 can improve the subjective / objective image quality by applying filtering to the restored signal. For example, the filtering unit 10b-50 can generate a modified restored image by applying various filtering methods to the restored image, and send the modified restored image to the memory 60, specifically, to the DPB of the memory 10b-60. The various filtering methods may include, for example, deblocking filtering, sample adaptive offset, adaptive loop filter, bilateral filtering unit, etc.

[0140] The (modified) restored picture stored in the DPB of the memory 10b-60 can be used as a reference picture in the inter-frame prediction unit 10b-31. The memory 10b-60 can store the motion information of the block from which the motion information in the current picture is derived (or decoded) and / or the motion information of the block in the restored picture. The stored motion information can be sent to the inter-frame prediction unit 10b-31 to be used as the motion information of the spatial neighboring blocks or the motion information of the temporal neighboring blocks. The memory 10b-60 can store the restored samples of the restored blocks in the current picture and send them to the intra-frame prediction unit 10b-32.

[0141] In the present disclosure, the examples described in the prediction unit 10b-30, dequantization unit 10b-21, inverse transform unit 10b-22 and filtering unit 10b-50 of the decoding device 10b are all the same or correspondingly applied to the prediction unit 10a-20, dequantization unit 10a-34, inverse transform unit 10a-35 and filtering unit 10a-60, etc. of the encoding device 10a.

[0142] As described above, when performing video encoding, prediction is performed to improve compression efficiency. Thus, a prediction block containing prediction samples for a current block as an encoding object block can be generated. Among them, the prediction block includes prediction samples in the spatial domain (or pixel domain). The predicted block is also derived in the encoding device and the decoding device, and the encoding device is signaled by the decoding device for the residual information (residual information) between the original block and the prediction block for the original block value itself that is not the original block, thereby improving the image coding efficiency. The decoding device can derive a residual block including residual samples based on the residual information, generate a restored block including a restored sample by combining the residual block and the predicted block, and can generate a restored picture including the restored block.

[0143] The residual information can be generated by a transformation and quantization process. For example, the encoding device derives a residual block between the original block and the predicted block, and performs a transformation process on the residual samples (residual sample array) included in the residual block to derive the transform coefficients, and performs a quantization process on the transform coefficients to derive the quantized transform coefficients, so that the relevant residual information is signaled by the decoding device (by the bitstream). Among them, the residual information may include information such as value information, position information, transformation technology, transformation kernel, and quantization parameter of the quantized transform coefficients. The decoding device can perform a dequantization / inverse transformation process based on the residual information and derive residual samples (or residual blocks). The decoding device can generate a restored picture based on the predicted block and the residual block. The encoding device can also dequantize / inverse transform the quantized transform coefficients to derive a residual block for reference for inter-frame prediction of subsequent pictures, and generate a restored picture based on this.

[0144] <Neural Processing Unit (NPU)> Figure 4 is a conceptual diagram schematically illustrating a neural processing unit according to the present disclosure.

[0145] Figure 4 The neural processing unit (NPU) 100 shown is a processor specialized for performing operations of an artificial neural network.

[0146] An artificial neural network is a network of artificial neural networks that multiply and add each weight when receiving multiple inputs or stimuli, and additionally add the value of the bias through the activation function deformation, thereby transmitting. The artificial neural network learned in this way can be used to output inference results from input data.

[0147] The NPU 100 is a semiconductor implemented by an electric / electronic circuit. The electric / electronic circuit refers to a semiconductor including a plurality of electronic elements (eg, transistors, capacitors).

[0148] In the case of a transformer and / or CNN-based artificial neural network model, the NPU 100 selects matrix multiplication operations, convolution operations, etc. according to the architecture of the artificial neural network and performs processing.

[0149] For example, in each layer of a convolutional neural network (CNN), the input feature map corresponding to the input data and the kernel corresponding to the weight can be a matrix consisting of multiple channels. The input feature map and the kernel are convolved to generate a convolution operation and a pooled output feature map in each channel. The activation map of the corresponding channel is generated by applying an activation function to the output feature map. After that, pooling of the activation map can be applied. Among them, the activation map can be collectively referred to as the output feature map.

[0150] However, the examples disclosed herein are not limited thereto, and the output feature map is applicable to matrix multiplication calculation or convolution operation, etc.

[0151] In detail, the output feature map according to the example of the present disclosure should be interpreted as a general meaning. For example, the output feature map can be the result value of a matrix multiplication operation or a convolution operation. Therefore, the multiple processing elements 110 can also be implemented in a modified manner to further include a processing circuit unit for additional algorithms.

[0152] The NPU 100 is configured to include a plurality of processing elements 110 for processing the convolutions and matrix multiplications required in the above-mentioned artificial neural network calculations.

[0153] The NPU 100 is configured to include each processing circuit that is optimal for matrix multiplication operations, soft maximum function operations, convolution operations, activation function operations, pooling operations, stride operations, batch normalization operations, skip connection operations, concatenation operations, quantization operations, cropping operations, and padding operations required for the above-mentioned artificial neural network operations.

[0154] For example, the NPU 100 is configured to include an SFU 150 for processing at least one of a soft maximum function operation, an activation function operation, a pooling operation, a stride operation, a batch normalization operation, a skip connection operation, a concatenation operation, a quantization operation, a cropping operation, and a padding operation in the above algorithm.

[0155] Specifically, the NPU 100 may include a plurality of processing elements (PEs) 110, an SFU 150, an NPU internal memory 120, an NPU controller 130, and an NPU interface 140. The plurality of processing elements 110, the SFU 150, the NPU internal memory 120, the NPU controller 130, and the NPU interface 140 are semiconductor circuits connected to a plurality of transistors, respectively. Therefore, some of these are difficult to identify and distinguish with the naked eye, and are only identified by operation.

[0156] For example, any circuit is operated by the plurality of processing elements 110 or is operated by the NPU controller 130. The NPU controller 130 is configured to perform the function of a control unit configured to control the artificial neural network inference operation of the NPU 100.

[0157] The NPU 100 may include an NPU internal memory 120 for storing parameters of an artificial neural network model inferred in the plurality of processing elements 110 and the SFU 150 , and an NPU controller 130 including a scheduler for controlling operation scheduling of the plurality of processing elements 110 , the SFU 150 , and the NPU internal memory 120 .

[0158] The NPU 100 may be configured to process a feature map corresponding to an encoding and decoding method using SVC or SFC.

[0159] The plurality of processing elements 110 may perform a portion of the operations for an artificial neural network.

[0160] The SFU 150 may perform other portions of the operation for the artificial neural network.

[0161] The NPU 100 may be configured to accelerate the operation of an artificial neural network from a hardware perspective using a plurality of processing elements 110 and an SFU 150 .

[0162] The NPU interface 140 can communicate with various components connected to the NPU 100 , such as a memory, through the system bus.

[0163] The NPU controller 130 may include a scheduler that controls operations of the plurality of processing elements 110 for inference operations of the neural processing unit 100 , operations of the SFU 150 , and a read and write order of the NPU internal memory 120 .

[0164] The scheduler in the NPU controller 130 may be configured to control the plurality of processing elements 110 , the SFU 150 , and the NPU internal memory 120 based on the data locality information or the information of the structure of the artificial neural network model.

[0165] The scheduler in the NPU controller 130 may analyze the structure of the artificial neural network model to operate in the plurality of processing elements 110 and the SFU 150, or may receive information that has been analyzed. The information may be contained in a compiled machine code or binary code. For example, the artificial neural network model may include artificial neural network data including node data (i.e., feature graphs) of each layer, arrangement data of each layer, regional information or structural information, and at least some of each weight data (i.e., weight kernel) of the connection network connecting the nodes of each layer.

[0166] The machine code or binary code may be determined according to the corresponding NPU 100. For example, when an artificial neural network model is processed in one NPU, the machine code or binary code may be compiled for one NPU. For example, when an artificial neural network model is processed in a distributed manner in a first NPU and a second NPU, the machine code or binary code may be compiled for each of the first NPU and the second NPU.

[0167] The compiled machine code or binary code may be stored in a memory provided inside the NPU controller 130 or the NPU internal memory 120 .

[0168] The scheduler in the NPU controller 130 can schedule the operation order of the artificial neural network model to be executed by the NPU 100 based on the data locality information or structure information of the artificial neural network model.

[0169] The scheduler in the NPU controller 130 can obtain the memory address value storing the feature map and weight data of the layer of the artificial neural network model based on the data locality information or the structure information of the artificial neural network model. For example, the scheduler in the NPU controller 130 can obtain the memory address value storing the feature map and weight data of the layer of the artificial neural network model stored in the memory. Therefore, the scheduler in the NPU controller 130 can retrieve the feature map and weight data of the layer of the artificial neural network model to be driven from the main memory and store them in the NPU internal memory 120.

[0170] The feature map of each layer may have corresponding memory address values.

[0171] Each weight data may have a corresponding memory address value.

[0172] The scheduler in the NPU controller 130 can execute the operation sequence of multiple processing elements 110 based on the data locality information or structural information of the artificial neural network model, for example, the configuration data, locality information or structural information of the layers of the artificial neural network of the artificial neural network model.

[0173] The scheduler in the NPU controller 130 performs scheduling based on the data locality information or structure information of the artificial neural network model, and therefore can operate differently from the scheduling concept of a general CPU. The scheduling of a general CPU considers fairness, efficiency, stability, response time, etc., so as to operate with the best efficiency. In other words, scheduling is performed by considering priority, operation time, etc. to perform the most processing in the same time.

[0174] Conventional CPUs use an algorithm that schedules jobs by taking into account data such as the priority of each process and the computational processing time.

[0175] Differently, the scheduler in the NPU controller 130 may control the NPU 100 based on the processing order of the NPU 100 determined by data locality information or structure information of the artificial neural network model.

[0176] In addition, the scheduler in the NPU controller 130 drives the NPU 100 based on the processing order determined by the data locality information or structure information of the artificial neural network model and / or the data locality information or structure information of the neural processing unit 100 to be used.

[0177] However, the present disclosure is not limited to data locality of the NPU 100 or information on the structure.

[0178] The scheduler within the NPU controller 130 is configured to store data locality information or information on the structure of the artificial neural network.

[0179] In other words, even if the scheduler in the NPU controller 130 uses at least only the data locality information or the information on the structure of the artificial neural network model, the processing order can be determined.

[0180] In addition, the scheduler in the NPU controller 130 may determine the processing order of the NPU 100 by considering the data locality information or information on the structure of the artificial neural network model and the data locality information or information on the structure of the NPU 100. In addition, the processing of the NPU 100 is optimized according to the determined processing order.

[0181] The plurality of processing elements 110 refers to a structure of a plurality of processing elements (PE1-PE12) configured with feature maps and weight data for computing artificial neural networks. Each processing element may include a multiply and accumulate (MAC) operator and / or an arithmetic logic unit (ALU) operator. However, examples according to the present disclosure are not limited thereto.

[0182] In order to process additional special functions, each processing element may also selectively include additional special function unit circuits.

[0183] For example, the processing element PE is deformedly implemented to further include a batch normalization circuit, an activation function circuit, an interpolation circuit, etc.

[0184] The SFU 150 may include a circuit unit that selects and processes activation function operations, pooling operations, stride operations, batch-normalization operations, skip-connection operations, concatenation operations, quantization operations, clipping operations, padding operations, etc. according to the architecture of the artificial neural network. That is, the SFU 150 may include a plurality of special function operation processing circuit units.

[0185] exist Figure 4 In the embodiment, although a plurality of processing elements are used as an example, a processing element may be configured in which a plurality of multipliers and adder trees are implemented in parallel, thereby replacing the MAC within the processing element. In this case, the plurality of processing elements 110 may also be referred to as at least one processing element including a plurality of operators.

[0186] The plurality of processing elements 110 are configured to include a plurality of processing elements PE1 to PE12 . Figure 4 The multiple processing elements PE1 to PE12 shown in the figure are merely examples for convenience of explanation, and the number of the multiple processing elements PE1 to PE12 is not limited. The size or number of the multiple processing elements 110 can be determined according to the number of the multiple processing elements PE1 to PE12. The size of the multiple processing elements 110 can be implemented in the form of an N×M matrix. Wherein N and M are integers greater than 0. The multiple processing elements 110 may include N×M processing elements. That is, the processing element may be more than one.

[0187] The sizes of the plurality of processing elements 110 may be designed in consideration of the characteristics of the artificial neural network model run by the NPU 100 .

[0188] The plurality of processing elements 110 are configured to perform the functions of addition, multiplication, accumulation, etc. required for artificial neural network operations. In other words, the plurality of processing elements 110 can be configured to perform multiplication and accumulation (MAC) operations.

[0189] The following description will be made by taking the first processing element PE1 among the plurality of processing elements 110 as an example.

[0190] Figure 5a is a conceptual diagram schematically illustrating one processing element among a plurality of processing elements applicable to the present disclosure.

[0191] The NPU 100 according to the example of the present disclosure includes a plurality of processing elements 110, an NPU internal memory 120 configured to store an artificial neural network model that can be inferred from the plurality of processing elements 110, and a plurality of processing elements 110 and an NPU controller 130 configured to control the NPU internal memory 120, the plurality of processing elements 110 are configured to perform MAC operations, and the plurality of processing elements 110 can be configured to quantize and output MAC operation results. However, the examples of the present disclosure are not limited thereto.

[0192] The NPU internal memory 120 can store all or part of the artificial neural network model according to the memory size and the data size of the artificial neural network model.

[0193] The first processing element PE1 may include a multiplier 111, an adder 112, an accumulator 113, and a bit quantization unit 114. However, examples according to the present disclosure are not limited thereto, and a plurality of processing elements 110 may be deformedly implemented in consideration of operational characteristics of an artificial neural network.

[0194] The multiplier 111 multiplies the input (N) bit data and (M) bit data. The operation value of the multiplier 111 is output as (N+M) bit data.

[0195] The multiplier 111 may be configured to receive a variable and a constant.

[0196] The accumulator 113 uses the adder 112 for (L) loops and accumulates the operation value of the multiplier 111 and the operation value of the accumulator 113. Therefore, the bit width of the data of the output unit and the input unit of the accumulator 113 can be output with (N+M+log2(L)) bits. Here, L is an integer greater than 0.

[0197] When the accumulation is completed, the accumulator 113 may receive an initialization signal ( ) and initialize the data stored in the accumulator 113 to 0. However, examples according to the present disclosure are not limited thereto.

[0198] The bit quantization unit 114 can reduce the bit width of the data output from the accumulator 113. The bit quantization unit 114 can be controlled by the NPU controller 130. The bit width of the quantized data can be output as (X)bit. Here, X is an integer greater than 0. According to the above configuration, the plurality of processing elements 110 are configured to perform MAC operations, and the plurality of processing elements 110 have the effect of quantizing and outputting the MAC operation results. In particular, as the (L)loops increase, this quantization has the effect of further reducing power consumption. In addition, reducing power consumption also has the effect of reducing heat generation. In particular, reducing heat generation has the effect of reducing the possibility of malfunction due to the high temperature of the NPU 100.

[0199] The output data (X) bit of the bit quantization unit 114 can be the node data of the next layer or the input data of the convolution. If the artificial neural network model is quantized, the bit quantization unit 114 can be configured to receive quantization information from the artificial neural network model. However, not limited to this, the NPU controller 130 can be configured to extract quantization information by analyzing the artificial neural network model. Therefore, the output data (X) bit can be transformed into a quantized bit width and output to correspond to the quantized data size. The output data (X) bit of the bit quantization unit 114 can be stored in the NPU internal memory 120 as a quantized bit width.

[0200] The plurality of processing elements 110 of the NPU 100 according to an example of the present disclosure includes a multiplier 111, an adder 112, and an accumulator 113. The bit quantization unit 114 may select a trade-off according to whether quantization is applicable or not.

[0201] Figure 5b is a conceptual diagram schematically illustrating an SFU applicable to the present disclosure.

[0202] refer to Figure 5b , SFU 150 includes multiple functional units. Each functional unit can be selectively operated. Each functional unit can be selectively turned on or off. That is, each functional unit can be set.

[0203] In other words, the SFU 150 may include various circuit units required for artificial neural network inference operations.

[0204] For example, the circuit units of SFU 150 include functional units for skip-connection operations, functional units for activation function operations, functional units for pooling operations, functional units for quantization operations, functional units for non-maximum suppression (NMS) operations, functional units for integer and floating-point conversion (INT to FP32) operations, functional units for batch-normalization operations, functional units for interpolation operations, functional units for concatenation operations, and functional units for bias operations.

[0205] The functional units of the SFU 150 are selectively turned on or off according to the data locality information of the artificial neural network model. The data locality information of the artificial neural network model may include control information for turning on or off the associated functional units when executing a specific layer of operation.

[0206] Among the functional units of the SFU 150, the activated units may be turned on. In this way, when some functional units of the SFU 150 are selectively turned off, the power consumption of the NPU 100 may be reduced. At the same time, some functional units may be turned off using power gating. Alternatively, some functional units may be turned off using clock gating.

[0207] Figure 6 It is shown Figure 4 An example diagram of a variation of neural processing unit 100 is shown in FIG.

[0208] Figure 6 The NPU 100 is shown with Figure 4 The processing units 100 shown are substantially the same except for the plurality of processing elements 110 , and thus repeated explanations may be omitted below for convenience of explanation only.

[0209] Figure 6 The plurality of processing elements 110 exemplarily shown in FIG. 1 include, in addition to the plurality of processing elements PE1 to PE12 , each register file RF1 to RF12 corresponding to each of the processing elements PE1 to PE12 .

[0210] Figure 6 The plurality of processing elements PE1 to PE12 and the plurality of register files RF1 to RF12 shown in FIG. 1 are merely examples for convenience of explanation, and the number of the plurality of processing elements PE1 to PE12 and the plurality of register files RF1 to RF12 is not limited.

[0211] The size or number of the plurality of processing elements 110 may be determined by the number of the plurality of processing elements PE1 to PE12 and the plurality of register files RF1 to RF12. The size of the plurality of processing elements 110 and the plurality of register files RF1 to RF12 may be implemented in the form of an N×M matrix, where N and M are integers greater than 0.

[0212] The array size of the plurality of processing elements 110 can be designed according to the characteristics of the artificial neural network model run by the NPU 100. Specifically, the memory size of the register file can be determined according to the data size of the artificial neural network model to be operated, the required operation speed, the required power consumption, etc.

[0213] The register files RF1 to RF12 of the NPU 100 are static storage units directly connected to the processing elements PE1 to PE12. The register files RF1 to RF12 may be composed of, for example, triggers and / or latches. The register files RF1 to RF12 may be configured to store MAC operation values ​​of the corresponding processing elements PE1 to PE12. The register files RF1 to RF12 may be configured to provide or receive the side NPU internal memory 120 and weight data and / or node data.

[0214] Register files RF1 to RF12 can also be configured to function as temporary storage for accumulators during MAC operations.

[0215] Figure 7 is a conceptual diagram schematically illustrating an exemplary artificial neural network model.

[0216] Below, the operation of an exemplary artificial neural network model 110 - 10 that may be executed on the NPU 100 is described.

[0217] Figure 7 An exemplary artificial neural network model 110-10 may be Figure 4 or Figure 6 The artificial neural network model may be an artificial neural network that performs various reasoning functions such as object recognition, speech recognition, etc., and is learned in the NPU 100 shown or in a separate machine learning device.

[0218] The artificial neural network model 110 - 10 may be a deep neural network (DNN).

[0219] However, the artificial neural network model 110 - 10 according to the examples of the present disclosure is not limited to deep neural networks.

[0220] However, the artificial neural network model 110 - 10 according to the examples of the present disclosure is not limited to deep neural networks.

[0221] For example, an artificial neural network model can be a model to be learned for performing reasoning such as object detection, object segmentation, image / video reconstruction, image / video enhancement, object tracking, event recognition, event prediction, anomaly detection, density estimation, event search, and measurement.

[0222] For example, the artificial neural network model may be a model such as Bisenet, Shelfnet, Alexnet, Densenet, Efficientnet, EfficientDet, Googlenet, Mnasnet, Mobilenet, Resnet, Shufflenet, Squeezenet, VGG, Yolo, RNN, CNN, DBN, RBM, LSTM, etc. For example, the artificial neural network model may be a model such as a Generative adversarial network (GAN), a transformer, etc. However, the present disclosure is not limited thereto, and new artificial neural network models to be operated on the NPU are constantly being announced.

[0223] However, the present disclosure is not limited to the above-mentioned models. In addition, the artificial neural network model 110-10 may be an integrated model based on at least two different models.

[0224] The artificial neural network model 110 - 10 may be stored in the NPU internal memory 120 of the NPU 100 .

[0225] Reference below Figure 7 , describing the neural network inference process performed by NPU 100 in an exemplary artificial neural network model 110 - 10 .

[0226] The artificial neural network model 110-10 includes an input layer 110-11, a first connection network 110-12, a first hidden layer 110-13, a second connection network 110-1, a second hidden layer 110-15, a third connection network 110-16, and an output layer 110-17. However, the present disclosure is not limited to Figure 7 The artificial neural network model shown. The first hidden layer 110-13 and the second hidden layer 110-15 can also be referred to as multiple hidden layers.

[0227] For example, the input layer 110 - 11 may include ×1 and ×2 input nodes. That is, the input layer 110 - 11 contains information on two input values. Figure 4 or Figure 6 The scheduler in the NPU controller 130 shown may set the memory address storing the information of the input value from the input layer 110-11 at Figure 4 or Figure 6 The NPU internal memory 120 is shown.

[0228] Exemplarily, the first connection network 110 - 12 may include information on 6 weight values ​​connecting each node of the input layer 110 - 11 to each node of the first hidden layer 110 - 13 . Figure 4 or Figure 6 The scheduler in the NPU controller 130 shown sets the memory address storing the information of the weight value for the first connection network 110-12 in the NPU internal memory 120. Each weight value is multiplied by the input node value, and the accumulated value of the multiplied value is stored in the first hidden layer 110-13. Here, the node can be called a feature map.

[0229] The first hidden layer 110 - 13 may exemplarily include nodes a1, a2, and a3. That is, the first hidden layer 110 - 13 may contain information on three node values. Figure 4 or Figure 6 The scheduler in the illustrated NPU controller 130 may set a memory address in the NPU internal memory 120 for storing information of node values ​​of the first hidden layer 110 - 13 .

[0230] The scheduler within the NPU controller 130 may be configured to schedule the order of operations so that the first processing element PE1 performs the MAC operation of the a1 node of the first hidden layer 110-13. The scheduler within the NPU controller 130 may be configured to schedule the order of operations so that the second processing element PE2 performs the MAC operation of the a2 node of the first hidden layer 110-13. The scheduler within the NPU controller 130 may be configured to schedule the order of operations so that the third processing element PE3 performs the MAC operation of the a3 node of the first hidden layer 110-13. Here, the scheduler in the NPU controller 130 may pre-schedule the order of operations so that the three processing elements respectively perform MAC operations simultaneously in parallel.

[0231] The second connection network 110 - 14 may exemplarily include information on 9 weight values ​​for connecting each node of the first hidden layer 110 - 13 with each node of the second hidden layer 110 - 15 . Figure 4 or Figure 6 The scheduler in the NPU controller 130 shown can set a memory address for storing the weight value information of the second connection network 110-14 in the NPU internal memory 120. The weight value of the second connection network 110-14 is multiplied by the node value input from the first hidden layer 110-13, and the accumulated value of the multiplication result is stored in the second hidden layer 110-15.

[0232] The second hidden layer 110-15 may exemplarily include b1, b2, and b3 nodes. That is, the second hidden layer 110-15 may contain information about three node values. The scheduler in the NPU controller 130 may set a memory address in the NPU internal memory 120 for storing information about the node values ​​of the second hidden layer 110-15.

[0233] The scheduler within the NPU controller 130 may be configured to schedule the operation order so that the fourth processing element PE4 performs the MAC operation of the b1 node of the second hidden layer 110-15. The scheduler within the NPU controller 130 may be configured to schedule the operation order so that the fifth processing element PE5 performs the MAC operation of the b2 node of the second hidden layer 110-15. The scheduler within the NPU controller 130 may be configured to schedule the operation order so that the sixth processing element PE6 performs the MAC operation of the b3 node of the second hidden layer 110-15.

[0234] Here, the scheduler in the NPU controller 130 may pre-schedule the operation order so that the three processing elements respectively and simultaneously execute MAC operations in parallel.

[0235] Here, the scheduler in the NPU controller 130 may determine the schedule so that the operation of the second hidden layer 110 - 15 is performed after the MAC operation of the first hidden layer 110 - 13 of the artificial neural network model.

[0236] In other words, the scheduler in the NPU controller 130 may be configured to control the plurality of processing units 100 and the NPU internal memory 120 based on the data locality or structure information of the artificial neural network model.

[0237] The third connection network 110-16 may exemplarily include information of 6 weight values ​​for connecting each node of the second hidden layer 110-15 with each node of the output layer 110-17. The scheduler in the NPU controller 130 may set a memory address for storing the weight value information of the third connection network 110-16 in the NPU internal memory 120. The weight value of the third connection network 110-16 is multiplied by the node value input from the second hidden layer 110-15, and the accumulated value of the multiplication result is stored in the output layer 110-17.

[0238] The output layer 110-17 may exemplarily include y1 and y2 nodes. That is, the output layer 110-17 may include information on two node values. The scheduler in the NPU controller 130 may set a memory address in the NPU internal memory 120 for storing information on the node values ​​of the output layer 110-17.

[0239] The scheduler in the NPU controller 130 may be configured to schedule the operation order so that the seventh processing element PE7 performs the MAC operation of the y1 node of the output layer 110-17. The scheduler in the NPU controller 130 may be configured to schedule the operation order so that the eighth processing element PE8 performs the MAC operation of the y2 node of the output layer 110-15.

[0240] Here, the scheduler in the NPU controller 130 may pre-schedule the operation order so that the two processing elements respectively and simultaneously execute the MAC operations in parallel.

[0241] Here, the scheduler in the NPU controller 130 may determine the schedule so that the operation of the output layer 110 - 17 is performed after the MAC operation of the second hidden layer 110 - 15 of the artificial neural network model.

[0242] In other words, the scheduler in the NPU controller 130 may be configured to control the plurality of processing units 100 and the NPU internal memory 120 based on the data locality or structure information of the artificial neural network model.

[0243] That is, the scheduler in the NPU controller 130 may analyze the structure of the artificial neural network model to be run in the plurality of processing units 100 or receive analysis information. The artificial neural network model may include information about the node value of each layer, regional information about the arrangement data of the layer or information about the structure, and information about each weight value of the connection network connecting the nodes of each layer.

[0244] The scheduler in the NPU controller 130 receives data locality information or structure information of the exemplary artificial neural network model 110-10, so the scheduler in the NPU controller 130 can grasp the order of operations from input to output of the artificial neural network model 110-10.

[0245] Therefore, the scheduler in the NPU controller 130 may set the memory address storing the MAC operation value of each layer in the NPU internal memory 120 in consideration of the scheduling order.

[0246] The NPU internal memory 120 may be configured to save the weight data of the connection network stored in the NPU internal memory 120 during the continuation of the inference operation of the NPU 100. Therefore, there is an effect of reducing memory read and write operations.

[0247] That is, the NPU internal memory 120 may be configured to reuse the MAC operation value stored in the NPU internal memory 120 during the duration of the inference operation.

[0248] Figure 8a It is a diagram illustrating the basic structure of a convolutional neural network.

[0249] refer to Figure 8a , a convolutional neural network can be a combination of one or more convolutional layers, pooling layers, and fully connected layers.

[0250] In the example of the present disclosure, the convolutional neural network has a kernel for each channel for extracting features of the input image of the channel. The kernel can be composed of a two-dimensional matrix and performs a convolution operation when traversing the input data. The size of the kernel can be determined arbitrarily, and the interval (stride) at which the kernel traverses the input data can also be determined arbitrarily. The result of each kernel convolving the entire input data can be called a feature map or an activation map. In the following, the kernel may include a set of weight values ​​or multiple sets of weight values. The number of kernels in each layer can be referred to as the number of channels. The kernel can be referred to as a weight in matrix form, or the kernel can also be referred to as a weight. In the following, the activation map is described as being contained in the feature map. Therefore, the activation map according to the example of the present disclosure can be referred to as a feature map.

[0251] Since the convolution operation is an operation that combines the input data and the kernel, an activation function can be applied afterwards to add non-linearity. When an activation function is applied to a feature map (the result of the convolution operation), it can be called an activation map.

[0252] Specifically, refer to Figure 8a , the convolutional neural network includes at least one convolutional layer, at least one pooling layer and at least one fully connected layer.

[0253] For example, a convolution can be defined by two main parameters: the size of the input data (usually a 1×1, 3×3, or 5×5 matrix) and the depth (number of kernels) of the output feature map. These main parameters can be calculated by convolution. These convolutions start at depth 32, go up to depth 64, and may end at depth 128 or 256. A convolution operation can mean sliding a 3×3 or 5×5 sized kernel over the input image matrix (i.e., input data), multiplying each element in the input image matrix that overlaps with each weight of the kernel, and then adding them all together.

[0254] The activation function is applied to the output feature map thus generated, and the activation map is finally output. In addition, the weights used by the current layer can be passed to the next layer through convolution. The pooling layer can perform a pooling operation that downsamples the output data (i.e., the activation map) to reduce the size of the feature map. For example, the pooling operation may include max pooling and / or average pooling, but is not limited thereto.

[0255] The max pooling operation uses a kernel, a feature map and a sliding kernel, outputting the maximum value in the area of ​​the feature map that overlaps with the kernel. The average pooling operation slides the feature map and the kernel, outputting the average value in the area of ​​the feature map that overlaps with the kernel. As the size of the feature map decreases during the pooling operation, the number of weights of the feature map also decreases.

[0256] The fully connected layer can classify the data output by the pooling layer into multiple classes (i.e., estimated values) and output the classified classes and their scores. The data output by the pooling layer is in the form of a 3D feature map, which can be transformed into a 1D vector and input to the fully connected layer.

[0257] Figure 8b is a comprehensive diagram showing the operation of a convolutional neural network for ease of understanding.

[0258] refer to Figure 8b , exemplarily showing the input image as a two-dimensional matrix with 6×6. In addition, Figure 8b An example is shown in which three channels, namely, channel 1, channel 2 and channel 3 are used.

[0259] First, the convolution operation is explained.

[0260] Input image (in Figure 8b ) in the first node with kernel 1 for channel 1 (in Figure 8b The output feature Figure 1 (exist Figure 8b is displayed in 4×4 size).

[0261] In addition, the input image (in Figure 8b ) in the second node with kernel 2 for channel 2 (in Figure 8b The output feature Figure 2 (exist Figure 8b The size of the display is 4×4).

[0262] In addition, the input image is combined with kernel 3 for channel 3 in the third node ( Figure 8b The output feature Figure 3 (exist Figure 8b The size of the display is 4×4).

[0263] To process each convolution, the processing elements PE1 to PE12 of the NPU 100 are configured to perform a MAC operation.

[0264] Next, the operation of the activation function is explained.

[0265] Activation functions can be applied to features output from convolution operations Figure 1 ,feature Figure 2 and Features Figure 3 (exist Figure 8b Each size is exemplarily shown as 4×4). The output after applying the activation function can be exemplarily of a size of 4×4.

[0266] Next, the pooling operation is described.

[0267] The features output from the above activation function Figure 1 ,feature Figure 2 and Features Figure 3 (Each size is exemplarily shown as 4×4 in FIG. 8b) is input to three nodes. Pooling can be performed by receiving the feature map output from the activation function as input. The above pooling can reduce the size or emphasize specific values ​​within the matrix. Pooling methods include maximum pooling, average pooling, and minimum pooling. Maximum pooling can be used to collect the maximum value within a specific area of ​​the matrix, and average pooling can be used to find the average within a specific area.

[0268] Figure 8b In the example, the feature map of size 4×4 is reduced to 2×2 by pooling.

[0269] Specifically, the first node converts the characteristics of channel 1 Figure 1 After receiving it as input and performing pooling, it is output as a 2×2 matrix, for example. The second node takes the features of channel 2 Figure 2 After receiving it as input and performing pooling, it is output as a 2×2 matrix, for example. The third node takes the features of channel 3 Figure 3 After receiving as input and performing pooling, the output is, for example, a 2×2 matrix.

[0270] The above convolution, activation function and pooling are repeated and finally output with full connection, such as Figure 8a As shown. The corresponding output can again be input to an artificial neural network for recognizing images. However, the present disclosure is not limited to the size of the feature map and the kernel.

[0271] <Video coding for Machines (VCM)> In recent years, with the development of various industries such as surveillance, intelligent transportation, smart cities, smart industry, and smart content, the amount of image or feature map data consumed by machines has continued to increase. In contrast, the traditional video compression methods currently used are technologies developed based on the characteristics of viewer visual perception (Human Vision), and therefore contain unnecessary information, resulting in low efficiency in performing machine tasks. Therefore, it is necessary to study video codec technologies that effectively compress feature maps to improve machine task performance.

[0272] Video Coding for Machine (VCM) technology is being discussed in the Moving Picture Experts Group (MPEG), an international standardization organization for multimedia coding. VCM is a coding technology based on images or feature maps of machine vision rather than human vision.

[0273] <Description of the present disclosure> Figures 9a to 9d An example diagram of an NPU including a VCM encoder and an NPU including a VCM decoder.

[0274] refer to Figure 9a , the first NPU 100a may include a VCM encoder 150a, and the second NPU 100b may include a VCM decoder 150b.

[0275] When the VCM encoder 150a in the first NPU 100a encodes the video and / or feature map and transmits it as a bitstream, the VCM decoder 150b in the second NPU 100b may decode the bitstream and output it. At this time, the VCM decoder 150b in the second NPU 100b may output one or more videos and / or feature maps. For example, the VCM decoder 150b in the second NPU 100b may output a first feature map for machine analysis and output a first image for viewing by a user. The first image may have a higher resolution than the first feature map described above.

[0276] The first NPU 100a outputs a feature map for decentralized operation at a specific layer (eg, the first to twelfth layers) of a specific artificial neural network model, and encodes the feature map in a bit stream through a VCM encoder 150a.

[0277] The second NPU 100b decodes the bit stream using the VCM decoder 150b and outputs a feature map for decentralized calculation of the remaining specific layers (eg, the thirteenth to twenty-eighth layers) of the specific artificial neural network model.

[0278] Reference Figure 9b , the first NPU 100a may include a feature extractor (FeatureExtractor) for extracting a feature map and a VCM encoder 150a.

[0279] The VCM encoder 150a in the first NPU 100a may include a feature encoder. The second NPU 100b may include a VCM decoder 150b. The VCM decoder 150b in the second NPU 100b may include a feature decoder and a video reconstructor. The feature decoder may decode a feature map from a bitstream and output a first feature map for machine analysis. The video reconstructor may regenerate and output a first video from a bitstream for a user to watch.

[0280] The first NPU 100a outputs a feature map for decentralized operation of a specific layer (e.g., the first to twelfth layers) of a specific artificial neural network model, and encodes the feature map in a bit stream through the VCM encoder 150a. The feature extractor can extract and encode specific features separately.

[0281] The second NPU 100b decodes the bitstream using the VCM decoder 150b and outputs a feature map for decentralized operation of the remaining specific layers (eg, the 13th to 28th layers) of the specific artificial neural network model. The feature extractor can decode specific features.

[0282] Reference Fig.9c , the first NPU 100a may include a feature extractor (FeatureExtractor) for extracting a feature map and a VCM encoder 150a.

[0283] The VCM encoder 150a in the first NPU 100a may include a feature encoder. The second NPU 100b may include a VCM decoder 150b. The VCM decoder 150b in the second NPU 100b may include a feature decoder. The feature decoder may decode a feature map from a bitstream and output a first feature map for machine analysis. That is, the bitstream may be encoded only by the feature map, not the image. Specifically, the feature map may be data containing feature information about a specific task based on an image processing machine.

[0284] The first NPU 100a outputs a feature map for decentralized operation of a specific layer (e.g., the first to twelfth layers) of a specific artificial neural network model, and encodes the feature map in a bit stream through the VCM encoder 150a. The feature extractor can extract and encode specific features separately.

[0285] The second NPU 100b decodes the bitstream using the VCM decoder 150b and outputs a feature map for decentralized operation of the remaining specific layers (eg, the 13th to 28th layers) of the specific artificial neural network model. The feature extractor can decode specific features.

[0286] refer to Figure 9d , the first NPU 100a may include a feature extractor (FeatureExtractor) for extracting a feature map and a VCM encoder 150a.

[0287] The VCM encoder 150a in the first NPU 100a may include a feature converter and a video encoder. The second NPU 100b may include a VCM decoder 150b. The VCM decoder 150b in the second NPU 100b may include a video decoder and an inverse converter.

[0288] The first NPU 100a outputs a feature map for decentralized operation of a specific layer (e.g., the first to twelfth layers) of a specific artificial neural network model, and encodes the feature map in a bit stream through the VCM encoder 150a. The feature extractor can extract and encode specific features separately.

[0289] The second NPU 100b decodes the bit stream using the VCM decoder 150b and outputs a feature map for decentralized calculation of the remaining specific layers (eg, the thirteenth to twenty-eighth layers) of the specific artificial neural network model.

[0290] Figure 9b to Figure 9d The feature extractor shown in can correspond to Figure 4 or Figure 6 100 in the NPU 100 shown in FIG. 100. Alternatively, Figure 9b to Figure 9d The feature extractor shown in may correspond to a combination of the plurality of PEs 110 within the aforementioned NPU 100 and the aforementioned SFU.

[0291] As mentioned above, in Figures 9a to 9d The first NPU 100a is shown to include at least a VCM encoder 150a, and the second NPU 100b includes at least a VCM decoder 150b. However, the VCM encoder 150a may be modified to include the first NPU 100a, or the VCM decoder 150b may be modified to include the second NPU 100b.

[0292] The first NPU 100a may process an artificial intelligence operation (e.g., convolution) to generate a feature map of a specific layer. The first NPU 100a may process an artificial intelligence operation, encode the feature map of a specific layer, and transmit it. However, the first NPU 100a may also encode the weight of a specific layer and transmit it.

[0293] The second NPU 100b may receive the encoded feature map of the specific layer. The second NPU 100b may decode the encoded feature map of the specific layer by processing artificial intelligence operations (e.g., deconvolution). However, the second NPU 100b may also decode the weight of the encoded specific layer.

[0294] To process artificial intelligence operations, an artificial neural network model of a specific structure may be used. For example, the NPU may process a convolution operation to extract a feature map of a specific layer. For example, the NPU may process a convolution operation to encode a feature map of a specific layer. For example, the NPU may process a deconvolution operation to decode a feature map of a specific encoding layer.

[0295] The artificial neural network model may have a multi-layer structure, and the artificial neural network model may include a backbone network. The feature graph generated by the artificial intelligence operation of the first NPU 100a may be a feature graph generated from a specific layer of the artificial neural network model of the multi-layer structure. In other words, the feature graph may be at least one feature graph generated from at least one layer of the artificial neural network model of the multi-layer structure. The feature graph generated from a specific layer of the artificial neural network model of the multi-layer structure may be a feature graph suitable for artificial intelligence analysis using a specific machine.

[0296] Fig.10a and 10b is an example diagram showing the location of a bit stream in an artificial neural network model.

[0297] refer to Fig.10a It can be seen that when the first NPU 100a or the VCM encoder receives a video, a separate feature map can be generated for each layer using an artificial neural network model (e.g., a convolutional network model). Fig.10a Feature maps from a fully connected layer, which is the last layer of a convolutional network model, are shown as an example of bitstream transmission.

[0298] Then, the second NPU 100b or the VCM decoder may decode the bitstream including the feature map using a deconvolutional network model.

[0299] On the one hand, reference Fig.10bAn example is shown in which a feature map in a fully connected layer is not transmitted in the form of a bit stream, but a feature map generated in an intermediate layer of an artificial neural network model (e.g., a convolutional network model) is transmitted in the form of a bit stream.

[0300] Fig.11a FIG. 1 is an example of a VCM encoder that compresses a feature map and transmits a bit stream, and a VCM decoder that restores the received bit stream. Fig.11b It is shown Fig.11a An example of generating feature maps P2 to P5 is shown in FIG. And, Fig.11c It shows the detailed display Fig.11a An example diagram of an MSFF execution unit is shown in FIG.

[0301] refer to Fig.11a It can be seen that the VCM encoder 150a (or transmitter) may include a multi-scale feature fusion (MSFF) execution unit 150a-1 and a single-stream feature codec (SSFC) encoder 150a-2. The VCM decoder 150b (or receiver) may include a SSFC decoder 150b-1 and a multi-scale feature restoration (MSFR) execution unit 150b-2.

[0302] After the MSFF execution unit 150a-1 aligns the feature maps P2 to P5 of the P layer, it performs concatenation. The feature maps P2 to P5 of the P layer can be as follows: Fig.11b is generated.

[0303] refer to Fig.11b , feature maps P2 to P5 can be generated using a combination of a first artificial neural network model (e.g., ResNet) and a second artificial neural network model (e.g., Feature Pyramid Network (FPN)). The first artificial neural network model can be called a backbone. The second artificial neural network model can be called a neck or a head.

[0304] like Fig.11bAs shown, when an image is input, the first artificial neural network model (e.g., ResNet) can perform multiple stages (e.g., Stage 1 to Stage 5) from bottom to top. Each stage may include, for example, convolution, batch normalization, and rectified linear unit (ReLU).

[0305] The output of each stage of the first artificial neural network model (e.g., ResNet) can be convolved to generate feature maps C2 to C5 of the C layer. The second artificial neural network model (e.g., FPN) can receive inputs of the feature maps C2 to C5 of the C layer, and accumulate and generate feature maps M2 to M5 of the M layer in a top-down manner. The feature maps M2 to M5 of the M layer are convolved respectively to generate feature maps P2 to P5 of the P layer.

[0306] Reference again Fig.11a , the MSFF execution unit 150a-1 aligns the feature maps P2 to P5 of the P layer and then concatenates them. Specifically, the MSFF execution unit 150a-1 downsamples the sizes of the feature maps P2 to P4 to match the size of the feature map P5 and then concatenates them.

[0307] Referring to FIG. 11c for a more detailed explanation, the MSFF execution unit 150a-1 performs pooling on each of the feature maps P2 to P5. Through the above-mentioned pooling, the feature maps P2 to P5 are respectively reduced to a specific size, such as a 64×64 size or a 32×32 size. Alternatively, the feature map P5 with the smallest size among the feature maps P2 to P5 can make the remaining feature maps (i.e., feature map P2, feature map P3, and P4) have the same size. That is, pooling can be performed on each of the remaining feature maps (i.e., feature map P2, feature map P3, and P4) to make them the same size as feature map P5.

[0308] Then, the MSFF execution unit 150a-1 may connect the feature maps P2 to P5. For example, if each of the feature maps P2 to P5 is a cube with a size of 64×64 and a channel of 256, after connection, it may become a cube with a size of 64×64 and a channel of 1024.

[0309] The above connected feature maps can be passed through SEblock.

[0310] refer to Fig.11a, SEBlock, as a processing operation of SE (Squeeze-and-Exitation), calculates the weight of each channel of the feature map and multiplies the weight by the output feature map of the residual unit. In addition, the MSFF execution unit can use convolution on the output of SEBlock to perform channel-wise reduction. The MSFF execution unit can ultimately send the output F to the SSFC encoder 150a-2. When the SSFC encoder 150a-2 receives the output F from the MSFF execution unit 150a-1, the bit stream can be transmitted after reducing the number of channels. To this end, the SSFC encoder 150a-2 may include a MAC circuit, a batch normalization circuit, and a Tanh function circuit.

[0311] The VCM decoder 150b (i.e., the receiving end) may include an MSFR execution unit (150b-1) and an SSFC decoder 150b-2. When the SSFC decoder 150b-2 receives the bit stream, an output F' may be generated. To this end, the SSFC decoder 150b-2 may include convolution, batch normalization, PReLu, etc.

[0312] The MSFR execution unit 150b-1 of the VCM decoder 150b can restore the feature maps P2' to P5' of the P' layer according to the output F' from the SSFC decoder 150b-2. To this end, the MSFR execution unit (150b-1) can perform upsampling / downsampling and accumulation, etc.

[0313] Fig.12 It is shown Fig.11a An example diagram of a modification of .

[0314] The following only describes Fig.11a Different parts, the same parts will not be repeated, and reference Fig.11a Description content.

[0315] refer to Fig.12 It can be seen that the VCM encoder 150a may further include a compressor (Compressor) 150a-3, and the VCM decoder 150b may further include a decompressor (Decompressor) 150b-3.

[0316] The MSFF execution unit 150a-1 of the VCM encoder 150a may selectively receive Figure 9b to Figure 9d At least one of the multiple feature maps extracted by the feature map extractor shown in . To give a specific example, suppose Figure 9b to Figure 9d The feature map extractor shown extracts Fig.11bThen, the MSFF execution unit 150a-1 of the VCM encoder 150a can selectively receive at least one of the feature maps P2 to P5 as needed. Fig.12 In the example of FIG. 1 , among the feature maps P2 to P5 , it is shown that the MSFF execution unit 150 a - 1 of the VCM encoder 150 a selectively receives the feature map P2 , the feature map P3 , and the feature map P5 .

[0317] The compressor 150a-3 in the VCM encoder 150a can compress the output from the SSFC encoder 150a-2. For the above compression, the compressor 150a-3 in the VCM encoder 150a can perform Versatile Video Coding (VVC) (or H.266), context-based adaptive binary arithmetic coding (context-based adaptive binary arithmetic coding, CABAC), principal component analysis (principal component analysis, PCA) and discrete cosine transform (discrete cosine transform, DCT), etc. Or for the compression, the compressor 150a-3 in the VCM encoder 150a can also use entropy coding technology, etc.

[0318] When compressed in this way, the size of the transmitted bitstream can be greatly reduced, which can save network bandwidth.

[0319] The decompressor 150b-3 of the VCM decoder 150b may decompress the received bit stream. To this end, the decompressor 150b-3 of the VCM decoder 150b may perform VVC (ie, H.266), inverse DCT (IDCT), inverse PCA (IPCA), and the like.

[0320] The MSFR execution unit 150b-1 of the VCM decoder 150b may restore feature maps P2' to P5' based on the output F' from the SSFC decoder 150b-2. In addition, the MSFR execution unit 150b-1 of the VCM decoder 150b may send at least one of the restored feature maps P2' to P5' to each task. For example, the tasks may include image classification, object detection, object tracking, object segmentation, etc. In other words, the MSFR execution unit 150b-1 of the VCM decoder 150b may send feature map P5' from the restored feature maps P2' to P5' to task #1, and send feature map P2', feature map P3', and feature map P4' to task #2.

[0321] Fig.13a is a conceptual diagram illustrating an example of distributed processing between an edge device and a cloud server according to an example of the present disclosure based on the data size of each layer in an exemplary artificial neural network model. Fig.13b It is shown Fig.13a An exemplary table of data size, etc. for each layer in the exemplary artificial neural network model shown in .

[0322] Fig.13a The exemplary artificial neural network model shown is Mobilenet V1. Fig.13a , the horizontal axis represents the layers in the exemplary artificial neural network model, and the vertical axis represents the size of the parameters of each layer. The parameters can be called weights, input feature maps, and output feature maps.

[0323] refer to Fig.13a and Fig.13b In the first layer shown, it can be seen that the size of the output feature map (OFMAP_SIZE) is larger than the size of the input feature map (IFMAP_SIZE).

[0324] However, reference Fig.13a and Fig.13b As shown, the size of the output feature map (OFMAP_SIZE) of the twelfth layer is 50,176 bytes (about 50Kb), which is not only smaller than the size of the input feature map (IFMAP_SIZE) of 200,704 bytes, but also the size of the output feature map (OFMAP_SIZE) of the twelfth layer is smaller than the size of the output feature maps of the previous and next adjacent layers.

[0325] In addition, the sum of the weights from the first to the twelfth layers is very small, only 133Kb. In addition, the size of the output feature map of the twelfth layer is also very small, about 50Kb.

[0326] Therefore, the NPU of each surveillance camera can transmit the output feature map of the twelfth layer in the form of a bit stream. This method of transmitting small feature maps will not cause a large load or congestion on the communication network even if the number of surveillance cameras increases significantly.

[0327] Therefore, if Fig.13a As shown, a method can be considered in which the first to twelfth layers are processed in the NPU of the edge terminal, and then the output feature map of the twelfth layer is transmitted to the cloud server via wired or wireless means.

[0328] The NPU of the cloud server can use the received output feature map of the twelfth layer as the input feature map of the thirteenth layer, and process the thirteenth to twenty-eighth layers.

[0329] At the same time, the output feature map of the twelfth layer may be transmitted in the form of a bit stream through the MSFF execution unit 150a-1 and the SSFC encoder 150a-2, such as Fig.11a That is, the size of the output feature map of the twelfth layer can be compressed by the MSFF execution unit and the SSFC encoder 150a-2 to make it smaller.

[0330] Therefore, in each layer of the artificial neural network model, if a smaller output feature map is selected and transmitted, the computational load can be dispersed while reducing network traffic. In other words, the transmission layer of the artificial neural network model distributed by the first NPU and the second NPU can select a layer with the smallest parameter size or a layer with a relatively small parameter size. Therefore, even if the number of processing elements and the internal memory capacity of the edge NPU are minimized, the required processing speed can be guaranteed.

[0331] Fig.14a yes Fig.13a Example diagram of feature maps of the 9th to 13th layers of the artificial neural network model shown.

[0332] refer to Fig.14a , when Fig.13a When any operation (such as convolution operation) is performed on the 9th layer of the artificial neural network model shown, feature map P9 is output, and feature map P9 is input to the 10th layer. When any operation (such as convolution operation) is performed on the above-mentioned 10th layer, feature map P10 is output, and feature map P10 is input to the 11th layer. When any operation (such as convolution operation) is performed on the above-mentioned 11th layer, feature map P11 is output and input to the 12th layer. Similarly, when any operation (such as convolution operation) is performed on the 12th layer, feature map P12 is output, and feature map P12 is input to the 13th layer.

[0333] Fig.14b It shows that according to Fig.14aAn example diagram of an example of the feature maps P9 to P12 shown in FIG. 1 being compressed, transmitted to a VCM encoder of a bit stream, and a VCM decoder that restores the received bit stream.

[0334] refer to Fig.14b It can be seen that the VCM encoder 150a (or transmitter) may include an MSFF execution unit 150a-1, an SSFC encoder 150a-2, and a compressor 150a-3. And the VCM decoder 150b (or receiver) may include an SSFC decoder 150b-1, an MSFR execution unit 150b-2, and a decompressor 150b-3. In the following, only the description related to Fig.11a and Fig.12 The same parts will not be described repeatedly, and reference Fig.11a and 12 Description.

[0335] The MSFF execution unit 150a-1 of the VCM encoder 150a may selectively receive Figure 9b to Figure 9d At least one of the multiple feature maps extracted by the feature map extractor shown in . To give a specific example, suppose Figure 9b to Figure 9d The feature map extractor shown extracts Fig.14a Then, the MSFF execution unit 150a-1 of the VCM encoder 150a can selectively receive at least one of the feature maps P9 to P12 as needed. Fig.14b In the example of FIG. 1 , among the feature maps P9 to P12 , it is shown that the MSFF execution unit 150 a - 1 of the VCM encoder 150 a selectively receives the feature map P9 , the feature map P10 , and the feature map P12 .

[0336] The MSFR execution unit 150b-1 of the VCM decoder 150b may restore feature maps P9' to P12' according to the output F' from the SSFC decoder 150b-2. Also, the MSFR execution unit 150b-1 of the VCM decoder 150b may send at least one of the restored feature maps P9' to P12' to each task.

[0337] Fig.15a is a diagram showing an example from the perspective of operating an NPU according to the present disclosure.

[0338] refer to Fig.15a It can be seen that the first NPU 100a on the transmitting side includes a plurality of first PEs 110a, a first NPU internal memory 120a, a first NPU controller 130a, a first NPU interface 140a and a VCM encoder 150a.

[0339] As shown in Figure 11, Fig.12 or Fig.14b As shown, the VCM encoder 150a may include an MSFF execution unit 150a-1, an SSFC encoder 150a-2, and a compressor 150a-3.

[0340] Furthermore, the second NPU 100b on the receiving side includes a plurality of second PEs 110b, a second NPU internal memory 120b, a second NPU controller 130b, a second NPU interface 140b, and a VCM decoder 150b.

[0341] As shown in Figure 11, Fig.12 or Fig.14b As shown, the VCM decoder 150b includes a SSFC decoder 150b-1, an MSFR performance unit 150b-2, and a decompressor 150b-3.

[0342] The bold lines shown represent the flow of data (eg, video / image, feature map, or bitstream), and the dashed lines represent the flow of control signals.

[0343] First, when the first NPU 100a on the transmitting side receives a video / image, the first NPU controller 130a within the first NPU 100a stores it in the first NPU internal memory 120a through the first NPU interface 140a.

[0344] Next, the first NPU controller 130a within the first NPU 100a enables the plurality of first PEs 110a to read the video / image from the first NPU internal memory 120a and then perform calculations on each layer of the ANN model.

[0345] The plurality of first PEs 110a store the feature maps outputted by executing the operation results of each layer of the ANN model in the first NPU internal memory 120a according to the control signal from the first NPU controller 130a. According to the feature maps stored according to the layers of the ANN model, the first NPU internal memory 120a may store a plurality of feature maps.

[0346] The VCM encoder 150a may select at least one feature map among a plurality of feature maps stored in the first NPU internal memory 120a as a transmission target according to a control signal from the first NPU controller 130a.

[0347] Then, the selected at least one feature map is converted into a bit stream through the SSFC decoder 150b-1, the MSFR performance unit 150b-2 and the decompressor 150b-3 in the VCM encoder 150a, as shown in FIG. Fig.12 or Fig.14b shown.

[0348] Afterwards, the VCM encoder 150a transmits the bit stream through the first NPU interface 140a according to the control signal from the first NPU controller 130a.

[0349] On the one hand, when the second NPU 100b on the receiving side receives the bit stream, the second NPU controller 130b in the second NPU 100b stores it in the second NPU internal memory 120b through the second NPU interface 140b.

[0350] The VCM decoder 150b in the second NPU 100b on the receiving side reads the bit stream from the second NPU internal memory 120b according to the control of the second NPU controller 130b.

[0351] The read bit stream is restored to at least one feature map after passing through the SSFC decoder 150b-1, the MSFR functional unit 150b-2 and the VCM decoder 150b in the VCM decoder 150b, as shown in FIG. 11, Fig.12 or Fig.14b shown.

[0352] Then, the VCM decoder 150b stores the restored at least one feature map in the second NPU internal memory 120b.

[0353] After the second NPU controller 130b in the second NPU 100b enables the plurality of second PEs 110b to read at least one restored feature map from the second NPU internal memory 120b, each layer of the ANN model is calculated.

[0354] like Fig.14b As shown, when there are multiple machine tasks, there may also be multiple ANN models.

[0355] Fig.15b is a flowchart showing the signal flow between the transmission-side NPU and the reception-side NPU.

[0356] refer to Fig.15b , showing a transmitting-side NPU 100a and a receiving-side NPU 100b.

[0357] The receiving side NPU 100b may send feedback information to the transmitting side NPU 100a. The feedback information may be information that the transmitting side NPU 100a selects at least one feature map as a sending target. For example, the feedback information may include information about the machine task used in the receiving side NPU 100b and the corresponding ANN model. Alternatively, the feedback information may include identification information of at least one layer among the multiple layers in the corresponding ANN model. Alternatively, the feedback information may include identification information of at least one feature map that becomes a sending target among the multiple feature maps generated by the transmitting side NPU 100a.

[0358] Then, the transmitting side NPU 100a may select at least one feature map from the plurality of feature maps as a transmission target based on the feedback information. Feature map restoration reference information including information on the selected at least one feature map may be transmitted to the receiving side NPU 100b. The feature map restoration reference information may include identification information of at least one layer to which the selected at least one feature map from the plurality of layers belongs. In addition, the feature map restoration reference information may include identification information of at least one feature map that becomes a transmission target among the plurality of feature maps generated by the transmitting side NPU 100a.

[0359] Furthermore, the NPU 100a on the sending side encodes and / or compresses the selected feature map and transmits it in the form of a bit stream.

[0360] After receiving the bit stream, the receiving side NPU 100b can restore the feature map by performing decoding and / or decompression. At this time, the feature map can be used to restore the reference information.

[0361] In addition, the receiving-side NPU 100b uses the restored feature map to perform calculations on subsequent layers of the ANN model, so that the machine task can be executed.

[0362] Fig.15c yes Fig.15b A variation of .

[0363] refer to Fig.15c , showing a transmitting-side NPU 100a and a receiving-side NPU 100b.

[0364] The transmitting side NPU 100a may select at least one feature map from the plurality of feature maps as a transmission target, and feature map restoration reference information including information on the selected at least one feature map may be transmitted to the receiving side NPU 100b.

[0365] In addition, after the transmitting side NPU 100a encodes and / or compresses the selected feature map, it can be transmitted in the form of a bit stream.

[0366] After receiving the bit stream, the receiving-side NPU 100b restores the feature map by performing decoding and / or decompression.

[0367] In addition, the receiving-side NPU 100b uses the restored feature map to perform calculations on subsequent layers of the ANN model, so that the machine task can be executed.

[0368] The feature map restoration reference information may include identification information of at least one layer belonging to the selected at least one feature map in multiple layers. In addition, the feature map restoration reference information may include identification information of at least one feature map that becomes a transmission target among multiple feature maps generated by the transmitting side NPU 100a.

[0369] Fig.15d yes Fig.15b Another variation of .

[0370] refer to Fig.15d , showing the sending side NPU 100a, the receiving side NPU 100b and the server 200.

[0371] First, the server 200 may transmit information of an ANN model to be used by the receiving-side NPU 100b to all or one of the transmitting-side NPU 100a and the receiving-side NPU 100b.

[0372] The receiving side NPU 100b may send feedback information. The feedback information may be information that the transmitting side NPU 100a selects at least one feature map as a sending target. For example, the feedback information may include identification information of at least one layer among multiple layers in the ANN model. Alternatively, the feedback information may include identification information of at least one feature map that becomes a sending target among multiple feature maps generated by the transmitting side NPU 100a.

[0373] Then, the transmitting side NPU 100a may select at least one feature map from the plurality of feature maps as a transmission target based on the feedback information. Feature map restoration reference information including information on the selected at least one feature map may be transmitted to the receiving side NPU 100b. The feature map restoration reference information may include identification information of at least one layer to which the selected at least one feature map from the plurality of layers belongs. In addition, the feature map restoration reference information may include identification information of at least one feature map that becomes a transmission target among the plurality of feature maps generated by the transmitting side NPU 100a.

[0374] In addition, after the transmitting side NPU 100a encodes and / or compresses the selected feature map, it can be transmitted in the form of a bit stream.

[0375] After receiving the bit stream, the receiving side NPU 100b can restore the feature map by performing decoding and / or decompression. At this time, the feature map can be used to restore the reference information.

[0376] In addition, the receiving-side NPU 100b uses the restored feature map to perform calculations on subsequent layers of the ANN model, so that the machine task can be executed.

[0377] Fig.16is an exemplary diagram showing an exemplary first scenario to which an example of the present disclosure is applied.

[0378] refer to Fig.16 It can be seen that a surveillance camera can be equipped with an edge NPU. An edge NPU is an NPU specifically designed for low power consumption. A camera can include a SoC containing a lens, an image sensor, an ISP, etc. The SoC can be configured to include an edge NPU or communicate with an edge NPU.

[0379] NPU can extract feature maps from captured images. Usually, due to the large size of feature maps, transmitting them via wired or wireless means requires a lot of network bandwidth and causes network congestion. In addition, it also wastes electricity.

[0380] Therefore, according to an example of the present disclosure, the NPU installed in the surveillance camera can select and transmit the output feature map with a smaller size between the layers of the artificial neural network model. Before transmitting the output feature map, the NPU installed in the surveillance camera can further reduce the size through a multi-scale feature fusion (MSFF) execution unit and a single-stream feature codec (SSFC) encoder, such as Fig.11a shown.

[0381] In this way, each surveillance camera selects a specific layer with a smaller size, such as the twelfth layer, and sends the output feature map of the twelfth layer. Then, each time inference is performed, more than 4 mege bytes of weights can be omitted.

[0382]

Table 1

[0383] Therefore, the NPU installed in the surveillance camera does not need to process all layers of the artificial neural network, thereby simplifying the complexity. For example, the NPU installed in the surveillance camera only needs to process any layer of all layers of the artificial neural network, and thus may include a smaller number of PEs. In addition, the NPU installed in the surveillance camera may include a smaller size of SRAM. In addition, the NPU installed in the camera may not include dynamic random-access memory (DRAM). Through this, the NPU to be installed in the surveillance camera can significantly reduce the manufacturing cost. In addition, the NPU to be installed in the surveillance camera can significantly reduce power consumption.

[0384] In other words, the system can be configured to take into account the performance of the NPU at the edge and only process the layers that the NPU at the edge can process in real time and transmit the next layer to the cloud server.

[0385] For example, if the NPU at the edge processes inference at a minimum of 30 frames per second (FPS), the NPU at the edge can be set to process only the layers that can achieve the target FPS. The minimum FPS can be calculated based on the amount of MAC calculations up to a specific layer.

[0386] For example, if the weight of the NPU at the edge is limited to 150Kb by the capacity of the internal memory, the total weight from the first layer to the twelfth layer is 133Kb, so the NPU at the edge can be set to process only the twelfth layer.

[0387] After each surveillance camera selects a specific layer with a smaller data size, the output feature map of the corresponding specific layer can be transmitted.

[0388] In this way, the output feature maps transmitted from multiple surveillance cameras can be multiplexed in a time-division manner through a multiplexer (MUX) and transmitted to the cloud server.

[0389] The cloud server may include a first memory and an NPU.

[0390] The first memory of the cloud server temporarily stores the multiple output feature maps multiplexed and received in a time-division manner. In addition, the first memory of the cloud server can store weights to be used in subsequent layers that are not processed by the NPU of the surveillance camera. Fig.12 As can be seen from Figure 13, the total weight size from the 13th layer to the 28th layer can be 4075Kb. Although the total weight from the 13th layer to the 28th layer is not small, as the number of surveillance cameras increases, the reuse rate of weights according to batch mode will also increase, so the efficiency will not decrease. In batch mode, if the artificial neural network model used by multiple surveillance cameras is the same, the weights of each layer of the artificial neural network model may be the same. Therefore, the NPU of the cloud server can use one weight to process multiple feature maps transmitted from multiple surveillance cameras. Therefore, one weight can be reused, and the reuse rate of weights can be improved as the number of cameras increases.

[0391] A memory controller controlling the first memory may manage memory addresses so that the plurality of output feature maps may be sequentially read from the first memory.

[0392] According to the control of the memory controller, the first memory transmits the plurality of output feature maps to the NPU in sequence.

[0393] The NPU of the cloud server uses the output feature map of the twelfth layer of each camera read from the first memory as the input feature map of the thirteenth layer. Then, the NPU of the cloud server performs an artificial neural network operation using the weights read from the first memory.

[0394] At the same time, the NPU of the cloud server is processing from the thirteenth layer to the twenty-eighth layer, and the size of the output feature map and the input feature map recorded or read in the first memory can be referred to Fig.12 As can be seen from Figure 13, it is not very large (200kb for each surveillance camera), so the cloud server can handle the neural network calculations for a large number of cameras.

[0395] Fig.17 is an example diagram illustrating an exemplary second example scenario to which an example of the present disclosure is applied.

[0396] like Fig.17 The vehicle camera shown may include an edge NPU. As described above, the NPU installed in the vehicle camera can select and transmit the smaller size of the output feature map in each layer of the artificial neural network model. Before transmitting the output feature map, the NPU installed in the surveillance camera such as Fig.11a As shown in Figure 5, the size is further reduced through a multi-scale feature fusion (MSFF) execution unit and a single-stream feature codec (SSFC) encoder.

[0397] The cloud server may include a server NPU. The NPU of the cloud server uses the output feature map from each vehicle-mounted camera as an input feature map and performs calculations. The NPU of the cloud server generates a vehicle control signal as a result of the calculation and can be transmitted to each vehicle.

[0398] The other details are the same as above, so they will not be explained again.

[0399] Fig.18 is an example diagram showing an example of a bit stream according to an example of the present disclosure.

[0400] refer to Fig.18 It can be seen that the bitstream according to an example of the present disclosure can be divided into a header and a payload. The header can include a serial number (SN) of the bitstream and feature map recovery reference information.

[0401] The payload may include a list of identifiers (IDs) of the included feature graphs. In addition, the payload may include multiple feature graphs identified by the list of identifiers.

[0402] Thus, when multiple feature maps are included in the payload, the list of identifiers can be represented as a bitmap.

[0403] For example, each identifier is represented by 3 bits, and if the payload contains 4 feature maps, the bitmap can be represented by 12 bits (= 3 bits × 4).

[0404] In another example, each identifier is represented by 6 bits. If the payload contains 4 feature maps, the bitmap can be represented by 24 bits (= 6 bits × 4). Fig.16 In the example shown, when multiple feature maps output from multiple surveillance cameras are multiplexed and transmitted, the identifier in the payload can be generated by combining the identifier of the surveillance camera and the identifier of the feature map. For example, if the identifier of the surveillance camera is 2 bits, and if the identifier of the feature map is 4 bits, the identifier in the payload can have a length of 2 bits + 4 bits = 6 bits.

[0405] The embodiments of the present disclosure and the accompanying drawings shown in the present disclosure are merely specific examples presented to easily illustrate the technical content of the present disclosure and to help understand the present disclosure, and are not intended to limit the scope of the present invention. It is obvious to those skilled in the art that other modifications may be implemented in addition to the examples described so far.

[0406] <Summary of Examples Provided in This Instruction> According to an example of the present specification, a neural processing unit (NPU) capable of executing coding is provided. The NPU includes at least one processing element (PE) that performs calculations for multiple layers in an artificial neural network to generate multiple output feature maps; and an encoder that encodes at least one arbitrary output feature map from the multiple output feature maps in a bit stream for transmission.

[0407] The encoder may select at least one arbitrary output feature map from the plurality of output feature maps based on arbitrary information.

[0408] The arbitrary information may be received from a server or other NPU that receives the bitstream.

[0409] The arbitrary information may include identification information of the at least one arbitrary output feature map or identification information of a layer corresponding to the arbitrary output feature map.

[0410] The arbitrary information is generated based on at least one of size information of the output feature map, information of a machine task performed by other neural processing units receiving the bit stream, and information of an artificial neural network model calculated by other neural processing units receiving the bit stream.

[0411] The bitstream is transmitted in order for other NPUs to perform calculations for subsequent layers within the artificial neural network.

[0412] The bitstream may include an identifier of the arbitrary output feature map. Also, the bitstream may be divided into a header and a payload.

[0413] According to an example of the present disclosure, a neural processing unit (NPU) capable of performing decoding is provided. The NPU includes: a decoder that decodes a received bit stream and restores it to at least one arbitrary feature map; and at least one processing element (PE) that performs calculations for an artificial neural network model. The restored at least one arbitrary feature map may be an output feature map of any layer among a plurality of layers within the artificial neural network. The at least one processing element utilizes the restored at least one arbitrary feature map as an input feature map of a subsequent layer of the arbitrary layer.

[0414] The NPU may transmit any information that the arbitrary layer or the at least one feature map can be selected by other neural processing units to which the bitstream is transmitted.

[0415] The arbitrary information may include identification information of the at least one arbitrary feature map or identification information of a layer corresponding to the at least one arbitrary feature map.

[0416] The arbitrary information is generated based on at least one of size information of a feature map, information of a machine task executed in the neural processing unit, and information of the artificial neural network model.

[0417] The bitstream may include an identifier of the arbitrary output feature map.The bitstream may be distinguished into a header and a payload.

[0418] According to an example of the present disclosure, a system is provided. The system may include: a first neural processing unit configured to process a first layer interval of an artificial neural network model including multiple layers; and a second neural processing unit configured to process a second layer interval of the artificial neural network model. The first neural processing unit may be configured to input an output feature map of the last layer of the first layer interval to the second neural processing unit.

[0419] The magnitude of the sum of the weights of the layers in the first layer interval may be relatively smaller than the magnitude of the sum of the weights of the layers in the second layer interval.

[0420] There are multiple first neural processing units, and the second neural processing unit can be configured to process each of the output feature maps output from the first neural processing unit in sequence.

[0421] The second neural processing unit can be configured at least to reuse weights corresponding to the number of the first neural processing units.

[0422] The size of the first memory of the first neural processing unit may be relatively smaller than the second memory of the second neural processing unit.

[0423] The first layer interval and the second layer interval of the artificial neural network model can be determined based on the size of the output feature map of a specific layer.

[0424] The first layer interval and the second layer interval of the artificial neural network model may be determined based on the size of the sum of weights of the respective layer intervals.

[0425] The magnitude of the first weight of the first layer interval of the artificial neural network model may be relatively smaller than the magnitude of the second weight of the second layer interval.

[0426] The claims set forth in this disclosure may be combined in a variety of ways. For example, the technical features of the method claims of this disclosure may be combined to be embodied by a device, and the technical features of the device claims of this disclosure may be combined to be embodied by a method. In addition, the technical features in the method claims of this disclosure may be combined with the technical features in the device claims and embodied by a device, and the technical features in the method claims of this disclosure may be combined with the technical features in the device claims and embodied by a method.

[0427]

Project Base Year

Project unique number

Detailed project number

Department name

Research leader

Project Execution Organization Name

Name of the project management (professional) organization

This year's project period

Project Name

Claims

1. A neural processing unit capable of performing encoding, wherein: The neural processing unit comprises: at least one processing element that performs computations for a plurality of layers within the artificial neural network to generate a plurality of output feature maps; and The encoder encodes at least one arbitrary output feature map from the multiple output feature maps in a bit stream for transmission.

2. The neural processing unit according to claim 1, wherein: The encoder selects at least one arbitrary output feature map from the plurality of output feature maps based on arbitrary information.

3. The neural processing unit according to claim 2, wherein: The arbitrary information is received from a server or other neural processing unit that receives the bitstream.

4. The neural processing unit according to claim 2, wherein: The arbitrary information includes identification information of the at least one arbitrary output feature map or identification information of a layer corresponding to the arbitrary output feature map.

5. The neural processing unit according to claim 2, wherein: The arbitrary information is generated based on at least one of size information of the output feature map, information of a machine task performed by other neural processing units receiving the bit stream, and information of an artificial neural network model calculated by other neural processing units receiving the bit stream.

6. The neural processing unit according to claim 1, wherein: The bit stream is transmitted in order for other neural processing units to perform calculations of subsequent layers within the artificial neural network.

7. The neural processing unit according to claim 1, wherein: The bitstream includes an identifier of the arbitrary output feature map, and The bit stream is divided into a header and a payload.

8. A neural processing unit capable of performing decoding, wherein: The neural processing unit comprises: A decoder, which decodes the received bit stream and restores it into at least one arbitrary feature map; and at least one processing element that performs calculations for the artificial neural network model, The at least one restored arbitrary feature map is an output feature map of any layer among the multiple layers in the artificial neural network. The at least one processing element utilizes the restored at least one arbitrary feature map as an input feature map of a subsequent layer of the arbitrary layer.

9. The neural processing unit according to claim 8, wherein: The neural processing unit transmits any information that the arbitrary layer or the at least one feature map can be selected by other neural processing units to which the bitstream is transmitted.

10. The neural processing unit according to claim 9, wherein: The arbitrary information includes identification information of the at least one arbitrary feature map or identification information of a layer corresponding to the at least one arbitrary feature map.

11. The neural processing unit according to claim 9, wherein: The arbitrary information is generated based on at least one of size information of a feature map, information of a machine task executed in the neural processing unit, and information of the artificial neural network model.

12. The neural processing unit according to claim 8, wherein: The bitstream includes an identifier of the arbitrary output feature map, and The bit stream is divided into a header and a payload.

13. A system comprising: A first neural processing unit is configured to process a first layer interval of an artificial neural network model including a plurality of layers; and A second neural processing unit is configured to process a second layer interval of the artificial neural network model, The first neural processing unit is configured to input the output feature map of the last layer of the first layer interval to the second neural processing unit.

14. The system according to claim 13, wherein: The magnitude of the sum of the weights of the layers in the first layer section is relatively smaller than the magnitude of the sum of the weights of the layers in the second layer section.

15. The system of claim 13, wherein: The first neural processing unit is multiple, The second neural processing unit is configured to sequentially process each of the output feature maps output from the first neural processing unit.

16. The system of claim 13, wherein: The second neural processing unit is at least configured to reuse weights corresponding to the number of the first neural processing units.

17. The system of claim 13, wherein: The size of the first memory of the first neural processing unit is relatively smaller than the second memory of the second neural processing unit.

18. The system of claim 13, wherein: The first layer interval and the second layer interval of the artificial neural network model are determined based on the size of the output feature map of a specific layer.

19. The system of claim 13, wherein: The first layer interval and the second layer interval of the artificial neural network model are determined based on the size of the sum of weights of the respective layer intervals.

20. The system of claim 13, wherein: The magnitude of the first weight of the first layer interval of the artificial neural network model is relatively smaller than the magnitude of the second weight of the second layer interval.

Images