Neural network processing method and device based on nested bit representation

Abstract

A neural network processing method and device based on nested bit representation are disclosed. The method includes: determining the bit width of a first layer of the neural network; obtaining a second weight of the first layer of the neural network by extracting at least one bit corresponding to the determined bit width from each of the first weights of the first layer of a source model corresponding to the first layer of the neural network; and processing input data of the first layer of the neural network by executing the first layer of the neural network based on the obtained second weight.

Description

[0001] This application claims the benefit of Korean Patent Application No. 10-2018-0165585, filed on December 19, 2018, in the Korean Intellectual Property Office, the disclosure of which is incorporated herein in its entirety for all purposes by reference. Technical Field

[0002] The following description relates to a neural network processing method and apparatus based on nested bit representation. Background Art

[0003] For example, technical automation of recognition has been achieved through processor-implemented neural network models, where the processor-implemented neural network models, as a dedicated computing architecture, can provide a computationally intuitive mapping between input patterns and output patterns after extensive training. The training ability to produce such a mapping can be referred to as the learning ability of the neural network. In addition, due to the specialized training, such a specially trained neural network can therefore have a generalization ability to generate relatively accurate outputs with respect to, for example, input patterns for which the neural network may not have been trained.

[0004] In the case where a neural network is used for object recognition (such as image recognition or speech recognition), the weights in the neural network may occupy a large memory space, which reduces the memory efficiency. Summary of the invention

[0005] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter.

[0006] In one general aspect, a neural network processing method includes: determining a bit width of a first layer of the neural network; obtaining a second weight of the first layer of the neural network by extracting at least one bit corresponding to the determined bit width from each of first weights of a first layer of a source model corresponding to the first layer of the neural network; and processing input data of the first layer of the neural network by executing the first layer of the neural network based on the obtained second weights.

[0007] The first weight may have a higher bit precision than the second weight.The second weight may be nested within the first weight.

[0008] The bit width of the first layer of the neural network may be determined based on processing characteristics corresponding to the first layer of the neural network. The processing characteristics may include at least one of a required processing speed, a required processing accuracy, a processing difficulty, or a terminal performance.

[0009] The processing method may also include: determining the bit width of the second layer of the neural network; obtaining a fourth weight of the second layer of the neural network by extracting at least one bit corresponding to the determined bit width of the second layer of the neural network from each of the third weights of the second layer of the source model corresponding to the second layer of the neural network; and processing input data of the second layer of the neural network by executing the second layer of the neural network based on the obtained fourth weight.

[0010] The third weight may have a higher bit precision than the fourth weight. The fourth weight may be nested within the third weight.

[0011] The first layer of the neural network executed based on the second weights can be configured to process a first task based on input data of the first layer, and the second layer of the neural network executed based on the fourth weights can be configured to process a second task different from the first task based on input data of the second layer.

[0012] In another general aspect, a neural network processing method includes: determining a bit width of a first neural network; obtaining second weights of the first neural network by extracting at least one bit corresponding to the determined bit width from each of first weights of a source model; and processing input data of the first neural network by executing the first neural network based on the obtained second weights.

[0013] The processing method may also include: determining a bit width of the second neural network based on a result of processing the input data by the first neural network; obtaining a third weight of the second neural network by extracting at least one bit corresponding to the determined bit width of the second neural network from each of the first weights; and processing the input data of the second neural network by executing the second neural network based on the obtained third weight.

[0014] The first neural network executed based on the second weights may be configured to process a first task based on input data of the first neural network, and the second neural network executed based on the third weights may be configured to process a second task different from the first task based on input data of the second neural network.

[0015] In another general aspect, a neural network training method includes: determining low bit-width weights corresponding to a first layer of the neural network by quantizing high bit-width weights corresponding to the first layer of the neural network; determining a loss value corresponding to the determined low bit-width weights by applying input data to the first layer; and updating the high bit-width weights based on the determined loss value.

[0016] The training method may further include determining a low bit width weight set corresponding to the first layer by quantizing the high bit width weight after the training associated with the high bit width weight is completed.

[0017] The low bit-width weight set may include a first bit-width weight set and a second bit-width weight set, the second bit-width weight set having a lower bit precision than the first bit-width weight set. The second bit-width weight set may be nested in the first bit-width weight set.

[0018] The low bit width weight may include a first weight of a first bit width and a second weight of a second bit width, wherein the first weight of the first bit width has a lower bit precision than the high bit width weight, and the second weight of the second bit width has a lower bit precision than the first weight of the first bit width.

[0019] The step of determining the weight of the low bit width may include: determining a first weight of the first bit width by quantizing the weight of the high bit width; and determining a second weight of the second bit width by extracting at least one bit from each of the determined first weights of the first bit width.

[0020] The step of determining the weight of the low bit width may include: determining the second weight of the second bit width by quantizing the weight of the high bit width; determining the determined second weight of the second bit width as the high bit group of the first weight of the first bit width; and determining the low bit group of the first weight of the first bit width by quantizing the weight of the high bit width.

[0021] The step of updating the high bit width weight may include: updating the high bit width weight based on the statistical information of the loss gradient corresponding to the determined loss value. The step of updating the high bit width weight may also include: calculating the statistical information by assigning a high weight value to the loss gradient corresponding to the weight with a high priority set among the low bit width weights.

[0022] In another general aspect, a neural network processing device includes: a processor; and a memory configured to store instructions that can be read by the processor. When the instructions are executed by the processor, the processor may be configured to: determine a bit width of a first layer of the neural network; obtain a second weight of the first layer of the neural network by extracting at least one bit corresponding to the determined bit width from each of the first weights of the first layer of a source model corresponding to the first layer of the neural network; and process input data of the first layer of the neural network by executing the first layer of the neural network based on the obtained second weights.

[0023] In a further general aspect, a neural network processing device includes: a processor; and a memory configured to store instructions that can be read by the processor. When the instructions are executed by the processor, the processor may be configured to: determine a bit width of a first neural network; obtain a second weight of the first neural network by extracting at least one bit corresponding to the determined bit width from each of the first weights of a source model; and process input data of the first neural network by executing the first neural network based on the obtained second weight.

[0024] In another general aspect, a neural network training device includes: a processor; and a memory configured to store instructions that can be read by the processor. When the instructions are executed by the processor, the processor can be configured to: determine the low bit width weights corresponding to the first layer of the neural network by quantizing the high bit width weights corresponding to the first layer of the neural network; determine the loss value corresponding to the determined low bit width weight by applying input data to the first layer; and update the high bit width weight based on the determined loss value.

[0025] In another general aspect, a method for identifying an object includes: receiving multimedia data as input data of a first layer of a neural network; determining a bit width of the first layer of the neural network based on a first task resource requirement, wherein the first layer includes at least one layer; obtaining a second weight of the first layer of the neural network by extracting at least one bit corresponding to the determined bit width from each of first weights of a first layer of a source model corresponding to the first layer of the neural network; and performing object recognition on the input data of the first layer of the neural network by executing the first layer of the neural network based on the obtained second weights.

[0026] The first weight may have a higher bit precision than the second weight.

[0027] The second weight may be nested within the first weight.

[0028] The higher the resource requirement of the first task, the higher the bit width of the first layer of the neural network is determined.

[0029] The first task resource requirement includes at least one of a required first processing speed, a required first processing accuracy, a first processing difficulty, a first quality of input data, and a first terminal performance.

[0030] The step of determining the bit width of the first layer of the neural network may include: the higher the first quality of the input data, the higher the bit width of the first layer of the neural network is determined to be.

[0031] The input data may be multi-level multimedia data including first-level input data and second-level input data, and the first quality of the first-level input data is lower than the first quality of the second-level input data. The step of performing object recognition may include: determining the bit width of the first layer of the neural network as the first bit width based on the first-level input data, and applying the first-level input data to the first layer of the neural network executed based on the first bit width weight to determine whether there is a predetermined object in the first-level input data; when it is determined that the predetermined object exists in the first-level input data, determining the bit width of the first layer of the neural network as a second bit width higher than the first bit width, and applying the second-level input data to the first layer of the neural network executed based on the second bit width weight to determine the location information of the predetermined object and / or verify the predetermined object.

[0032] The multimedia data may be image data. The higher the resolution of the image data, the higher the first quality.

[0033] The method may also include: determining a bit width of a second layer of the neural network based on a second task resource requirement, wherein the second layer includes at least one layer; obtaining a fourth weight of the second layer of the neural network by extracting at least one bit corresponding to the determined bit width of the second layer of the neural network from each of the third weights of the second layer of the source model corresponding to the second layer of the neural network; and performing object recognition on input data of the second layer of the neural network by executing the second layer of the neural network based on the obtained fourth weight, wherein the input data of the second layer corresponds to output data of the first layer, wherein the second task resource requirement includes at least one of a required second processing speed, a required second processing accuracy, a second processing difficulty, a second quality of the input data, and a second terminal performance, wherein the second task resource requirement is the same as or different from the first task resource requirement.

[0034] The third weight may have a higher bit precision than the fourth weight.

[0035] The fourth weight may be nested within the third weight.

[0036] The first layer of the neural network executed based on the second weights can be configured to process a first task corresponding to the resource requirement of the first task based on the input data of the first layer, and the second layer of the neural network executed based on the fourth weights can be configured to process a second task corresponding to the resource requirement of the second task different from the first task based on the input data of the second layer.

[0037] The multimedia data may include at least one of text, voice, dynamic image, and static image.

[0038] In another general aspect, a method for identifying an object includes: receiving multimedia data as input data of a neural network; determining a bit width of a first neural network based on a first task resource requirement; obtaining a second weight of the first neural network by extracting at least one bit corresponding to the determined bit width from each of first weights of a source model; and performing object recognition on the input data of the first neural network by executing the first neural network based on the obtained second weights.

[0039] The first weight may have a higher bit precision than the second weight.

[0040] The second weight may be nested within the first weight.

[0041] The method may also include: determining whether a second neural network is needed based on a result of processing the input data by the first neural network; in response to determining that the second neural network is needed, determining a bit width of the second neural network based on a second task resource requirement; obtaining a third weight of the second neural network by extracting at least one bit corresponding to the determined bit width of the second neural network from each of the first weights; and performing object recognition on the input data of the second neural network by executing the second neural network based on the obtained third weight.

[0042] The result may indicate whether the object is present in the multimedia data, wherein it is determined that a second neural network is needed based on the result indicating that the object is present in the multimedia data.

[0043] The first weight may have a higher bit precision than the third weight.

[0044] The second weight and the third weight may be nested within the first weight.

[0045] The first neural network executed based on the second weights may be configured to process a first task corresponding to a first task resource requirement based on input data of the first neural network, and the second neural network executed based on the third weights may be configured to process a second task corresponding to a second task resource requirement different from the first task based on input data of the second neural network.

[0046] The first task resource requirement may include at least one of a required processing speed, a required processing accuracy, a processing difficulty, a quality of input data, and a terminal performance.

[0047] The quality of the input data of the first neural network is lower than the quality of the input data of the second neural network, and the bit precision of the second weight is lower than the bit precision of the third weight.

[0048] The first task is to detect whether a predetermined object exists in input data, and the second task is to detect information about at least one of a position and a classification of the object.

[0049] The first task is to detect whether the input data corresponds to speech, and the second task is to detect keywords or verify the speaker from the input data.

[0050] In another general aspect, a training method for a neural network for identifying an object includes: receiving multimedia data as training data for the neural network; determining low bit-width weights corresponding to a first layer of the neural network by quantizing high bit-width weights corresponding to the first layer of the neural network; determining a loss value corresponding to the determined low bit-width weights by applying the multimedia data to the first layer; and updating the high bit-width weights based on the determined loss value.

[0051] The training method may further include determining a low bit width weight set corresponding to the first layer by quantizing the high bit width weight after the training associated with the high bit width weight is completed.

[0052] The low bit-width weight set may include a first bit-width weight set and a second bit-width weight set, the second bit-width weight set having a bit precision lower than the bit precision of the first bit-width weight set, wherein the second bit-width weight set is nested in the first bit-width weight set.

[0053] The low bit width weight may include a first weight of a first bit width and a second weight of a second bit width, wherein the first weight of the first bit width has a bit precision lower than the bit precision of the high bit width weight, and the second weight of the second bit width has a bit precision lower than the bit precision of the first weight of the first bit width.

[0054] The step of determining the weight of the low bit width may include: determining a first weight of the first bit width by quantizing the weight of the high bit width; and determining a second weight of the second bit width by extracting at least one bit from each of the determined first weights of the first bit width.

[0055] The step of determining the weight of the low bit width may include: determining the second weight of the second bit width by quantizing the weight of the high bit width; determining the determined second weight of the second bit width as the high bit group of the first weight of the first bit width; and determining the low bit group of the first weight of the first bit width by quantizing the weight of the high bit width.

[0056] The step of updating the high bit width weight may include updating the high bit width weight based on statistical information of the loss gradient corresponding to the determined loss value.

[0057] The step of updating the high bit width weight may further include calculating statistical information by assigning a high weight value to a loss gradient corresponding to a weight to which a high priority is set among the low bit width weights.

[0058] In another general aspect, a device for identifying an object includes: a processor; and a memory configured to store instructions that can be read by the processor, wherein, when the instructions are executed by the processor, the processor is configured to: receive multimedia data as input data of a first layer of a neural network; determine a bit width of the first layer of the neural network based on a first task resource requirement, wherein the first layer includes at least one layer; obtain a second weight of the first layer of the neural network by extracting at least one bit corresponding to the determined bit width from each of the first weights of the first layer of a source model corresponding to the first layer of the neural network; and perform object recognition on the input data of the first layer of the neural network by executing the first layer of the neural network based on the obtained second weights.

[0059] The processor may be configured to: determine the bit width of a second layer of the neural network based on a second task resource requirement, wherein the second layer includes at least one layer; obtain a fourth weight of the second layer of the neural network by extracting at least one bit corresponding to the determined bit width of the second layer of the neural network from each of the third weights of the second layer of the source model corresponding to the second layer of the neural network; and perform object recognition on input data of the second layer of the neural network by executing the second layer of the neural network based on the obtained fourth weight, wherein the input data of the second layer corresponds to output data of the first layer, wherein the second task resource requirement includes at least one of a required second processing speed, a required second processing accuracy, a second processing difficulty, and a second terminal performance, and wherein the second task resource requirement is the same as or different from the first task resource requirement.

[0060] In another general aspect, a device for identifying an object includes: a processor; and a memory configured to store instructions that can be read by the processor, wherein when the instructions are executed by the processor, the processor is configured to: receive multimedia input data as input data of a neural network; determine a bit width of a first neural network based on a first task resource requirement; obtain a second weight of the first neural network by extracting at least one bit corresponding to the determined bit width from each of the first weights of a source model; and perform object recognition on the input data of the first neural network by executing the first neural network based on the obtained second weights.

[0061] The processor may be configured to: determine whether a second neural network is needed based on a result of processing the input data by the first neural network; determine a bit width of the second neural network based on a second task resource requirement in response to determining that the second neural network is needed; obtain a third weight of the second neural network by extracting at least one bit corresponding to the determined bit width of the second neural network from each of the first weights; and perform object recognition on the input data of the second neural network by executing the second neural network based on the obtained third weight.

[0062] The result may indicate whether the object is present in the multimedia data, wherein the processor is configured to determine that a second neural network is needed based on the result indicating that the object is present in the multimedia data.

[0063] In another general aspect, a training device for a neural network for identifying an object includes: a processor; and a memory configured to store instructions that can be read by the processor, wherein, when the instructions are executed by the processor, the processor is configured to: receive multimedia data as training data for the neural network; determine low bit-width weights corresponding to a first layer of the neural network by quantizing high bit-width weights corresponding to the first layer of the neural network; determine a loss value corresponding to the determined low bit-width weights by applying the multimedia data to the first layer; and update the high bit-width weights based on the determined loss values.

[0064] The low bit width weight may include a first weight of a first bit width and a second weight of a second bit width, wherein the first weight of the first bit width has a bit precision lower than the bit precision of the high bit width weight, and the second weight of the second bit width has a bit precision lower than the bit precision of the first weight of the first bit width.

[0065] The processor may be configured to: determine a first weight of a first bit width by quantizing weights of a high bit width; and determine a second weight of a second bit width by extracting at least one bit from each of the determined first weights of the first bit width.

[0066] The processor can be configured to: determine the second weight of the second bit width by quantizing the weight of the high bit width; determine the determined second weight of the second bit width as the high bit group of the first weight of the first bit width; and determine the low bit group of the first weight of the first bit width by quantizing the weight of the high bit width.

[0067] Other features and aspects will be apparent from the following detailed description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] Figure 1 is a diagram illustrating an example of a nested bit representation-based neural network processing device.

[0069] Figure 2 is a diagram illustrating an example of nested bit representation.

[0070] Figure 3 is a diagram showing an example of weights based on nested bit representation.

[0071] Figure 4 is a diagram showing an example of adjusting the bit width of a weight for every at least one layer.

[0072] Figure 5 is a diagram showing an example of adjusting the bit width of weights for each neural network.

[0073] Figure 6 is a diagram illustrating an example of a training device based on a nested bit representation.

[0074] Figure 7 is a diagram illustrating an example of forward processing for training.

[0075] Figure 8 is a diagram illustrating another example of forward processing for training.

[0076] Fig. 9 is a diagram illustrating an example of backward processing for training.

[0077] Fig.10 is a diagram showing an example of processing a single task using nested bit representations.

[0078] Fig.11 is a diagram illustrating an example of using nested bit representations to process multiple tasks.

[0079] Fig.12 is a diagram illustrating an example of multitasking for face authentication.

[0080] Fig.13 is a diagram illustrating an example of multi-tasking for object classification.

[0081] Fig.14 is a diagram illustrating an example of a device for sensing a face with low power using multitasking.

[0082] Fig.15 is a diagram illustrating an example of a device for recognizing a voice command with low power using multitasking.

[0083] Fig.16 is a diagram illustrating an example of a neural network processing device.

[0084] Fig.17 is a diagram illustrating an example of an electronic device.

[0085] Fig.18 is a flow chart illustrating an example of a neural network processing method.

[0086] Fig.19 is a flowchart illustrating another example of a neural network processing method.

[0087] Fig. 20 is a flow chart illustrating an example of a neural network training method.

[0088] Throughout the drawings and detailed description, unless otherwise described or provided, the same figure reference numerals will be understood to refer to the same elements, features, and structures. The drawings may not be drawn to scale, and the relative sizes, proportions, and depictions of the elements in the drawings may be exaggerated for clarity, illustration, and convenience. DETAILED DESCRIPTION

[0089] The following specific embodiments are provided to help the reader obtain a comprehensive understanding of the methods, devices and / or systems described herein. However, after understanding the disclosure of the present application, various changes, modifications and equivalents of the methods, devices and / or systems described herein will be clear. For example, the order of operations described herein is merely an example and is not limited to the order of operations set forth herein, but the order of operations can be changed as clearly after understanding the disclosure of the present application, except for the operations that must occur in a specific order. In addition, for greater clarity and simplicity, the description of features known after understanding the disclosure of the present application can be omitted.

[0090] The features described herein may be implemented in different forms and are not to be construed as being limited to the examples described herein. Rather, the examples described herein are provided to illustrate only some of the many possible ways to implement the methods, devices, and / or systems described herein that will be apparent after understanding the disclosure of the present application.

[0091] Throughout the specification, when an element such as a layer, region, or substrate is described as being "on," "connected to," or "bonded to" another element, the element may be directly "on," "connected to," or "bonded to" the other element, or one or more other elements may be present between them. In contrast, when an element is described as being "directly on," "directly connected to," or "directly bonded to" another element, no other elements may be present between them. As used herein, the term "and / or" includes any one or any combination of any two or more of the relevant listed items.

[0092] Although terms such as "first," "second," and "third" may be used herein to describe various members, components, regions, layers, or portions, these members, components, regions, layers, or portions are not limited by these terms. Instead, these terms are only used to distinguish one member, component, region, layer, or portion from another member, component, region, layer, or portion. Therefore, without departing from the teachings of the examples described herein, the first member, first component, first region, first layer, or first portion referred to may also be referred to as the second member, second component, second region, second layer, or second portion.

[0093] The terms used herein are only used to describe various examples and are not intended to limit the present disclosure. Unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. The terms "comprise", "include" and "have" indicate the presence of the features, quantities, operations, components, elements and / or combinations thereof set forth, but do not exclude the presence or addition of one or more other features, quantities, operations, components, elements and / or combinations thereof.

[0094] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those generally understood by those skilled in the art based on the understanding of the disclosure of this application and the art to which this disclosure belongs. Unless explicitly defined as such herein, terms (such as those defined in general dictionaries) should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and not to be interpreted in an idealized or overly formal manner.

[0095] Furthermore, in the description of the exemplary embodiments, when it is considered that a detailed description of a structure or function known after understanding the disclosure of the present application will lead to vague interpretation of the exemplary embodiments, such description will be omitted.

[0096] Hereinafter, examples will be described in detail with reference to the accompanying drawings, and like reference numerals in the drawings denote like elements throughout.

[0097] Figure 1 is a diagram showing an example of a neural network processing device based on nested bit representation. The neural network processing device will be simply referred to as a processing device hereinafter.

[0098] Reference Figure 1 , the processing device 100 processes the operation of the neural network 120. For example, the operation of the neural network 120 may include object recognition (e.g., identifying or verifying an object in an input image). It will be understood that "object recognition" here is a broad concept and may include object sensing or detection, object verification or recognition, object classification, recognition of keywords in speech, recognition of the meaning of text included in text, etc., but is not limited thereto. In this case, the processing device may be a device for object recognition. At least a portion of the processing operations associated with the neural network 120 to be described below may be implemented by software, hardware including a neural processor, or a combination of software and hardware.

[0099] The neural network 120 may include all types of neural networks that can operate based on weights. For example, the neural network 120 may be a deep neural network (DNN) including, for example, a fully connected network, a deep convolutional network, and a recurrent neural network (RNN). Although the following description focuses on a convolutional neural network (CNN) as an example of the neural network 120, it may also be applicable to other types of neural networks that operate based on weights.

[0100] The neural network 120 can perform object recognition and object verification by mapping input data and output data in a nonlinear relationship based on deep learning. Deep learning refers to a machine learning method for recognizing images or voices (or speech) from a large data set. Deep learning can be interpreted as an optimized problem-solving process, wherein the problem-solving process locates a point where energy is minimized while training the neural network 120 using prepared training data. Deep learning can be divided into supervised learning and unsupervised learning, wherein, through supervised learning and unsupervised learning, weights corresponding to the architecture or model of the neural network 120 are obtained. Through the weights obtained in this way, input data and output data can be mapped. In CNN, deep learning can be performed based on the weights of the weight kernel used in the convolution layer.

[0101] The neural network 120 may include a plurality of layers, wherein the plurality of layers include an input layer, at least one hidden layer, and an output layer. For example, as shown, the first layer and the second layer included in the neural network 120 may be at least a portion of the plurality of layers. Figure 1 In the example of FIG. 1 , two layers are shown to be included in the neural network 120, but this example is provided for convenience of description, so a larger number of layers may be included in the neural network 120. In one example, the input data may be multimedia data. For example, the multimedia data may include at least one of text, voice, dynamic image, and static image.

[0102] Reference Figure 1 , the source model 110 includes weights of the first bit width X1 to the nth bit width Xn. In this example, when n of Xn increases, the bit width increases. The processing device 100 executes the neural network 120 by selecting the weight of the desired bit width from the source model 110. That is, the selective variable bit width can be applied to the neural network 120. The source model 110 selects the weight of the desired bit width based on the processing characteristics of the neural network 120. For example, the processing characteristics may include at least one of the required processing speed, the required processing accuracy, the processing difficulty, and the terminal performance. In this example, the accuracy may correspond to the false acceptance rate (FAR) or the verification rate (VR). Here, the processing characteristics may also be referred to as task resource requirements. That is, the task resource requirements may include at least one of the required speed of processing tasks, the required accuracy of processing tasks, the difficulty of processing tasks, the quality of input data, and the performance of the terminal of the processing task. For example, the bit width of the first layer of the neural network 120 may be related to the task resource requirements of the first layer (e.g., the first task resource requirements), and the bit width of the second layer of the neural network 120 may be related to the task resource requirements of the second layer (e.g., the second task resource requirements). The higher the task resource requirement, the higher the bit width of the corresponding layer of the neural network.

[0103] For example, in the case where a high processing speed is required or data is processed in a mobile terminal with insufficient resources, the source model 110 may execute the neural network 120 with a first bit width X1 corresponding to a low bit width. For another example, in the case where a high processing accuracy is required or a high processing difficulty is required, the source model 110 may execute the neural network 120 with an nth bit width Xn corresponding to a high bit width. For example, in the case where the quality of the input data is low (for example, the resolution of the input image is low), the source model 110 may execute the neural network 120 with a first bit width X1 corresponding to a low bit width. For another example, in the case where the quality of the input data is high (for example, the resolution of the input image is high), the source model 110 may execute the neural network 120 with an nth bit width Xn corresponding to a high bit width. The bit precision corresponds to the bit width, and thus a variable bit width may indicate a variable bit precision.

[0104] In one example, the bit width of the weight may be adjusted for each at least one layer. For example, the weight corresponding to the first bit width X1 may be applied to the first layer, and the weight corresponding to the second bit width X2 may be applied to the second layer of the neural network 120, so that, as described above, the bit width of the weight may be adjusted for each layer. Alternatively, the weight corresponding to the first bit width X1 may be applied to the first to third layers of the neural network 120, and the weight corresponding to the second bit width X2 may be applied to the fourth and fifth layers of the neural network 120, so that, as described above, the bit width of the weight may be adjusted for each at least two layers.

[0105] In this example, the bit width of the weight to be applied to at least one layer can be determined based on the processing characteristics of the layer. For example, a low bit width weight can be applied to a layer that requires a high processing speed or a layer that processes data in a mobile terminal with insufficient resources or a layer where low-quality data is input. That is, a low bit width weight can be applied to a layer that performs a task that requires a high processing speed or a layer that performs a task in a mobile terminal with insufficient resources. On the contrary, a high bit width weight can be applied to a layer that requires a high processing accuracy or a layer with a high processing difficulty. That is, a high bit width weight can be applied to a layer that performs a task that requires a high processing accuracy or a layer that performs a task with a high processing difficulty.

[0106] In another example, the bit width of the weights may be adjusted for each neural network. For example, a weight corresponding to the first bit width X1 may be applied to the first neural network, and a weight corresponding to the second bit width X2 may be applied to the second neural network. In this example, the bit width of the weights to be applied to each neural network may be determined based on the processing characteristics of the corresponding neural network. For example, a low bit width weight may be applied to a neural network that processes an operation with relatively low difficulty (such as, for example, facial sensing or detection). Conversely, a high bit width weight may be applied to a neural network that processes an operation with relatively high difficulty (such as, for example, facial verification or recognition).

[0107] However, when the weight of the first bit width X1 to the weight of the nth bit width Xn included in the source model 110 occupy independent memory space, memory efficiency may be reduced. Therefore, the bits corresponding to the weight of the first bit width X1 to the weight of the nth bit width Xn may be nested, so that the memory space may be used efficiently. For example, the weight of the first bit width X1 to the weight of the nth bit width Xn included in the source model 110 may be nested. In this example, the weight of the first bit width X1 may be nested in the weight of the second bit width X2 and the weight of the nth bit width Xn, and the weight of the second bit width X2 may be nested in the weight of the nth bit width Xn. Figure 2 Such nested bit representations are described in detail.

[0108] Figure 2 is a diagram showing an example of nested bit representation. Figure 2 In the example, the weights of the CNN weight kernel are represented by nested bits. Such nested bit representation can be applied to other types of neural networks. Figure 2 , there are 3×3 weight kernels 210, 220, and 230. Each element of the weight kernel may correspond to a weight or a weight value.

[0109] In this example, the weight of weight core 210 is represented by X1 bit, the weight of weight core 220 is represented by X2 bit, and the weight of weight core 230 is represented by X3 bit. In this example, X2 is a value greater than X1, and X3 is a value greater than X2. For example, X1 bit can be 2 bits, X2 bit can be 4 bits, and X3 bit can be 8 bits. However, the values ​​of X1 to X3 are not limited to the example values ​​described above, and in addition to X1 to X3, the weight can be represented by more types or fewer types of bit widths.

[0110] The weight cores 210, 220, and 230 may share bit values. For example, the first element at a particular position of the weight core 210 may have a weight value of 01. (2) , the second element at the corresponding position of the weight kernel 220 may have a weight value of 0111 (2) , the third element at the corresponding position of the weight kernel 230 may have a weight value of 01110100 (2) In this example, (2) Indicates binary representation. Therefore, the first to third elements can share the bit value 01 (2) , and the second and third elements can share the bit value 0111 (2) , so the first element to the third element can be represented by being nested.

[0111] As described above, weight cores 210, 220, and 230 may share bit values ​​among corresponding elements and thus may be merged to define a nested bit pattern 250. To store the nested bit pattern 250, only memory space for the weight core 230 may be required, and thus through the nested bit pattern 250, memory efficiency may be improved.

[0112] Return to reference Figure 1 The weight of the first bit width X1 to the weight of the nth bit width Xn included in the source model 110 may correspond to the above reference Figure 2 The nested bit model 250 described above. For example, the weight of the first bit width X1 may correspond to the weight core 210, the weight of the second bit width X2 may correspond to the weight core 220, and the weight of the n-th bit width Xn may correspond to the weight core 230. Therefore, all weights of the first bit width X1 to the n-th bit width Xn may be stored in the memory space for the weight of the n-th bit width Xn.

[0113] The neural network 120 may be trained based on the training data in the training process, and perform an inference operation (such as, for example, classification, recognition, and detection) on the input data in the inference process. The weight of the first bit width X1 to the weight of the nth bit width Xn included in the source model 110 may be determined by the training process. The training may be performed offline or online. Due to the recent introduction of hardware with accelerated training (such as, for example, a neural processor), online training may be feasible.

[0114] The weight kernel may be predetermined, which indicates that the weight kernel is determined before the neural network 120 is started. The neural network 120 being started may indicate that the neural network 120 is ready for inference. For example, the neural network 120 being started may indicate that the neural network 120 is loaded into a memory, or that input data for inference is input into the neural network 120 after the neural network 120 is loaded into a memory.

[0115] Figure 3 is a diagram showing an example of weights based on nested bit representation.

[0116] Reference Figure 3 , the output data 330 is determined based on the operation between the input data 310 and the weight core 320. In this example, the input data 310 has a size of H×W×C, each of the weight cores 320 has a size of K×K×C, and the weight cores 320 are classified into D groups. The output data 330 has a size of H×W×D. In this example, the input data 310, the weight core 320, and the output data 330 may be associated with the first layer.

[0117] The weight kernel 320 includes a weight kernel 116 (10) In this example, 116(10) Represented as 01110100 (2) ,in, (10) Indicates decimal representation, (2) Indicates binary representation. The source model includes 01110100 with the first element (2) The corresponding weight value is 350.

[0118] The processing device determines the bit width of the first layer (hereinafter referred to as the first bit width). For example, the processing device determines the first bit width based on the task resource requirements of object recognition. In addition, the processing device obtains the weight value of the first element by extracting at least one bit corresponding to the first bit width from the weight value 350. For example, as shown, in the case where the determined bit width is 2 bits, the processing device extracts 01 from the weight value 350. (2) To obtain the weight value 351. In the case where the determined bit width is 4 bits, the processing device extracts 0111 from the weight value 350 (2) To obtain the weight value 352. In the case where the determined bit width is 8 bits, the processing device extracts 01110100 from the weight value 350. (2) to obtain the weight value 353. In this example, 2 bits, 4 bits, and 8 bits are provided as example bit widths that can be represented by the source model, and thus the source model can represent a larger number or a smaller number of bit widths.

[0119] The processing device also obtains the weight values ​​of the remaining elements of the weight core 320 by extracting at least one bit corresponding to the first bit width from the weight values ​​of the source model corresponding to the remaining elements of the weight core 320. The processing device processes the input data 310 by executing the first layer based on the obtained weight values. For example, as shown, the processing device performs an operation between the input data 310 and the weight core 320 by executing the first layer, and determines the output data 330 as a result of the execution of the operation. Although Figure 3 The examples focus on a single layer, but by referring to Figure 3 The weights of all individual neural networks can be set by the process described corresponding to the process described.

[0120] exist Figure 3 In the example, 01 (2) Corresponds to 1 (10) ,0111 (2) Corresponds to 7 (10) ,01110100 (2) Corresponds to 116 (10) This indicates that the weight value changes based on bit extraction. Therefore, the model of the nested bit representation may need to be trained to correspond to the change of each weight value based on the change of the bit width. In the following, training the model or source model of the nested bit representation will be described in detail.

[0121] Figure 4 is a diagram showing an example of adjusting the bit width of a weight for every at least one layer.

[0122] Reference Figure 4 , the processing device determines the bit width of the first layer. Figure 4 In the example of , the bit width of the first layer is determined to be 4 bits. The processing device obtains the weight 420 of the first layer by extracting at least one bit corresponding to the bit width of 4 bits from each of the weights 410 of the first layer of the source model corresponding to the first layer. For example, as shown, the processing device extracts at least one bit corresponding to the bit width of 4 bits from the elements 01110100 in the weight 410. (2) Extract the high 4 bits 0111 (2) To obtain the corresponding element in weight 420.

[0123] The processing device obtains the weight 420 of the first layer by extracting 4 bits from each of the remaining elements in the weight 410. By such bit extraction, 01110100 (2) and 0111 (2) The processing device processes input data 430 of the first layer by executing the first layer based on the obtained weights 420 .

[0124] Processing corresponding to the processing performed on the first layer may be performed on the second layer. The processing device determines a bit width for the second layer. The processing device may determine a bit width for the second layer that is the same as the bit width determined for the first layer or a bit width that is different from the bit width determined for the first layer. Figure 4 In the example of , the bit width of the second layer is determined to be 2 bits.

[0125] The processing device obtains the weight 450 of the second layer by extracting at least one bit corresponding to the bit width of 2 bits from each of the weights 440 of the second layer of the source model corresponding to the second layer. For example, as shown, the processing device extracts at least one bit corresponding to the bit width of 2 bits from the element 01000101 in the weight 440. (2) Extract the high 2 bits 01 (2) To obtain the corresponding element in weight 450. The processing device obtains the weight 450 of the second layer by extracting 2 bits from each of the remaining elements in weight 440. In this example, 01000101 (2) and 01 (2) The processing device processes input data 460 of the second layer by executing the second layer based on the obtained weights 450 .

[0126] A process corresponding to the process performed on the first layer and the second layer may be performed on a subsequent layer of the second layer. As described above, the processing device may adjust the bit width of the weight for each layer, or adjust the bit width of the weight for every at least two layers. The bit width of the weight to be applied to at least one layer may be determined based on the processing characteristics of the layer. The processing characteristics may include at least one of the required processing speed, the required processing accuracy, the processing difficulty, and the terminal performance. In other words, the task resource requirements may include at least one of the required processing speed, the required processing accuracy, the processing difficulty, the quality of the input data, and the performance of the terminal processing the task.

[0127] Figure 5 is a diagram showing an example of adjusting the bit width of weights for each neural network.

[0128] Reference Figure 5 , the processing device determines the bit width of the neural network 510. Figure 5 In the example of , the bit width of the neural network 510 is determined to be 2 bits. The processing device obtains the second weight of the neural network 510 by extracting at least one bit corresponding to the bit width of 2 bits from each of the first weights of the source model 500. The processing device processes the input data of the neural network 510 by executing the neural network 510 based on the obtained second weight. For example, a processing device for object recognition receives input data (e.g., multimedia data, wherein the multimedia data may include at least one of text, voice, dynamic image, and static image). The processing device determines the bit width of the neural network 510 to be 2 bits based on the task resource requirement of object recognition. The processing device obtains the second weight of the neural network 510 by extracting at least one bit corresponding to the bit width of 2 bits from each of the first weights of the source model 500. The processing device performs object recognition (e.g., facial sensing) on ​​the input data of the neural network 510 by executing the neural network 510 based on the obtained second weight.

[0129] Processing corresponding to the aforementioned processing performed on neural network 510 may also be performed on neural network 520. The processing device determines the bit width of neural network 520. Figure 5In an example of , the processing device determines the bit width of the neural network 520 based on the result of the processing of the input data by the neural network 510. For example, the processing device determines whether the neural network 520 needs to be executed based on the result of the processing of the input data by the neural network 510. In response to determining that the neural network 520 needs to be executed, the processing device determines the bit width of the neural network 520 based on the task resource requirements of the neural network 520. For example, the neural network 510 may process face sensing, and the neural network 520 may process face detection. In this example, when the neural network 510 successfully senses a face, the processing device may determine that the neural network 520 needs to be executed, and determine the bit width of the neural network 520 to execute the neural network 520. For example, for face sensing with relatively less difficulty, the bit width of the neural network 510 may be determined to be 2 bits, and for face detection with relatively greater difficulty, the bit width of the neural network 520 may be determined to be 4 bits.

[0130] The processing device obtains a third weight of the neural network 520 by extracting at least one bit corresponding to a bit width of 4 bits from each of the first weights of the source model 500. The processing device processes input data of the neural network 520 by executing the neural network 520 based on the obtained third weight. The input data of the neural network 520 may have a relatively high quality or resolution compared to the quality or resolution of the input data of the neural network 510. For example, a low-resolution image may be provided as input data to the neural network 510 in a low-resolution mode. In this example, when the neural network 510 successfully senses a face, a high-resolution image may be provided as input data to the neural network 520 in a high-resolution mode.

[0131] Figure 6 is a diagram showing an example of a training device based on nested bit representation. The training device based on nested bit representation may also be referred to as a neural network processing device, and will be referred to as a training device hereinafter.

[0132] Reference Figure 6 , the training device 600 includes a memory 610 and a processor 620. The memory 610 stores a neural network 615. The processor 620 obtains the neural network 615 from the memory 610 and trains the neural network 615 based on the training data. The memory 610 may store instructions that can be read by the processor 620, and when the instructions stored in the memory 610 are executed by the processor 620, the processor 620 may perform operations for training the neural network 615.

[0133] The training data may include training input and training output. The training input may be input data to be input to the neural network 615, and the input data may be multimedia data, such as text, dynamic images, static images, and voice, etc., but is not limited thereto. As data mapped to the training input, the training output may be, for example, a label output from the neural network 615 in response to the training input being input to the neural network 615. The training data may also be referred to as input data.

[0134] Processor 620 trains neural network 615 so that training output is obtained from training input. Training of neural network 615 may include: training parameters of neural network 615, updating neural network 615, or updating parameters of neural network 615.

[0135] For example, the processor 620 may determine the low bit width weight (also referred to as the low bit width weight) corresponding to the first layer of the neural network 615 by quantizing the high bit width weight (also referred to as the high bit width weight) corresponding to the first layer of the neural network 615, and determine the loss value corresponding to the determined low bit width weight by applying the training input to the first layer. The processor 620 may update the high bit width weight based on the determined loss value. Such aforementioned processing may be repeated for all layers of the neural network 615.

[0136] After the training associated with the high bit width weights is completed, the processor 620 may determine a low bit width weight set by quantizing the high bit width weights. For example, the low bit width weight set may correspond to Figure 1 For example, the low bit-width weight set may include the first bit-width weight to the nth bit-width weight represented as nested.

[0137] The processing of such training of the neural network 615 may include forward processing for determining a loss value and backward processing for updating weights by back propagation. According to one example, weights of various bit widths may be trained by a single training process. The forward processing and backward processing for training the neural network 615 will be described in detail below.

[0138] Figure 7 is a diagram illustrating an example of forward processing for training.

[0139] exist Figure 7In the example of , input data 710, high bit width weight 720, and low bit width weights 730 to 750 are shown. Weight 720 has a higher bit precision than weights 730 to 750. For example, as shown, weights 730, 740, and 750 are represented by a bit width of 8 bits, a bit width of 4 bits, and a bit width of 2 bits, respectively. Weight 720 is represented by a bit width of 32 bits (e.g., a floating point type). The number of bits shown is provided only as an example, and thus weights 720 to 750 may be represented by various other number of bits. Weight 730 has a higher bit precision than weight 740, and weight 740 has a higher bit precision than weight 750. In Figure 7 In this example, input data 710, high bit-width weights 720, and low bit-width weights 730-750 may correspond to a first layer of a neural network.

[0140] The training device determines weights 730 to 750 by quantizing weight 720. The training device determines weight 730 by quantizing weight 720, determines weight 740 by extracting at least one bit from each of weight 730, and determines weight 750 by extracting at least one bit from each of weight 740. Through such processing, weights 730 to 750 can share bits. For example, as shown, weight 730 and weight 740 share 4 bits, and weight 740 and weight 750 share 2 bits.

[0141] The training device determines loss values ​​corresponding to weights 730, 740, and 750, respectively, by applying input data 710 to the first layer (e.g., as shown in Loss 8 、Loss 4 and Loss 2 ). Loss 8 、Loss 4 and Loss 2 The weights 730, 740, and 750 may correspond to the output of the neural network output by the first layer and the subsequent layers of the first layer. In the forward process, the weights 730, 740, and 750 are used to determine the loss value Loss 8 、Loss 4 and Loss 2 In the backward process, the weight 720 is based on the loss value Loss 8 、Loss 4 and Loss 2 For example, the input data 710 may be multimedia data, and the output of the neural network may be the result of object recognition. The backward processing will be described in detail below.

[0142] When the training weights 720 are completed through such the aforementioned processing, the source model is determined by quantizing the trained weights 720 to the bit widths corresponding to the weights 730 to 750. For example, a first weight set having a bit width of 8 bits can be determined by quantizing the trained weights 720, a second weight set having a bit width of 4 bits can be determined by extracting at least one bit from each of the weights of the first weight set, and a third weight set having a bit width of 2 bits can be determined by extracting at least one bit from each of the weights of the second weight set. The number of bits described above is provided only as an example, and thus the weight set may be represented by various number of bits corresponding to the weights 730 to 750. The determined weight set may correspond to Figure 1 The source model 110.

[0143] The weight 720 may be optimized based on the weights 730 to 750, and the weight set may have a bit width corresponding to the weights 730 to 750. Therefore, although the weight value changes due to the bit extraction, each weight value may be maintained in an optimized state. In addition, although the weights of various bit widths are trained through a single training process, low bit width weights with different bit widths may affect each other, so the weights may be maintained in an optimized state.

[0144] Figure 8 is a diagram illustrating another example of forward processing for training.

[0145] exist Figure 8 In the example of , input data 810, high bit width weight 820, and low bit width weights 830 to 850 are shown. Weight 820 has a bit precision higher than the bit precision of weights 830 to 850. For example, as shown, weights 830, 840, and 850 are represented by a bit width of 2 bits, a bit width of 4 bits, and a bit width of 8 bits, respectively. Weight 820 is represented by a bit width of 32 bits (e.g., a floating point type). The number of bits shown is provided only as an example, and thus weights 820 to 850 may be represented by various other number of bits. Weight 830 has a bit precision lower than the bit precision of weight 840, and weight 840 has a bit precision lower than the bit precision of weight 850. In Figure 8 In this example, input data 810, high bit-width weights 820, and low bit-width weights 830-850 may correspond to a first layer of a neural network.

[0146] The training device determines weights 830 to 850 by quantizing weight 820. The training device determines weight 830 by quantizing weight 820, and determines weight 840 based on weights 820 and 830. For example, the training device may determine weight 830 as a high-order group of weight 840, and determine a low-order group of weight 840 by quantizing weight 820. The training device may perform quantization so that weight 840 has a value corresponding to weight 820 while the high-order group of weight 840 is fixed.

[0147] Through a process corresponding to the process of determining weight 840 described above, the training device determines weight 850 based on weight 820 and weight 840. For example, the training device may determine weight 840 as the upper group of weight 850, and determine the lower group of weight 850 by quantizing weight 820. Figure 8 In the example of (2) Determine the weight value 0111 included in the weight 840 (2) , based on the weight value 0111 included in the weight 840 (2) To determine the weight value 01110100 included in weight 850 (2) .

[0148] The training device determines loss values ​​corresponding to weights 830, 840, and 850, respectively, by applying input data 810 to the first layer (e.g., as shown in Loss 2 、Loss 4 and Loss 8 ). Loss 2 、Loss 4 and Loss 8 The weights 820 may correspond to the outputs of the neural network output by the first layer and the subsequent layers of the first layer. In the backward process, the weights 820 are based on the loss value Loss 2 、Loss 4 and Loss 8 When the training weight 820 is completed through such a foregoing process, the source model is determined by quantizing the trained weight 820 to a bit width corresponding to the weights 830 to 850. Figure 7 The forward processing described can be applied to reference Figure 8 Describe the forward processing.

[0149] In reference Figure 7 In the forward process described, due to bit extraction, the bit values ​​of low-bitwidth weights can depend on the bit values ​​of high-bitwidth weights, so linear quantization can be based on reference Figure 7 The forward processing described in the reference Figure 8 In the forward processing described, only a portion of the bit value of the low bit width weight can be extracted from the bit value of the high bit width weight, and the remaining portion of the bit value of the low bit width weight can be determined by quantization based on the high bit width weight. Figure 8 The forward process described above performs nonlinear quantization. Therefore, Figure 8 The bit sharing between weights 830 to 850 in the example of Figure 7 In the example of , bits are shared between weights 730 and 750.

[0150] Fig. 9 is a diagram illustrating an example of backward processing for training.

[0151] exist Fig. 9 In the example of , a high bit width weight 910 and low bit width weights 920 to 940 are shown. Weight 910 has a bit precision higher than the bit precision of weights 920 to 940. Weight 920 has a bit width of n bits, weight 930 has a bit width of n-1 bits, and weight 940 has a bit width of 1 bit. Fig. 9 In the example of , the weight having a bit width of 2 bits and the weight having a bit width of n-2 bits are omitted. Fig. 9 In the example of , each interval between the bit widths is shown as 1 bit, but the intervals between the bit widths can be defined as various numbers of bits. In addition, the intervals can be the same or different from each other. Fig. 9 In the example of , high bit width weight 910 and low bit width weight 920 to 940 may correspond to a first layer of a neural network.

[0152] Training equipment based on loss value Loss n 、Loss n-1 and Loss 1 To update the weight 910. The training device determines the loss value Loss n 、Loss n-1 and Loss 1 The corresponding loss gradient or loss gradient, and update the weight 910 based on the statistical information of the determined loss gradient. Here, the loss gradient may include The first loss gradient represented by The second loss gradient is represented by , where i represents an integer value between 1 and n, L, w and w q Represents loss, weight, and quantized weight respectively. For example, the loss value Loss n 、Loss n-1 and Loss 1 May correspond to the result of object recognition.

[0153] The training device updates the weights based on the statistical information of the second loss gradient 910. For example, the statistical information may be the average value of the second loss gradient represented by Equation 1, or the weighted average value of the second loss gradient represented by Equation 2.

[0154] [Equation 1]

[0155]

[0156] [Equation 2]

[0157]

[0158] In Equation 2, α represents a weighted value. The training device may calculate statistical information by assigning a high weighted value to a loss gradient corresponding to a weight set with a high priority among weights 920 to 940. For example, a higher weighted value may be assigned to a weight with a higher bit precision, or a higher weighted value may be assigned to a weight with a lower bit precision. Here, the weighted value may be determined based on the result of the training or the purpose of the training.

[0159] When the training weight 910 is completed through such the aforementioned process, the source model is determined by quantizing the trained weight 910 to a bit width corresponding to the weights 920 to 940 .

[0160] Fig.10 is a diagram showing an example of processing a single task using nested bit representations.

[0161] Reference Fig.10 , a neural network 1020 is executed based on a weight 1021 having a bit width of 2 bits, a weight 1022 having a bit width of 4 bits, and a weight 1023 having a bit width of 8 bits. The weights 1021 to 1023 may correspond to a nested bit model. The number of bits shown is provided only as an example, and thus the weights 1021 to 1023 may be represented by various other bit numbers. Although in Fig.10 Although three types of bit widths are shown in the example of FIG. 1 , the neural network 1020 may also be performed based on weights having two or four or more types of bit widths.

[0162] The neural network 1020 generates output data 1030 by processing the input data 1010. When the input data 1010 is input, the neural network 1020 selects a weight corresponding to at least one bit width from the weights 1021 to 1023 to process the input data 1010. For example, the neural network 1020 for object recognition generates output data 1030 by processing the input data 1010. The input data 1010 may be text, voice, dynamic image, static image, etc., but is not limited thereto. When the input data 1010 is input, the neural network 1020 selects a weight corresponding to at least one bit width from the weights 1021 to 1023 based on the task resource requirements of object recognition to perform object recognition on the input data 1010.

[0163] The neural network 1020 is executed based on weights having different bit widths for each at least one layer. For example, the first layer of the neural network 1020 may be executed based on weights of the first bit width, and the second layer of the neural network 1020 may be executed based on weights of the second bit width. Alternatively, the neural network 1020 may be executed based on weights having different bit widths for each network. For example, after the neural network 1020 is executed based on weights of the first bit width, the neural network 1020 may be executed again based on weights of the second bit width. In this example, the neural network 1020 executed based on the weights of the first bit width may be referred to as the first neural network, and the neural network 1020 executed based on the weights of the second bit width may be referred to as the second neural network.

[0164] exist Fig.10 In the example of , the neural network 1020 processes a single task based on weights 1021 to 1023 of various bit widths. In this example, the bit precision corresponding to the bit width of the weights 1021 to 1023 may affect the accuracy of processing the single task. For example, the output data 1030 generated based on the weight 1023 with a relatively high bit width may be more accurate than the output data 1030 generated based on the weight 1021 corresponding to a relatively low bit width. For example, such processing accuracy may correspond to FAR or VR. Therefore, the bit width may be determined based on the processing accuracy required for the single task.

[0165] Fig.11 is a diagram illustrating an example of using nested bit representations to process multiple tasks.

[0166] Reference Fig.11 , when the multi-level input data 1110 is applied to the neural network 1120, the neural network 1120 processes the multi-level input data 1110 to generate multi-level output data 1130. The multi-level input data 1110 may be multi-level image data or multi-level speech data. For example, the input data 1111 as the first-level input data may correspond to a low-resolution image, the input data 1112 as the second-level input data may correspond to a medium-resolution image, and the input data 1113 as the third-level input data may correspond to a high-resolution image. For another example, the input data 1111 may correspond to a low-resolution image, the input data 1112 may correspond to a high-resolution image, and the input data 1113 may correspond to a cropped image.

[0167] The neural network 1120 is performed based on a weight 1121 having a first bit width (e.g., a bit width of 2 bits), a weight 1122 having a second bit width (e.g., a bit width of 4 bits), and a weight 1123 having a third bit width (e.g., a bit width of 8 bits). The weights 1121 to 1123 may be trained to perform different tasks. For example, the weight 1121 may be trained to sense a face in an image, the weight 1122 may be trained to detect the position of a face in an image, and the weight 1123 may be trained to verify a face in an image. In this example, the output data 1131 may indicate whether a face is sensed, the output data 1132 may indicate information about the position of a face, and the output data 1133 may indicate whether a face is verified.

[0168] For another example, weight 1121 may be trained to sense an object in an image, weight 1122 may be trained to detect the location of an object in an image and identify a superclass or coarse-grained class of the object, and weight 1123 may be trained to identify a subclass or fine-grained class of an object. In this example, output data 1131 may indicate whether an object is sensed, output data 1132 may indicate information about the location of the object and the superclass of the object, and output data 1133 may indicate a subclass of the object. Multi-level output data 1130 may be used for subsequent tasks. For example, output data 1131 generated based on weight 1121 corresponding to a first task may be used to trigger a second task.

[0169] Weights 1121 to 1123 may correspond to a nested bit model. Weights 1121 to 1123 may include shared bits and individual bits. For example, a 2-bit weight of 01 (2) and 4-bit weight 0111 (2) There may be a shared bit 01 and a unique bit 11. Neural network 1120 may be trained to perform multiple tasks through unique bits. For example, when weight 1121 is trained to sense a face, weight 1122 and weight 1123 may be trained to detect the position of the face through the unique bit.

[0170] Neural network 1120 may be executed based on weights having different bit widths for each at least one layer, or based on weights having different bit widths for each network. For example, in the case where multiple tasks need to be processed in one network, neural network 1120 may be executed based on weights having different bit widths for each at least one layer. In this example, a first layer group of neural network 1120 may perform facial sensing based on weight 1121, a second layer group of neural network 1120 may perform facial position detection based on weight 1122, and a third layer group of neural network 1120 may perform facial verification based on weight 1123. In this example, each layer group may include at least one layer.

[0171] For another example, in the case of using multiple networks to process multiple tasks, the neural network 1120 may be executed based on weights having different bit widths for each network. For example, the neural network 1120 may perform object sensing by being executed based on the weight 1121, perform object position detection and superclass recognition by being executed based on the weight 1122, and perform subclass recognition by being executed based on the weight 1123. In this example, the neural network 1120 may be referred to as a first neural network, a second neural network, and a third neural network corresponding to these tasks, respectively.

[0172] For a more detailed description of the neural network 1120, please refer to the above reference Fig.10 A description of neural network 1020 is provided.

[0173] Fig.12 is a diagram illustrating an example of multitasking for face authentication.

[0174] Reference Fig.12 , the neural network 1220 processes the multi-stage input data 1210. The processing device executes at least one layer of the neural network 1220 based on the weight 1221 having a bit width of 2 bits. At least one layer of the neural network 1220 executed based on the weight 1221 outputs face sensing information indicating whether there is a face in the low-quality image 1211. The processing device performs operation 1230 based on the output generated by at least one layer of the neural network 1220 based on the weight 1221. If the face does not exist, the processing device applies the low-quality image 1211 to the at least one layer of the neural network 1220 executed based on the weight 1221 again.

[0175] When the face exists, the processing device executes at least one layer of the neural network 1220 based on the weight 1222 having a bit width of 4 bits, and applies the high-quality image 1212 to the at least one layer of the neural network 1220 executed based on the weight 1222. At least one layer of the neural network 1220 executed based on the weight 1222 detects the position of the face in the high-quality image 1212 to output information indicating the position of the face in operation 1240.

[0176] In one example, the processing device generates a cropped image 1213 from the high-quality image 1212 based on the position of the face. The processing device executes at least one layer of the neural network 1220 based on the weight 1223 having a bit width of 8 bits, and applies the cropped image 1213 to at least one layer of the neural network 1220 executed based on the weight 1223. At least one layer of the neural network 1220 executed based on the weight 1223 performs facial verification through the cropped image 1213, and outputs a result of the facial verification in operation 1250.

[0177] For a more detailed description of neural network 1220, please refer to the above reference Fig.10 The description of neural network 1020 provided above is Fig.11 A description of neural network 1120 is provided.

[0178] Fig.13 is a diagram illustrating an example of multi-tasking for object classification. Fig.13 , the multi-level input data 1310 includes a low-resolution image 1311, a high-resolution image 1312, and a cropped image 1313. At least one layer of the neural network 1320 executed based on the weight 1321 having a bit width of 2 bits outputs object sensing information indicating whether an object exists in the low-resolution image 1311, and whether an object exists in the low-resolution image 1311 is determined based on the object sensing information in operation 1330. In operation 1340, at least one layer of the neural network 1320 executed based on the weight 1322 having a bit width of 4 bits detects a superclass of an object and a position of the object from the high-resolution image 1312, and outputs information indicating the superclass and position of the object. In operation 1350, at least one layer of the neural network 1320 executed based on the weight 1323 having a bit width of 8 bits classifies an object through the cropped image 1313, and outputs information about a subclass or a fine-grained class of the object. For a more detailed description of the neural network 1320, reference may be made to the above reference. Fig.10 The description of the neural network 1020 provided above refers to Fig.11 The description of the neural network 1120 provided and the references above Fig.12 A description of neural network 1220 is provided.

[0179] Fig.14 is a diagram illustrating an example of a device for sensing a face with low power using multitasking.

[0180] Reference Fig.14 , the image sensor 1400 includes pixels 1410, logic 1420, and an interface (I / F) 1430. The image sensor 1400 can operate at all times. In order to perform such an always-on operation, low power consumption may be required. Therefore, the image sensor 1400 can reduce power consumption by operating in a low-resolution mode or a high-resolution mode based on whether a face is sensed using a nested bit model.

[0181] In the low-resolution mode, the pixel 1410 activates only a portion of the pixel 1410 to generate a low-resolution image, and the logic 1420 executes the low-bitwidth network 1421 based on the low-bitwidth weight corresponding to the low-resolution mode to sense whether there is a face in the low-resolution image. For example, the low bitwidth can be 2 bits or 4 bits.

[0182] When a face is present in the low-resolution image, the operating mode of the image sensor 1400 changes from the low-resolution mode to the high-resolution mode. In the high-resolution mode, the pixel 1410 activates a portion of the pixel 1410 that is larger than the portion of the pixel 1410 activated in the low-resolution mode to generate a high-resolution image. For example, in the high-resolution mode, the pixel 1410 may activate the entire area of ​​the pixel 1410. The logic 1420 executes the high-bitwidth network 1422 based on the high-bitwidth weights corresponding to the high-resolution mode to detect a region of interest (ROI) corresponding to the face from the high-resolution image. For example, the high bitwidth may be 4 bits or 8 bits.

[0183] The interface 1430 transmits a wake-up interrupt and the cropped ROI image to the application 1450. The application 1450 wakes up in response to the wake-up interrupt and then performs user authentication using the cropped ROI image.

[0184] The low bit width network 1421 and the high bit width network 1422 may be executed based on a nested bit model. The nested bit model may occupy less memory space, so the nested bit model may be run in the logic 1420 even when the logic 1420 has a low capacity memory, and the image sensor 1400 may therefore provide a low resolution mode and a high resolution mode. In the low resolution mode, only a portion of the pixel 1410 may be activated, and a low resolution image with a small processing capability may be processed in the logic 1420, so the power to be consumed in the low resolution mode may be reduced. Therefore, by the image sensor 1400 based on the nested bit model, facial sensing or detection may always be performed at low power. In addition, the application 1450 may be run after the facial ROI is detected, so the power consumed by the operation of the application 1450 may be minimized.

[0185] Fig.15 is a diagram illustrating an example of a device for recognizing a voice command with low power using multitasking.

[0186] Reference Fig.15 , the microphone module 1500 includes a microphone 1510 and a digital signal processor (DSP) 1520. The microphone module 1500 may operate all the time. The microphone module 1500 may use a nested bit model to reduce power consumption for such an always-on operation.

[0187] Microphone 1510 outputs an audio signal in such a constant working state. DSP 1520 executes low bit width network 1521 based on low bit width weight, and uses low bit width network 1521 to determine whether the audio signal corresponds to voice or speech. For example, the low bit width can be 2 bits. When the audio signal is determined to correspond to voice or speech, DSP 1520 executes medium bit width network 1522 based on medium bit width weight. For example, the medium bit width can be 4 bits. DSP 1520 detects keywords from the audio signal using medium bit width network 1522. Keywords can be commands for waking up application 1550 (e.g., "Hi, XXX" and "Okay, XXX").

[0188] When a keyword is detected from the audio signal, the DSP 1520 executes a high bit width network 1523 based on a high bit width weight. For example, the high bit width may be 8 bits. The DSP 1520 verifies the speaker using the high bit width network 1523. When the speaker is verified, or when the speaker corresponds to a registered user, a wake-up interrupt and an audio signal in a speech interval are sent to the application 1550. The application 1550 wakes up in response to the wake-up interrupt, and then performs voice or speech recognition using the audio signal in the speech interval.

[0189] The low bit width network 1521, the medium bit width network 1522, and the high bit width network 1523 may be executed based on a nested bit model. The nested bit model may occupy less memory space, so the nested bit model may be run in the DSP 1520 even when the DSP 1520 has a low-capacity memory. The DSP 1520 may operate the low bit width network 1521 normally, and may operate the medium bit width network 1522 and the high bit width network 1523 only when the audio signal corresponds to voice or speech and when a keyword is detected, so as to reduce power consumption. In addition, the application 1550 may be run after the speaker is authenticated, so the power consumed by the operation of the application 1550 may be minimized.

[0190] Fig.16 is a diagram illustrating an example of a processing device.

[0191] Reference Fig.16, the processing device 1600 receives input data. The processing device 1600 uses a neural network to process the input data. For example, the input data may be multimedia data. For example, the processing device 1600 may use a neural network to process identification or verification of the input data. The database (DB) 1630 may store a source model.

[0192] The processing device 1600 may perform at least one of the operations for facial verification described herein and provide a result of the facial verification to the user. Fig.16 In the example of , the processing device 1600 includes at least one processor 1610 and a memory 1620. The memory 1620 may be connected to the processor 1610 and store instructions that can be executed by the processor 1610, and data processed by the processor 1610 or data to be processed by the processor 1610. The memory 1620 may include a non-transitory computer-readable medium (e.g., a high-speed random access memory (RAM)) and / or a non-volatile computer-readable storage medium (e.g., at least one disk storage device, a flash memory device, and other non-volatile solid-state memory devices).

[0193] The processor 1610 can execute instructions to perform the above reference Figures 1 to 15 At least one of the operations described. For example, the processor 1610 may determine the bit width of the first layer of the neural network, obtain the second weight of the first layer of the neural network by extracting at least one bit corresponding to the determined bit width from each of the first weights of the first layer of the source model corresponding to the first layer of the neural network, and process the input data of the first layer of the neural network by executing the first layer of the neural network based on the obtained second weight. For example, the processor 1610 may receive multimedia data as input data, determine the bit width of the first layer of the neural network based on the task resource requirement of object recognition, obtain the second weight of the first layer of the neural network by extracting at least one bit corresponding to the determined bit width from each of the first weights of the first layer of the source model corresponding to the first layer of the neural network, and process the input data of the first layer of the neural network by executing the first layer of the neural network based on the obtained second weight to perform object recognition. Alternatively, the processor 1610 may determine the bit width of the first neural network, obtain the second weight of the first neural network by extracting at least one bit corresponding to the determined bit width from each of the first weights of the source model, and process the input data of the first neural network by executing the first neural network based on the obtained second weight.

[0194] Fig.17 is a diagram illustrating an example of an electronic device.

[0195] The electronic device 1700 may receive an input image and process the operation of a neural network associated with the input image. For example, the operation of the neural network may include identifying or verifying an object in the input image. The electronic device 1700 may use the above-mentioned source model to process the operation of the neural network. The electronic device 1700 may include the above-mentioned reference Figures 1 to 16 The processing device described above, or performs the above reference Figures 1 to 16 Describes the functionality of a processing device.

[0196] Reference Fig.17 , the electronic device 1700 includes a processor 1710, a memory 1720, a camera 1730, a storage device 1740, an input device 1750, an output device 1760, and a network interface 1770. The processor 1710, the memory 1720, the camera 1730, the storage device 1740, the input device 1750, the output device 1760, and the network interface 1770 can communicate with each other through a communication bus 1780.

[0197] The processor 1710 may execute functions and instructions in the electronic device 1700. For example, the processor 1710 may process instructions stored in the memory 1720 or the storage device 1740. The processor 1710 may execute the above-mentioned Figures 1 to 16 At least one of the operations described.

[0198] The memory 1720 may store information to be used for processing the operation of the neural network. The memory 1720 may include a computer-readable storage medium or a computer-readable storage device. The memory 1720 may store instructions to be executed by the processor 1710 and store relevant information while the electronic device 1700 is executing a software program or application.

[0199] The camera 1730 may capture still images, video images, or both. The camera 1730 may capture an image of a facial region input by a user for facial verification. The camera 1730 may also provide a three-dimensional (3D) image including depth information of an object.

[0200] The storage device 1740 may include a computer-readable storage medium or a computer-readable storage device. The storage device 1740 may store a DB including information to be used for processing a neural network (such as, for example, a source model, a weight kernel, and a weight value). For example, compared with the memory 1720, the storage device 1740 may store a larger amount of information for a longer period of time. For example, the storage device 1740 may include a magnetic hard disk, an optical disk, a flash memory, a floppy disk, and other types of non-volatile memory known in the relevant technical field.

[0201] The input device 1750 may receive input from the user through a conventional input method (e.g., a keyboard and a mouse) and / or a new input method (e.g., a touch input, a voice input, and an image input). For example, the input device 1750 may include a keyboard, a mouse, a touch screen, a microphone, and other devices that may detect input from the user and send the detected input to the electronic device 1700.

[0202] The output device 1760 can provide the output of the electronic device 1700 to the user through visual, auditory or tactile channels. For example, the output device 1760 may include a display, a touch screen, a speaker, a vibration generator, and other devices that can provide output to the user. The network interface 1770 can communicate with external devices through a wired network or a wireless network.

[0203] Fig.18 is a flowchart illustrating an example of a neural network processing method. In one example, the neural network processing method may be a method for identifying an object. Fig.18 In operation 1810, the processing device determines a bit width of a first layer of the neural network. In operation 1820, the processing device obtains a second weight of the first layer of the neural network by extracting at least one bit corresponding to the determined bit width from each of the first weights of the first layer of the source model corresponding to the first layer of the neural network. In operation 1830, the processing device processes input data of the first layer of the neural network by executing the first layer of the neural network based on the obtained second weight. For a detailed description of the operation of the neural network processing method, reference may be made to the above reference. Figures 1 to 17 Provide a description.

[0204] Fig.19 is a flowchart showing another example of a neural network processing method. In one example, the neural network processing method may be a method for identifying an object. Fig.19 In operation 1910, the processing device determines a bit width of the first neural network. In operation 1920, the processing device obtains a second weight of the first neural network by extracting at least one bit corresponding to the determined bit width from each of the first weights of the source model. In operation 1930, the processing device processes input data of the first neural network by executing the first neural network based on the obtained second weight. For a detailed description of the operation of the neural network processing method, reference may be made to the above reference. Figures 1 to 17 Provide a description.

[0205] Fig. 20 is a flow chart showing an example of a neural network training method. Fig. 20In operation 210, the training device determines the low bit width weights corresponding to the first layer of the neural network by quantizing the high bit width weights corresponding to the first layer of the neural network. In operation 220, the training device determines the loss value corresponding to the determined low bit width weights by applying the input data to the first layer. In operation 230, the training device updates the high bit width weights based on the determined loss value. For a detailed description of the operation of the neural network training method, reference may be made to the above reference. Figures 1 to 17 Provide a description.

[0206] In this regard Figure 1 , Figure 6 , Fig.16 and Fig.17The processing equipment, training equipment and other equipment, units, modules, devices and other components described are implemented by hardware components. Examples of hardware components that can be used to perform the operations described in this application include: controllers, sensors, generators, drivers, memories, comparators, arithmetic logic units, adders, subtractors, multipliers, dividers, integrators and any other electronic components configured to perform the operations described in this application. In other examples, one or more hardware components in the hardware components that perform the operations described in this application are implemented by computing hardware (for example, by one or more processors or computers). Processors or computers can be implemented by one or more processing elements (such as logic gate arrays, controllers and arithmetic logic units, digital signal processors, microcomputers, programmable logic controllers, field programmable gate arrays, programmable logic arrays, microprocessors or any other devices or combinations of devices configured to respond and execute instructions in a limited manner to achieve the desired result). In one example, a processor or computer includes or is connected to one or more memories storing instructions or software executed by a processor or computer. The hardware components implemented by a processor or a computer can execute instructions or software (such as an operating system (OS) and one or more software applications running on the OS) for performing the operations described in this application. The hardware components can also access, manipulate, process, create and store data in response to the execution of instructions or software. For simplicity, the singular term "processor" or "computer" can be used for the description of the examples described in this application, but in other examples, multiple processors or computers can be used, or the processor or computer can include multiple processing elements or multiple types of processing elements or both. For example, a single hardware component or two or more hardware components can be implemented by a single processor, or two or more processors, or a processor and a controller. One or more hardware components can be implemented by one or more processors, or a processor and a controller, and one or more other hardware components can be implemented by one or more other processors, or another processor and another controller. One or more processors, or a processor and a controller can implement a single hardware component or two or more hardware components. The hardware components may have any one or more of different processing configurations, where examples of different processing configurations include: a single processor, independent processors, parallel processors, single instruction single data (SISD) multiprocessing, single instruction multiple data (SIMD) multiprocessing, multiple instruction single data (MISD) multiprocessing, and multiple instruction multiple data (MIMD) multiprocessing.

[0207] exist Figures 1 to 5 , Figures 7 to 15 as well as Figures 18 to 20The method for performing the operations described in the present application shown in the embodiment is performed by computing hardware (e.g., by one or more processors or computers), wherein the computing hardware is implemented as an execution instruction or software as described above to perform the operations performed by the method described in the present application. For example, a single operation or two or more operations may be performed by a single processor or two or more processors, or a processor and a controller. One or more operations may be performed by one or more processors, or a processor and a controller, and one or more other operations may be performed by one or more other processors, or another processor and another controller. One or more processors, or a processor and a controller may perform a single operation or two or more operations.

[0208] The instructions or software for controlling the processor or computer to implement the hardware components and perform the methods described above are written as computer programs, code segments, instructions or any combination thereof to individually or collectively instruct or configure the processor or computer to operate as a machine or special-purpose computer to perform the operations performed by the hardware components and methods described above. In one example, the instructions or software include machine code (such as machine code generated by a compiler) directly executed by the processor or computer. In another example, the instructions or software include high-level code executed by the processor or computer using an interpreter. A programmer of ordinary skill in the art can easily write instructions or software based on the block diagrams and flow charts shown in the accompanying drawings and the corresponding descriptions in the specification, wherein the block diagrams and flow charts shown in the accompanying drawings and the corresponding descriptions in the specification disclose algorithms for performing the operations performed by the hardware components and methods described above.

[0209] The instructions or software for controlling a processor or computer to implement the hardware components and perform the methods described above, and any associated data, data files, and data structures are recorded, stored, or fixed in or on one or more non-transitory computer-readable storage media. Examples of non-transitory computer-readable storage media include: read-only memory (ROM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disk storage, hard disk drive (HDD), solid state drive (SSD), card storage (such as, multimedia mini card or card (for example, Secure Digital (SD) or Extreme Digital (XD)), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid state disk and any other device, wherein any other device is configured to store instructions or software and any associated data, data files and data structures in a non-transitory manner and provide the instructions or software and any associated data, data files and data structures to a processor or computer so that the processor or computer can execute the instructions.

[0210] Although the present disclosure includes specific examples, it will be clear to those skilled in the art that various changes in form and detail may be made in these examples without departing from the spirit and scope of the claims and their equivalents. The examples described herein are considered to be descriptive only and not for limiting purposes. The description of the features or aspects in each example will be considered to be applicable to similar features or aspects in other examples. If the described techniques are performed in a different order, and / or if the components in the described systems, architectures, devices, or circuits are combined in different ways, and / or replaced or supplemented by other components or their equivalents, suitable results may be achieved. Therefore, the scope of the present disclosure is not limited by specific embodiments, but by the claims and their equivalents, and all changes within the scope of the claims and their equivalents should be interpreted as included in the present disclosure.

Claims

1. A neural network processing method, comprising: Obtaining a first weight of a first layer of a source model corresponding to a first layer of a neural network; Determine the bit width of the first layer of the neural network; obtaining a second weight of the first layer of the neural network by extracting at least one bit corresponding to the determined bit width from each of the first weights of the first layer of the source model corresponding to the first layer of the neural network; as well as processing input data of a first layer of the neural network by executing the first layer of the neural network based on the obtained second weight, wherein the input data of the first layer of the neural network includes multimedia data, and the multimedia data includes at least one of text, voice, dynamic image and static image, Among them, the source model is trained based on the following processing: By quantizing the first weight of the high bit width of the first layer of the source model, a low bit width weight corresponding to the first weight of the high bit width is determined, determining the loss value corresponding to the determined low bit-width weights by applying the training input data to the first layer of the source model, and Update the first weight of the high bit width based on the determined loss value, The step of updating the first weight of the high bit width includes: updating the first weight of the high bit width based on statistical information of the loss gradient corresponding to the determined loss value, The step of updating the first weight of the high bit width further includes: calculating statistical information by assigning a high weight value to a loss gradient corresponding to a weight with a high priority set among the weights of the low bit width.

2. The neural network processing method according to claim 1, wherein: The first weight has a bit precision higher than a bit precision of the second weight.

3. The neural network processing method according to claim 1, wherein: The second weight is nested within the first weight.

4. The neural network processing method according to claim 1, wherein: determining the bit width of the first layer of the neural network based on processing characteristics corresponding to the first layer of the neural network, The processing characteristics include at least one of a required processing speed, a required processing accuracy, a processing difficulty, and a terminal performance.

5. The neural network processing method according to claim 1, further comprising: Determine the bit width of the second layer of the neural network; Obtaining a third weight of a second layer of the source model corresponding to a second layer of the neural network; obtaining a fourth weight of the second layer of the neural network by extracting at least one bit corresponding to the bit width of the second layer of the determined neural network from each of the third weights of the second layer of the source model corresponding to the second layer of the neural network; as well as Input data of a second layer of the neural network is processed by executing the second layer of the neural network based on the obtained fourth weight, wherein the input data of the second layer corresponds to the output data of the first layer.

6. The neural network processing method according to claim 5, wherein: The third weight has a higher bit precision than the fourth weight.

7. The neural network processing method according to claim 5, wherein: The fourth weight is nested within the third weight.

8. The neural network processing method according to claim 5, wherein: The first layer of the neural network executed based on the second weights is configured to: process a first task corresponding to the first processing characteristic based on input data of the first layer, and The second layer of the neural network executed based on the fourth weight is configured to process a second task corresponding to a second processing characteristic different from the first task based on input data of the second layer.

9. A neural network processing method, comprising: Obtaining a first weight of a source model of a first neural network; determining a bit width of the first neural network; obtaining a second weight of the first neural network by extracting at least one bit corresponding to the determined bit width from each of the first weights of the source model; as well as processing input data of the first neural network by executing the first neural network based on the obtained second weight, wherein the input data of the first neural network includes multimedia data, and the multimedia data includes at least one of text, voice, dynamic image and static image, Among them, the source model is trained based on the following processing: By quantizing the first weight of the high bit width of the source model, the weight of the low bit width corresponding to the first weight of the high bit width is determined, determining the loss value corresponding to the determined low bit-width weights by applying the training input data to the source model, and Update the first weight of the high bit width based on the determined loss value, The step of updating the first weight of the high bit width includes: updating the first weight of the high bit width based on statistical information of the loss gradient corresponding to the determined loss value, The step of updating the first weight of the high bit width further includes: calculating statistical information by assigning a high weight value to a loss gradient corresponding to a weight with a high priority set among the weights of the low bit width.

10. The neural network processing method according to claim 9, wherein: The first weight has a bit precision higher than a bit precision of the second weight.

11. The neural network processing method according to claim 9, wherein: The second weight is nested within the first weight.

12. The neural network processing method according to claim 9, further comprising: determining a bit width of the second neural network based on a result of processing the input data performed by the first neural network; obtaining a third weight of the second neural network by extracting at least one bit corresponding to the determined bit width of the second neural network from each of the first weights; as well as Input data of the second neural network is processed by executing the second neural network based on the obtained third weights.

13. The neural network processing method according to claim 12, wherein: The first weight has a bit precision higher than a bit precision of the third weight.

14. The neural network processing method according to claim 12, wherein: The second weight and the third weight are nested within the first weight.

15. The neural network processing method according to claim 12, wherein: The first neural network executed based on the second weights is configured to: process a first task corresponding to the first processing characteristic based on input data of the first neural network, and The second neural network executed based on the third weights is configured to process a second task corresponding to a second processing characteristic different from the first task based on input data of the second neural network.

16. The neural network processing method according to claim 9, wherein: determining a bit width of the first neural network based on processing characteristics corresponding to the first neural network, The processing characteristics include at least one of a required processing speed, a required processing accuracy, a processing difficulty, and a terminal performance.

17. A neural network training method comprising: Determining a low bit width weight corresponding to the first layer of the neural network by quantizing a high bit width weight corresponding to the first layer of the neural network; Determining a loss value corresponding to the determined low bit width weight by applying input data to the first layer, wherein the input data includes multimedia data, and the multimedia data includes at least one of text, speech, dynamic image, and static image; and Update the high bit width weights based on the determined loss value, The step of updating the high bit width weight includes: updating the high bit width weight based on statistical information of the loss gradient corresponding to the determined loss value, The step of updating the high bit width weights further includes: calculating statistical information by assigning high weight values ​​to loss gradients corresponding to weights with high priorities set among the low bit width weights.

18. The neural network training method according to claim 17, further comprising: After the training associated with the high bit-width weights is completed, a set of low bit-width weights corresponding to the first layer is determined by quantizing the high bit-width weights.

19. The neural network training method according to claim 18, wherein: The low bit-width weight set includes a first bit-width weight set and a second bit-width weight set, the second bit-width weight set having a bit precision lower than the bit precision of the first bit-width weight set, Among them, the weight set of the second width is nested in the weight set of the first width.

20. The neural network training method according to claim 17, wherein: The low bit width weight includes a first weight of a first bit width and a second weight of a second bit width, wherein the first weight of the first bit width has a lower bit precision than the high bit width weight, and the second weight of the second bit width has a lower bit precision than the first weight of the first bit width.

21. The neural network training method according to claim 20, wherein: The steps to determine the weight of the low bit width include: Determine a first weight of the first bit width by quantizing the weight of the high bit width; and By extracting at least one bit from each of the determined first weights for the first bit width, a second weight for the second bit width is determined.

22. The neural network training method according to claim 20, wherein: The steps to determine the weight of the low bit width include: Determine a second weight of a second bit width by quantizing a weight of a high bit width; Determine the determined second weight of the second bit width as a high bit group of the first weight of the first bit width; and The low bit group of the first weight of the first bit width is determined by quantizing the weight of the high bit width.

23. A neural network processing device, comprising: processor; as well as a memory configured to store instructions readable by a processor, Wherein, when the instruction is executed by the processor, the processor is configured to: Obtaining a first weight of a first layer of a source model corresponding to a first layer of a neural network; Determine the bit width of the first layer of the neural network; obtaining a second weight of the first layer of the neural network by extracting at least one bit corresponding to the determined bit width from each of the first weights of the first layer of the source model corresponding to the first layer of the neural network; and processing input data of a first layer of the neural network by executing the first layer of the neural network based on the obtained second weight, wherein the input data of the first layer of the neural network includes multimedia data, and the multimedia data includes at least one of text, voice, dynamic image and static image, Among them, the source model is trained based on the following processing: By quantizing the first weight of the high bit width of the first layer of the source model, a low bit width weight corresponding to the first weight of the high bit width is determined, determining the loss value corresponding to the determined low bit-width weights by applying the training input data to the first layer of the source model, and Update the first weight of the high bit width based on the determined loss value, The step of updating the first weight of the high bit width includes: updating the first weight of the high bit width based on statistical information of the loss gradient corresponding to the determined loss value, The step of updating the first weight of the high bit width further includes: calculating statistical information by assigning a high weight value to a loss gradient corresponding to a weight with a high priority set among the weights of the low bit width.

24. The neural network processing device according to claim 23, wherein: The processor is configured as: Determine the bit width of the second layer of the neural network; Obtaining a third weight of a second layer of the source model corresponding to a second layer of the neural network; obtaining a fourth weight of the second layer of the neural network by extracting at least one bit corresponding to the bit width of the second layer of the determined neural network from each of the third weights of the second layer of the source model corresponding to the second layer of the neural network; as well as Input data of a second layer of the neural network is processed by executing the second layer of the neural network based on the obtained fourth weight, wherein the input data of the second layer corresponds to the output data of the first layer.

25. A device for identifying an object, comprising: processor; as well as a memory configured to store instructions readable by a processor, Wherein, when the instruction is executed by a processor, the processor is configured to: Obtaining a first weight of a source model of a first neural network; determining a bit width of the first neural network; obtaining a second weight of the first neural network by extracting at least one bit corresponding to the determined bit width from each of the first weights of the source model; and processing input data of the first neural network by executing the first neural network based on the obtained second weight, wherein the input data of the first neural network includes multimedia data, and the multimedia data includes at least one of text, voice, dynamic image and static image, Among them, the source model is trained based on the following processing: By quantizing the first weight of the high bit width of the source model, the weight of the low bit width corresponding to the first weight of the high bit width is determined, determining the loss value corresponding to the determined low bit-width weights by applying the training input data to the source model, and Update the first weight of the high bit width based on the determined loss value, The step of updating the first weight of the high bit width includes: updating the first weight of the high bit width based on statistical information of the loss gradient corresponding to the determined loss value, The step of updating the first weight of the high bit width further includes: calculating statistical information by assigning a high weight value to a loss gradient corresponding to a weight with a high priority set among the weights of the low bit width.

26. The apparatus of claim 25, wherein: The processor is configured as: determining a bit width of the second neural network based on a result of processing the input data by the first neural network; obtaining a third weight of the second neural network by extracting at least one bit corresponding to the determined bit width of the second neural network from each of the first weights; as well as Input data of the second neural network is processed by executing the second neural network based on the obtained third weights.

27. A neural network training device comprising: processor; as well as a memory configured to store instructions readable by a processor, Wherein, when the instruction is executed by the processor, the processor is configured to: Determining a low bit width weight corresponding to the first layer of the neural network by quantizing a high bit width weight corresponding to the first layer of the neural network; Determining a loss value corresponding to the determined low bit width weight by applying input data to the first layer, wherein the input data includes multimedia data, and the multimedia data includes at least one of text, speech, dynamic image, and static image; and Update the high bit width weights based on the determined loss value, The step of updating the high bit width weight includes: updating the high bit width weight based on statistical information of the loss gradient corresponding to the determined loss value, The step of updating the high bit width weights further includes: calculating statistical information by assigning high weight values ​​to loss gradients corresponding to weights with high priorities set among the low bit width weights.

28. The neural network training apparatus according to claim 27, wherein: The low bit width weight includes a first weight of a first bit width and a second weight of a second bit width, wherein the first weight of the first bit width has a lower bit precision than the high bit width weight, and the second weight of the second bit width has a lower bit precision than the first weight of the first bit width.

29. The neural network training apparatus according to claim 28, wherein: The processor is configured as: Determine a first weight of the first bit width by quantizing the weight of the high bit width; and By extracting at least one bit from each of the determined first weights for the first bit width, a second weight for the second bit width is determined.

30. The neural network training apparatus according to claim 28, wherein: The processor is configured as: Determine a second weight of a second bit width by quantizing a weight of a high bit width; Determine the determined second weight of the second bit width as a high bit group of the first weight of the first bit width; as well as The low bit group of the first weight of the first bit width is determined by quantizing the weight of the high bit width.

Images