Method and apparatus for recognizing an image by a neural network

Abstract

A method and apparatus for recognizing an image through a neural network are provided. The method includes obtaining an image to be recognized as an input image, determining an input feature map corresponding to the input image, obtaining a first weight kernel of a weight model of the neural network and pruning information of the first weight kernel, determining a processing range of the input feature map for each weight element vector of the first weight kernel based on the pruning information, performing a convolution operation between the input feature map and the first weight kernel based on the determined processing range, generating an output feature map based on an operation result of the convolution operation, and outputting an image recognition result based on the output feature map.

Description

[0001] This application claims the benefit of Korean Patent Application No. 10-2020-0087449, filed with the Korean Intellectual Property Office on July 15, 2020, the entire disclosure of which is incorporated herein by reference for all purposes. Technical Field

[0002] The following description relates to methods and apparatuses with neural network computation processing. Background Technology

[0003] The automation of recognition processing techniques can be achieved, for example, using neural network models implemented in processors as specialized computational structures, which can provide a computationally intuitive mapping between input and output patterns after considerable training. The ability to be trained to generate such a mapping can be referred to as the "trainability of the neural network." Furthermore, due to dedicated training, such specialized and trained neural networks can generalize to generate relatively accurate outputs for untrained input patterns. Summary of the Invention

[0004] This summary is provided to introduce, in a simplified form, the selection of concepts further described in the following detailed description. This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to help determine the scope of the claimed subject matter.

[0005] In one general aspect, a method for recognizing images using a neural network includes: obtaining an image to be recognized as an input image; determining an input feature map corresponding to the input image; obtaining a first weight kernel of a weight model of the neural network and pruning information of the first weight kernel; determining a processing range of the input feature map for each weight element vector of the first weight kernel based on the pruning information; performing a convolution operation between the input feature map and the first weight kernel based on the determined processing range; generating an output feature map based on the result of the convolution operation; and outputting an image recognition result based on the output feature map.

[0006] In another general aspect, an apparatus for recognizing images via a neural network includes: a processor configured to: acquire an image to be recognized as an input image; determine an input feature map corresponding to the input image; acquire a first weight kernel of a weight model of the neural network and pruning information of the first weight kernel; based on the pruning information, determine a processing range of the input feature map for each weight element vector of the first weight kernel; perform a convolution operation between the input feature map and the first weight kernel based on the determined processing range; generate an output feature map based on the result of the convolution operation; and output an image recognition result based on the output feature map.

[0007] In another general aspect, an electronic device includes: a camera configured to generate an input image; and a processor configured to: determine an input feature map corresponding to the input image; obtain a first weight kernel of a weight model of a neural network and pruning information of the first weight kernel; determine a processing range of the input feature map for each weight element vector of the first weight kernel based on the pruning information; perform a convolution operation between the input feature map and the first weight kernel based on the determined processing range; generate an output feature map based on the result of the convolution operation; and generate a recognition result of the input image based on the output feature map.

[0008] In another general aspect, a processor-implemented neural network method includes: obtaining a first weight kernel of a weight model and pruning information of the first weight kernel; determining, based on the pruning information, a processing range of an input feature map for each weight element vector of the first weight kernel; performing a convolution operation between the input feature map and the first weight kernel based on the determined processing range; and generating an output feature map of the neural network layer based on the result of the convolution operation.

[0009] The pruning information may include information about the number of weight element vectors in the first weight kernel and information about the position of the weight element vectors in the first weight kernel, and each weight element vector in the first weight kernel may correspond to a valid vector, which is different from the invalid vectors removed from the first weight kernel in the pruning process.

[0010] The step of determining the processing range for each weight element vector may include: determining a first processing range in the input feature map corresponding to the first weight element vector of the first weight kernel based on pruning information, and the step of performing convolution operation may include: generating a first intermediate output plane by performing a convolution operation between the first weight element vector and the first processing range.

[0011] The convolution operation between the first weight element vector and the first processing range may include a 1×1 convolution operation.

[0012] The convolution operation between the first weight element vector and the first processing range may include a 1×1 convolution operation between each input element vector of the first processing range and the first weight element vector.

[0013] The steps of generating the output feature map may include: generating a first output plane of the output feature map by accumulating intermediate output planes that include a first intermediate output plane.

[0014] The steps for generating the output feature map may include: generating the first output plane of the output feature map by accumulating the intermediate output planes corresponding to the results of the convolution operation.

[0015] The first weighting kernel and the first output plane can correspond to the first output channel.

[0016] The weight model may include multiple weight kernels, the multiple weight kernels including a first weight kernel, and the weight element vectors of the multiple weight kernels may correspond to non-zero vectors.

[0017] The sparsity of each of the multiple weight kernels in the weight model can be increased by using training processing with the corresponding basic weight kernels and scaling matrices.

[0018] The training process may include, for each of the plurality of weight kernels: inducing sparsity into a scaling matrix based on a loss function; and generating a weight kernel based on pointwise multiplication between the base weight kernel and the sparsity-induced scaling matrix.

[0019] The method may include: generating an input image using a camera; and determining an input feature map, the input feature map corresponding to the input image.

[0020] A non-transitory computer-readable storage medium stores instructions that, when executed by a processor, configure the processor to perform the method.

[0021] In another general aspect, a neural network device includes: a processor configured to: obtain a first weight kernel of a weight model and pruning information of the first weight kernel; determine, based on the pruning information, a processing range of an input feature map for each weight element vector of the first weight kernel; perform a convolution operation between the input feature map and the first weight kernel based on the determined processing range; and generate an output feature map of a neural network layer based on the result of the convolution operation.

[0022] The pruning information may include information about the number of weight element vectors in the first weight kernel and information about the position of the weight element vectors in the first weight kernel, and each weight element vector in the first weight kernel may correspond to a valid vector, which is different from the invalid vectors removed from the first weight kernel in the pruning process.

[0023] The processor can be configured to: determine a first processing range in the input feature map corresponding to the first weight element vector of the first weight kernel based on pruning information in order to determine the processing range for each weight element vector; and generate a first intermediate output plane by performing a convolution operation between the first weight element vector and the first processing range in order to perform a convolution operation.

[0024] In order to generate the output feature map, the processor can be configured to generate the first output plane of the output feature map by accumulating intermediate output planes that include the first intermediate output plane.

[0025] To generate the output feature map, the processor can be configured to generate the first output plane of the output feature map by accumulating the intermediate output planes corresponding to the results of the convolution operation.

[0026] The weight model may include multiple weight kernels, including a first weight kernel, and the weight element vectors of the multiple weight kernels may correspond to non-zero vectors.

[0027] The sparsity of each of the multiple weight kernels in the weight model can be increased by using training processing with the corresponding basic weight kernels and scaling matrices.

[0028] The device may be an electronic device including a camera configured to generate an input image, a processor configured to determine an input feature map, and the input feature map may correspond to an input image.

[0029] The device may include: a memory storing instructions that, when executed by a processor, configure the processor to perform steps such as obtaining a first weight kernel, determining a processing range, performing convolution, and generating an output feature map.

[0030] In another general aspect, an electronic device includes: a camera configured to generate an input image; and a processor configured to: determine an input feature map corresponding to the input image; obtain a first weight kernel of a weighted model and pruning information of the first weight kernel; determine a processing range of the input feature map for each weight element vector of the first weight kernel based on the pruning information; perform a convolution operation between the input feature map and the first weight kernel based on the determined processing range; and generate an output feature map of a neural network layer based on the result of the convolution operation.

[0031] The processor can be configured to: determine a first processing range in the input feature map corresponding to the first weight element vector of the first weight kernel based on pruning information in order to determine the processing range for each weight element vector; and generate a first intermediate output plane by performing a convolution operation between the first weight element vector and the first processing range in order to perform a convolution operation.

[0032] The processor can be configured to generate recognition results of the input image based on the output feature map.

[0033] In another general aspect, a processor-implemented neural network method includes: for each valid vector of a pruned weight kernel, determining a processing range of an input feature map based on the position of the valid vector within the pruned weight kernel; for each valid vector, performing a convolution operation between the valid vector and the corresponding processing range of the input feature map; and generating channels of the output feature map of the neural network layer by accumulating the results of the convolution operations performed for each valid vector.

[0034] The position of each processing range within the input feature map can correspond to the position of the corresponding effective vector within the pruned weight kernel.

[0035] An effective vector can correspond to a non-zero vector of the pruned weight kernel.

[0036] In another general aspect, a processor-implemented neural network method includes: obtaining effective vectors of pruned weight kernels and pruning information of the effective vectors; determining a processing range of an input feature map based on the pruning information for each effective vector; generating an intermediate output plane for each effective vector by performing a convolution operation between the effective vector and the corresponding processing range; and generating an output feature map by accumulating an output plane corresponding to the intermediate output plane of the weight kernel for each weight kernel to generate an output feature map.

[0037] Other features and aspects will become clear from the following detailed description, the accompanying drawings, and the claims. Attached Figure Description

[0038] Figure 1 This illustrates an example of the operation of a computational processing device associated with convolution operations.

[0039] Figure 2 Examples of data related to convolution operations are shown.

[0040] Figure 3 An example of the data structure for a weighted model is shown.

[0041] Figure 4 An example of a data structure showing pruning information is provided.

[0042] Figure 5 An example of the processing range corresponding to each weight element vector is shown.

[0043] Figure 6 This illustrates an example of processing that generates the output plane through an accumulative convolution operation.

[0044] Figure 7 This illustrates an example of the process of generating output feature maps through convolution operations.

[0045] Figure 8 An example of training a neural network using a scaling matrix is ​​shown.

[0046] Figure 9 An example is shown showing changes to the convolution weights based on training.

[0047] Figure 10 Examples of computational processing methods related to convolution operations are shown.

[0048] Figure 11 An example of the configuration of a computing processing device is shown.

[0049] Figure 12 An example of an electronic device is shown.

[0050] Throughout the accompanying drawings and detailed embodiments, unless otherwise described or provided, the same reference numerals will be understood to denote the same elements, features, and structures. The drawings may not be to scale, and for clarity, illustration, and convenience, the relative dimensions, scale, and depiction of elements in the drawings may be exaggerated. Detailed Implementation

[0051] The following detailed embodiments are provided to aid the reader in gaining a comprehensive understanding of the methods, apparatus, and / or systems described herein. However, various changes, modifications, and equivalents of the methods, apparatus, and / or systems described herein will become apparent upon understanding this disclosure. For example, the order of operations described herein is merely illustrative and is not limited to those orders set forth herein, but may be changed as will become clear upon understanding this disclosure, except for operations that must occur in a specific order. Furthermore, for clarity and conciseness, descriptions of known features may be omitted.

[0052] The features described herein may be implemented in different forms and should not be construed as limited to the examples described herein. Rather, the examples described herein have been provided merely to illustrate some of the many feasible ways of implementing the methods, apparatus, and / or systems described herein that will be clear upon understanding the disclosure of this application.

[0053] In the following description, some exemplary embodiments will be described in detail with reference to the accompanying drawings. Various modifications may be made to the exemplary embodiments. Here, the exemplary embodiments are not to be construed as limited to the disclosure, but should be understood to include all changes, equivalents, and substitutions within the scope of the disclosed ideas and techniques.

[0054] The terminology used herein is for the purpose of describing exemplary embodiments only and is not intended to limit this disclosure. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. As used herein, the term "and / or" includes any one and any combination of any two or more of the associated listed items. As used herein, the terms "comprising," "including," and "having" indicate the presence of the described features, integrals, steps, operations, elements, components, quantities, and / or combinations thereof, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, quantities, and / or combinations thereof. The term "may" (e.g., regarding what an example or embodiment may include or implement) used herein with respect to examples or embodiments indicates the presence of at least one example or embodiment that includes or implements such a feature, and all examples are not limited thereto.

[0055] Unless otherwise defined, all terms used herein (including technical or scientific terms) shall have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains and as understood after understanding this disclosure. Unless expressly defined herein, terms (such as those defined in a general dictionary) shall be interpreted as having a meaning matching the context in the relevant field and the meaning in this disclosure, and shall not be interpreted as having an idealized or overly formal meaning.

[0056] When describing exemplary embodiments with reference to the accompanying drawings, the same reference numerals denote the same constituent elements, and repetitive descriptions related to the same constituent elements will be omitted. In the description of exemplary embodiments, detailed descriptions of well-known related structures or functions will be omitted where such detailed descriptions would lead to a vague interpretation of this disclosure.

[0057] Furthermore, the terms “first,” “second,” “A,” “B,” “(a),” “(b),” etc., are used herein to describe components according to the example embodiments. Each of these terms is not intended to define the nature, order, or sequence of the corresponding component, but only to distinguish the corresponding component from one or more other components. Although the terms “first” or “second” are used herein to describe various elements, components, regions, layers, or portions, these elements, components, regions, layers, or portions are not limited by these terms. Rather, these terms are only used to distinguish one element, component, region, layer, or portion from another element, component, region, layer, or portion. Therefore, without departing from the teachings of the examples, the first element, first component, first region, first layer, or first portion referred to in the examples described herein may also be referred to as a second element, second component, second region, second layer, or second portion.

[0058] Throughout this specification, when an element such as a layer, region, or substrate is described as being "on" another element, "connected to," or "bonded to" another element, the element may be directly "on" the other element, "connected to," or "bonded to" the other element, or there may be one or more other elements in between. Conversely, when an element is described as being "directly on" another element, "directly connected to," or "directly bonded to" another element, there may be no other elements in between. Similarly, expressions such as "between" and "immediately between," and "adjacent to" and "closely adjacent to" can also be interpreted as described above.

[0059] Components that share common functionality with components included in one example embodiment are described using the same names in another example embodiment. Unless otherwise described, descriptions in one example embodiment are applicable to another example embodiment, and detailed descriptions within a redundant scope are omitted.

[0060] Figure 1 An example of the operation of a computational processing device (e.g., computational processing device 100) associated with convolution operations is shown. (Refer to...) Figure 1 The computing processing device 100 can process operations related to the neural network 110. In response to the processing of operations related to the neural network 110, the neural network 110 can perform operations for training the neural network 110. For example, the operations of the neural network 110 may include image recognition (such as object classification, object detection, object identification, object recognition, or user authentication). At least a portion of the processing operations related to the neural network 110 may be implemented by hardware including a neural processor or a combination of hardware and software. The computing processing device 100 may be, for example, a mobile phone, desktop computer, laptop computer, tablet PC, wearable device, smart TV, smart vehicle, security system, smart home system, smart home appliance, etc., or may be installed in, for example, a mobile phone, desktop computer, laptop computer, tablet PC, wearable device, smart TV, smart vehicle, security system, smart home system, smart home appliance, etc.

[0061] The neural network 110 may correspond to a deep neural network (DNN) comprising multiple layers. The multiple layers may include an input layer, at least one hidden layer, and an output layer. Figure 1 The first layer 111, the second layer 112, and the nth layer 113 can be at least a subset of multiple layers. For example, the input layer can be the first layer 111, at least one hidden layer can include the second layer 112 through the (n-1)th layer, and the output layer can be the nth layer. The neural network 110 can include any one or any combination of a fully connected network (FCN), a convolutional neural network (CNN), and a recurrent neural network (RNN). For example, at least a subset of the multiple layers in the neural network 110 can correspond to a CNN, and another subset can correspond to an FCN. In this example, the CNN can be referred to as a convolutional layer, and the FCN can be referred to as a fully connected layer.

[0062] In a CNN, the data input to each layer is called an "input feature map," and the data output from each layer is called an "output feature map." Input and output feature maps can also be referred to as activation data. A layer's output feature map can be the input feature map for subsequent layers, or it can be used to generate the input feature map for subsequent layers. When a convolutional layer corresponds to an input layer, the input feature map of that input layer can correspond to the input data. For example, the input data can be an input image or data generated from initial processing of the input image. In one example, the first layer of the neural network receives the input image and outputs its output feature map as the input feature map for the second layer. The i-th layer of the neural network outputs its own output feature map based on the (i-1)-th layer's output feature map, where i is an integer greater than 1. The last layer of the neural network generates and outputs a recognition result corresponding to the input image based on the output feature map of the previous layer.

[0063] The neural network 110 can be trained based on deep learning and can perform inferences suitable for the training purpose by mapping input and output data in a non-linear relationship. Deep learning can be a machine learning scheme for solving problems such as image recognition based on large datasets. Deep learning can be understood as the process of solving an optimization problem to find the point that minimizes energy while training the neural network 110 based on prepared training data.

[0064] Through supervised or unsupervised learning in deep learning, the structure of the neural network 110, or the weights corresponding to the model, can be obtained or determined, and input and output data can be mapped to each other through the weights. For example, when the width and depth of the neural network 110 are large enough, the neural network 110 can have the ability to perform arbitrary functions. Optimal performance can be achieved when the neural network 110 is trained on a sufficiently large amount of training data through appropriate training processing.

[0065] In the following description, neural network 110 may be expressed as "pre-trained," where "pre" may indicate the state of neural network 110 before it is "started." A "started" neural network 110 may indicate that neural network 110 is ready for inference. For example, "starting" neural network 110 may include loading neural network 110 into memory, or, after loading neural network 110 into memory, input image data for inference into neural network 110.

[0066] The computational processing device 100 can perform convolution operations between the input feature map and the weight kernel of each convolutional layer to process operations associated with each convolutional layer, and can generate an output feature map based on the results of the convolution operations. To process operations associated with the neural network 110, multiple operations, including multiplication and accumulation (MAC) operations, can be processed. Furthermore, processing these operations may consume significant computational resources and time. The computational processing device 100 of one or more embodiments can make the neural network 110 lightweight and perform high-speed operations, thereby reducing the aforementioned consumption of computational resources and time, and enabling the neural network 110 to be effectively implemented in resource-constrained environments (such as sensor environments or embedded environments (e.g., mobile terminals)).

[0067] In one example, the computational processing device 100 may perform convolutional operations associated with convolutional layers in the neural network 110 based on a weight model and pruning information. In this example, pointwise pruning may be applied to the weight model. Pruning can be a scheme to lightweight the model by removing redundant weight values ​​(e.g., zero values ​​or by setting redundant weights to zero values), and may include structured pruning and unstructured pruning. Structured pruning may be a scheme that uses structuring elements (e.g., channels) as the unit of pruning, while unstructured pruning may be a scheme that prunes redundant values ​​instead of using structuring elements.

[0068] For example, channel pruning, as a form of structured pruning, can prune the model with a value of "0" for each weight kernel corresponding to the output channel. For instance, in channel pruning, each weight kernel in the same channel that is determined to be redundant can be pruned. Based on structured pruning, acceleration and lightweighting can be achieved relatively easily; however, performance may degrade due to the removal of important weights (e.g., by pruning entire weight kernels that may have some important weights). Based on unstructured pruning, the proportion of "0" can be increased, but acceleration and lightweighting may be difficult to achieve without hardware assistance due to the lack of regularity. Pointwise pruning can have the characteristics of structured pruning, but the performance degradation can be minimized by using relatively small units (e.g., point tensors) as the pruning units.

[0069] Figure 2 Examples of data related to convolution operations are shown. (See reference) Figure 2The weighting model may include a kernel, which comprises D weight kernels, each corresponding to an output channel. For example, the weight kernels may correspond one-to-one with the output channels. Alternatively, the number of weight kernels may equal the number of output channels. Each weight kernel may include multiple weight element vectors. For example, the size of the weight kernel may be defined as "K1×K2×C" based on height × width × input channels. The size of the weight kernel may indicate the number of weight elements included in the weight kernel. In other words, the weight kernel may include "K1×K2×C" weight elements. Here, C may indicate the number of input channels. The weight kernel may include "K1×K2" weight element vectors. When K1=K2=K, the weight kernel may include "K..." 2 ×C” weight elements and “K” 2 A weight element vector. The weight element vector can correspond to a point tensor. For example, the size of the weight element vector can be defined as "1×1×C".

[0070] Pruning information can indicate the structural features of a pruned weight model. Pointwise pruning corresponds to structured pruning, thus the data of the weight model can have structural meaning, and this structural meaning can be recorded in the pruning information. For example, the pruning information can include information about the position of the weight element vectors included in each weight kernel and / or information about the number (e.g., total number) of weight element vectors included in each weight kernel. In this example, the weight element vectors included in each weight kernel can correspond to valid vectors. The computational processing device 100 can analyze the structural features of each weight kernel of the weight model based on the pruning information and can perform convolution operations based on the structural features.

[0071] Valid vectors can be weight element vectors retained in the pruning process or as a result of the pruning process, while invalid vectors can be weight element vectors removed during the pruning process. For example, valid vectors can correspond to non-zero vectors, and invalid vectors can be zero vectors. For example, the first weight kernel of the weight model can include the third, sixth, and seventh weight element vectors, which are valid vectors among the first to ninth weight element vectors. In this example, pruning information can include information about the number of valid vectors (e.g., the total number) (e.g., three) and information about the positions of the valid vectors (e.g., "3, 6, and 7" corresponding to matrix positions [1, 3], [2, 3], and [3, 1], respectively).

[0072] In one example, the processing device 100 can perform an accumulative convolution operation. The accumulative convolution operation can be performed using intermediate results from the accumulative convolution operation. For example, a first intermediate output plane can be generated by a convolution operation between a first effective vector of the first weight kernel and a first processing range of the input feature map. Through the above processing, an intermediate output plane can be generated by a convolution operation between each effective vector of the first weight kernel and the corresponding processing range of the input feature map. The generated intermediate output planes can be accumulated to generate the first output plane of the output feature map corresponding to the first weight kernel. The corresponding processing range of the input feature map can be determined based on the positional information (e.g., offset) of the effective vectors. Furthermore, the first weight kernel and the first output plane can correspond to the first output channel of the output feature map.

[0073] Based on the above cumulative convolution operation, the computing processing device 100 of one or more embodiments can perform convolution operations to generate output feature maps without performing buffering operations as performed by typical computing processing devices when performing convolution operations, such as transforming the weight kernel or input feature map into a form suitable for convolution and storing the weight kernel or input feature map in a buffer. Therefore, the convolution operation of the computing processing device 100 of one or more embodiments is faster than the convolution operation of a typical computing processing device. In addition, the cumulative convolution operation of the computing processing device 100 of one or more embodiments can be performed on a per-valid vector basis, so pruning can be applied to each point tensor. Furthermore, the cumulative convolution operation of the computing processing device 100 of one or more embodiments is applicable to pointwise pruning that skips zeros for each point tensor.

[0074] Figure 3 An example of the data structure for a weighted model (e.g., weighted model 200) is shown. In one example, weighted model 200 could be a weighted model with pointwise pruning applied. See reference. Figure 3 The weight model 200 includes weight element vectors W1_2, W1_3, W1_4, W1_6, W1_8, W2_2, W2_4, W2_6, W2_8, W3_2, W3_4, W3_5, W3_6, W4_3, W4_6, and W4_7 (referred to as "weight element vectors W1_2 to W4_7"). Weight element vectors W1_2 to W4_7 can each correspond to an effective vector. Weight element vector Wa_b can indicate the b-th weight element vector of the a-th weight kernel. In other words, a can be the identification information of the weight kernel, and b can be the positional information (e.g., offset) of the weight element vector. The weight model 200 can be visually represented as... Figure 3 The lower part shows a three-dimensional (3D) block. Figure 3The diagram illustrates a "3×3×C" weight kernel, which is a weight kernel that satisfies "K1=K2=3". An example of a "3×3×C" weight kernel will be described below; however, this is merely an example for ease of description. For instance, K1 and K2 can have any value other than "3".

[0075] The weight model 200 may include weight element vectors W1_2 to W4_7 corresponding to valid vectors among the weight element vectors W1_1 to W4_9 of the weight kernels W1 to W4. For example, when pointwise pruning is applied to the weight kernel W1 during training, weight element vectors W1_2, W1_3, W1_4, W1_6, and W1_8 may be identified as valid vectors, and weight element vectors W1_1, W1_5, W1_7, and W1_9 may be identified as invalid vectors. Similarly, based on pointwise pruning, each of the weight element vectors in the weight kernels W2 to W4 may be identified as a valid or invalid vector. When the weight model 200 includes only each valid vector as continuous data, structural information associated with each valid vector can be used to interpret each valid vector. This structural information can be provided through pruning information.

[0076] Figure 4 An example of a data structure for pruning information (e.g., pruning information 300) is shown. (See reference...) Figure 4 The pruning information 300 may include effective vectors with weight kernels W1 to W4 (e.g., Figure 3 Information related to the weight element vectors (W1_2 to W4_7). For example, pruning information 300 may include information about the number of valid vectors included in each of the weight kernels W1 to W4 and information about the position of each valid vector in each of the weight kernels W1 to W4.

[0077] In one example, refer to Figure 3 and Figure 4 The pruning information 300 may include "5" indicating the number of valid vectors included in weight kernel W1 and "2, 3, 4, 6, 8" indicating the position of valid vectors in weight kernel W1; may include "4" indicating the number of valid vectors included in weight kernel W2 and "2, 4, 6, 8" indicating the position of valid vectors in weight kernel W2; may include "4" indicating the number of valid vectors included in weight kernel W3 and "2, 4, 5, 6" indicating the position of valid vectors in weight kernel W3; and may include "3" indicating the number of valid vectors included in weight kernel W4 and "3, 6, 7" indicating the position of valid vectors in weight kernel W4.

[0078] The pruning information 300 can be provided as header information for the weight model. In one example, 2 bytes can be allocated to each of the weight kernels W1 through W4 in the pruning information 300. In this example, the pruning information 300 may include "0000101011101010" for weight kernel W1. Here, "0000101" indicates the number of valid vectors, and "011101010" indicates the position of each valid vector.

[0079] For example, the binary number "0000101" can represent the decimal number "5". Each bit position of "011101010" can correspond to each position in a "3×3×C" weight kernel, with "1" indicating a valid vector and "0" indicating an invalid vector. In other words, "011101010" can indicate that the second, third, fourth, sixth, and eighth weight element vectors correspond to valid vectors. Similarly, pruning information 300 can include "0000100010101010" for weight kernel W2, "0000100010111000" for weight kernel W3, and "0000011001001100" for weight kernel W4. In one example, when the bit order corresponds to the order of the width and height directions of the weight kernel, the position information of the valid vector in weight kernel W1 can be displayed as "011101010" as described above. In another example, when the bit order corresponds to the order of the height and width directions of the weight kernel, the position information of the effective vector in the weight kernel W1 can be displayed as "010101110" (i.e., in reverse order).

[0080] Figure 5 An example of the processing range corresponding to each weight element vector is shown. As described above, based on the example, one or more cumulative convolution operations of various embodiments can be performed. For cumulative convolution operations, the processing range of the input feature map corresponding to each valid vector can be determined. For example, when performing a convolution operation between the weight kernel and the input feature map based on a sliding window scheme, the path through which the weight kernel slides can be specified in the input feature map. Therefore, the region traversed by each weight element vector of the weight kernel can be specified in the input feature map. According to the example, the processing range of the input feature map corresponding to the weight element vector can correspond to the region of the input feature map through which the weight element vector passes.

[0081] For example, refer to Figure 5The first processing range 421 of the input feature map can correspond to the first weight element vector of the weight kernel 410. Similarly, the second processing range 422 of the input feature map can correspond to the second weight element vector of the weight kernel 410, and the third processing range 423 of the input feature map can correspond to the third weight element vector of the weight kernel 410. The fourth processing range 424 to the ninth processing range 429 of the input feature map can correspond one-to-one with the fourth to ninth weight element vectors of the weight kernel 410, respectively. The processing ranges defined above can be used in the cumulative type convolution operation of one or more embodiments described further below.

[0082] Figure 6 This illustrates an example of processing that generates the output plane using an accumulative convolution operation. (See also...) Figure 6 It can perform cumulative convolution operations between the input feature map 510 and the weight kernel 520. The convolution operation between the input feature map 510 and the weight kernel 520 can include individual convolution operations between each valid vector of the input feature map 510 and the weight kernel 520.

[0083] Despite Figure 6 The weight kernel 520 is shown as a whole to aid understanding, but convolution operations can be performed with respect to the effective vectors of the weight kernel 520, rather than the entire weight kernel 520. In other words, although the weight kernel 520 is shown in a "3×3" form, it can function as if three 1×1 weight kernels were used. Within the weight kernel 520, the third, sixth, and seventh weight element vectors correspond to the effective vectors. Therefore, convolution operations on the weight kernel 520 can be performed based on the third, sixth, and seventh weight element vectors.

[0084] For example, an intermediate output plane 531 can be generated as the result of the convolution operation between the processing range 511 of the input feature map 510 and the third weight element vector of the weight kernel 520. Similarly, an intermediate output plane 532 can be generated as the result of the convolution operation between the processing range 512 of the input feature map 510 and the sixth weight element vector of the weight kernel 520, and an intermediate output plane 533 can be generated as the result of the convolution operation between the processing range 513 of the input feature map 510 and the seventh weight element vector of the weight kernel 520. In one example, the input feature map 510 may be zero-padding, such that each of the processing ranges 511, 512, and 513 includes a zero-padding portion.

[0085] Each processing range of the input feature map 510 can be specified or determined based on pruning information associated with the weight kernel 520. For example, a computational processing device (e.g., computational processing device 100) can analyze or determine the correspondence between each weight element vector of the weight kernel and each processing range of the input feature map before the convolution operation (e.g., such as...). Figure 5 As shown in the figure, the processing range corresponding to the effective vector of the weight kernel 520 in the input feature map 510 can be specified based on the correspondence.

[0086] The convolution operation between each weight element vector and each processing range may include a 1×1 convolution operation. For example, the convolution operation between the third weight element vector and processing range 511 may include a 1×1 convolution operation between the input element vector of processing range 511 and the third weight element vector, such that the result of the 1×1 convolution operation corresponds to an output element of the intermediate output plane 531. A 1×1 convolution operation may be performed between each input element vector of processing range 511 and the third weight element vector to determine all output elements of the intermediate output plane 531. Therefore, in one example, the width and height of the intermediate output planes 531, 532, and 533 may be equal to the width and height of the corresponding processing ranges 511, 512, and 513.

[0087] Intermediate output planes 531, 532, and 533 can be accumulated to generate output plane 540 of the output feature map. Weight kernel 520 and output plane 540 can correspond to output channels. Weight kernel 520 can correspond to one of multiple weight kernels included in the weight model, and each of the multiple weight kernels can correspond to an output channel. An output plane corresponding to each output channel can be generated based on an accumulative convolution operation of each weight kernel. When all output planes corresponding to all output channels are generated, the output feature map is fully generated.

[0088] Figure 7 This illustrates an example of the process of generating output feature maps through convolution operations. Figure 7 This illustrates the operations associated with the nth layer (e.g., a convolutional layer) of a neural network. (See reference...) Figure 7 In operation 610, the computational processing device (e.g., computational processing device 100) can obtain the d-th weight kernel and can obtain or determine pruning information. The total number of weight kernels in the n-th layer can be "D", and d is less than or equal to D. The computational processing device can obtain or determine information related to the effective vectors of the d-th weight kernel (e.g., the number of effective vectors and / or the positions of effective vectors) by parsing the pruning information.

[0089] In operation 620, the processing device can obtain or determine the c-th weight element vector. The obtained weight element vector may correspond to a valid vector. In operation 630, the processing device can obtain or determine the input feature map of the n-th layer. In operation 640, the processing device can perform a convolution operation. The processing device can specify or determine the processing range in the input feature map corresponding to the weight element vector, and can perform a 1×1 convolution operation between the weight element vector and each input feature vector in the processing range. Based on the convolution operation of operation 640, the c-th intermediate output plane can be generated. In operation 650, the processing device can accumulate the intermediate output planes.

[0090] In operation 660, the processing device compares c with Valid-cnt. Valid-cnt indicates the total number of valid vectors in the d-th weight kernel. In one example, when c and Valid-cnt are different (i.e., when the operation on one of the valid vectors in the d-th weight kernel has not yet been completed), c can be incremented by "1", and operation 620 can be performed. In another example, when c and Valid-cnt are the same (i.e., when the operation associated with all valid vectors in the d-th weight kernel has been completed), operation 670 can be performed. The same c and Valid-cnt indicates that all intermediate output planes of the d-th weight kernel have been accumulated, resulting in the output plane of the d-th weight kernel being generated.

[0091] In operation 670, the processing device compares "d" and "D". In one example, when "d" and "D" are different (i.e., when the operation of one of the "D" weight kernels in the nth layer has not yet been completed), "d" can be incremented by "1", and operation 610 can be executed. In another example, when "d" and "D" are the same (i.e., when the operation associated with all "D" weight kernels in the nth layer has been completed), the convolution operation associated with the nth layer can be terminated. "d" and "D" being the same indicates that all output planes (i.e., "D" output channels) of the "D" weight kernels have been generated, resulting in the generation of the output feature map of the nth layer. This can be repeated for each layer of the neural network. Figure 7 In the above processing, the output feature map of a layer can be, or can be used to, generate the input feature map of subsequent layers, and the output feature map of the final layer can be, or can be used to generate the output data of the neural network. For example, the output data can correspond to the results of image recognition (e.g., object classification, object detection, object identification, object recognition, or user authentication).

[0092] Figure 8This illustrates an example of training a neural network using a scaling matrix. The neural network may include convolutional layers. The training output can be determined based on the convolution operation between the training inputs of the convolutional layers and the convolutional weights. The neural network can be trained based on the loss of the training output.

[0093] The convolution weight kernel can be determined based on the basic weight kernel and the scaling matrix. In the following text, the convolution weight kernel and the basic weight kernel may be referred to as "convolution weight" and "basic weight," respectively. The basic weight may have a size of "K1×K2×C," and the scaling matrix may have a size of "K1×K2." As a non-limiting example, such as... Figure 8 As shown, K1 = K2 = 3. In one example, the convolution weights can be determined based on pointwise multiplication between the base weights and the scaling matrix.

[0094] Figure 8 The first convolution weights 731 to the Dth convolution weights 733 can correspond to the weight kernels of a predetermined convolutional layer. As described above, the first convolution weight 731 can be determined based on the pointwise multiplication operation between the first basic weight 711 and the first scaling matrix 721, the second convolution weight 732 can be determined based on the pointwise multiplication operation between the second basic weight 712 and the second scaling matrix 722, and the Dth convolution weight 733 can be determined based on the pointwise multiplication operation between the Dth basic weight 713 and the Dth scaling matrix 723.

[0095] Trainable neural networks can minimize the loss of the training output of convolutional layers. For example, the training output can be determined based on the convolution operation between the training input and the convolution weights, and the difference between the training output and the label can be determined as the loss. The convolution weights can be adjusted to reduce the loss. By repeating the above process, the convolution weights that minimize the loss can be determined.

[0096] The loss function may include a task loss component (e.g., cross-entropy) and a sparsity-inducing penalty component. Due to the sparsity-inducing penalty component of the loss function, sparsity can be induced in the convolution weights. For example, the loss function may be defined as shown in Equation 1 below.

[0097] Equation 1:

[0098]

[0099] In Equation 1, L represents the loss function, x represents the training input (e.g., the training image), W represents the base weights, r represents the scaling matrix, and f represents the convolution operation. Therefore, W*r represents the convolution weights, and f(x,W*r) represents the convolution operation between the training input and the convolution weights. Furthermore, y represents the label, and l represents the loss. Therefore, l(f(x,W*r),y) represents the loss between the label and the result of the convolution operation. Additionally, γ represents the scale hyperparameter, and g represents the sparsity-induced penalty (e.g., the L1 norm).

[0100] Figure 9 This illustrates an example of changes to convolutional weights based on training. (See reference...) Figure 9 The sparsity of convolutional weights 831 can be increased by using training processing with basic weights 811 and scaling matrix 821. For example, since the sparsity of the loss function induces a penalty component, sparsity can be induced in scaling matrix 821, so scaling matrix 821 can be changed to or trained to have sparsity as scaling matrix 822. Furthermore, the change from scaling matrix 821 to scaling matrix 822 may result in a difference between basic weights 811 and basic weights 812.

[0101] Convolution weights 831 can be determined based on pointwise multiplication between the base weights 811 and the scaling matrix 821, and convolution weights 832 can be determined based on pointwise multiplication between the base weights 812 and the scaling matrix 822. Due to the sparsity of the scaling matrix 822, convolution weights 832 can also have sparsity (e.g., the same sparsity). Pruning can be applied to convolution weights 832 for each point tensor; therefore, the above pruning can be referred to as "pointwise pruning".

[0102] A weight model (e.g., weight model 200) can be determined based on the convolution weights 832. Furthermore, pruning information can be determined based on the distribution of effective vectors within the convolution weights 832. For example, pointwise pruning can be applied. Figure 8 The first convolutional weights 731 to the Dth convolutional weights 733 are used to construct the weight model. Furthermore, pruning information (e.g., pruning information 300) can be generated for the weight model. For example, the pruning information can be stored as the head of the weight model.

[0103] Figure 10 Examples of computational processing methods related to convolution operations are shown. (See reference...) Figure 10In operation 910, the computational processing device (e.g., computational processing device 100) can obtain or determine a first weight kernel of the weight model and pruning information of the first weight kernel. In operation 920, the computational processing device can specify or determine a processing range in the input feature map corresponding to each weight element vector of the first weight kernel based on the pruning information. In operation 930, the computational processing device can perform a convolution operation between the input feature map and the first weight kernel based on the specified processing range. For example, the computational processing device can specify a first processing range in the input feature map corresponding to the first weight element vector of the first weight kernel based on the pruning information, and can perform a convolution operation between the first weight element vector and the first processing range to generate a first intermediate output plane. In operation 940, the computational processing device can generate an output feature map based on the result of the convolution operation. For example, the computational processing device can generate a first output plane of the output feature map by accumulating intermediate output planes including the first intermediate output plane. As described above. Figures 1 to 9 , Figure 11 and Figure 12 The description and operation are applicable to the arithmetic processing methods, and will not be repeated here.

[0104] Figure 11 An example configuration of a computing processing device (e.g., computing processing device 1000) is shown. (Refer to...) Figure 11 The computing device 1000 includes a processor 1010 (e.g., one or more processors) and a memory 1020 (e.g., one or more memories). The memory 1020 may be connected to the processor 1010 and may store instructions executable by the processor 1010, data to be computed by the processor 1010, or data processed by the processor 1010. The memory 1020 may include, for example, non-transitory computer-readable storage media (e.g., high-speed random access memory (RAM)) and / or non-volatile computer-readable storage media (e.g., at least one disk storage device, flash memory device, or other non-volatile solid-state memory device). In one example, the computing device 1000 may be... Figure 1 The computing and processing device 100.

[0105] Processor 1010 can execute instructions as described above. Figures 1 to 10 and the following Figure 12 At least one or all of the described operations. For example, processor 1010 can obtain a first weight kernel of a weight model and pruning information of the first weight kernel, specify a processing range in the input feature map corresponding to each weight element vector of the first weight kernel based on the pruning information, perform a convolution operation between the input feature map and the first weight kernel based on the specified processing range, and generate an output feature map based on the result of the convolution operation. Additionally, Figures 1 to 10 and Figure 12The description also applies to the computing device 1000, and further description will not be repeated here.

[0106] Figure 12 An example of an electronic device (e.g., electronic device 1100) is shown. Electronic device 1100 may be, or may include, Figure 1 The computing processing device 100 or Figure 11 The electronic device 1100 is a computing processing device. It can acquire an input image and perform operations related to the input image using a neural network. Operations related to the input image may include image recognition (e.g., object classification, object detection, object tracking, object identification, object recognition, or user authentication). For example, the electronic device 1100 can determine an input feature map corresponding to the input image, generate an output feature map corresponding to the input feature map through at least one layer of a neural network, and generate image recognition results related to the input image (such as any one or any combination of classification results, detection results, tracking results, identification results, and authentication results) based on the output feature map.

[0107] Electronic device 1100 includes a processor 1110 (e.g., one or more processors), a memory 1120 (e.g., one or more memories), a camera 1130, a storage device 1140, an input device 1150, an output device 1160, and a network interface 1170. The processor 1110, memory 1120, camera 1130, storage device 1140, input device 1150, output device 1160, and network interface 1170 can communicate with each other via a communication bus 1180. For example, electronic device 1100 can be implemented as at least a part of, for example, a mobile device (such as a mobile phone, smartphone, personal digital assistant (PDA), netbook, tablet computer, or laptop computer), a wearable device (such as a smartwatch, smart bracelet, or smart glasses), a computing device (such as a desktop computer or server), a home appliance (such as a television (TV), smart TV, or refrigerator), a security device (such as a door lock), or a vehicle (such as a smart vehicle).

[0108] Processor 1110 can execute instructions and functions in electronic device 1100. For example, processor 1110 can process instructions stored in memory 1120 or storage device 1140. Processor 1110 can execute the instructions described above. Figures 1 to 12 At least one of the operations described.

[0109] Memory 1120 may store data used for image recognition. Memory 1120 may include a non-transitory computer-readable storage medium or a non-transitory computer-readable storage device. Memory 1120 may store instructions to be executed by processor 1110, and may also store information related to software and / or applications while electronic device 1100 is executing software and / or applications.

[0110] Camera 1130 can capture photos and / or videos and generate input images. For example, camera 1130 can generate user images that include a user (e.g., a face). In one example, camera 1130 can provide a three-dimensional (3D) image that includes depth information associated with an object.

[0111] Storage device 1140 may include a non-transitory computer-readable storage medium or a non-transitory computer-readable storage device. In one example, storage device 1140 may store a larger amount of information than memory 1120 over a relatively long period of time. For example, storage device 1140 may include a magnetic hard disk, optical disk, flash memory, floppy disk, or other forms of non-volatile memory known in the art.

[0112] Input device 1150 can receive input from a user through conventional input schemes using a keyboard and mouse, as well as through new input schemes such as touch input, voice input, and image input. Input device 1150 may include, for example, a keyboard, mouse, touch screen, microphone, or other devices configured to detect input from the user and transmit the detected input to electronic device 1100.

[0113] Output device 1160 can provide the output of electronic device 1100 to a user through a visual channel, an auditory channel, or a tactile channel. Output device 1160 may include, for example, a display, a touch screen, a speaker, a vibration generator, or any other device configured to provide output to a user. Network interface 1170 can communicate with external devices via a wired or wireless network.

[0114] Regarding Figures 1 to 12The described computing devices, processors, memories, electronic devices, cameras, storage devices, input devices, output devices, network interfaces, communication buses, computing processing devices 100 and 1000, processor 1010, memory 1020, electronic devices 1100 and 1110, memory 1120, cameras 1130, storage devices 1140, input devices 1150, output devices 1160, network interfaces 1170, communication buses 1180, devices, units, modules, apparatuses, and other components are implemented or represented by hardware components. Examples of hardware components that can be used to perform the operations described in this application include, where appropriate, controllers, sensors, generators, drivers, memories, comparators, arithmetic logic units, adders, subtractors, multipliers, dividers, integrators, and any other electronic components configured to perform the operations described in this application. In other examples, one or more of the hardware components performing the operations described in this application are implemented by computing hardware (e.g., by one or more processors or computers). A processor or computer may be implemented by one or more processing elements, such as logic gate arrays, controllers and arithmetic logic units, digital signal processors, microcomputers, programmable logic controllers, field-programmable gate arrays, programmable logic arrays, microprocessors, or any other means or combination of means configured to respond to and execute instructions in a defined manner to achieve a desired result. In one example, the processor or computer includes or is connected to one or more memories storing instructions or software executed by the processor or computer. Hardware components implemented by the processor or computer may execute instructions or software (such as an operating system (OS) and one or more software applications running on the OS) for performing the operations described herein. The hardware components may also access, manipulate, process, create, and store data in response to the execution of instructions or software. For simplicity, the singular terms “processor” or “computer” may be used in the description of the examples described herein, but in other examples, multiple processors or computers may be used, or a processor or computer may include multiple processing elements or multiple types of processing elements or both. For example, a single hardware component or two or more hardware components may be implemented by a single processor, or two or more processors, or a processor and a controller. One or more hardware components may be implemented by one or more processors, or processors and controllers, and one or more other hardware components may be implemented by one or more other processors, or additional processors and additional controllers. One or more processors, or processors and controllers, may implement a single hardware component or two or more hardware components.The hardware components can have any one or more different processing configurations, examples of which include: a single processor, a discrete processor, a parallel processor, a single instruction single data (SISD) multiprocessing, a single instruction multiple data (SIMD) multiprocessing, multiple instruction single data (MISD) multiprocessing, and multiple instruction multiple data (MIMD) multiprocessing.

[0115] Figures 1 to 12 The methods for performing the operations described in this application, as shown, are executed by computing hardware (e.g., by one or more processors or a computer), which is implemented to execute instructions or software as described above to perform the operations performed by the methods described in this application. For example, a single operation or two or more operations may be executed by a single processor, or two or more processors, or a processor and a controller. One or more operations may be executed by one or more processors, or a processor and a controller, and one or more other operations may be executed by one or more other processors, or additional processors and additional controllers. One or more processors, or a processor and a controller, may execute a single operation or two or more operations.

[0116] Instructions or software for controlling computing hardware (e.g., one or more processors or computers) to implement hardware components and perform the methods described above can be written as computer programs, code segments, instructions, or any combination thereof to individually or collectively instruct or configure one or more processors or computers, such as machines or special-purpose computers, to perform operations performed by the hardware components and methods described above. In one example, the instructions or software include machine code (such as machine code generated by a compiler) that is directly executed by one or more processors or computers. In another example, the instructions or software include high-level code that is executed by one or more processors or computers using an interpreter. The instructions or software can be written in any programming language based on the block diagrams and flowcharts shown in the accompanying drawings and the corresponding descriptions used herein, wherein the block diagrams and flowcharts shown in the accompanying drawings and the corresponding descriptions used herein disclose algorithms for performing operations performed by the hardware components and methods described above.

[0117] Instructions or software for controlling computing hardware (e.g., one or more processors or computers) to implement hardware components and perform the methods described above, along with any associated data, data files, and data structures, may be recorded, stored, or fixed on one or more non-transitory computer-readable storage media. Examples of non-transitory computer-readable storage media include: read-only memory (ROM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-RLTH, BD-RE, etc. Blu-ray or optical disc storage, hard disk drive (HDD), solid-state drive (SSD), flash memory, card storage (such as multimedia cards or microcards (e.g., Secure Digital (SD) or Extreme Digital (XD))), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid-state drive, and any other device configured to store instructions or software and any associated data, data files, and data structures in a non-transitory manner and to provide the instructions or software and any associated data, data files, and data structures to one or more processors or computers, such that one or more processors or computers can execute the instructions. In one example, the instructions or software and any associated data, data files, and data structures are distributed across a networked computer system, such that the instructions and software and any associated data, data files, and data structures are stored, accessed, and executed in a distributed manner by one or more processors or computers.

[0118] While this disclosure includes specific examples, it will be clear upon understanding this disclosure that various changes in form and detail may be made in these examples without departing from the spirit and scope of the claims and their equivalents. The examples described herein are to be considered descriptive only and not for limiting purposes. The description of features or aspects in each example should be considered applicable to similar features or aspects in other examples. Suitable results may be achieved if the described techniques are performed in a different order, and / or if components in the described system, architecture, apparatus, or circuit are combined in a different manner, and / or replaced or supplemented by other components or their equivalents.

Claims

1. A method for recognizing images using a neural network, the method comprising: Obtain the image to be identified as the input image; Determine the input feature map corresponding to the input image; Obtain the first weight kernel and the pruning information of the first weight kernel in the weight model of the neural network; Based on the pruning information, the processing range of the input feature map for each weight element vector of the first weight kernel is determined; Perform convolution operations between the input feature map and the first weight kernel based on the defined processing range; The output feature map is generated based on the result of the convolution operation; and Based on the output feature map, the image recognition result is output. The pruning information includes: Information including the number of effective vectors corresponding to non-zero vectors in the first weight kernel, and Information about the positions of effective vectors in the first weight kernel. Here, the effective vector is the weight element vector that is retained in the pruning process or as a result of the pruning process. Each weight element vector corresponds to a point tensor, and the size of each weight element vector is 1×1×C, where C is the number of input channels. In this context, each weight element vector in the first weight kernel corresponds to a valid vector, which is distinct from the invalid vector removed from the first weight kernel during the pruning process.

2. The method according to claim 1, wherein, The step of determining the processing range for each weight element vector includes: based on pruning information, determining a first processing range in the input feature map corresponding to the first weight element vector of the first weight kernel, and... The steps for performing a convolution operation include generating a first intermediate output plane by performing a convolution operation between a first weight element vector and a first processing range.

3. The method according to claim 2, wherein, The convolution operation between the first weight element vector and the first processing range includes a 1 × 1 convolution operation.

4. The method according to claim 3, wherein, The convolution operation between the first weight element vector and the first processing range includes a 1 × 1 convolution operation between each input element vector of the first processing range and the first weight element vector.

5. The method according to claim 2, wherein, The steps for generating the output feature map include: generating the first output plane of the output feature map by accumulating intermediate output planes including the first intermediate output plane.

6. The method according to claim 1, wherein, The steps for generating the output feature map include: generating the first output plane of the output feature map by accumulating the intermediate output planes corresponding to the results of the convolution operation.

7. The method according to claim 6, wherein, The first weighting kernel and the first output plane correspond to the first output channel.

8. The method according to claim 1, wherein, The weighting model includes: multiple weight kernels, wherein the multiple weight kernels include a first weight kernel, and The weight element vectors of the multiple weight kernels correspond to non-zero vectors.

9. The method according to claim 8, wherein, The sparsity of each of the multiple weight kernels in the weight model is increased by using training processing with corresponding basic weight kernels and scaling matrices.

10. The method according to claim 9, wherein, The training process includes, for each of the plurality of weight kernels: Based on the loss function, sparsity is induced to scale the matrix; and The weight kernel is generated by pointwise multiplication between the basic weight kernel and the scaling matrix that induces sparsity.

11. The method according to any one of claims 1 to 10, further comprising: Use a camera to generate the input image.

12. A non-transitory computer-readable storage medium storing instructions, which, when executed by a processor, configure the processor to perform the method according to any one of claims 1 to 11.

13. An apparatus for recognizing images via a neural network, the apparatus comprising: The processor is configured as follows: Obtain the image to be identified as the input image; Determine the input feature map corresponding to the input image; Obtain the first weight kernel and the pruning information of the first weight kernel in the weight model of the neural network; Based on the pruning information, the processing range of the input feature map for each weight element vector of the first weight kernel is determined; Perform convolution operations between the input feature map and the first weight kernel based on the defined processing range; The output feature map is generated based on the result of the convolution operation; and Based on the output feature map, the image recognition result is output. The pruning information includes: Information including the number of effective vectors corresponding to non-zero vectors in the first weight kernel, and Information about the positions of effective vectors in the first weight kernel. Here, the effective vector is the weight element vector that is retained in the pruning process or as a result of the pruning process. Each weight element vector corresponds to a point tensor, and the size of each weight element vector is 1×1×C, where C is the number of input channels. In this context, each weight element vector in the first weight kernel corresponds to a valid vector, which is distinct from the invalid vector removed from the first weight kernel during the pruning process.

14. The device according to claim 13, wherein, The processor is configured as follows: To determine the processing range for each weight element vector, a first processing range corresponding to the first weight element vector of the first weight kernel in the input feature map is determined based on pruning information; and To perform the convolution operation, a first intermediate output plane is generated by performing a convolution operation between the first weight element vector and the first processing range.

15. The device according to claim 14, wherein, In order to generate the output feature map, the processor is configured to generate the first output plane of the output feature map by accumulating the intermediate output planes that include the first intermediate output plane.

16. The device according to claim 13, wherein, In order to generate the output feature map, the processor is configured to generate the first output plane of the output feature map by accumulating the intermediate output plane corresponding to the result of the convolution operation.

17. The device according to claim 13, wherein, The weighting model includes: multiple weight kernels, wherein the multiple weight kernels include a first weight kernel, and The weight element vectors of the multiple weight kernels correspond to non-zero vectors.

18. The device according to claim 17, wherein, The sparsity of each of the multiple weight kernels in the weight model is increased by using training processing with corresponding basic weight kernels and scaling matrices.

19. The device according to any one of claims 13 to 18, wherein, The device is an electronic device that includes a camera configured to generate an input image.

20. The device according to any one of claims 13 to 18, further comprising: The memory stores instructions that, when executed by the processor, configure the processor to perform the following steps: obtaining an image to be recognized, determining an input feature map corresponding to the input image, obtaining a first weight kernel, determining a processing range, performing convolution, generating an output feature map, and outputting an image recognition result.

21. An electronic device, comprising: The camera is configured to generate the input image; and The processor is configured as follows: Determine the input feature map corresponding to the input image; Obtain the first weight kernel and the pruning information of the first weight kernel in the weight model of the neural network; Based on the pruning information, the processing range of the input feature map for each weight element vector of the first weight kernel is determined; Perform convolution operations between the input feature map and the first weight kernel based on the defined processing range; The output feature map is generated based on the result of the convolution operation; and The recognition result of the input image is generated based on the output feature map. The pruning information includes: Information including the number of effective vectors corresponding to non-zero vectors in the first weight kernel, and Information about the positions of effective vectors in the first weight kernel. Here, the effective vector is the weight element vector that is retained in the pruning process or as a result of the pruning process. Each weight element vector corresponds to a point tensor, and the size of each weight element vector is 1×1×C, where C is the number of input channels. In this context, each weight element vector in the first weight kernel corresponds to a valid vector, which is distinct from the invalid vector removed from the first weight kernel during the pruning process.

22. The electronic device according to claim 21, wherein, The processor is configured as follows: To determine the processing range for each weight element vector, a first processing range corresponding to the first weight element vector of the first weight kernel in the input feature map is determined based on pruning information; and To perform the convolution operation, a first intermediate output plane is generated by performing a convolution operation between the first weight element vector and the first processing range.

Images