Image recognition method, device and electronic device

Abstract

Disclosed are a method and apparatus for image recognition and an electronic device. The method includes obtaining image data for image recognition, performing a respective operation for each layer of a neural network based on the image data to obtain a result of the image recognition, and outputting the result of the image recognition, wherein the performing the respective operation includes extracting a first target feature vector from a target feature map, extracting a first weight vector matching the first target feature vector from a first type weight element based on a matching relationship for a depth-wise convolution operation, generating a first intermediate feature vector by performing a multiplication operation between the first target feature vector and the first weight vector, generating a first hidden feature vector by accumulating the first intermediate feature vector and a second intermediate feature vector generated based on a second target feature vector, and generating a first output feature vector of an output feature map based on a point-wise convolution operation between the first hidden feature vector and a second type weight element.

Description

[0001] This application claims the benefit of Korean Patent Application No. 10-2020-0028360 filed on March 6, 2020, in the Korean Intellectual Property Office, the disclosure of which is incorporated herein by reference in its entirety for all purposes. Technical Field

[0002] The following description relates to methods and apparatus for processing convolution operations on layers in a neural network. Background Art

[0003] Automated recognition processing has been achieved through neural network models implemented by processors as specialized computing structures. After training, these neural network models provide intuitive mappings for computations between input patterns and output patterns. The ability to train to generate such mappings is the learning capability of the neural network. Furthermore, neural networks trained and specialized through specialized training have the ability to generalize, for example, to provide relatively accurate outputs for untrained input patterns. Summary of the Invention

[0004] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter.

[0005] In one general aspect, a method for image recognition includes: obtaining image data for image recognition as input data of a neural network; performing corresponding operations on each layer of the neural network based on the image data to obtain a result of image recognition; and outputting the result of image recognition, wherein the step of performing corresponding operations on each layer of at least one of the layers included in the neural network includes: extracting a first target feature vector from a target feature map; extracting a first weight vector matching the first target feature vector from the first type weight element based on a matching relationship between the target feature vector of the target feature map and the weight vector of the first type weight element for a depth-wise convolution operation; generating a first intermediate feature vector by performing a multiplication operation between the first target feature vector and the first weight vector; generating a first hidden feature vector by accumulating the first intermediate feature vector generated based on the first target feature vector and the second intermediate feature vector generated based on the second target feature vector; and generating a first output feature vector of the output feature map based on a point-by-point convolution operation between the first hidden feature vector and the second type weight element.

[0006] In another general aspect, an image recognition device includes: a memory configured to store executable instructions; and a processor configured to execute the instructions to: obtain image data for image recognition as input data of a neural network; perform corresponding operations on each layer of the neural network based on the image data to obtain a result of image recognition; and output the result of image recognition, wherein the processing of performing the corresponding operations on each layer of at least one layer included in the layers of the neural network includes: extracting a first target feature vector from a target feature map; extracting a first weight vector matching the first target feature vector from the first type weight element based on a matching relationship between the target feature vector of the target feature map and the weight vector of the first type weight element for a depth-wise convolution operation; generating a first intermediate feature vector by performing a multiplication operation between the first target feature vector and the first weight vector; generating a first hidden feature vector by accumulating the first intermediate feature vector generated based on the first target feature vector and the second intermediate feature vector generated based on the second target feature vector; and generating a first output feature vector of the output feature map based on a point-by-point convolution operation between the first hidden feature vector and the second type weight element.

[0007] In another general aspect, an electronic device includes: an input device configured to receive image data for image recognition; an output device configured to output a result of the image recognition; a memory configured to store executable instructions; and a processor configured to execute the instructions to: obtain the image data as input data of a neural network; perform corresponding operations on each layer of the neural network based on the image data to obtain a result of the image recognition; and output the result of the image recognition, wherein the processing of performing the corresponding operations on each layer of at least one of the layers included in the neural network includes: extracting a first target feature vector from a target feature map; performing the corresponding operations based on the target A matching relationship between a target feature vector of the feature map and a weight vector of a first type of weight element for a depth-wise convolution operation, extracting a first weight vector matching the first target feature vector from the first type of weight element; generating a first intermediate feature vector by performing a multiplication operation between the first target feature vector and the first weight vector; generating a first hidden feature vector by accumulating a first intermediate feature vector generated based on the first target feature vector and a second intermediate feature vector generated based on the second target feature vector; and generating a first output feature vector of the output feature map based on a point-by-point convolution operation between the first hidden feature vector and the second type of weight element.

[0008] In another general aspect, a processor-implemented method for processing a convolution operation on a layer in a neural network, the method comprising: extracting a first target feature vector from a target feature map; extracting a first weight vector that matches the first target feature vector from the first type of weight element based on a matching relationship between the target feature vector of the target feature map and a weight vector of the first type of weight element for a depth-wise convolution operation; generating a first intermediate feature vector by performing a multiplication operation between the first target feature vector and the first weight vector; generating a first hidden feature vector by accumulating a first intermediate feature vector generated based on the first target feature vector and a second intermediate feature vector generated based on the second target feature vector; and generating a first output feature vector of an output feature map based on a point-by-point convolution operation between the first hidden feature vector and the second type of weight element.

[0009] The first hidden feature vector may include the first intermediate feature vector and the second intermediate feature vector, and may be completed in response to all required elements being accumulated.

[0010] The step of generating the first hidden feature vector may include generating the first hidden feature vector based on accumulating the first intermediate feature vector and the second intermediate feature vector in the first space of the hidden buffer.

[0011] In response to the first output feature vector being generated, the first space of the hidden buffer may be reused to accumulate intermediate feature vectors for generating a second hidden feature vector.

[0012] A plurality of weight vectors including a first weight vector may be matched with the first target feature vector based on the matching relationship, and a plurality of hidden vectors may be generated based on a multiplication operation between the first target feature vector and each of the plurality of weight vectors.

[0013] The target feature map, the first type weight elements, the second type weight elements, and the output feature map may each be in an interleaved format.

[0014] The first target feature vector, the first weight vector, the first intermediate feature vector, the second intermediate feature vector, the first hidden feature vector, and the first output feature vector may each correspond to a channel direction.

[0015] The method may further include: extracting a second target feature vector from the target feature map; extracting a second weight vector matching the second target feature vector from the first type weight element based on the matching relationship; and generating a second intermediate feature vector by performing a multiplication operation between the second target feature vector and the second weight vector.

[0016] The step of generating the first output feature vector may include generating the first output feature vector by performing a point-by-point convolution operation between the first hidden feature vector and respective weight vectors of the second type weight elements.

[0017] The depthwise convolution operation and the pointwise convolution operation may constitute at least a portion of a depthwise separable convolution (DSC) operation.

[0018] The first type of weight elements may be used to extract spatial features, and the second type of weight elements may be used to extract combined features.

[0019] The target feature map can correspond to the input feature map or the hidden feature map.

[0020] The depth-wise convolution operation and the point-wise convolution operation may each be processed for each single instruction multiple data (SIMD) operation.

[0021] The required elements are determined based on the first target feature vector and the first weight vector.

[0022] The required elements are determined based on a plurality of target feature vectors of the target feature map and a plurality of weight vectors respectively matched with the plurality of target feature vectors.

[0023] In another general aspect, a device for processing a convolution operation on a layer in a neural network includes: a memory configured to store executable instructions, and a processor configured to execute the instructions to: extract a first target feature vector from a target feature map; extract a first weight vector matching the first target feature vector from the first type weight element based on a matching relationship between the target feature vector of the target feature map and the weight vector of the first type weight element for a depth-wise convolution operation; generate a first intermediate feature vector by performing a multiplication operation between the first target feature vector and the first weight vector; generate a first hidden feature vector by accumulating the first intermediate feature vector generated based on the first target feature vector and the second intermediate feature vector generated based on the second target feature vector; and generate a first output feature vector of the output feature map based on the point-by-point convolution operation between the first hidden feature vector and the second type weight element.

[0024] The first hidden feature vector may include the first intermediate feature vector and the second intermediate feature vector, and may be completed in response to all required elements being accumulated.

[0025] The processor may be configured to generate a first hidden feature vector based on accumulating the first intermediate feature vector and the second intermediate feature vector in the first space of the hidden buffer, and in response to the first output feature vector being generated, the first space of the hidden buffer may be reused to accumulate the intermediate feature vectors for generating the second hidden feature vector.

[0026] A plurality of weight vectors including a first weight vector may be matched with the first target feature vector based on the matching relationship, and a plurality of hidden vectors may be generated based on a multiplication operation between the first target feature vector and each of the plurality of weight vectors.

[0027] The required elements may be determined based on the first target feature vector and the first weight vector.

[0028] In another general aspect, an electronic device includes: a memory configured to store executable instructions, and a processor configured to execute the instructions to: extract a first target feature vector from a target feature map; extract a first weight vector matching the first target feature vector from the first type weight element based on a matching relationship between the target feature vector of the target feature map and the weight vector of the first type weight element for a depth-wise convolution operation; generate a first intermediate feature vector by performing a multiplication operation between the first target feature vector and the first weight vector; generate a first hidden feature vector by accumulating the first intermediate feature vector generated based on the first target feature vector and the second intermediate feature vector generated based on the second target feature vector; and generate a first output feature vector of the output feature map based on a point-by-point convolution operation between the first hidden feature vector and the second type weight element.

[0029] The first hidden feature vector may include the first intermediate feature vector and the second intermediate feature vector, and may be completed in response to all required elements being accumulated.

[0030] Other features and aspects will be apparent from the following detailed description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 An example of processing a convolution operation by a processing device is shown.

[0032] Figure 2 and Figure 3 Showing an example of depthwise separable convolution (DSC).

[0033] Figure 4 An example of processing DSC is shown.

[0034] Figure 5 and Figure 6 An example of determining a matching relationship between a target feature vector and a weight vector is shown.

[0035] Figure 7 and Figure 8 An example of generating and storing intermediate feature vectors is shown.

[0036] Figure 9An example of generating a latent feature vector based on the accumulation of intermediate feature vectors is shown.

[0037] Figure 10 An example of generating an output feature vector through a point-by-point convolution operation is shown.

[0038] Figure 11 An example of reusing a buffer is shown.

[0039] Figure 12 An example of processing a convolution operation is shown.

[0040] Figure 13 An example of a processing device for processing a convolution operation is shown.

[0041] Figure 14 An example of an electronic device is shown.

[0042] Throughout the drawings and detailed description, unless otherwise described or provided, the same drawing reference numerals will be understood to refer to the same elements, features, and structures. The drawings may not be to scale, and the relative sizes, proportions, and depictions of the elements in the drawings may be exaggerated for clarity, illustration, and convenience. DETAILED DESCRIPTION

[0043] The following detailed description is provided to help the reader gain a comprehensive understanding of the methods, devices, and / or systems described herein. However, various changes, modifications, and equivalents of the methods, devices, and / or systems described herein will be apparent upon understanding the disclosure of this application. For example, the order of operations described herein is merely exemplary and is not limited to those described herein, but may be changed as will be apparent upon understanding the disclosure of this application, except for operations that must occur in a particular order. In addition, descriptions of features known in the art may be omitted for clarity and conciseness.

[0044] The features described herein may be implemented in different forms and are not to be construed as limited to the examples described herein. Rather, the examples described herein have been provided to illustrate only some of the many possible ways to implement the methods, devices, and / or systems described herein that will be apparent upon understanding the disclosure of this application.

[0045] Although terms such as "first," "second," and "third" may be used herein to describe various members, components, regions, layers, or portions, these members, components, regions, layers, or portions are not limited by these terms. Instead, these terms are used only to distinguish one member, component, region, layer, or portion from another member, component, region, layer, or portion. Thus, without departing from the teachings of the examples described herein, a first member, first component, first region, first layer, or first portion referred to in the examples may also be referred to as a second member, second component, second region, second layer, or second portion.

[0046] Throughout the specification, when a component is described as being "connected to" or "coupled to" another component, the component may be directly "connected to" or "coupled to" the other component, or there may be one or more other components interposed therebetween. Conversely, when an element is described as being "directly connected to" or "directly coupled to" another element, there may not be other elements interposed therebetween. Similarly, similar expressions (e.g., "between" versus "immediately between" and "adjacent" versus "immediately adjacent") may also be interpreted in the same manner. As used herein, the term "and / or" includes any one of the relevant listed items, or any combination of any two or more thereof.

[0047] The terms used herein are intended only to describe various examples and are not intended to limit the disclosure. Unless the context clearly indicates otherwise, singular forms are intended to include plural forms. The terms "comprise," "include," and "have" indicate the presence of stated features, quantities, operations, components, elements, and / or combinations thereof, but do not preclude the presence or addition of one or more other features, quantities, operations, components, elements, and / or combinations thereof.

[0048] Hereinafter, examples will be described in detail with reference to the accompanying drawings. Throughout the drawings, the same reference numerals are used for the same elements.

[0049] Figure 1 An example of processing a convolution operation by a processing device is shown. Figure 1 , the processing device 100 may include a neural network 110 and process operations on the neural network 110. For example, the operations on the neural network 110 may include convolution operations on layers in the neural network 110. When the operations on the neural network 110 are processed, output data of the neural network 110 may be generated, and the output data of the neural network 110 may be used for techniques of image recognition (such as object recognition, user authentication, etc.).

[0050] The neural network 110 can perform an object recognition operation or a user authentication operation by mapping input data and output data in a nonlinear relationship based on deep learning. Deep learning is a machine learning technology for solving problems (such as image recognition or speech recognition) from large data sets. Deep learning is interpreted as solving an optimization problem of such a process: while training the neural network 110 using prepared training data, finding a point that minimizes energy. Through supervised learning or unsupervised learning of deep learning, the structure of the neural network 110 or the weight corresponding to the model is obtained, and the input data and output data are mapped to each other through the weights.

[0051] The neural network 110 may correspond to a deep neural network (DNN) including a plurality of layers. The plurality of layers may include an input layer, at least one hidden layer, and an output layer. Figure 1 The first layer, the second layer, and the nth layer shown in the figure may be at least a portion of a plurality of layers, where n is an integer greater than 1. The neural network 110 may include any one or any combination of networks such as a fully connected network, a convolutional neural network (CNN), and a recurrent neural network (RNN). For example, at least a portion of the plurality of layers in the neural network 110 may correspond to a CNN, and another portion of the plurality of layers in the neural network 110 may correspond to a fully connected network.

[0052] The data input to each layer in a CNN can be referred to as an input feature map, and the data output from each layer can be referred to as an output feature map. The input feature map and the output feature map can also be referred to as activation data. In an input layer, the input feature map can correspond to the input data. For example, when image data is input to the first layer in a CNN, the image data can be the input feature map corresponding to the first layer. For example, when data output from the first layer is input to the second layer, the data can be the input feature map corresponding to the second layer.

[0053] To process operations related to the neural network 110, the processing device 100 may process various operation blocks. The operation block may include any one or any combination of at least one convolution operation (e.g., a single convolution operation or multiple convolution operations), a skip connection (also known as a "residual connection"), and a pooling operation. For example, the operation block may include a convolution operation on a layer (e.g., a convolution layer) in the neural network 110. The processing device 100 may perform a convolution operation between an input feature map and a weight kernel for each convolution layer, and generate an output feature map based on the result of the convolution operation.

[0054] The processing device 100 can perform a convolution operation by processing the operation elements included in the convolution operation in various ways. The processing device 100 can perform the convolution operation in a depth-wise separable convolution (DSC) manner. DSC is a convolution operation for performing a convolution operation by distinguishing between spatial feature extraction and combined feature extraction. In this example, spatial features can be extracted by a depth-wise convolution operation, and combined features can be extracted by a point-wise convolution operation.

[0055] If the width and depth of the neural network 110 are large enough, the neural network 110 may have sufficient capabilities to implement predetermined functions. When learning a sufficiently large amount of training data through an appropriate training process, the neural network 110 may achieve optimized performance.

[0056] Hereinafter, neural network 110 or weight kernels may be referred to as being "pre-trained." Here, "pre-trained" means before neural network 110 is "activated." A "activated" neural network 110 refers to a neural network 110 that is ready for inference. For example, a "activated" neural network 110 may include a neural network 110 that is loaded into memory, or a neural network 110 that receives input data for inference after being loaded into memory.

[0057] Figure 2 and Figure 3 An example of DSC is shown. As described above, DSC is a convolution operation for performing a convolution operation by distinguishing spatial feature extraction and combined feature extraction, and may include at least one depth-wise convolution operation and at least one point-wise convolution operation. For DSC, at least one depth-wise convolution operation and at least one point-wise convolution operation can be combined in various modes. Figure 2 and Figure 3 An example of DSC is shown.DSC can be performed by combining different modes of at least one depth-wise convolution operation and at least one point-wise convolution operation.

[0058] Reference Figure 2 DSC includes depth-wise convolution operations and point-wise convolution operations. The hidden feature map 230 is generated based on the depth-wise convolution operation between the input feature map 210 and the weight element 220, and the output feature map 250 is generated based on the point-wise convolution operation between the hidden feature map 230 and the weight element 240.

[0059] exist Figure 2In FIG, the input feature map 210, the weight element 220, the hidden feature map 230, and the output feature map 250 are each shown in the form of a set of planes, and the weight element 240 is shown in the form of bars. Each plane and each bar can be distinguished for each channel. Each plane can represent data arranged in two dimensions, and each bar can represent data arranged in one dimension. Here, the data can be an element. The planes arranged in succession can be represented in the form of a three-dimensional (3D) box. For example, Figure 2 The input feature map 210 can be expressed as Figure 3 The input feature map 310 is provided in this form for a better understanding of the processing operation. In practice, the data can be stored in a memory in one dimension.

[0060] Input feature map 210 includes input feature planes 2101, 2102, ..., 210C. Each of input feature planes 2101, 2102, ..., 210C may have a width W and a height H. The number of input feature planes 2101, 2102, ..., 210C may be C. Input feature planes 2101, 2102, ..., 210C may correspond to different input channels. Therefore, C may represent the number of input channels.

[0061] Here, W and H may represent the number of input elements. That is, each of the input feature planes 2101, 2102, ..., 210C may include W input elements in the horizontal direction and H input elements in the vertical direction. hereinafter, unless otherwise mentioned, it may be assumed that the elements of the convolution operation (such as the input feature map 210, the weight element 220, the hidden feature map 230, the weight element 240, and the output feature map 250) are configured element-by-element.

[0062] Weight element 220 includes weight planes 2201, 2202, ..., 220C. Each of weight planes 2201, 2202, ..., 220C may have a width K1 and a height K2. Hereinafter, for ease of description, it may be assumed that K1 and K2 are equal to K. For example, it may be assumed that the size of each of weight planes 2201, 2202, ..., 220C is 3×3. The number of weight planes 2201, 2202, ..., 220C may be C. Like the input feature planes 2101, 2102, ..., 210C, weight planes 2201, 2202, ..., 220C may correspond to different input channels.

[0063] The hidden feature map 230 is generated based on a depthwise convolution operation between the input feature map 210 and the weight elements 220. The weight elements 220 are used for spatial feature extraction, and hereinafter, the type of the weight elements 220 will be referred to as the first type.

[0064] The depthwise convolution operation may include a two-dimensional (2D) convolution operation between the input feature map 210 and the weight element 220. The 2D convolution operation between the input feature plane and the weight plane of the input channels corresponding to each other may be performed. The 2D convolution operation for the depthwise convolution operation may include, for example, a 2D convolution operation between the input feature plane 2101 and the weight plane 2201, a 2D convolution operation between the input feature plane 2102 and the weight plane 2202, ..., and a 2D convolution operation between the input feature plane 210C and the weight plane 220C.

[0065] Hidden feature map 230 includes hidden feature planes 2301, 2302, ..., 230C. Each of hidden feature planes 2301, 2302, ..., 230C may have a width W and a height H. It is assumed that the size of hidden feature planes 2301, 2302, ..., 230C is W×H, which is the same as the size of input feature planes 2101, 2102, ..., 210C. However, in another example, the size of hidden feature planes 2301, 2302, ..., 230C may be W2×H2, which is different from the size of input feature planes 2101, 2102, ..., 210C. The number of hidden feature planes 2301, 2302, ..., 230C may be C. Like the input feature planes 2101 , 2102 , . . . , 210C and the weight planes 2201 , 2202 , . . . , 220C, the hidden feature planes 2301 , 2302 , . . . , 230C may correspond to different input channels.

[0066] The hidden feature plane of the corresponding input channel can be generated according to each 2D convolution operation. For example, the hidden feature plane 2301 can be generated according to the 2D convolution operation between the input feature plane 2101 and the weight plane 2201, the hidden feature plane 2302 can be generated according to the 2D convolution operation between the input feature plane 2102 and the weight plane 2202, ..., the hidden feature plane 230C can be generated according to the 2D convolution operation between the input feature plane 210C and the weight plane 220C.

[0067] The output feature map 250 is generated based on a point-by-point convolution operation between the hidden feature map 230 and the weight elements 240. The weight elements 240 include weight vectors 2401, 2402, ..., 240N. Each of the weight vectors 2401, 2402, ..., 240N may have a size of 1×1×C, and the number of weight vectors 2401, 2402, ..., 240N may be N. The weight vectors 2401, 2402, ..., 240N may correspond to different output channels. Therefore, N may represent the number of output channels. The weight elements 240 are used for combined feature extraction. Hereinafter, the type of weight element 240 will be referred to as the second type.

[0068] The point-by-point convolution operation may include a 1×1 convolution operation between the hidden feature map 230 and the weight element 240. In one example, the 1×1 convolution operation between each of the weight vectors 2401, 2402, ..., 240N and the hidden feature map 230 may be performed based on a sliding window. The 1×1 convolution operation based on a sliding window for the point-by-point convolution operation may include, for example, a 1×1 convolution operation based on a sliding window between the hidden feature map 230 and the weight vector 2401, a 1×1 convolution operation based on a sliding window between the hidden feature map 230 and the weight vector 2402, ..., and a 1×1 convolution operation based on a sliding window between the hidden feature map 230 and the weight vector 240N.

[0069] Output feature map 250 includes output feature planes 2501, 2502, ..., 250N. Each of output feature planes 2501, 2502, ..., 250N may have a width W and a height H. It is assumed that the size of output feature planes 2501, 2502, ..., 250N is W×H, the same as the size of input feature planes 2101, 2102, ..., 210C. However, in another example, the size of output feature planes 2501, 2502, ..., 250N may be W3×H3, which is different from the size of input feature planes 2101, 2102, ..., 210C. The number of output feature planes 2501, 2502, ..., 250N may be N. Like weight vectors 2401, 2402, ..., 240N, output feature planes 2501, 2502, ..., 250N may correspond to different output channels.

[0070] A 1×1 convolution operation can be performed by sliding the hidden feature map 230 onto each of the weight vectors 2401, 2402, ..., 240N, and an output feature plane of the corresponding output channel can be generated based on each sliding window-based 1×1 convolution operation. For example, a hidden feature vector can be obtained by extracting elements having the same offset (e.g., the first offset) from each of the hidden feature planes 2301, 2302, ..., 230C, and an output element having the offset (e.g., the first offset) in the output feature plane 2501 can be generated based on a 1×1 convolution operation between the obtained hidden feature vector and the weight vector 2401.

[0071] Hidden feature vectors corresponding to other offsets of the hidden feature planes 2301, 2302, ..., 230C can be obtained through a sliding window, and the output feature plane 2501 can be completed by performing a 1×1 convolution operation between the obtained hidden feature vector and the weight vector 2401. Completing the output feature plane 2501 may indicate determining the value of the output element in the output feature plane 2501. Similarly, the output feature map 2502 may be generated based on a sliding window-based 1×1 convolution operation between the hidden feature map 230 and the weight vector 2402, ..., and the output feature map 250N may be generated based on a sliding window-based 1×1 convolution operation between the hidden feature map 230 and the weight vector 240N.

[0072] Reference Figure 3 , according to another example, the DSC includes a depth-wise convolution operation and a point-wise convolution operation. Figure 2 Different from DSC, Figure 3 The DSC includes two point-by-point convolution operations. The first point-by-point convolution operation can be used for expansion, and the second point-by-point convolution operation can be used for compression.

[0073] Hidden feature map 330 is generated based on a point-by-point convolution operation between input feature map 310 and weight element 320, and hidden feature map 350 is generated based on a depth-wise convolution operation between hidden feature map 330 and weight element 340. Output feature map 370 is generated based on a point-by-point convolution operation between hidden feature map 350 and weight element 360. Figure 2 The description can be applied to the operation processing of point-by-point convolution operation and depth-by-depth convolution operation. C can represent the number of input channels, N can represent the number of hidden channels, and M can represent the number of output channels. Figure 3 The “*” in represents a convolution operation.

[0074] In the following, the term "target feature map" will be used. The hidden feature map can be generated based on a depth-wise convolution operation on the target feature map, and the output feature map can be generated based on a point-wise convolution operation on the hidden feature map. In one example, the target feature map can be Figure 2 The input feature map 210 or Figure 3 The hidden feature map 330 of . In addition, the hidden feature map generated based on the target feature map can be Figure 2 Hidden feature map 230 or Figure 3 The hidden feature map 350 of . The output feature map can be Figure 2 The output feature map 250 or Figure 3 The output feature map 370 of .

[0075] Data can be stored in a memory in various sorting modes. For example, data sorting modes may include a planar mode and an interleaved mode. Data sorting is applied in the order of width, height, and channel according to the planar mode, and data sorting is applied in the order of channel, width, and height according to the interleaved mode.

[0076] Before the DSC operation, an interleaved format may be applied to the data to be used for the DSC operation. For example, an interleaved format may be applied to Figure 2 The input feature map 210, weight element 220 and weight element 240 of FIG, and the interleaved format can be applied to Figure 3 Input feature map 310, weight element 320, weight element 340 and weight element 360 of.

[0077] Furthermore, the interleaved format of the data can be maintained until the output feature map is generated. For example, Figure 2 , the interleaved hidden feature map 230 may be generated by performing a depthwise convolution operation between the interleaved input feature map 210 and the interleaved weight elements 220. In addition, the interleaved output feature map 250 may be generated by performing a depthwise convolution operation between the interleaved hidden feature map 230 and the interleaved weight elements 240. Similarly, in Figure 3 In the example of , the interleaved format of the data may be maintained until the output feature map 370 in the interleaved format is generated.

[0078] The interleaving method can be beneficial to ensure the continuity of the data. Therefore, memory access can be greatly reduced by applying an interleaved format to the data. In addition, the interleaved method can be beneficial to single instruction multiple data (SIMD) processing. SIMD represents an operation process in which a processor uses a single instruction to process multiple data. In addition, according to the example explained below, SIMD processing can be performed by loading data according to SIMD operation units (e.g., 4, 8, 16, 32, 64, 128, etc.), whereby the efficiency of SIMD processing can be maximized. For example, depth-wise convolution operations and point-by-point convolution operations can each be processed according to SIMD operation units. In addition, since the data in the interleaved format is maintained until the output feature map is generated, an additional column buffer for converting the data format may not be required.

[0079] In addition, according to an example, a depth-wise convolution operation is performed for each target feature vector using a matching relationship between the target feature vector and the weight vector, wherein the operation is performed for each target feature vector arranged in an interleaved manner, so that an additional column buffer for performing SIMD operations is not required. For each target feature vector, the result of the depth-wise convolution operation is accumulated in a hidden buffer, and the point-by-point convolution operation is performed for each single hidden feature vector when the hidden feature vector is completed. For example, if a first hidden feature vector is generated for each target feature vector by a depth-wise convolution operation, a first output feature vector can be generated for the first hidden feature vector by a point-by-point convolution operation. If the first output feature vector is generated, the first hidden feature vector is not further used for DSC. Therefore, the memory space used for the first hidden feature vector can be reused to store another hidden feature vector. Therefore, the memory space used to store the hidden feature map can be saved.

[0080] Figure 4 An example of processing DSC is shown. Figure 4 The operations in the embodiment may be performed in the order and manner as shown, however, the order of some operations may be changed or some operations may be omitted without departing from the spirit and scope of the illustrative examples described. Figure 4 Many of the operations shown in FIG. 1 may be performed in parallel or simultaneously. Figure 4 One or more blocks and combinations of blocks may be implemented by a computer based on dedicated hardware (such as a processor that performs specific functions) or a combination of dedicated hardware and computer instructions. Figure 4 In addition to the description, Figures 1 to 3 The description can also be applied to Figure 4 , and are incorporated herein by reference. Therefore, the above description may not be repeated here.

[0081] Reference Figure 4 In operation 410, the processing device applies an interleaved format to the target feature map and the weight elements. As described above, the target feature map may correspond to Figure 2 The input feature map 210 and Figure 3 Hidden feature map 330. However, in Figure 3 In the example of , the interleaved format may be applied to the input feature map 310. In this example, the interleaved format hidden feature map 330 may be generated by a point-by-point convolution operation between the interleaved format input feature map 310 and the interleaved format weight elements 320. Therefore, there may be no need to perform a separate format conversion for the hidden feature map 330.

[0082] Furthermore, the weight elements to which format conversion is applied in this operation may correspond to Figure 2 The weight elements 220 and 240 and Figure 3 However, during the training process, the interleaved format may be applied to the second type weight elements (e.g., Figure 2 The weight element 240 and Figure 3 Therefore, separate format conversion may not be performed for the second type weight elements.

[0083] In operation 420, the processing device obtains the target feature vector T i The processing device can extract the target feature vector T from the target feature map i To obtain the target feature vector T i Target feature vector T i It may correspond to a portion of the target feature map. For example, the size of the target feature map may be W×H×C or W×H×N, and the size of the target feature vector may be 1×1×C or 1×1×N. In this example, the target feature map may include W×H target feature vectors T i , where i can be a natural number between 1 and W×H. i can be initially set to 1.

[0084] In operation 430, the processing device obtains the target feature vector T i Matching weight vector. The weight vector can be part of a weight element. Here, the weight element can be of the first type. For example, the size of the weight element can be K×K×C or K×K×N, and the size of the weight vector can be 1×1×C or 1×1×N. In this example, the weight element can include K×K weight vectors.

[0085] In the following, the weight vector can be represented by Wl m Here, l represents the type of weight element to which the weight vector belongs, and m represents the index of the weight vector. For example, W11 may be the first weight vector belonging to the first type of weight element, and W29 may be the ninth weight vector belonging to the second type of weight element.

[0086] The processing device may use the matching relationship between the target feature vector and the weight vector to perform a depth-wise convolution operation. During the sliding window processing for the depth-wise convolution operation, a matching relationship between the target feature vector and at least one weight vector that needs to be operated with the target feature vector may be formed. Figure 5 and Figure 6 Describes the matching relationship between the target feature vector and the weight vector.

[0087] Figure 5 and Figure 6 An example of determining the matching relationship between the target feature vector and the weight vector for the depth-wise convolution operation is shown. Figure 5 , the target feature map 510 includes target feature vectors T1, T2, T3, .... The area corresponding to the weight element 520 is indicated by a dotted line. The correspondence between the weight element 520 and the target feature map 510 can be known in a sliding window process performed by the position of the weight element 520 over time t.

[0088] like Figure 6 As shown in the weight element 610, the weight vector of the weight element 520 can be indicated as W11 to W19. In this example, the matching relationship between the target feature vector T1 and the weight vector W15 can be Figure 5 In addition, a matching relationship between the target feature vector T1 and the weight vector W14 can be formed at t=2, a matching relationship between the target feature vector T1 and the weight vector W12 can be formed at t=α+1, and a matching relationship between the target feature vector T1 and the weight vector W11 can be formed at t=α+2, where α is an integer greater than 2. Finally, matching relationships between the target feature vector T1 and the weight vectors W15, W14, W12, and W11 can be formed.

[0089] Reference Figure 6 , the matching relationship between some target feature vectors and at least one weight vector is shown. As described above, the matching relationship between the target feature vector T1 and the weight vectors W15, W14, W12 and W11 can be formed. In addition, the matching relationship between the target feature vector T2 and the weight vectors W16, W15, W14, W13, W12 and W11 can be formed. For example, the matching relationship between the target feature vector T2 and the weight vector W16 is formed at t=1, the matching relationship between the target feature vector T2 and the weight vector W15 is formed at t=2, the matching relationship between the target feature vector T2 and the weight vector W14 is formed at t=3, the matching relationship between the target feature vector T2 and the weight vector W13 is formed at t=α+1, the matching relationship between the target feature vector T2 and the weight vector W12 is formed at t=α+2, and the matching relationship between the target feature vector T2 and the weight vector W11 is formed at t=α+3.

[0090] Similarly, a matching relationship can be formed between the target feature vector T3 and the weight vectors W16 to W11, a matching relationship can be formed between the target feature vector T5 and the weight vectors W18, W17, W15, W14, W12, and W11, and a matching relationship can be formed between each of the target feature vectors T6 and T7 and the weight vectors W19 to W11. The matching relationship between each of the remaining target feature vectors and at least one weight vector can be determined in a similar manner.

[0091] Figure 5The sliding window processing according to time t is shown. However, the sliding window processing is provided only for the sake of ease of description of the process of determining the matching relationship, and the convolution operation is not limited to the above sliding window processing. For example, if the matching relationship between the target feature vector and the weight vector is determined by Figure 5 The sliding window processing is predetermined, and the predetermined matching relationship can be used for the convolution operation as described above.

[0092] Refer again Figure 4 In operation 440, the processing device performs the target feature vector T i The intermediate feature vector is generated by multiplying the weight vector T with the target feature vector T. Here, the multiplication operation may correspond to element-by-element multiplication. For example, the xth element of the intermediate feature vector may be based on the target feature vector T i The product of the xth element of and the xth element of the weight vector is determined. In this example, the unit for performing the multiplication operation can be determined based on the SIMD operation unit (e.g., 4, 8, 16, 32, 64, 128, ...). Therefore, efficient SIMD processing is possible.

[0093] In the following, the intermediate eigenvector (or intermediate element) can be represented by M1 n1n2 Here, l may represent the type of weight element used to generate the intermediate feature vector, n1 may represent the index of the buffer space storing the intermediate feature vector, and n2 may represent the index indicating the sequential position of the intermediate feature vector accumulated in the buffer space. For example, M1 11 It can be an intermediate feature vector generated based on the first type weight element and first stored in the first buffer space. 66 The intermediate feature vector may be generated based on the first type weight element and stored sixthly in the sixth buffer space.

[0094] In operation 450, the processing device accumulates the intermediate feature vectors in the hidden buffer. The hidden buffer may correspond to a memory space for storing the hidden feature vectors. The processing device may allocate memory space to the hidden buffer and generate the hidden feature vector by accumulating the intermediate feature vectors in the hidden buffer. In operation 460, the processing device determines whether there is a completed hidden feature vector in the hidden buffer. If all the intermediate feature vectors required to complete the hidden feature vector (hereinafter referred to as required elements) are accumulated in the buffer space, the hidden feature vector can be completed. In one example, the required elements can be determined based on a plurality of target feature vectors of the target feature map and a plurality of weight vectors respectively matched with the plurality of target feature vectors. In the following, the hidden feature vector can be determined by H p Here, p can represent the index of the hidden feature vector.

[0095] The required elements for completing the hidden feature vector can be determined based on the target feature vector and weight vector for each 2D convolution operation for the depthwise convolution operation. For example, to generate the hidden feature vector H1 based on the 2D convolution operation, the intermediate feature vector M1 generated based on the multiplication operation between the target feature vector T1 and the weight vector W15 is 11 , the intermediate feature vector M1 generated based on the multiplication operation between the target feature vector T2 and the weight vector W16 12 , the intermediate feature vector M1 generated based on the multiplication operation between the target feature vector T5 and the weight vector W18 13 , and the intermediate feature vector M1 generated based on the multiplication operation between the target feature vector T6 and the weight vector W19 14 Can be needed.

[0096] For example, the processing device can process the intermediate feature vector M1 11 、M1 12 、M1 13 and M1 14 Accumulate to generate hidden feature vector H1. If the intermediate feature vector M1 11 、M1 12 、M1 13 and M1 14 All are accumulated in the hidden buffer, then the hidden feature vector H1 can be completed. For example, if the intermediate feature vector M1 11 is generated, the processing device can convert the intermediate feature vector M1 11 is stored in the first space of the hidden buffer. Here, space can represent memory space. Afterwards, if the intermediate feature vector M1 12 is generated, the processing device can load the intermediate feature vector M1 from the first space 11 , by the intermediate eigenvector M1 11 and the intermediate eigenvector M1 12 Accumulation is performed to generate cumulative data, and the cumulative data is stored in the first space. Afterwards, if the intermediate feature vector M1 13 is generated, the processing device can load the accumulated data from the first space, and perform the calculation on the accumulated data and the intermediate feature vector M1 13 Accumulate to generate new accumulated data, and store the new accumulated data in the first space. 14 is accumulated in the first space, the hidden feature vector H1 can be completed and stored in the first space.

[0097] If the completed hidden feature vector does not exist in the hidden buffer, operation 420 is performed again after increasing i (e.g., by 1). If the completed hidden feature vector exists in the hidden buffer, operation 470 is performed. In operation 470, the processing device generates an output feature vector O by performing a point-by-point convolution operation. j The point-wise convolution operation may perform a multiplication and accumulation (MAC) operation between the hidden feature vector and the weight vector of the second type weight elements.

[0098] In operation 480, the processing device determines whether j is equal to W×H. The output feature map may include W×H output feature vectors. Therefore, j equal to W×H indicates that the W×Hth output feature vector O W×H is generated in operation 470, and when the output feature vector O W×H When generated, the output feature map may be completed. In this example, the completed output feature map may be returned and DSC may be terminated. If j is not equal to W×H (i.e., if j is less than W×H), operation 420 is performed again after increasing i and j (e.g., by 1).

[0099] Figure 7 and Figure 8 As described above, the multiplication operation can be performed based on each target feature vector T i A matching relationship with the weight vectors W11 to W19 is performed, and the intermediate feature vectors may be accumulated in the hidden buffer 730 or 830 according to a multiplication operation. Figure 7 shows the operation process associated with the target feature vector T1, Figure 8 The operation process associated with the target feature vector T2 is shown. Figure 8 and Figure 7 The same process as in is repeated for the remaining target feature vectors T3, T4, ..., so that the hidden feature vectors can be generated in the hidden buffer 730 or 830. Figure 7 and Figure 8 In the example, the target feature maps 710 and 810 and the weight elements 720 and 820 may each be in an interleaved format. In addition, the target feature vectors T1, T2, ..., the weight vectors W11, W12, ..., and the intermediate feature vector M1 11 、M1 12 ...can each correspond to a channel direction.

[0100] Reference Figure 7, a multiplication operation may be performed between the target feature vector T1 of the target feature map 710 and the weight vectors W15, W14, W12, and W11 of the weight elements 720. Here, the weight elements 720 may be of the first type, and the multiplication operation may correspond to element-by-element multiplication. In this example, the unit for performing the multiplication operation may be determined based on the SIMD operation unit. Figure 7 An example of 4-SIMD is shown. The intermediate eigenvector M1 11 、M1 21 、M1 41 and M1 51 The intermediate feature vector M1 is generated according to the multiplication operation and stored in the hidden buffer 730. 11 、M1 21 、M1 41 and M1 51 can be stored in registers 1 to 4 during the multiplication process, and if the multiplication is completed, the intermediate feature vector M1 11 、M1 21 、M1 41 and M1 51 It can be stored in spaces S1, S2, S5, and S6 of the hidden buffer 730.

[0101] Reference Figure 8 , a multiplication operation between the target feature vector T2 of the target feature map 810 and the weight vectors W16, W15, W14, W13, W12, and W11 of the weight elements 820 may be performed. Here, the multiplication operation may correspond to element-by-element multiplication. The intermediate feature vector M1 12 、M1 22 、M1 31 、M1 42 、M1 52 and M1 61 M1 can be generated according to the multiplication operation and stored in the hidden buffer 830. 12 、M1 22 、M1 31 、M1 42 、M1 52 and M1 61 can be stored in registers 1 to 6 during the multiplication process, and if the multiplication is completed, the intermediate feature vector M1 12 、M1 22 、M1 31 、M1 42 、M1 52 and M1 61 It can be stored in spaces S1, S2, S3, S5, S6, and S7 of the hidden buffer 830.

[0102] Since the intermediate eigenvector M1 11 pass Figure 7 The processing is stored in space S1, so the intermediate feature vector M1 11 and the intermediate eigenvector M1 12 Available through Figure 8 The processing is accumulated in space S1. For example, the processing device can store the intermediate feature vector M1 in space S1 11 Load into register, for the intermediate feature vector M1 11 and the intermediate eigenvector M1 12 The accumulation is performed and the accumulated result is stored in the space S1. If the accumulation process as described above is performed for the intermediate feature vector M1 11 、M1 12 、M1 13 and M1 14 is repeated, then the hidden feature vector H1 can be generated in the space S1. Therefore, it can be considered that multiple hidden feature vectors can be generated based on the multiplication operation between the target feature vector T1 and each of the multiple weight vectors W15, W14, W12 and W11, or multiple hidden feature vectors can be generated based on the multiplication operation between the target feature vector T2 and each of the multiple weight vectors W16, W15, W14, W13, W12 and W11.

[0103] Figure 9 An example of generating a latent feature vector based on the accumulation of intermediate feature vectors is shown. Figure 9 , the intermediate eigenvector M1 11 The intermediate feature vector M1 is generated according to the multiplication operation between the target feature vector T1 and the weight vector W15. 12 The intermediate feature vector M1 is generated according to the multiplication operation between the target feature vector T2 and the weight vector W16. 13 The intermediate feature vector M1 is generated by the multiplication operation between the target feature vector T5 and the weight vector W18. 14 The intermediate feature vector M1 is generated by the multiplication operation between the target feature vector T6 and the weight vector W19. 11 to M1 14 is accumulated in the space S1 of the hidden buffer 910. If the intermediate feature vector M1 11 to M1 14 All are accumulated in space S1, and then hidden feature vector H1 can be generated in space S1. When the above process is repeated, hidden feature vectors H2, H3, ... can also be generated in other spaces S2, S3, ... of the hidden buffer 910.

[0104] Figure 10An example of generating an output feature vector by point-by-point convolution operation is shown. As mentioned above, if the hidden feature vector H p is completed, then for the hidden feature vector H p The point-wise convolution operation can be performed to output the feature vector O j can be generated. The weight elements 1010 and the output feature maps 1020 can each be in an interleaved format. In addition, the hidden feature vector H1, the weight vectors W21, W22, ..., and the output feature vectors O1, O2, ... can each correspond to a channel direction. Although Figure 10 Not shown in , but each of the other hidden feature vectors H2, H3, ... may also correspond to a channel direction.

[0105] Reference Figure 10 , intermediate elements M21 to M2 N Weight vectors W21 to W2 based on the hidden feature vector H1 and the weight element 1010 N A MAC operation is generated between them. Here, the weight element 1010 may be of the second type, and the multiplication operation of the MAC operation may correspond to element-by-element multiplication. For example, the MAC operation between the hidden feature vector H1 and the weight vector W21 may include an element-by-element multiplication operation between each hidden element of the hidden feature vector H1 and each weight element of the weight vector W21, and an accumulation operation of the elements corresponding to the results of the element-by-element multiplication. In this example, the MAC operation may be performed based on a SIMD operation unit.

[0106] During the MAC operation process, the intermediate elements M21 to M2 N Can be stored in registers 1 to N. When the MAC operation is completed, concatenation (also known as series connection, cascade) can be performed on the intermediate elements M21 to M2 N is executed, so that the output feature vector O1 can be generated. If W×H output feature vectors O j By being generated through the processing as described above, the output feature map 1020 can be completed.

[0107] Figure 11An example of reusing a buffer is shown. As described above, a depth-wise convolution operation can be performed for each target feature vector, and if the hidden feature vector is completed by the depth-wise convolution operation, a point-wise convolution operation can be performed for each completed hidden feature vector. For example, if the first hidden feature vector is generated by a depth-wise convolution operation for each target feature vector, the first output feature vector can be generated by a point-wise convolution operation for the first hidden feature vector. If the first output feature vector is generated, the first hidden feature vector is not further used for DSC. Therefore, the memory space for the first hidden feature vector can be reused to store another hidden feature vector. For example, the first space used for the first hidden feature vector can be reused to accumulate intermediate feature vectors to be used to generate the second hidden feature vector.

[0108] Reference Figure 11 , a multiplication operation between the target feature vector T9 of the target feature map 1110 and the weight vectors W18, W17, W15, W14, W12 and W11 of the weight element 1120 can be performed. The intermediate feature vector M1 55 、M1 67 、M1 93 、M1 104 、M1 131 and M1 141 Can be generated according to the multiplication operation and stored in the hidden buffer 1130. In this example, the first row of the hidden buffer 1130 can be empty. Since the point-by-point convolution operation on the hidden feature vectors in the spaces S1, S2, S3, and S4 has been completed, and the output feature vectors corresponding to the hidden feature vectors in the spaces S1, S2, S3, and S4 have been generated, the hidden feature vectors in the spaces S1, S2, S3, and S4 may not be used further. Therefore, such spaces S1, S2, S3, and S4 can be initialized to empty, or can be reused without initialization to store other data (e.g., other hidden feature vectors).

[0109] exist Figure 11 In the example, the intermediate feature vector M1 55 、M1 67 、M1 93 and M1 104 Can be stored with Space S5, S6, S9 and S 10 The existing intermediate feature vectors in are accumulated and then stored in spaces S5, S6, S9 and S 10 and the intermediate eigenvector M1 131 and M1 141can be stored in empty spaces S1 and S2. In the above example, spaces S1 and S2 may be spaces previously used to generate hidden feature vectors H1 and H2. After the output feature vectors O1 and O2 corresponding to the hidden feature vectors H1 and H2 are generated, spaces S1 and S2 may be reused to store the intermediate feature vector M1. 131 and M1 141 .

[0110] For example, in Figure 11 In the example of , it can be assumed that two rows of zero padding are applied to the horizontal and vertical directions of the target feature map 1110, and the width of the target feature map 1110 is the same as the width of the hidden buffer 1130. In this example, the height of the hidden buffer 1130 may be "3", which may be smaller than the height of the target feature map 1110. If there is no reuse operation as described above, a memory space of the same size as the target feature map 1110 may be required to store the hidden feature map. Therefore, the memory space used to store the hidden feature map can be saved by the reuse operation.

[0111] Figure 12 An example of processing a convolution operation is shown. Figure 12 The operations in the embodiment may be performed in the order and manner as shown, however, the order of some operations may be changed or some operations may be omitted without departing from the spirit and scope of the illustrative examples described. Figure 12 Many of the operations shown in FIG. 1 may be performed in parallel or simultaneously. Figure 12 One or more blocks and combinations of blocks may be implemented by a computer based on dedicated hardware (such as a processor that performs specific functions) or a combination of dedicated hardware and computer instructions. Figure 12 In addition to the description, Figures 1 to 11 The description can also be applied to Figure 12 , and is incorporated herein by reference. Therefore, the above description may not be repeated here. In one embodiment, the method of image recognition may include: obtaining image data for image recognition as input data of a neural network; based on the image data, performing corresponding operations on each layer of the neural network to obtain a result of image recognition; and outputting the result of image recognition. Here, the corresponding operations performed on each layer of the neural network may be the same as or different from each other according to the function of the layer, and the present application does not limit this. In one embodiment, for each layer of at least one of the layers included in the neural network, the step of performing corresponding operations may include: Figure 12 Operations S1210 to S1250 shown in FIG.

[0112] Reference Figure 12In operation 1210, the processing device extracts a first target feature vector from the target feature map. In operation 1220, the processing device extracts a first weight vector that matches the first target feature vector from the first type weight element based on the matching relationship for the depth-wise convolution operation. In operation 1230, the processing device generates a first intermediate feature vector by performing a multiplication operation between the first target feature vector and the first weight vector. In operation 1240, the processing device generates a first hidden feature vector by accumulating the first intermediate feature vector generated based on the first target feature vector and the second intermediate feature vector generated based on the second target feature vector. In operation 1250, the processing device generates a first output feature vector of the output feature map based on the point-by-point convolution operation between the first hidden feature vector and the second type weight element.

[0113] Figure 13 An example of a processing device for processing a convolution operation is shown. Figure 13 , the processing device 1300 includes a processor 1310 and a memory 1320. The memory 1320 is connected to the processor 1310 and may store instructions that can be executed by the processor 1310, data to be calculated by the processor 1310, or data processed by the processor 1310. The memory 1320 may include a non-transitory computer-readable medium (e.g., a high-speed random access memory) and / or a non-volatile computer-readable medium (e.g., at least one disk storage device, a flash memory device, or another non-volatile solid-state memory device).

[0114] The processor 1310 may be configured to execute a reference Figures 1 to 12 For example, the processor 1310 may extract a first target feature vector from a target feature map, extract a first weight vector matching the first target feature vector from the first type of weight element based on a matching relationship between the target feature vector of the target feature map and the weight vector of the first type of weight element, generate a first intermediate feature vector by performing a multiplication operation between the first target feature vector and the first weight vector, generate a first hidden feature vector by accumulating the first intermediate feature vector generated based on the first target feature vector and the second intermediate feature vector generated based on the second target feature vector, and generate a first output feature vector of the output feature map based on a point-by-point convolution operation between the first hidden feature vector and the second type of weight element.

[0115] Figure 14 An example of an electronic device is shown. Figure 14 , the electronic device 1400 may structurally and / or functionally include Figure 1 The processing device 100 and Figure 13 Processing device 1300.

[0116] The electronic device 1400 may include a processor 1410, a memory 1420, a camera 1430, a storage device 1440, an input device 1450, an output device 1460, and a network interface 1470. The processor 1410, the memory 1420, the camera 1430, the storage device 1440, the input device 1450, the output device 1460, and the network interface 1470 may communicate with each other via a communication bus 1480. For example, the electronic device 1400 may be implemented as at least a portion of a mobile device (such as a mobile phone, a smart phone, a PDA, a netbook, a tablet computer, or a laptop computer), a wearable device (such as a smart watch, a smart band, or smart glasses), a computing device (such as a desktop computer or a server), a home appliance (such as a television, a smart TV, or a refrigerator), a security device (such as a door lock), and a vehicle (such as a smart car).

[0117] The processor 1410 executes instructions or functions to be executed in the electronic device 1400. For example, the processor 1410 may process instructions stored in the memory 1420 or the storage device 1440. The processor 1410 may execute instructions or functions to be executed in the electronic device 1400. Figures 1 to 13 Describes one or more operations.

[0118] The memory 1420 stores data used for image recognition including biometric detection (such as facial detection). The memory 1420 may include a computer-readable storage medium or a computer-readable storage device. The memory 1420 may store instructions to be executed by the processor 1410 and may store relevant information when software and / or applications are executed by the electronic device 1400.

[0119] Camera 1430 can capture photos and / or videos. For example, camera 1430 can capture a facial image including a user's face. Camera 1430 can provide a 3D image including depth information related to an object. In one example, camera 1430 can be included in input device 1450 as an image / video information providing device or be considered as an input device.

[0120] Storage device 1440 includes a computer-readable storage medium or a computer-readable storage device. Storage device 1440 can store a larger amount of information and store it for a longer period of time than memory 1420. For example, storage device 1440 may include a magnetic hard disk, an optical disk, a flash memory, a floppy disk, or other non-volatile memory known in the art.

[0121] The input device 1450 can receive input from the user in a conventional input manner such as a keyboard and a mouse, as well as in a new input manner such as touch input, voice input, and image input. For example, the input device 1450 may include a keyboard, a mouse, a touch screen, a microphone, or any other device that detects input from the user and transmits the detected input to the electronic device 1400.

[0122] Output device 1460 can provide output from electronic device 1400 to a user via visual, auditory, or tactile channels. Output device 1460 may include, for example, a display, a touch screen, a speaker, a vibration generator, or any other device that provides output to a user. In one example, output device 1460 may output the results of image recognition. Network interface 1470 can communicate with external devices via a wired or wireless network.

[0123] Processing equipment 100, processing equipment 1300, electronic device 1400 and other devices, units, modules, devices and other components described herein are implemented by hardware components. Examples of hardware components that can be used to perform the operations described in this application include, where appropriate, controllers, sensors, generators, drivers, memories, comparators, arithmetic logic units, adders, subtractors, multipliers, dividers, integrators, and any other electronic components configured to perform the operations described in this application. In other examples, one or more hardware components in the hardware components that perform the operations described in this application are implemented by computing hardware (e.g., by one or more processors or computers). A processor or computer can be implemented by one or more processing elements (such as logic gate arrays, controllers and arithmetic logic units, digital signal processors, microcomputers, programmable logic controllers, field programmable gate arrays, programmable logic arrays, microprocessors, or any other device or combination of devices configured to respond and execute instructions in a limited manner to achieve desired results). In one example, a processor or computer includes or is connected to one or more memories storing instructions or software executed by a processor or computer. The hardware components implemented by a processor or a computer can execute instructions or software (such as an operating system (OS) and one or more software applications running on the OS) for performing the operations described in this application. The hardware components can also access, manipulate, process, create and store data in response to the execution of instructions or software. For simplicity, the singular term "processor" or "computer" can be used in the description of the examples described in this application, but in other examples, multiple processors or computers can be used, or the processor or computer can include multiple processing elements, or multiple types of processing elements, or both. For example, a single hardware component, or two or more hardware components can be implemented by a single processor, or two or more processors, or a processor and a controller. One or more hardware components can be implemented by one or more processors, or a processor and a controller, and one or more other hardware components can be implemented by one or more other processors, or another processor and another controller. One or more processors, or a processor and a controller can implement a single hardware component, or two or more hardware components.The hardware components may have any one or more of different processing configurations, examples of which include: a single processor, independent processors, parallel processors, single instruction single data (SISD) multiprocessing, single instruction multiple data (SIMD) multiprocessing, multiple instruction single data (MISD) multiprocessing, and multiple instruction multiple data (MIMD) multiprocessing, a controller and arithmetic logic unit (ALU), a DSP, a microcomputer, an FPGA, a programmable logic unit (PLU), a central processing unit (CPU), a graphics processing unit (GPU), a neural processing unit (NPU), or any other device capable of responding to and executing instructions in a defined manner.

[0124] The method for performing the operations described in this application is performed by computing hardware (e.g., by one or more processors or computers), which is implemented to execute instructions or software as described above to perform the operations performed by the methods described in this application. For example, a single operation, or two or more operations may be performed by a single processor, or two or more processors, or a processor and a controller. One or more operations may be performed by one or more processors, or a processor and a controller, and one or more other operations may be performed by one or more other processors, or another processor and another controller. One or more processors, or a processor and a controller may perform a single operation, or two or more operations.

[0125] The instructions or software for controlling a processor or computer to implement the hardware components and perform the methods described above are written as a computer program, code segments, instructions, or any combination thereof to individually or collectively instruct or configure the processor or computer to operate as a machine or special-purpose computer to perform the operations performed by the hardware components and methods described above. In one example, the instructions or software include at least one of an applet, a dynamic link library (DLL), middleware, firmware, a device driver, and an application program storing a method for processing convolution operations on layers in a neural network. In another example, the instructions or software include machine code (such as machine code generated by a compiler) that is directly executed by the processor or computer. In another example, the instructions or software include high-level code that is executed by the processor or computer using an interpreter. A programmer of ordinary skill in the art can easily write instructions or software based on the block diagrams and flow charts shown in the accompanying drawings and the corresponding description in the specification, which disclose algorithms for performing the operations performed by the hardware components and methods described above.

[0126] The instructions or software for controlling a processor or computer to implement the hardware components and perform the methods described above, as well as any associated data, data files, and data structures, are recorded, stored, or fixed in or on one or more non-transitory computer-readable storage media. Examples of non-transitory computer-readable storage media include read-only memory (ROM), random-access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random-access memory (RAM), dynamic random-access memory (DRAM), static random-access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disc storage, hard disk drive (HDD), solid state drive (SSD), card memory (such as, multimedia card or micro card (for example, Secure Digital (SD) or Extreme Digital (XD))), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid state disk and any other device, any other device is configured to store instructions or software and any associated data, data files and data structures in a non-transitory manner and provide the instructions or software and any associated data, data files and data structures to a processor or computer so that the processor or computer can execute the instructions. In one example, the instructions or software and any associated data, data files and data structures are distributed on a networked computer system so that the instructions and software and any associated data, data files and data structures are stored, accessed and executed in a distributed manner by one or more processors or computers.

[0127] Although the present disclosure includes specific examples, it will be clear after understanding the disclosure of the present application that various changes in form and detail can be made in these examples without departing from the spirit and scope of the claims and their equivalents. The examples described herein are to be considered illustrative only and not for purposes of limitation. The description of the features or aspects in each example is considered to be applicable to similar features or aspects in other examples. Suitable results can be achieved if the described techniques are performed in a different order, and / or if the components in the described systems, architectures, devices, or circuits are combined in different ways, and / or replaced or supplemented by other components or their equivalents. Therefore, the scope of the disclosure is not limited by the specific embodiments, but by the claims and their equivalents, and all changes within the scope of the claims and their equivalents will be interpreted as included in the disclosure.

Claims

1. A method for image recognition, comprising: Obtaining image data for image recognition as input data of a neural network; Based on the image data, corresponding operations are performed on each layer of the neural network to obtain the image recognition results; and Output the result of image recognition, The step of performing corresponding operations on each layer of at least one of the layers included in the neural network includes: extracting a first target feature vector from the target feature map; Extracting a first weight vector matching the first target feature vector from the first type of weight element based on a matching relationship between the target feature vector of the target feature map and the weight vector of the first type of weight element for a depthwise convolution operation; generating a first intermediate feature vector by performing a multiplication operation between the first target feature vector and the first weight vector; generating a first hidden feature vector by accumulating a first intermediate feature vector generated based on the first target feature vector and a second intermediate feature vector generated based on the second target feature vector; and A first output feature vector of the output feature map is generated based on a point-by-point convolution operation between the first hidden feature vector and the second type weight element.

2. The method according to claim 1, wherein The first hidden feature vector includes the first intermediate feature vector and the second intermediate feature vector, and is completed in response to all required elements being accumulated.

3. The method according to claim 2, wherein: The required elements are determined based on a plurality of target feature vectors of the target feature map and a plurality of weight vectors respectively matched with the plurality of target feature vectors.

4. The method according to any one of claims 1, 2 and 3, wherein: The step of generating the first hidden feature vector includes generating the first hidden feature vector based on accumulating the first intermediate feature vector and the second intermediate feature vector in the first space of the hidden buffer.

5. The method according to claim 4, wherein In response to the first output feature vector being generated, the first space of the hidden buffer is reused to accumulate intermediate feature vectors for generating a second hidden feature vector.

6. The method according to any one of claims 1, 2 and 3, wherein: A plurality of weight vectors including a first weight vector are matched with the first target feature vector based on the matching relationship, and A plurality of hidden feature vectors are generated based on a multiplication operation between the first target feature vector and each of the plurality of weight vectors including the first weight vector.

7. The method according to claim 1, wherein The target feature map, the first type weight elements, the second type weight elements, and the output feature map are each in an interleaved format.

8. The method according to claim 1, wherein The first target feature vector, the first weight vector, the first intermediate feature vector, the second intermediate feature vector, the first hidden feature vector, and the first output feature vector each correspond to a channel direction.

9. The method according to any one of claims 1, 2 and 3, further comprising: extracting a second target feature vector from the target feature map; Extracting a second weight vector that matches the second target feature vector from the first type weight element based on the matching relationship; and A second intermediate feature vector is generated by performing a multiplication operation between the second target feature vector and the second weight vector.

10. The method according to any one of claims 1, 2 and 3, wherein: The step of generating the first output feature vector includes generating the first output feature vector by performing a point-by-point convolution operation between the first hidden feature vector and each weight vector of the second type weight element.

11. The method according to claim 1, wherein The depthwise convolution operation and the pointwise convolution operation constitute at least a part of the depthwise separable convolution operation.

12. The method according to claim 1, wherein The first type of weight elements are used to extract spatial features, and the second type of weight elements are used to extract combined features.

13. The method according to claim 1, wherein The target feature map corresponds to the input feature map or the hidden feature map.

14. The method according to claim 1, wherein Depth-wise convolution operations and point-wise convolution operations are each processed for each SIMD operation.

15. A non-transitory computer-readable storage medium storing instructions, which, when executed by a processor, cause the processor to perform the method of claim 1.

16. A device for image recognition, comprising: a memory configured to store executable instructions; and A processor configured to execute the instructions to: Obtaining image data for image recognition as input data of a neural network; Based on the image data, corresponding operations are performed on each layer of the neural network to obtain the image recognition results; and Output the result of image recognition, The process of performing corresponding operations on each layer of at least one of the layers included in the neural network includes: extracting a first target feature vector from the target feature map; Extracting a first weight vector matching the first target feature vector from the first type of weight element based on a matching relationship between the target feature vector of the target feature map and the weight vector of the first type of weight element for a depthwise convolution operation; generating a first intermediate feature vector by performing a multiplication operation between the first target feature vector and the first weight vector; generating a first hidden feature vector by accumulating a first intermediate feature vector generated based on the first target feature vector and a second intermediate feature vector generated based on the second target feature vector; and A first output feature vector of the output feature map is generated based on a point-by-point convolution operation between the first hidden feature vector and the second type weight element.

17. The apparatus according to claim 16, wherein The first hidden feature vector includes the first intermediate feature vector and the second intermediate feature vector, and is completed in response to all required elements being accumulated.

18. The apparatus according to claim 16 or 17, wherein The processor is configured to generate a first hidden feature vector based on accumulating the first intermediate feature vector and the second intermediate feature vector in the first space of the hidden buffer, and In response to the first output feature vector being generated, the first space of the hidden buffer is reused to accumulate intermediate feature vectors for generating a second hidden feature vector.

19. The apparatus according to claim 16 or 17, wherein A plurality of weight vectors including a first weight vector are matched with the first target feature vector based on the matching relationship, and A plurality of hidden feature vectors are generated based on a multiplication operation between the first target feature vector and each of the plurality of weight vectors.

20. An electronic device comprising: an input device configured to receive image data for image recognition; an output device configured to output a result of image recognition; a memory configured to store executable instructions; and A processor configured to execute the instructions to: Obtaining image data for image recognition as input data of a neural network; Based on the image data, corresponding operations are performed on each layer of the neural network to obtain the image recognition results; and Output the result of image recognition, The process of performing corresponding operations on each layer of at least one of the layers included in the neural network includes: extracting a first target feature vector from the target feature map; Extracting a first weight vector matching the first target feature vector from the first type of weight element based on a matching relationship between the target feature vector of the target feature map and the weight vector of the first type of weight element for a depthwise convolution operation; generating a first intermediate feature vector by performing a multiplication operation between the first target feature vector and the first weight vector; generating a first hidden feature vector by accumulating a first intermediate feature vector generated based on the first target feature vector and a second intermediate feature vector generated based on the second target feature vector; and A first output feature vector of the output feature map is generated based on a point-by-point convolution operation between the first hidden feature vector and the second type weight element.

21. The electronic device according to claim 20, wherein: The first hidden feature vector includes the first intermediate feature vector and the second intermediate feature vector, and is completed in response to all required elements being accumulated.

Images