Convolution operation method and device, electronic equipment and readable storage medium

Abstract

Embodiments of the invention provide a convolution operation method and apparatus, an electronic device and a readable storage medium. The method comprises the steps of obtaining input data and an operator stream file; the operator stream file comprises operators; determining the operator type of the operator in the operator stream file; when the operator type of the operator is a specified operator type, determining a convolution kernel type of the operator; and according to the first specified convolution kernel type and the convolution kernel type of the operator, correspondingly adopting the multiplication and accumulation unit to carry out convolution operation on the input data to obtain a convolution operation result. The neural network processor provided by the embodiment of the invention comprises the acceleration hardware aiming at the convolution operation of the first specified convolution kernel type, so that the multiplication and accumulation unit is correspondingly adopted to perform the convolution operation on the input data based on the first specified convolution kernel type and the convolution kernel type of the operator, the convolution operation efficiency is improved, and the convolution operation speed is improved. And the waste of hardware resources in the neural network processor is avoided.

Description

Technical Field

[0001] Embodiments of the present invention relate to the field of computer technology, and in particular, to a convolution operation method, a convolution operation device, an electronic device, and a computer-readable storage medium. Background Art

[0002] With the rise of deep learning neural networks, the advent of the era of artificial intelligence and big data, CPU (Central Processing Unit) and GPU (Graphics Processing Unit) are gradually unable to meet the needs of deep learning. Faced with the growing demand and the vast expected market, it is very necessary to design an efficient and intelligent processor specifically for neural network deep learning. Therefore, NPU (Neural-network Processing Unit) came into being.

[0003] However, the current design of neural network processors is not perfect. When using neural network processors for convolution operations, some MAC (Multiply-Accumulate) units in the neural network processors are idle, resulting in low efficiency of convolution operations of the neural network processors and a waste of hardware resources of the neural network processors. Summary of the invention

[0004] An embodiment of the present invention provides a convolution operation method, device, electronic device and computer-readable storage medium to solve the problem that the convolution operation efficiency of a neural network processor is low and the hardware resources of the neural network processor are wasted.

[0005] An embodiment of the present invention discloses a convolution operation method, which relates to a neural network processor, wherein the neural network processor includes a multiplication-accumulation unit, and the neural network processor includes an accelerated multiplication-accumulation unit for a convolution operation of a first specified convolution kernel type. The method includes:

[0006] Obtain input data and an operator stream file; the operator stream file includes an operator;

[0007] Determine the operator type of the operator in the operator stream file;

[0008] When the operator type of the operator is a specified operator type, determining a convolution kernel type of the operator;

[0009] According to the first specified convolution kernel type and the convolution kernel type of the operator, the multiplication and accumulation unit is correspondingly used to perform a convolution operation on the input data to obtain a convolution operation result.

[0010] Optionally, determining the operator type of the operator in the operator stream file includes:

[0011] When it is determined that the operator in the operator stream file is an operator that can be executed on a specified hardware platform, determining the operator type of the operator as a hardware operator;

[0012] When it is determined that the operator in the operator stream file is an operator that cannot be executed on the specified hardware platform, the operator type of the operator is determined to be a software operator.

[0013] Optionally, when the operator type of the operator is a specified operator type, determining the convolution kernel type of the operator includes:

[0014] When the operator type is a hardware operator, the operator type of the operator is determined to be a specified operator type, and a convolution kernel type of the operator is determined.

[0015] Optionally, according to the first specified convolution kernel type and the convolution kernel type of the operator, the multiplication and accumulation unit is used to perform a convolution operation on the input data to obtain a convolution operation result, including:

[0016] When the convolution kernel type of the operator is not the first specified convolution kernel type, determining whether the convolution kernel type of the operator is a second specified convolution kernel type;

[0017] When the convolution kernel type of the operator is a second specified convolution kernel type, merging every preset number of input channels in the input data to obtain merged input data;

[0018] Determining the number of combined input channels of the combined input data;

[0019] According to the preset stride, the number of merged input channels and the first specified convolution kernel type, a convolution operation is performed on the merged input data to obtain a convolution operation result.

[0020] Optionally, the first specified convolution kernel type includes a 3*3 convolution kernel, the second specified convolution kernel type includes a 1*1 convolution kernel, and the preset step size includes 3.

[0021] Optionally, according to the first specified convolution kernel type and the convolution kernel type of the operator, the multiplication and accumulation unit is used to perform a convolution operation on the input data to obtain a convolution operation result, including:

[0022] When the convolution kernel type of the operator is the first specified convolution kernel type, determining the number of input channels of the input data;

[0023] According to the preset stride, the number of input channels and the first specified convolution kernel type, the multiplication and accumulation unit is correspondingly used to perform a convolution operation on the input data to obtain a convolution operation result.

[0024] Optionally, after determining the operator type of the operator in the operator stream file, the method further includes:

[0025] When the operator type of the operator is not a specified operator type, a preset software algorithm of the operator is called to perform a convolution operation on the input data.

[0026] Optionally, before obtaining the input data and the operator stream file, the method further includes:

[0027] An operator stream file is generated by encoding using a preset artificial intelligence compiler; the operator stream file includes a header, an operator stream, a hardware instruction set, and a quantized network weight set, and the operator stream includes an operator;

[0028] Storing the operator stream file in a designated memory;

[0029] When the neural network processor is started, the operator stream file is loaded into a random access memory, so as to obtain the operator stream file from the random access memory for convolution operation.

[0030] Optionally, after performing a convolution operation on the input data according to the first specified convolution kernel type and the convolution kernel type of the operator and obtaining a convolution operation result by using the multiplication and accumulation unit, the method further includes:

[0031] After the operator in the operator stream file is executed once, the convolution operation result is converted into a specified format; the specified format at least includes a floating point format.

[0032] The embodiment of the present invention further discloses a convolution operation device, which relates to a neural network processor, wherein the neural network processor includes a multiplication-accumulation unit, and the neural network processor includes an accelerated multiplication-accumulation unit for convolution operations of a first specified convolution kernel type, and the device includes:

[0033] A data acquisition module, used to acquire input data and operator stream files; the operator stream files include operators;

[0034] An operator type determination module, used to determine the operator type of the operator in the operator stream file;

[0035] A convolution kernel type determination module, used for determining the convolution kernel type of the operator when the operator type of the operator is a specified operator type;

[0036] A convolution operation module is used to perform a convolution operation on the input data using the multiplication and accumulation unit according to the first specified convolution kernel type and the convolution kernel type of the operator to obtain a convolution operation result.

[0037] Optionally, the operator type determination module is used to:

[0038] When it is determined that the operator in the operator stream file is an operator that can be executed on a specified hardware platform, determining the operator type of the operator as a hardware operator;

[0039] When it is determined that the operator in the operator stream file is an operator that cannot be executed on the specified hardware platform, the operator type of the operator is determined to be a software operator.

[0040] Optionally, the convolution kernel type determination module is used to:

[0041] When the operator type is a hardware operator, the operator type of the operator is determined to be a specified operator type, and a convolution kernel type of the operator is determined.

[0042] Optionally, the convolution operation module is used to:

[0043] When the convolution kernel type of the operator is not the first specified convolution kernel type, determining whether the convolution kernel type of the operator is a second specified convolution kernel type;

[0044] When the convolution kernel type of the operator is a second specified convolution kernel type, merging every preset number of input channels in the input data to obtain merged input data;

[0045] Determining the number of combined input channels of the combined input data;

[0046] According to the preset stride, the number of merged input channels and the first specified convolution kernel type, a convolution operation is performed on the merged input data to obtain a convolution operation result.

[0047] Optionally, the first specified convolution kernel type includes a 3*3 convolution kernel, the second specified convolution kernel type includes a 1*1 convolution kernel, and the preset step size includes 3.

[0048] Optionally, the convolution operation module is used to:

[0049] When the convolution kernel type of the operator is the first specified convolution kernel type, determining the number of input channels of the input data;

[0050] According to the preset stride, the number of input channels and the first specified convolution kernel type, the multiplication and accumulation unit is correspondingly used to perform a convolution operation on the input data to obtain a convolution operation result.

[0051] Optionally, the device further comprises: a software operator convolution operation module, configured to:

[0052] When the operator type of the operator is not a specified operator type, a preset software algorithm of the operator is called to perform a convolution operation on the input data.

[0053] Optionally, the device further includes: an operator stream file generation module, configured to:

[0054] An operator stream file is generated by encoding using a preset artificial intelligence compiler; the operator stream file includes a header, an operator stream, a hardware instruction set, and a quantized network weight set, and the operator stream includes an operator;

[0055] Storing the operator stream file in a designated memory;

[0056] When the neural network processor is started, the operator stream file is loaded into a random access memory, so as to obtain the operator stream file from the random access memory for convolution operation.

[0057] Optionally, the device further includes: a format conversion module, configured to:

[0058] After the operator in the operator stream file is executed once, the convolution operation result is converted into a specified format; the specified format at least includes a floating point format.

[0059] The embodiment of the present invention further discloses an electronic device, comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;

[0060] The memory is used to store computer programs;

[0061] The processor is used to implement the method described in the embodiment of the present invention when executing the program stored in the memory.

[0062] The embodiment of the present invention further discloses a computer program product, which is stored in a storage medium and is executed by at least one processor to implement the method described in the embodiment of the present invention.

[0063] The embodiment of the present invention further discloses a computer-readable storage medium having instructions stored thereon, which, when executed by one or more processors, enables the processors to execute the method described in the embodiment of the present invention.

[0064] The embodiments of the present invention include the following advantages:

[0065] In an embodiment of the present invention, a neural network processor is involved. The neural network processor includes a multiplication-accumulation unit. The neural network processor includes acceleration hardware for convolution operations of a first specified convolution kernel type. When performing a convolution operation, input data and an operator stream file are obtained, and the operator type of the operator in the operator stream file is determined. When the operator type of the operator is a specified operator type, the convolution kernel type of the operator is determined, so that according to the first specified convolution kernel type and the convolution kernel type of the operator, the multiplication-accumulation unit in the neural network processor is used to perform a convolution operation on the input data to obtain a convolution operation result. The neural network processor of the embodiment of the present invention includes acceleration hardware for convolution operations of a first specified convolution kernel type. Therefore, based on the first specified convolution kernel type and the convolution kernel type of the operator, the multiplication-accumulation unit in the neural network processor is used to perform a convolution operation on the input data, which can improve the efficiency of the convolution operation and avoid wasting hardware resources in the neural network processor. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] Figure 1 is a flow chart of the steps of a convolution operation method provided in an embodiment of the present invention;

[0067] Figure 2 is a schematic diagram of input channel merging of input data provided in an embodiment of the present invention;

[0068] Figure 3 is a schematic diagram of a flow chart of a convolution operation in a neural network processor provided in an embodiment of the present invention;

[0069] Figure 4 is a structural block diagram of a convolution operation device provided in an embodiment of the present invention;

[0070] Figure 5 It is a schematic diagram of the hardware structure of an electronic device for implementing various embodiments of the present invention. DETAILED DESCRIPTION

[0071] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0072] In specific implementations, convolution operations play an important role in tasks ranging from image classification to target detection, semantic segmentation, image generation, and style transfer. At present, when designing a neural network processor (NPU), hardware acceleration optimization is usually performed for some convolution operations, usually for 3x3 convolution operations (using 3x3 convolution kernels for convolution operations), and then large convolution kernel operations are implemented by superimposing multiple 3x3 convolution operations. For example, a 5x5 convolution operation can be implemented by 4 3x3 convolution operations, and a 7x7 convolution operation can be implemented by 9 3x3 convolution operations. The 3x3 convolution operation hardware unit has 9 MAC units. If the 3x3 convolution operation hardware unit is directly used to complete the 1x1 convolution operation, then 8 MAC units in the neural network processor will be idle, resulting in a waste of hardware resources (MAC units) in the neural network processor.

[0073] In response to the above problems, an embodiment of the present invention provides a convolution operation method, which can perform convolution operations on input data and operator stream files according to convolution operations of specified convolution kernel types (for example, 3x3 convolution operations) that have been hardware accelerated and optimized in a neural network processor. This can improve the efficiency of convolution operations and avoid energy waste caused by idle MAC units in the neural network processor.

[0074] Reference Figure 1 , shows a flow chart of steps of a convolution operation method provided in an embodiment of the present invention, involving a neural network processor, the neural network processor includes a multiplication-accumulation unit, the neural network processor includes an accelerated multiplication-accumulation unit for convolution operations of a first specified convolution kernel type, and the method may specifically include the following steps:

[0075] Step 101: Obtain input data and operator stream files; the operator stream files include operators.

[0076] In a specific implementation, a neural network processor (NPU) is a hardware processor designed to accelerate neural network calculations. Compared with a central processing unit (CPU) and a graphics processing unit (GPU), a neural network processor performs hardware optimization for convolution operations of a first specified convolution kernel type in a neural network. Therefore, when using a neural network processor to perform convolution operations of a first specified convolution kernel type, the hardware resources in the neural network processor, namely, a multiplication-accumulation unit (MAC unit), can be fully utilized, wherein the multiplication-accumulation unit is a hardware unit for performing multiplication and accumulation operations. In the convolution operation, the multiplication-accumulation unit is used to calculate the product between the convolution kernel and the local area of ​​the input feature map (input data, which can be related data of tasks to be performed such as target detection, semantic segmentation, image generation, and style migration), and accumulate the product. The input data includes the number of input channels, a dimension of the number of input channels, and the number of feature maps of the input data. Each input channel corresponds to a feature map, and the feature map is a two-dimensional matrix obtained by sliding the convolution kernel on the input data and performing a convolution operation. The convolution kernel can be represented as a three-dimensional tensor, such as k*k*c, where k is the size of the convolution kernel and c is the depth of the convolution kernel, which must be the same as the number of input channels of the input data. The input data can be represented as a three-dimensional tensor, such as H*W*C, where H is the height, W is the width, and C is the number of input channels. The convolution kernel slides on each input channel of the input data and performs a dot product operation at each position to extract the local features of the input data, and then adds the results of all channels to obtain an element of the output feature map.

[0077] Exemplarily, the first specified convolution kernel type may be a 3x3 convolution, and the convolution operation of the first specified convolution kernel type is a convolution operation of a 3x3 convolution kernel. An operator stream file refers to a file format used to describe a data processing flow in certain specific computing frameworks or systems. The operator stream file defines a series of operations (operators) and the data flow relationship between operators, thereby forming a complete data processing or computing flow. Specifically, a plurality of multiplication-accumulation units are designed in the neural network processor to execute convolution operations of the first specified convolution kernel type such as the injected 3x3 convolution operation in parallel to improve the efficiency of the convolution operation, and a dedicated instruction set is designed for the convolution operation of the first specified convolution kernel type to reduce instruction overhead, so that when the neural network processor executes the convolution operation of the first specified convolution kernel type, the convolution operation can be accelerated to improve the efficiency of the convolution operation.

[0078] In an embodiment of the present invention, when a convolution operation needs to be performed on input data, the neural network processor can execute the corresponding convolution operation task according to the operator stream file.

[0079] Step 102: Determine the operator type of the operator in the operator stream file.

[0080] Step 103: When the operator type of the operator is a specified operator type, determine the convolution kernel type of the operator.

[0081] Step 104: According to the first specified convolution kernel type and the convolution kernel type of the operator, the multiplication and accumulation unit is used to perform a convolution operation on the input data to obtain a convolution operation result.

[0082] In a specific implementation, an operator has its corresponding convolution kernel type. For example, the convolution kernel type of an operator can be a 1x1 convolution kernel, a 3x3 convolution kernel, a 5x5 convolution kernel, and other convolution kernels. The multiplication and accumulation unit is used to perform multiplication and accumulation operations in the convolution operation. For a convolution kernel, each convolution kernel element is multiplied by the element at the corresponding position of the input data, and then all the products are accumulated. Multiple multiplication and accumulation units in the neural network processor are required to execute in parallel. For example, a 1x1 convolution kernel requires 1 multiplication and accumulation unit to execute, a 3x3 convolution kernel requires 9 multiplication and accumulation units to execute, and a 5x5 convolution kernel requires 25 multiplication and accumulation units to execute.

[0083] In an embodiment of the present invention, there may be multiple types of operators in an operator stream file, such as software operators or hardware operators, etc. The specified operator type refers to the operator type that needs to use a neural network processor to perform a convolution operation. Therefore, when the operator type of the operator in the operator stream file is a specified type, that is, it is necessary to use the multiplication and accumulation unit in the neural network processor to perform a convolution operation, it is necessary to further determine the convolution kernel type of the operator, and then determine the number of multiplication and accumulation units in the neural network processor that need to be used. Specifically, in an embodiment of the present invention, according to the first specified convolution kernel type and the convolution kernel type of the operator, the multiplication and accumulation unit in the neural network processor can be used to perform a convolution operation on the input data to obtain a convolution operation result.

[0084] In the above-mentioned convolution operation method, a neural network processor is involved, the neural network processor includes a multiplication accumulation unit, and the neural network processor includes acceleration hardware for convolution operation of a first specified convolution kernel type. When performing a convolution operation, the input data and the operator stream file are obtained, and the operator type of the operator in the operator stream file is determined. When the operator type of the operator is a specified operator type, the convolution kernel type of the operator is determined, so that according to the first specified convolution kernel type and the convolution kernel type of the operator, the multiplication accumulation unit in the neural network processor is used to perform a convolution operation on the input data to obtain a convolution operation result. The neural network processor of an embodiment of the present invention includes acceleration hardware for convolution operation of a first specified convolution kernel type. Therefore, based on the first specified convolution kernel type and the convolution kernel type of the operator, the multiplication accumulation unit in the neural network processor is used to perform a convolution operation on the input data, which can improve the efficiency of the convolution operation and avoid wasting hardware resources in the neural network processor.

[0085] In one embodiment of the present invention, determining the operator type of the operator in the operator stream file includes:

[0086] When it is determined that the operator in the operator stream file is an operator that can be executed on a specified hardware platform, determining the operator type of the operator as a hardware operator;

[0087] When it is determined that the operator in the operator stream file is an operator that cannot be executed on the specified hardware platform, the operator type of the operator is determined to be a software operator.

[0088] In the specific implementation, a sequence of hardware instructions that are executed continuously is implemented as an OP_HW operator, that is, an operator that can be executed on a specified hardware platform is a hardware operator, and the specified hardware platform may include a neural network processor. All operations that cannot be implemented by hardware must be implemented as software operators.

[0089] In one embodiment of the present invention, after determining the operator type of the operator in the operator stream file, the method further includes:

[0090] When the operator type of the operator is not a specified operator type, a preset software algorithm of the operator is called to perform a convolution operation on the input data.

[0091] In an embodiment of the present invention, the operator flow file is parsed and the operators are executed in sequence. If it is determined that the operator type of the operator is a software operator, that is, the operator type of the operator is not a specified operator type, the preset software algorithm corresponding to the operator is called to perform a convolution operation on the input data, thereby obtaining a convolution operation result.

[0092] In one embodiment of the present invention, when the operator type of the operator is a specified operator type, determining the convolution kernel type of the operator includes:

[0093] When the operator type is a hardware operator, the operator type of the operator is determined to be a specified operator type, and a convolution kernel type of the operator is determined.

[0094] In an embodiment of the present invention, if it is determined that the operator type of the operator is a hardware operator, it can be determined that the operator type of the operator is a specified operator type, and then it is necessary to further determine the convolution kernel type of the operator, and then based on the convolution kernel type of the operator, combined with the first specified convolution kernel type that has been hardware optimized in the neural network processor, a multiplication and accumulation unit is used to perform a convolution operation on the input data to obtain a convolution operation result, so as to improve the efficiency of the convolution operation on the input data.

[0095] In an embodiment of the present invention, according to the first specified convolution kernel type and the convolution kernel type of the operator, the multiplication and accumulation unit is used to perform a convolution operation on the input data to obtain a convolution operation result, including:

[0096] When the convolution kernel type of the operator is not the first specified convolution kernel type, determining whether the convolution kernel type of the operator is a second specified convolution kernel type;

[0097] When the convolution kernel type of the operator is a second specified convolution kernel type, merging every preset number of input channels in the input data to obtain merged input data;

[0098] Determining the number of combined input channels of the combined input data;

[0099] According to the preset stride, the number of merged input channels and the first specified convolution kernel type, a convolution operation is performed on the merged input data to obtain a convolution operation result.

[0100] Exemplarily, the first specified convolution kernel type includes a 3*3 convolution kernel, the second specified convolution kernel type includes a 1*1 convolution kernel, the preset step size includes 3, and each preset number can be 9.

[0101] In an embodiment of the present invention, when the convolution kernel type of the operator is not the first specified convolution kernel type, that is, when the convolution kernel type of the operator is not a 3*3 convolution kernel, the multiplication and accumulation unit of the first specified convolution kernel type that has been hardware optimized in the neural network processor cannot be used to perform accelerated convolution operations. Therefore, at this time, it is further determined whether the convolution kernel type of the operator is the second specified convolution kernel type, that is, whether the convolution kernel type of the operator is a 1*1 convolution kernel. When the convolution kernel type of the operator is a 1*1 convolution kernel, every 9 input channels in the input data are merged to obtain merged input data. Then, the number of merged input channels of the merged input data is determined. According to the preset step (for example, 3), the number of merged input channels and the 3*3 convolution kernel, the merged input data is accelerated by using the multiplication and accumulation unit to quickly obtain the convolution operation result.

[0102] In an embodiment of the present invention, according to the first specified convolution kernel type and the convolution kernel type of the operator, the multiplication and accumulation unit is used to perform a convolution operation on the input data to obtain a convolution operation result, including:

[0103] When the convolution kernel type of the operator is the first specified convolution kernel type, determining the number of input channels of the input data;

[0104] According to the preset stride, the number of input channels and the first specified convolution kernel type, the multiplication and accumulation unit is correspondingly used to perform a convolution operation on the input data to obtain a convolution operation result.

[0105] When the convolution kernel type of the operator is the first specified convolution kernel type, that is, when the convolution kernel type of the operator is a 3*3 convolution kernel, the neural network processor is directly started at this time, and the multiplication and accumulation unit is used to accelerate the convolution operation according to the preset step size, number of input channels and 3*3 convolution kernel to quickly obtain the convolution operation result.

[0106] In a specific embodiment, after obtaining the input data and the operator stream file, the operator stream file is parsed, and the operators in the operator stream file are executed in sequence. If the operator is a software operator, the relevant software logic is called to perform the operation; if the operator is a hardware operator, the convolution kernel type of the operator is determined. If the operator is a 3x3 convolution kernel, the hardware instruction parameters obtained by parsing the operator stream file are directly used to configure the neural network processor, and then the neural network processor is started to accelerate the convolution operation; if the operator is a 1x1 convolution kernel, every 9 input channels are merged into one channel. For details, please refer to Figure 2 , and then update the convolution operation channel data, row and column data according to the following formula:

[0107] (1) Calculate the number of channels after merging the 9 input channels: ch_num = (ch_num + 8) / 9;

[0108] (2) The number of rows after the 9 input channels are merged is expanded to 3 times: hardware_in_row*=3;

[0109] (3) The number of columns after merging the 9 input channels is expanded to 3 times: hardware_in_col*=3;

[0110] (4) The stride after merging the 9 input channels is expanded from 1 to 3;

[0111] Then, the neural network processor is started to accelerate the 3x3 convolution operation with a stride of 3.

[0112] Exemplarily, the convolution operation is a multiplication and addition process, that is, each input channel data performs matrix multiplication and then accumulation. If the multiplication and accumulation unit of the 3x3 convolution operation is directly used to complete the 1x1 convolution operation, there will be 8 multiplication and accumulation units that are idle. In order to make full use of the hardware performance, every 9 input channel data are merged into one input channel. In order to avoid repeated operations, the convolution operation stride needs to be expanded from 1 to 3; for example, 12 (number of rows) * 12 (number of columns) * 18 (number of input channels) input data performs a 1x1 convolution operation. After the input channels are merged, it becomes 36 (number of rows) * 36 (number of columns) * 2 (number of input channels) input data for a 3x3 convolution operation.

[0113] In one embodiment of the present invention, before obtaining the input data and the operator stream file, the method further includes:

[0114] An operator stream file is generated by encoding using a preset artificial intelligence compiler; the operator stream file includes a header, an operator stream, a hardware instruction set, and a quantized network weight set, and the operator stream includes an operator;

[0115] Storing the operator stream file in a designated memory;

[0116] When the neural network processor is started, the operator stream file is loaded into a random access memory, so as to obtain the operator stream file from the random access memory for convolution operation.

[0117] In the embodiment of the present invention, an operator stream file is generated by encoding through an AI compiler (artificial intelligence compiler). The operator stream file generated by the AI ​​compiler is divided into four parts:

[0118] (1) The first part is the header. The length of the header is fixed. The definitions of the header fields are shown in Table 1:

[0119] Table 1: All variable fields in the header

[0120]

[0121] (2) The second part is the operator stream, and the starting position is shown in the header field ops_offset. A sequence of hardware instructions that are executed continuously is implemented as an OP_HW operator (hardware operator). All operations that cannot be implemented by hardware must be implemented as software operators, and the operating parameters are placed in the operator parameter list.

[0122] (3) The third part is a hardware instruction set, the starting position of which is shown in the header field cmd_offset in Table 1 above;

[0123] (4) The fourth part is the quantized network weight set, and the starting position is shown in the header field weight_offset in Table 1.

[0124] The operator stream file generated by the AI ​​compiler encoding will be solidified and stored in a specified memory, such as flash. When the hardware system starts, the operator stream file will be loaded into RAM (Random Access Memory) for application access.

[0125] In addition, in the embodiment of the present invention, the decoding middleware is initialized to inform the AI ​​compiler of the location of the relevant operator stream file, the input data location input and the output data storage location output of the AI ​​compiler.

[0126] In an embodiment of the present invention, after performing a convolution operation on the input data using the multiplication and accumulation unit according to the first specified convolution kernel type and the convolution kernel type of the operator to obtain a convolution operation result, the method may further include:

[0127] After the operator in the operator stream file is executed once, the convolution operation result is converted into a specified format; the specified format at least includes a floating point format.

[0128] In an embodiment of the present invention, after all operators in the operator stream file are executed and the convolution operation result is obtained, the convolution operation result is converted into a specified format, and relevant post-processing operations of the AI ​​algorithm are performed. Specifically, the convolution operation result is usually in a fixed-point format, and the convolution operation result can be converted from a fixed-point format to a floating-point format.

[0129] One of the inventive points created by the embodiment of the present invention is: to provide a method for accelerating 1x1 convolution operations, by rearranging data, merging 9 input channels of input data into one channel, and then performing convolution operations on the input data after the channels are merged according to a 3x3 convolution kernel with a stride of 3, thereby improving the computational efficiency of the convolution operations of the 1x1 convolution kernel and avoiding the waste of hardware resources caused by idle multiplication and accumulation units in the neural network processor.

[0130] In order to enable those skilled in the art to better understand the embodiments of the present invention, a specific example is used below for illustration. Figure 3 , is a flow chart of a convolution operation in a neural network processor provided in an embodiment of the present invention, and the specific steps include:

[0131] Step 301: Load the operator stream file generated by compilation;

[0132] Step 302: Parse the operator stream file and initialize the context environment;

[0133] Step 303: Execute the operators in the operator stream file in sequence;

[0134] Step 304: Determine whether the operator is a hardware operator; if so, execute step 305; if not, execute step 309;

[0135] Step 305: determine whether the convolution kernel is 1x1;

[0136] Step 306, merging the input channels of the input data;

[0137] Step 307: Update the number of channels and the number of rows and columns of the convolution operation;

[0138] Step 308: The multiplication and accumulation unit of the 3x3 convolution operation in the NPU performs accelerated convolution operation;

[0139] Step 309: Execute corresponding software algorithm;

[0140] Step 310: store the calculation result of this layer (convolution operation result);

[0141] Step 311: Determine whether all operators in the operator stream file have been executed; if so, execute step 312; if not, execute step 303;

[0142] Step 312: Export the final result.

[0143] By applying the embodiment of the present invention, data of every 9 input channels of input data are merged into one channel through data rearrangement, and then a 3x3 convolution operation with a stride of 3 is performed, thereby improving the efficiency of the 1x1 convolution operation and avoiding energy waste caused by idle multiplication and accumulation units in the NPU.

[0144] It should be noted that, for the sake of simplicity, the method embodiments are described as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

[0145] Reference Figure 4 , shows a structural block diagram of a convolution operation device provided in an embodiment of the present invention, which relates to a neural network processor, the neural network processor includes a multiplication-accumulation unit, the neural network processor includes an accelerated multiplication-accumulation unit for convolution operations of a first specified convolution kernel type, and the device may specifically include the following modules:

[0146] The data acquisition module 401 is used to acquire input data and operator stream files; the operator stream files include operators;

[0147] An operator type determination module 402, used to determine the operator type of the operator in the operator stream file;

[0148] A convolution kernel type determination module 403, used to determine the convolution kernel type of the operator when the operator type of the operator is a specified operator type;

[0149] The convolution operation module 404 is used to perform a convolution operation on the input data using the multiplication and accumulation unit according to the first specified convolution kernel type and the convolution kernel type of the operator to obtain a convolution operation result.

[0150] In one embodiment of the present invention, the operator type determination module 402 is used to:

[0151] When it is determined that the operator in the operator stream file is an operator that can be executed on a specified hardware platform, determining the operator type of the operator as a hardware operator;

[0152] When it is determined that the operator in the operator stream file is an operator that cannot be executed on the specified hardware platform, the operator type of the operator is determined to be a software operator.

[0153] In one embodiment of the present invention, the convolution kernel type determination module 403 is used to:

[0154] When the operator type is a hardware operator, the operator type of the operator is determined to be a specified operator type, and a convolution kernel type of the operator is determined.

[0155] In one embodiment of the present invention, the convolution operation module 404 is used to:

[0156] When the convolution kernel type of the operator is not the first specified convolution kernel type, determining whether the convolution kernel type of the operator is a second specified convolution kernel type;

[0157] When the convolution kernel type of the operator is a second specified convolution kernel type, merging every preset number of input channels in the input data to obtain merged input data;

[0158] Determining the number of combined input channels of the combined input data;

[0159] According to the preset stride, the number of merged input channels and the first specified convolution kernel type, a convolution operation is performed on the merged input data to obtain a convolution operation result.

[0160] In one embodiment of the present invention, the first specified convolution kernel type includes a 3*3 convolution kernel, the second specified convolution kernel type includes a 1*1 convolution kernel, and the preset stride includes 3.

[0161] In one embodiment of the present invention, the convolution operation module 404 is used to:

[0162] When the convolution kernel type of the operator is the first specified convolution kernel type, determining the number of input channels of the input data;

[0163] According to the preset stride, the number of input channels and the first specified convolution kernel type, the multiplication and accumulation unit is correspondingly used to perform a convolution operation on the input data to obtain a convolution operation result.

[0164] In one embodiment of the present invention, the device further comprises: a software operator convolution operation module, configured to:

[0165] When the operator type of the operator is not a specified operator type, a preset software algorithm of the operator is called to perform a convolution operation on the input data.

[0166] In one embodiment of the present invention, the device further includes: an operator stream file generation module, which is used to:

[0167] An operator stream file is generated by encoding using a preset artificial intelligence compiler; the operator stream file includes a header, an operator stream, a hardware instruction set, and a quantized network weight set, and the operator stream includes an operator;

[0168] Storing the operator stream file in a designated memory;

[0169] When the neural network processor is started, the operator stream file is loaded into a random access memory, so as to obtain the operator stream file from the random access memory for convolution operation.

[0170] In an embodiment of the present invention, the device further includes: a format conversion module, configured to:

[0171] After the operator in the operator stream file is executed once, the convolution operation result is converted into a specified format; the specified format at least includes a floating point format.

[0172] In an embodiment of the present invention, a neural network processor is involved. The neural network processor includes a multiplication-accumulation unit. The neural network processor includes acceleration hardware for convolution operations of a first specified convolution kernel type. When performing a convolution operation, input data and an operator stream file are obtained, and the operator type of the operator in the operator stream file is determined. When the operator type of the operator is a specified operator type, the convolution kernel type of the operator is determined, so that according to the first specified convolution kernel type and the convolution kernel type of the operator, the multiplication-accumulation unit in the neural network processor is used to perform a convolution operation on the input data to obtain a convolution operation result. The neural network processor of the embodiment of the present invention includes acceleration hardware for convolution operations of a first specified convolution kernel type. Therefore, based on the first specified convolution kernel type and the convolution kernel type of the operator, the multiplication-accumulation unit in the neural network processor is used to perform a convolution operation on the input data, which can improve the efficiency of the convolution operation and avoid wasting hardware resources in the neural network processor.

[0173] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0174] In addition, an embodiment of the present invention further provides an electronic device, including: a processor, a memory, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, the various processes of the above-mentioned convolution operation method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be described here.

[0175] The embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, each process of the above-mentioned convolution operation method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it is not repeated here. The computer-readable storage medium is, for example, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0176] An embodiment of the present invention also provides a computer program product, which is stored in a storage medium. The program product is executed by at least one processor to implement the various processes of the above-mentioned convolution operation method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0177] Figure 5 A schematic diagram of the hardware structure of an electronic device for implementing various embodiments of the present invention.

[0178] The electronic device 500 includes but is not limited to: a radio frequency unit 501, a network module 502, an audio output unit 503, an input unit 504, a sensor 505, a display unit 506, a user input unit 507, an interface unit 508, a memory 509, a processor 510, and a power supply 511. Those skilled in the art will appreciate that Figure 5 The electronic device structure shown in the figure does not constitute a limitation on the electronic device, and the electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently. In the embodiments of the present invention, the electronic device includes but is not limited to a mobile phone, a tablet computer, a laptop computer, a PDA, a vehicle-mounted terminal, a wearable device, and a pedometer.

[0179] It should be understood that in the embodiment of the present invention, the radio frequency unit 501 can be used for receiving and sending signals during information transmission or communication. Specifically, after receiving downlink data from the base station, it is sent to the processor 510 for processing; in addition, uplink data is sent to the base station. Generally, the radio frequency unit 501 includes but is not limited to an antenna, at least one amplifier, a transceiver, a coupler, a low noise amplifier, a duplexer, etc. In addition, the radio frequency unit 501 can also communicate with the network and other devices through a wireless communication system.

[0180] The electronic device provides users with wireless broadband Internet access through the network module 502, such as helping users to send and receive emails, browse web pages, and access streaming media.

[0181] The audio output unit 503 can convert the audio data received by the RF unit 501 or the network module 502 or stored in the memory 509 into an audio signal and output it as sound. Moreover, the audio output unit 503 can also provide audio output related to a specific function performed by the electronic device 500 (for example, a call signal reception sound, a message reception sound, etc.). The audio output unit 503 includes a speaker, a buzzer, a receiver, etc.

[0182] The input unit 504 is used to receive audio or video signals. The input unit 504 may include a graphics processor (GPU) 5041 and a microphone 5042. The graphics processor 5041 processes the image data of a static picture or video obtained by an image capture device (such as a camera) in a video capture mode or an image capture mode. The processed image frame can be displayed on the display unit 506. The image frame processed by the graphics processor 5041 can be stored in the memory 509 (or other storage medium) or sent via the radio frequency unit 501 or the network module 502. The microphone 5042 can receive sound and can process such sound into audio data. The processed audio data can be converted into a format output that can be sent to a mobile communication base station via the radio frequency unit 501 in the case of a telephone call mode.

[0183] The electronic device 500 also includes at least one sensor 505, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor includes an ambient light sensor and a proximity sensor, wherein the ambient light sensor can adjust the brightness of the display panel 5061 according to the brightness of the ambient light, and the proximity sensor can turn off the display panel 5061 and / or the backlight when the electronic device 500 is moved to the ear. As a type of motion sensor, the accelerometer sensor can detect the magnitude of acceleration in each direction (generally three axes), and can detect the magnitude and direction of gravity when stationary, which can be used to identify the posture of the electronic device (such as horizontal and vertical screen switching, related games, magnetometer posture calibration), vibration recognition related functions (such as pedometer, tapping), etc.; the sensor 505 can also include a fingerprint sensor, a pressure sensor, an iris sensor, a molecular sensor, a gyroscope, a barometer, a hygrometer, a thermometer, an infrared sensor, etc., which will not be repeated here.

[0184] The display unit 506 is used to display information input by the user or information provided to the user. The display unit 506 may include a display panel 5061, which may be configured in the form of a liquid crystal display (LCD), an organic light-emitting diode (OLED), or the like.

[0185] The user input unit 507 can be used to receive input digital or character information, and to generate key signal input related to user settings and function control of the electronic device. Specifically, the user input unit 507 includes a touch panel 5071 and other input devices 5072. The touch panel 5071, also known as a touch screen, can collect the user's touch operation on or near it (such as the user's operation on the touch panel 5071 or near the touch panel 5071 using any suitable object or accessory such as a finger, stylus, etc.). The touch panel 5071 may include two parts: a touch detection device and a touch controller. Among them, the touch detection device detects the user's touch orientation, detects the signal brought by the touch operation, and transmits the signal to the touch controller; the touch controller receives the touch information from the touch detection device, converts it into the contact point coordinates, and then sends it to the processor 510, receives the command sent by the processor 510 and executes it. In addition, the touch panel 5071 can be implemented using multiple types such as resistive, capacitive, infrared and surface acoustic waves. In addition to the touch panel 5071, the user input unit 507 may also include other input devices 5072. Specifically, other input devices 5072 may include but are not limited to a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and a joystick, which are not described in detail here.

[0186] Furthermore, the touch panel 5071 may be overlaid on the display panel 5061. When the touch panel 5071 detects a touch operation on or near it, it transmits the information to the processor 510 to determine the type of the touch event. Then, the processor 510 provides a corresponding visual output on the display panel 5061 according to the type of the touch event. Figure 5 In the figure, the touch panel 5071 and the display panel 5061 are used as two independent components to realize the input and output functions of the electronic device. However, in some embodiments, the touch panel 5071 and the display panel 5061 can be integrated to realize the input and output functions of the electronic device, which is not limited here.

[0187] The interface unit 508 is an interface for connecting an external device to the electronic device 500. For example, the external device may include a wired or wireless headset port, an external power supply (or battery charger) port, a wired or wireless data port, a memory card port, a port for connecting a device with an identification module, an audio input / output (I / O) port, a video I / O port, a headphone port, etc. The interface unit 508 may be used to receive input (e.g., data information, power, etc.) from an external device and transmit the received input to one or more elements within the electronic device 500 or may be used to transmit data between the electronic device 500 and an external device.

[0188] The memory 509 can be used to store software programs and various data. The memory 509 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area can store data created according to the use of the mobile phone (such as audio data, a phone book, etc.), etc. In addition, the memory 509 can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0189] The processor 510 is the control center of the electronic device. It uses various interfaces and lines to connect various parts of the entire electronic device. By running or executing software programs and / or modules stored in the memory 509 and calling data stored in the memory 509, it performs various functions of the electronic device and processes data, thereby monitoring the electronic device as a whole. The processor 510 may include one or more processing units; preferably, the processor 510 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 510.

[0190] The electronic device 500 may also include a power supply 511 (such as a battery) for supplying power to each component. Preferably, the power supply 511 may be logically connected to the processor 510 through a power management system, thereby implementing functions such as charging, discharging, and power consumption management through the power management system.

[0191] In addition, the electronic device 500 includes some functional modules not shown, which will not be described in detail here.

[0192] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0193] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for enabling a terminal (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0194] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the enlightenment of the present invention, ordinary technicians in this field can also make many forms without departing from the scope of protection of the present invention and the claims, all of which are within the protection of the present invention.

[0195] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed in the embodiments of the present invention can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0196] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0197] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0198] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0199] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0200] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard disks, ROM, RAM, magnetic disks, or optical disks.

[0201] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A convolution operation method, characterized in that: A neural network processor is provided, the neural network processor comprising a multiplication-accumulation unit, the neural network processor comprising a multiplication-accumulation unit for accelerating a convolution operation of a first specified convolution kernel type, the method comprising: Obtain input data and an operator stream file; the operator stream file includes an operator; Determine the operator type of the operator in the operator stream file; When the operator type of the operator is a specified operator type, determining a convolution kernel type of the operator; According to the first specified convolution kernel type and the convolution kernel type of the operator, the multiplication and accumulation unit is correspondingly used to perform a convolution operation on the input data to obtain a convolution operation result.

2. The method according to claim 1, characterized in that Determining the operator type of the operator in the operator stream file includes: When it is determined that the operator in the operator stream file is an operator that can be executed on a specified hardware platform, determining the operator type of the operator as a hardware operator; When it is determined that the operator in the operator stream file is an operator that cannot be executed on the specified hardware platform, the operator type of the operator is determined to be a software operator.

3. The method according to claim 2, characterized in that When the operator type of the operator is a specified operator type, determining the convolution kernel type of the operator includes: When the operator type is a hardware operator, the operator type of the operator is determined to be a specified operator type, and a convolution kernel type of the operator is determined.

4. The method according to claim 3, characterized in that According to the first specified convolution kernel type and the convolution kernel type of the operator, the multiplication and accumulation unit is used to perform a convolution operation on the input data to obtain a convolution operation result, including: When the convolution kernel type of the operator is not the first specified convolution kernel type, determining whether the convolution kernel type of the operator is a second specified convolution kernel type; When the convolution kernel type of the operator is a second specified convolution kernel type, merging every preset number of input channels in the input data to obtain merged input data; Determining the number of combined input channels of the combined input data; According to the preset stride, the number of merged input channels and the first specified convolution kernel type, a convolution operation is performed on the merged input data to obtain a convolution operation result.

5. The method according to claim 4, characterized in that The first specified convolution kernel type includes a 3*3 convolution kernel, the second specified convolution kernel type includes a 1*1 convolution kernel, and the preset step size includes 3.

6. The method according to claim 3, characterized in that: According to the first specified convolution kernel type and the convolution kernel type of the operator, the multiplication and accumulation unit is used to perform a convolution operation on the input data to obtain a convolution operation result, including: When the convolution kernel type of the operator is the first specified convolution kernel type, determining the number of input channels of the input data; According to the preset stride, the number of input channels and the first specified convolution kernel type, the multiplication and accumulation unit is correspondingly used to perform a convolution operation on the input data to obtain a convolution operation result.

7. The method according to claim 3, characterized in that After determining the operator type of the operator in the operator stream file, the method further includes: When the operator type of the operator is not a specified operator type, a preset software algorithm of the operator is called to perform a convolution operation on the input data.

8. The method according to claim 1, characterized in that Before obtaining the input data and the operator stream file, the method further includes: An operator stream file is generated by encoding using a preset artificial intelligence compiler; the operator stream file includes a header, an operator stream, a hardware instruction set, and a quantized network weight set, and the operator stream includes an operator; Storing the operator stream file in a designated memory; When the neural network processor is started, the operator stream file is loaded into a random access memory, so as to obtain the operator stream file from the random access memory for convolution operation.

9. The method according to claim 1, characterized in that: After performing a convolution operation on the input data using the multiplication and accumulation unit according to the first specified convolution kernel type and the convolution kernel type of the operator to obtain a convolution operation result, the method further includes: After the operator in the operator stream file is executed once, the convolution operation result is converted into a specified format; the specified format at least includes a floating point format.

10. A convolution operation device, characterized in that: A neural network processor is provided, the neural network processor comprising a multiplication-accumulation unit, the neural network processor comprising a multiplication-accumulation unit for accelerating a convolution operation of a first specified convolution kernel type, the device comprising: A data acquisition module, used to acquire input data and operator stream files; the operator stream files include operators; An operator type determination module, used to determine the operator type of the operator in the operator stream file; A convolution kernel type determination module, used for determining the convolution kernel type of the operator when the operator type of the operator is a specified operator type; A convolution operation module is used to perform a convolution operation on the input data using the multiplication and accumulation unit according to the first specified convolution kernel type and the convolution kernel type of the operator to obtain a convolution operation result.

11. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; The memory is used to store computer programs; The processor is used to implement the method according to any one of claims 1 to 9 when executing the program stored in the memory.

12. A computer-readable storage medium having instructions stored thereon, which, when executed by one or more processors, cause the processors to perform the method according to any one of claims 1 to 9.

Images