Data processing method and device based on artificial intelligence chip

Abstract

The present disclosure provides an artificial intelligence chip-based data processing method and device, relates to the technical field of artificial intelligence, and particularly relates to the field of AI chips and deep learning technology. The artificial intelligence chip-based data processing method comprises the following steps: inputting to-be-processed data into a three-dimensional convolutional neural network, the three-dimensional convolutional neural network comprising a plurality of three-dimensional operators; obtaining an acceleration library of a target artificial intelligence chip used for executing a data processing process of the three-dimensional convolutional neural network; for any three-dimensional operator in the plurality of three-dimensional operators: in response to a first parameter of the three-dimensional operator satisfying a preset condition, converting the three-dimensional operator into a two-dimensional operator; and calling a function corresponding to the two-dimensional operator in the acceleration library to enable the target artificial intelligence chip to execute the three-dimensional operator.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence (AI), and more particularly to the fields of AI chips and deep learning technology. Specifically, it relates to a data processing method and apparatus based on an artificial intelligence chip, an electronic device, a computer-readable storage medium, and a computer program product. Background Technology

[0002] AI chips, also known as AI accelerators or computing cards, are chips specifically designed for handling AI computing tasks. Compared to traditional computer chips, AI accelerator chips offer higher performance and efficiency in processing complex AI algorithms and large-scale data.

[0003] The methods described in this section are not necessarily methods that had been previously conceived or adopted. Unless otherwise specified, no method described in this section should be assumed to be prior art simply because it is included in this section. Similarly, unless otherwise specified, the issues mentioned in this section should not be considered to be accepted in any prior art. Summary of the Invention

[0004] This disclosure provides a data processing method and apparatus, electronic device, computer-readable storage medium, and computer program product based on an artificial intelligence chip.

[0005] According to one aspect of this disclosure, a data processing method based on an artificial intelligence chip is provided, comprising: inputting data to be processed into a three-dimensional convolutional neural network, the three-dimensional convolutional neural network including a plurality of three-dimensional operators, each of the plurality of three-dimensional operators having a first parameter for describing the three-dimensional data processing process of the three-dimensional operator; acquiring an acceleration library for a target artificial intelligence chip for executing the data processing process of the three-dimensional convolutional neural network, the acceleration library including a plurality of functions, each of the plurality of functions corresponding to an operator in the neural network, and for allocating hardware resources for the operator to enable the target artificial intelligence chip to execute the operator; for any three-dimensional operator among the plurality of three-dimensional operators: in response to the first parameter of the three-dimensional operator satisfying a preset condition, converting the three-dimensional operator into a two-dimensional operator; and calling a function in the acceleration library corresponding to the two-dimensional operator to enable the target artificial intelligence chip to execute the three-dimensional operator.

[0006] According to one aspect of this disclosure, a data processing apparatus based on an artificial intelligence chip is provided, comprising: an input module configured to input data to be processed into a three-dimensional convolutional neural network, the three-dimensional convolutional neural network including a plurality of three-dimensional operators, each of the plurality of three-dimensional operators having a first parameter for describing the three-dimensional data processing process of the three-dimensional operator; an acquisition module configured to acquire an acceleration library for a target artificial intelligence chip for executing the data processing process of the three-dimensional convolutional neural network, the acceleration library including a plurality of functions, each of the plurality of functions corresponding to an operator in the neural network, and configured to allocate hardware resources for the operator to enable the target artificial intelligence chip to execute the operator; and an execution module configured to, for any three-dimensional operator among the plurality of three-dimensional operators: in response to the first parameter of the three-dimensional operator satisfying a preset condition, convert the three-dimensional operator into a two-dimensional operator; and call the function in the acceleration library corresponding to the two-dimensional operator to enable the target artificial intelligence chip to execute the three-dimensional operator.

[0007] According to one aspect of this disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor, the memory storing instructions executable by the at least one processor, the instructions being executed by the at least one processor to cause the at least one processor to perform the method described above.

[0008] According to one aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are used to cause the computer to perform the above-described method.

[0009] According to one aspect of this disclosure, a computer program product is provided, including a computer program, wherein the computer program, when executed by a processor, is capable of implementing the above-described method.

[0010] According to one or more embodiments of this disclosure, the computational efficiency of AI chips for three-dimensional convolutional neural networks can be improved.

[0011] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0012] The accompanying drawings exemplify embodiments and form part of the specification, serving together with the textual description to explain exemplary implementations of the embodiments. The illustrated embodiments are for illustrative purposes only and do not limit the scope of the claims. Throughout the drawings, the same reference numerals refer to similar but not necessarily identical elements.

[0013] Figure 1A flowchart is shown showing a data processing method based on an artificial intelligence chip according to some embodiments of the present disclosure;

[0014] Figure 2 A schematic diagram is shown illustrating the reduction of a three-dimensional convolution operator to a two-dimensional convolution operator according to some embodiments of the present disclosure;

[0015] Figure 3 A schematic diagram is shown illustrating the reduction of a three-dimensional pooling operator to a two-dimensional pooling operator according to some embodiments of the present disclosure;

[0016] Figure 4 A structural block diagram of a data processing apparatus based on an artificial intelligence chip according to some embodiments of the present disclosure is shown; and

[0017] Figure 5 A structural block diagram of an exemplary electronic device that can be used to implement embodiments of the present disclosure is shown. Detailed Implementation

[0018] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0019] In this disclosure, unless otherwise stated, the use of terms such as "first," "second," etc., to describe various elements is not intended to limit the positional, temporal, or importance relationships of these elements; such terms are merely used to distinguish one element from another. In some examples, the first element and the second element may refer to the same instance of that element, while in other cases, based on the context, they may refer to different instances.

[0020] The terminology used in the description of the various examples described in this disclosure is for the purpose of describing particular examples only and is not intended to be limiting. Unless the context explicitly indicates otherwise, an element may be one or more unless the number of elements is specifically limited. Furthermore, the term "and / or" as used in this disclosure covers any one of the listed items and all possible combinations thereof. "More than" means two or more.

[0021] In the technical solutions disclosed herein, the collection, storage, use, processing, transmission, provision, and disclosure of user personal information shall all comply with the provisions of relevant laws and regulations and shall not violate public order and good morals.

[0022] AI chips can perform neural network calculations. Since AI chips have limited hardware resources (such as computing units and memory), to achieve efficient utilization of these resources, AI chip manufacturers typically configure acceleration libraries for their chips. These libraries include processing functions for various neural network operators (such as convolution operators, pooling operators, and activation operators). The processing functions in the acceleration library allocate hardware resources appropriately to the corresponding operators through software acceleration, enabling the AI ​​chip to execute those operators efficiently, thereby optimizing and accelerating neural network calculations.

[0023] Some AI chips can perform calculations using both two-dimensional (e.g., 2D convolution and pooling) and three-dimensional (e.g., 3D convolution and pooling) operators. Correspondingly, the acceleration libraries of these AI chips include processing functions for both 2D and 3D operators. Because 3D operators are more complex than 2D operators, it is more difficult to optimize their processing functions. Furthermore, 3D operators have fewer application scenarios (typically video classification) and lower usage frequency, so AI chip manufacturers typically do not optimize their processing functions promptly and thoroughly. In contrast, 2D operators are the most commonly used operators in deep learning, and AI chip manufacturers usually focus on optimizing them. Therefore, AI chips generally perform better on 2D operators than on 3D operators.

[0024] Video classification tasks can be achieved using 3D convolutional neural networks (CNNs). CNNs contain numerous 3D operators, such as 3D convolution operators and 3D pooling operators. The inference process (i.e., the forward computation process) of a CNN for video classification can be executed by an AI chip. In related technologies, the processing functions of 3D operators in acceleration libraries are typically called directly to perform the computation of the CNN. However, as mentioned above, because the processing functions of 3D operators in acceleration libraries have not been optimized in a timely and sufficient manner, the processing performance of AI chips for 3D operators is relatively low. Directly calling the processing functions of 3D operators in acceleration libraries to perform the computation of the CNN results in low computational efficiency of the AI ​​chip, failing to fully utilize the performance advantages of the AI ​​chip, and thus affecting the efficiency of video classification.

[0025] To address the aforementioned problems, embodiments of this disclosure provide a data processing method based on an artificial intelligence chip. During the forward computation of a three-dimensional convolutional neural network, a three-dimensional operator meeting preset conditions is reduced to a two-dimensional operator. Then, the execution of the three-dimensional operator is achieved by calling a fully optimized and more efficient two-dimensional operator processing function from an acceleration library, thereby improving the computational efficiency of the AI ​​chip for the three-dimensional convolutional neural network. Since the three-dimensional convolutional neural network is used for video classification, embodiments of this disclosure further improve the video classification efficiency of the AI ​​chip.

[0026] The embodiments of this disclosure will now be described in detail with reference to the accompanying drawings.

[0027] Figure 1 A flowchart of a data processing method 100 based on an artificial intelligence chip according to an embodiment of the present disclosure is shown. Method 100 is performed by an electronic device (e.g., equipped with a target artificial intelligence chip) Figure 1 The server 120 in the middle executes the operation. In embodiments of this disclosure, the target artificial intelligence chip refers to an artificial intelligence chip used to perform three-dimensional convolutional neural network calculations. Figure 1 As shown, method 100 includes steps S110-S140.

[0028] In step S110, the data to be processed is input into a three-dimensional convolutional neural network. The three-dimensional convolutional neural network includes multiple three-dimensional operators, and each of the multiple three-dimensional operators has a first parameter for describing the three-dimensional data processing process of that three-dimensional operator.

[0029] In step S120, an acceleration library for the target AI chip used to perform the data processing of a three-dimensional convolutional neural network is obtained. This acceleration library includes multiple functions, each corresponding to an operator in the neural network, and is used to allocate hardware resources for that operator so that the target AI chip can execute it.

[0030] For any three-dimensional operator among multiple three-dimensional operators, execute steps S130 and S140.

[0031] In step S130, in response to the first parameter of the three-dimensional operator satisfying a preset condition, the three-dimensional operator is converted into a two-dimensional operator.

[0032] In step S140, the function corresponding to the two-dimensional operator in the acceleration library is called to make the target artificial intelligence chip execute the three-dimensional operator.

[0033] According to embodiments of this disclosure, during the forward computation of a three-dimensional convolutional neural network, a three-dimensional operator that meets preset conditions is reduced to a two-dimensional operator. Then, the three-dimensional operator is executed by calling the processing function of the fully optimized and more efficient two-dimensional operator in the acceleration library, thereby improving the computational efficiency of the AI ​​chip for the three-dimensional convolutional neural network.

[0034] Since the three-dimensional convolutional neural network of this disclosure is used for video classification, the embodiments of this disclosure further improve the video classification efficiency of AI chips.

[0035] The following details each step of method 100.

[0036] In step S110, the data to be processed is input into a three-dimensional convolutional neural network.

[0037] The three-dimensional convolutional neural network of this disclosure can be used for video classification. Accordingly, the data to be processed can be videos to be classified. The dimension of the input data of the three-dimensional convolutional neural network can be represented, for example, as a five-dimensional vector [N,D,H,W,C], where N is the number of videos to be classified (i.e., batch size), D, H, and W are the depth (i.e., the number of image frames included in a single video), height (i.e., the height of each image frame), and width (i.e., the width of each image frame), respectively, and C is the number of channels in a single video. It can be understood that, compared to a two-dimensional convolutional neural network used for image processing, the three-dimensional convolutional neural network adds a time dimension (D). The dimension of the input data of a two-dimensional convolutional neural network can be represented, for example, as a four-dimensional vector [N,H,W,C].

[0038] A three-dimensional convolutional neural network includes multiple three-dimensional operators, each having a first parameter describing the three-dimensional data processing procedure of that operator. It should be noted that, in the embodiments of this disclosure, to facilitate the distinction between the parameters of the three-dimensional operators and the parameters of the two-dimensional operators, the parameters of the three-dimensional operators are referred to as the first parameter, and the parameters of the two-dimensional operators are referred to as the second parameter.

[0039] The first parameters of a 3D operator may include, for example, the dimension parameters of the input data of the 3D operator, the dimension parameters of the processing kernel used by the 3D operator, and the stride, padding, and dilation of the processing kernel in each dimension.

[0040] For example, a 3D operator can include a 3D convolution operator (conv3d). The processing kernel used by the 3D convolution operator is a convolution kernel. Accordingly, the first parameters of the 3D convolution operator include the dimension parameters of the input data of the 3D convolution operator, the dimension parameters of the convolution kernel, the stride, padding, and dilation of the convolution kernel in each dimension.

[0041] The dimensionality parameter of the input data for a 3D convolution operator indicates the amount of data in the input. For example, the dimensionality parameter can be represented as a five-dimensional vector [N, D, H, W, C], where N is the number of videos to be classified (i.e., batch size), C is the number of channels corresponding to each video (i.e., the number of feature maps), and D, H, and W are the depth, height, and width of the feature map for each channel, respectively. It can be understood that when the 3D convolution operator is the first operator in a 3D convolutional neural network, the input of this 3D convolution operator is equivalent to the input of the 3D convolutional neural network.

[0042] The dimensional parameters of a convolution kernel can be represented as a five-dimensional vector [Co,kd,kh,kw,Ci], where Co is the number of output channels (i.e., the number of convolution sums), Ci is the number of input channels (i.e., the number of channels included in the input data, i.e., the number of input feature maps), and kd, kh, and kw are the depth, height, and width of a single convolution kernel, respectively.

[0043] It can be understood that the convolutional kernel moves in the depth (D), height (H), and width (W) directions of the input data, performing a convolution operation at each position. Therefore, in the embodiments of this disclosure, depth, height, and width are denoted as directional dimensions. The convolutional kernel has movement parameters in each directional dimension, including stride, padding, and dilation. The stride of the convolutional kernel in the depth, height, and width directions can be denoted as stride_d, stride_h, and stride_w, respectively; the padding of the convolutional kernel in the depth, height, and width directions can be denoted as pad_d1, pad_d2, pad_h1, pad_h2, pad_w1, and pad_w2, respectively (it can be understood that the padding at both ends of a certain directional dimension can be different, so each directional dimension corresponds to two paddings); and the dilation of the convolutional kernel in the depth, height, and width directions can be denoted as dilation_d, dilation_h, and dilation_w, respectively.

[0044] In addition to 3D convolution operators, 3D convolutional neural networks typically also include 3D pooling operators. The processing kernel used by the 3D pooling operator is called a pooling kernel. Accordingly, the first parameters of the 3D pooling operator include the dimension parameters of the input data, the dimension parameters of the pooling kernel, the stride of the pooling kernel in each dimension, padding, etc.

[0045] The dimension parameter of the input data for a 3D pooling operator indicates the amount of data in the input data. The dimension parameter of the input data can be represented as a five-dimensional vector [N,D,H,W,C], where N is the number of videos to be classified (i.e., batch size), C is the number of channels corresponding to each video (i.e., the number of feature maps), and D, H, and W are the depth, height, and width of the feature map for each channel, respectively.

[0046] The dimensional parameters of a pooling kernel can be represented as a three-dimensional vector [kd,kh,kw], where kd, kh, and kw are the depth, height, and width of a single pooling kernel, respectively.

[0047] As can be understood, the pooling kernel moves along the depth (D), height (H), and width (W) directions of the input data and performs pooling operations. The pooling kernel has movement parameters in each dimension, including stride and padding. The stride of the pooling kernel in the depth, height, and width directions can be denoted as stride_d, stride_h, and stride_w, respectively, and the padding in the depth, height, and width directions can be denoted as pad_d1, pad_d2, pad_h1, pad_h2, pad_w1, and pad_w2, respectively (it can be understood that the padding at both ends of a certain dimension can be different, therefore each dimension corresponds to two padding values).

[0048] In step S120, an acceleration library for the target artificial intelligence chip used to perform the data processing of a three-dimensional convolutional neural network is obtained.

[0049] The target AI chip can be any AI chip deployed in an electronic device capable of performing three-dimensional convolutional neural network calculations. The target AI chip is pre-configured with an acceleration library (usually pre-configured by the target AI chip manufacturer). This acceleration library includes multiple processing functions (hereinafter referred to as "functions"), each function being used to process an operator in the neural network, such as convolution operators, pooling operators, activation operators, etc. Specifically, each function is used to allocate hardware resources in the target AI chip for the corresponding neural network operator and control the target AI chip to execute the operator using the allocated hardware resources.

[0050] The target AI chip's ability to perform 3D convolutional neural network calculations means that its acceleration library includes functions for both 3D and 2D operators. However, the 3D operator functions are often not optimized in a timely and sufficient manner, and their optimization level is lower than that of the 2D operator functions. Therefore, the target AI chip has lower computational efficiency for 3D operators and cannot fully utilize the performance advantages of AI chips.

[0051] During the forward computation of a three-dimensional convolutional neural network, steps S130 and S140 are executed for any three-dimensional operator among multiple three-dimensional operators.

[0052] In step S130, in response to the first parameter of the three-dimensional operator satisfying a preset condition, the three-dimensional operator is converted into a two-dimensional operator.

[0053] In step S140, the function corresponding to the two-dimensional operator in the acceleration library is called to make the target artificial intelligence chip execute the three-dimensional operator.

[0054] According to some embodiments, as described above, the first parameter of the three-dimensional operator includes the dimension parameter of the input data of the three-dimensional operator, the dimension parameter of the processing kernel used by the three-dimensional operator, the step size and padding of the processing kernel in each directional dimension, etc. Accordingly, the preset condition in step S130 can be: the dimension parameter of the input data, the dimension parameter of the processing kernel, the step size and padding of the processing kernel in each directional dimension enable one directional dimension of the input data to be deleted or merged with an adjacent dimension, and enable one directional dimension of the processing kernel to be deleted or merged with an adjacent directional dimension. Thus, both the input data and the processing kernel of the three-dimensional operator can be reduced by one dimension, thereby reducing its dimensionality to a two-dimensional operator.

[0055] According to some embodiments, a three-dimensional operator can be a three-dimensional convolution operator. The processing kernel of the three-dimensional convolution operator is a convolution kernel. A three-dimensional convolution operator can be transformed into a two-dimensional convolution operator if its first parameter satisfies any of the following sub-conditions.

[0056] First sub-condition: The dimensions of any two adjacent directions in the convolution kernel are both 1, the stride of the preceding direction is 1, the dimension of the following direction in the input data is divisible by the stride of the following direction, and the padding in the following direction is 0. In this case, adjacent directions can be merged, thereby reducing one direction dimension and reducing the 3D convolution operator to a 2D convolution operator.

[0057] For example, the input data of a 3D convolution operator has dimensions [N, D, H, W, C]. The dimensions of two adjacent directions can be H and W, or D and H.

[0058] For two adjacent directional dimensions H and W, if kh = 1, kw = 1, stride_h = 1, W%stride_w = 0 (% represents the modulo operation), and pad_w = 0, then the directional dimensions H and W can be merged (H is folded into W), thereby reducing the 3D convolution operator to a 2D convolution operator. The input data dimension of the 2D convolution operator after dimensionality reduction is [N,D,HW,C], the convolution kernel dimension is [Co,kd,1,Ci], the stride of the convolution kernel is [stride_d,stride_w], the padding is [pad_d1,pad_d2,pad_h1*W,pad_h2*W], and the dilation is [dilation_d,1].

[0059] It's understandable that when kh = 1 and kw = 1, the convolution kernel is [Co, kd, 1, 1, Ci]. The convolution kernel transforms from a three-dimensional cube of length kd*kh*kw into a line segment of length kd, which can also be viewed as a two-dimensional plane of length kd*1. A one-dimensional convolution kernel kd performs convolution on a five-dimensional input [N, D, H, W, C], with no coupling in the H and W dimensions, which is equivalent to a two-dimensional convolution kernel kd*1 performing convolution on a four-dimensional input [N, D, H*W, C]. Considering parameters such as stride, padding, and dilation, when stride_h = 1, to ensure a unified stride after merging the H and W dimensions, W%stride_w needs to be 0. Dilation represents the spacing on the convolution kernel; since kh = 1 and kw = 1, dilation_h and dilation_w must also be 1. If padding_w is not equal to 0, the data in the H and W directions will be discontinuous; therefore, padding_w = ​​0. After merging the H and W dimensions, pad_h becomes pad_h*W after being folded to W. Therefore, the pad of the two-dimensional convolution operator is [pad_d1,pad_d2,pad_h1*W,pad_h2*W].

[0060] Similarly, for two adjacent directional dimensions D and H, if kd = 1, kh = 1, stride_d = 1, H%strided_h = 0, and pad_h = 0, then the directional dimensions D and H can be merged (D is folded into H), thereby reducing the 3D convolution operator to a 2D convolution operator. The input data dimension of the 2D convolution operator after dimensionality reduction is [N,DH,W,C], the convolution kernel dimension is [Co,1,kw,Ci], the stride of the convolution kernel is [stride_h,stride_w], the padding is [pad_d1*H,pad_d2*H,pad_w1,pad_w2], and the dilation is [1,dilation_w].

[0061] The second sub-condition is that the first direction dimension of the convolution kernel has a size of 1, a stride of 1, and padding of 0. In this case, this direction dimension can be merged with the prior batch size dimension N, thereby reducing one direction dimension and reducing the 3D convolution operator to a 2D convolution operator.

[0062] For example, the input data dimension of a 3D convolution operator is [N,D,H,W,C]. Here, the first direction dimension is D. If kd=1, stride_d=1, and pad_d=0 simultaneously, the direction dimension D can be merged with the batch size dimension N, thus reducing the 3D convolution operator to a 2D convolution operator. The input data dimension of the 2D convolution operator after dimensionality reduction is [ND,H,W,C], the kernel dimension is [Co,kh,kw,Ci], the kernel stride is [stride_h,stride_w], the padding is [pad_h1,pad_h2,pad_w1,pad_w2], and the dilation is [dilation_h,dilation_w].

[0063] The third sub-condition: The dimension of any one of the first two directions in the input data is 1, and the stride of that direction is 1. In this case, that direction dimension can be deleted, thereby reducing the dimension by one and reducing the 3D convolution operator to a 2D convolution operator.

[0064] For example, the input data of a 3D convolution operator has dimensions [N,D,H,W,C]. The dimensions of the first two directions are D and H.

[0065] If the orientation dimension D is 1 (i.e., D = 1), and stride_d = 1, then the orientation dimension D can be removed, reducing the 3D convolution operator to a 2D convolution operator. The input data dimension of the 2D convolution operator after dimensionality reduction is [N, H, W, C], the kernel dimension is [Co, kh, kw, Ci], the kernel stride is [stride_h, stride_w], the padding is [pad_d1*(H+pad_h1+pad_h2)+pad_h1,pad_d2*(H+pad_h1+pad_h2)+pad_h2,pad_w1,pad_w2], and the dilation is [dilation_h, dilation_w].

[0066] It should be noted that in the above embodiment, since D = 1, kd must necessarily be 1. Since pad_d may not be equal to 0, stride_d may be greater than 1. When D = 1 and stride_d = 1, the original input data [N,D,H,W,C] degenerates into [N,H,W,C]. Because kd = 1, dilation_d must necessarily be 1. If pad_d1 and pad_d2 are not 0, it can be considered that pad_d1 and pad_d2 are merged into the H dimension. Therefore, pad_h1 becomes pad_d1*(H+pad_h1+pad_h2)+padh1, and pad_h2 becomes pad_d2*(H+pad_h1+pad_h2)+pad_h2.

[0067] Similarly, when the orientation dimension H is 1 (H = 1), if stride_h = 1, the orientation dimension H can be deleted, thus reducing the 3D convolution operator to a 2D convolution operator. The input data dimension of the 2D convolution operator after dimensionality reduction is [N,D,W,C], the kernel dimension is [Co,kd,kw,Ci], the kernel stride is [stride_d,stride_w], and the padding is [pad_d1,pad_d2,pad_h1*(W+pad_w1+pad_w2)+pad_w1,pad_h2*(W+pad_w1+pad_w2)+pad_w2].

[0068] The fourth sub-condition: The dimension of the last direction in the input data is 1, and the padding of that direction is 0. In this case, that direction dimension can be deleted, thereby reducing the dimension by one and reducing the 3D convolution operator to a 2D convolution operator.

[0069] For example, the input data of a 3D convolution operator has dimensions [N,D,H,W,C]. The last direction dimension is W. If W=1 and pad_w=0 simultaneously, the direction dimension W can be deleted, thus reducing the 3D convolution operator to a 2D convolution operator. The input data of the reduced 2D convolution operator has dimensions [N,D,H,C], the kernel dimension is [Co,kd,kw,Ci], the kernel stride is [stride_d,stride_h], and the padding is [pad_d1,pad_d2,pad_h1,pad_h2].

[0070] In the above embodiment, since W=1 and pad_w=0, it can be concluded that kw=1, stride_w=1, and dilation_w=1.

[0071] According to some embodiments, the three-dimensional operator can be a three-dimensional pooling operator. The processing kernel of the three-dimensional pooling operator is a pooling kernel. The three-dimensional pooling operator can be transformed into a two-dimensional pooling operator if its first parameter satisfies any of the following sub-conditions.

[0072] The fifth sub-condition: the dimensions of two adjacent directions in the pooling kernel are both 1, the step size of the earlier direction dimension is 1, the dimension of the later direction dimension in the input data is divisible by the step size of the later direction dimension, and the padding of the later direction dimension is 0. In this case, adjacent directions can be merged, thereby reducing one direction dimension and reducing the 3D pooling operator to a 2D pooling operator.

[0073] For example, the input data of a 3D pooling operator has dimensions [N,D,H,W,C]. The dimensions of two adjacent directions can be H and W, or D and H.

[0074] For two adjacent directional dimensions H and W, if kh = 1, kw = 1, stride_h = 1, W%stride_w = 0, and pad_w = 0, then the directional dimensions H and W can be merged (H is folded into W), thereby reducing the 3D pooling operator to a 2D pooling operator. The input data dimension of the 2D pooling operator after dimensionality reduction is [N,D,HW,C], the dimension of the pooling kernel is [kd,1], the stride of the pooling kernel is [stride_d,stride_w], and the padding is [pad_d1,pad_d2,pad_h1*W,pad_h2*W].

[0075] Similarly, for two adjacent directional dimensions D and H, if kd = 1, kh = 1, stride_d = 1, H%strided_h = 0, and pad_h = 0, then the directional dimensions D and H can be merged (D is folded into H), thereby reducing the 3D pooling operator to a 2D pooling operator. The input data dimension of the 2D pooling operator after dimensionality reduction is [N,DH,W,C], the dimension of the pooling kernel is [1,kw], the stride of the pooling kernel is [stride_h,stride_w], and the padding is [pad_d1*H,pad_d2*H,pad_w1,pad_w2].

[0076] The sixth sub-condition: The first directional dimension in the pooling kernel has a size of 1, a stride of 1, and padding of 0. In this case, this directional dimension can be merged with the prior batch size dimension N, thereby reducing one directional dimension and reducing the 3D convolution operator to a 2D convolution operator.

[0077] For example, the input data dimension of a 3D pooling operator is [N,D,H,W,C]. Here, the first directional dimension is D. If kd=1, stride_d=1, and pad_d=0 simultaneously, the directional dimension D can be merged with the batch size dimension N, thus reducing the 3D pooling operator to a 2D pooling operator. The input data dimension of the reduced 2D pooling operator is [ND,H,W,C], the pooling kernel dimension is [kh,kw], the stride of the pooling kernel is [stride_h,stride_w], and the padding is [pad_h1,pad_h2,pad_w1,pad_w2].

[0078] The seventh sub-condition: The dimension of either of the first two directional dimensions in the input data is 1, and the step size of that directional dimension is 1. In this case, that directional dimension can be deleted, thereby reducing the dimensionality of one directional dimension and reducing the 3D pooling operator to a 2D pooling operator.

[0079] For example, the input data of a three-dimensional pooling operator has dimensions [N,D,H,W,C]. The dimensions of the first two directions are D and H.

[0080] If the dimension D of the direction is 1 (i.e., D = 1), and stride_d = 1, then the dimension D of the direction can be deleted, reducing the three-dimensional pooling operator to a two-dimensional pooling operator. The input data dimension of the reduced two-dimensional pooling operator is [N, H, W, C], the dimension of the pooling kernel is [kh, kw], the stride of the pooling kernel is [stride_h, stride_w], and the padding is [pad_d1*(H+pad_h1+pad_h2)+pad_h1,pad_d2*(H+pad_h1+pad_h2)+pad_h2,pad_w1,pad_w2].

[0081] Similarly, when the dimension H of the orientation dimension is 1 (i.e., H = 1), if stride_h = 1 is also satisfied, the orientation dimension H can be deleted, thereby reducing the 3D pooling operator to a 2D pooling operator. The dimension of the input data of the 2D pooling operator after dimensionality reduction is [N,D,W,C], the dimension of the pooling kernel is [kd,kw], the stride of the pooling kernel is [stride_d,stride_w], and the padding is [pad_d1,pad_d2,pad_h1*(W+pad_w1+pad_w2)+pad_w1,pad_h2*(W+pad_w1+pad_w2)+pad_w2].

[0082] The eighth sub-condition: The dimension of the last direction in the input data is 1, and the padding of that direction is 0. In this case, the direction dimension can be deleted, thereby reducing the dimension by one and reducing the 3D pooling operator to a 2D pooling operator.

[0083] For example, the input data of a 3D pooling operator has dimensions [N,D,H,W,C]. The last directional dimension is W. If W=1 and pad_w=0 simultaneously, the directional dimension W can be deleted, thus reducing the 3D pooling operator to a 2D pooling operator. The input data of the reduced 2D pooling operator has dimensions [N,D,H,C], the pooling kernel has dimensions [kd,kh], the stride of the pooling kernel is [stride_d,stride_h], and the padding is [pad_d1,pad_d2,pad_h1,pad_h2].

[0084] It is understood that in step S130, for each sub-condition described above, the first parameter of the three-dimensional operator can be transformed into the second parameter used to describe the two-dimensional data processing process, thereby obtaining the corresponding two-dimensional operator.

[0085] For example, for a 3D convolution operator, according to the first to fourth sub-conditions mentioned above, the first parameters of the 3D convolution operator (including the dimension of the input data, the dimension of the 3D convolution kernel, the stride of the 3D convolution kernel, padding, and dilation) can be transformed into the second parameters of a 2D convolution operator (the dimension of the input data, the dimension of the 2D convolution kernel, the stride of the 2D convolution kernel, padding, and dilation). For a 3D pooling operator, according to the fifth to eighth sub-conditions mentioned above, the first parameters of the 3D pooling operator (including the dimension of the input data, the dimension of the 3D convolution kernel, the stride of the 3D convolution kernel, and padding) can be transformed into the second parameters of a 2D pooling operator (the dimension of the input data, the dimension of the 2D convolution kernel, the stride of the 2D convolution kernel, and padding).

[0086] Corresponding to obtaining the second parameter of the two-dimensional operator in step S130, in step S140, based on the second parameter, a function corresponding to the two-dimensional operator in the acceleration library can be called to enable the target AI chip to execute the three-dimensional operator. When calling the function corresponding to the two-dimensional operator, the second parameter is used as the input parameter of the function. Since the two-dimensional operator is obtained by dimensionality reduction of the three-dimensional operator, the calculation process of the two-dimensional operator is equivalent to the calculation process of the original three-dimensional operator. By calling the function of the two-dimensional operator, the target AI chip can execute the two-dimensional operator and obtain the execution result of the two-dimensional operator. The execution result of the two-dimensional operator is the execution result of the original three-dimensional operator.

[0087] According to some embodiments, method 100 further includes step S150. In step S150, in response to the first parameter of the three-dimensional operator not satisfying a preset condition, a function corresponding to the three-dimensional operator in the acceleration library is called to cause the target artificial intelligence chip to execute the three-dimensional operator.

[0088] According to the above embodiments, if the first parameter of the three-dimensional operator does not meet the preset conditions, the dimensionality reduction is not performed, and the function corresponding to the three-dimensional operator is directly called to make the target artificial intelligence chip execute the three-dimensional operator.

[0089] It's important to note that dimensionality reduction is not performed when the first parameter of a 3D operator does not meet the preset conditions. This is not because the 3D operator cannot be objectively reduced to a 2D operator, but because dimensionality reduction in this case involves high computational complexity, requiring significant computational resources and time, thus making it difficult to improve computational performance. For example, a dimensionality reduction algorithm can be used to reduce a 3D operator to a 2D operator when the first parameter does not meet the preset conditions. However, this algorithm itself is highly complex and consumes substantial computational resources and time. Therefore, even if dimensionality reduction is achieved using this algorithm, it still cannot improve the computational efficiency of the target AI chip for 3D convolutional neural networks, limiting the application of the target AI chip in video classification.

[0090] According to some embodiments, before performing the forward computation of a three-dimensional convolutional neural network, the three-dimensional operators that meet the conditions in the three-dimensional convolutional neural network can be reduced to two-dimensional operators by graph optimization, thus obtaining an optimized three-dimensional convolutional neural network.

[0091] Figure 2 A schematic diagram is shown illustrating the reduction of a three-dimensional convolution operator to a two-dimensional convolution operator during the graph optimization stage according to some embodiments of the present disclosure.

[0092] like Figure 2 As shown, the three-dimensional convolutional neural network 210 includes a three-dimensional convolution operator (conv3d) 211, a bias operator (bias_add) 212, an activation operator (e.g., ReLU, Rectified Linear Unit) 213, a three-dimensional convolution operator (conv3d) 214, a bias operator (bias_add) 215, and an activation operator (e.g., ReLU) 216.

[0093] The three-dimensional convolution operators 211 and 214 satisfy the preset conditions of this embodiment (the first to fourth sub-conditions mentioned above), therefore, they can be dimensionality-reduced to obtain a three-dimensional convolutional neural network 220. Specifically, the three-dimensional convolution operator 211 is transformed into a reshape operator 2111, a two-dimensional convolution operator (conv2d) 2112, and a reshape operator 2113. The reshape operator 2111 is used to reshape the input data of the three-dimensional convolution operator 211 to conform to the format of the input data of the two-dimensional convolution operator 2112. The reshape operator 2113 is used to reshape the result data of the two-dimensional convolution operator 2112 into the format of the result data of the three-dimensional convolution operator. Similarly, the three-dimensional convolution operator 214 is transformed into a reshape operator 2141, a two-dimensional convolution operator (conv2d) 2142, and a reshape operator 2143.

[0094] It's understandable that bias and activation operators are insensitive to the orientation dimension. Insensitivity means the processing they perform on the input data is independent of the input data's orientation dimension. For bias and activation operators, regardless of the values ​​of the input data's various orientation dimensions (D, H, W), the processing involves adding a bias or activating each data point. Conversely, orientation-sensitive operations are related to the input data's orientation dimension. For example, convolution operators can set different shift parameters for different orientation dimensions, thus performing different processing on those dimensions; aggregation operators (reduce) can process data in a specific orientation dimension, such as accumulating, finding maximum / minimum values, or averaging data in that dimension.

[0095] According to some embodiments, for operators that are insensitive to the orientation dimension and located after a 3D convolution operator, these operators can be moved before the second reshaping operator after dimensionality reduction. For example, see Figure 2 The three-dimensional convolutional neural network 220 can move the bias operator 212 and activation operator 213, which are insensitive to the orientation dimension, before the reshaping operator 2113, and move the bias operator 215 and activation operator 216, which are insensitive to the orientation dimension, before the reshaping operator 2143, to obtain the three-dimensional convolutional neural network 230.

[0096] According to some embodiments, two adjacent reshaping operators can be merged, where the dimension of the input data of the merged reshaping operator is the same as the dimension of the input data of the first reshaping operator, and the dimension of the output data of the merged reshaping operator is the same as the dimension of the output data of the second reshaping operator. For example, see Figure 2The 3D convolutional neural network 230 in the diagram can merge adjacent reshaping operators 2113 and 2141 to obtain a reshaping operator 242. The input data dimension of the reshaping operator 242 is the same as that of the reshaping operator 2113, and the output data dimension of the reshaping operator 242 is the same as that of the reshaping operator 2141. Further, the 2D convolution operator 2112, bias operator 212, and activation operator 213 are represented as a fused 2D convolution operator (fused_conv2d) 241, and the 2D convolution operator 2142, bias operator 215, and activation operator 216 are represented as a fused 2D convolution operator (fused_conv2d) 243, thus obtaining the graph-optimized 3D convolutional neural network 240.

[0097] Furthermore, according to some embodiments, in two adjacent reshaping operators, if the dimension of the input data of the first reshaping operator is the same as the dimension of the output data of the second reshaping operator, then these two reshaping operators can be eliminated, that is, removed from the three-dimensional convolutional neural network. For example, see... Figure 2 In the 3D convolutional neural network 230, if the dimension of the input data of the reshaping operator 2113 is the same as the dimension of the output data of the reshaping operator 2141, then the reshaping operators 2113 and 2141 can be eliminated. Thus, the graph-optimized 3D convolutional neural network 240 only includes the reshaping operator 2111, the fused 2D convolutional operator 241, the fused 2D convolutional operator 243, and the reshaping operator 2143, and no longer includes the reshaping operator 242.

[0098] Figure 3 A schematic diagram is shown illustrating the reduction of a three-dimensional pooling operator to a two-dimensional pooling operator during the graph optimization stage according to some embodiments of the present disclosure.

[0099] like Figure 3 As shown, the three-dimensional convolutional neural network 310 includes pre-operators (pre_ops) 311, a three-dimensional pooling operator 312, and back-operators (back_ops) 313. It can be understood that there can be multiple pre-operators 311, including convolution operators, activation operators, pooling operators, etc. There can also be multiple back-operators 313, including, for example, batch normalization operators, convolution operators, activation operators, pooling operators, etc.

[0100] The three-dimensional pooling operator 312 satisfies the preset conditions of this embodiment (the fifth to eighth sub-conditions mentioned above), therefore it can be dimensionality-reduced to obtain a three-dimensional convolutional neural network 320. Specifically, as... Figure 3 As shown, the three-dimensional pooling operator 312 is transformed into the reshaping operator 3121, the two-dimensional pooling operator 3122, and the reshaping operator 3123.

[0101] Figure 4A structural block diagram of a data processing apparatus 400 based on an artificial intelligence chip according to an embodiment of the present disclosure is shown. Figure 4 As shown, the device 400 includes an input module 410, an acquisition module 420, and an execution module 430.

[0102] The input module 410 is configured to input the data to be processed into a three-dimensional convolutional neural network. The three-dimensional convolutional neural network includes multiple three-dimensional operators, each of which has a first parameter describing the three-dimensional data processing procedure of that three-dimensional operator.

[0103] The acquisition module 420 is configured to acquire an acceleration library for a target AI chip used to perform data processing for a three-dimensional convolutional neural network. This acceleration library includes multiple functions, each corresponding to an operator in the neural network, and is used to allocate hardware resources for that operator so that the target AI chip can execute it.

[0104] The execution module 430 is configured to, for any three-dimensional operator among a plurality of three-dimensional operators: in response to the first parameter of the three-dimensional operator satisfying a preset condition, convert the three-dimensional operator into a two-dimensional operator; and call the function in the acceleration library corresponding to the two-dimensional operator to cause the target artificial intelligence chip to execute the three-dimensional operator.

[0105] According to embodiments of this disclosure, during the forward computation of a three-dimensional convolutional neural network, a three-dimensional operator that meets preset conditions is reduced to a two-dimensional operator. Then, the three-dimensional operator is executed by calling the processing function of the fully optimized and more efficient two-dimensional operator in the acceleration library, thereby improving the computational efficiency of the AI ​​chip for the three-dimensional convolutional neural network.

[0106] Since the three-dimensional convolutional neural network of this disclosure is used for video classification, the embodiments of this disclosure further improve the video classification efficiency of AI chips.

[0107] According to some embodiments, the first parameter includes the dimension parameter of the input data of the three-dimensional operator, the dimension parameter of the processing kernel used by the three-dimensional operator, and the step size and padding of the processing kernel in each directional dimension. The preset conditions include: the dimension parameter of the input data, the dimension parameter of the processing kernel, the step size and padding of the processing kernel in each directional dimension enable one directional dimension of the input data to be deleted or merged with an adjacent dimension, and enable one directional dimension of the processing kernel to be deleted or merged with an adjacent directional dimension.

[0108] According to some embodiments, the three-dimensional operator includes a three-dimensional convolution operator, the processing kernel includes a convolution kernel, and the preset condition includes any one of the following sub-conditions: First sub-condition: the dimensions of two adjacent directional dimensions in the convolution kernel are both 1, the stride of the earlier directional dimension is 1, the dimension of the later directional dimension in the input data is divisible by the stride of the later directional dimension, and the padding of the later directional dimension is 0; Second sub-condition: the dimension of the first directional dimension in the convolution kernel is 1, the stride of the directional dimension is 1, and the padding of the directional dimension is 0; Third sub-condition: the dimension of either of the first two directional dimensions in the input data is 1, and the stride of the directional dimension is 1; Fourth sub-condition: the dimension of the last directional dimension in the input data is 1, and the padding of the directional dimension is 0.

[0109] According to some embodiments, the three-dimensional operator includes a three-dimensional pooling operator, the processing kernel includes a pooling kernel, and the preset condition includes any of the following sub-conditions: Fifth sub-condition: the dimensions of two adjacent directional dimensions in the pooling kernel are both 1, the step size of the earlier directional dimension in the two directional dimensions is 1, the dimension of the later directional dimension in the input data is divisible by the step size of the later directional dimension, and the padding of the later directional dimension is 0; Sixth sub-condition: the dimension of the first directional dimension in the pooling kernel is 1, the step size of the directional dimension is 1, and the padding of the directional dimension is 0; Seventh sub-condition: the dimension of either of the first two directional dimensions in the input data is 1, and the step size of the directional dimension is 1; Eighth sub-condition: the dimension of the last directional dimension in the input data is 1, and the padding of the directional dimension is 0.

[0110] According to some embodiments, the execution module is further configured to: convert the first parameter into a second parameter for describing the two-dimensional data processing process to obtain the two-dimensional operator; and based on the second parameter, call a function in the acceleration library corresponding to the two-dimensional operator to cause the target artificial intelligence chip to execute the three-dimensional operator.

[0111] According to some embodiments, the execution module is further configured to: in response to the first parameter of the three-dimensional operator not satisfying the preset condition, call the function corresponding to the three-dimensional operator in the acceleration library to cause the target artificial intelligence chip to execute the three-dimensional operator.

[0112] According to some embodiments, the three-dimensional convolutional neural network is used for video classification, and the data to be processed is the video to be classified.

[0113] According to some embodiments, the three-dimensional operator includes a three-dimensional convolution operator and a three-dimensional pooling operator. The dimensional parameters of the input data of the three-dimensional convolution operator or the three-dimensional pooling operator include: the number of videos to be classified, the number of channels corresponding to each video, and the depth, height, and width of the feature map of each channel.

[0114] It should be understood that Figure 4 The various modules of the device 400 shown can be used with reference to Figure 1 The steps of method 100 described correspond to each other. Therefore, the operations, features, and advantages of the steps described above for method 100 also apply to the various modules of device 400. For the sake of brevity, some operations, features, and advantages will not be repeated here.

[0115] According to another aspect of this disclosure, an electronic device is also provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor, the memory storing instructions executable by the at least one processor, the instructions being executed by the at least one processor to cause the at least one processor to perform the data processing method 100 based on the artificial intelligence chip described above.

[0116] According to another aspect of this disclosure, a non-transitory computer-readable storage medium storing computer instructions is also provided. These computer instructions are used to cause the computer to execute the aforementioned data processing method 100 based on an artificial intelligence chip.

[0117] According to another aspect of this disclosure, a computer program product is also provided, including computer program instructions that, when executed by a processor, implement the aforementioned data processing method 100 based on an artificial intelligence chip.

[0118] refer to Figure 5 The present invention describes a structural block diagram of an electronic device 500 that can serve as a server or client of the present disclosure, which is an example of a hardware device that can be applied to various aspects of the present disclosure. The electronic device is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0119] like Figure 5As shown, device 500 includes a computing unit 501, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 502 or a computer program loaded from storage unit 508 into random access memory (RAM) 503. RAM 503 may also store various programs and data required for the operation of device 500. The computing unit 501, ROM 502, and RAM 503 are interconnected via bus 504. Input / output (I / O) interface 505 is also connected to bus 504.

[0120] Multiple components in device 500 are connected to I / O interface 505, including: input unit 506, output unit 507, storage unit 508, and communication unit 509. Input unit 506 can be any type of device capable of inputting information to device 500. Input unit 506 can receive input numerical or character information and generate key signal inputs related to user settings and / or function control of the electronic device, and may include, but is not limited to, a mouse, keyboard, touchscreen, trackpad, trackball, joystick, microphone, and / or remote control. Output unit 507 can be any type of device capable of presenting information, and may include, but is not limited to, a monitor, speaker, video / audio output terminal, vibrator, and / or printer. Storage unit 508 may include, but is not limited to, a hard disk and an optical disk. Communication unit 509 allows device 500 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks, and may include, but is not limited to, modems, network cards, infrared communication devices, wireless communication transceivers, and / or chipsets, such as Bluetooth devices, 802.5 devices, Wi-Fi devices, WiMAX devices, cellular communication devices, and / or the like.

[0121] The computing unit 501 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 501 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 501 performs the various methods and processes described above, such as the AI ​​chip-based data processing method 100. For example, in some embodiments, the AI ​​chip-based data processing method 100 can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 508. In some embodiments, part or all of the computer program can be loaded and / or installed on device 500 via ROM 502 and / or communication unit 509. When the computer program is loaded into RAM 503 and executed by the computing unit 501, one or more steps of the AI ​​chip-based data processing method 100 described above can be performed. Alternatively, in other embodiments, computing unit 501 may be configured to perform data processing method 100 based on artificial intelligence chip by any other suitable means (e.g., by means of firmware).

[0122] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0123] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0124] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0125] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0126] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), the Internet, and blockchain networks.

[0127] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0128] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0129] While embodiments or examples of this disclosure have been described with reference to the accompanying drawings, it should be understood that the methods, systems, and devices described above are merely exemplary embodiments or examples, and the scope of this disclosure is not limited to these embodiments or examples. Various elements in the embodiments or examples may be omitted or replaced by their equivalents. Furthermore, the steps may be performed in a different order than that described in this disclosure. Further, various elements in the embodiments or examples may be combined in various ways. Importantly, as the technology evolves, many elements described herein can be replaced by equivalents that appear after this disclosure.

Claims

1. A data processing method based on an artificial intelligence chip, comprising: The data to be processed is input into a three-dimensional convolutional neural network, wherein the three-dimensional convolutional neural network is used for video classification, the data to be processed is the video to be classified, and the three-dimensional convolutional neural network includes multiple three-dimensional operators, each of the multiple three-dimensional operators having a first parameter for describing the three-dimensional data processing process of the three-dimensional operator; An acceleration library is obtained for a target AI chip to perform the data processing of the three-dimensional convolutional neural network. The acceleration library includes multiple functions, each corresponding to an operator in the neural network, and is used to allocate hardware resources to the operator so that the target AI chip can execute the operator. For any three-dimensional operator among the plurality of three-dimensional operators: In response to the first parameter of the three-dimensional operator satisfying a preset condition, the three-dimensional operator is converted into a two-dimensional operator; and The function corresponding to the two-dimensional operator in the acceleration library is called to cause the target artificial intelligence chip to execute the three-dimensional operator. The first parameter includes the dimension parameter of the input data of the 3D operator, the dimension parameter of the processing kernel used by the 3D operator, and the step size and padding of the processing kernel in each directional dimension. The preset conditions include: The dimension parameters of the input data, the dimension parameters of the processing kernel, the step size and padding of the processing kernel in each directional dimension enable one directional dimension of the input data to be deleted or merged with an adjacent dimension, and enable one directional dimension of the processing kernel to be deleted or merged with an adjacent directional dimension.

2. The method according to claim 1, wherein, The three-dimensional operator includes a three-dimensional convolution operator, the processing kernel includes a convolution kernel, and the preset condition includes any one of the following sub-conditions: First sub-condition: The dimensions of two adjacent directional dimensions in the convolution kernel are both 1, the stride of the earlier directional dimension is 1, the dimension of the later directional dimension in the input data is divisible by the stride of the later directional dimension, and the padding of the later directional dimension is 0. Second sub-condition: The size of the first direction dimension in the convolution kernel is 1, the stride of the direction dimension is 1, and the padding of the direction dimension is 0; Third sub-condition: The dimension of any one of the first two directional dimensions in the input data is 1, and the step size of that directional dimension is 1; Fourth sub-condition: The dimension of the last direction in the input data is 1, and the padding of that direction dimension is 0.

3. The method according to claim 1, wherein, The three-dimensional operator includes a three-dimensional pooling operator, the processing kernel includes a pooling kernel, and the preset condition includes any of the following sub-conditions: Fifth sub-condition: The dimensions of two adjacent directional dimensions in the pooling kernel are both 1, the step size of the earlier directional dimension is 1, the dimension of the later directional dimension in the input data is divisible by the step size of the later directional dimension, and the padding of the later directional dimension is 0. Sixth sub-condition: The size of the first directional dimension in the pooling kernel is 1, the step size of the directional dimension is 1, and the padding of the directional dimension is 0; Seventh sub-condition: The dimension of any one of the first two directional dimensions in the input data is 1, and the step size of that directional dimension is 1; Eighth sub-condition: The dimension of the last direction in the input data is 1, and the padding of that direction dimension is 0.

4. The method according to any one of claims 1-3, wherein, The step of converting the three-dimensional operator into a two-dimensional operator includes: converting the first parameter into a second parameter used to describe the two-dimensional data processing process, so as to obtain the two-dimensional operator. Furthermore, the step of calling the function corresponding to the two-dimensional operator in the acceleration library to cause the target AI chip to execute the three-dimensional operator includes: based on the second parameter, calling the function corresponding to the two-dimensional operator in the acceleration library to cause the target AI chip to execute the three-dimensional operator.

5. The method according to claim 1, further comprising: In response to the first parameter of the three-dimensional operator not satisfying the preset condition, the function corresponding to the three-dimensional operator in the acceleration library is called to cause the target artificial intelligence chip to execute the three-dimensional operator.

6. The method according to claim 1, wherein, The three-dimensional operators include a three-dimensional convolution operator and a three-dimensional pooling operator. The dimensional parameters of the input data of the three-dimensional convolution operator or the three-dimensional pooling operator include: the number of videos to be classified, the number of channels corresponding to each video, and the depth, height, and width of the feature map of each channel.

7. A data processing device based on an artificial intelligence chip, comprising: An input module is configured to input data to be processed into a three-dimensional convolutional neural network, wherein the three-dimensional convolutional neural network is used for video classification, the data to be processed is a video to be classified, and the three-dimensional convolutional neural network includes a plurality of three-dimensional operators, each of the plurality of three-dimensional operators having a first parameter for describing the three-dimensional data processing process of the three-dimensional operator. An acquisition module is configured to acquire an acceleration library for a target AI chip used to perform the data processing of the three-dimensional convolutional neural network, wherein the acceleration library includes multiple functions, each function corresponding to an operator in the neural network, and is used to allocate hardware resources to the operator so that the target AI chip executes the operator; and The execution module is configured to work for any one of the plurality of three-dimensional operators: In response to the first parameter of the three-dimensional operator satisfying a preset condition, the three-dimensional operator is converted into a two-dimensional operator; and The function corresponding to the two-dimensional operator in the acceleration library is called to cause the target artificial intelligence chip to execute the three-dimensional operator. The first parameter includes the dimension parameter of the input data of the 3D operator, the dimension parameter of the processing kernel used by the 3D operator, and the step size and padding of the processing kernel in each directional dimension. The preset conditions include: The dimension parameters of the input data, the dimension parameters of the processing kernel, the step size and padding of the processing kernel in each directional dimension enable one directional dimension of the input data to be deleted or merged with an adjacent dimension, and enable one directional dimension of the processing kernel to be deleted or merged with an adjacent directional dimension.

8. The apparatus according to claim 7, wherein, The three-dimensional operator includes a three-dimensional convolution operator, the processing kernel includes a convolution kernel, and the preset condition includes any one of the following sub-conditions: First sub-condition: The dimensions of two adjacent directional dimensions in the convolution kernel are both 1, the stride of the earlier directional dimension is 1, the dimension of the later directional dimension in the input data is divisible by the stride of the later directional dimension, and the padding of the later directional dimension is 0. Second sub-condition: The size of the first direction dimension in the convolution kernel is 1, the stride of the direction dimension is 1, and the padding of the direction dimension is 0; Third sub-condition: The dimension of any one of the first two directional dimensions in the input data is 1, and the step size of that directional dimension is 1; Fourth sub-condition: The dimension of the last direction in the input data is 1, and the padding of that direction dimension is 0.

9. The apparatus according to claim 7, wherein, The three-dimensional operator includes a three-dimensional pooling operator, the processing kernel includes a pooling kernel, and the preset condition includes any of the following sub-conditions: Fifth sub-condition: The dimensions of two adjacent directional dimensions in the pooling kernel are both 1, the step size of the earlier directional dimension is 1, the dimension of the later directional dimension in the input data is divisible by the step size of the later directional dimension, and the padding of the later directional dimension is 0. Sixth sub-condition: The size of the first directional dimension in the pooling kernel is 1, the step size of the directional dimension is 1, and the padding of the directional dimension is 0; Seventh sub-condition: The dimension of any one of the first two directional dimensions in the input data is 1, and the step size of that directional dimension is 1; Eighth sub-condition: The dimension of the last direction in the input data is 1, and the padding of that direction dimension is 0.

10. The apparatus according to any one of claims 7-9, wherein, The execution module is further configured as follows: The first parameter is converted into a second parameter to describe the two-dimensional data processing procedure, thus obtaining the two-dimensional operator; and Based on the second parameter, the function corresponding to the two-dimensional operator in the acceleration library is called to cause the target artificial intelligence chip to execute the three-dimensional operator.

11. The apparatus according to claim 7, wherein, The execution module is further configured as follows: In response to the first parameter of the three-dimensional operator not satisfying the preset condition, the function corresponding to the three-dimensional operator in the acceleration library is called to cause the target artificial intelligence chip to execute the three-dimensional operator.

12. The apparatus according to claim 7, wherein, The three-dimensional operators include a three-dimensional convolution operator and a three-dimensional pooling operator. The dimensional parameters of the input data of the three-dimensional convolution operator or the three-dimensional pooling operator include: the number of videos to be classified, the number of channels corresponding to each video, and the depth, height, and width of the feature map of each channel.

13. An electronic device, comprising: At least one processor; as well as The memory that is communicatively connected to the at least one processor The memory stores instructions that can be executed by the at least one processor to cause the at least one processor to perform the method of any one of claims 1-6.

14. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-6.

15. A computer program product comprising computer program instructions, wherein, The computer program instructions, when executed by a processor, implement the method according to any one of claims 1-6.

Images