Data processing method and device based on artificial intelligence chip

Abstract

The invention provides a data processing method and device based on an artificial intelligence chip, and relates to the technical field of artificial intelligence, in particular to the technical field of AI chips and deep learning. The artificial intelligence chip-based data processing method comprises the steps of inputting to-be-processed data into a three-dimensional convolutional neural network, wherein the three-dimensional convolutional neural network comprises a plurality of three-dimensional operators; obtaining an acceleration library of a target artificial intelligence chip used for executing a data processing process of the three-dimensional convolutional neural network; for any three-dimensional operator in the plurality of three-dimensional operators, responding to a condition that a first parameter of the three-dimensional operator meets a preset condition, and converting the three-dimensional operator into a two-dimensional operator; and calling a function corresponding to the two-dimensional operator in an acceleration library to enable the target artificial intelligence chip to execute the three-dimensional operator.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence (AI), in particular to the field of AI chips and deep learning technology, and specifically to a data processing method and device based on an AI chip, an electronic device, a computer-readable storage medium, and a computer program product. Background Art

[0002] AI chips, also known as AI accelerators or computing cards, are chips specifically designed to handle AI computing tasks. Compared with traditional computer chips, AI accelerator chips have higher performance and efficiency in processing complex AI algorithms and large-scale data.

[0003] The methods described in this section are not necessarily methods that have been previously conceived or employed. Unless otherwise indicated, it should not be assumed that any method described in this section is considered to be prior art simply because it is included in this section. Similarly, unless otherwise indicated, the issues mentioned in this section should not be considered to have been recognized in any prior art. Summary of the invention

[0004] The present disclosure provides a data processing method and device based on an artificial intelligence chip, an electronic device, a computer-readable storage medium, and a computer program product.

[0005] According to one aspect of the present disclosure, there is provided a data processing method based on an artificial intelligence chip, comprising: inputting data to be processed into a three-dimensional convolutional neural network, the three-dimensional convolutional neural network comprising a plurality of three-dimensional operators, each of the plurality of three-dimensional operators having a first parameter for describing a three-dimensional data processing process of the three-dimensional operator; obtaining an acceleration library of a target artificial intelligence chip for executing the data processing process of the three-dimensional convolutional neural network, the acceleration library comprising a plurality of functions, each of the plurality of functions corresponding to an operator in the neural network and being used to allocate hardware resources to the operator so that the target artificial intelligence chip executes the operator; for any three-dimensional operator among the plurality of three-dimensional operators: in response to the first parameter of the three-dimensional operator satisfying a preset condition, converting the three-dimensional operator into a two-dimensional operator; and calling the function corresponding to the two-dimensional operator in the acceleration library so that the target artificial intelligence chip executes the three-dimensional operator.

[0006] According to one aspect of the present disclosure, there is provided a data processing device based on an artificial intelligence chip, comprising: an input module, configured to input data to be processed into a three-dimensional convolutional neural network, the three-dimensional convolutional neural network comprising a plurality of three-dimensional operators, each of the plurality of three-dimensional operators having a first parameter for describing a three-dimensional data processing process of the three-dimensional operator; an acquisition module, configured to acquire an acceleration library of a target artificial intelligence chip for executing the data processing process of the three-dimensional convolutional neural network, the acceleration library comprising a plurality of functions, each of the plurality of functions corresponding to an operator in the neural network and used to allocate hardware resources to the operator so that the target artificial intelligence chip executes the operator; an execution module, configured to: for any three-dimensional operator among the plurality of three-dimensional operators: in response to the first parameter of the three-dimensional operator satisfying a preset condition, convert the three-dimensional operator into a two-dimensional operator; and call the function corresponding to the two-dimensional operator in the acceleration library so that the target artificial intelligence chip executes the three-dimensional operator.

[0007] According to one aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor, the memory storing instructions executable by the at least one processor, the instructions being executed by the at least one processor so that the at least one processor executes the method.

[0008] According to one aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the above method.

[0009] According to one aspect of the present disclosure, a computer program product is provided, including a computer program, wherein the computer program can implement the above method when executed by a processor.

[0010] According to one or more embodiments of the present disclosure, the computing efficiency of AI chips for three-dimensional convolutional neural networks can be improved.

[0011] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The accompanying drawings exemplarily illustrate the embodiments and constitute a part of the specification, and together with the text description of the specification, are used to explain the exemplary implementation of the embodiments. The embodiments shown are for illustrative purposes only and do not limit the scope of the claims. In all drawings, the same reference numerals refer to similar but not necessarily identical elements.

[0013] Figure 1A flowchart of a data processing method based on an artificial intelligence chip according to some embodiments of the present disclosure is shown;

[0014] Figure 2 A schematic diagram of reducing the dimension of a three-dimensional convolution operator to a two-dimensional convolution operator according to some embodiments of the present disclosure is shown;

[0015] Figure 3 A schematic diagram showing a method of reducing the dimension of a three-dimensional pooling operator to a two-dimensional pooling operator according to some embodiments of the present disclosure is shown;

[0016] Figure 4 A structural block diagram of a data processing device based on an artificial intelligence chip according to some embodiments of the present disclosure is shown; and

[0017] Figure 5 A structural block diagram of an exemplary electronic device that can be used to implement the embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0018] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope of the present disclosure. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0019] In the present disclosure, unless otherwise specified, the use of the terms "first", "second", etc. to describe various elements is not intended to limit the positional relationship, timing relationship, or importance relationship of these elements, and such terms are only used to distinguish one element from another element. In some examples, the first element and the second element may refer to the same instance of the element, and in some cases, based on the description of the context, they may also refer to different instances.

[0020] The terms used in the description of various examples in this disclosure are only for the purpose of describing specific examples and are not intended to be limiting. Unless the context clearly indicates otherwise, if the number of elements is not specifically limited, the element can be one or more. In addition, the term "and / or" used in this disclosure covers any one of the listed items and all possible combinations. "Multiple" refers to two or more.

[0021] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are all subject to the provisions of relevant laws and regulations and do not violate public order and good morals.

[0022] AI chips can perform neural network calculations. AI chips have limited hardware resources (such as computing units, memory, etc.). In order to achieve efficient use of hardware resources, AI chip manufacturers usually configure acceleration libraries for AI chips. The acceleration library includes processing functions for each neural network operator (such as convolution operator, pooling operator, activation operator, etc.). The processing functions in the acceleration library reasonably allocate hardware resources to the corresponding operators through software acceleration, so that the AI ​​chip can efficiently execute the operator, thereby optimizing and accelerating neural network calculations.

[0023] Some AI chips can perform calculations of both two-dimensional operators (such as two-dimensional convolution operators, two-dimensional pooling operators, etc.) and three-dimensional operators (such as three-dimensional convolution operators, three-dimensional pooling operators, etc.). Accordingly, the acceleration libraries of these AI chips include both processing functions for two-dimensional operators and processing functions for three-dimensional operators. Since the operations of three-dimensional operators are more complex than those of two-dimensional operators, it is also more difficult to optimize the processing functions of three-dimensional operators. At the same time, three-dimensional operators have fewer usage scenarios (usually used for video classification) and are used less frequently, so AI chip manufacturers usually do not optimize the processing functions of three-dimensional operators in a timely and sufficient manner. As the most commonly used operator in the field of deep learning, two-dimensional operators are usually optimized by AI chip manufacturers. Therefore, the processing performance of AI chips for two-dimensional operators is usually better than that for three-dimensional operators.

[0024] The video classification task can be implemented using a three-dimensional convolutional neural network. The three-dimensional convolutional neural network contains a large number of three-dimensional operators, such as three-dimensional convolution operators, three-dimensional pooling operators, etc. The reasoning process (i.e., the forward calculation process) of the three-dimensional convolutional neural network for video classification can be performed by an AI chip. In the related art, the processing function of the three-dimensional operator in the acceleration library is usually directly called to realize the calculation of the three-dimensional convolutional neural network. However, as mentioned above, since the processing function of the three-dimensional operator in the acceleration library has not been timely and fully optimized, the processing performance of the AI ​​chip for the three-dimensional operator is low. If the processing function of the three-dimensional operator in the acceleration library is directly called to calculate the three-dimensional convolutional neural network, the computing efficiency of the AI ​​chip will be low, and the performance advantages of the AI ​​chip cannot be fully utilized, which in turn affects the efficiency of video classification.

[0025] In response to the above problems, the embodiments of the present disclosure provide a data processing method based on an artificial intelligence chip. In the forward calculation process of the three-dimensional convolutional neural network, the three-dimensional operator that meets the preset conditions is reduced to a two-dimensional operator, and then the execution of the three-dimensional operator is realized by calling the fully optimized and more efficient processing function of the two-dimensional operator in the acceleration library, thereby improving the computing efficiency of the AI ​​chip for the three-dimensional convolutional neural network. Since the three-dimensional convolutional neural network is used for video classification, the embodiments of the present disclosure further improve the video classification efficiency of the AI ​​chip.

[0026] The embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings.

[0027] Figure 1 1 shows a flow chart of a data processing method 100 based on an artificial intelligence chip according to an embodiment of the present disclosure. The method 100 is performed by an electronic device (e.g. Figure 1 In the embodiment of the present disclosure, the target artificial intelligence chip refers to an artificial intelligence chip for performing three-dimensional convolutional neural network calculations. Figure 1 As shown, the method 100 includes steps S110 - S140 .

[0028] In step S110, the data to be processed is input into a three-dimensional convolutional neural network. The three-dimensional convolutional neural network includes a plurality of three-dimensional operators, each of which has a first parameter for describing a three-dimensional data processing process of the three-dimensional operator.

[0029] In step S120, an acceleration library of a target artificial intelligence chip for executing a data processing process of a three-dimensional convolutional neural network is obtained. The acceleration library includes a plurality of functions, each of which corresponds to an operator in a neural network and is used to allocate hardware resources to the operator so that the target artificial intelligence chip executes the operator.

[0030] For any three-dimensional operator among the multiple three-dimensional operators, steps S130 and S140 are performed.

[0031] In step S130 , in response to the first parameter of the three-dimensional operator satisfying a preset condition, the three-dimensional operator is converted into a two-dimensional operator.

[0032] In step S140, the function corresponding to the two-dimensional operator in the acceleration library is called to enable the target artificial intelligence chip to execute the three-dimensional operator.

[0033] According to an embodiment of the present disclosure, in the forward calculation process of a three-dimensional convolutional neural network, a three-dimensional operator that meets preset conditions is reduced to a two-dimensional operator, and then the execution of the three-dimensional operator is realized by calling a fully optimized and more efficient processing function of the two-dimensional operator in the acceleration library, thereby improving the computing efficiency of the AI ​​chip for the three-dimensional convolutional neural network.

[0034] Since the three-dimensional convolutional neural network of the embodiment of the present disclosure is used for video classification, the embodiment of the present disclosure further improves the video classification efficiency of the AI ​​chip.

[0035] The following is a detailed description of each step of method 100.

[0036] In step S110, the data to be processed is input into the three-dimensional convolutional neural network.

[0037] The three-dimensional convolutional neural network of the disclosed embodiment can be used for video classification. Accordingly, the data to be processed can be a video to be classified. The dimension of the input data of the three-dimensional convolutional neural network can be expressed as a five-dimensional vector [N, D, H, W, C], for example, where N is the number of videos to be classified (i.e., the batch size), D, H, and W are the depth (i.e., the number of image frames included in a single video), height (i.e., the height of each image frame), and width (i.e., the width of each image frame) of a single video, respectively, and C is the number of channels of a single video. It can be understood that compared to the two-dimensional convolutional neural network used for image processing, the three-dimensional convolutional neural network adds a time dimension (D). The dimension of the input data of the two-dimensional convolutional neural network can be expressed as a four-dimensional vector [N, H, W, C].

[0038] The three-dimensional convolutional neural network includes a plurality of three-dimensional operators, each of which has a first parameter for describing the three-dimensional data processing process of the three-dimensional operator. It should be noted that in the embodiments of the present disclosure, in order to facilitate the distinction between the parameters of the three-dimensional operator and the parameters of the two-dimensional operator, the parameters of the three-dimensional operator are recorded as the first parameters, and the parameters of the two-dimensional operator are recorded as the second parameters.

[0039] The first parameters of the three-dimensional operator include, for example, the dimension parameters of the input data of the three-dimensional operator, the dimension parameters of the processing core used by the three-dimensional operator, the stride, padding, dilation, etc. of the processing core in each direction.

[0040] For example, the three-dimensional operator may include a three-dimensional convolution operator (conv3d). The processing kernel used by the three-dimensional convolution operator is a convolution kernel (conv kernel). Accordingly, the first parameter of the three-dimensional convolution operator includes the dimension parameter of the input data of the three-dimensional convolution operator, the dimension parameter of the convolution kernel, the step size, padding, expansion, etc. of the convolution kernel in each direction dimension.

[0041] The dimension parameter of the input data of the three-dimensional convolution operator indicates the data volume of its input data. The dimension parameter of the input data can be expressed as a five-dimensional vector [N, D, H, W, C], for example, where N is the number of videos to be classified (i.e., the batch size), C is the number of channels corresponding to each video (i.e., the number of feature maps), and D, H, and W are the depth, height, and width of the feature map of each channel, respectively. It can be understood that when the three-dimensional convolution operator is the first operator in the three-dimensional convolutional neural network, the input of the three-dimensional convolution operator is equivalent to the input of the three-dimensional convolutional neural network.

[0042] The dimensional parameters of the convolution kernel can be expressed as a five-dimensional vector [Co, kd, kh, kw, Ci], for example, where Co is the number of output channels (i.e., the number of convolution sums), Ci is the number of input channels (i.e., the number of channels included in the input data, that is, the number of input feature maps), and kd, kh, and kw are the depth, height, and width of a single convolution kernel, respectively.

[0043] It can be understood that the convolution kernel moves in the three directions of depth (D), height (H) and width (W) of the input data, and a convolution operation is performed each time it moves to a position. Therefore, in the embodiments of the present disclosure, the depth, height and width are recorded as directional dimensions. The convolution kernel has movement parameters in various directional dimensions, including step size, padding, expansion, etc. The step size of the convolution kernel in the depth, height and width directions can be recorded as stride_d, stride_h, stride_w, respectively, and the padding of the convolution kernel in the depth, height and width directions can be recorded as pad_d1, pad_d2, pad_h1, pad_h2, pad_w1, pad_w2 (it can be understood that the padding at both ends of a certain directional dimension can be different, so each directional dimension corresponds to two paddings), and the expansion of the convolution kernel in the depth, height and width directions can be recorded as dilation_d, dilation_h, dilation_w, respectively.

[0044] In addition to the three-dimensional convolution operator, the three-dimensional convolutional neural network usually also includes a three-dimensional pooling operator. The processing kernel used by the three-dimensional pooling operator is a pooling kernel. Accordingly, the first parameter of the three-dimensional pooling operator includes the dimension parameter of the input data of the three-dimensional pooling operator, the dimension parameter of the pooling kernel, the step size of the pooling kernel in each direction dimension, padding, etc.

[0045] The dimension parameter of the input data of the three-dimensional pooling operator indicates the amount of data of its input data. The dimension parameter of the input data can be expressed as a five-dimensional vector [N, D, H, W, C], for example, where N is the number of videos to be classified (i.e., the batch size), C is the number of channels corresponding to each video (i.e., the number of feature maps), and D, H, and W are the depth, height, and width of the feature map of each channel, respectively.

[0046] The dimension parameter of the pooling kernel can be expressed as a three-dimensional vector [kd, kh, kw], where kd, kh, and kw are the depth, height, and width of a single pooling kernel, respectively.

[0047] It can be understood that the pooling core moves in the three directions of depth (D), height (H) and width (W) of the input data and performs pooling operations. The pooling core has movement parameters in various directional dimensions, including stride, padding, etc. The stride of the pooling core in the depth, height, and width directions can be recorded as stride_d, stride_h, and stride_w, respectively, and the padding of the pooling core in the depth, height, and width directions can be recorded as pad_d1, pad_d2, pad_h1, pad_h2, pad_w1, and pad_w2, respectively (it can be understood that the padding at both ends of a certain directional dimension can be different, so each directional dimension corresponds to two paddings).

[0048] In step S120, an acceleration library of a target artificial intelligence chip for executing a data processing process of a three-dimensional convolutional neural network is obtained.

[0049] The target artificial intelligence chip can be any artificial intelligence chip deployed in an electronic device that can perform three-dimensional convolutional neural network calculations. The target artificial intelligence chip is pre-configured with an acceleration library (usually pre-configured by the manufacturer of the target artificial intelligence chip), which includes multiple processing functions (hereinafter referred to as "functions"), each of which is used to process an operator in a neural network, such as a convolution operator, a pooling operator, an activation operator, etc. Specifically, each function is used to allocate hardware resources in the target artificial intelligence chip to the corresponding neural network operator, and control the target artificial intelligence chip to execute the operator using the allocated hardware resources.

[0050] The target AI chip is capable of performing three-dimensional convolutional neural network calculations, which means that the acceleration library of the target AI chip includes both three-dimensional operator functions and two-dimensional operator functions. However, the functions of three-dimensional operators are often not optimized in a timely and sufficient manner, and their optimization level is lower than that of two-dimensional operator functions. Therefore, the target AI chip has low computing efficiency for three-dimensional operators and cannot give full play to the performance advantages of AI chips.

[0051] In the forward calculation process of the three-dimensional convolutional neural network, for any three-dimensional operator among the multiple three-dimensional operators, steps S130 and S140 are executed.

[0052] In step S130 , in response to the first parameter of the three-dimensional operator satisfying a preset condition, the three-dimensional operator is converted into a two-dimensional operator.

[0053] In step S140, the function corresponding to the two-dimensional operator in the acceleration library is called to enable the target artificial intelligence chip to execute the three-dimensional operator.

[0054] According to some embodiments, as described above, the first parameter of the three-dimensional operator includes the dimension parameter of the input data of the three-dimensional operator, the dimension parameter of the processing core used by the three-dimensional operator, the step size and padding of the processing core in each direction dimension, etc. Accordingly, the preset conditions in step S130 may be: the dimension parameter of the input data, the dimension parameter of the processing core, the step size and padding of the processing core in each direction dimension enable one direction dimension in the input data to be deleted or merged with an adjacent dimension, and enable one direction dimension in the processing core to be deleted or merged with an adjacent direction dimension. Thus, the input data and the processing core of the three-dimensional operator can both reduce one dimension, so that it can be reduced to a two-dimensional operator.

[0055] According to some embodiments, the three-dimensional operator may be a three-dimensional convolution operator. The processing kernel of the three-dimensional convolution operator is a convolution kernel. When the first parameter of the three-dimensional convolution operator satisfies any of the following sub-conditions, the three-dimensional convolution operator may be converted into a two-dimensional convolution operator.

[0056] The first sub-condition: the size of the two adjacent directional dimensions in the convolution kernel is 1, the stride of the first directional dimension of the two directional dimensions is 1, the size of the second directional dimension in the input data is divisible by the stride of the second directional dimension, and the padding of the second directional dimension is 0. In this case, the two adjacent directional dimensions can be merged, thereby reducing one directional dimension and reducing the dimensionality of the three-dimensional convolution operator to a two-dimensional convolution operator.

[0057] For example, the dimension of the input data of a three-dimensional convolution operator is [N, D, H, W, C], where two adjacent directional dimensions can be H and W or D and H.

[0058] For two adjacent directional dimensions H and W, if kh=1, kw=1, stride_h=1, W%stride_w=0 (% denotes remainder operation), pad_w=0, then the directional dimensions H and W can be merged (H is folded to W), thereby reducing the dimensionality of the three-dimensional convolution operator to a two-dimensional convolution operator. The dimension of the input data of the reduced two-dimensional convolution operator is [N, D, HW, C], the dimension of the convolution kernel is [Co, kd, 1, Ci], the stride of the convolution kernel is [stride_d, stride_w], the padding is [pad_d1, pad_d2, pad_h1*W, pad_h2*W], and the expansion is [dilation_d, 1].

[0059] It can be understood that when kh=1 and kw=1, the convolution kernel is [Co,kd,1,1,Ci], and the convolution kernel changes from a three-dimensional cube of kd*kh*kw to a line segment of length kd, which can also be regarded as a two-dimensional plane of kd*1. The one-dimensional convolution kernel kd performs convolution on the five-dimensional input [N,D,H,W,C], and there is no coupling relationship in the H and W dimensions, which is equivalent to the two-dimensional convolution kernel kd*1 performing convolution on the four-dimensional input [N,D,H*W,C]. Considering parameters such as stride, padding, and dilation at the same time, when stride_h=1, in order to ensure a unified stride after merging the H dimension and the W dimension, it is necessary to satisfy W%stride_w=0. Dilation represents the interval on the convolution kernel. Since kh=1 and kw=1, dilation_h and dilation_w must also be 1. If pad_w is not equal to 0, the data in the H and W directions will be discontinuous, so pad_w=0. After merging the H dimension and the W dimension, pad_h becomes pad_h*W after being folded to W, so the pad of the two-dimensional convolution operator is [pad_d1,pad_d2,pad_h1*W,pad_h2*W].

[0060] Similarly, for two adjacent directional dimensions D and H, if kd=1, kh=1, stride_d=1, H%strided_h=0, pad_h=0, then the directional dimensions D and H can be merged (D is folded to H), thereby reducing the dimensionality of the three-dimensional convolution operator to a two-dimensional convolution operator. The dimension of the input data of the reduced two-dimensional convolution operator is [N, DH, W, C], the dimension of the convolution kernel is [Co, 1, kw, Ci], the stride of the convolution kernel is [stride_h, stride_w], the padding is [pad_d1*H, pad_d2*H, pad_w1, pad_w2], and the expansion is [1, dilation_w].

[0061] Second sub-condition: the size of the first directional dimension in the convolution kernel is 1, the stride of the directional dimension is 1, and the padding of the directional dimension is 0. In this case, the directional dimension can be merged with the previous batch processing volume dimension N, thereby reducing one directional dimension and reducing the dimensionality of the three-dimensional convolution operator to a two-dimensional convolution operator.

[0062] For example, the dimension of the input data of the three-dimensional convolution operator is [N, D, H, W, C]. The first directional dimension is D. If kd = 1, stride_d = 1, and pad_d = 0 are satisfied at the same time, the directional dimension D can be merged with the batch processing dimension N, thereby reducing the dimensionality of the three-dimensional convolution operator to a two-dimensional convolution operator. The dimension of the input data of the two-dimensional convolution operator after dimensionality reduction is [ND, H, W, C], the dimension of the convolution kernel is [Co, kh, kw, Ci], the stride of the convolution kernel is [stride_h, stride_w], the padding is [pad_h1, pad_h2, pad_w1, pad_w2], and the dilation is [dilation_h, dilation_w].

[0063] The third sub-condition: the size of any one of the first two directional dimensions in the input data is 1, and the step size of the directional dimension is 1. In this case, the directional dimension can be deleted, thereby reducing one directional dimension and reducing the dimensionality of the three-dimensional convolution operator to a two-dimensional convolution operator.

[0064] For example, the dimension of the input data of a 3D convolution operator is [N, D, H, W, C], where the dimensions of the first two directions are D and H.

[0065] When the size of the directional dimension D is 1, that is, D = 1, if stride_d = 1 is satisfied at the same time, the directional dimension D can be deleted, and the three-dimensional convolution operator can be reduced to a two-dimensional convolution operator. The dimension of the input data of the reduced two-dimensional convolution operator is [N, H, W, C], the dimension of the convolution kernel is [Co, kh, kw, Ci], the stride of the convolution kernel is [stride_h, stride_w], the padding is [pad_d1*(H+pad_h1+pad_h2)+pad_h1,pad_d2*(H+pad_h1+pad_h2)+pad_h2,pad_w1,pad_w2], and the expansion is [dilation_h, dilation_w].

[0066] It should be noted that in the above embodiment, since D=1, kd=1 is inevitable. Since pad_d may not be equal to 0, stride_d may be greater than 1. When D=1 and stride_d=1, the original input data [N, D, H, W, C] degenerates to [N, H, W, C]. Because kd=1, dilation_d=1 is inevitable. If pad_d1 and pad_d2 are not 0, it can be regarded as merging pad_d1 and pad_d2 into the H dimension, so pad_h1 becomes pad_d1*(H+pad_h1+pad_h2)+padh1, and pad_h2 becomes pad_d2*(H+pad_h1+pad_h2)+pad_h2.

[0067] Similarly, when the size of the directional dimension H is 1, that is, H = 1, if stride_h = 1 is satisfied at the same time, the directional dimension H can be deleted, thereby reducing the dimensionality of the three-dimensional convolution operator to a two-dimensional convolution operator. The dimension of the input data of the reduced two-dimensional convolution operator is [N, D, W, C], the dimension of the convolution kernel is [Co, kd, kw, Ci], the stride of the convolution kernel is [stride_d, stride_w], and the padding is [pad_d1, pad_d2, pad_h1*(W+pad_w1+pad_w2)+pad_w1, pad_h2*(W+pad_w1+pad_w2)+pad_w2].

[0068] Fourth sub-condition: the size of the last directional dimension in the input data is 1, and the padding of the directional dimension is 0. In this case, the directional dimension can be deleted, thereby reducing one directional dimension and reducing the dimensionality of the three-dimensional convolution operator to a two-dimensional convolution operator.

[0069] For example, the dimension of the input data of the three-dimensional convolution operator is [N, D, H, W, C]. The last directional dimension is W. If W = 1 and pad_w = 0 are satisfied at the same time, the directional dimension W can be deleted, thereby reducing the dimension of the three-dimensional convolution operator to a two-dimensional convolution operator. The dimension of the input data of the two-dimensional convolution operator after dimensionality reduction is [N, D, H, C], the dimension of the convolution kernel is [Co, kd, kw, Ci], the stride of the convolution kernel is [stride_d, stride_h], and the padding is [pad_d1, pad_d2, pad_h1, pad_h2].

[0070] In the above embodiment, since W=1 and pad_w=0, it can be concluded that kw=1, stride_w=1, and dilation_w=1.

[0071] According to some embodiments, the three-dimensional operator may be a three-dimensional pooling operator. The processing core of the three-dimensional pooling operator is a pooling core. When the first parameter of the three-dimensional pooling operator satisfies any of the following sub-conditions, the three-dimensional pooling operator may be converted into a two-dimensional pooling operator.

[0072] The fifth sub-condition: the size of the two adjacent directional dimensions in the pooling kernel is 1, the step size of the first directional dimension of the two directional dimensions is 1, the size of the second directional dimension in the input data is divisible by the step size of the second directional dimension, and the padding of the second directional dimension is 0. In this case, the two adjacent directional dimensions can be merged, thereby reducing one directional dimension and reducing the dimensionality of the three-dimensional pooling operator to a two-dimensional pooling operator.

[0073] For example, the dimension of the input data of the three-dimensional pooling operator is [N, D, H, W, C], where two adjacent directional dimensions can be H and W or D and H.

[0074] For two adjacent directional dimensions H and W, if kh=1, kw=1, stride_h=1, W%stride_w=0, pad_w=0 are satisfied at the same time, the directional dimensions H and W can be merged (H is folded to W), thereby reducing the dimensionality of the 3D pooling operator to a 2D pooling operator. The dimension of the input data of the 2D pooling operator after dimensionality reduction is [N, D, HW, C], the dimension of the pooling kernel is [kd, 1], the stride of the pooling kernel is [stride_d, stride_w], and the padding is [pad_d1, pad_d2, pad_h1*W, pad_h2*W].

[0075] Similarly, for two adjacent directional dimensions D and H, if kd=1, kh=1, stride_d=1, H%strided_h=0, pad_h=0 are satisfied at the same time, the directional dimensions D and H can be merged (D is folded to H), thereby reducing the dimensionality of the 3D pooling operator to a 2D pooling operator. The dimension of the input data of the 2D pooling operator after dimensionality reduction is [N, DH, W, C], the dimension of the pooling kernel is [1, kw], the stride of the pooling kernel is [stride_h, stride_w], and the padding is [pad_d1*H, pad_d2*H, pad_w1, pad_w2].

[0076] The sixth sub-condition: the size of the first directional dimension in the pooling kernel is 1, the stride of the directional dimension is 1, and the padding of the directional dimension is 0. In this case, the directional dimension can be merged with the previous batch processing dimension N, thereby reducing one directional dimension and reducing the dimensionality of the three-dimensional convolution operator to a two-dimensional convolution operator.

[0077] For example, the dimension of the input data of the 3D pooling operator is [N, D, H, W, C]. The first directional dimension is D. If kd = 1, stride_d = 1, and pad_d = 0 are satisfied at the same time, the directional dimension D can be merged with the batch processing dimension N, thereby reducing the dimensionality of the 3D pooling operator to a 2D pooling operator. The dimension of the input data of the 2D pooling operator after dimensionality reduction is [ND, H, W, C], the dimension of the pooling kernel is [kh, kw], the stride of the pooling kernel is [stride_h, stride_w], and the padding is [pad_h1, pad_h2, pad_w1, pad_w2].

[0078] Seventh sub-condition: the size of any one of the first two directional dimensions in the input data is 1, and the step size of the directional dimension is 1. In this case, the directional dimension can be deleted, thereby reducing one directional dimension and reducing the dimensionality of the three-dimensional pooling operator to a two-dimensional pooling operator.

[0079] For example, the dimension of the input data of the 3D pooling operator is [N, D, H, W, C], where the dimensions of the first two directions are D and H.

[0080] When the size of the directional dimension D is 1, that is, D = 1, if stride_d = 1 is satisfied at the same time, the directional dimension D can be deleted, and the 3D pooling operator can be reduced to a 2D pooling operator. The dimension of the input data of the 2D pooling operator after dimensionality reduction is [N, H, W, C], the dimension of the pooling kernel is [kh, kw], the stride of the pooling kernel is [stride_h, stride_w], and the padding is [pad_d1*(H+pad_h1+pad_h2)+pad_h1,pad_d2*(H+pad_h1+pad_h2)+pad_h2,pad_w1,pad_w2].

[0081] Similarly, when the size of the directional dimension H is 1, that is, H = 1, if stride_h = 1 is satisfied at the same time, the directional dimension H can be deleted, thereby reducing the dimensionality of the 3D pooling operator to a 2D pooling operator. The dimension of the input data of the 2D pooling operator after dimensionality reduction is [N, D, W, C], the dimension of the pooling kernel is [kd, kw], the stride of the pooling kernel is [stride_d, stride_w], and the padding is [pad_d1, pad_d2, pad_h1*(W+pad_w1+pad_w2)+pad_w1, pad_h2*(W+pad_w1+pad_w2)+pad_w2].

[0082] Eighth sub-condition: the size of the last directional dimension in the input data is 1, and the padding of the directional dimension is 0. In this case, the directional dimension can be deleted, thereby reducing one directional dimension and reducing the dimensionality of the three-dimensional pooling operator to a two-dimensional pooling operator.

[0083] For example, the dimension of the input data of the 3D pooling operator is [N, D, H, W, C]. The last directional dimension is W. If W = 1 and pad_w = 0 are satisfied at the same time, the directional dimension W can be deleted, thereby reducing the dimensionality of the 3D pooling operator to a 2D pooling operator. The dimension of the input data of the 2D pooling operator after dimensionality reduction is [N, D, H, C], the dimension of the pooling kernel is [kd, kh], the stride of the pooling kernel is [stride_d, stride_h], and the padding is [pad_d1, pad_d2, pad_h1, pad_h2].

[0084] It can be understood that in step S130, for each sub-condition described above, the first parameter of the three-dimensional operator can be converted into a second parameter for describing the two-dimensional data processing process, thereby obtaining a corresponding two-dimensional operator.

[0085] For example, for a three-dimensional convolution operator, according to the first to fourth sub-conditions above, the first parameter of the three-dimensional convolution operator (including the dimension of the input data, the dimension of the three-dimensional convolution kernel, the step size of the three-dimensional convolution kernel, padding and dilation) can be converted into the second parameter of the two-dimensional convolution operator (the dimension of the input data, the dimension of the two-dimensional convolution kernel, the step size of the two-dimensional convolution kernel, padding and dilation). For a three-dimensional pooling operator, according to the fifth to eighth sub-conditions above, the first parameter of the three-dimensional pooling operator (including the dimension of the input data, the dimension of the three-dimensional convolution kernel, the step size of the three-dimensional convolution kernel and padding) can be converted into the second parameter of the two-dimensional pooling operator (the dimension of the input data, the dimension of the two-dimensional convolution kernel, the step size of the two-dimensional convolution kernel and padding).

[0086] Corresponding to the second parameter of the two-dimensional operator obtained in step S130, in step S140, based on the second parameter, the function corresponding to the two-dimensional operator in the acceleration library can be called to enable the target artificial intelligence chip to execute the three-dimensional operator. When calling the function corresponding to the two-dimensional operator, the second parameter is used as the input parameter of the function. Since the two-dimensional operator is obtained by reducing the dimension of the three-dimensional operator, the calculation process of the two-dimensional operator is equivalent to the calculation process of the original three-dimensional operator. By calling the function of the two-dimensional operator, the target artificial intelligence chip can execute the two-dimensional operator and obtain the execution result of the two-dimensional operator. The execution result of the two-dimensional operator is the execution result of the original three-dimensional operator.

[0087] According to some embodiments, method 100 further includes step S150. In step S150, in response to the first parameter of the three-dimensional operator not satisfying a preset condition, a function corresponding to the three-dimensional operator in the acceleration library is called to enable the target artificial intelligence chip to execute the three-dimensional operator.

[0088] According to the above embodiment, if the first parameter of the three-dimensional operator does not meet the preset condition, the dimensionality reduction is not performed on it, and the function corresponding to the three-dimensional operator is directly called to enable the target artificial intelligence chip to execute the three-dimensional operator.

[0089] It should be noted that if the first parameter of the three-dimensional operator does not meet the preset conditions, the dimensionality reduction is not performed on it. This is not because the three-dimensional operator cannot be objectively reduced to a two-dimensional operator, but because the computational complexity of reducing its dimensionality in this case is high, and it requires more computing resources and computing time, so it is difficult to bring about an improvement in computing performance. For example, if the first parameter of the three-dimensional operator does not meet the preset conditions, it can also be reduced to a two-dimensional operator through a certain dimensionality reduction algorithm. However, the complexity of the dimensionality reduction algorithm itself is high, and it will consume a lot of computing resources and computing time. Therefore, even if the dimensionality reduction of the three-dimensional operator is completed according to the dimensionality reduction algorithm, it is still impossible to improve the computational efficiency of the target artificial intelligence chip for the three-dimensional convolutional neural network, which limits the application of the target artificial intelligence chip in the field of video classification.

[0090] According to some embodiments, before executing the forward calculation of the three-dimensional convolutional neural network, the qualified three-dimensional operators in the three-dimensional convolutional neural network can be reduced to two-dimensional operators through graph optimization to obtain an optimized three-dimensional convolutional neural network.

[0091] Figure 2 A schematic diagram is shown of reducing the dimension of a three-dimensional convolution operator to a two-dimensional convolution operator in the graph optimization stage according to some embodiments of the present disclosure.

[0092] like Figure 2 As shown, the three-dimensional convolutional neural network 210 includes a three-dimensional convolution operator (conv3d) 211, a bias operator (bias_add) 212, an activation operator (e.g., ReLU, Rectified Linear Unit) 213, a three-dimensional convolution operator (conv3d) 214, a bias operator (bias_add) 215 and an activation operator (e.g., ReLU) 216.

[0093] The three-dimensional convolution operators 211 and 214 meet the preset conditions of the embodiment of the present disclosure (the first to fourth sub-conditions mentioned above), so the two can be reduced in dimension respectively to obtain the three-dimensional convolution neural network 220. Specifically, the three-dimensional convolution operator 211 is converted into a reshape operator (reshape) 2111, a two-dimensional convolution operator (conv2d) 2112, and a reshape operator (reshape) 2113. Among them, the reshape operator 2111 is used to reshape the input data of the three-dimensional convolution operator 211 so that it conforms to the format of the input data of the two-dimensional convolution operator 2112. The reshape operator 2113 is used to reshape the result data of the two-dimensional convolution operator 2112 into the format of the result data of the three-dimensional convolution operator. Similarly, the three-dimensional convolution operator 214 is converted into a reshape operator (reshape) 2141, a two-dimensional convolution operator (conv2d) 2142, and a reshape operator (reshape) 2143.

[0094] It can be understood that the bias operator and the activation operator are operators that are insensitive to the directional dimension. The operator is insensitive to the directional dimension, which means that the processing performed by the operator on the input data is independent of the directional dimension of the input data. For the bias operator and the activation operator, no matter what the values ​​of the various directional dimensions (i.e., D, H, and W) of the input data are, the processing performed is to add a bias or activate each data therein. The operator is sensitive to the directional dimension, which means that the processing performed by the operator on the input data is related to the directional dimension of the input data. For example, the convolution operator can set different movement parameters for different directional dimensions, thereby performing different processing on different directional dimensions; the aggregation operator (reduce) can process data in a certain directional dimension, such as accumulating, maximizing, averaging, etc. the data in a certain directional dimension.

[0095] According to some embodiments, for operators that are insensitive to the directional dimension and are located after the 3D convolution operator, these operators can be moved before the second reshaping operator after the dimensionality reduction. For example, see Figure 2 In the three-dimensional convolutional neural network 220, the bias operator 212 and the activation operator 213 that are insensitive to the directional dimension can be moved before the reshaping operator 2113, and the bias operator 215 and the activation operator 216 that are insensitive to the directional dimension can be moved before the reshaping operator 2143 to obtain the three-dimensional convolutional neural network 230.

[0096] According to some embodiments, two adjacent reshaping operators may be merged, and the dimension of the input data of the merged reshaping operator is the same as the dimension of the input data of the first reshaping operator, and the dimension of the output data of the merged reshaping operator is the same as the dimension of the output data of the second reshaping operator. For example, see Figure 2In the three-dimensional convolutional neural network 230, adjacent reshaping operators 2113 and 2141 can be merged to obtain a reshaping operator 242. The dimension of the input data of the reshaping operator 242 is the same as the dimension of the input data of the reshaping operator 2113, and the dimension of the output data of the reshaping operator 242 is the same as the dimension of the output data of the reshaping operator 2141. Further, the two-dimensional convolution operator 2112, the bias operator 212 and the activation operator 213 are represented as a fused two-dimensional convolution operator (fused_conv2d) 241, and the two-dimensional convolution operator 2142, the bias operator 215 and the activation operator 216 are represented as a fused two-dimensional convolution operator (fused_conv2d) 243, thereby obtaining the three-dimensional convolutional neural network 240 after graph optimization.

[0097] Further, according to some embodiments, in two adjacent reshaping operators, if the dimension of the input data of the first reshaping operator is the same as the dimension of the output data of the second reshaping operator, the two reshaping operators can be eliminated, that is, the two reshaping operators are removed from the three-dimensional convolutional neural network. For example, see Figure 2 In the three-dimensional convolutional neural network 230, if the dimension of the input data of the reshaping operator 2113 is the same as the dimension of the output data of the reshaping operator 2141, the reshaping operator 2113 and the reshaping operator 2141 can be eliminated. In this way, the three-dimensional convolutional neural network 240 after graph optimization only includes the reshaping operator 2111, the fused two-dimensional convolution operator 241, the fused two-dimensional convolution operator 243 and the reshaping operator 2143, and no longer includes the reshaping operator 242.

[0098] Figure 3 A schematic diagram is shown of reducing the dimension of a three-dimensional pooling operator to a two-dimensional pooling operator in the graph optimization stage according to some embodiments of the present disclosure.

[0099] like Figure 3 As shown, the three-dimensional convolutional neural network 310 includes a prior operator (pre_ops) 311, a three-dimensional pooling operator 312, and a back operator (back_ops) 313. It can be understood that there can be multiple prior operators 311, including convolution operators, activation operators, pooling operators, etc. There can also be multiple back operators 313, including, for example, batch normalization operators, convolution operators, activation operators, pooling operators, etc.

[0100] The three-dimensional pooling operator 312 meets the preset conditions of the embodiment of the present disclosure (the fifth to eighth sub-conditions mentioned above), so it can be reduced in dimension to obtain a three-dimensional convolutional neural network 320. Specifically, Figure 3 As shown, the three-dimensional pooling operator 312 is converted into a reshaping operator 3121 , a two-dimensional pooling operator 3122 , and a reshaping operator 3123 .

[0101] Figure 4FIG. 4 shows a structural block diagram of a data processing device 400 based on an artificial intelligence chip according to an embodiment of the present disclosure. Figure 4 As shown, the apparatus 400 includes an input module 410 , an acquisition module 420 and an execution module 430 .

[0102] The input module 410 is configured to input the data to be processed into the three-dimensional convolutional neural network. The three-dimensional convolutional neural network includes multiple three-dimensional operators, each of which has a first parameter for describing the three-dimensional data processing process of the three-dimensional operator.

[0103] The acquisition module 420 is configured to acquire an acceleration library of a target artificial intelligence chip for executing a data processing process of a three-dimensional convolutional neural network. The acceleration library includes a plurality of functions, each of which corresponds to an operator in a neural network and is used to allocate hardware resources to the operator so that the target artificial intelligence chip executes the operator.

[0104] The execution module 430 is configured to: for any three-dimensional operator among multiple three-dimensional operators: in response to a first parameter of the three-dimensional operator satisfying a preset condition, convert the three-dimensional operator into a two-dimensional operator; and call a function corresponding to the two-dimensional operator in the acceleration library so that the target artificial intelligence chip executes the three-dimensional operator.

[0105] According to an embodiment of the present disclosure, in the forward calculation process of a three-dimensional convolutional neural network, a three-dimensional operator that meets preset conditions is reduced to a two-dimensional operator, and then the execution of the three-dimensional operator is realized by calling a fully optimized and more efficient processing function of the two-dimensional operator in the acceleration library, thereby improving the computing efficiency of the AI ​​chip for the three-dimensional convolutional neural network.

[0106] Since the three-dimensional convolutional neural network of the embodiment of the present disclosure is used for video classification, the embodiment of the present disclosure further improves the video classification efficiency of the AI ​​chip.

[0107] According to some embodiments, the first parameter includes dimension parameters of the input data of the three-dimensional operator, dimension parameters of the processing core used by the three-dimensional operator, and the step size and padding of the processing core in each direction dimension, and the preset condition includes: the dimension parameters of the input data, the dimension parameters of the processing core, the step size and padding of the processing core in each direction dimension enable one direction dimension in the input data to be deleted or merged with an adjacent dimension, and enable one direction dimension in the processing core to be deleted or merged with an adjacent direction dimension.

[0108] According to some embodiments, the three-dimensional operator includes a three-dimensional convolution operator, the processing core includes a convolution kernel, and the preset condition includes any of the following sub-conditions: the first sub-condition: the sizes of two adjacent directional dimensions in the convolution kernel are both 1, the step size of the preceding directional dimension of the two directional dimensions is 1, the size of the succeeding directional dimension of the two directional dimensions in the input data is divisible by the step size of the succeeding directional dimension, and the padding of the succeeding directional dimension is 0; the second sub-condition: the size of the first directional dimension in the convolution kernel is 1, the step size of the directional dimension is 1, and the padding of the directional dimension is 0; the third sub-condition: the size of any one of the first two directional dimensions in the input data is 1, and the step size of the directional dimension is 1; the fourth sub-condition: the size of the last directional dimension in the input data is 1, and the padding of the directional dimension is 0.

[0109] According to some embodiments, the three-dimensional operator includes a three-dimensional pooling operator, the processing core includes a pooling core, and the preset condition includes any of the following sub-conditions: the fifth sub-condition: the sizes of two adjacent directional dimensions in the pooling core are both 1, the step size of the preceding directional dimension of the two directional dimensions is 1, the size of the succeeding directional dimension of the two directional dimensions in the input data is divisible by the step size of the succeeding directional dimension, and the padding of the succeeding directional dimension is 0; the sixth sub-condition: the size of the first directional dimension in the pooling core is 1, the step size of the directional dimension is 1, and the padding of the directional dimension is 0; the seventh sub-condition: the size of any one of the first two directional dimensions in the input data is 1, and the step size of the directional dimension is 1; the eighth sub-condition: the size of the last directional dimension in the input data is 1, and the padding of the directional dimension is 0.

[0110] According to some embodiments, the execution module is further configured to: convert the first parameter into a second parameter for describing a two-dimensional data processing process to obtain the two-dimensional operator; and based on the second parameter, call the function corresponding to the two-dimensional operator in the acceleration library so that the target artificial intelligence chip executes the three-dimensional operator.

[0111] According to some embodiments, the execution module is further configured to: in response to the first parameter of the three-dimensional operator not satisfying the preset condition, call the function corresponding to the three-dimensional operator in the acceleration library so that the target artificial intelligence chip executes the three-dimensional operator.

[0112] According to some embodiments, the three-dimensional convolutional neural network is used for video classification, and the data to be processed is a video to be classified.

[0113] According to some embodiments, the three-dimensional operator includes a three-dimensional convolution operator and a three-dimensional pooling operator, and the dimensional parameters of the input data of the three-dimensional convolution operator or the three-dimensional pooling operator include: the number of videos to be classified, the number of channels corresponding to each video, and the depth, height and width of the feature map of each channel.

[0114] It should be understood that Figure 4 The various modules of the apparatus 400 shown in FIG. Figure 1 Therefore, the operations, features and advantages of the various steps described above for the method 100 are also applicable to the various modules of the device 400. For the sake of brevity, some operations, features and advantages are not repeated here.

[0115] According to another aspect of the present disclosure, there is also provided an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor, the memory storing instructions executable by the at least one processor, the instructions being executed by the at least one processor so that the at least one processor executes the data processing method 100 based on the artificial intelligence chip.

[0116] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is also provided. The computer instructions are used to enable the computer to execute the above-mentioned data processing method 100 based on the artificial intelligence chip.

[0117] According to another aspect of the present disclosure, a computer program product is also provided, including computer program instructions, which implement the above-mentioned artificial intelligence chip-based data processing method 100 when executed by a processor.

[0118] refer to Figure 5 , a block diagram of an electronic device 500 that can be used as a server or client of the present disclosure will now be described, which is an example of a hardware device that can be applied to various aspects of the present disclosure. The electronic device is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0119] like Figure 5As shown, the device 500 includes a computing unit 501, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 502 or a computer program loaded from a storage unit 508 into a random access memory (RAM) 503. In the RAM 503, various programs and data required for the operation of the device 500 can also be stored. The computing unit 501, the ROM 502, and the RAM 503 are connected to each other via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0120] A plurality of components in the device 500 are connected to the I / O interface 505, including: an input unit 506, an output unit 507, a storage unit 508, and a communication unit 509. The input unit 506 may be any type of device capable of inputting information to the device 500, the input unit 506 may receive input digital or character information, and generate key signal inputs related to user settings and / or function control of the electronic device, and may include but is not limited to a mouse, a keyboard, a touch screen, a track pad, a track ball, a joystick, a microphone, and / or a remote controller. The output unit 507 may be any type of device capable of presenting information, and may include but is not limited to a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. The storage unit 508 may include but is not limited to a disk, an optical disk. The communication unit 509 allows the device 500 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks, and may include but is not limited to a modem, a network card, an infrared communication device, a wireless communication transceiver, and / or a chipset, such as a Bluetooth device, an 802.5 device, a Wi-Fi device, a WiMAX device, a cellular communication device, and / or the like.

[0121] The computing unit 501 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 501 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 501 performs the various methods and processes described above, such as the data processing method 100 based on the artificial intelligence chip. For example, in some embodiments, the data processing method 100 based on the artificial intelligence chip may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 508. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 500 via the ROM 502 and / or the communication unit 509. When the computer program is loaded into the RAM 503 and executed by the computing unit 501, one or more steps of the data processing method 100 based on the artificial intelligence chip described above may be performed. Alternatively, in other embodiments, the computing unit 501 may be configured to execute the artificial intelligence chip-based data processing method 100 in any other appropriate manner (e.g., by means of firmware).

[0122] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0123] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0124] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0125] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0126] The systems and techniques described herein can be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), the Internet, and a blockchain network.

[0127] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0128] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be performed in parallel, sequentially or in a different order, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document is not limited here.

[0129] Although the embodiments or examples of the present disclosure have been described with reference to the accompanying drawings, it should be understood that the above-mentioned methods, systems and devices are merely exemplary embodiments or examples, and the scope of the present disclosure is not limited by these embodiments or examples. Various elements in the embodiments or examples may be omitted or may be replaced by their equivalent elements. In addition, each step may be performed in an order different from that described in the present disclosure. Further, the various elements in the embodiments or examples may be combined in various ways. It is important that with the evolution of technology, many elements described herein may be replaced by equivalent elements that appear after the present disclosure.

Claims

1. A data processing method based on artificial intelligence chip, include: Inputting the data to be processed into a three-dimensional convolutional neural network, wherein the three-dimensional convolutional neural network includes a plurality of three-dimensional operators, and each of the plurality of three-dimensional operators has a first parameter for describing a three-dimensional data processing process of the three-dimensional operator; Obtaining an acceleration library of a target artificial intelligence chip for executing the data processing process of the three-dimensional convolutional neural network, wherein the acceleration library includes a plurality of functions, each of the plurality of functions corresponds to an operator in the neural network and is used to allocate hardware resources to the operator so that the target artificial intelligence chip executes the operator; and For any three-dimensional operator among the multiple three-dimensional operators: In response to a first parameter of the three-dimensional operator satisfying a preset condition, converting the three-dimensional operator into a two-dimensional operator; and Call the function corresponding to the two-dimensional operator in the acceleration library to enable the target artificial intelligence chip to execute the three-dimensional operator.

2. The method according to claim 1, in, The first parameter includes the dimension parameter of the input data of the three-dimensional operator, the dimension parameter of the processing core used by the three-dimensional operator, and the step size and padding of the processing core in each direction dimension. The preset conditions include: The dimensional parameters of the input data, the dimensional parameters of the processing core, and the stride and padding of the processing core in each directional dimension enable a directional dimension in the input data to be deleted or merged with an adjacent dimension, and enable a directional dimension in the processing core to be deleted or merged with an adjacent directional dimension.

3. The method according to claim 2, in, The three-dimensional operator includes a three-dimensional convolution operator, the processing core includes a convolution core, and the preset condition includes any of the following sub-conditions: The first sub-condition: the sizes of the two adjacent directional dimensions in the convolution kernel are both 1, the stride of the first directional dimension of the two directional dimensions is 1, the size of the second directional dimension of the two directional dimensions in the input data is divisible by the stride of the second directional dimension, and the padding of the second directional dimension is 0; Second sub-condition: the size of the first directional dimension in the convolution kernel is 1, the step size of the directional dimension is 1, and the padding of the directional dimension is 0; The third sub-condition: the size of any one of the first two directional dimensions in the input data is 1, and the step length of the directional dimension is 1; Fourth sub-condition: the size of the last directional dimension in the input data is 1, and the padding of the directional dimension is 0.

4. The method according to claim 2, in, The three-dimensional operator includes a three-dimensional pooling operator, the processing core includes a pooling core, and the preset condition includes any of the following sub-conditions: Fifth sub-condition: the sizes of two adjacent directional dimensions in the pooling kernel are both 1, the stride of the preceding directional dimension of the two directional dimensions is 1, the size of the succeeding directional dimension of the two directional dimensions in the input data is divisible by the stride of the succeeding directional dimension, and the padding of the succeeding directional dimension is 0; Subcondition 6: The size of the first directional dimension in the pooling kernel is 1, the stride of the directional dimension is 1, and the padding of the directional dimension is 0; Seventh sub-condition: the size of any one of the first two directional dimensions in the input data is 1, and the step length of the directional dimension is 1; Eighth sub-condition: the size of the last directional dimension in the input data is 1, and the padding of the directional dimension is 0.

5. The method according to any one of claims 1 to 4, in, The converting the three-dimensional operator into a two-dimensional operator comprises: converting the first parameter into a second parameter for describing a two-dimensional data processing process to obtain the two-dimensional operator, And wherein, calling the function corresponding to the two-dimensional operator in the acceleration library so that the target artificial intelligence chip executes the three-dimensional operator includes: based on the second parameter, calling the function corresponding to the two-dimensional operator in the acceleration library so that the target artificial intelligence chip executes the three-dimensional operator.

6. The method according to any one of claims 1 to 5, further comprising: include: In response to the first parameter of the three-dimensional operator not satisfying the preset condition, calling the function corresponding to the three-dimensional operator in the acceleration library to enable the target artificial intelligence chip to execute the three-dimensional operator.

7. The method according to any one of claims 1 to 6, in, The three-dimensional convolutional neural network is used for video classification, and the data to be processed is the video to be classified.

8. The method according to claim 7, in, The three-dimensional operator includes a three-dimensional convolution operator and a three-dimensional pooling operator. The dimension parameters of the input data of the three-dimensional convolution operator or the three-dimensional pooling operator include: the number of videos to be classified, the number of channels corresponding to each video, and the depth, height and width of the feature map of each channel.

9. A data processing device based on artificial intelligence chip, include: An input module is configured to input the data to be processed into a three-dimensional convolutional neural network, wherein the three-dimensional convolutional neural network includes a plurality of three-dimensional operators, and each of the plurality of three-dimensional operators has a first parameter for describing a three-dimensional data processing process of the three-dimensional operator; an acquisition module configured to acquire an acceleration library of a target artificial intelligence chip for executing the data processing process of the three-dimensional convolutional neural network, wherein the acceleration library includes a plurality of functions, each of the plurality of functions corresponds to an operator in the neural network and is used to allocate hardware resources to the operator so that the target artificial intelligence chip executes the operator; and An execution module is configured to, for any three-dimensional operator of the multiple three-dimensional operators: In response to a first parameter of the three-dimensional operator satisfying a preset condition, converting the three-dimensional operator into a two-dimensional operator; and Call the function corresponding to the two-dimensional operator in the acceleration library to enable the target artificial intelligence chip to execute the three-dimensional operator.

10. The device according to claim 9, in, The first parameter includes the dimension parameter of the input data of the three-dimensional operator, the dimension parameter of the processing core used by the three-dimensional operator, and the step size and padding of the processing core in each direction dimension. The preset conditions include: The dimensional parameters of the input data, the dimensional parameters of the processing core, and the stride and padding of the processing core in each directional dimension enable a directional dimension in the input data to be deleted or merged with an adjacent dimension, and enable a directional dimension in the processing core to be deleted or merged with an adjacent directional dimension.

11. The device according to claim 10, in, The three-dimensional operator includes a three-dimensional convolution operator, the processing core includes a convolution core, and the preset condition includes any of the following sub-conditions: The first sub-condition: the sizes of the two adjacent directional dimensions in the convolution kernel are both 1, the stride of the first directional dimension of the two directional dimensions is 1, the size of the second directional dimension of the two directional dimensions in the input data is divisible by the stride of the second directional dimension, and the padding of the second directional dimension is 0; Second sub-condition: the size of the first directional dimension in the convolution kernel is 1, the step size of the directional dimension is 1, and the padding of the directional dimension is 0; The third sub-condition: the size of any one of the first two directional dimensions in the input data is 1, and the step length of the directional dimension is 1; Fourth sub-condition: the size of the last directional dimension in the input data is 1, and the padding of the directional dimension is 0.

12. The device according to claim 10, in, The three-dimensional operator includes a three-dimensional pooling operator, the processing core includes a pooling core, and the preset condition includes any of the following sub-conditions: Fifth sub-condition: the sizes of two adjacent directional dimensions in the pooling kernel are both 1, the stride of the preceding directional dimension of the two directional dimensions is 1, the size of the succeeding directional dimension of the two directional dimensions in the input data is divisible by the stride of the succeeding directional dimension, and the padding of the succeeding directional dimension is 0; Subcondition 6: The size of the first directional dimension in the pooling kernel is 1, the stride of the directional dimension is 1, and the padding of the directional dimension is 0; Seventh sub-condition: the size of any one of the first two directional dimensions in the input data is 1, and the step length of the directional dimension is 1; Eighth sub-condition: the size of the last directional dimension in the input data is 1, and the padding of the directional dimension is 0.

13. The device according to any one of claims 9 to 12, in, The execution module is further configured to: converting the first parameter into a second parameter for describing a two-dimensional data processing process to obtain the two-dimensional operator; and Based on the second parameter, the function corresponding to the two-dimensional operator in the acceleration library is called to enable the target artificial intelligence chip to execute the three-dimensional operator.

14. The device according to any one of claims 9 to 13, in, The execution module is further configured to: In response to the first parameter of the three-dimensional operator not satisfying the preset condition, calling the function corresponding to the three-dimensional operator in the acceleration library to enable the target artificial intelligence chip to execute the three-dimensional operator.

15. The device according to any one of claims 9 to 14, in, The three-dimensional convolutional neural network is used for video classification, and the data to be processed is the video to be classified.

16. The device according to claim 15, in, The three-dimensional operator includes a three-dimensional convolution operator and a three-dimensional pooling operator. The dimension parameters of the input data of the three-dimensional convolution operator or the three-dimensional pooling operator include: the number of videos to be classified, the number of channels corresponding to each video, and the depth, height and width of the feature map of each channel.

17. An electronic device, include: at least one processor; as well as a memory communicatively coupled to the at least one processor, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 8.

18. A non-transitory computer-readable storage medium storing computer instructions, in, The computer instructions are used to make a computer execute the method according to any one of claims 1-8.

19. A computer program product comprising computer program instructions, in, The computer program instructions, when executed by a processor, implement the method according to any one of claims 1-8.

Images