Tensor serialization method and related product

Abstract

The invention discloses a tensor serialization method and a related product, and generally relates to the technical field of artificial intelligence. The method comprises the following steps: responding to a view-class operator subgraph with a to-be-processed tensor, and obtaining an original tensor of the view-class operator subgraph; and according to the original tensor, carrying out serialization on the tensor to be processed. According to the method, the device, the chip, the system and the computer readable storage medium provided by the disclosure, the to-be-processed tensor has the view type operator subgraph in response to the to-be-processed tensor, the view type operator subgraph can be utilized to restore the to-be-processed tensor into the original tensor, and then continuous processing is performed based on the original tensor. Furthermore, in some embodiments, a high-performance computing library kernel operator corresponding to the view-class operator subgraph is used for carrying out data handling, so that the continuity of the tensor to be processed is realized at a relatively high speed, and the running speed of the neural network model is further improved.

Description

Technical Field

[0001] The present disclosure generally relates to the field of artificial intelligence technology. More specifically, the present disclosure relates to a method for continuousizing tensors and related products. Background Art

[0002] Neural network models can automatically learn complex patterns and features from large amounts of data, thus having powerful reasoning capabilities. In recent years, the application of neural network models has been widely popularized at an unprecedented rate, showing strong influence and application value in various fields represented by computer vision and natural language processing.

[0003] As a bridge between the theory and practical application of neural network models, the deep learning framework provides developers with a series of convenient interfaces, modules and tools to easily build, train, optimize and deploy neural networks. The operation of neural network models involves a large number of multidimensional array operations. In the deep learning framework, these multidimensional arrays are usually modeled as tensors. However, there are view operators such as transpose and slice in the framework. Such operators can easily cause tensors to be discontinuous in memory. In the prior art, the continuation of tensors is achieved by moving memory data element by element, but this continuation method is inefficient, resulting in the problem of slow operation of neural network models.

[0004] In view of this, there is an urgent need to provide a solution for continuation of tensors so as to achieve continuation of tensors at a faster speed, thereby improving the running speed of the neural network model. Summary of the invention

[0005] In order to at least solve the problems described in the above background technology section, the present disclosure proposes the following technical solutions and multiple embodiments thereof.

[0006] In a first aspect, the present disclosure proposes a method for continuating tensors, the method comprising: in response to a tensor to be processed having a view-class operator subgraph, obtaining the original tensor of the view-class operator subgraph; and continuating the tensor to be processed according to the original tensor.

[0007] In a second aspect, the present disclosure further discloses a device, comprising: a processor configured to execute program instructions; and a memory configured to store the program instructions, so that when the program instructions are loaded and executed by the processor, the device executes the method according to the first aspect.

[0008] In a third aspect, the present disclosure further discloses a computer-readable storage medium, wherein program instructions are stored, wherein the program instructions are suitable for being loaded by a processor and executing the method according to the first aspect.

[0009] In a fourth aspect, the present disclosure discloses a chip, comprising: an input terminal configured to receive a tensor; and a processor configured to execute the method according to the first aspect.

[0010] In a fifth aspect, the present disclosure discloses an integrated circuit, which includes the chip described in the fourth aspect.

[0011] The method, device, chip, system and computer-readable storage medium proposed in the present disclosure can use the view-class operator subgraph to restore the tensor to be processed to the original tensor in response to the tensor to be processed having a view-class operator subgraph, and then perform serial processing based on the original tensor. Furthermore, in some embodiments, the kernel operator of the high-performance computing library corresponding to the view-class operator subgraph is used to carry out data transfer, so as to realize the serialization of the tensor to be processed at a faster speed, thereby improving the running speed of the neural network model. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] By reading the detailed description below with reference to the accompanying drawings, the above and other objects, features and advantages of the exemplary embodiments of the present disclosure will become readily understood. In the accompanying drawings, several embodiments of the present disclosure are shown in an exemplary and non-limiting manner, and the same or corresponding reference numerals represent the same or corresponding parts, wherein:

[0013] Figure 1 A schematic diagram showing a Tensor data structure of some embodiments of the present disclosure;

[0014] Figure 2 This is an exemplary diagram of a slice-type operator leading to an interval-type discontinuous Tensor;

[0015] Figure 3 This is an exemplary diagram of an extended class operator leading to an overlapping class of discontinuous Tensors;

[0016] Figure 4 This is an exemplary schematic diagram of a transposition operator leading to a non-interval-non-overlapping discontinuous Tensor;

[0017] Figure 5 An exemplary method flow chart of tensor continuation according to some embodiments of the present disclosure is shown;

[0018] Figure 6 A flowchart of an exemplary method for serializing a tensor to be processed according to an original tensor according to some embodiments of the present disclosure is shown;

[0019] Figure 7 A schematic diagram showing an exemplary method of serializing Tensors by data transfer according to some embodiments of the present disclosure is shown;

[0020] Figure 8 A flowchart of an exemplary method for determining a second target rearrangement class operator for serializing a tensor to be processed according to some embodiments of the present disclosure is shown;

[0021] Fig. 9 An exemplary schematic diagram of operator fusion in some embodiments of the present disclosure is shown;

[0022] Fig.10 The block diagram shows the hardware configuration of a computing device that can implement the method of serializing a tensor according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0023] The following will be combined with the drawings in the embodiments of the present disclosure to clearly and completely describe the technical solutions in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present disclosure.

[0024] It should be understood that the terms "include" and "comprising" used in the specification and claims of the present disclosure indicate the presence of described features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or collections thereof.

[0025] It should also be understood that the terms used in this disclosure are only for the purpose of describing specific embodiments and are not intended to limit the disclosure. As used in this disclosure and claims, the singular forms of "a", "an", and "the" are intended to include the plural forms unless the context clearly indicates otherwise. It should also be further understood that the term "and / or" used in this disclosure and claims refers to any combination of one or more of the associated listed items and all possible combinations, including these combinations.

[0026] As used in this specification and claims, the term "if" may be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" may be interpreted as meaning "upon determination" or "in response to determining" or "upon detection of [described condition or event]" or "in response to detecting [described condition or event]," depending on the context.

[0027] The specific implementation of the present disclosure is described in detail below with reference to the accompanying drawings.

[0028] The neural network model is composed of layers arranged in a certain order. Different layers have different functions in the neural network model. Through the connection and data transmission between layers, multiple layers work together. Take the Convolutional Neural Network (CNN) in the field of image processing as an example. When CNN performs image classification tasks, the original image is input into the network in the form of a pixel matrix from the input layer. The neurons in the input layer receive the pixel signals and pass them to the subsequent convolution layer. The convolution layer uses different convolution kernels to slide on the image, perform convolution operations with the pixels in the local area of ​​the image, extract local features to obtain feature maps, and pass the feature maps to the subsequent pooling layer. The pooling layer usually uses maximum pooling or average pooling to downsample the feature map. Subsequently, the convolution and pooling operations are repeated to progressively build more complex and abstract high-level features based on low-level features. Finally, the extracted features are flattened and input into the fully connected layer. The fully connected layer outputs the category probability distribution through functions such as softmax. The category with the highest probability is the image category predicted by the network.

[0029] The various data processed by the neural network model often exist in the form of multidimensional arrays. For example, in the image processing field mentioned above, for a grayscale image with a resolution of 6*6, it can be represented by a two-dimensional array, where the first dimension of the array represents the height and the second dimension represents the width, and the shape of the two-dimensional array is (6, 6); and for a RGB three-channel color image with a resolution of 6*6, it can be represented by a three-dimensional array, where the first dimension of the array represents the channel, the second dimension represents the height, and the third dimension represents the width, and the shape of the three-dimensional array is (3, 6, 6).

[0030] Not only the input data, but also the structure of the neural network often exists in the form of multi-dimensional arrays. For example, the convolution layer mentioned above uses the convolution kernel to slide on the image, which actually forms a convolution connection, and the weight of the convolution kernel is the connection weight. For a convolution kernel with a shape of 3*3, it can be represented by a two-dimensional array with a shape of (3, 3); and for three such convolution kernels, they can be represented by a three-dimensional array with a shape of (3, 3, 3). As a result, a large number of multi-dimensional array operations are formed in the neural network model. For example, the convolution operation mentioned above, for an input RGB three-channel color image with a resolution of 6*6, it can be A 3,6,j Represents image data. When a 3*3 convolution kernel is used with a step size of 1 for convolution operation, B 3,3 Represents the convolution kernel data, then the convolution operation can be expressed as Thus, a 4*4 feature map is output; for the same input image, when three 3*3 convolution kernels are used with a step size of 1 for convolution operation, B3,3,3 Represents the convolution kernel data, then the convolution operation can be expressed as This outputs three 4*4 feature maps.

[0031] As a bridge between the theory and practical application of neural network models, deep learning frameworks provide developers with a series of convenient application programming interfaces (APIs), modules, and tools to easily build, train, optimize, and deploy neural networks. Currently, some common deep learning algorithm frameworks include: TensorFlow, PyTorch, Keras, Caffe, MXNet, and Theano. In deep learning frameworks, multidimensional array data is usually modeled as tensors.

[0032] Figure 1 A schematic diagram of a tensor data structure in some embodiments of the present disclosure is shown. Figure 1 As shown in the figure, the common properties of tensors include data type, shape, stride, and storage, where: the data type specifies the size of the space occupied by the elements in the multidimensional array when stored in the memory and the storage precision; storage is used to store multidimensional array data, which corresponds to a piece of memory space in the memory (for example, memory DRAM and cache RAM). Although the multidimensional array has multiple dimensions, the layout of the memory is always one-dimensional, so storage is a one-dimensional array; size corresponds to the "shape" mentioned above, which describes the length of the multidimensional array of the Tensor in each dimension; and stride refers to the step required to jump from one element to the next element in the specified dimension.

[0033] The operators in the deep learning framework need to process a large amount of multi-dimensional array data. In this scenario, batch processing of array elements is more helpful to fully utilize the computing power of multi-core processors than processing array elements one by one, thereby improving the calculation speed of the operator. Different operators in the deep learning framework can realize batch processing of multi-dimensional array data in different data units. For example, the convolution calculation of image data mentioned above can calculate the pixel values ​​of different channels of the same pixel point of the input image with the convolution kernel at the same time, thereby improving the calculation speed of the convolution operation. In other words, the convolution calculation can realize batch processing of image data with the multi-channel pixel values ​​of the same pixel point as the data unit; there is also the pooling operation mentioned above, which can realize batch processing of image data with the pixel matrix corresponding to different channels of the input image as the data unit.

[0034] In the deep learning framework, Tensor can adopt a certain continuous memory layout to store multidimensional arrays, so that operators can read data units continuously to achieve batch processing. Among them, continuous memory layout includes but is not limited to: contiguous_format, channels_last and channels_last_3d. Table 1 shows the relationship between size and stride for different continuous memory layouts. Then, if Tensor stores multidimensional arrays according to the continuous memory layout, it can meet the requirements of the corresponding operator to continuously obtain data in specific data units, thereby realizing batch processing of multidimensional array data. At this time, Tensor is called continuous.

[0035]

[0036] Table 1

[0037] Corresponding to the various calculations required by neural networks, the deep learning framework provides a rich API that encapsulates a series of operators. Among these operators, there is a series of view operators, which only change the size and stride attributes of Tensor when operating on Tensor, without making corresponding changes to storage. View operators include but are not limited to: transpose, permute, select, chunk, narrow, slice, expand, view, etc. As shown in Table 2, view operators can be divided into three categories according to their impact on the size of multidimensional array data.

[0038] Scale Change Representative Operator Operator class name The scale remains unchanged transpose, permute, view Rearrangement Operator Downsizing select, chunk, narrow, slice Slice Operators Scaling expand Extended class operator

[0039] Table 2

[0040] Since the view operator changes the size and stride attributes of the Tensor, it is easy to make a continuous Tensor discontinuous.

[0041] Figure 2 This is an exemplary diagram of a slice operator leading to an interval discontinuous Tensor. Figure 2 The three-dimensional Tensor X shown in (a) has a size of (2, 2, 3). Figure 2 (c) shows that its storage is a one-dimensional vector [1,2,3,4,5,6,7,8,9,10,11,12]; a slice-like operation is performed on X in the third dimension. For example, in PyTorch, let Y = X[:,:,:2], as Figure 2As shown in (b), the size of Y is (2, 2, 2), but the storage is still Figure 2 (c) As shown in Figure 1, [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12] is maintained. According to the storage and size of Y, its stride can be determined to be (6, 3, 1). Then for Tensor Y, its size and stride no longer satisfy the relationship between size and stride under the above continuous memory layout, and thus become discontinuous. In particular, in Tensor Y, element 3 in the storage does not exist, that is, there is a gap between elements 2 and 4 in the memory storage corresponding to Tensor Y. Similarly, there are gaps between elements 5 and 7, between elements 8 and 10, and after element 11. In this case, Tensor Y is said to have gap-type discontinuity.

[0042] Figure 3 This is an exemplary diagram of the extended class operator leading to overlapping class discontinuous Tensors. Figure 3 The initial one-dimensional Tensor X shown in (a) has a size of (6). Figure 3 (c) shows that its storage is a one-dimensional vector [1,2,3,4,5,6]; an expansion operation is performed on X. For example, in PyTorch, let Y = X.expand(2,6), as Figure 3 As shown in (b), the size of Y is (2, 6), but the storage is still Figure 3 (c) As shown in Figure 1, keep [1,2,3,4,5,6]. According to the storage and size of Y, its stride can be determined to be (0,1). Then for Tensor Y, its size and stride no longer satisfy the relationship between size and stride under the above continuous memory layout, and thus become discontinuous. In particular, in Tensor Y, Y[0,0] corresponds to element 1 in storage, and Y[1,0] also corresponds to element 1 in storage. That is to say, the memory storage locations of Y[0,0] and Y[1,0] in Tensor Y overlap. Similarly, the memory storage locations of Y[0,1] and Y[1,1], ..., Y[0,5] and Y[1,5] also overlap. At this time, Tensor Y is said to have overlapping discontinuity.

[0043] Figure 4 This is an exemplary diagram of a transpose operator leading to a non-interval-non-overlapping discontinuous Tensor. Figure 4 The initial three-dimensional Tensor X shown in (a) has a size of (2, 2, 3). Figure 4(c) shows that its storage is a one-dimensional vector [1,2,3,4,5,6,7,8,9,10,11,12]; perform a rearrangement operation on X. For example, in PyTorch, let Y = X.permute(0,2,1), as Figure 4 As shown in (b), the size of Y is (2, 3, 2), but the storage is still Figure 4 (c) As shown in Figure 1, [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12] is maintained. According to the storage and size of Y, its stride can be determined to be (6, 1, 3). Then for Tensor Y, its size and stride no longer satisfy the relationship between size and stride under the above continuous memory layout, and thus become discontinuous. However, for Tensor Y, the number of its elements is the same as storage, so there is no interval or overlap discontinuity. In this case, Tensor Y is said to have non-interval-non-overlap discontinuity.

[0044] If the Tensor is not continuous, that is, the storage method of its multidimensional array does not meet the continuous memory layout, then it is necessary to process the Tensor by calling a specific operator (for example, the copy operator) to move the memory data element by element, so as to meet the operator's requirement of continuously taking data in specific data units. In this case, the running speed of the neural network model will be significantly reduced. In view of this, the present disclosure provides a method for continuousizing tensors, thereby achieving the continuation of discontinuous tensors at a faster speed, thereby improving the running speed of the neural network model.

[0045] Figure 5 An exemplary flow chart of a method for tensor continuation according to an embodiment of the present disclosure is shown. Figure 5 As shown, in step 501: in response to the tensor to be processed having a view class operator subgraph, the original tensor of the view class operator subgraph is obtained.

[0046] In the deep learning framework, a computational graph is a directed graph consisting of nodes and edges, where nodes correspond to operators. Tensors are passed between nodes in the computational graph. Each time a tensor passes through a node, it is calculated as the input of the node's operation, and the result of the calculation flows along the node's output edge to the following node. A subgraph refers to a part of the computational graph that has a specific function. Furthermore, a view operator subgraph refers to a computational substructure formed by combining multiple view operators in a neural network model.

[0047] In some embodiments, a method for obtaining a view class operator subgraph of a tensor to be processed may include: for the tensor to be processed, traversing the operators that process the tensor, and when the operator encountered during the traversal is a view class operator, extracting the operator to construct a view class operator subgraph.

[0048] In some embodiments, the view class operator subgraph includes: description information of the view class operator associated with the tensor to be processed in the computation graph. It is understandable that each operator has attributes to identify relevant information when the operator is executed. Common attributes include: operator name, operator type, operator input data, operator output data, and operation parameters. In some embodiments, the description information of the operator in the view class operator subgraph includes at least one of the above common attributes.

[0049] It can be understood that the view class operator subgraph is a set of view class operators that operate on the current Tensor, so the original tensor of the view class operator subgraph is the original tensor input into the view class operator subgraph. In some embodiments, obtaining the original tensor of the view class operator subgraph includes: obtaining the original tensor according to the description information of the input data of the view class operator subgraph.

[0050] In step 502: the tensor to be processed is serialized according to the original tensor.

[0051] Figure 6 An exemplary flow chart of a method for serializing a to-be-processed tensor according to the original tensor in an embodiment of the present disclosure is shown, as shown in FIG. Figure 6 As shown, in some embodiments, the process of continuating the tensor to be processed according to the original tensor includes: step 601, in response to the original tensor being non-continuous and satisfying non-overlapping and non-interval conditions, determining a first target rearrangement class operator for continuating the original tensor, and determining the continuation result of the original tensor according to the first target rearrangement class operator.

[0052] It can be understood that the original tensor is discontinuous and satisfies non-overlapping and non-interval, which means that the continuity of the original tensor corresponds to the non-interval-non-overlapping type of discontinuity described above. For the original tensor, by changing its size and stride, it can satisfy the relationship followed by size and stride under the above-mentioned continuous memory layout, so that it becomes continuous. As an optional method, the original tensor is processed using a rearrangement operator to obtain a continuous result. For example, for Figure 4 For the discontinuous Tensor Y shown in the figure, let Z = Y.permute(0, 2, 1), then the size of Z is (2, 2, 3), but the storage is still Figure 4(c) As shown in Figure 2, [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12] is maintained, then Z is essentially the same Tensor as X, which is continuous.

[0053] Step 602: Expand the view class operator subgraph according to the first target rearrangement class operator to obtain an extended view class operator subgraph.

[0054] Step 601 determines the first target reordering operator for the discontinuous original tensor and obtains the continuous result according to the first target reordering operator, and when the first target reordering operator satisfies certain conditions (explained below), the first target reordering operator is called on the continuous result to restore the original tensor. Exemplarily, for the continuous result Z in the above text, the same reordering operator is called again, i.e., Z.permute(0, 2, 1), and the discontinuous tensor Tensor Y can be restored. In this way, the obtained extended view operator subgraph can be understood as performing a series of view operator operations on the continuous result.

[0055] In some embodiments, the first target rearrangement class operator is added to the front end of the view class operator subgraph, thereby obtaining an extended view class operator subgraph.

[0056] Step 603: for the serialization result, call the kernel corresponding to the extended view class operator subgraph to perform data transfer processing to serialize the tensor to be processed.

[0057] The result of the continuation is a continuous Tensor. After being processed by a series of view operators, the continuous Tensor becomes discontinuous. The reason is that the view operator only changes the size and stride of the Tensor but does not change the storage of the Tensor, so that the size and stride of the processed Tensor no longer satisfy the relationship between the size and stride under the above continuous memory layout. Then, if the corresponding kernel is called based on the information recorded in the extended view operator subgraph, while changing the size and stride of the Tensor, the corresponding data transfer is performed on its storage, so that the size and stride of the modified Tensor can maintain the relationship under the continuous memory layout.

[0058] Figure 7 FIG. 2 shows an exemplary schematic diagram of a method for converting a Tensor into a continuous state by data transfer in an embodiment of the present disclosure. Figure 7 As shown, for Figure 4As shown in the figure, the permute operator converts a continuous Tensor into a discontinuous Tensor. While permuting (0, 2, 1) Tensor X, the corresponding data is moved to its storage. The size of the resulting Tensor Y is (2, 3, 2), and the storage is as follows: Figure 7 (d) shows [1,4,2,5,3,6,7,10,8,11,9,12]. Based on the storage and size of Y, its stride can be determined to be (6, 2, 1). For Tensor Y, its size and stride satisfy the relationship followed in the above continuous memory layout, that is, they are continuous.

[0059] The high-performance computing library in deep learning provides highly optimized kernel operators for the common operators that deep neural networks rely on, thereby improving the training and reasoning speed of deep learning models. For each operator in the view class subgraph, you can select the corresponding kernel from the high-performance computing library to perform the corresponding data handling. For example, for the permute operator, you can call the transpose kernel in the high-performance computing library (such as CNNL) to implement the data rearrangement function. For another example, for the expand operator, you can call the expand kernel in CNNL to implement the data expansion function.

[0060] Therefore, based on the order of the extended view class operator subgraphs or the optimized order of the extended view class operator subgraphs, by traversing each view class operator and performing kernel calls, the tensor data can be converted from a discontinuous memory state to a continuous memory state. Compared with the prior art of moving memory data element by element, calling the kernel to move data can greatly shorten the processing time and improve memory access efficiency. As mentioned above, through the first target rearrangement class operator, the original tensor can be converted to a continuous result, and the continuous result can also be converted to the original tensor. In some embodiments, determining the first target rearrangement class operator for continuousizing the original tensor includes: step 701, sorting the stride of the original tensor; wherein, in response to the stride i =stride j And size i =1 and size j ≥2, set stride i Relative stride j The direction is consistent with the descending direction of stride.

[0061] According to Table 1, for tensors with a memory layout of contiguous_format, their stride is decreasing, so the stride and size of the original tensor can be rearranged based on this feature. When sorting the stride of the original tensor, it can be sorted in descending order or ascending order, and the disclosed embodiment has no limitation in this regard. However, the inventors of the present disclosure have found that: for a tensor with a memory layout of contiguous_format, if the size i =1 and i≠0, it will result in stride i-1 =stride i , so for stride i =stride j In the case of a non-overlapping and non-interval Tensor A, it is very important to determine the front and back positions of these two elements. As an example, suppose there is a discontinuous Tensor A that satisfies non-overlapping and non-interval, with a size of (2, 3, 1, 3) and a stride of (9, 1, 3, 3). Then directly sort its stride in descending order, and the sort result is (9, 3, 3, 1). Then the sort index can be (0, 2, 3, 1) or (0, 3, 2, 1):

[0062] If (0, 2, 3, 1) is used as the parameter of the first target reordering operator, and A.permute(0,2,3,1) is executed on Tensor A, Tensor B is obtained. The size of B is (2, 1, 3, 3) and the stride is (9, 3, 3, 1). If B.permute(0,2,3,1) is executed on Tensor B, the size of the obtained Tensor is (2, 3, 3, 1), which is obviously inconsistent with Tensor A. That is to say, although the sorting index (0, 2, 3, 1) is used as the parameter of the first target reordering operator, the discontinuous Tensor A can be converted into a continuous Tensor B, but Tensor B cannot be restored to Tensor A.

[0063] However, if (0, 3, 2, 1) is used as the parameter of the first target reordering operator, and A.permute(0, 3, 2, 1) is executed on Tensor A, Tensor B is obtained. The size of B is (2, 3, 1, 3) and the stride is (9, 3, 3, 1). When B.permute(0, 3, 2, 1) is executed on Tensor B, the size of the obtained Tensor is (2, 3, 1, 3) and the stride is (9, 1, 3, 3), which is consistent with TensorA. That is to say, using the sorting index (0, 3, 2, 1) as the parameter of the first target reordering operator, we can convert the discontinuous Tensor A into a continuous Tensor B, or restore Tensor B to Tensor A.

[0064] Therefore, in this step, we also set: i =stride j And size i =1 and size j ≥2, set stride i Relative stride j The direction is consistent with the descending direction of stride. For the above example, according to this setting:

[0065] If the stride of TensorA is sorted in descending order, considering stride2=stride3=3, size2=1 and size3=3≥2, the direction of stride2 relative to stride3 is set to be consistent with the direction of stride descending, that is, stride2 is placed after stride3, and the sorting index used is (0, 3, 2, 1).

[0066] If the stride of TensorA is sorted in ascending order, considering stride2=stride3=3, size2=1 and size3=3≥2, the direction of stride2 relative to stride3 is set to be consistent with the direction of stride descending, that is, stride2 is placed in front of stride3, and the sorting index used is (1, 2, 3, 0).

[0067] Therefore, the "setting stride i Relative stride j The direction is consistent with the descending direction of stride" specifically means:

[0068] If it is sorted in ascending order, you need to change stride i Place on stride j in front of;

[0069] If it is sorted in descending order, you need to change stride i Place on stride j Behind.

[0070] Step 702: Determine the parameters of the first target rearrangement operator according to the sorting index of the stride. It can be understood from the above description that:

[0071] If the stride is sorted in descending order, the sorting index of the stride is directly used as the parameter of the first target rearrangement operator. For example, in the above text, the stride of TensorA is sorted in descending order to obtain the sorting index (0, 3, 2, 1), and (0, 3, 2, 1) is directly used as the parameter of the first target rearrangement operator.

[0072] If the stride is sorted in ascending order, the sort index of the stride is inverted and used as the parameter of the first target rearrangement operator. For example, in the above text, the stride of TensorA is sorted in ascending order to obtain the sort index (1, 2, 3, 0), then (1, 2, 3, 0) is inverted to obtain (0, 3, 2, 1), and (0, 3, 2, 1) is used as the parameter of the first target rearrangement operator.

[0073] In some embodiments, serializing the tensor to be processed according to the original tensor includes: in response to the original tensor being continuous, calling the kernel corresponding to the view class operator subgraph for the tensor to be processed to perform data transfer processing to serialize the tensor to be processed.

[0074] The previous article has combined Figure 7 Explanation: For the discontinuous Tensor output by the view operator subgraph, the discontinuous tensor can be made continuous by calling the kernel corresponding to the view operator subgraph to perform data transfer processing on the continuous tensor input into the view operator subgraph. I will not go into details here.

[0075] In some embodiments, continuating the tensor to be processed according to the original tensor includes: in response to the original tensor being discontinuous and satisfying overlap or interval, moving elements of the tensor to be processed one by one to continuate the tensor to be processed.

[0076] It is understandable that when the original tensor is discontinuous and overlaps or is spaced, the size of the original tensor is inconsistent with the size of its continuous result, and it is impossible to make it continuous by constructing a rearrangement operator. At this time, the only way to make the processed tensor continuous is to move the elements of the processed tensor one by one. In these embodiments, it is assumed that the size of the processed tensor is (s0, s1, s2, ..., s i ), stride is (y0,y1,y2,…,y i ), then the tensor is at the point (x0,x1,x2,…,x i ) can be calculated according to the following formula (1):

[0077]

[0078] Where dptr is the starting position of the tensor corresponding to the memory storage, dtype is the data type of the tensor, storage offset The offset of the first element of the tensor relative to the starting position of storage.

[0079] In some embodiments, in response to the tensor to be processed having no view operator subgraph and satisfying overlap or interval, the elements of the tensor to be processed are moved one by one to make the tensor to be processed continuous. Similarly, the memory location corresponding to the tensor at any point can be calculated according to formula (1) when moving the elements of the tensor to be processed one by one.

[0080] In some embodiments, in response to the tensor to be processed having no view class operator subgraph and satisfying non-overlapping and non-interval conditions, a second target rearrangement class operator is determined for making the tensor to be processed continuous; and for the tensor to be processed, a kernel corresponding to the second target rearrangement class operator is called to perform data transfer processing to make the tensor to be processed continuous.

[0081] The previous article has introduced in detail the principle of making non-overlapping and non-interval discontinuous tensors continuous through the rearrangement operator, and the principle of making discontinuous tensors continuous by calling the corresponding kernel according to the view operator to carry out data transfer, which will not be repeated here.

[0082] In some embodiments, determining a second target rearrangement operator for serializing the tensor to be processed includes: sorting the strides of the tensor to be processed; and determining a parameter of the second target rearrangement operator according to the sorting index of the stride.

[0083] When sorting the stride of the tensor to be processed, it can be sorted in descending order or ascending order, and the disclosed embodiment has no limitation in this regard. If the stride of the tensor to be processed is sorted in descending order, the sorting index of the stride is directly used as the parameter of the second target rearrangement operator. If the stride of the tensor to be processed is sorted in ascending order, the sorting index of the stride is inverted and then used as the parameter of the first target rearrangement operator. It can be understood that the first target rearrangement operator in the above steps 601-603 not only needs to satisfy the requirement that the original tensor can be converted into a continuous result, but also needs to satisfy the requirement that the continuous result can be converted into the original tensor, while for the second target rearrangement operator, it only needs to satisfy the requirement that the tensor to be processed can be continuous. Therefore, in the confirmation method of the second target rearrangement operator, the stride can be sorted directly without combining size to process different dimensions with the same stride value.

[0084] Figure 8 An exemplary method flow chart of determining a second target rearrangement class operator for serializing the tensor to be processed in some embodiments of the present disclosure is shown. Figure 8 As shown, in some embodiments, determining a second target reordering type operator for serializing the tensor to be processed includes: step 801, determining a first reordering type operator; the first reordering type operator is used to convert the memory layout of the tensor to be processed into a first continuous memory layout.

[0085] As mentioned above, the contiguous memory layout includes but is not limited to contiguous_format, channels_last, and channels_last_3d. The first contiguous memory layout can be any contiguous memory layout. The principle of making non-overlapping and non-interval discontinuous tensors continuous by rearrangement operators has been described in detail above, so I will not repeat it here.

[0086] Step 802, determining a second reordering type operator; the second reordering type operator is used to convert the first continuous memory layout into a second continuous memory layout.

[0087] It can be understood that the second continuous memory layout is a continuous memory layout other than the first continuous memory layout. In particular, when the first continuous memory layout is contiguous_format and the second continuous memory layout is channels_last, for example, converting the first continuous memory layout to the second continuous memory layout only requires exchanging the second dimension and the fourth dimension, and the second rearrangement class operator at this time can be permute(0, 3, 2, 1).

[0088] Step 803, obtaining the second target rearrangement class operator according to the first rearrangement class operator and the second rearrangement class operator. In some embodiments, obtaining the second target rearrangement class operator according to the first rearrangement class operator and the second rearrangement class operator includes: fusing the second rearrangement class operator with the first rearrangement class operator to obtain the second target rearrangement class operator.

[0089] Fig. 9 An exemplary schematic diagram of operator fusion in some embodiments of the present disclosure is shown to better understand operator fusion. Fig. 9 As shown, the first rearrangement operator is permute(0, 2, 3, 1), and the second rearrangement operator is permute(0, 3, 2, 1). The second rearrangement operator is superimposed on the first rearrangement operator, and the second dimension of the first rearrangement operator is exchanged with the fourth dimension to obtain the fusion operator permute(0, 1, 3, 2).

[0090] It is understandable that when the tensor to be processed is a continuous tensor, no continuation processing is required for it. Therefore, the method for continuation of the tensors to be processed in the above-mentioned embodiment of the present disclosure is for the tensors to be processed that are discontinuous. The method for continuation of tensors according to the embodiment of the present disclosure is described in detail with reference to the accompanying drawings. From the above description, it can be seen that the method for continuation of the tensors to be processed in the embodiment of the present disclosure, by extending the view class operator subgraph to optimize the continuation method, improves the IO transfer efficiency. Compared with the prior art of transferring memory data element by element, calling the kernel for data transfer can greatly shorten the processing time and improve the memory access efficiency.

[0091] Fig.10 A block diagram showing the hardware configuration of a computing device 100 that can implement the method of serializing tensors according to an embodiment of the present disclosure. Fig.10 As shown, the computing device 100 may include a processor 101 and a memory 102. Fig.10 In the computing device 100, only the components related to this embodiment are shown. Therefore, it is obvious to those skilled in the art that the computing device 100 may also include Fig.10 The constituent elements shown in are different from the common constituent elements.

[0092] The computing device 100 may correspond to a computing device having various processing functions. For example, the computing device 100 may be implemented as various types of devices, such as a personal computer (PC), a server device, a mobile device, etc.

[0093] The processor 101 can control the operation of the computing device 100. For example, the processor 101 controls the operation of the computing device 100 by executing a program stored in the memory 102 on the computing device 100. The processor 101 can be implemented by a central processing unit (CPU), a graphics processing unit (GPU), an application processor (AP), an artificial intelligence processor chip (IPU), etc. provided in the computing device 100. However, the present disclosure is not limited to this. In the present embodiment, the processor 101 can be implemented in any appropriate manner. For example, the processor 101 can take the form of a computer-readable medium, a logic gate, a switch, an application-specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller, etc., such as a microprocessor or a processor and a computer-readable program code (such as software or firmware) that can be executed by the (micro) processor.

[0094] The memory 102 may be used to store hardware of various data and instructions processed in the computing device 100. For example, the memory 102 may store processed data and data to be processed in the computing device 100. The memory 102 may store data sets that have been processed or to be processed by the processor 101, such as tensors to be processed, etc. In addition, the memory 102 may store applications, drivers, etc. to be driven by the computing device 100. For example: the memory 102 may store various programs related to file classification to be executed by the processor 101, etc. The memory 102 may be a DRAM, but the present disclosure is not limited thereto. The memory 102 may include at least one of a volatile memory or a non-volatile memory. The non-volatile memory may include a read-only memory (ROM), a programmable ROM (PROM), an electrically programmable ROM (EPROM), an electrically erasable programmable ROM (EEPROM), a flash memory, a phase change RAM (PRAM), a magnetic RAM (MRAM), a resistive RAM (RRAM), a ferroelectric RAM (FRAM), etc. The volatile memory may include dynamic RAM (DRAM), static RAM (SRAM), synchronous DRAM (SDRAM), PRAM, MRAM, RRAM, ferroelectric RAM (FeRAM), etc. In an embodiment, the memory 102 may include at least one of a hard disk drive (HDD), a solid state drive (SSD), a high-density flash memory (CF), a secure digital (SD) card, a micro secure digital (Micro-SD) card, a mini secure digital (Mini-SD) card, an extreme digital (xD) card, caches, or a memory stick.

[0095] In summary, the specific functions implemented by the memory 102 and the processor 101 of the computing device 100 provided in the embodiments of this specification can be explained in comparison with the aforementioned embodiments in this specification, and can achieve the technical effects of the aforementioned embodiments, and will not be repeated here.

[0096] The embodiments of the present disclosure also provide a computer-readable storage medium on which a computer program is stored. When the computer program is executed at a processor of a computing device, the processor is caused to execute the methods described in the previous embodiments of the present disclosure.

[0097] Based on the above description, those skilled in the art can also understand that the method of serializing tensors in the embodiment of the present disclosure can also be implemented in a chip. The chip may include an input terminal configured to receive a received tensor; further, the chip may include a processor configured to execute the method described in the previous embodiments of the present disclosure. In addition, given that the chip here can be included in an integrated circuit, the present disclosure also discloses an integrated circuit for serializing tensors, which includes the aforementioned chip.

[0098] It should be noted that, for the purpose of simplicity, the present disclosure describes some methods and embodiments thereof as a series of actions and combinations thereof, but those skilled in the art will appreciate that the scheme of the present disclosure is not limited by the order of the actions described. Therefore, based on the disclosure or teaching of the present disclosure, those skilled in the art will appreciate that some of the steps therein may be performed in other orders or simultaneously. Further, those skilled in the art will appreciate that the embodiments described in the present disclosure may be regarded as optional embodiments, i.e., the actions or modules involved therein may not necessarily be necessary for the implementation of one or some of the schemes of the present disclosure. In addition, depending on the different schemes, the present disclosure also has different focuses on the description of some embodiments. In view of this, those skilled in the art may appreciate the parts that are not described in detail in a certain embodiment of the present disclosure, and may also refer to the relevant descriptions of other embodiments.

Claims

1. A method for continuousizing a tensor, comprising: In response to the tensor to be processed having a view class operator subgraph, obtaining the original tensor of the view class operator subgraph; The tensor to be processed is serialized according to the original tensor.

2. According to the method of claim 1, serializing the tensor to be processed according to the original tensor comprises: In response to the original tensor being non-continuous and satisfying non-overlapping and non-interval conditions, determining a first target rearrangement class operator for serializing the original tensor, and Determine the continuation result of the original tensor according to the first target rearrangement operator; Expanding the view class operator subgraph according to the first target rearrangement class operator to obtain an extended view class operator subgraph; as well as For the serialization result, the kernel corresponding to the extended view class operator subgraph is called to perform data transfer processing to serialize the tensor to be processed.

3. According to the method of claim 2, determining the first target rearrangement class operator for serializing the original tensor comprises: Sort the stride of the original tensor; in which, in response to stride i =stride j And size i =1 and size j ≥2, set stride i Relative stride j The direction is consistent with the descending direction of stride; Determine the parameters of the first target rearrangement operator according to the sorting index of the stride; Here, i and j represent array indices, and size is the shape of the original tensor.

4. According to the method of claim 1, the view class operator subgraph comprises: Description of the view operator in the computation graph associated with the tensor to be processed.

5. According to the method of claim 4, obtaining the original tensor of the view class operator subgraph comprises: The original tensor is obtained according to the description information of the input data of the view class operator subgraph.

6. The method according to claim 1, wherein the step of serializing the tensor to be processed according to the original tensor comprises: In response to the original tensor being non-continuous and satisfying overlap or gap, elements of the tensor to be processed are moved one by one to make the tensor to be processed continuous.

7. The method according to claim 1, wherein the step of serializing the tensor to be processed according to the original tensor comprises: In response to the original tensor being continuous, the kernel corresponding to the view class operator subgraph is called for the tensor to be processed to perform data transfer processing to make the tensor to be processed continuous.

8. The method according to claim 1, further comprising: In response to the tensor to be processed having no view operator subgraph and satisfying overlap or interval, elements of the tensor to be processed are moved one by one to make the tensor to be processed continuous.

9. The method according to claim 1, further comprising: In response to the tensor to be processed having no view operator subgraph and satisfying non-overlapping and non-interval conditions, determining a second target rearrangement operator for making the tensor to be processed continuous; as well as For the tensor to be processed, call the kernel corresponding to the second target rearrangement operator to perform data transfer processing to make the tensor to be processed continuous.

10. According to the method of claim 9, determining the second target rearrangement type operator for serializing the tensor to be processed comprises: Sort the stride of the tensor to be processed; The parameters of the second target rearrangement operator are determined according to the sorting index of the stride.

11. According to the method of claim 9, determining the second target rearrangement type operator for serializing the tensor to be processed comprises: Determine the first rearrangement class operator; The first reordering operator is used to convert the memory layout of the tensor to be processed into a first continuous memory layout; Determining a second reordering type operator; the second reordering type operator is used to convert the first continuous memory layout into a second continuous memory layout; The second target rearrangement operator is obtained according to the first rearrangement operator and the second rearrangement operator.

12. The method according to claim 11, obtaining the second target rearrangement operator according to the first rearrangement operator and the second rearrangement operator comprises: The second rearrangement operator and the first rearrangement operator are fused to obtain the second target rearrangement operator.

13. An apparatus comprising: a processor configured to execute program instructions; as well as A memory configured to store the program instructions, which, when loaded and executed by the processor, causes the apparatus to perform the method according to any one of claims 1 to 12.

14. A computer-readable storage medium storing program instructions, wherein the program instructions are suitable for being loaded by a processor and executing the method according to any one of claims 1 to 12.

15. A chip, comprising: An input terminal, which is configured to receive tensors; A processor configured to execute the method according to any one of claims 1-12.

16. An integrated circuit comprising the chip according to claim 15.

Images