Optimization method, compiler, electronic device and storage medium for artificial intelligence model

Abstract

At least one embodiment of the present disclosure provides an optimization method, compiler, electronic device and storage medium for an artificial intelligence model. The optimization method of the artificial intelligence model includes: compiling the primary representation of at least one first operator into a first intermediate representation; compiling the primary representation of the second operator into a second intermediate representation, wherein the calculation content of the second intermediate representation is rewritten so that the second intermediate representation includes the function of converting the shifted first output data from a non-continuous tensor to a continuous tensor; based on the first intermediate representation and the second intermediate representation, at least one first operator and the second operator are fused to obtain a first fused operator; the second output data of the first fused operator is used as the first input data of the target operator, wherein the first input data is a continuous tensor. The optimization method of the artificial intelligence model improves the computational efficiency of the artificial intelligence model, improves the operating performance of the chip, and expands the support of the deep learning framework for shift operations.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to an optimization method, a compiler, an electronic device, and a storage medium for an artificial intelligence model. Background Art

[0002] In the field of artificial intelligence (AI), deep neural networks (DNNs) have become the foundation of advanced technologies and the core driving force of many applications. With the continuous generalization of deep learning application scenarios, computing tasks performed by deep neural networks need to be deployed on different computing devices and hardware architectures. Actual deployment or training scenarios often have more radical requirements for the performance of deep neural networks, such as the need to customize computing codes based on the characteristics of the deployed hardware. These requirements are difficult to meet in general AI frameworks. Using AI compilers to compile and optimize deep learning computing tasks can meet the hardware performance requirements of actual deployment or training scenarios of computing tasks. Summary of the invention

[0003] At least one embodiment of the present disclosure provides an optimization method for an artificial intelligence model, wherein the artificial intelligence model includes at least one first operator, a second operator and a target operator, wherein the at least one first operator precedes the target operator in a calculation order, and the second operator is used to perform a shift operation on the first output data of the at least one first operator to obtain the shifted first output data, and the shifted first output data is a non-continuous tensor. The optimization method for the artificial intelligence model includes: compiling the primary representation of the at least one first operator into a first intermediate representation; compiling the primary representation of the second operator into a second intermediate representation, wherein in the process of compiling the primary representation of the second operator into the second intermediate representation, rewriting the calculation content of the second intermediate representation so that the second intermediate representation includes the function of converting the shifted first output data from the non-continuous tensor to a continuous tensor; based on the first intermediate representation and the second intermediate representation, fusing the at least one first operator and the second operator to obtain a first fused operator; using the second output data of the first fused operator as the first input data of the target operator, wherein the first input data is the continuous tensor.

[0004] In the optimization method of an artificial intelligence model provided by at least one embodiment of the present disclosure, before compiling the primary representation of the second operator into a second intermediate representation, the optimization method of the artificial intelligence model also includes: registering the second operator into a compilation list of the compiler.

[0005] In the optimization method of the artificial intelligence model provided in at least one embodiment of the present disclosure, the at least one first operator and the second operator are fused based on the first intermediate representation and the second intermediate representation to obtain a first fused operator, including: fusing the first intermediate representation and the second intermediate representation to obtain a fused intermediate representation; using the fused intermediate representation to generate the first fused operator.

[0006] In the optimization method of the artificial intelligence model provided by at least one embodiment of the present disclosure, after the calculation content of the second intermediate representation is rewritten, the calculation content of the second intermediate representation includes: performing continuous processing on the shifted tensor data to convert the shifted first output data from the non-continuous tensor to a continuous tensor.

[0007] In the optimization method of the artificial intelligence model provided by at least one embodiment of the present disclosure, the first output data of the at least one first operator has a first shape attribute, a first step length attribute and a first data attribute, and the first data attribute matches the first shape attribute and the first step length attribute; the shifted first output data has a second shape attribute, a second step length attribute and the first data attribute, and the first data attribute does not match the second shape attribute and the second step length attribute; the second output data of the first fusion operator has the second shape attribute, the second step length attribute and the second data attribute, and the second data attribute matches the second shape attribute and the second step length attribute.

[0008] In the optimization method of the artificial intelligence model provided by at least one embodiment of the present disclosure, after the calculation content of the second intermediate representation is rewritten, the calculation content of the second intermediate representation includes: calculating the second data attribute matching the second shape attribute and the second step attribute according to the second shape attribute and the second step attribute of the shifted first output data; converting the first data attribute of the shifted first output data into the second data attribute to obtain the second output data.

[0009] In the optimization method of the artificial intelligence model provided in at least one embodiment of the present disclosure, after rewriting the calculation content of the second intermediate representation, the calculation content of the second intermediate representation further includes:

[0010] The shifted first output data is read into a temporary variable; and the second output data is written from the temporary variable into a storage device.

[0011] In the optimization method of the artificial intelligence model provided in at least one embodiment of the present disclosure, the first fusion operator is an operator written in the Triton language.

[0012] At least one embodiment of the present disclosure further provides a compiler for optimizing an artificial intelligence model, wherein the artificial intelligence model includes at least one first operator, a second operator and a target operator, wherein the at least one first operator precedes the target operator in a calculation order, and the second operator is used to perform a shift operation on the first output data of the at least one first operator to obtain the shifted first output data, and the shifted first output data is a non-continuous tensor, and the compiler includes: a first compilation tool, configured to compile the primary representation of the at least one first operator into a first intermediate representation; a second compilation tool, configured to compile the primary representation of the second operator into a first intermediate representation; two intermediate representations, wherein, in the process of compiling the primary representation of the second operator into the second intermediate representation, the calculation content of the second intermediate representation is rewritten so that the second intermediate representation includes the function of converting the shifted first output data from the non-continuous tensor to a continuous tensor; a first fusion tool is configured to fuse the at least one first operator and the second operator based on the first intermediate representation and the second intermediate representation to obtain a first fused operator; a first processing tool is configured to use the second output data of the first fused operator as the first input data of the target operator, wherein the first input data is the continuous tensor.

[0013] At least one embodiment of the present disclosure further provides an electronic device, including the compiler provided by any embodiment of the present disclosure, and the electronic device further includes a storage device, wherein the storage device is configured to store the second output data of the first fusion operator.

[0014] At least one embodiment of the present disclosure also provides an electronic device, comprising: a processor; a memory, comprising one or more computer program modules; wherein the one or more computer program modules are stored in the memory and configured to be executed by the processor, and the one or more computer program modules are used to implement the optimization method of the artificial intelligence model provided in any embodiment of the present disclosure.

[0015] At least one embodiment of the present disclosure further provides a storage medium storing non-temporary computer-readable instructions, which, when executed by a computer, implements the optimization method of the artificial intelligence model provided by any embodiment of the present disclosure.

[0016] The method, compiler, electronic device, and storage medium provided by at least one embodiment of the present disclosure can make the input data of the target operator a continuous tensor by adding a function of converting non-continuous tensors into continuous tensors in the second intermediate representation during the compilation process of the operator, and compiling the first operator and the second operator into a first fusion operator, without the need to add other operators for continuous operations or re-develop the target operator, thereby reducing the computational complexity, improving the computational efficiency of the artificial intelligence model, thereby improving the operating performance of the chip, and expanding the support of the deep learning framework for shift operations. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the drawings of the embodiments will be briefly introduced below. Obviously, the drawings in the following description only relate to some embodiments of the present disclosure, but are not intended to limit the present disclosure.

[0018] Figure 1 An exemplary flow chart of an optimization method of an artificial intelligence model provided for at least one embodiment of the present disclosure;

[0019] Figure 2 A schematic diagram of an example of a calculation method in an artificial intelligence model;

[0020] Figure 3 A schematic diagram of an example of an optimization method for an artificial intelligence model provided by at least one embodiment of the present disclosure;

[0021] Figure 4 A schematic diagram of another example of an optimization method for an artificial intelligence model provided in at least one embodiment of the present disclosure;

[0022] Figure 5 A schematic block diagram of a compiler provided for at least one embodiment of the present disclosure;

[0023] Figure 6 A schematic block diagram of an electronic device provided for at least one embodiment of the present disclosure;

[0024] Figure 7 A schematic block diagram of another electronic device provided for at least one embodiment of the present disclosure;

[0025] Figure 8 A schematic block diagram of another electronic device provided for at least one embodiment of the present disclosure; and

[0026] Fig. 9 A schematic diagram of a storage medium provided for at least one embodiment of the present disclosure. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical solution and advantages of the embodiments of the present disclosure clearer, the technical solution of the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings of the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. Based on the described embodiments of the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.

[0028] Unless otherwise defined, the technical terms or scientific terms used in the present disclosure should be understood by people with ordinary skills in the field to which the present disclosure belongs. The "first", "second" and similar words used in the present disclosure do not indicate any order, quantity or importance, but are only used to distinguish different components. Similarly, similar words such as "one", "one" or "the" do not indicate quantity restrictions, but indicate that there is at least one. Similar words such as "include" or "comprise" mean that the elements or objects appearing before the word cover the elements or objects listed after the word and their equivalents, without excluding other elements or objects. Similar words such as "connect" or "connected" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0029] The present disclosure is described below through several specific embodiments. In order to keep the following description of the embodiments of the present disclosure clear and concise, detailed descriptions of known functions and known components may be omitted. When any component of the embodiments of the present disclosure appears in more than one figure, the component is represented by the same or similar reference numeral in each figure.

[0030] Deep learning computing tasks are composed of multiple operators (Operators, Ops). In artificial intelligence models, operators correspond to the computing logic of layers or nodes. For example, the convolution computing task in the convolution layer can be an operator; the weight sum computing task in the fully-connected layer (FC layer) can also be an operator. When implementing a neural network, operators can be implemented in software (for example, through a computer program) or in hardware (for example, through a circuit).

[0031] During the training or reasoning process of an AI model, multiple operators in the AI ​​model need to be called to perform predetermined computing tasks. For example, traditional AI frameworks use manual methods to optimize operators, while AI compilers can automatically generate operators to complete computing tasks during the optimization process. For example, specifically, using an AI compiler for deep learning compilation and optimization is the process of translating and optimizing the current deep learning computing task through one or more layers of intermediate representation (IR), and finally converting it into a machine executable file that can run on the chip. By using an AI compiler, the efficiency of software and hardware collaboration can be improved, further meeting the requirements for hardware performance in actual deployment of computing tasks or training scenarios.

[0032] For example, in some emerging deep learning frameworks (e.g., PyTorch 2.0 framework, etc.), high-performance compilers (e.g., Inductor compiler, etc.) can be used to generate high-performance code for the backend of the accelerator, and high-performance programming languages ​​(e.g., Triton language, etc.) can be used to improve the versatility of the design to achieve high-performance computing. For example, the operator generated by the compiler can be run by calling the processing unit (or processing core) in the artificial intelligence chip, and the processing unit needs to access the corresponding storage device to read and store the tensor data of the operator (e.g., the input data and output data of the operator).

[0033] For example, taking the PyTorch framework as an example, after using the Inductor compiler to compile the operators in the artificial intelligence model, if the generated compilation graph (for example, a machine executable file) includes a shift operation (for example, an as_strided operation), then for the operator that performs a specific calculation, the tensor data of the calculation operator is subjected to a shift operation, and the shape attribute and stride attribute of the tensor data after the shift operation will be modified; since the shift operation only changes the shape attribute and stride attribute of the tensor data, but does not change the tensor data stored in the storage device (for example, no data movement is performed on the video memory), the data attribute of the shifted tensor data will not be modified, resulting in the data attribute of the tensor data not matching its shape attribute and stride attribute, and the shifted tensor data can be called a non-continuous tensor.

[0034] For example, when performing a shift operation on tensor data of a specific computation operator, there are the following two cases:

[0035] (1) If the operator supports non-continuous tensors as input data, the kernel function of the operator will read and write non-continuous tensors according to the stride rule;

[0036] (2) If the calculation operator does not support non-continuous tensors as input data, it is necessary to first call an operator that performs a continuous operation on the non-continuous tensor (for example, call an operator related to the contiguous operation) to convert the non-continuous tensor into a continuous tensor, and then use the converted continuous tensor as the input data of the calculation operator.

[0037] For example, taking the batch matrix multiplication (bmm) operator as an example, in the artificial intelligence model, the output data of the previous operator before the bmm operator in the calculation order is used as the input data of the bmm operator. For example, the previous operator can be the operator Triton_fused written in the Triton language, and its output data is buf43; when the compilation graph of the artificial intelligence model contains a shift operation, buf43 needs to be shifted before being input into the bmm operator. For example, it can be simply represented by pseudo code as as_strided(buf43), and the shifted buf43 (that is, as_strided(buf43)) is a non-continuous tensor; further, the shifted buf43 is used as the input data of the bmm operator to perform the corresponding batch matrix multiplication operation.

[0038] For example, if a computing operator (for example, a BMM operator developed by some manufacturers) supports non-continuous tensors as input data, the shifted tensor data can be used directly to perform the corresponding operation; however, in order to set the computing operator to support non-continuous tensors, complex design and development are required in the chip, which is extremely difficult and costly to implement.

[0039] For example, if a computing operator (for example, certain BMM operators, softmax operators, etc.) does not support non-continuous tensors as input data, it is necessary to call operators related to the contiguous operation to perform continuous processing on the shifted tensor data to convert it into a continuous tensor as the input data of the computing operator; however, calling the contiguous operation requires calling other related operators to read and write data, which increases the computational complexity, reduces the computational efficiency of the artificial intelligence model, and thereby reduces the operating performance of the chip.

[0040] At least one embodiment of the present disclosure provides an optimization method for an artificial intelligence model, the artificial intelligence model includes at least one first operator, a second operator and a target operator, and at least one first operator is before the target operator in the calculation order. The second operator is used to perform a shift operation on the first output data of at least one first operator to obtain the shifted first output data, and the shifted first output data is a non-continuous tensor. The optimization method of the artificial intelligence model includes: compiling the primary representation of at least one first operator into a first intermediate representation; compiling the primary representation of the second operator into a second intermediate representation, wherein, in the process of compiling the primary representation of the second operator into the second intermediate representation, rewriting the calculation content of the second intermediate representation so that the second intermediate representation includes the function of converting the shifted first output data from a non-continuous tensor to a continuous tensor; based on the first intermediate representation and the second intermediate representation, fusing at least one first operator and the second operator to obtain a first fused operator; using the second output data of the first fused operator as the first input data of the target operator, wherein the first input data is a continuous tensor.

[0041] At least one embodiment of the present disclosure also provides at least one compiler, an electronic device, and a storage medium.

[0042] The method, compiler, electronic device, and storage medium provided by at least one embodiment of the present disclosure can make the input data of the target operator a continuous tensor by adding a function of converting non-continuous tensors into continuous tensors in the second intermediate representation during the compilation process of the operator, and compiling the first operator and the second operator into a first fusion operator, without the need to add other operators for continuous operations or re-develop the target operator, thereby reducing the computational complexity, improving the computational efficiency of the artificial intelligence model, thereby improving the operating performance of the chip, and expanding the support of the deep learning framework for shift operations.

[0043] Hereinafter, at least one embodiment of the present disclosure will be described in detail with reference to the accompanying drawings. It should be noted that the same reference numerals in different drawings will be used to refer to the same elements described.

[0044] Figure 1 An exemplary flow chart of a method for optimizing an artificial intelligence model provided for at least one embodiment of the present disclosure.

[0045] For example, Figure 1 As shown, at least one embodiment of the present disclosure provides an optimization method for an artificial intelligence model. For example, the artificial intelligence model includes at least one first operator, a second operator and a target operator, and at least one first operator precedes the target operator in the calculation order. For example, the second operator is used to perform a shift operation on the first output data of at least one first operator to obtain the shifted first output data, and the shifted first output data is a non-continuous tensor.

[0046] For example, Figure 1 As shown, the optimization method of the artificial intelligence model includes the following steps S110~S140.

[0047] Step S110: compile a primary representation of at least one first operator into a first intermediate representation;

[0048] Step S120: compile the primary representation of the second operator into a second intermediate representation;

[0049] Step S130: Based on the first intermediate representation and the second intermediate representation, fuse at least one first operator and the second operator to obtain a first fused operator;

[0050] Step S140: using the second output data of the first fusion operator as the first input data of the target operator.

[0051] For example, in steps S110 and S120, the primary representation of the first operator and the primary representation of the second operator may be layer-level representations (Graph IR), where the layer IR stores the knowledge or information of the compilation process in the graph, and the corresponding operator is described by operating on objects (nodes, edges, lists, and trees) in the graph. For example, the primary representation of the first operator and the primary representation of the second operator may also be selected as other types according to actual application scenarios, and the embodiments of the present disclosure are not limited to this.

[0052] For example, in steps S110 and S120, the primary representation of the first operator may be assembled downward to obtain a first intermediate representation, and the primary representation of the second operator may be assembled downward to obtain a second intermediate representation. For example, the first intermediate representation and the second intermediate representation may be IRs of the next one or more levels of the layer IR, such as a tensor intermediate representation (Tensor IR), a low-level virtual machine intermediate representation (LLVM IR), etc., and the embodiments of the present disclosure are not limited to this.

[0053] For example, the number of first operators may be one or more; when the number of first operators is multiple, in the process of compiling the primary representations of multiple first operators into the first intermediate representation, the fusion of the multiple first operators at the IR level may be achieved, that is, the first intermediate representation may be a fused intermediate representation corresponding to the multiple first operators.

[0054] For example, in step S120, in the process of compiling the primary representation of the second operator into the second intermediate representation, the calculation content of the second intermediate representation is rewritten so that the second intermediate representation includes the function of converting the shifted first output data from a non-continuous tensor to a continuous tensor.

[0055] For example, after the second operator performs a shift operation on the first output data of the first operator, the shifted first output data obtained is a non-continuous tensor; by rewriting the calculation content of the second intermediate representation corresponding to the second operator in step S120, the shifted first output data can be converted from a non-continuous tensor to a continuous tensor for direct use by the target operator.

[0056] It should be noted that the shift operation may be an as_strided operation or other operations for shifting tensor data. The specific operation may be selected according to actual needs, and the embodiments of the present disclosure are not limited to this.

[0057] For example, in step S130, based on the first intermediate representation of the first operator and the second intermediate representation of the second operator, the first operator and the second operator may be fused to obtain a first fused operator. For example, since the second intermediate representation includes a function of converting the shifted first output data from a non-continuous tensor to a continuous tensor, the first fused operator also includes a function of converting the shifted first output data from a non-continuous tensor to a continuous tensor.

[0058] For example, in step S140, since at least one first operator precedes the target operator in the calculation order, if the operator fusion operation of step S130 is not performed, the first output data of at least one first operator can be used as input data of the target operator after the shift operation of the second operator; after executing the operator fusion operation of step S130, since the first fused operator also includes the function of converting the shifted first output data from a non-continuous tensor to a continuous tensor, the second output data output by the first fused operator is a continuous tensor and can be directly used as the first input data by the target operator.

[0059] In some examples, the target operator may be an operator used to perform specific calculations, such as a bmm operator, a softmax operator, etc.; the first operator may be an operator that precedes the target operator in the calculation order in the artificial intelligence model, such as an operator written in the Triton language, or other types of calculation operators; the second operator may be an operator that performs shift operations such as as_strided; the specific types and implementation methods of the target operator, the first operator, and the second operator may be selected according to actual needs, and the embodiments of the present disclosure are not limited to this.

[0060] In some examples, the first fusion operator may be an operator written in the Triton language. Depending on the actual application scenario, the first fusion operator may also be of other types. The specific selection may be based on actual needs, and the embodiments of the present disclosure are not limited to this.

[0061] It should be noted that Figure 1 Steps S110 to S140 may be executed in the Inductor compiler in the PyTorch framework, or may be executed in other types of AI compilers according to actual needs, and the embodiments of the present disclosure are not limited to this.

[0062] Figure 2 A schematic diagram of an example of a calculation method in an artificial intelligence model; Figure 3 A schematic diagram of an example of an optimization method for an artificial intelligence model provided by at least one embodiment of the present disclosure. For example, Figure 3 yes Figure 1 A specific example of the optimization method for the artificial intelligence model shown.

[0063] For example, in Figure 2 and Figure 3 In the example, the artificial intelligence model includes n first operators, second operators and target operators, where n is a positive integer. For example, the n first operators (first operator 1, first operator 2...first operator n) are before the target operator in the calculation order, and the second operator is used to perform a shift operation on the first output data of the n first operators to obtain the shifted first output data, and the shifted first output data is a non-continuous tensor.

[0064] For example, Figure 2 As shown, without executing the optimization method of the artificial intelligence model provided by at least one embodiment of the present disclosure, the second operator performs a shift operation on the first output data of n first operators to obtain shifted first output data, which is a non-continuous tensor; the shifted first output data is used as input data of the target operator.

[0065] For example, when the target operator does not support non-continuous tensors, it is necessary to first call related operators to perform continuous operations on non-continuous tensors (for example, call operators related to contiguous operations) to convert non-continuous tensors into continuous tensors, and use the converted continuous tensors as input data for the target operator. Since calling operations such as contiguous requires calling other related operators to read and write data, the computational complexity increases and the computational efficiency of the artificial intelligence model is reduced. Alternatively, the target operator needs to be redeveloped so that the target operator supports non-continuous tensors as input data, which increases the design and development cost and complexity of the chip.

[0066] For example, Figure 3 As shown, based on Figure 1 The first intermediate representation and the second intermediate representation generated by steps S110 and S120 ( Figure 3 ), execute Figure 1Step S130, fusing the n first operators with the second operators to obtain a first fused operator; further, executing Figure 1 In step S140, the second output data of the first fused operator is used as the first input data of the target operator. For example, since the second intermediate representation includes a function of converting the shifted first output data from a non-continuous tensor to a continuous tensor, the first fused operator also includes a function of converting the shifted first output data from a non-continuous tensor to a continuous tensor; the second output data output by the first fused operator is a continuous tensor and can be directly used by the target operator as the first input data.

[0067] The optimization method of the artificial intelligence model provided by at least one embodiment of the present disclosure does not need to add other operators to perform continuous operations on the shifted first output data, nor does it need to redevelop the target operator when the target operator does not support non-continuous tensors. This reduces the design and development cost and complexity of the chip, reduces the computational complexity, improves the computational efficiency of the artificial intelligence model, thereby improving the operating performance of the chip, and expands the support of the deep learning framework for shift operations.

[0068] In some examples, Figure 1 Before step S120, the optimization method of the artificial intelligence model provided by at least one embodiment of the present disclosure may further include the following step S150.

[0069] Step S150: register the second operator into the compilation list of the compiler.

[0070] For example, in step S150, by registering the second operator in the compilation list of the compiler, the primary representation of the second operator can be assembled downward (lowering) in the compiler to compile it into a second intermediate representation. For example, taking the compiler as an Inductor compiler as an example, the compilation list is a list used for downward assembly in the Inductor compiler (that is, an Inductor lowering list); the compilation list can also be selected as other types of lists according to actual application scenarios, and the embodiments of the present disclosure are not limited to this.

[0071] For example, the compiler may compile the primary representation of the second operator into a second intermediate representation (i.e., execute Figure 1 Step S120), and automatically fuse the first operator and the second operator based on the first intermediate representation corresponding to the first operator and the second intermediate representation to obtain a first fused operator (i.e., perform Figure 1 In step S130), the second output data of the first fusion operator is a continuous tensor.

[0072] In some examples, Figure 1The step S130 may further include the following steps S131~S132.

[0073] Step S131: Fusing the first intermediate representation with the second intermediate representation to obtain a fused intermediate representation;

[0074] Step S132: Generate a first fusion operator using the fused intermediate representation.

[0075] For example, in order to fuse at least one first operator and a second operator, first in step S131, the compiler fuses the first intermediate representation with the second intermediate representation to obtain a fused intermediate representation; according to the internal mechanism of the compiler, the first intermediate representation and the second intermediate representation can be automatically fused, and the intermediate representation of the target operator used for a specific calculation does not participate in the fusion.

[0076] For example, further, in step S132, the compiler can use the fused intermediate representation to generate a first fusion operator; since the second intermediate representation includes the function of converting non-continuous tensors into continuous tensors, the fused intermediate representation also includes the function of converting non-continuous tensors into continuous tensors, and the second output data output by the first fusion operator is a continuous tensor.

[0077] In some examples, for example, after rewriting the computation content of the second intermediate representation, the computation content of the second intermediate representation may include the following step S1201.

[0078] Step S1201: performing a continuous processing on the shifted tensor data to convert the shifted first output data from a non-continuous tensor into a continuous tensor.

[0079] For example, in step S1201, by rewriting the calculation content of the second intermediate representation, the second intermediate representation may include relevant calculation content for performing continuous processing (such as performing contiguous operations) on the shifted tensor data, thereby converting the shifted first output data from a non-continuous tensor to a continuous tensor.

[0080] In some examples, the first output data of at least one first operator has a first shape attribute, a first step length attribute, and a first data attribute, and the first data attribute matches the first shape attribute and the first step length attribute; the shifted first output data has a second shape attribute, a second step length attribute, and the first data attribute, and the first data attribute does not match the second shape attribute and the second step length attribute; the second output data of the first fusion operator has a second shape attribute, a second step length attribute, and a second data attribute, and the second data attribute matches the second shape attribute and the second step length attribute.

[0081] For example, the first data attribute of the first output data of the first operator matches the first shape attribute and the first step length attribute, that is, the first output data itself is continuous. For example, after performing a shift operation on the first output data, the shape attribute and the step length attribute of the shifted first output data will be modified; since the shift operation will not change the data stored in the storage device, the data attribute of the shifted first output data will not be modified; that is, the first data attribute of the shifted first output data does not match the modified second shape attribute and the second step length attribute, so the shifted first output data is discontinuous. For example, by performing Figure 1 In steps S110 to S130, the second data attribute of the second output data of the first fusion operator obtained matches the second shape attribute and the second step attribute, so the second output data is continuous.

[0082] For example, after rewriting the calculation content of the second intermediate representation, the calculation content of the second intermediate representation may include the following steps S121-S122.

[0083] Step S121: calculating a second data attribute that matches the second shape attribute and the second step attribute according to the second shape attribute and the second step attribute of the shifted first output data;

[0084] Step S122: converting the first data attribute of the shifted first output data into the second data attribute to obtain second output data.

[0085] For example, in order to convert the shifted first output data from a non-continuous tensor to a continuous tensor, first in step S121, the second data attribute is calculated based on the second shape attribute and the second step attribute of the shifted first output data, and the second data attribute matches the second shape attribute and the second step attribute.

[0086] For example, further in step S122, the first data attribute of the shifted first output data is converted into a second data attribute to obtain second output data; the second data attribute of the second output data matches the second shape attribute and the second step attribute, so the second output data is continuous.

[0087] For example, the following is a specific example of steps S121-S122:

[0088] First output data: shape1, stride1, data1;

[0089] The first output data after shifting: shape2, stride2, data1;

[0090] Second output data: shape2, stride2, data2.

[0091] For example, in this example, the first data attribute (data1) of the first output data of the first operator matches the first shape attribute (shape1) and the first step length attribute (stride1), that is, the first output data itself is continuous.

[0092] For example, in this example, after performing a shift operation on the first output data, the shape attribute and the step attribute of the shifted first output data will be modified; since the shift operation will not change the data stored in the storage device, the data attributes of the shifted first output data will not be modified; that is, the first data attribute (data1) of the shifted first output data does not match the modified second shape attribute (shape2) and the second step attribute (stide2), so the shifted first output data is discontinuous.

[0093] For example, in this example, in step S121, based on the second shape attribute (shape2) and the second step attribute (stide2) of the shifted first output data, the second data attribute (data2) that matches the second shape attribute (shape2) and the second step attribute (stide2) is calculated; further, in step S122, the first data attribute (data1) of the shifted first output data is converted into the second data attribute (data2), so that the second output data can be obtained.

[0094] For example, in this example, the second data attribute (data2) of the second output data of the first fused operator matches the second shape attribute (shape2) and the second step attribute (stide2), so the second output data is a continuous tensor.

[0095] In some examples, after rewriting the computation content of the second intermediate representation, the computation content of the second intermediate representation may further include the following steps S123 - S124 .

[0096] Step S123: reading the shifted first output data into a temporary variable;

[0097] Step S124: writing the second output data from the temporary variable into the storage device.

[0098] For example, after obtaining the second output data, in step S123, the shifted first output data can be read into a temporary variable on the chip, that is, through step S123, the shifted first output data can be temporarily stored on the chip. For example, on the chip, the compiler can execute steps S121-S122 described above to obtain continuous second output data, which is also temporarily stored on the chip. For example, further, in step S124, the obtained second output data is written from the temporary variable to a storage device (e.g., an external video memory, etc.) as the first input data of the target operator.

[0099] Figure 4 A schematic diagram of another example of an optimization method for an artificial intelligence model provided in at least one embodiment of the present disclosure. For example, Figure 4 yes Figure 1 Another specific example of the optimization method of the artificial intelligence model shown.

[0100] For example, in Figure 4 In the example, the artificial intelligence model includes n first operators, second operators and target operators, where n is a positive integer. For example, the n first operators (first operator 1, first operator 2...first operator n) are before the target operator in the calculation order, and the second operator is used to perform a shift operation on the first output data of the n first operators to obtain the shifted first output data, and the shifted first output data is a non-continuous tensor.

[0101] For example, Figure 4 As shown, in Figure 1 In step S110, the compiler may compile the primary representations of the n first operators into a first intermediate representation; Figure 1 In step S120, the compiler may compile the primary representation of the second operator into a second intermediate representation. For example, in the process of executing step S120, the compiler may rewrite the calculation content of the second intermediate representation so that the second intermediate representation includes a function of converting the shifted first output data from a non-continuous tensor to a continuous tensor.

[0102] For example, Figure 4 As shown, in Figure 1 In step S130, based on the first intermediate representation and the second intermediate representation, the compiler may fuse at least one first operator and the second operator to obtain a first fused operator. For example, specifically, in step S131, the compiler may fuse the first intermediate representation and the second intermediate representation to obtain a fused intermediate representation; in step S132, the compiler may generate the first fused operator using the fused intermediate representation.

[0103] For example, since the second intermediate representation includes the function of converting non-continuous tensors into continuous tensors, the fused intermediate representation also includes the function of converting non-continuous tensors into continuous tensors. The second output data output by the first fusion operator is a continuous tensor, which can be used in Figure 1 Step S140 ( Figure 4 (not shown in the figure) is directly used as the first input data by the target operator; accordingly, the first input data is also a continuous tensor.

[0104] For example, Figure 4 As shown in FIG. 1 , for the generation process of the target operator, the compiler can first compile the primary representation of the target operator into the intermediate representation of the target operator. For example, according to the internal mechanism of the compiler, the intermediate representation of the target operator does not participate in the fusion; further, the compiler can compile the intermediate representation of the target operator into a target operator that can be run on the chip to perform the corresponding operation.

[0105] It should be noted that the compiler can be the Inductor compiler in the PyTorch framework, or it can be selected as other types of AI compilers according to actual needs, and the embodiments of the present disclosure are not limited to this.

[0106] It should be noted that Figure 2~Figure 4 The examples described in the description are only for illustration, and the specific implementation method can also be selected according to actual needs, and the embodiments of the present disclosure are not limited to this.

[0107] At least one embodiment of the present disclosure provides an optimization method for an artificial intelligence model. By adding a function of converting non-continuous tensors into continuous tensors in the second intermediate representation during the compilation of the operator, and compiling the first operator and the second operator into a first fusion operator, the input data of the target operator can be made into a continuous tensor, without the need to add other operators for continuous operations or to redevelop the target operator. This reduces the computational complexity, improves the computational efficiency of the artificial intelligence model, further improves the operating performance of the chip, and expands the support of the deep learning framework for shift operations.

[0108] Figure 5 A schematic block diagram of a compiler provided for at least one embodiment of the present disclosure.

[0109] For example, Figure 5 As shown, at least one embodiment of the present disclosure provides a compiler 200 for optimizing an artificial intelligence model. For example, the artificial intelligence model includes at least one first operator, a second operator, and a target operator, and at least one first operator precedes the target operator in the calculation order. For example, the second operator is used to perform a shift operation on the first output data of at least one first operator to obtain the shifted first output data, and the shifted first output data is a non-continuous tensor.

[0110] For example, Figure 5 As shown, compiler 200 includes a first compilation tool 210 , a second compilation tool 220 , a first fusion tool 230 and a first processing tool 240 .

[0111] For example, the first compilation tool 210 is configured to compile the primary representation of at least one first operator into a first intermediate representation. That is, the first compilation tool 210 may be configured to perform, for example Figure 1 Step S110 is shown.

[0112] For example, the second compilation tool 220 is configured to compile the primary representation of the second operator into the second intermediate representation. For example, in the process of compiling the primary representation of the second operator into the second intermediate representation, the second compilation tool 220 can be configured to rewrite the calculation content of the second intermediate representation so that the second intermediate representation includes the function of converting the shifted first output data from a non-continuous tensor to a continuous tensor. That is, the second compilation tool 220 can be configured to perform, for example Figure 1 Step S120 is shown.

[0113] For example, the first fusion tool 230 is configured to fuse at least one first operator and a second operator based on the first intermediate representation and the second intermediate representation to obtain a first fused operator. That is, the first fusion tool 230 can be configured to perform, for example Figure 1 Step S130 is shown.

[0114] For example, the first processing tool 240 is configured to use the second output data of the first fusion operator as the first input data of the target operator. For example, the first input data is a continuous tensor. That is, the first processing tool 240 can be configured to perform, for example Figure 1 Step S140 is shown.

[0115] In some examples, the compiler 200 further includes a second processing tool (not shown in the figure). For example, before compiling the primary representation of the second operator into the second intermediate representation, the second processing tool is configured to register the second operator into a compilation list of the compiler.

[0116] In some examples, the first fusion tool 230 is further configured to: fuse the first intermediate representation with the second intermediate representation to obtain a fused intermediate representation; and generate a first fusion operator using the fused intermediate representation.

[0117] In some examples, after rewriting the computational content of the second intermediate representation, the computational content of the second intermediate representation includes: performing continuous processing on the shifted tensor data to convert the shifted first output data from a non-continuous tensor to a continuous tensor.

[0118] In some examples, the first output data of at least one first operator has a first shape attribute, a first step length attribute, and a first data attribute, and the first data attribute matches the first shape attribute and the first step length attribute; the shifted first output data has a second shape attribute, a second step length attribute, and the first data attribute, and the first data attribute does not match the second shape attribute and the second step length attribute; the second output data of the first fusion operator has a second shape attribute, a second step length attribute, and a second data attribute, and the second data attribute matches the second shape attribute and the second step length attribute.

[0119] In some examples, after the calculation content of the second intermediate representation is rewritten, the calculation content of the second intermediate representation includes: calculating a second data attribute that matches the second shape attribute and the second step attribute based on the second shape attribute and the second step attribute of the shifted first output data; converting the first data attribute of the shifted first output data into the second data attribute to obtain the second output data.

[0120] In some examples, after rewriting the computational content of the second intermediate representation, the computational content of the second intermediate representation further includes: reading the shifted first output data into a temporary variable; and writing the second output data from the temporary variable into a storage device.

[0121] In some examples, the first fusion operator is an operator written in Triton language. For example, the first fusion operator may also be other types of operators, which may be selected according to actual application scenarios, and the embodiments of the present disclosure are not limited thereto.

[0122] Since in the above description, for example Figure 1 In the process of optimizing the artificial intelligence model shown in FIG. 1 , the details of the operations of the compiler 200 have been introduced, so for the sake of brevity, they will not be repeated here. For relevant details, please refer to the above Figure 1~Figure 4 Description.

[0123] It should be noted that Figure 5 The various tools described above in the compiler 200 shown can be configured as software, hardware, firmware, or any combination of the above items to perform specific functions. For example, these tools can correspond to dedicated integrated circuits, pure software codes, or tools that combine software and hardware. Figure 5 The compiler described may be an Inductor compiler in the PyTorch framework, or may be another type of AI compiler according to actual needs, but is not limited thereto.

[0124] In addition, the above reference Figure 5The compiler 200 described is not limited to including the tools described above, but may also add some other tools (eg, reading and writing tools, storage tools, etc.) as needed, or the above tools may also be combined.

[0125] At least one embodiment of the present disclosure further provides an electronic device, which includes the compiler provided by at least one embodiment of the present disclosure. For example, the electronic device also includes a storage device, which is configured to store the second output data of the first fusion operator.

[0126] Figure 6 A schematic block diagram of an electronic device provided for at least one embodiment of the present disclosure.

[0127] For example, Figure 6 As shown, the first electronic device 300 includes a compiler 200 and a storage device 310. For example, the storage device 310 is configured to store the second output data of the first fusion operator. For example, the storage device 310 can also be configured to store other data generated during the execution of the optimization method of the artificial intelligence model provided in at least one embodiment of the present disclosure, which can be selected according to actual needs, and the embodiments of the present disclosure are not limited to this. For example, the storage device 310 can be an external video memory of a chip, or it can be selected as other hardware types according to actual needs, and the embodiments of the present disclosure are not limited to its type and implementation method.

[0128] For example, the compiler 200 may be a compiler provided by any embodiment of the present disclosure. The structure and function of the compiler 200 may refer to Figure 5 The corresponding description will not be repeated here.

[0129] At least one embodiment of the present disclosure also provides an electronic device, which includes a processor and a memory; the memory includes one or more computer program modules; the one or more computer program modules are stored in the memory and configured to be executed by the processor, and the one or more computer program modules include an optimization method for implementing the artificial intelligence model provided by at least one embodiment of the present disclosure described above.

[0130] Figure 7 A schematic block diagram of another electronic device provided for at least one embodiment of the present disclosure.

[0131] For example, Figure 7As shown, the second electronic device 400 includes a processor 410 and a memory 420. For example, the memory 420 is used to store non-transitory computer-readable instructions (e.g., one or more computer program modules). The processor 410 is used to run non-transitory computer-readable instructions, and when the non-transitory computer-readable instructions are run by the processor 410, one or more steps of the optimization method according to the artificial intelligence model described above can be executed. The memory 420 and the processor 410 can be interconnected through a bus system and / or other forms of connection mechanisms (not shown).

[0132] For example, the processor 410 may be a digital signal processor (DSP) or other forms of processing units having optimization capabilities of artificial intelligence models and / or program execution capabilities, such as X86, ARM architecture, or field programmable gate array (FPGA), etc. The processor 410 may be a general-purpose processor or a dedicated processor, and may control other components in the second electronic device 400 to perform desired functions.

[0133] For example, the memory 420 may include any combination of one or more computer program products, and the computer program product may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory (cache), etc. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, erasable programmable read-only memory (EPROM), portable compact disk read-only memory (CD-ROM), USB memory, flash memory, etc. One or more computer program modules may be stored on the computer-readable storage medium, and the processor 410 may run one or more computer program modules to implement various functions of the second electronic device 400. Various applications and various data, as well as various data used and / or generated by the application, etc. may also be stored in the computer-readable storage medium.

[0134] It should be noted that in the embodiments of the present disclosure, the specific functions and technical effects of the second electronic device 400 can refer to the above description of the optimization method of the artificial intelligence model provided by at least one embodiment of the present disclosure, and will not be repeated here.

[0135] Figure 8 A schematic block diagram of another electronic device provided for at least one embodiment of the present disclosure.

[0136] For example, Figure 8 As shown, the third electronic device 500 is suitable for implementing the optimization method of the artificial intelligence model provided by the embodiment of the present disclosure. It should be noted that, Figure 8The third electronic device 500 shown is only an example and does not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0137] For example, Figure 8 As shown, the third electronic device 500 may include a processing device 51 (such as a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory ROM 52 or a program loaded from a storage device 310 to a random access memory RAM 53. In RAM 53, various programs and data required for the operation of the third electronic device 500 are also stored. The processing device 51, ROM 52 and RAM 53 are connected to each other through a bus 54. An input / output I / O interface 55 is also connected to the bus 54. Generally, the following devices can be connected to the I / O interface 55: an input device 56 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 57 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 310 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 59. The communication device 59 can allow the third electronic device 500 to communicate with other electronic devices wirelessly or by wire to exchange data.

[0138] Although Figure 8 The third electronic device 500 is shown to have various devices, but it should be understood that it is not required to implement or possess all the devices shown, and the third electronic device 500 may alternatively implement or possess more or fewer devices.

[0139] It should be noted that, for the sake of clarity and brevity, the embodiments of the present disclosure do not provide all the components of the first electronic device 300, the second electronic device 400, and the third electronic device 500. To implement the necessary functions of the electronic device, those skilled in the art may provide and set other components not shown according to specific needs, and the embodiments of the present disclosure are not limited thereto.

[0140] For the detailed description and technical effects of the first electronic device 300, the second electronic device 400 and the third electronic device 500, please refer to the above description of the optimization method of the artificial intelligence model, which will not be repeated here.

[0141] Fig. 9 A schematic diagram of a storage medium provided for at least one embodiment of the present disclosure.

[0142] For example, Fig. 9 As shown, the storage medium 600 stores non-transitory computer-readable instructions 610. For example, when the non-transitory computer-readable instructions 610 are executed by a computer, one or more steps in the optimization method according to the artificial intelligence model described above are performed.

[0143] For example, the storage medium 600 can be applied to Figure 7 The storage medium 600 may be the memory 420 in the second electronic device 400. For example, the description of the storage medium 600 may refer to Figure 7 The corresponding description of the memory 420 in the second electronic device 400 is shown and will not be repeated here.

[0144] There are a few points to note about this disclosure:

[0145] (1) In the drawings of the embodiments of the present disclosure, only the structures related to the embodiments of the present disclosure are involved, and other structures can refer to the general design.

[0146] (2) Unless there is a conflict, the features of the same embodiment or different embodiments of the present disclosure may be combined with each other.

[0147] The above are only specific embodiments of the present disclosure, but the protection scope of the present disclosure is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present disclosure, which should be included in the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be based on the protection scope of the claims.

Claims

1. A method for optimizing an artificial intelligence model, characterized in that: The artificial intelligence model includes at least one first operator, a second operator and a target operator, wherein the at least one first operator precedes the target operator in a calculation order, The second operator is used to perform a shift operation on the first output data of the at least one first operator to obtain the shifted first output data, wherein the shifted first output data is a non-continuous tensor, The optimization method of the artificial intelligence model includes: compiling a primary representation of the at least one first operator into a first intermediate representation; Compiling the primary representation of the second operator into a second intermediate representation, wherein in the process of compiling the primary representation of the second operator into the second intermediate representation, rewriting the computation content of the second intermediate representation so that the second intermediate representation includes a function of converting the shifted first output data from the non-continuous tensor to a continuous tensor; Based on the first intermediate representation and the second intermediate representation, fusing the at least one first operator and the second operator to obtain a first fused operator; The second output data of the first fusion operator is used as the first input data of the target operator, wherein the first input data is the continuous tensor.

2. The optimization method of the artificial intelligence model according to claim 1, characterized in that: Before compiling the primary representation of the second operator into the second intermediate representation, the optimization method of the artificial intelligence model further includes: Register the second operator in the compilation list of the compiler.

3. The optimization method of the artificial intelligence model according to claim 1, characterized in that: The fusing the at least one first operator and the second operator based on the first intermediate representation and the second intermediate representation to obtain a first fused operator includes: fusing the first intermediate representation with the second intermediate representation to obtain a fused intermediate representation; The first fusion operator is generated using the fused intermediate representation.

4. The optimization method of the artificial intelligence model according to claim 1, characterized in that: After rewriting the calculation content of the second intermediate representation, the calculation content of the second intermediate representation includes: The shifted tensor data is processed continuously to convert the shifted first output data from the non-continuous tensor into a continuous tensor.

5. The optimization method of the artificial intelligence model according to claim 1, characterized in that: The first output data of the at least one first operator has a first shape attribute, a first step length attribute and a first data attribute, and the first data attribute matches the first shape attribute and the first step length attribute; The shifted first output data has a second shape attribute, a second step attribute and the first data attribute, and the first data attribute does not match the second shape attribute and the second step attribute; The second output data of the first fusion operator has the second shape attribute, the second step attribute and the second data attribute, and the second data attribute matches the second shape attribute and the second step attribute.

6. The optimization method of the artificial intelligence model according to claim 5, characterized in that: After rewriting the calculation content of the second intermediate representation, the calculation content of the second intermediate representation includes: Calculating the second data attribute matching the second shape attribute and the second step attribute according to the second shape attribute and the second step attribute of the shifted first output data; The first data attribute of the shifted first output data is converted into the second data attribute to obtain the second output data.

7. The optimization method of the artificial intelligence model according to claim 1, characterized in that: After rewriting the calculation content of the second intermediate representation, the calculation content of the second intermediate representation further includes: Reading the shifted first output data into a temporary variable; The second output data is written from the temporary variable into a storage device.

8. The method for optimizing an artificial intelligence model according to any one of claims 1 to 7, characterized in that: The first fusion operator is an operator written in Triton language.

9. A compiler for optimizing an artificial intelligence model, characterized in that: The artificial intelligence model includes at least one first operator, a second operator and a target operator, wherein the at least one first operator precedes the target operator in a calculation order, The second operator is used to perform a shift operation on the first output data of the at least one first operator to obtain the shifted first output data, wherein the shifted first output data is a non-continuous tensor, The compiler comprises: a first compilation tool configured to compile a primary representation of the at least one first operator into a first intermediate representation; a second compilation tool configured to compile the primary representation of the second operator into a second intermediate representation, wherein, in the process of compiling the primary representation of the second operator into the second intermediate representation, the calculation content of the second intermediate representation is rewritten so that the second intermediate representation includes a function of converting the shifted first output data from the non-continuous tensor to a continuous tensor; a first fusion tool configured to fuse the at least one first operator with the second operator based on the first intermediate representation and the second intermediate representation to obtain a first fused operator; The first processing tool is configured to use the second output data of the first fusion operator as the first input data of the target operator, wherein the first input data is the continuous tensor.

10. An electronic device, characterized in that: The electronic device comprises the compiler according to claim 9, The electronic device further includes a storage device, wherein the storage device is configured to store the second output data of the first fusion operator.

11. An electronic device, characterized in that: The electronic device comprises: processor; a memory including one or more computer program modules; Wherein, the one or more computer program modules are stored in the memory and configured to be executed by the processor, and the one or more computer program modules are used to implement the optimization method of the artificial intelligence model described in any one of claims 1-8.

12. A storage medium, characterized in that: The storage medium stores non-transitory computer-readable instructions, which, when executed by a computer, implement the optimization method of the artificial intelligence model described in any one of claims 1 to 8.

Images