Compiling method and device based on dynamic shape scene, storage medium and electronic device

Abstract

The application discloses a compiling method and device based on a dynamic shape scene, a storage medium and an electronic device. The compiling method based on the dynamic shape scene comprises the following steps: acquiring the shape of each operator in an IR graph; marking the target dimension of the dynamic shape; calculating the dynamic data carrying parameter of the dynamic shape, and generating a binary file carrying the dynamic data carrying parameter and an AI model of a dynamic seq, so that the compiling time is reduced, and the technical problem of low compiling efficiency of the dynamic shape scene in the prior art is solved.

Description

Technical Field

[0001] This invention relates to the field of computer science, and more specifically, to a compilation method, apparatus, storage medium, and electronic device based on dynamic shape scenarios. Background Technology

[0002] As large AI models become more prevalent, these models must first be compiled into binary files for the target platform before they can run. However, most platforms currently only support compilation of static shape operators, meaning they only support the compilation and execution of large static models.

[0003] Pre-compile small shapes for various operators into binary files. When running the large model, for each dynamic shape operator, combine the pre-compiled small shape operators into a dynamic shape using a "base shape" approach, and then run. For example, an operator with a shape of [1, 64, 128] can be formed by combining two operators with shapes of [1, 8, 128] and two operators with shapes of [1, 24, 128]. At runtime, the first two shape operators are executed directly, resolving the dynamic shape issue. Each time the model runs, each operator needs to dynamically match multiple pre-compiled base shape .o files, combine them, and then run. The compilation efficiency for dynamic shape scenarios is low. Summary of the Invention

[0004] This invention provides a compilation method, apparatus, storage medium, and electronic device based on dynamic shape scenes, to at least solve the technical problem of low compilation efficiency of dynamic shape scenes in the prior art.

[0005] According to one aspect of the present invention, a compilation method based on a dynamic shape scenario is provided, comprising: obtaining the shape of each operator in an IR graph; marking the target dimension of the dynamic shape therein; calculating the dynamic data transfer parameters of the dynamic shape; and generating a binary file carrying the dynamic data transfer parameters.

[0006] Optionally, after obtaining the shape of each operator in the IR graph, the method includes: setting the maximum number supported by the target dimension; and allocating global memory based on the maximum number supported by the target dimension.

[0007] Optionally, the calculation of the dynamic data transfer parameters of the dynamic shape includes: when the target dimension is seq, multi-core partitioning is performed to partition the batch dimension and channel dimension in the IR diagram, and the shape size for single-core computation is calculated; the amount of data transferred in a single operation is calculated based on the maximum number supported by the seq and the channel dimension; or when the target dimension is batch, multi-core partitioning is performed to partition the seq dimension and channel dimension in the IR diagram, and the shape size for single-core computation is calculated; the amount of data transferred in a single operation is calculated based on the maximum number supported by the batch dimension and the channel dimension.

[0008] Optionally, marking the target dimension of the dynamic shape includes: when the target dimension is a batch, resetting the batch dimension of each operator shape to mark the shape of the dynamic target dimension.

[0009] Optionally, when the target dimension is seq, the step of calculating the dynamic data transfer parameters of the shape of the dynamic target dimension includes:

[0010] Calculate the maximum amount of data that can be moved in a single pass, s_max, along the seq dimension:

[0011] s_max=MAX_ONLIINE_MEM / c,

[0012] The number of times the target dimension is moved is s_num = s / s_max.

[0013] One data transfer operation [1, s_max, c] takes a total of b*s_num transfers.

[0014] Wherein, MAX_ONLIINE_MEM is a preset constant, c is the value corresponding to the channel dimension, and b is the value corresponding to the batch dimension.

[0015] Optionally, when the target dimension is batch, the dynamic data transport parameters for calculating the shape of the dynamic target dimension include:

[0016] Calculate the maximum amount of data that can be moved in a single batch, b_max:

[0017] b_max=MAX_ONLIINE_MEM / sc

[0018] The number of times the target dimension is moved is b_num = b / b_max.

[0019] One data transfer operation [b_max, s, c] takes a total of b_num transfers.

[0020] Wherein, MAX_ONLIINE_MEM is a preset constant, sc is the product of the value corresponding to the seq dimension and the value corresponding to the channel dimension, s is the value corresponding to the seq dimension, c is the value corresponding to the channel dimension, and b is the value corresponding to the batch dimension.

[0021] Optionally, the method further includes: when the compiled model is run, after each operator is calculated, the actual shape is used as the parameter input for the next operator, until all operators have been executed.

[0022] According to a first aspect of the embodiments of this application, a compilation apparatus based on a dynamic shape scenario is provided, comprising: an acquisition unit for acquiring the shape of each operator in an IR diagram; a marking unit for marking the target dimension of the dynamic shape therein; and a generation unit for calculating dynamic data transfer parameters of the dynamic shape and generating a binary file carrying the dynamic data transfer parameters.

[0023] According to a first aspect of the embodiments of this application, a computer-readable storage medium is provided, characterized in that the storage medium stores a computer program, wherein the computer program is configured to execute the above-described compilation method based on a dynamic shape scene at runtime.

[0024] According to a first aspect of the embodiments of this application, an electronic device is provided, including a memory and a processor, characterized in that the memory stores a computer program, and the processor is configured to run the computer program to execute the above-described compilation method based on a dynamic shape scene.

[0025] In this embodiment of the invention, the shape of each operator in the IR graph is obtained; the target dimension of the dynamic shape is marked; the dynamic data transfer parameters of the dynamic shape are calculated, and a binary file carrying the dynamic data transfer parameters and an AI model of dynamic seq are generated. The model is compiled once and run multiple times, reducing compilation time and thus solving the technical problem of low compilation efficiency in dynamic shape scenarios in the prior art. Attached Figure Description

[0026] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, illustrate exemplary embodiments of the invention and, together with their description, serve to explain the invention and do not constitute an undue limitation thereof. In the drawings:

[0027] Figure 1This is a hardware structure block diagram of a mobile terminal based on an optional compilation method for dynamic shape scenes according to an embodiment of the present invention.

[0028] Figure 2 This is a flowchart of an optional compilation method based on a dynamic shape scene according to an embodiment of the present invention;

[0029] Figure 3 This is a flowchart of an optional dynamic compilation method based on dynamic Seq according to an embodiment of the present invention;

[0030] Figure 4 This is a flowchart of an optional dynamic compilation based on dynamic batch according to an embodiment of the present invention;

[0031] Figure 5 This is a diagram of an optional compilation device based on a dynamic shape scene according to an embodiment of the present invention. Detailed Implementation

[0032] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0033] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a sequence of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0034] The compilation method based on dynamic shape scenes provided in this application can be executed on a mobile terminal, computer terminal, or similar computing device. Taking running on a mobile terminal as an example, Figure 1 This is a hardware structure block diagram of a mobile terminal based on a compilation method for dynamic shape scenes, according to an embodiment of the present invention. Figure 1 As shown, the mobile terminal 10 may include one or more ( Figure 1 Only one is shown in the diagram. A processor 102 (which may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.) and a memory 104 for storing data are also shown. Optionally, the mobile terminal may further include a transmission device 106 for communication functions and an input / output device 108. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the mobile terminal described above. For example, the mobile terminal 10 may also include components that are more... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0035] The memory 104 can be used to store computer programs, such as application software programs and modules, like the computer program corresponding to the compilation method based on dynamic shape scenes in this embodiment of the invention. The processor 102 executes various functional applications and data processing by running the computer programs stored in the memory 104, thereby implementing the above-described method. The memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the mobile terminal 10 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0036] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of the mobile terminal 10. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module, used for wireless communication with the Internet.

[0037] This application involves many related concepts such as operators, dynamic shapes, and just-in-time compilation. To better understand the solutions of this application, the relevant terms and concepts that may be involved in this application are introduced below. It should be understood that the explanation of related concepts may be limited due to the specific circumstances of this application, but it does not mean that this application is limited to only that specific situation. The specific circumstances of different embodiments may also differ, and no specific limitation is made here.

[0038] (1) Operator

[0039] An operator is a general term for a certain mathematical operation, such as the convolution operator and the matrix multiplication operator. Specifically, an operator can be considered as a mapping O: X → X from one function space to another. In a general sense, operators can be extended to any space, such as the inner product space.

[0040] (2) Tensor

[0041] In the field of artificial intelligence, a tensor is a multidimensional array. Specifically, a tensor can also be defined, given a basis, as a set of numbers that satisfy a certain transformation law. These numbers, written in a vertical column, are a first-order tensor; written as a two-dimensional array, they are a second-order tensor; written as a three-dimensional array, they are a third-order tensor, and so on. Tensors with dimensions higher than three are called higher-order tensors.

[0042] (3) shape

[0043] The literal meaning of "shape" is simply "shape." In applications like AI and HPC, computationally intensive operations (called operators) such as matrix multiplication, convolution, and vector operations are needed. Mathematically, these operators are all functions performed on multidimensional arrays (i.e., tensors). Each dimension of a multidimensional array has a specific numerical value; for example, a two-dimensional array has a height H and a width W; a three-dimensional array has a height H, a width W, and a depth D. These numerical values ​​across different dimensions are collectively referred to as the shape of the multidimensional array (i.e., tensor).

[0044] The shape of a tensor refers to the shape assigned to it when it is defined. For example, a 5x5 matrix has a length of 5 and a width of 5; similarly, a tensor used for convolution calculations, defined as a 3x3 kernel (filter), has a length of 3 and a width of 3. As an example, suppose the pixels of the image being convolved have a shape of 6x6 and a depth of 3 (representing the three primary colors R / G / B); the convolution kernel is a 3x3x3 cube. Therefore, the convolution kernel is a tensor with a shape of (3, 3, 3), denoted as Tensor1; the image being convolved can be represented as a tensor with a shape of (6, 6, 3), denoted as Tensor2. The operation of convolving these two tensors, Tensor1 and Tensor2, is called an operator, denoted as Convolution.

[0045] (4) Dynamic shape of tensors

[0046] Since each tensor represents a physical meaning, such as pixels in an image or a convolution kernel, if the shape of the tensor to be output at each layer remains unchanged during the model's training or inference computation, then the operator used in this type of computation can be called a fixed-shape operator, and the tensor to be output is a fixed-shape tensor. Conversely, if the shape of the tensor to be output at each layer can change during the model's training or inference computation, then the operator used in this type of computation can be called a dynamic-shape operator, and the tensor to be output is a dynamic-shape tensor.

[0047] IR (Intermediate Representation) is a type of intermediate representation.

[0048] This embodiment also provides a compilation method based on dynamic shape scenarios. Figure 2 This is a flowchart of a compilation method based on a dynamic shape scene according to an embodiment of the present invention, such as... Figure 2 As shown, the compilation method based on dynamic shape scenarios includes the following steps:

[0049] Step S202: Obtain the shape of each operator in the IR graph.

[0050] Step S204: Mark the target dimension of the dynamic shape.

[0051] Step S206: Calculate the dynamic data transfer parameters of the dynamic shape and generate a binary file carrying the dynamic data transfer parameters.

[0052] In this application embodiment, the compilation method based on dynamic shape scenarios includes, but is not limited to, large model inference / training scenarios. It supports dynamic Seq shape compilation of models, compiles once and runs multiple times, reduces compilation time, and solves the problem of not being able to support dynamic shapes.

[0053] In this embodiment, shape can be represented as [b, s, c], where b represents the number of samples in the batch dimension (i.e., batch_size), s represents the input or output dimension of the seq dimension (i.e., seqLen), and c represents the channel dimension.

[0054] To speed up computation, large shapes can be divided into multiple blocks or cores, with each block executing a smaller portion. For example, [2, 24] can be divided into factors of [2, 1], with each block or core executing [1, 24].

[0055] In this embodiment, the operators in the main file can be binary files.

[0056] For large shapes, local memory is limited and cannot accommodate them. Therefore, a data transfer strategy needs to be determined to ensure that the data transferred each time fills the local memory, and the operation is performed in multiple times.

[0057] The embodiments provided in this application obtain the shape of each operator in the IR diagram; mark the target dimension of the dynamic shape; calculate the dynamic data transfer parameters of the dynamic shape and generate a binary file carrying the dynamic data transfer parameters, an AI model of dynamic seq, and compile once and run multiple times to reduce compilation time, thereby solving the technical problem of low compilation efficiency in dynamic shape scenarios in the prior art.

[0058] Optionally, after obtaining the shape of each operator in the IR graph, the above method may include: setting the maximum number supported by the target dimension; and allocating global memory based on the maximum number supported by the target dimension.

[0059] In this embodiment, the maximum number supported by the seq dimension can be set; global memory allocation is performed based on the maximum number supported by the seq dimension.

[0060] Optionally, the dynamic data transfer parameters for calculating the dynamic shape may include: when the target dimension is seq, multi-core partitioning is performed to partition the batch dimension and channel dimension in the IR diagram, and the shape size for single-core computation is calculated; the amount of data transferred in a single transaction is calculated based on the maximum number of seq and the channel dimension; or when the target dimension is batch, multi-core partitioning is performed to partition the seq dimension and channel dimension in the IR diagram, and the shape size for single-core computation is calculated; the amount of data transferred in a single transaction is calculated based on the maximum number of batch dimension and the channel dimension.

[0061] Optionally, marking the target dimension of the dynamic shape may include: when the target dimension is a batch, resetting the batch dimension of each operator shape to mark the shape of the dynamic target dimension.

[0062] In this embodiment, the batch dimension of each operator shape can be reset to 0; for the data with a batch dimension of 0 for each operator, the main file is generated by parameterizing the batch.

[0063] Optionally, when the target dimension is seq, the step of calculating the dynamic data transfer parameters of the shape of the dynamic target dimension may include:

[0064] Calculate the maximum amount of data that can be moved in a single pass, s_max, along the seq dimension:

[0065] s_max=MAX_ONLIINE_MEM / c,

[0066] The number of times the target dimension is moved is s_num = s / s_max.

[0067] One data transfer operation [1, s_max, c] takes a total of b*s_num transfers.

[0068] Wherein, MAX_ONLIINE_MEM is a preset constant, c is the value corresponding to the channel dimension, and b is the value corresponding to the batch dimension.

[0069] Wherein, when the target dimension is batch, the calculation of the dynamic data transport parameters for the shape of the dynamic target dimension includes:

[0070] Calculate the maximum amount of data that can be moved in a single batch, b_max:

[0071] b_max=MAX_ONLIINE_MEM / sc

[0072] The number of times the target dimension is moved is b_num = b / b_max.

[0073] One data transfer operation [b_max, s, c] takes a total of b_num transfers.

[0074] Wherein, MAX_ONLIINE_MEM is a preset constant, sc is the product of the value corresponding to the seq dimension and the value corresponding to the channel dimension, s is the value corresponding to the seq dimension, c is the value corresponding to the channel dimension, and b is the value corresponding to the batch dimension.

[0075] Optionally, the above method may also include: when the compiled model is run, after each operator is calculated, the actual shape is used as the parameter input for the next operator, until all operators have been executed.

[0076] As an optional embodiment, this application also provides a dynamic compilation method based on dynamic sequence number (seq). Figure 3 The diagram shows a schematic of a dynamic compilation method based on dynamic seq.

[0077] Step S31: Given [b, s, c] of arbitrary shape, where s is dynamically changing and b and c are fixed.

[0078] Step S32: Reset the seq dimension to 0 to prepare for subsequent parameterization, and set the maximum number of seq dimensions that can be supported.

[0079] Step S33, multi-core segmentation (multi-block), the segmentation factor only supports two dimensions b and c, so calculate the shape size of single-core calculation.

[0080] Step S34, Single-core (single-block) data transfer strategy: Calculate the amount of data transferred in a single transaction based on the maximum dimension supported by seq and the c-dimensional dimension.

[0081] Step S35, global memory allocation, global memory allocation is performed using the maximum number supported by the seq dimension.

[0082] Step S36: For each operator's seq=0 data, parameterize seq to generate the main file.

[0083] Step S37, execution state: After each operator is calculated, the actual shape is used as the parameter input for the next operator, until all operators have been executed.

[0084] In this embodiment, the segmentation and local memory scheme ([b, s, c]) is as follows:

[0085] 1. Determine the local memory size of the chip platform: MAX_ONLIINE_MEM.

[0086] 2. Fixed the maximum dimension supported by seqLen: SEQ_MAX.

[0087] 3. During compilation, the compilation operator is the xxx.o file.

[0088] 3.1. Marking: For each operator, the seqLen dimension of the shape of each dynamic shape operator is marked as 0.

[0089] 3.2 Segmentation: For each operator, perform multi-core (multi-block) segmentation. For [b*knums, s, c], [b, s*knums,c], or [b, s*knums,c], the multi-core segmentation result is [b, s, c], where knums represents the number of multi-cores. For dynamic seq, only the b-dimensional and c-dimensional segments are segmented. Taking [b*knums, s, c] as an example, such as [16, s, c], the segmentation factor is [16, 1, 1], and the single-core calculation shape is [1, s, c].

[0090] 3.3 Transfer: Based on the MAX_ONLIINE_MEM constraint, calculate the maximum amount of data that can be transferred in a single step in dimension s, s_max = MAX_ONLIINE_MEM / c. The number of transfers for the current dimension is s_num = s / s_max. Transfer this amount of data [1, s_max, c] in one step, and transfer it a total of b * s_num times.

[0091] 3.4 Compilation: Query each operator. If the current operator's shape specifies a dimension of 0, send it to codegen via parameters and generate the operator file xxx.o.

[0092] 3.5 Memory: Allocate the local memory required by the operator based on the maximum seqLen, i.e. SEQ_MAX.

[0093] 3.6 Code: codegen generates xxx.o operator files with parameters.

[0094] 4. During runtime, the actual shape is calculated by issuing parameters and then issued to the next node in sequence until all operators have been executed.

[0095] In this embodiment, the compiler is mainly divided into front-end and back-end, respectively handling hardware-independent and hardware-dependent processing. Each part has its own IR (Intermediate Representation), and each part is also optimized.

[0096] The global memory allocation scheme is as follows: 1. Fix the maximum dimension supported by Seq; 2. Allocate global memory based on the maximum dimension supported by Seq.

[0097] In this embodiment, during the execution of the large AI model, the compilation of dynamic seq segmentation and dynamic seq memory allocation, and the AI ​​model of dynamic seq are compiled once and run multiple times, reducing compilation time.

[0098] For large model inference / training scenarios, it supports dynamic seq shape compilation of models, allowing for one-time compilation and multiple runs, reducing compilation time and solving the problem of not being able to support dynamic shapes.

[0099] As an optional embodiment, this application also provides a dynamic compilation method based on dynamic batches. For example... Figure 4 The diagram shows a schematic of a dynamic compilation method based on dynamic batches.

[0100] Step S41: Given [b, s, c] of arbitrary shape, where b is dynamically changing and s and c are fixed.

[0101] Step S42: Reset the batch dimension to 0 to prepare for subsequent parameterization, and set the maximum number of batch dimensions that can be supported.

[0102] Step S43, multi-core segmentation (multi-block), the segmentation factor only supports two dimensions, s and c, so calculate the shape size of single-core calculation.

[0103] Step S44, Single-core (single-block) data transfer strategy: Calculate the amount of data transferred in a single batch based on the maximum dimension supported by the batch, as well as the s-dimensional and c-dimensional dimensions.

[0104] Step S45, global memory allocation: global memory allocation is performed based on the maximum number supported by the batch dimension.

[0105] Step S46: For each operator with batch=0 data, parameterize the batch and generate the main file.

[0106] Step S47, execution state: After each operator is calculated, the actual shape is used as the parameter input for the next operator, until all operators have been executed.

[0107] In this embodiment, the segmentation and local memory scheme ([b, s, c]) is as follows:

[0108] 1. Determine the on-chip memory size MAX_ONLIINE_MEM of the chip platform.

[0109] 2. Fixed maximum dimension BATCH_MAX supported by batch.

[0110] 3. During compilation, the compilation operator is the xxx.o file.

[0111] 3.1. Marking: For each operator, the batch dimension of the shape of each dynamic shape operator is marked as 0;

[0112] 3.2 Segmentation: For each operator, perform multi-core (multi-block) segmentation. For [b*knums, s, c], [b, s*knums,c], or [b, s*knums,c], the multi-core segmentation result is [b, s, c], only segmenting the s and c dimensions, where knums represents the number of multi-cores. For dynamic batches, only the s and c dimensions are segmented. Taking [b, s*knums,c] as an example, such as [b, 16, c], the segmentation factor is [1, 16, 1], and the single-core calculation shape is [b, 1, c].

[0113] 3.3 Transportation: Calculate the transportation strategies for the s and c dimensions based on the MAX_ONLIINE_MEM constraint;

[0114] 3.4 Compilation: For each operator, if the specified dimension of the current operator's shape is 0, then send it to codegen via parameters and generate the operator file xxx.o;

[0115] 3.5 Memory: Allocate local memory required by the operator based on the maximum batch size, i.e., BATCH_MAX;

[0116] 3.6 Code: codegen generates xxx.o operator files with parameters.

[0117] 4. During runtime, the actual shape is calculated by issuing parameters and then issued to the next node in sequence until all operators have been executed.

[0118] The global memory allocation scheme includes: a) fixing the maximum dimension supported by the batch; b) allocating global memory based on the maximum dimension supported by the batch.

[0119] In this embodiment, for large model inference / training scenarios, dynamic batch shape compilation of the model is supported, allowing for multiple runs after a single compilation, reducing compilation time and solving the problem of not being able to support dynamic shapes.

[0120] When running a large AI model, dynamic batch splitting and dynamic batch memory allocation are used in the compilation process. The AI ​​model is compiled once and run multiple times, reducing compilation time.

[0121] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0122] This embodiment also provides a compilation apparatus based on a dynamic shape scene, which is used to implement the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the apparatus described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0123] Figure 5 This is a structural block diagram of a compilation device based on a dynamic shape scene according to an embodiment of the present invention, such as... Figure 5 As shown, the compilation device based on dynamic shape scenes includes:

[0124] Acquisition unit 51 is used to acquire the shape of each operator in the IR diagram.

[0125] The labeling unit 53 is used to label the target dimension of the dynamic shape therein.

[0126] The generation unit 55 is used to calculate the dynamic data transport parameters of the dynamic shape and generate a binary file carrying the dynamic data transport parameters.

[0127] Through the embodiments provided in this application, the acquisition unit 51 acquires the shape of each operator in the IR diagram; the marking unit 53 marks the target dimension of the dynamic shape; the generation unit 55 calculates the dynamic data transfer parameters of the dynamic shape and generates a binary file carrying the dynamic data transfer parameters, an AI model of dynamic seq, which can be compiled once and run multiple times, reducing compilation time, thereby solving the technical problem of low compilation efficiency in dynamic shape scenarios in the prior art.

[0128] It should be noted that the above modules can be implemented by software or hardware. For the latter, they can be implemented in the following ways, but are not limited to: all the above modules are located in the same processor; or, the above modules are located in different processors in any combination.

[0129] Embodiments of the present invention also provide a storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above method embodiments when running.

[0130] Optionally, in this embodiment, the storage medium may be configured to store a computer program for performing the following steps:

[0131] S1, obtain the shape of each operator in the IR plot;

[0132] S2, marking the target dimensions of the dynamic shape;

[0133] S3 calculates the dynamic data transfer parameters of the dynamic shape and generates a binary file carrying the dynamic data transfer parameters.

[0134] Optionally, in this embodiment, the storage medium may include, but is not limited to, various media capable of storing computer programs, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0135] Embodiments of the present invention also provide an electronic device including a memory and a processor, the memory storing a computer program and the processor being configured to run the computer program to perform the steps in any of the above method embodiments.

[0136] Optionally, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor and the input / output device is connected to the processor.

[0137] Optionally, in this embodiment, the processor can be configured to perform the following steps via a computer program:

[0138] S1, obtain the shape of each operator in the IR plot;

[0139] S2, marking the target dimensions of the dynamic shape;

[0140] S3 calculates the dynamic data transfer parameters of the dynamic shape and generates a binary file carrying the dynamic data transfer parameters.

[0141] Optionally, specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementations, and will not be repeated here.

[0142] It is obvious to those skilled in the art that the modules or steps of the present invention described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. Optionally, they can be implemented using computer-executable program code, thereby storing them in a storage device for execution by a computing device. In some cases, the steps shown or described can be performed in a different order than those presented herein, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, the present invention is not limited to any particular combination of hardware and software.

[0143] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, or improvements made within the principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for compiling based on a dynamic shape scenario, characterized in that, The method comprises the following steps: acquiring the shape of each operator in an IR graph; labeling the target dimension of a dynamic shape; calculating a dynamic data transfer parameter of the dynamic shape and generating a binary file carrying the dynamic data transfer parameter; during the compilation of a compiled model, each operator calculates to the end, and a real shape is taken as a parameter input of a next operator until all operators are executed, wherein the real shape is calculated by the dynamic data transfer parameter in the binary file delivered; the calculation of the dynamic data transfer parameter of the dynamic shape comprises the following steps: in a case where the target dimension is seq, multi-core splitting is performed to split a batch dimension and a channel dimension in the IR graph, and the shape size of single-core calculation is calculated; the maximum data s_max that can be transferred once in the seq dimension is calculated as follows: s_max=MAX_ONLIINE_MEM / c, the target dimension transfer number s_num is s / s_max, data [1, s_max, c] is transferred once, and b*s_num times of data transfer are performed in total; wherein, the MAX_ONLIINE_MEM is the local memory size of a chip platform, the c is a corresponding value of the channel dimension, and the b is a corresponding value of the batch dimension; or in a case where the target dimension is batch, multi-core splitting is performed to split a seq dimension and a channel dimension in the IR graph, and the shape size of single-core calculation is calculated; the maximum data b_max that can be transferred once in the batch dimension is calculated according to the maximum number supported by the batch dimension and the channel dimension; b_max=MAX_ONLIINE_MEM / sc, the target dimension transfer number b_num is b / b_max, data [b_max, s, c] is transferred once, and b_num times of data transfer are performed in total; wherein, the MAX_ONLIINE_MEM is the local memory size of a chip platform, the sc is the product of a corresponding value of the seq dimension and a corresponding value of the channel dimension, the s is the corresponding value of the seq dimension, the c is the corresponding value of the channel dimension, and the b is the corresponding value of the batch dimension. After the shape of each operator in the IR graph is acquired, the method comprises the following steps:

2. The method of claim 1, wherein, setting the maximum number supported by the target dimension; performing global memory allocation according to the maximum number supported by the target dimension. The labeling of the target dimension of the dynamic shape comprises the following steps:

3. The method of claim 1, wherein, in a case where the target dimension is batch, the batch dimension of the shape of each operator is reset to label the shape of the dynamic target dimension. The method comprises the following steps:

4. A device for compiling based on a dynamic shape scenario, characterized in that, an acquiring unit is configured to acquire the shape of each operator in an IR graph; a labeling unit is configured to label the target dimension of a dynamic shape; ​ The generating unit is configured to calculate dynamic data carrying parameters of the dynamic shape and generate a binary file carrying the dynamic data carrying parameters. During the compilation and running of the compiled model, each operator calculation ends, and a real shape is input as a parameter of a next operator until all operators are executed, wherein the real shape is calculated by the dynamic data carrying parameters in the binary file. The generating unit is configured to calculate dynamic data carrying parameters of the dynamic shape, including: In the case that the target dimension is seq, multi-core splitting is performed to split batch and channel dimensions in the IR graph, and a shape size of single-core calculation is calculated; a maximum data s_max that can be carried by the seq dimension once is calculated: s_max=MAX_ONLIINE_MEM / c, a target dimension carrying number s_num is s / s_max, data [1, s_max, c] is carried once, and b*s_num times in total are carried; wherein, the MAX_ONLIINE_MEM is a chip platform local memory size, the c is a channel dimension corresponding value, and the b is a batch dimension corresponding value. Or In the case that the target dimension is batch, multi-core splitting is performed to split seq and channel dimensions in the IR graph, and a shape size of single-core calculation is calculated; According to a maximum number supported by the batch dimension and a channel dimension, a data amount carried once is calculated; a maximum data b_max that can be carried by the batch dimension once is calculated: b_max=MAX_ONLIINE_MEM / sc, a target dimension carrying number b_num is b / b_max, data [b_max, s, c] is carried once, and b_num times in total are carried; wherein, the MAX_ONLIINE_MEM is a chip platform local memory size, the sc is a product of a seq dimension corresponding value and a channel dimension corresponding value, the s is the seq dimension corresponding value, the c is the channel dimension corresponding value, and the b is the batch dimension corresponding value.

5. A computer readable storage medium, characterized in that, The storage medium has a computer program stored therein, wherein the computer program is configured to execute the method in any one of claims 1 to 3 when running. 6.An electronic device comprising a memory and a processor, the electronic device comprising: The memory has a computer program stored therein, and the processor is configured to execute the computer program to execute the method in any one of claims 1 to 3.

Images