Compiling method and device based on dynamic shape scene, storage medium and electronic device

Abstract

The invention discloses a compiling method and device based on a dynamic shape scene, a storage medium and an electronic device. The compiling method based on the dynamic shape scene comprises the following steps: acquiring a shape of each operator in an IR graph; marking the target dimension of the dynamic shape in the target dimension; according to the method, the dynamic data handling parameters of the dynamic shape are calculated, the binary file carrying the dynamic data handling parameters and the AI model of the dynamic seq are generated, multiple times of operation are carried out through one-time compiling, the compiling time is shortened, and the technical problem that in the prior art, the compiling efficiency of the dynamic shape scene is low is solved.

Description

Technical Field

[0001] The present invention relates to computer-related fields, and in particular to a compilation method, device, storage medium and electronic device based on a dynamic shape scene. Background Art

[0002] As large AI models become increasingly common, they must first be compiled into binary files for the target platform before they can be run. Currently, most platforms only support operator compilation for static shapes, meaning they only support the compilation and execution of large static models.

[0003] Precompile the small shapes of various operators into binary files. When running a large model, for each dynamic shape operator, combine the pre-compiled small shape operators into a dynamic shape using a "base shape" method, and then run it. For example, an operator with a shape of [1, 64, 128] can be combined into [1, 64, 128] using two operators with a shape of [1, 8, 128] and two operators with a shape of [1, 24, 128]. The first two shape operators are then directly executed at runtime, resolving the dynamic shape issue. During each run, each operator needs to dynamically match multiple pre-compiled base shape .o files, combine them, and then run. This results in low compilation efficiency for dynamic shape scenarios. Summary of the Invention

[0004] The embodiments of the present invention provide a compilation method, device, storage medium and electronic device based on a dynamic shape scene, so as to at least solve the technical problem of low compilation efficiency of dynamic shape scenes in the prior art.

[0005] According to one aspect of an embodiment of the present invention, a compilation method based on a dynamic shape scenario is provided, including: obtaining the shape of each operator in an IR graph; marking the target dimension of the dynamic shape; calculating dynamic data handling parameters of the dynamic shape, and generating a binary file carrying the dynamic data handling parameters.

[0006] Optionally, after obtaining the shape of each operator in the IR graph, the method includes: setting the maximum number supported by the target dimension; and performing global memory allocation according to the maximum number supported by the target dimension.

[0007] Optionally, the calculation of the dynamic data handling parameters of the dynamic shape includes: when the target dimension is seq, multi-core splitting, splitting the batch dimension and channel dimension in the IR map, and calculating the shape size of the single-core calculation; calculating the amount of data handled in a single time based on the maximum number and channel dimension supported by the seq; or when the target dimension is batch, multi-core splitting, splitting the seq dimension and channel dimension in the IR map, and calculating the shape size of the single-core calculation; calculating the amount of data handled in a single time based on the maximum number and channel dimension supported by the batch dimension.

[0008] Optionally, marking the target dimension of the dynamic shape includes: when the target dimension is batch, resetting the batch dimension of each operator shape, and marking the shape of the dynamic target dimension.

[0009] Optionally, when the target dimension is seq, the step of calculating the dynamic data handling parameter of shape of the dynamic target dimension includes: Calculate the maximum data s_max that can be transported in a single time in the seq dimension: s_max=MAX_ONLIINE_MEM / c, The target dimension transfer times s_num=s / s_max, The data [1, s_max, c] is moved once, and a total of b*s_num times; Among them, the MAX_ONLIINE_MEM is a preset constant, the c is the value corresponding to the channel dimension, and the b is the value corresponding to the batch dimension.

[0010] Optionally, when the target dimension is batch, the dynamic data handling parameters for calculating the shape of the dynamic target dimension include: Calculate the maximum data b_max that can be moved in a single batch dimension: b_max=MAX_ONLIINE_MEM / sc, The target dimension transfer times b_num=b / b_max, Move the data [b_max, s, c] once, and move a total of b_num times; Among them, the MAX_ONLIINE_MEM is a preset constant, the sc is the product of the value corresponding to the seq dimension and the value corresponding to the channel dimension, the s is the value corresponding to the seq dimension, the c is the value corresponding to the channel dimension, and the b is the value corresponding to the batch dimension.

[0011] Optionally, the method further includes: when the compiled model is run, after each operator is calculated, the actual shape is used as a parameter input of the next operator until all operators are executed.

[0012] According to a first aspect of an embodiment of the present application, a compilation device based on a dynamic shape scenario is provided, including: an acquisition unit for acquiring the shape of each operator in an IR graph; a marking unit for marking the target dimension of the dynamic shape; and a generation unit for calculating dynamic data handling parameters of the dynamic shape and generating a binary file carrying the dynamic data handling parameters.

[0013] According to a first aspect of an embodiment of the present application, a computer-readable storage medium is provided, characterized in that a computer program is stored in the storage medium, wherein the computer program is configured to execute the above-mentioned compilation method based on the dynamic shape scene during runtime.

[0014] According to a first aspect of an embodiment of the present application, an electronic device is provided, comprising a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute the above-mentioned compilation method based on the dynamic shape scene.

[0015] In an embodiment of the present invention, the shape of each operator in the IR graph is obtained; the target dimension of the dynamic shape is marked; the dynamic data handling parameters of the dynamic shape are calculated, and a binary file carrying the dynamic data handling parameters is generated. The AI ​​model of the dynamic seq is compiled once and run multiple times, reducing the compilation time, thereby solving the technical problem of low compilation efficiency in dynamic shape scenarios in the existing technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings: Figure 1 1 is a hardware structure block diagram of a mobile terminal according to an optional compilation method based on a dynamic shape scene according to an embodiment of the present invention; Figure 2is a flowchart of an optional compilation method based on a dynamic shape scene according to an embodiment of the present invention; Figure 3 is a flowchart of an optional dynamic compilation method based on dynamic Seq according to an embodiment of the present invention; Figure 4 is a flow chart of an optional dynamic compilation based on dynamic batch according to an embodiment of the present invention; Figure 5 4 is an optional diagram of a compilation device based on a dynamic shape scene according to an embodiment of the present invention. DETAILED DESCRIPTION

[0017] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0018] It should be noted that the terms "first," "second," and the like in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or precedence. It should be understood that the numbers used in this manner are interchangeable where appropriate so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having," as well as any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus that includes a sequence of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to these processes, methods, products, or apparatus.

[0019] The embodiment of the compilation method based on the dynamic shape scene provided in the embodiment of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Taking running on a mobile terminal as an example, Figure 1 This is a hardware structure diagram of a mobile terminal based on a compilation method of a dynamic shape scene according to an embodiment of the present invention. Figure 1 As shown, the mobile terminal 10 may include one or more ( Figure 1Only one is shown) a processor 102 (the processor 102 may include but is not limited to a microprocessor MCU or a programmable logic device FPGA and other processing devices) and a memory 104 for storing data. Optionally, the mobile terminal may also include a transmission device 106 and an input / output device 108 for communication functions. It will be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the mobile terminal. Figure 1 More or fewer components than shown, or with Figure 1 Different configurations shown.

[0020] The memory 104 can be used to store computer programs, such as software programs and modules of application software, such as the computer program corresponding to the dynamic shape scene-based compilation method in the embodiment of the present invention. The processor 102 executes the computer programs stored in the memory 104 to execute various functional applications and data processing, thereby implementing the above-mentioned method. The memory 104 may include high-speed random access memory and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories may be connected to the mobile terminal 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0021] Transmission device 106 is used to receive or transmit data via a network. A specific example of the aforementioned network may include a wireless network provided by the telecommunications provider of mobile terminal 10. In one embodiment, transmission device 106 includes a network interface controller (NIC), which can be connected to other network devices via a base station to enable communication with the Internet. In another embodiment, transmission device 106 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0022] The embodiments of this application involve a lot of knowledge about operators, dynamic shapes, just-in-time compilation, etc. In order to better understand the solutions of the embodiments of this application, the following first introduces the relevant terms and concepts that may be involved in the embodiments of this application. It should be understood that the interpretation of related concepts may be limited by the specific circumstances of the embodiments of this application, but it does not mean that this application is limited to this specific situation. The specific circumstances of different embodiments may also vary, and are not specifically limited here.

[0023] (1) Operator An operator is a general term for a mathematical operation, such as the convolution operator or the matrix multiplication operator. Specifically, an operator can be thought of as a mapping O:X→X from one function space to another. In a broad sense, operators can be extended to any space, such as the inner product space.

[0024] (2) Tensor In the field of artificial intelligence, a tensor is a multidimensional array. Specifically, a tensor can be defined as a set of numbers that satisfy a certain transformation law within a given basis. These numbers written as a vertical row are first-order tensors; written as a two-dimensional array is a second-order tensor; written as a three-dimensional array is a third-order tensor, and so on. Tensors with dimensions higher than third order are called higher-order tensors.

[0025] (3) shape The literal meaning of "shape" is "shape." Applications like AI and HPC require computationally intensive operations (called operators) such as matrix multiplication, convolution, and vector manipulation. These operators are mathematically functions that compute on multidimensional arrays (tensors). Each dimension of a multidimensional array has a specific value, such as the height H and width W of a two-dimensional array, or the height H, width W, and depth D of a three-dimensional array. These values ​​are collectively referred to as the shape of the multidimensional array (tensor).

[0026] The shape of a tensor refers to the shape assigned to it when it is defined. For example, the shape of a 5*5 matrix is ​​5 in length and 5 in width. For another example, the shape of the convolution kernel (filter) used for convolution calculation is 3*3, and its shape is 3 in length and 3 in width. As an example, suppose the shape of the pixels of the image being convolved is 6*6 and the depth is 3 (representing the three primary colors R / G / B); the convolution kernel is a 3*3*3 cube. Therefore, the convolution kernel is a tensor with a shape of (3, 3, 3), which can be recorded as Tensor1; the convolved image can be represented as a tensor with a shape of (6, 6, 3), which can be recorded as Tensor2. The operation of performing a convolution operation on these two tensors is called an operator, recorded as Convolution.

[0027] (4) Dynamic shape of tensors Since each tensor expresses a physical meaning, such as image pixels, convolution kernels, etc., if the shape of the tensor to be output at each layer does not change during model training or inference calculations, then the operator used in such calculations can be called a fixed-shape operator, and the tensor to be output is a tensor of fixed shape. Conversely, if the shape of the tensor to be output at each layer can change during model training or inference calculations, then the operator used in such calculations can be called a dynamic-shape operator, and the tensor to be output is a tensor of dynamic shape.

[0028] IR (Intermediate Representation) intermediate representation.

[0029] In this embodiment, a compilation method based on a dynamic shape scene is also provided. Figure 2 : is a flowchart of a compilation method based on a dynamic shape scene according to an embodiment of the present invention, such as Figure 2 As shown, the compilation method based on the dynamic shape scene includes the following steps: Step S202: Obtain the shape of each operator in the IR graph.

[0030] Step S204: Mark the target dimension of the dynamic shape.

[0031] Step S206 , calculating dynamic data handling parameters of the dynamic shape, and generating a binary file carrying the dynamic data handling parameters.

[0032] In the embodiment of the present application, the compilation method based on the dynamic shape scenario includes but is not limited to large model reasoning / training scenarios, supports model dynamic Seq shape compilation, compiles once and runs multiple times, reduces compilation time, and solves the problem of not being able to support dynamic shape.

[0033] In this embodiment, shape can be expressed as [b, s, c], where b represents the number of samples in the batch dimension (ie, batch_size), s represents the seq dimension input or output dimension (ie, seqLen), and c represents the channel dimension.

[0034] To speed up the calculation, for large shapes, they can be divided into multiple blocks or cores to execute a small piece each. For example, [2, 24] can be split into [2, 1], and each block or core executes [1, 24] respectively.

[0035] In an embodiment, the operator in the main file may be a binary file.

[0036] For large shapes, local memory is limited and cannot accommodate large shapes. It is necessary to determine a transfer strategy to ensure that the data transferred each time fills the local memory and is executed in multiple times.

[0037] Through the embodiments provided in this application, the shape of each operator in the IR graph is obtained; the target dimension of the dynamic shape is marked; the dynamic data handling parameters of the dynamic shape are calculated, and a binary file carrying the dynamic data handling parameters is generated. The AI ​​model of the dynamic seq is compiled once and run multiple times, reducing the compilation time, thereby solving the technical problem of low compilation efficiency of dynamic shape scenarios in the existing technology.

[0038] Optionally, after obtaining the shape of each operator in the IR graph, the above method may include: setting the maximum number supported by the target dimension; and performing global memory allocation according to the maximum number supported by the target dimension.

[0039] In this embodiment, the maximum number supported by the seq dimension may be set; and global memory allocation is performed based on the maximum number supported by the seq dimension.

[0040] Optionally, calculating dynamic data handling parameters for dynamic shape may include: when the target dimension is seq, multi-core splitting, splitting the batch dimension and channel dimension in the IR map, and calculating the shape size of single-core calculation; calculating the amount of data handled in a single time based on the maximum number and channel dimension supported by seq; or when the target dimension is batch, multi-core splitting, splitting the seq dimension and channel dimension in the IR map, and calculating the shape size of single-core calculation; calculating the amount of data handled in a single time based on the maximum number and channel dimension supported by the batch dimension.

[0041] Optionally, marking the target dimension of the dynamic shape may include: when the target dimension is a batch, resetting the batch dimension of each operator shape, and marking the shape of the dynamic target dimension.

[0042] In this embodiment, the batch dimension of each operator shape can be reset to 0; for the data with a batch dimension of 0 for each operator, the main file is generated by parameterizing the batch.

[0043] Optionally, when the target dimension is seq, the step of calculating the dynamic data handling parameter of the shape of the dynamic target dimension may include: Calculate the maximum data s_max that can be transported in a single time in the seq dimension: s_max=MAX_ONLIINE_MEM / c, The target dimension transfer times s_num=s / s_max, The data [1, s_max, c] is moved once, and a total of b*s_num times; Among them, the MAX_ONLIINE_MEM is a preset constant, the c is the value corresponding to the channel dimension, and the b is the value corresponding to the batch dimension.

[0044] Wherein, when the target dimension is batch, the dynamic data handling parameters of the shape of the dynamic target dimension are calculated, including: Calculate the maximum data b_max that can be moved in a single batch dimension: b_max=MAX_ONLIINE_MEM / sc, The target dimension transfer times b_num=b / b_max, Move the data [b_max, s, c] once, and move a total of b_num times; Among them, the MAX_ONLIINE_MEM is a preset constant, the sc is the product of the value corresponding to the seq dimension and the value corresponding to the channel dimension, the s is the value corresponding to the seq dimension, the c is the value corresponding to the channel dimension, and the b is the value corresponding to the batch dimension.

[0045] Optionally, the above method may further include: when the compiled model is run, after each operator is calculated, the actual shape is used as a parameter input of the next operator until all operators are executed.

[0046] As an optional embodiment, the present application also provides a dynamic compilation method based on dynamic seq. Figure 3 As shown, it is a schematic diagram of the dynamic compilation method based on dynamic seq.

[0047] Step S31: Given an arbitrary shape [b, s, c], where s is dynamically changing and b and c are fixed.

[0048] Step S32: reset the seq dimension to 0 to prepare for subsequent parameterization, and set the maximum number supported by the Seq dimension.

[0049] Step S33: multi-core segmentation (multi-block). The segmentation factor only supports two dimensions, b and c, to calculate the shape size of single-core calculation.

[0050] Step S34, single-core (single-block) transfer strategy: calculate the amount of data transferred in a single time based on the maximum dimension and c dimension supported by seq.

[0051] Step S35: global memory allocation, performing global memory allocation according to the maximum number supported by the seq dimension.

[0052] Step S36: for the data of seq=0 of each operator, parameterize seq to generate the main file.

[0053] Step S37, execution state, after each operator is calculated, the real shape is used as the parameter input of the next operator until all operators are executed.

[0054] In this embodiment, the segmentation and local memory solution ([b, s, c]): 1. Determine the chip platform local memory size MAX_ONLIINE_MEM.

[0055] 2. Fixed the maximum dimension SEQ_MAX supported by seqLen.

[0056] 3. During compilation, the compilation operator is the xxx.o file.

[0057] 3.1. Marking: Query each operator and mark the seqLen dimension of each dynamic shape operator shape as 0.

[0058] 3.2. Splitting: Query each operator and perform multi-core (multi-block) splitting. For [b*knums, s, c], [b, s*knums,c], or [b, s*knums,c], the multi-core splitting result is [b, s, c], where knums represents the number of multi-cores. For dynamic sequences, only the b and c dimensions are split. For example, for [b*knums, s, c], for example, [16, s, c], the split factor is [16, 1, 1], and the single-core calculation shape is [1, s, c].

[0059] 3.3. Transport: According to the MAX_ONLIINE_MEM constraint, calculate the maximum data that can be transported in the s dimension at a time: s_max = MAX_ONLIINE_MEM / c. The number of times the current dimension is transported: s_num = s / s_max. Transporting this much data at a time: [1, s_max, c], a total of b*s_num times.

[0060] 3.4. Compilation: Query each operator. If the current operator shape specifies a dimension of 0, send it to codegen through parameters and generate the operator file xxx.o.

[0061] 3.5. Memory: Based on the maximum seqLen, i.e. SEQ_MAX, apply for the local memory required by the operator.

[0062] 3.6. Code: codegen generates the xxx.o operator file with parameters.

[0063] 4. During the runtime, the actual shape is calculated by sending parameters and then sent to the next node in turn until all operators are executed.

[0064] In this embodiment, the compiler is mainly divided into the front-end and back-end, respectively for hardware-independent and hardware-dependent processing. Each part has its own IR (Intermediate Representation), and each part will also be optimized.

[0065] Among them, the global memory allocation scheme is: 1. Fix the maximum dimension supported by Seq; 2. Perform global memory allocation based on the maximum dimension supported by Seq.

[0066] In this embodiment, when the AI ​​large model is running, dynamic seq segmentation and dynamic seq memory allocation are compiled, and the dynamic seq AI model is compiled once and run multiple times, reducing compilation time.

[0067] In large-model inference / training scenarios, it supports dynamic seq shape compilation of the model, allowing one compilation and multiple runs, reducing compilation time and solving the problem of not being able to support dynamic shapes.

[0068] As an optional embodiment, the present application also provides a dynamic compilation method based on dynamic batch. Figure 4 As shown in FIG, a schematic diagram of a dynamic compilation method based on dynamic batch.

[0069] Step S41: Given an arbitrary shape [b, s, c], where b is dynamically changing and s and c are fixed.

[0070] Step S42: Reset the batch dimension to 0 to prepare for subsequent parameterization and set the maximum number of batch dimensions supported. Step S43: multi-core segmentation (multi-block). The segmentation factor only supports two dimensions, s and c, to calculate the shape size of single-core calculation.

[0071] Step S44, single core (single block) each transfer strategy: Calculate the amount of data transferred in a single time based on the maximum dimension supported by the batch and the s and c dimensions. Step S45: Global memory allocation: global memory allocation is performed using the maximum number supported by the batch dimension.

[0072] Step S46: For the data of batch=0 of each operator, parameterize the batch to generate the main file.

[0073] Step S47, execution state, after each operator is calculated, the real shape is used as the parameter input of the next operator until all operators are executed.

[0074] In this embodiment, the segmentation and local memory solution ([b, s, c]): 1. Determine the chip platform on-chip memory size MAX_ONLIINE_MEM.

[0075] 2. Fixed the maximum dimension BATCH_MAX supported by the batch.

[0076] 3. During compilation, the compilation operator is the xxx.o file.

[0077] 3.1. Marking: Query each operator and mark the batch dimension of each dynamic shape operator shape as 0; 3.2. Splitting: Query each operator and perform multi-core (multi-block) splitting of the operator. For [b*knums, s, c], [b, s*knums,c], or [b, s*knums,c], the multi-core splitting result is [b, s, c], which only splits the s and c dimensions, where knums represents the number of multi-cores. For dynamic batches, only the s and c dimensions are split. Taking [b, s*knums,c] as an example, for example: [b, 16, c], the splitting factor is [1, 16, 1], and the single-core calculation shape is [b, 1, c]; 3.3. Transport: Calculate the transport strategy for the s and c dimensions based on the MAX_ONLINE_MEM constraint; 3.4. Compile: Query each operator. If the current operator shape specifies a dimension of 0, send it to codegen through parameters and generate the operator file xxx.o. 3.5. Memory: Based on the maximum batch dimension, BATCH_MAX, apply for the local memory required by the operator; 3.6. Code: codegen generates the xxx.o operator file with parameters.

[0078] 4. During the runtime, the actual shape is calculated by sending parameters and then sent to the next node in turn until all operators are executed.

[0079] Among them, the global memory allocation scheme is: a. fix the maximum dimension supported by the batch; b. perform global memory allocation based on the maximum dimension supported by the batch.

[0080] In this embodiment, large model inference / training scenarios support dynamic batch shape compilation of the model, which allows one compilation and multiple runs, reducing compilation time and solving the problem of not being able to support dynamic shapes.

[0081] When a large AI model is running, dynamic batch splitting and dynamic batch memory allocation are compiled. The AI ​​model of dynamic batch can be compiled once and run multiple times, reducing compilation time.

[0082] Through the description of the above embodiments, those skilled in the art will clearly understand that the methods according to the above embodiments can be implemented using software plus the necessary general-purpose hardware platform. Of course, hardware can also be used, but in many cases the former is a more preferred embodiment. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, or optical disk) and includes a number of instructions for enabling a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in various embodiments of the present invention.

[0083] This embodiment also provides a compilation device based on a dynamic shape scene, which is used to implement the above-mentioned embodiments and preferred embodiments. Details already described will not be repeated here. As used below, the term "module" may refer to a combination of software and / or hardware that implements a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, implementation using hardware, or a combination of software and hardware, is also possible and contemplated.

[0084] Figure 5 is a structural block diagram of a compilation device based on a dynamic shape scene according to an embodiment of the present invention, such as Figure 5 As shown, the compilation device based on the dynamic shape scene includes: The acquisition unit 51 is used to acquire the shape of each operator in the IR graph.

[0085] The marking unit 53 is used to mark the target dimension of the dynamic shape.

[0086] The generating unit 55 is configured to calculate the dynamic data handling parameters of the dynamic shape and generate a binary file carrying the dynamic data handling parameters.

[0087] Through the embodiments provided in this application, the acquisition unit 51 acquires the shape of each operator in the IR graph; the marking unit 53 marks the target dimension of the dynamic shape therein; the generation unit 55 calculates the dynamic data handling parameters of the dynamic shape, and generates a binary file carrying the dynamic data handling parameters. The AI ​​model of the dynamic seq is compiled once and run multiple times, reducing the compilation time, thereby solving the technical problem of low compilation efficiency of dynamic shape scenarios in the existing technology.

[0088] It should be noted that the above modules can be implemented through software or hardware. For the latter, it can be implemented in the following ways, but not limited to: the above modules are all located in the same processor; or the above modules are located in different processors in any combination.

[0089] An embodiment of the present invention further provides a storage medium storing a computer program, wherein the computer program is configured to execute the steps of any of the above method embodiments when running.

[0090] Optionally, in this embodiment, the storage medium may be configured to store a computer program for performing the following steps: S1, obtain the shape of each operator in the IR graph; S2, marks the target dimension of the dynamic shape; S3 calculates the dynamic data handling parameters of the dynamic shape and generates a binary file carrying the dynamic data handling parameters.

[0091] Optionally, in this embodiment, the above-mentioned storage medium may include but is not limited to: a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk, and other media that can store computer programs.

[0092] An embodiment of the present invention further provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0093] Optionally, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0094] Optionally, in this embodiment, the processor may be configured to execute the following steps through a computer program: S1, obtain the shape of each operator in the IR graph; S2, marks the target dimension of the dynamic shape; S3 calculates the dynamic data handling parameters of the dynamic shape and generates a binary file carrying the dynamic data handling parameters.

[0095] Optionally, specific examples in this embodiment may refer to the examples described in the above embodiments and optional implementation modes, and this embodiment will not be described in detail here.

[0096] Obviously, those skilled in the art will appreciate that the various modules or steps of the present invention described above can be implemented using a general-purpose computing device, centralized on a single computing device, or distributed across a network of multiple computing devices. Alternatively, they can be implemented using program code executable by a computing device, which can then be stored in a storage device and executed by the computing device. In some cases, the steps shown or described can be performed in a different order than that shown, or can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, the present invention is not limited to any particular combination of hardware and software.

[0097] The foregoing description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the principles of the present invention are intended to be within the scope of protection of the present invention.

Claims

1. A compilation method based on dynamic shape scenes, characterized in that: include: Get the shape of each operator in the IR graph; Mark the target dimension of the dynamic shape; Dynamic data handling parameters of the dynamic shape are calculated, and a binary file carrying the dynamic data handling parameters is generated.

2. The method according to claim 1, characterized in that After obtaining the shape of each operator in the IR graph, the method includes: Set the maximum number supported by the target dimension; Global memory allocation is performed with the maximum supported size of the target dimension.

3. The method according to claim 2, characterized in that The calculating of the dynamic data handling parameters of the dynamic shape includes: When the target dimension is seq, multi-core splitting is used to split the batch dimension and channel dimension in the IR image and calculate the shape size of the single-core calculation; Calculate the amount of data transferred in a single transaction based on the maximum number of channels supported by the seq; or When the target dimension is batch, multi-core splitting is used to split the seq dimension and channel dimension in the IR graph, and the shape size of the single-core calculation is calculated; Calculate the amount of data transferred in a single batch based on the maximum number supported by the batch dimension and the channel dimension.

4. The method according to claim 2, characterized in that The target dimension of the dynamic shape is marked, including: When the target dimension is batch, the batch dimension of each operator shape is reset and marked to obtain the shape of the dynamic target dimension.

5. The method according to claim 4, characterized in that In the case where the target dimension is seq, the steps of calculating the dynamic data handling parameters of the shape of the dynamic target dimension include: Calculate the maximum data s_max that can be transported in a single time in the seq dimension: s_max=MAX_ONLIINE_MEM / c, The target dimension transfer times s_num=s / s_max, The data [1, s_max, c] is moved once, and a total of b*s_num times; Among them, the MAX_ONLIINE_MEM is a preset constant, the c is the value corresponding to the channel dimension, and the b is the value corresponding to the batch dimension.

6. The method according to claim 4, characterized in that When the target dimension is batch, the calculation of the dynamic data handling parameters of the shape of the dynamic target dimension includes: Calculate the maximum data b_max that can be moved in a single batch dimension: b_max=MAX_ONLIINE_MEM / sc, The target dimension transfer times b_num=b / b_max, Move the data [b_max, s, c] once, and move a total of b_num times; Among them, the MAX_ONLIINE_MEM is a preset constant, the sc is the product of the value corresponding to the seq dimension and the value corresponding to the channel dimension, the s is the value corresponding to the seq dimension, the c is the value corresponding to the channel dimension, and the b is the value corresponding to the batch dimension.

7. The method according to claim 1, characterized in that The method further comprises: When the compiled model is run, each operator is calculated and the actual shape is used as the parameter input of the next operator until all operators are executed.

8. A compilation device based on dynamic shape scenes, characterized in that: include: The acquisition unit is used to obtain the shape of each operator in the IR graph; The marking unit is used to mark the target dimension of the dynamic shape; A generating unit is used to calculate the dynamic data handling parameters of the dynamic shape and generate a binary file carrying the dynamic data handling parameters.

9. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, wherein the computer program is configured to execute the method according to any one of claims 1 to 7 when executed.

10. An electronic device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to run the computer program to perform the method according to any one of claims 1 to 7.

Images