A data processing method, device, equipment and computer storage medium

Abstract

The present invention provides a data processing method, apparatus, device and computer storage medium, the method comprising: obtaining an intermediate representation of a deep learning model; wherein the loop expansion factor is related to the information of the intermediate representation when executed by a back-end hardware device and / or the device information of the back-end hardware device; loop expansion is performed on the intermediate representation according to the loop expansion factor to obtain an optimized intermediate representation; the optimized intermediate representation is compiled to obtain a target code executable by the back-end hardware device, so that the back-end hardware device executes the target code and realizes the function of the target code. By adopting the embodiment of the present invention, the loop expansion factor can be calculated through the information of the back-end hardware device when executed and / or the device information of the back-end hardware device to obtain a more accurate loop expansion factor, and the intermediate representation is loop expanded through the loop expansion factor to perform instruction scheduling in a larger range, thereby improving the portability of the intermediate representation.

Description

Technical Field

[0001] The present invention relates to the field of computer software technology, and in particular to a data processing method, device, equipment and computer storage medium. Background Art

[0002] The new wave of AI enthusiasm represented by machine learning and deep learning has been heating up for several years. Deep learning has been a research hotspot in the field of AI in recent years and has achieved breakthrough progress in many fields. There are many deep learning frameworks and hardware platforms that support multiple deep learning frameworks. The diversity of deep learning frameworks and hardware brings great benefits to users and is crucial to maintaining the healthy development of the AI ​​ecosystem, but supporting multiple frameworks and hardware requires a huge workload, which also brings considerable challenges to AI developers.

[0003] As deep learning is increasingly used, the efficiency of deep learning algorithm training and reasoning on different hardware architectures has attracted more and more attention. Since deep learning has many different front-ends and back-ends, a bridge is needed to effectively optimize and map them. As the intermediary between source code and target code in the program compilation process, the design of IR (Intermediate Representation) is very critical for the compiler. It is necessary to consider the completeness of the compilation from source code to target code, the ease of use of compilation optimization, and the performance. At present, since most of the running time of IR programs is occupied by loops, it is very important to optimize the loop code in the program. Summary of the invention

[0004] The technical problem to be solved by the embodiments of the present invention is to provide a data processing method, apparatus, device and computer storage medium, which can further improve the compilation optimization effect of the intermediate representation by optimizing the loop code in the intermediate representation, realize the versatility of the intermediate representation that is independent of the front-end language in a variety of back-end hardware devices, and improve the portability of the intermediate representation.

[0005] In a first aspect, an embodiment of the present invention provides a data processing method, including:

[0006] Acquire an intermediate representation of a deep learning model; wherein the deep learning model is a model built based on a front-end model framework; a loop expansion factor is set for the intermediate representation; the loop expansion factor is related to information of the intermediate representation when executed by a back-end hardware device and / or device information of the back-end hardware device;

[0007] Performing loop expansion on the intermediate representation according to the loop expansion factor to obtain an optimized intermediate representation;

[0008] The optimized intermediate representation is compiled to obtain a target code executable by the back-end hardware device, so that the back-end hardware device executes the target code to realize the function of the target code.

[0009] Compared with the prior art, a data processing method provided in the first aspect of an embodiment of the present invention can calculate the loop expansion factor through the information of the back-end hardware device during execution and / or the device information of the back-end hardware device to obtain a more accurate loop expansion factor, and use the loop expansion factor to perform loop expansion on the intermediate representation to perform instruction scheduling in a larger range and better develop instruction-level parallelism, thereby achieving further optimization of the loop code in the intermediate representation, improving the compilation optimization effect of the intermediate representation, achieving the universality of the intermediate representation that is independent of the front-end language in a variety of back-end hardware devices, and improving the portability of the intermediate representation.

[0010] As an optional embodiment of the first aspect, the information of the intermediate representation when executed on the back-end hardware device includes at least one of the M loop structures in the intermediate representation and the number of iterations corresponding to each of the M loop structures; M is a positive integer greater than 1.

[0011] It is worth noting that since the loop code accounts for a large proportion of the intermediate representation and some parameter information in the loop code cannot be determined during the program compilation stage, the loop expansion factor can be calculated by considering the parameter information that can only be determined when the program is running, that is, the information when the intermediate representation is executed on the back-end hardware device. This can obtain a more accurate loop expansion factor, thereby improving the optimization effect of the intermediate representation.

[0012] As an optional embodiment of the first aspect, the device information of the backend hardware device includes at least one of a register parameter, a code volume parameter, and a functional component parameter;

[0013] Among them, the register parameter is used to indicate the number of registers contained in the back-end hardware device; the code volume parameter is used to indicate the size of the program code that the back-end hardware device can run; the functional component parameter is used to indicate the number of functional components contained in the back-end hardware device; the functional component is used to characterize the component that completes the computing function.

[0014] It is worth noting that when performing loop unrolling, if the loop unrolling factor is too small, the parallelism that can be mined is insufficient, and if the unrolling factor is too large, the register pressure will increase, and even register overflow will occur. Therefore, the selection of the loop unrolling factor is very important. The embodiment of the present invention can obtain a more accurate loop unrolling factor by introducing the device information of the back-end hardware device when calculating the loop unrolling factor, and select a more appropriate number of loop unrolling times for loop unrolling, thereby achieving further optimization of the loop code in the intermediate representation.

[0015] As an optional embodiment of the first aspect, after obtaining the intermediate representation of the deep learning model, the method further includes:

[0016] An iteration number threshold N is set for the intermediate representation. During the i-th loop iteration of the intermediate representation, the loop unrolling factor K calculated in the i-th loop iteration is determined. i and the loop unrolling factor K calculated in the i-1th loop iteration i-1 Are they the same? Where i≥2, N≥2;

[0017] If K i With K i-1 If the same, the loop iteration process ends and K i With K i-1 Any one of is used as the final loop unrolling factor;

[0018] If K i With K i-1 If they are not the same, then determine whether i is satisfied. <N;

[0019] If satisfied, the intermediate representation is iterated for the i+1th loop, and the loop expansion factor K calculated in the i+1th loop iteration is determined. i+1 and the loop unrolling factor K calculated in the i-th loop iteration i until the first loop expansion factor calculated in the current loop iteration is the same as the second loop expansion factor calculated in the previous loop iteration, the loop iteration process ends, and any one of the first loop expansion factor and the second loop expansion factor is used as the final loop expansion factor;

[0020] If not, the loop iteration process ends, and the differences of the loop expansion factors calculated in two adjacent loop iterations are respectively calculated, the minimum difference among the differences is obtained, and any one of the two loop expansion factors corresponding to the minimum difference is used as the final loop expansion factor;

[0021] Then, the intermediate representation is loop expanded according to the loop expansion factor to obtain an optimized intermediate representation, specifically:

[0022] The intermediate representation is loop-unrolled according to the final loop-unrolling factor to obtain the optimized intermediate representation.

[0023] It is worth mentioning that by using a method similar to iterative compilation to continuously update the loop expansion factor, it is possible to comprehensively consider multiple factors to select a suitable loop expansion factor and perform loop optimization, thereby optimizing the loop code in the program, tapping into data-level parallelism and instruction-level parallelism in the program, and significantly improving program performance.

[0024] As an optional embodiment of the first aspect, after obtaining the intermediate representation in the deep learning model, the method further includes:

[0025] A dynamic SCOP and a static SCOP are acquired based on the intermediate representation, the dynamic SCOP is converted into a static SCOP, and a first intermediate representation is obtained according to the acquired static SCOP and the converted static SCOP; wherein the dynamic SCOP is used to represent the program code represented by the polyhedral model, and the dynamic SCOP carries parameter information that the back-end hardware device cannot recognize when compiling the SCOP; the static SCOP is used to represent the program code represented by the polyhedral model, and the static SCOP does not carry parameter information that the back-end hardware device cannot recognize when compiling the SCOP;

[0026] Performing polyhedron optimization on the first intermediate representation to obtain a second intermediate representation;

[0027] Then, the intermediate representation is loop expanded according to the loop expansion factor to obtain an optimized intermediate representation, specifically:

[0028] The second intermediate representation is loop-unrolled according to the loop-unrolling factor to obtain an optimized intermediate representation.

[0029] It is worth noting that for loops containing runtime parameter information, runtime information is obtained through runtime optimization, and a more accurate loop unrolling factor is obtained according to the actual requirements of the program code, thereby better exerting the effectiveness of loop unrolling.

[0030] As an optional embodiment of the first aspect, converting the dynamic SCOP into a static SCOP includes:

[0031] Acquire parameter information of the dynamic SCOP when the backend hardware device is running;

[0032] According to the acquired parameter information, the parameter information carried by the dynamic SCOP and which cannot be identified by the backend hardware device when compiling the SCOP is adjusted to obtain the converted static SCOP.

[0033] It is worth noting that some loop codes in the code contain parameter information that can only be determined at runtime. The polyhedron tool can only perform static detection, and dynamic SCOP and static SCOP cannot be detected. Since more information about the actual structure of the loop can be obtained at runtime, more loops can be optimized. Therefore, in order to obtain more information and optimize more loops, runtime optimization is required to obtain better performance optimization results.

[0034] As an optional embodiment of the first aspect, performing polyhedron optimization on the first intermediate representation to obtain a second intermediate representation includes:

[0035] Performing polyhedral modeling on the first intermediate representation to obtain an iteration domain, a memory access function, and an affine schedule corresponding to the first intermediate representation; wherein the iteration domain is used to represent the multidimensional iteration space domain of each statement in the SCOP in a multi-layer loop; the memory access function is used to represent the mapping relationship between the iteration domain and an array with an affine subscript; and the affine schedule is used to represent the mapping relationship between the iteration domain and the logical execution time of each statement in the SCOP;

[0036] The first intermediate representation is optimized according to the iteration domain, memory access function and affine scheduling corresponding to the first intermediate representation to obtain an intermediate representation of the deep learning model that can be loop unrolled.

[0037] It is worth noting that compared with using loop transformation and iterative compilation separately, combining the polyhedron model with iterative compilation technology, mining the optimal loop transformation sequence and the optimal transformation parameters, and automatically performing code transformation and parameter search can achieve better performance optimization results.

[0038] In a second aspect, an embodiment of the present invention provides a data processing method, including:

[0039] Get the target code and the image to be processed;

[0040] The image to be processed is processed according to the target code to obtain an image processing result; wherein the target code is obtained by the data processing method described in any embodiment of the first aspect above.

[0041] It is worth noting that in the prior art, most of the running time of many applications is occupied by loops, especially in wireless communication and video image processing applications, which contain a large number of loop programs. Therefore, optimizing the loop code in the program can further improve the performance of the program.

[0042] In a third aspect, an embodiment of the present invention provides a data processing device, including a receiving unit and a processing unit:

[0043] The receiving unit is used to obtain an intermediate representation of a deep learning model; wherein the deep learning model is a model built based on a front-end model framework; a loop expansion factor is set for the intermediate representation; the loop expansion factor is related to information of the intermediate representation when executed by a back-end hardware device and / or device information of the back-end hardware device;

[0044] The processing unit is used to perform loop expansion on the intermediate representation according to the loop expansion factor to obtain an optimized intermediate representation;

[0045] The processing unit is further used to compile the optimized intermediate representation to obtain the target code executable by the back-end hardware device, so that the back-end hardware device executes the target code to realize the function of the target code.

[0046] As an optional embodiment of the third aspect, the information of the intermediate representation when executed on the back-end hardware device includes at least one of the M loop structures in the intermediate representation and the number of iterations corresponding to each of the M loop structures; M is a positive integer greater than 1.

[0047] As an optional embodiment of the third aspect, the device information of the backend hardware device includes at least one of a register parameter, a code volume parameter, and a functional component parameter;

[0048] Among them, the register parameter is used to indicate the number of registers contained in the back-end hardware device; the code volume parameter is used to indicate the size of the program code that the back-end hardware device can run; the functional component parameter is used to indicate the number of functional components contained in the back-end hardware device; the functional component is used to characterize the component that completes the computing function.

[0049] As an optional embodiment of the third aspect, after the receiving unit obtains the intermediate representation of the deep learning model, the processing unit is further used to:

[0050] An iteration number threshold N is set for the intermediate representation. During the i-th loop iteration of the intermediate representation, the loop unrolling factor K calculated in the i-th loop iteration is determined. iand the loop unrolling factor K calculated in the i-1th loop iteration i-1 Are they the same? Where i≥2, N≥2;

[0051] If K i With K i-1 If the same, the loop iteration process ends and K i With K i-1 Any one of is used as the final loop unrolling factor;

[0052] If K i With K i-1 If they are not the same, then determine whether i is satisfied. <N;

[0053] If satisfied, the intermediate representation is iterated for the i+1th loop, and the loop expansion factor K calculated in the i+1th loop iteration is determined. i+1 and the loop unrolling factor K calculated in the i-th loop iteration i until the first loop expansion factor calculated in the current loop iteration is the same as the second loop expansion factor calculated in the previous loop iteration, the loop iteration process ends, and any one of the first loop expansion factor and the second loop expansion factor is used as the final loop expansion factor;

[0054] If not, the loop iteration process ends, and the differences of the loop expansion factors calculated in two adjacent loop iterations are respectively calculated, the minimum difference among the differences is obtained, and any one of the two loop expansion factors corresponding to the minimum difference is used as the final loop expansion factor;

[0055] Then, the processing unit performs loop expansion on the intermediate representation according to the loop expansion factor to obtain an optimized intermediate representation, specifically:

[0056] The intermediate representation is loop-unrolled according to the final loop-unrolling factor to obtain the optimized intermediate representation.

[0057] As an optional embodiment of the third aspect, after the receiving unit obtains the intermediate representation of the deep learning model, the processing unit is further used to:

[0058] A dynamic SCOP and a static SCOP are acquired based on the intermediate representation, the dynamic SCOP is converted into a static SCOP, and a first intermediate representation is obtained according to the acquired static SCOP and the converted static SCOP; wherein the dynamic SCOP is used to represent the program code represented by the polyhedral model, and the dynamic SCOP carries parameter information that the back-end hardware device cannot recognize when compiling the SCOP; the static SCOP is used to represent the program code represented by the polyhedral model, and the static SCOP does not carry parameter information that the back-end hardware device cannot recognize when compiling the SCOP;

[0059] Performing polyhedron optimization on the first intermediate representation to obtain a second intermediate representation;

[0060] Then, the processing unit performs loop expansion on the intermediate representation according to the loop expansion factor to obtain an optimized intermediate representation, specifically:

[0061] The second intermediate representation is loop-unrolled according to the loop-unrolling factor to obtain an optimized intermediate representation.

[0062] As an optional embodiment of the third aspect, the processing unit converts the dynamic SCOP into a static SCOP, including:

[0063] Acquire parameter information of the dynamic SCOP when the backend hardware device is running;

[0064] According to the acquired parameter information, the parameter information carried by the dynamic SCOP and which cannot be identified by the backend hardware device when compiling the SCOP is adjusted to obtain the converted static SCOP.

[0065] As an optional embodiment of the third aspect, the processing unit performs polyhedron optimization on the first intermediate representation to obtain a second intermediate representation, including:

[0066] Performing polyhedral modeling on the first intermediate representation to obtain an iteration domain, a memory access function, and an affine schedule corresponding to the first intermediate representation; wherein the iteration domain is used to represent the multidimensional iteration space domain of each statement in the SCOP in a multi-layer loop; the memory access function is used to represent the mapping relationship between the iteration domain and an array with an affine subscript; and the affine schedule is used to represent the mapping relationship between the iteration domain and the logical execution time of each statement in the SCOP;

[0067] The first intermediate representation is optimized according to the iteration domain, memory access function and affine scheduling corresponding to the first intermediate representation to obtain an intermediate representation of the deep learning model that can be loop unrolled.

[0068] It should be noted that the specific implementation schemes and beneficial effects of each embodiment of a data processing device provided in the third aspect of an embodiment of the present invention are the same as the specific implementation schemes and beneficial effects of each embodiment of a data processing method provided in the first aspect of an embodiment of the present invention, and will not be elaborated here.

[0069] In a fourth aspect, an embodiment of the present invention provides a data processing device, including a receiving unit and a processing unit:

[0070] The receiving unit is used to obtain the target code and the image to be processed;

[0071] The processing unit is used to process the image to be processed according to the target code to obtain an image processing result; wherein the target code is obtained by the data processing method described in any embodiment of the first aspect above.

[0072] It is worth noting that in the prior art, most of the running time of many applications is occupied by loops, especially in wireless communication and video image processing applications, which contain a large number of loop programs. Therefore, optimizing the loop code in the program can further improve the performance of the program.

[0073] In a fifth aspect, an embodiment of the present invention provides a data processing device, comprising: a memory and a processor, wherein the memory is used to store programs, and the processor executes the programs stored in the memory. When the programs stored in the memory are executed, the processor is used to execute the data processing method described in any embodiment of the first aspect or the data processing method described in any embodiment of the second aspect.

[0074] In the sixth aspect, an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium includes a stored computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute the data processing method described in any embodiment of the first aspect above or the data processing method described in any embodiment of the second aspect above.

[0075] Compared with the prior art, the data processing method, apparatus, device and computer storage medium provided by the embodiments of the present invention can calculate the loop expansion factor through the information of the back-end hardware device during execution and / or the device information of the back-end hardware device to obtain a more accurate loop expansion factor, and use the loop expansion factor to perform loop expansion on the intermediate representation to perform instruction scheduling in a larger range and better develop instruction-level parallelism, thereby achieving further optimization of the loop code in the intermediate representation, improving the compilation optimization effect of the intermediate representation, achieving the universality of the intermediate representation that is independent of the front-end language in a variety of back-end hardware devices, and improving the portability of the intermediate representation. BRIEF DESCRIPTION OF THE DRAWINGS

[0076] Figure 1 is a flow chart of an embodiment of a data processing method provided by the first aspect of the present invention;

[0077] Figure 2 is a flow chart of an embodiment of a data processing method provided by the second aspect of the present invention;

[0078] Figure 3 is a structural schematic diagram of an embodiment of a data processing device provided by the third aspect of the present invention;

[0079] Figure 4 is a structural schematic diagram of an embodiment of a data processing device provided by the fourth aspect of the present invention;

[0080] Figure 5 It is a structural diagram of an embodiment of a data processing device provided by the fifth aspect of the present invention. DETAILED DESCRIPTION

[0081] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0082] In a first aspect, an embodiment of the present invention provides a data processing method, see Figure 1 , is a flow chart of an embodiment of a data processing method provided by the first aspect of the present invention, the method comprising steps S101 to S103:

[0083] S101. Obtain an intermediate representation of a deep learning model; wherein the deep learning model is a model built based on a front-end model framework; a loop expansion factor is set for the intermediate representation; the loop expansion factor is related to information of the intermediate representation when executed on a back-end hardware device and / or device information of the back-end hardware device.

[0084] In step S101, it is necessary to obtain an intermediate representation of a deep learning model, which is a model built based on a front-end model framework, and a loop expansion factor is set in the obtained intermediate representation, and the specific value of the loop expansion factor is related to the information of the intermediate representation when it is executed on the back-end hardware device and / or the device information of the back-end hardware device. Specifically, the loop expansion factor can be obtained by calculating only based on the information of the intermediate representation when it is executed on the back-end hardware device, or by calculating only based on the device information of the back-end hardware device, and the loop expansion factor can be obtained accordingly. The information of the intermediate representation when it is executed on the back-end hardware device can also be combined with the device information of the back-end hardware device to calculate the loop expansion factor.

[0085] It is understandable that the new wave of artificial intelligence represented by machine learning and deep learning has been heating up for several years. Deep learning has been a research hotspot in the field of artificial intelligence in recent years, and has achieved breakthrough progress in many fields. There are many deep learning frameworks and various hardware platforms that support multiple deep learning frameworks. As the application of deep learning becomes more and more widespread, people are increasingly concerned about the efficiency of deep learning algorithm training and reasoning on different hardware architectures. Since deep learning has many different front-ends and back-ends, a bridge is needed to effectively optimize and map them. IR (Intermediate Representation) is the intermediary for translation between source code and target code during program compilation. The design of IR is very critical for compilers. IR must consider the completeness of compilation from source code to target code, the ease of use of compilation optimization, and performance. Therefore, the competition for intermediate representation will be an important part of the future framework competition.

[0086] In addition, it should be noted that the loop expansion factor refers to the number of times the loop body is copied or the number of times the loop body is expanded, which determines the optimization effect of loop expansion. Among them, the timing of determining the loop expansion factor can be divided into compile time and run time. If various parameter information in the loop code of the intermediate representation can be obtained by the back-end hardware device at compile time, the loop expansion factor can be determined at compile time. If part of the loop parameter information can only be obtained at runtime through the back-end hardware device, it is necessary to collect the parameter information of the code at runtime, that is, the information of the intermediate representation when the back-end hardware device executes, and determine the loop expansion factor by iterative compilation, so as to calculate a more accurate loop expansion factor.

[0087] S102: Perform loop expansion on the intermediate representation according to the loop expansion factor to obtain an optimized intermediate representation.

[0088] In step S102, after the intermediate representation of the deep learning model and the loop expansion factor set in the intermediate representation are obtained according to step S101, the intermediate representation is optimized by loop expansion by the obtained loop expansion factor, and the optimized intermediate representation is obtained accordingly. Since loop expansion is an important loop optimization method in the field of compilation. Loop expansion refers to copying the loop body code multiple times, or extracting one or more iterations of the loop head and tail from the loop. Loop expansion is closely related to the target machine architecture, the actual situation of the loop code, the selection of the loop expansion factor and the timing of loop expansion, and many factors will affect the effect of loop expansion. Therefore, after the intermediate representation is loop expanded according to the loop expansion factor, the loop body code can be copied multiple times according to the selected loop expansion factor to expand the basic block, expand the data reorganization window, and explore more data reorganization opportunities, thereby further optimizing and improving the compilation effect of the intermediate representation.

[0089] It should be noted that there are many types of loop transformations, including loop unrolling, loop joining, loop distribution, loop separation, loop merging, loop tiling, loop filling, loop swapping, loop soft pipelining, and scalar expansion, etc. In the actual application process, the appropriate type of loop transformation can be selected according to actual needs to optimize the intermediate representation. The loop optimization method of unrolling the intermediate representation according to the loop unrolling factor described in this embodiment is only one of the implementation methods. All other embodiments obtained by ordinary technicians in this field without creative work and implemented by applying other loop transformation methods are within the scope of protection of the present invention.

[0090] It is worth noting that in actual applications, the types of intermediate representations are ever-changing, and the difficulty of optimizing the intermediate representation will increase exponentially with the growth of the scale of the expression items in the intermediate representation. Since loop unrolling can expand the basic block and perform instruction scheduling in a larger range, it can better develop instruction-level parallelism. At the same time, loop unrolling can expand the data reorganization window, discover more data reorganization opportunities, and better develop data parallelism. Therefore, loop unrolling of the intermediate representation through the loop unrolling factor can further optimize the intermediate representation and improve the compilation effect.

[0091] S103: compile the optimized intermediate representation to obtain a target code executable by the back-end hardware device, so that the back-end hardware device executes the target code to realize the function of the target code.

[0092] In step S103, after obtaining the optimized intermediate representation according to step S101 and step S102, the optimized intermediate representation is optimized into target code executable by the back-end hardware device through the compiler, so as to further improve the adaptability and general performance of the deep learning algorithm on different hardware structures and artificial intelligence development frameworks based on the optimized intermediate representation.

[0093] It is understandable that, with the increasing application of deep learning, the efficiency of deep learning algorithm training and reasoning on different hardware architectures has attracted more and more attention. As an important link in the compiler, a good intermediate representation must have the ability to accurately express the source program information and be convenient for transformation. In the process of transformation, the intermediate representation will be operated multiple times, thus becoming very complicated. Therefore, in order to improve the running performance of the compiled target code, it is often necessary to optimize the intermediate representation. Compared with the prior art, a data processing method provided by an embodiment of the present invention can calculate the loop expansion factor through the information of the back-end hardware device during execution and / or the device information of the back-end hardware device to obtain a more accurate loop expansion factor, and loop expansion of the intermediate representation through the loop expansion factor, so as to perform instruction scheduling in a larger range and better develop instruction-level parallelism, thereby achieving further optimization of the loop code in the intermediate representation, improving the compilation optimization effect of the intermediate representation, and achieving the versatility of the intermediate representation independent of the front-end language in a variety of back-end hardware devices, and improving the portability of the intermediate representation.

[0094] As an optional embodiment of the first aspect, the information of the intermediate representation when executed on the back-end hardware device includes at least one of the M loop structures in the intermediate representation and the number of iterations corresponding to each of the M loop structures; M is a positive integer greater than 1.

[0095] In this embodiment, the information of the intermediate representation when executed by the back-end hardware device includes at least one of the M loop structures in the intermediate representation and the number of iterations corresponding to each of the M loop structures. Accordingly, when calculating the loop expansion factor based on the information of the intermediate representation when executed by the back-end hardware device, the loop expansion factor can be calculated based on only the M loop structures included in the intermediate representation, or the loop expansion factor can be calculated based on only the number of iterations corresponding to each of the M loop structures included in the intermediate representation. The loop expansion factor can also be calculated by combining the M loop structures included in the intermediate representation with the number of iterations corresponding to each of the M loop structures, so that the loop expansion factor can be calculated based on the device information of the back-end hardware device to further improve the accuracy of the loop expansion factor, thereby obtaining a more optimized intermediate representation.

[0096] It is understandable that in the prior art, most of the running time of many applications is occupied by loops. Optimizing the loop code in the program can improve the performance of the program. However, the parameter information that the loop code can obtain during compilation is limited. For example, the nonlinear cache of the loop program can only obtain more parameter information of the actual structure of the loop during runtime. Therefore, in order to obtain more information and optimize more loops, it is very necessary to optimize the loop code during runtime.

[0097] It should be noted that both M and N are set according to the actual code structure of the intermediate representation and are not limited here.

[0098] It is worth noting that since the loop code accounts for a large proportion of the intermediate representation and some parameter information in the loop code cannot be determined during the program compilation stage, the loop expansion factor can be calculated by considering the parameter information that can only be determined when the program is running, that is, the information when the intermediate representation is executed on the back-end hardware device. This can obtain a more accurate loop expansion factor, thereby improving the optimization effect of the intermediate representation.

[0099] As an optional embodiment of the first aspect, the device information of the backend hardware device includes at least one of a register parameter, a code volume parameter, and a functional component parameter;

[0100] Among them, the register parameter is used to indicate the number of registers contained in the back-end hardware device; the code volume parameter is used to indicate the size of the program code that the back-end hardware device can run; the functional component parameter is used to indicate the number of functional components contained in the back-end hardware device; the functional component is used to characterize the component that completes the computing function.

[0101] In this embodiment, the device information of the back-end hardware device includes at least one of register parameters, code volume parameters, and functional component parameters. Accordingly, when calculating the loop expansion factor according to the device information of the back-end hardware device, the calculation can be performed only according to the register parameters in the device information of the back-end hardware device, or only according to the code volume parameters, or only according to the functional component parameters; or, any two parameters in the device information can be combined to calculate the loop expansion factor; or, all three parameters in the device information can be combined to calculate the loop expansion factor. In actual application, the three parameters contained in the device information can be combined in different forms according to the actual application scenarios and requirements to calculate different loop expansion factors, so as to perform targeted loop optimization.

[0102] It is understandable that the selection of loop expansion factor is very important for loop optimization. When calculating the loop expansion factor, many factors need to be considered, including register parameters, code volume parameters, and functional component parameters. Assuming that the number of registers available in the backend hardware device is NR, the register parameters need to meet the following constraints:

[0103] P reg ≤NR

[0104] 1≤R i ≤NR

[0105] Among them, P reg Indicates register parameters, R i Represents the number of variables in the i-th layer loop.

[0106] The code size parameters need to meet the following constraints:

[0107] P code ≤IC

[0108] Among them, P code Represents the code size parameter, and IC represents the instruction cache size available in the backend hardware device.

[0109] For feature parameters Assume that the backend hardware device has If there are i functional units of type i, then through instruction-level parallelism, the number of execution cycles required for such operations processed by the i functional units is: After the loop is unrolled, the execution cost of the loop body is:

[0110]

[0111] It is worth noting that when performing loop unrolling, if the loop unrolling factor is too small, the parallelism that can be mined is insufficient, and if the unrolling factor is too large, the register pressure will increase, and even register overflow will occur. Therefore, the selection of the loop unrolling factor is very important. The embodiment of the present invention can obtain a more accurate loop unrolling factor by introducing the device information of the back-end hardware device when calculating the loop unrolling factor, and select a more appropriate number of loop unrolling times for loop unrolling, thereby achieving further optimization of the loop code in the intermediate representation.

[0112] As an optional embodiment of the first aspect, after step S101, the method further includes:

[0113] An iteration number threshold N is set for the intermediate representation. During the i-th loop iteration of the intermediate representation, the loop unrolling factor K calculated in the i-th loop iteration is determined. iand the loop unrolling factor K calculated in the i-1th loop iteration i-1 Are they the same? Where i≥2, N≥2;

[0114] If K i With K i-1 If the same, the loop iteration process ends and K i With K i-1 Any one of is used as the final loop unrolling factor;

[0115] If K i With K i-1 If they are not the same, then determine whether i is satisfied. <N;

[0116] If satisfied, the intermediate representation is iterated for the i+1th loop, and the loop expansion factor K calculated in the i+1th loop iteration is determined. i+1 and the loop unrolling factor K calculated in the i-th loop iteration i until the first loop expansion factor calculated in the current loop iteration is the same as the second loop expansion factor calculated in the previous loop iteration, the loop iteration process ends, and any one of the first loop expansion factor and the second loop expansion factor is used as the final loop expansion factor;

[0117] If not, the loop iteration process ends, and the differences of the loop expansion factors calculated in two adjacent loop iterations are respectively calculated, the minimum difference among the differences is obtained, and any one of the two loop expansion factors corresponding to the minimum difference is used as the final loop expansion factor;

[0118] Then, the step S102 is specifically as follows:

[0119] The intermediate representation is loop-unrolled according to the final loop-unrolling factor to obtain the optimized intermediate representation.

[0120] In this embodiment, an iteration number threshold N is also set in the intermediate representation. The loop expansion factor calculated in each loop iteration can be further optimized through the loop iteration process of the intermediate representation. The specific calculation method of the loop expansion factor can refer to the method described in the above embodiment and will not be repeated here.

[0121] Exemplarily, assuming that i=2, the intermediate representation is in the process of the second loop iteration, and it is determined whether the loop expansion factor K2 calculated in the second loop iteration is the same as the loop expansion factor K1 calculated in the first loop iteration;

[0122] If K2 is the same as K1, the entire loop iteration process of the intermediate representation ends, and either K2 or K1 is used as the final loop unrolling factor;

[0123] If K2 is different from K1, it is judged whether the current loop iteration count i satisfies i < N;

[0124] Assume that the iteration count threshold N = 3. If the loop iteration count i is less than the iteration count threshold N, the third loop iteration is performed on the intermediate representation, and it is judged whether the loop unrolling factor K3 obtained in the third loop iteration is the same as the loop unrolling factor K2 obtained in the second loop iteration;

[0125] If K3 is the same as K2, the entire loop iteration process of the intermediate representation ends, and either K3 or K2 is used as the final loop unrolling factor;

[0126] If K3 is different from K2, it is judged whether the current loop iteration count i satisfies i < N. Since the current loop iteration count is i + 1 and i + 1 = N = 3, the loop iteration process ends, and the differences between the loop unrolling factors of K2 and K1 and K3 and K2 are calculated respectively, the minimum difference among the differences is obtained, and either one of the two loop unrolling factors corresponding to the minimum difference is used as the final loop unrolling factor.

[0127] After obtaining the final loop unrolling factor, the intermediate representation is loop unrolled according to the final loop unrolling factor obtained through loop iteration to obtain an optimized intermediate representation.

[0128] In addition, it should be noted that in the first loop iteration process, that is, in the special case of i = 1, after the loop unrolling factor is calculated in the loop optimization module, the loop unrolling factor will be recorded and passed to the polyhedron optimization module; after the polyhedron optimization module receives the loop unrolling factor and obtains the information when the intermediate representation is executed on the backend hardware device, it will perform polyhedron modeling on the loop code in the intermediate representation according to this information to obtain a program transformation space, and search the program transformation space for program transformation to obtain the intermediate representation after program transformation. Further, after obtaining the intermediate representation after program transformation, the polyhedron optimization module will pass the intermediate representation after program transformation to the loop optimization module. Subsequently, the loop optimization module will perform the next loop iteration according to the received intermediate representation, that is, perform the next round of calculation and detection of the loop unrolling factor. At this time, the loop iteration count is i + 1 = 2, and the loop iteration can be continuously performed according to the above judgment method until the loop iteration ends and the final loop unrolling factor is obtained.

[0129] It should be noted that the iteration number threshold is a positive integer greater than or equal to 2. In actual application, the size of the iteration number threshold can be set according to actual needs or experiments and is not limited here.

[0130] It is worth noting that, as one of the optimization methods of adaptive optimization, iterative compilation refers to using several different expansion factors to expand the loop, running the expanded code and collecting runtime information, and iterating repeatedly to obtain a better loop expansion factor. By using a method similar to iterative compilation to continuously update the loop expansion factor, it is possible to comprehensively consider multiple factors to select the appropriate loop expansion factor and perform loop optimization, thereby optimizing the loop code in the program, exploring the data-level parallelism and instruction-level parallelism in the program, and significantly improving program performance.

[0131] As an optional embodiment of the first aspect, after step S102, the method further includes:

[0132] A dynamic SCOP and a static SCOP are acquired based on the intermediate representation, the dynamic SCOP is converted into a static SCOP, and a first intermediate representation is obtained according to the acquired static SCOP and the converted static SCOP; wherein the dynamic SCOP is used to represent the program code represented by the polyhedral model, and the dynamic SCOP carries parameter information that the back-end hardware device cannot recognize when compiling the SCOP; the static SCOP is used to represent the program code represented by the polyhedral model, and the static SCOP does not carry parameter information that the back-end hardware device cannot recognize when compiling the SCOP;

[0133] Performing polyhedron optimization on the first intermediate representation to obtain a second intermediate representation;

[0134] Then, the step S102 is specifically as follows:

[0135] The second intermediate representation is loop-unrolled according to the loop-unrolling factor to obtain an optimized intermediate representation.

[0136] In this embodiment, after obtaining the optimized intermediate representation according to step S102, it is necessary to detect the type of SCOP contained in the intermediate representation, wherein the type of SCOP includes dynamic SCOP and static SCOP, both of which are used to characterize the program code represented by the polyhedral model, and the dynamic SCOP carries parameter information that the back-end hardware device cannot recognize when compiling the SCOP, and the static SCOP does not carry parameter information that the back-end hardware device cannot recognize when compiling the SCOP; further, if it is detected in the intermediate representation that there is no parameter information that the back-end hardware device cannot recognize when compiling the SCOP, it means that the intermediate representation only contains static SCOP and does not contain dynamic SCOP, and the intermediate representation can be directly optimized; and if it is detected in the intermediate representation that there is parameter information that the back-end hardware device cannot recognize when compiling the SCOP, such as parameter information such as pointer alias and array, it means that the intermediate representation contains dynamic SCOP. At this time, it is necessary to convert the detected dynamic SCOP into static SCOP, and after detecting that only the static SCOP exists in the intermediate representation, the first intermediate representation is obtained according to the static SCOP originally existing in the intermediate representation and the converted static SCOP;

[0137] Further, after obtaining the first intermediate representation, the first intermediate representation is subjected to polyhedral optimization to obtain an optimized intermediate representation, i.e., a second intermediate representation, and the second intermediate representation is subjected to loop expansion according to the loop expansion factor calculated by the method described in the above embodiment to obtain an optimized intermediate representation. It can be understood that SCOP refers to a program segment that can be represented by a polyhedron. Static SCOP refers to a program segment that can be represented by a basic polyhedron model and can be analyzed and transformed during compilation, and the static SCOP can be directly detected by existing polyhedron tools. Dynamic SCOP refers to a program segment that can be represented by a basic polyhedron model, but requires runtime information to detect this type of SCOP, and cannot be directly detected by polyhedron tools.

[0138] It is worth noting that for loops containing runtime parameter information, runtime information is obtained through runtime optimization, and a more accurate loop unrolling factor is obtained according to the actual requirements of the program code, thereby better exerting the effectiveness of loop unrolling.

[0139] As an optional embodiment of the first aspect, converting the dynamic SCOP into a static SCOP includes:

[0140] Acquire parameter information of the dynamic SCOP when the backend hardware device is running;

[0141] According to the acquired parameter information, the parameter information carried by the dynamic SCOP and which cannot be identified by the backend hardware device when compiling the SCOP is adjusted to obtain the converted static SCOP.

[0142] In this embodiment, after the presence of a dynamic SCOP is detected in the intermediate representation, the dynamic SCOP needs to be converted into a static SCOP. Specifically, since some parameter information contained in the dynamic SCOP can only be recognized at runtime, it can only be replaced by parameter names at the compilation stage. Therefore, if the parameter value corresponding to the parameter information is obtained at the stage when the back-end hardware device runs the SCOP, the parameter value can be substituted into the source code corresponding to the intermediate representation to obtain the known parameter value corresponding to the parameter information. Further, through these known parameter values, the dynamic SCOP in the source code corresponding to the intermediate representation can be converted into a static SCOP, thereby adjusting the parameter information that the back-end hardware device cannot recognize when compiling the SCOP when carrying the dynamic SCOP, and obtaining the converted static SCOP.

[0143] It is worth noting that some loop codes in the code contain parameter information that can only be determined at runtime. The polyhedron tool can only perform static detection, and dynamic SCOP and static SCOP cannot be detected. Since more information about the actual structure of the loop can be obtained at runtime, more loops can be optimized. Therefore, in order to obtain more information and optimize more loops, runtime optimization is required to obtain better performance optimization results.

[0144] As an optional embodiment of the first aspect, performing polyhedron optimization on the first intermediate representation to obtain a second intermediate representation includes:

[0145] Performing polyhedral modeling on the first intermediate representation to obtain an iteration domain, a memory access function, and an affine schedule corresponding to the first intermediate representation; wherein the iteration domain is used to represent the multidimensional iteration space domain of each statement in the SCOP in a multi-layer loop; the memory access function is used to represent the mapping relationship between the iteration domain and an array with an affine subscript; and the affine schedule is used to represent the mapping relationship between the iteration domain and the logical execution time of each statement in the SCOP;

[0146] The first intermediate representation is optimized according to the iteration domain, memory access function and affine scheduling corresponding to the first intermediate representation to obtain an intermediate representation of the deep learning model that can be loop unrolled.

[0147] In this embodiment, the process of performing polyhedral optimization on the intermediate representation mainly includes loop analysis and detection, parameter reception, polyhedral modeling, and program transformation. Specifically, the polyhedral optimization module first analyzes and detects the type of SCOP contained in the intermediate representation, and after obtaining the first intermediate representation through the method of the above embodiment, obtains the information of the intermediate representation when it is executed by the backend hardware device and receives the loop expansion factor transmitted by the loop optimization module, and performs polyhedral modeling on the loop code in the intermediate representation based on this information to obtain the iteration domain, memory access function and affine scheduling of each statement. After completing the polyhedral modeling, the intermediate representation that has completed the polyhedral modeling is subjected to program transformation to obtain the program transformation space, and the program transformation space is searched and program transformation is performed, and the intermediate representation that has undergone program transformation is the second intermediate representation after polyhedral optimization.

[0148] It should be noted that several commonly used basic concepts in the polyhedron model are explained here, as follows:

[0149] Affine function: If with respect to one or more variables x1,x2,x3,…,x n The function can be expressed as a constant plus a constant times the sum of these variables, that is, c0x1+c1x2+c2x3+…+c n x n , where c1, c2, c3, …, c n are all constants, then the function is affine.

[0150] Convex polyhedron: The iteration space of a loop nested structure is defined as the combination of all loop subscript variable references in the nested structure. A loop nested structure with a depth of d can be modeled as a d-dimensional space. The dimensions of the space are ordered, and the kth dimension represents the kth loop from the outermost loop in the nested structure. A point in this space (x1, x2, x3, ..., x d ) represents the value of all loop subscripts. The value of the outermost loop subscript is x1, the value of the second loop subscript is x2, and so on. The value of the innermost loop subscript is x d。Not every point in this space represents a subscript access combination that actually occurs during the execution of the loop nest structure. As an affine function of the outer loop subscript, the upper and lower bounds of each loop define an inequality that divides the space into two halves: the part corresponding to loop iterations (i.e., the positive half-space) and the part not corresponding to iterations (i.e., the negative half-space). The intersection (logical AND) of all linear inequalities represents the intersection of these positive half-spaces, which defines a convex polyhedron. This convex polyhedron is called the iteration space of this loop nest structure. A convex polyhedron has the following property: if two points are within the polyhedron, then all points on the line connecting them are also within the polyhedron. Using the loop bound inequalities to describe the polyhedron, each iteration of the loop can be represented by a point with integer coordinates within the polyhedron, and each integer point within the polyhedron represents an iteration that the loop nest structure executes at some point.

[0151] Iteration domain: An execution of a statement is called a statement instance. A statement outside the loop has only one instance, while a statement within the loop may have multiple instances. The iteration domain in a loop nest of depth d is represented as: where \(Z\) represents the set of integers, \(A\) is a \(d\times d\) integer matrix, is an integer vector of length d, and is a vector consisting of d zeros.

[0152] Memory access function:

[0153] Affine scheduling:

[0154] Furthermore, it can be further understood that before any program transformation, the original program and its transformation must be analyzed in depth to ensure that this transformation does not change the semantics of the original program. In the polyhedral model, if all memory accesses are through arrays with affine subscripts, and the affine subscripts depend on the outer loop iteration factors and global parameters, then precise analysis can be performed on it.

[0155] Exemplarily, the iteration domain is mapped to a multi-dimensional vector representing the logical execution time of the statement to record the logical execution time of the statement, and the lexicographical order is used to compare the execution order of the statements. The affine scheduling is represented as: A loop nest structure executes each iteration in its iteration space in ascending lexicographical order. A vector is lexicographically less than another vector denoted as \(i < i'\) if and only if there exists an \(m < \min(n, n')\) such that \(m\) can be equal to 0.

[0156] The linear inequalities extracted from the lower and upper bounds of a loop body define a set of iterations on a convex polyhedron. This representation does not assume any execution order between iterations in the iteration domain. The original program imposes a serial order on the iterations, i.e., a lexicographic ordering of the loop index variables from the outside to the inside. However, the iterations in this space can be executed in any order as long as the data dependencies between them are respected (i.e., the order of write / read operations on any array element performed by different assignment statements in the loop nesting structure does not change). How to choose an order that respects data dependencies while optimizing data locality and parallelism is a complex problem. If no ordering information is provided, it means that statement instances may be executed in any order, but some statement instances may be dependent on other statement instances, and in this case, it is critical to execute them in a given order, so additional information is required. When there is a dependency between two statement instances, the program transformation cannot violate the dependency, otherwise it will cause the dependent statement instances to be executed in the wrong order or in parallel. In order to be able to calculate data dependencies, all domain, scheduling, and access constraints must be affine functions.

[0157] For example, if the scheduling function is θ s (i,j)=(j,i), then the corresponding transformation is cyclic exchange.

[0158] Multidimensional affine scheduling is very useful for scheduling the original program. The idea is to build an abstract syntax tree of the program and read in the schedule for each statement. These schedules depend on the iteration factor and give the execution order of each instance of each statement. Using such affine schedules, new code can be regenerated.

[0159] It is worth noting that compared with using loop transformation and iterative compilation separately, combining the polyhedron model with iterative compilation technology, mining the optimal loop transformation sequence and the optimal transformation parameters, and automatically performing code transformation and parameter search can achieve better performance optimization results.

[0160] In specific applications, taking image processing as an example, in the second aspect, the embodiment of the present invention provides a data processing method, see Figure 2 , is a flow chart of an embodiment of a data processing method provided by the second aspect of the present invention, wherein the data processing method comprises steps S201 to S202:

[0161] S201, obtaining a target code and an image to be processed;

[0162] S202. Process the image to be processed according to the target code to obtain an image processing result; wherein the target code is obtained by the data processing method described in any embodiment of the first aspect above.

[0163] Specifically, this embodiment can be executed by a graphics processing unit (GPU). After obtaining the target code and the image to be processed, the image processor processes the image to be processed according to the target code, and obtains the image processing result accordingly. The target code can be obtained by optimizing the data processing method described in any of the above embodiments, and the image processing can include but is not limited to any one of convolution processing, classification processing and grayscale processing. By optimizing the intermediate representation and then compiling to obtain the target code, the running efficiency of the target code can be improved, thereby improving the image processing efficiency.

[0164] It is understandable that the new wave of AI enthusiasm represented by machine learning and deep learning has been heating up for several years. Deep learning has been a research hotspot in the field of AI in recent years, and has achieved breakthroughs in many fields. There are many deep learning frameworks and hardware platforms that support multiple deep learning frameworks. The diversity of deep learning frameworks and hardware brings great benefits to users and is crucial to maintaining the healthy development of the AI ​​ecosystem, but supporting multiple frameworks and hardware requires a huge workload, which also brings considerable challenges to AI developers. Now deep learning has many different front ends and many different back ends, and a bridge is needed to effectively optimize and map them. As the application of deep learning becomes more and more widespread, people are increasingly concerned about the efficiency of deep learning algorithms in training and reasoning on different hardware architectures. The competition for intermediate representation will be an important part of the future framework competition.

[0165] It is worth noting that in the prior art, most of the running time of many applications is occupied by loops, especially in wireless communication and video image processing applications, which contain a large number of loop programs. Therefore, optimizing the loop code in the program can further improve the performance of the program.

[0166] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should know that the present disclosure is not limited by the order of the actions described, because according to the present disclosure, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily required by the present disclosure.

[0167] It should be further explained that Figure 1 For example, although Figure 1The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 1 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0168] Correspondingly, in a third aspect, an embodiment of the present invention further provides a data processing device capable of implementing all processes of the data processing method in the above embodiment.

[0169] See also Figure 3 , is a schematic diagram of the structure of an embodiment of a data processing device provided by the third aspect of the present invention, wherein the data processing device comprises a receiving unit 301 and a processing unit 302:

[0170] The receiving unit 301 is used to obtain an intermediate representation of a deep learning model; wherein the deep learning model is a model built based on a front-end model framework; a loop expansion factor is set for the intermediate representation; the loop expansion factor is related to information of the intermediate representation when executed by a back-end hardware device and / or device information of the back-end hardware device;

[0171] The processing unit 302 is used to perform loop expansion on the intermediate representation according to the loop expansion factor to obtain an optimized intermediate representation;

[0172] The processing unit 302 is further used to compile the optimized intermediate representation to obtain a target code executable by the back-end hardware device, so that the back-end hardware device executes the target code to implement the function of the target code.

[0173] It is worth noting that, compared with the prior art, a data processing device provided by an embodiment of the present invention can calculate the loop expansion factor through the information of the back-end hardware device during execution and / or the device information of the back-end hardware device to obtain a more accurate loop expansion factor, and perform loop expansion on the intermediate representation through the loop expansion factor to perform instruction scheduling in a larger range and better develop instruction-level parallelism, thereby achieving further optimization of the loop code in the intermediate representation, improving the compilation optimization effect of the intermediate representation, achieving the universality of the intermediate representation that is independent of the front-end language in a variety of back-end hardware devices, and improving the portability of the intermediate representation.

[0174] As an optional embodiment of the third aspect, the information of the intermediate representation when executed on the back-end hardware device includes at least one of the M loop structures in the intermediate representation and the number of iterations corresponding to each of the M loop structures; M is a positive integer greater than 1.

[0175] As an optional embodiment of the third aspect, the device information of the backend hardware device includes at least one of a register parameter, a code volume parameter, and a functional component parameter;

[0176] Among them, the register parameter is used to indicate the number of registers contained in the back-end hardware device; the code volume parameter is used to indicate the size of the program code that the back-end hardware device can run; the functional component parameter is used to indicate the number of functional components contained in the back-end hardware device; the functional component is used to characterize the component that completes the computing function.

[0177] As an optional embodiment of the third aspect, after the receiving unit obtains the intermediate representation of the deep learning model, the processing unit 302 is further used to:

[0178] An iteration number threshold N is set for the intermediate representation. During the i-th loop iteration of the intermediate representation, the loop unrolling factor K calculated in the i-th loop iteration is determined. i and the loop unrolling factor K calculated in the i-1th loop iteration i-1 Are they the same? Where i≥2, N≥2;

[0179] If K i With K i-1 If the same, the loop iteration process ends and K i With K i-1 Any one of is used as the final loop unrolling factor;

[0180] If K i With K i-1 If they are not the same, then determine whether i is satisfied. <N;

[0181] If satisfied, the intermediate representation is iterated for the i+1th loop, and the loop expansion factor K calculated in the i+1th loop iteration is determined. i+1 and the loop unrolling factor K calculated in the i-th loop iteration i until the first loop expansion factor calculated in the current loop iteration is the same as the second loop expansion factor calculated in the previous loop iteration, the loop iteration process ends, and any one of the first loop expansion factor and the second loop expansion factor is used as the final loop expansion factor;

[0182] If not, the loop iteration process ends, and the differences of the loop expansion factors calculated in two adjacent loop iterations are respectively calculated, the minimum difference among the differences is obtained, and any one of the two loop expansion factors corresponding to the minimum difference is used as the final loop expansion factor;

[0183] Then, the processing unit 302 performs loop expansion on the intermediate representation according to the loop expansion factor to obtain an optimized intermediate representation, which is specifically:

[0184] The intermediate representation is loop-unrolled according to the final loop-unrolling factor to obtain the optimized intermediate representation.

[0185] As an optional embodiment of the third aspect, after the receiving unit obtains the intermediate representation of the deep learning model, the processing unit 302 is further used to:

[0186] A dynamic SCOP and a static SCOP are acquired based on the intermediate representation, the dynamic SCOP is converted into a static SCOP, and a first intermediate representation is obtained according to the acquired static SCOP and the converted static SCOP; wherein the dynamic SCOP is used to represent the program code represented by the polyhedral model, and the dynamic SCOP carries parameter information that the back-end hardware device cannot recognize when compiling the SCOP; the static SCOP is used to represent the program code represented by the polyhedral model, and the static SCOP does not carry parameter information that the back-end hardware device cannot recognize when compiling the SCOP;

[0187] Performing polyhedron optimization on the first intermediate representation to obtain a second intermediate representation;

[0188] Then, the processing unit 302 performs loop expansion on the intermediate representation according to the loop expansion factor to obtain an optimized intermediate representation, which is specifically:

[0189] The second intermediate representation is loop-unrolled according to the loop-unrolling factor to obtain an optimized intermediate representation.

[0190] As an optional embodiment of the third aspect, the processing unit 302 converts the dynamic SCOP into a static SCOP, including:

[0191] Acquire parameter information of the dynamic SCOP when the backend hardware device is running;

[0192] According to the acquired parameter information, the parameter information carried by the dynamic SCOP and which cannot be identified by the backend hardware device when compiling the SCOP is adjusted to obtain the converted static SCOP.

[0193] As an optional embodiment of the third aspect, the processing unit 302 performs polyhedron optimization on the first intermediate representation to obtain a second intermediate representation, including:

[0194] Performing polyhedral modeling on the first intermediate representation to obtain an iteration domain, a memory access function, and an affine schedule corresponding to the first intermediate representation; wherein the iteration domain is used to represent the multidimensional iteration space domain of each statement in the SCOP in a multi-layer loop; the memory access function is used to represent the mapping relationship between the iteration domain and an array with an affine subscript; and the affine schedule is used to represent the mapping relationship between the iteration domain and the logical execution time of each statement in the SCOP;

[0195] The first intermediate representation is optimized according to the iteration domain, memory access function and affine scheduling corresponding to the first intermediate representation to obtain an intermediate representation of the deep learning model that can be loop unrolled.

[0196] In addition, it should be noted that the specific implementation schemes and beneficial effects of each embodiment of a data processing device provided in the third aspect of an embodiment of the present invention are the same as the specific implementation schemes and beneficial effects of each embodiment of a data processing method provided in the first aspect of an embodiment of the present invention, and will not be repeated here.

[0197] In specific applications, this embodiment takes image processing as an example. In a fourth aspect, the embodiment of the present invention provides a data processing device, see Figure 4 , is a schematic diagram of a structure of an embodiment of a data processing device provided by the fourth aspect of the present invention. The data processing device includes a receiving unit 401 and a processing unit 402;

[0198] The receiving unit 401 is used to obtain the target code and the image to be processed;

[0199] The processing unit 402 is used to process the image to be processed according to the target code to obtain an image processing result; wherein the target code is obtained by the data processing method described in any embodiment of the first aspect above.

[0200] Specifically, this embodiment can be executed by a graphics processing unit (GPU). After obtaining the target code and the image to be processed, the image processor processes the image to be processed according to the target code, and obtains the image processing result accordingly. The target code can be obtained by optimizing the data processing method described in any of the above embodiments, and the image processing can include but is not limited to any one of convolution processing, classification processing and grayscale processing. By optimizing the intermediate representation and then compiling to obtain the target code, the running efficiency of the target code can be improved, thereby improving the image processing efficiency.

[0201] It is understandable that the new wave of AI enthusiasm represented by machine learning and deep learning has been heating up for several years. Deep learning has been a research hotspot in the field of AI in recent years, and has achieved breakthroughs in many fields. There are many deep learning frameworks and hardware platforms that support multiple deep learning frameworks. The diversity of deep learning frameworks and hardware brings great benefits to users and is crucial to maintaining the healthy development of the AI ​​ecosystem, but supporting multiple frameworks and hardware requires a huge workload, which also brings considerable challenges to AI developers. Now deep learning has many different front ends and many different back ends, and a bridge is needed to effectively optimize and map them. As the application of deep learning becomes more and more widespread, people are increasingly concerned about the efficiency of deep learning algorithms in training and reasoning on different hardware architectures. The competition for intermediate representation will be an important part of the future framework competition.

[0202] It is worth noting that in the prior art, most of the running time of many applications is occupied by loops, especially in wireless communication and video image processing applications, which contain a large number of loop programs. Therefore, optimizing the loop code in the program can further improve the performance of the program.

[0203] In addition, it should be noted that the specific implementation schemes and beneficial effects of each embodiment of a data processing device provided in the fourth aspect of an embodiment of the present invention are the same as the specific implementation schemes and beneficial effects of each embodiment of a data processing method provided in the second aspect of an embodiment of the present invention, and will not be repeated here.

[0204] In a fifth aspect, an embodiment of the present invention provides a data processing device, see Figure 5 , is a schematic diagram of the structure of an embodiment of a data processing device provided by the fifth aspect of the present invention. The data processing device includes a memory 501 and a processor 502, the memory 501 is used to store programs, and the processor 502 executes the programs stored in the memory 501. When the program stored in the memory 501 is executed, the processor 502 is used to execute the data processing method described in any embodiment of the first aspect or the data processing method described in the embodiment of the second aspect.

[0205] As an optional embodiment, the computer program may be divided into one or more modules / units (such as computer program 1, computer program 2, ...), and the one or more modules / units are stored in the memory 501 and executed by the processor 502 to complete the present invention. The one or more modules / units may be a series of computer program instruction segments capable of completing specific functions, and the instruction segments are used to describe the execution process of the computer program in the data processing device.

[0206] The processor 502 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor, or the processor 502 may be any conventional processor. The processor 502 is the control center of the data processing device, and various parts of the data processing device are connected using various interfaces and lines.

[0207] The memory 501 mainly includes a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function, etc., and the data storage area can store related data, etc. In addition, the memory 501 can be a high-speed random access memory, or a non-volatile memory, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, and a flash card (Flash Card), etc., or the memory 501 can also be other volatile solid-state storage devices.

[0208] It should be noted that the above data processing device may include, but is not limited to, a processor and a memory. Those skilled in the art will understand that Figure 5 The structural diagram is only an example of the above-mentioned data processing device and does not constitute a limitation of the above-mentioned data processing device. It may include more or less components than shown in the figure, or combine certain components, or different components.

[0209] In the sixth aspect, an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium includes a stored computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute the data processing method described in any embodiment of the first aspect above or the data processing method described in any embodiment of the second aspect above.

[0210] If the various component modules of the above-mentioned device are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer product is stored in a computer-readable storage medium.

[0211] The computer-readable storage medium may be an internal storage unit of the device of the aforementioned embodiment, such as a hard disk or a memory. The computer-readable storage medium may also be an external storage device of the device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. Further, the computer-readable storage medium may also include both an internal storage unit of the device and an external storage device. The computer-readable storage medium is used to store the computer program and other programs and data required by the device. The computer-readable storage medium may also be used to temporarily store data that has been output or is to be output.

[0212] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing related hardware through a computer program, and the computer program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. The aforementioned storage medium includes: ROM, RAM, magnetic disk or optical disk and other media that can store program codes.

[0213] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0214] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0215] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0216] The above-mentioned integrated unit implemented in the form of a software functional unit can be stored in a computer-readable storage medium. The above-mentioned software functional unit is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor (English: processor) to perform some steps of the methods of each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (English: Read-Only Memory, abbreviated: ROM), random access memory (English: Random Access Memory, abbreviated: RAM), disk or optical disk and other media that can store program codes.

[0217] The above is a specific implementation of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A data processing method, characterized in that: include: Obtain an intermediate representation of a deep learning model; wherein the deep learning model is a model built based on a front-end model framework; a loop expansion factor is set for the intermediate representation; the loop expansion factor is related to information of the intermediate representation when executed by a back-end hardware device and / or device information of the back-end hardware device; the loop expansion factor is used to characterize the number of times a loop body contained in the intermediate representation is copied or the number of times a loop body is expanded; the timing for determining the loop expansion factor is divided into compile time and run time, if various parameter information in the loop code of the intermediate representation can be obtained by the back-end hardware device at compile time, then the loop expansion factor is determined at compile time; if part of the parameter information in the loop code of the intermediate representation can only be obtained by the back-end hardware device at run time, then the information of the intermediate representation when executed by the back-end hardware device is collected, and the loop expansion factor is determined by iterative compilation; Performing loop expansion on the intermediate representation according to the loop expansion factor to obtain an optimized intermediate representation; The optimized intermediate representation is compiled to obtain a target code executable by the back-end hardware device, so that the back-end hardware device executes the target code to realize the function of the target code.

2. The data processing method according to claim 1, characterized in that: The information of the intermediate representation when executed by the backend hardware device includes at least one of the M loop structures in the intermediate representation and the number of iterations corresponding to each of the M loop structures; M is a positive integer greater than 1.

3. The data processing method according to claim 1, characterized in that: The device information of the backend hardware device includes at least one of register parameters, code volume parameters, and function component parameters; Among them, the register parameter is used to indicate the number of registers contained in the back-end hardware device; the code volume parameter is used to indicate the size of the program code that the back-end hardware device can run; the functional component parameter is used to indicate the number of functional components contained in the back-end hardware device; the functional component is used to characterize the component that completes the computing function.

4. The data processing method according to any one of claims 1 to 3, characterized in that: After obtaining the intermediate representation of the deep learning model, the method further includes: An iteration number threshold N is set for the intermediate representation. During the i-th loop iteration of the intermediate representation, the loop unrolling factor K calculated in the i-th loop iteration is determined. i and the loop unrolling factor K calculated in the i-1th loop iteration i-1 Are they the same? Where i≥2, N≥2; If K i With K i-1 If the same, the loop iteration process ends and K i With K i-1 Any one of is used as the final loop unrolling factor; If K i With K i-1 If they are not the same, then determine whether i is satisfied. <N; If satisfied, the intermediate representation is iterated for the i+1th loop, and the loop expansion factor K calculated in the i+1th loop iteration is determined. i+1 and the loop unrolling factor K calculated in the i-th loop iteration i until the first loop expansion factor calculated in the current loop iteration is the same as the second loop expansion factor calculated in the previous loop iteration, the loop iteration process ends, and any one of the first loop expansion factor and the second loop expansion factor is used as the final loop expansion factor; If not, the loop iteration process ends, and the differences of the loop expansion factors calculated in two adjacent loop iterations are respectively calculated, the minimum difference among the differences is obtained, and any one of the two loop expansion factors corresponding to the minimum difference is used as the final loop expansion factor; Then, the intermediate representation is loop expanded according to the loop expansion factor to obtain an optimized intermediate representation, specifically: The intermediate representation is loop-unrolled according to the final loop-unrolling factor to obtain the optimized intermediate representation.

5. The data processing method according to claim 1, characterized in that: After acquiring the intermediate representation in the deep learning model, the method further includes: A dynamic SCOP and a static SCOP are acquired based on the intermediate representation, the dynamic SCOP is converted into a static SCOP, and a first intermediate representation is obtained according to the acquired static SCOP and the converted static SCOP; wherein the dynamic SCOP is used to represent the program code represented by the polyhedral model, and the dynamic SCOP carries parameter information that the back-end hardware device cannot recognize when compiling the SCOP; the static SCOP is used to represent the program code represented by the polyhedral model, and the static SCOP does not carry parameter information that the back-end hardware device cannot recognize when compiling the SCOP; Performing polyhedron optimization on the first intermediate representation to obtain a second intermediate representation; Then, the intermediate representation is loop expanded according to the loop expansion factor to obtain an optimized intermediate representation, specifically: The second intermediate representation is loop-unrolled according to the loop-unrolling factor to obtain an optimized intermediate representation.

6. The data processing method according to claim 5, characterized in that: The converting the dynamic SCOP into a static SCOP comprises: Acquire parameter information of the dynamic SCOP when the backend hardware device is running; According to the acquired parameter information, the parameter information carried by the dynamic SCOP and which cannot be identified by the backend hardware device when compiling the SCOP is adjusted to obtain the converted static SCOP.

7. The data processing method according to claim 5, characterized in that: The performing polyhedron optimization on the first intermediate representation to obtain a second intermediate representation includes: Performing polyhedral modeling on the first intermediate representation to obtain an iteration domain, a memory access function, and an affine schedule corresponding to the first intermediate representation; wherein the iteration domain is used to represent the multidimensional iteration space domain of each statement in the SCOP in a multi-layer loop; the memory access function is used to represent the mapping relationship between the iteration domain and an array with an affine subscript; and the affine schedule is used to represent the mapping relationship between the iteration domain and the logical execution time of each statement in the SCOP; The first intermediate representation is optimized according to the iteration domain, memory access function and affine scheduling corresponding to the first intermediate representation to obtain an intermediate representation of the deep learning model that can be loop unrolled.

8. A data processing method, characterized in that: include: Get the target code and the image to be processed; The image to be processed is processed according to the target code to obtain an image processing result; wherein the target code is obtained by the data processing method according to any one of claims 1 to 7.

9. A data processing device, characterized in that: Including receiving unit and processing unit: The receiving unit is used to obtain an intermediate representation of a deep learning model; wherein the deep learning model is a model built based on a front-end model framework; a loop expansion factor is set for the intermediate representation; the loop expansion factor is related to information of the intermediate representation when executed by a back-end hardware device and / or device information of the back-end hardware device; the loop expansion factor is used to characterize the number of times a loop body contained in the intermediate representation is copied or the number of times a loop body is expanded; the timing for determining the loop expansion factor is divided into compile time and run time, if various parameter information in the loop code of the intermediate representation can be obtained by the back-end hardware device at compile time, then the loop expansion factor is determined at compile time; if part of the parameter information in the loop code of the intermediate representation can only be obtained by the back-end hardware device at run time, then the information of the intermediate representation when executed by the back-end hardware device is collected, and the loop expansion factor is determined by iterative compilation; The processing unit is used to perform loop expansion on the intermediate representation according to the loop expansion factor to obtain an optimized intermediate representation; The processing unit is further used to compile the optimized intermediate representation to obtain the target code executable by the back-end hardware device, so that the back-end hardware device executes the target code to realize the function of the target code.

10. The data processing device according to claim 9, characterized in that: The information of the intermediate representation when executed by the backend hardware device includes at least one of the M loop structures in the intermediate representation and the number of iterations corresponding to each of the M loop structures; M is a positive integer greater than 1.

11. The data processing device according to claim 9, characterized in that: The device information of the backend hardware device includes at least one of register parameters, code volume parameters, and function component parameters; Among them, the register parameter is used to indicate the number of registers contained in the back-end hardware device; the code volume parameter is used to indicate the size of the program code that the back-end hardware device can run; the functional component parameter is used to indicate the number of functional components contained in the back-end hardware device; the functional component is used to characterize the component that completes the computing function.

12. The data processing device according to any one of claims 9 to 11, characterized in that: After the receiving unit obtains the intermediate representation of the deep learning model, the processing unit is further used to: An iteration number threshold N is set for the intermediate representation. During the i-th loop iteration of the intermediate representation, the loop unrolling factor K calculated in the i-th loop iteration is determined. i and the loop unrolling factor K calculated in the i-1th loop iteration i-1 Are they the same? Where i≥2, N≥2; If K i With K i-1 If the same, the loop iteration process ends and K i With K i-1 Any one of is used as the final loop unrolling factor; If K i With K i-1 If they are not the same, then determine whether i is satisfied. <N; If satisfied, the intermediate representation is iterated for the i+1th loop, and the loop expansion factor K calculated in the i+1th loop iteration is determined. i+1 and the loop unrolling factor K calculated in the i-th loop iteration i until the first loop expansion factor calculated in the current loop iteration is the same as the second loop expansion factor calculated in the previous loop iteration, the loop iteration process ends, and any one of the first loop expansion factor and the second loop expansion factor is used as the final loop expansion factor; If not, the loop iteration process ends, and the differences of the loop expansion factors calculated in two adjacent loop iterations are respectively calculated, the minimum difference among the differences is obtained, and any one of the two loop expansion factors corresponding to the minimum difference is used as the final loop expansion factor; Then, the processing unit performs loop expansion on the intermediate representation according to the loop expansion factor to obtain an optimized intermediate representation, specifically: The intermediate representation is loop-unrolled according to the final loop-unrolling factor to obtain the optimized intermediate representation.

13. The data processing device according to claim 9, characterized in that: After the receiving unit obtains the intermediate representation of the deep learning model, the processing unit is further used to: A dynamic SCOP and a static SCOP are acquired based on the intermediate representation, the dynamic SCOP is converted into a static SCOP, and a first intermediate representation is obtained according to the acquired static SCOP and the converted static SCOP; wherein the dynamic SCOP is used to represent the program code represented by the polyhedral model, and the dynamic SCOP carries parameter information that the back-end hardware device cannot recognize when compiling the SCOP; the static SCOP is used to represent the program code represented by the polyhedral model, and the static SCOP does not carry parameter information that the back-end hardware device cannot recognize when compiling the SCOP; Performing polyhedron optimization on the first intermediate representation to obtain a second intermediate representation; Then, the processing unit performs loop expansion on the intermediate representation according to the loop expansion factor to obtain an optimized intermediate representation, specifically: The second intermediate representation is loop-unrolled according to the loop-unrolling factor to obtain an optimized intermediate representation.

14. The data processing device according to claim 13, characterized in that: The processing unit converts the dynamic SCOP into a static SCOP, including: Acquire parameter information of the dynamic SCOP when the backend hardware device is running; According to the acquired parameter information, the parameter information carried by the dynamic SCOP and which cannot be identified by the backend hardware device when compiling the SCOP is adjusted to obtain the converted static SCOP.

15. The data processing device according to claim 13, characterized in that: The processing unit performs polyhedron optimization on the first intermediate representation to obtain a second intermediate representation, including: Performing polyhedral modeling on the first intermediate representation to obtain an iteration domain, a memory access function, and an affine schedule corresponding to the first intermediate representation; wherein the iteration domain is used to represent the multidimensional iteration space domain of each statement in the SCOP in a multi-layer loop; the memory access function is used to represent the mapping relationship between the iteration domain and an array with an affine subscript; and the affine schedule is used to represent the mapping relationship between the iteration domain and the logical execution time of each statement in the SCOP; The first intermediate representation is optimized according to the iteration domain, memory access function and affine scheduling corresponding to the first intermediate representation to obtain an intermediate representation of the deep learning model that can be loop unrolled.

16. A data processing device, characterized in that: Including receiving unit and processing unit: The receiving unit is used to obtain the target code and the image to be processed; The processing unit is used to process the image to be processed according to the target code to obtain an image processing result; wherein the target code is obtained by the data processing method according to any one of claims 1 to 7.

17. A data processing device, characterized in that: include: A memory and a processor, wherein the memory is used to store a program, and the processor executes the program stored in the memory. When the program stored in the memory is executed, the processor is used to execute the data processing method according to any one of claims 1 to 7 or the data processing method according to claim 8.

18. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored computer program, wherein when the computer program is executed, the device where the computer-readable storage medium is located is controlled to execute the data processing method according to any one of claims 1 to 7 or the data processing method according to claim 8.

Images