A data processing method, device, equipment and computer storage medium

Abstract

The present invention discloses a data processing method, device, equipment and computer storage medium, the method comprising: obtaining a first dynamic intermediate representation of a deep learning model, the first dynamic intermediate representation comprising at least one computational subgraph, the computational subgraph comprising N nodes and directed edges connecting different nodes, the directed edges carrying dynamic tensor data; screening at least one computational subgraph according to the runtime corresponding to each of the at least one computational subgraphs, obtaining at least one computational subgraph to be optimized; optimizing at least one computational subgraph to be optimized, obtaining a second dynamic intermediate representation; compiling the second dynamic intermediate representation, obtaining a target code executable by a back-end hardware device, and the back-end hardware device executing the target code to implement the function of the target code. The present invention can achieve front-end framework independence and back-end platform independence by optimizing the dynamic intermediate representation, thereby improving the portability of the intermediate representation to be applicable to a variety of hardware platforms.

Description

Technical Field

[0001] The present invention relates to the field of computer software technology, and in particular to a data processing method, device, equipment and computer storage medium. Background Art

[0002] The new wave of AI enthusiasm represented by machine learning and deep learning has been heating up for several years. Deep learning has been a research hotspot in the field of AI in recent years and has achieved breakthrough progress in many fields. There are many deep learning frameworks and hardware platforms that support multiple deep learning frameworks. The diversity of deep learning frameworks and hardware brings great benefits to users and is crucial to maintaining the healthy development of the AI ​​ecosystem, but supporting multiple frameworks and hardware requires a huge workload, which also brings considerable challenges to AI developers.

[0003] As deep learning is increasingly used, people are increasingly concerned about the efficiency of deep learning algorithm training and reasoning on different hardware architectures. Since deep learning has many different front-ends and back-ends, a bridge is needed to effectively optimize and map them. IR (Intermediate Representation) is the medium for translating between source code and target code during program compilation. The design of IR is very critical for compilers. IR should consider the completeness of compilation from source code to target code, the ease of use and performance of compilation optimization. Therefore, the competition of intermediate representation will be an important part of the future framework competition. Referring to the experience of traditional compiler design, NNVM (Neural Network Virtual Machine), TVM (Tensor Virtual Machine) and XLA (Accelerated Linear Algebra) have all made good attempts. They are all optimization and compilation tools built around specific intermediate representations. However, they have not been tested and optimized for adaptation on multiple hardware platforms and artificial intelligence development frameworks, resulting in low portability of intermediate representations, and they need to be re-optimized for different hardware platforms. Moreover, in actual applications, the types of intermediate representations vary greatly, and the difficulty of optimizing intermediate representations increases exponentially with the growth of the number of expressions in the intermediate representations. Based on this, how to optimize the intermediate representation is an urgent problem to be solved. Summary of the invention

[0004] The technical problem to be solved by the embodiments of the present invention is to provide a data processing method, device, equipment and computer storage medium, which can achieve front-end framework independence and back-end platform independence by optimizing the dynamic intermediate representation, thereby improving the portability of the intermediate representation and being suitable for a variety of hardware platforms.

[0005] In a first aspect, an embodiment of the present invention provides a data processing method, including:

[0006] Obtain a first dynamic intermediate representation of a deep learning model; wherein the first dynamic intermediate representation includes at least one computational subgraph, the computational subgraph includes N nodes and directed edges connecting different nodes, the directed edges carry dynamic tensor data, and N>1;

[0007] Filter the at least one computation subgraph according to the runtime corresponding to each of the at least one computation subgraphs to obtain at least one computation subgraph to be optimized;

[0008] Optimizing the at least one computational subgraph to be optimized to obtain a second dynamic intermediate representation; wherein the second dynamic intermediate representation at least includes the optimized computational subgraph;

[0009] The second dynamic intermediate representation is compiled to obtain a target code executable by a backend hardware device, so that the backend hardware device executes the target code to implement the function of the target code.

[0010] This embodiment screens the computational subgraphs in the dynamic intermediate representation to obtain the computational subgraphs to be optimized, and optimizes the computational subgraphs to be optimized. This can achieve front-end framework independence and back-end platform independence, improve the portability of the intermediate representation, and make it suitable for a variety of hardware platforms.

[0011] Optionally, in the embodiment of the present invention, the at least one computation subgraph is screened according to the runtime corresponding to each of the at least one computation subgraphs to obtain at least one computation subgraph to be optimized, which may be implemented in the following possible ways:

[0012] In a first possible implementation, the screening of the at least one computation subgraph according to the runtime corresponding to each of the at least one computation subgraphs to obtain at least one computation subgraph to be optimized includes:

[0013] For the changed computation subgraph, obtaining the running time of the changed computation subgraph executed on the backend hardware device;

[0014] When the running time is greater than the first threshold, the computation subgraph corresponding to the running time is used as the computation subgraph to be optimized.

[0015] In this embodiment, the computation subgraph with a longer running time is optimized during the running process to obtain the optimal computation subgraph, and then the optimized dynamic intermediate representation is obtained. In a second possible implementation, the at least one computation subgraph is screened according to the running time corresponding to each of the at least one computation subgraph to obtain at least one computation subgraph to be optimized, including:

[0016] For the changed computational subgraph, the running time of the changed computational subgraph executed on the backend hardware device is obtained to obtain a ranking result; wherein, in the ranking result, the longer the running time, the higher the ranking;

[0017] The computation subgraphs corresponding to the running times ranked in the top L in the sorting result are used as the computation subgraphs to be optimized, where L>0.

[0018] This embodiment optimizes some computation subgraphs with a long running time during the running process, thereby obtaining an optimized dynamic intermediate representation. In this implementation, the performance of the device in executing the target code can be improved.

[0019] In a third possible implementation, the screening of the at least one computation subgraph according to the runtime corresponding to each of the at least one computation subgraphs to obtain at least one computation subgraph to be optimized includes:

[0020] For the same computational subgraph that has changed, obtaining a first running time of the computational subgraph after the change being executed on the backend hardware device, and a second running time of the computational subgraph before the change being executed on the backend hardware device;

[0021] When the difference between the first running time and the second running time is greater than a second threshold, the computation subgraph corresponding to the first running time is used as the computation subgraph to be optimized.

[0022] This embodiment optimizes the computation subgraphs with the largest difference in running time before and after the change during the running process to obtain the optimal computation subgraph, and then obtains the optimized dynamic intermediate representation. In this implementation, the performance of the device in executing the target code can be improved.

[0023] In a second aspect, an embodiment of the present invention further provides a data processing method, including:

[0024] Get the target code and the image to be processed;

[0025] The image to be processed is processed according to the target code to obtain an image processing result; wherein the target code is obtained by the data processing method described in any one of the first aspects above.

[0026] This embodiment optimizes the dynamic intermediate representation and then compiles it to obtain the target code, which can improve the running efficiency of the target code and thus improve the image processing efficiency.

[0027] In a third aspect, an embodiment of the present invention further provides a data processing device, including a receiving unit and a processing unit:

[0028] The receiving unit is used to obtain a first dynamic intermediate representation of a deep learning model; wherein the first dynamic intermediate representation includes at least one computational subgraph, the computational subgraph includes N nodes and directed edges connecting different nodes, and the directed edges carry dynamic tensor data, N>1;

[0029] The processing unit is used to screen the at least one computational subgraph according to the running time corresponding to each of the at least one computational subgraphs to obtain at least one computational subgraph to be optimized; optimize the at least one computational subgraph to be optimized to obtain a second dynamic intermediate representation; wherein the second dynamic intermediate representation at least includes the optimized computational subgraph; compile the second dynamic intermediate representation to obtain a target code executable by a backend hardware device, so that the backend hardware device executes the target code to implement the function of the target code.

[0030] Optionally, in the embodiment of the present invention, the processing unit screens the at least one computation subgraph according to the runtime corresponding to each of the at least one computation subgraphs, and obtains at least one computation subgraph to be optimized, which may be implemented in the following possible ways:

[0031] In a first possible implementation manner, the processing unit screens the at least one computation subgraph according to the runtime corresponding to each of the at least one computation subgraphs to obtain at least one computation subgraph to be optimized, specifically including:

[0032] For the changed computation subgraph, obtaining the running time of the changed computation subgraph executed on the backend hardware device;

[0033] When the running time is greater than the first threshold, the computation subgraph corresponding to the running time is used as the computation subgraph to be optimized.

[0034] In a second possible implementation, the processing unit screens the at least one computation subgraph according to the runtime corresponding to each of the at least one computation subgraphs to obtain at least one computation subgraph to be optimized, specifically including:

[0035] For the changed computational subgraph, the running time of the changed computational subgraph executed on the backend hardware device is obtained to obtain a ranking result; wherein, in the ranking result, the longer the running time, the higher the ranking;

[0036] The computation subgraphs corresponding to the running times ranked in the top L in the sorting result are used as the computation subgraphs to be optimized, where L>0.

[0037] In a third possible implementation, the processing unit screens the at least one computation subgraph according to the runtime corresponding to each of the at least one computation subgraphs to obtain at least one computation subgraph to be optimized, specifically including:

[0038] For the same computational subgraph that has changed, obtaining a first running time of the computational subgraph after the change being executed on the backend hardware device, and a second running time of the computational subgraph before the change being executed on the backend hardware device;

[0039] When the difference between the first running time and the second running time is greater than a second threshold, the computation subgraph corresponding to the first running time is used as the computation subgraph to be optimized.

[0040] It should be noted that the relevant specific implementation schemes and beneficial effects of each embodiment of a data processing device provided in the third aspect of an embodiment of the present invention are the same as the specific implementation schemes and beneficial effects of each embodiment of a data processing method provided in the first aspect of an embodiment of the present invention, and will not be repeated here.

[0041] In a fourth aspect, an embodiment of the present invention further provides a data processing device, including a receiving unit and a processing unit:

[0042] The receiving unit is used to obtain the target code and the image to be processed;

[0043] The processing unit is used to process the image to be processed according to the target code to obtain an image processing result; wherein the target code is obtained by the data processing method described in any one of the first aspects above.

[0044] In a fifth aspect, an embodiment of the present invention further provides a data processing device, comprising: a memory and a processor, wherein the memory is used to store programs, and the processor executes the programs stored in the memory. When the programs stored in the memory are executed, the processor is used to execute the data processing method described in any one of the first aspect above, or the data processing method described in the second aspect above.

[0045] In the sixth aspect, an embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium includes a stored computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute the data processing method described in any one of the first aspect above, or the data processing method described in the second aspect above.

[0046] Compared with the prior art, the beneficial effects of a data processing method, device, equipment and computer storage medium provided by an embodiment of the present invention are: by obtaining a first dynamic intermediate representation of a deep learning model; wherein the first dynamic intermediate representation includes at least one computational subgraph, the computational subgraph includes N nodes and directed edges connecting different nodes, and the directed edges carry dynamic tensor data; the at least one computational subgraph is screened according to the corresponding running time of each of the at least one computational subgraphs to obtain at least one computational subgraph to be optimized; the at least one computational subgraph to be optimized is optimized to obtain a second dynamic intermediate representation; wherein the second dynamic intermediate representation includes at least the optimized computational subgraph; the second dynamic intermediate representation is compiled to obtain a target code executable by a backend hardware device, so that the backend hardware device executes the target code and realizes the function of the target code. The embodiment of the present invention can achieve front-end framework independence and back-end platform independence by optimizing the dynamic intermediate representation, and improves the portability of the intermediate representation to be applicable to a variety of hardware platforms. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 It is a flow chart of an embodiment of a data processing method provided by the present invention;

[0048] Figure 2 is a flow chart of another embodiment of a data processing method provided by the present invention;

[0049] Figure 3 is a flow chart of another embodiment of a data processing method provided by the present invention;

[0050] Figure 4 It is a flow chart of a data processing method provided by the present invention applied to image processing;

[0051] Figure 5 is a structural schematic diagram of an embodiment of a data processing device provided by the present invention;

[0052] Figure 6 is a structural schematic diagram of another embodiment of a data processing device provided by the present invention;

[0053] Figure 7 It is a structural schematic diagram of an embodiment of a data processing device provided by the present invention. DETAILED DESCRIPTION

[0054] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0055] See also Figure 1 , Figure 1 The following is a flow chart of an embodiment of a data processing method provided by the present invention. The data processing method comprises:

[0056] S101, obtaining a first dynamic intermediate representation of a deep learning model; wherein, the first dynamic intermediate representation includes at least one computational subgraph, the computational subgraph includes N nodes and directed edges connecting different nodes, the directed edges carry dynamic tensor data, and N>1.

[0057] Specifically, in the initial stage of deep learning, each deep learning researcher needs to write a lot of repetitive code. In order to improve work efficiency, these researchers wrote these codes into a framework and put it online for all researchers to use. Then, different frameworks appeared on the Internet. Over time, the most useful frameworks were widely used and became popular. At present, the most popular deep learning frameworks in the world are PaddlePaddle, Tensorflow, Caffe, Theano, MXNet, Torch and PyTorch. A deep learning framework is an interface, library or tool that enables us to build deep learning models more easily and quickly without having to deeply understand the details of the underlying algorithms. The deep learning framework defines the model using a pre-built and optimized set of components, providing a clear and concise method for building deep learning models. Since deep learning has many different front-ends and back-ends, IR (Intermediate Representation) acts as an intermediary for translation between source code and target code, which can express the data structure of the source program and related algorithms.

[0058] Exemplarily, the first dynamic intermediate representation of the deep learning model obtained in this embodiment includes at least one computational subgraph, wherein each computational subgraph includes N nodes and directed edges connecting different nodes, the directed edges carry dynamic tensor data, and N is a positive integer greater than 1.

[0059] It should be noted that the data of the deep learning model can be classified through the trained neural network, and static data streams and dynamic data streams can be obtained accordingly. The first dynamic intermediate representation can be obtained according to the obtained dynamic data stream.

[0060] S102: Screen the at least one computation subgraph according to the runtime corresponding to each of the at least one computation subgraphs to obtain at least one computation subgraph to be optimized.

[0061] Specifically, by executing each computation subgraph on the backend hardware device, the corresponding running time of each computation subgraph executed on the backend hardware device can be obtained. By screening all corresponding computation subgraphs according to the running time, at least one computation subgraph to be optimized can be obtained.

[0062] S103, optimizing the at least one computation subgraph to be optimized to obtain a second dynamic intermediate representation; wherein the second dynamic intermediate representation at least includes the optimized computation subgraph.

[0063] Specifically, the intermediate representation, as an intermediary for translating between source code and target code, can express the data structure and related algorithms of the source program. By compiling the intermediate representation, the target code executable by the back-end hardware device can be obtained. A good intermediate representation must have the ability to accurately express the source program information and be easy to transform. During the transformation process, the intermediate representation will undergo multiple operations, making it very complicated. Therefore, in order to improve the running performance of the compiled target code, it is necessary to optimize the computational subgraph to be optimized in the first dynamic intermediate representation obtained.

[0064] It should be noted that the first dynamic intermediate representation obtained from the deep learning model includes at least one computational subgraph. Therefore, when screening the computational subgraphs to obtain the computational subgraphs to be optimized, some computational subgraphs that do not need to be optimized may appear, and the second dynamic intermediate representation obtained after optimization may include only the optimized computational subgraphs, or may include the optimized computational subgraphs and computational subgraphs that do not need to be optimized.

[0065] S104, compile the second dynamic intermediate representation to obtain a target code executable by a backend hardware device, so that the backend hardware device executes the target code to implement the function of the target code.

[0066] Specifically, the optimized second dynamic intermediate representation is compiled to obtain a target code executable by the back-end hardware device. The back-end hardware device executes the target code to implement the function of the target code.

[0067] This embodiment screens the computational subgraphs in the dynamic intermediate representation to obtain the computational subgraphs to be optimized, and optimizes the computational subgraphs to be optimized. This can achieve front-end framework independence and back-end platform independence, improve the portability of the intermediate representation, and make it suitable for a variety of hardware platforms.

[0068] As an optional embodiment, the step of screening the at least one computation subgraph according to the runtime corresponding to each of the at least one computation subgraphs to obtain at least one computation subgraph to be optimized includes:

[0069] For the changed computation subgraph, obtaining the running time of the changed computation subgraph executed on the backend hardware device;

[0070] When the running time is greater than the first threshold, the computation subgraph corresponding to the running time is used as the computation subgraph to be optimized.

[0071] Exemplarily, the first dynamic intermediate representation includes at least one computational subgraph, and the computational subgraph changes dynamically. For each computational subgraph that changes, the running time of the changed computational subgraph executed on the backend hardware device is obtained. The running time is compared with a preset first time threshold, and the case where the running time is greater than the preset first time threshold is screened. If the running time is greater than the preset first time threshold, the computational subgraph corresponding to the running time is used as the computational subgraph to be optimized.

[0072] This embodiment optimizes the computation subgraphs with a long running time during the running process to obtain the optimal computation subgraph, and then obtains the optimized dynamic intermediate representation. In this implementation, the performance of the device in executing the target code can be improved.

[0073] As an optional embodiment, the step of screening the at least one computation subgraph according to the runtime corresponding to each of the at least one computation subgraphs to obtain at least one computation subgraph to be optimized includes:

[0074] For the changed computational subgraph, the running time of the changed computational subgraph executed on the backend hardware device is obtained to obtain a ranking result; wherein, in the ranking result, the longer the running time, the higher the ranking;

[0075] The computation subgraphs corresponding to the running times ranked in the top L in the sorting result are used as the computation subgraphs to be optimized, where L>0.

[0076] Exemplarily, the first dynamic intermediate representation includes at least one computational subgraph, and the computational subgraph is dynamically changing. For each computational subgraph that changes, the runtime of the changed computational subgraph executed on the backend hardware device is obtained. All the acquired runtimes are sorted in order from large to small to obtain a sorting result. The computational subgraphs corresponding to the runtimes ranked in the top L in the sorting result are used as the computational subgraphs to be optimized. Wherein, L is a positive integer greater than 0.

[0077] This embodiment optimizes some computation subgraphs with a long running time during the running process, thereby obtaining an optimized dynamic intermediate representation. In this implementation, the performance of the device in executing the target code can be improved.

[0078] As an optional embodiment, the step of screening the at least one computation subgraph according to the runtime corresponding to each of the at least one computation subgraphs to obtain at least one computation subgraph to be optimized includes:

[0079] For the same computational subgraph that has changed, obtaining a first running time of the computational subgraph after the change being executed on the backend hardware device, and a second running time of the computational subgraph before the change being executed on the backend hardware device;

[0080] When the difference between the first running time and the second running time is greater than a second threshold, the computation subgraph corresponding to the first running time is used as the computation subgraph to be optimized.

[0081] Exemplarily, the first dynamic intermediate representation includes at least one computational subgraph, and the computational subgraph changes dynamically. For the same computational subgraph that changes, the first running time of the computational subgraph after the change on the backend hardware device and the second running time of the computational subgraph before the change on the backend hardware device are obtained. The difference between the first running time and the second running time is calculated, and the difference is compared with a preset second time threshold. The case where the difference is greater than the preset second time threshold is screened. If the difference is greater than the second threshold, the computational subgraph corresponding to the first running time is used as the computational subgraph to be optimized.

[0082] This embodiment optimizes the computation subgraphs with a large difference in running time before and after the change during the running process to obtain the optimal computation subgraph, and then obtains the optimized dynamic intermediate representation.

[0083] The present invention also provides a data processing method to optimize the static intermediate representation. Figure 2 , Figure 2 1 is a flow chart of another embodiment of a data processing method provided by the present invention. The data processing method comprises:

[0084] S201, obtaining a first static intermediate representation of a deep learning model; wherein the deep learning model is a model built based on a front-end model framework, and the first static intermediate representation has a graph structure, the graph structure includes M nodes and directed edges connecting different nodes, and the directed edges carry static tensor data, M>1.

[0085] Specifically, in the initial stage of deep learning, each deep learning researcher needs to write a lot of repetitive code. In order to improve work efficiency, these researchers wrote these codes into a framework and put it online for all researchers to use. Then, different frameworks appeared on the Internet. Over time, the most useful frameworks were widely used and became popular. At present, the most popular deep learning frameworks in the world are PaddlePaddle, Tensorflow, Caffe, Theano, MXNet, Torch and PyTorch. A deep learning framework is an interface, library or tool that enables us to build deep learning models more easily and quickly without having to deeply understand the details of the underlying algorithms. The deep learning framework defines the model using a pre-built and optimized set of components, providing a clear and concise method for building deep learning models. Since deep learning has many different front-ends and back-ends, IR (Intermediate Representation) acts as an intermediary for translation between source code and target code, which can express the data structure of the source program and related algorithms.

[0086] Exemplarily, the first static intermediate representation of the deep learning model obtained in this embodiment has a graph structure, which includes M nodes and directed edges connecting different nodes, and the directed edges carry static tensor data, where M is a positive integer greater than 1. The graph structure is a computational graph, and the workload of the deep learning model from the front-end model framework is represented by an intermediate representation in the form of a computational graph.

[0087] It should be noted that the data of the deep learning model can be classified through the trained neural network, and static data streams and dynamic data streams can be obtained accordingly. The first static intermediate representation can be obtained according to the obtained static data stream.

[0088] S202: Convert the first static intermediate representation into a second static intermediate representation that is independent of the front-end model framework.

[0089] Specifically, the first static intermediate representation of the deep learning model obtained in S201 is related to the front-end model framework. In order to make the optimized static intermediate representation applicable to a variety of back-end hardware devices, it is necessary to convert the first static intermediate representation related to the front-end model framework into a second static intermediate representation that is independent of the front-end model framework.

[0090] It should be noted that the first static intermediate representation can be converted into the form of a decoupled framework. The form of the decoupled framework specifically means that the converted intermediate representation has nothing to do with the upper-level model framework, thereby obtaining a second static intermediate representation that is independent of the front-end model framework.

[0091] S203: Optimize the second static intermediate representation to obtain a third static intermediate representation.

[0092] Specifically, the intermediate representation, as an intermediary for translating between source code and target code, can express the data structure and related algorithms of the source program. By compiling the intermediate representation, the target code executable by the back-end hardware device can be obtained. A good intermediate representation must have the ability to accurately express the source program information and be easy to transform. During the transformation process, the intermediate representation will undergo multiple operations, making it very complicated. Therefore, in order to improve the running performance of the compiled target code, the second static intermediate representation needs to be optimized.

[0093] Since the static intermediate representation is a computational graph that contains various attribute mapping information, when it is optimized in the form of a decoupled framework, that is, when the second static intermediate representation is optimized, the computational graph can have more attributes (for example, adding some new operators and adjusting attributes for specific operators) or become another new computational graph.

[0094] S204, compile the third static intermediate representation to obtain a target code executable by a backend hardware device, so that the backend hardware device executes the target code to implement the function of the target code.

[0095] Specifically, the optimized third static intermediate representation is compiled to obtain a target code executable by the back-end hardware device. The back-end hardware device executes the target code to implement the function of the target code.

[0096] This embodiment converts the static intermediate representation into a static intermediate representation that is independent of the front-end model framework and optimizes the converted static intermediate representation. This can achieve front-end framework independence and back-end platform independence, improve the portability of the intermediate representation, and make it suitable for a variety of hardware platforms.

[0097] As an optional embodiment, the step S202 of converting the first static intermediate representation into a second static intermediate representation that is independent of the front-end model framework includes:

[0098] The first static intermediate representation is decoupled, and the nodes and static tensor data related to the front-end model framework in the first static intermediate representation are removed to obtain the second static intermediate representation that is independent of the front-end model framework.

[0099] Exemplarily, the first static intermediate representation has a graph structure, which includes M nodes and directed edges connecting different nodes, and the directed edges carry static tensor data. The NNVM compiler can be used to decouple the first static intermediate representation, remove the nodes and static tensor data related to the front-end model framework in the first static intermediate representation, and obtain a second static intermediate representation that is unrelated to the front-end model framework. Among them, when decoupling the first static intermediate representation, the compiler used is not limited to the above-mentioned NNVM compiler, and can also be a TVM compiler, an XLA compiler, etc., as long as the compiler can achieve decoupling of the static intermediate representation.

[0100] This embodiment can achieve front-end framework independence and back-end platform independence by converting a first static intermediate representation related to the front-end model framework into a second static intermediate representation independent of the front-end model framework, thereby improving the portability of the intermediate representation to be suitable for multiple hardware platforms.

[0101] As an optional embodiment, the optimizing the second static intermediate representation includes:

[0102] The nodes included in the second static intermediate representation are merged to merge multiple nodes with the same attribute in the second static intermediate representation into the same kernel function.

[0103] Specifically, perform operator fusion on the second static intermediate representation, group all nodes contained in the second static intermediate representation according to node attributes, obtain at least one group of nodes, all nodes in each group of nodes have the same attribute, and fuse each group of nodes with the same attribute in the second static intermediate representation into the same kernel function, so that the intermediate results do not need to be saved back to the global memory, reduce the migration of intermediate data, and avoid frequent reading and writing back and forth.

[0104] As an optional embodiment, the optimizing the second static intermediate representation includes:

[0105] Obtaining a layout requirement corresponding to a tensor operation of each node in the second static intermediate representation;

[0106] Perform data layout conversion on the static tensor data of the second static intermediate representation according to the layout requirement.

[0107] Specifically, the data layout of the second static intermediate representation is converted. Tensor operations are basic operators of the computational graph. Tensors may have different layout requirements in different operators. The layout requirements corresponding to the tensor operations of each node in the second static intermediate representation are obtained. According to the layout requirements, the data layout of the static tensor data of the second static intermediate representation is converted to optimize the local memory access efficiency. For example, a node in the second static intermediate representation may use a 4x4 tensor operation, so the original 2x2 static tensor data needs to be converted into 4x4 blocks for storage to optimize the local memory access efficiency.

[0108] As an optional embodiment, the optimizing the second static intermediate representation includes:

[0109] Fusing the nodes included in the second static intermediate representation, so as to fuse multiple nodes having the same attribute in the second static intermediate representation into the same kernel function, and obtain at least one fused node;

[0110] Obtaining a layout requirement corresponding to the tensor operation of each of the fusion nodes;

[0111] The data layout of the static tensor data of the fused second static intermediate representation is converted according to the layout requirement.

[0112] Specifically, perform operator fusion and data layout conversion on the second static intermediate representation, group all nodes included in the second static intermediate representation according to node attributes, obtain at least one group of nodes, all nodes in each group of nodes have the same attribute, and fuse each group of nodes with the same attribute in the second static intermediate representation into the same kernel function to obtain at least one fused node. Obtain the layout requirements corresponding to the tensor operation of each fused node, and perform data layout conversion on the static tensor data of the fused second static intermediate representation according to the layout requirements. For example, a fused node in the second static intermediate representation may use a 4x4 tensor operation, so the original 2x2 static tensor data needs to be converted into 4x4 blocks for storage to optimize local memory access efficiency.

[0113] This embodiment optimizes the operator fusion and data layout conversion of the second static intermediate representation, so that the intermediate results do not need to be saved back to the global memory, the migration of intermediate data is reduced, frequent back-and-forth reading and writing is avoided, and the local memory access efficiency is optimized.

[0114] The present invention also provides a data processing method, which optimizes static intermediate representation and dynamic intermediate representation at the same time. Figure 3 , Figure 3 FIG. 1 is a flow chart of another embodiment of a data processing method provided by the present invention. The data processing method comprises:

[0115] S301, obtaining a static intermediate representation and a dynamic intermediate representation of a deep learning model; wherein the deep learning model is a model built based on a front-end model framework; in a computational subgraph corresponding to the static intermediate representation, each directed edge carries static tensor data; in a computational subgraph corresponding to the dynamic intermediate representation, each directed edge carries dynamic tensor data.

[0116] Specifically, in the initial stage of deep learning, each deep learning researcher needs to write a lot of repetitive code. In order to improve work efficiency, these researchers wrote these codes into a framework and put it online for all researchers to use. Then, different frameworks appeared on the Internet. Over time, the most useful frameworks were widely used and became popular. At present, the most popular deep learning frameworks in the world are PaddlePaddle, Tensorflow, Caffe, Theano, MXNet, Torch and PyTorch. A deep learning framework is an interface, library or tool that enables us to build deep learning models more easily and quickly without having to deeply understand the details of the underlying algorithms. The deep learning framework defines the model using a pre-built and optimized set of components, providing a clear and concise method for building deep learning models. Since deep learning has many different front-ends and back-ends, IR (Intermediate Representation) acts as an intermediary for translation between source code and target code, which can express the data structure of the source program and related algorithms.

[0117] Exemplarily, the static intermediate representation and dynamic intermediate representation of the deep learning model obtained in this embodiment each include at least one computational subgraph, wherein each computational subgraph includes N nodes and directed edges connecting different nodes. In the computational subgraph corresponding to the static intermediate representation, each directed edge carries static tensor data; in the computational subgraph corresponding to the dynamic intermediate representation, each directed edge carries dynamic tensor data.

[0118] It should be noted that the data of the deep learning model can be classified through the trained neural network, and static data streams and dynamic data streams can be obtained accordingly. The static intermediate representation can be obtained based on the obtained static data stream, and the dynamic intermediate representation can be obtained based on the obtained dynamic data stream.

[0119] S302, converting the static intermediate representation into a static intermediate representation that is independent of the front-end model framework, and optimizing the converted static intermediate representation to obtain an optimized static intermediate representation.

[0120] Specifically, the static intermediate representation of the deep learning model obtained in S301 is related to the front-end model framework. In order to make the optimized static intermediate representation applicable to a variety of back-end hardware devices, the static intermediate representation related to the front-end model framework needs to be first converted into a static intermediate representation that is independent of the front-end model framework. The NNVM compiler can be used to decouple the static intermediate representation, remove the nodes and static tensor data related to the front-end model framework in the static intermediate representation, and obtain a static intermediate representation that is independent of the front-end model framework, that is, the converted static intermediate representation. When decoupling the static intermediate representation, the compiler used is not limited to the above-mentioned NNVM compiler, but can also be a TVM compiler, an XLA compiler, etc., as long as the compiler can achieve decoupling of the static intermediate representation. Then, the converted static intermediate representation that is independent of the front-end model framework is optimized to improve the running performance of the compiled target code.

[0121] It should be noted that the static intermediate representation can be converted into the form of a decoupled framework. The form of a decoupled framework specifically means that the converted intermediate representation has no relationship with the upper-level model framework, thereby obtaining a static intermediate representation that is independent of the front-end model framework. Since the static intermediate representation is a calculation graph containing various attribute mapping information, when optimizing the form of a decoupled framework, that is, optimizing the static intermediate representation that is independent of the front-end model framework, the calculation graph can have more attributes (for example, adding some new operators and adjusting attributes for specific operators) or become another new calculation graph.

[0122] S303: In the computation subgraph corresponding to the dynamic intermediate representation, the at least one computation subgraph is screened according to the runtime corresponding to each of the at least one computation subgraph to obtain at least one computation subgraph to be optimized.

[0123] Specifically, for the computational subgraphs corresponding to the dynamic intermediate representation, each computational subgraph is executed on the backend hardware device, and the corresponding running time of each computational subgraph executed on the backend hardware device can be obtained. All corresponding computational subgraphs are screened according to the running time, and at least one computational subgraph to be optimized can be obtained.

[0124] S304, optimizing the at least one computational subgraph to be optimized to obtain an optimized dynamic intermediate representation; wherein the optimized dynamic intermediate representation at least includes the optimized computational subgraph.

[0125] Specifically, the intermediate representation, as an intermediary for translating between source code and target code, can express the data structure and related algorithms of the source program. By compiling the intermediate representation, the target code executable by the back-end hardware device can be obtained. A good intermediate representation must have the ability to accurately express the source program information and be easy to transform. During the transformation process, the intermediate representation will undergo multiple operations, making it very complicated. Therefore, in order to improve the running performance of the compiled target code, it is necessary to optimize the computational subgraph to be optimized in the first dynamic intermediate representation obtained.

[0126] It should be noted that the dynamic intermediate representation obtained from the deep learning model includes at least one computational subgraph. Therefore, when screening the computational subgraphs to obtain the computational subgraphs to be optimized, some computational subgraphs that do not need to be optimized may appear, and the optimized dynamic intermediate representation may include only the optimized computational subgraphs, or may include the optimized computational subgraphs and computational subgraphs that do not need to be optimized.

[0127] S305, compile the optimized static intermediate representation and the optimized dynamic intermediate representation to obtain a target code executable by a backend hardware device, so that the backend hardware device executes the target code to implement the function of the target code.

[0128] Specifically, the optimized static intermediate representation and the optimized dynamic intermediate representation are compiled to obtain a target code executable by the back-end hardware device. The back-end hardware device executes the target code to realize the function of the target code.

[0129] This embodiment optimizes the static intermediate representation and the dynamic intermediate representation at the same time, so that the optimization effect is better, the front-end framework and the back-end platform can be achieved, and the portability of the intermediate representation is improved to be applicable to various hardware platforms.

[0130] As an optional embodiment, the optimizing the converted static intermediate representation includes:

[0131] Fusing the nodes contained in the converted static intermediate representation to fuse multiple nodes with the same attribute in the converted static intermediate representation into the same kernel function;

[0132] or,

[0133] Obtain the layout requirements corresponding to the tensor operations of each node in the converted static intermediate representation;

[0134] The data layout of the static tensor data of the converted static intermediate representation is converted according to the layout requirement.

[0135] Specifically, the operator fusion is performed on the converted static intermediate representation, and all nodes contained in the converted static intermediate representation are grouped according to the node attributes to obtain at least one group of nodes, and all nodes in each group of nodes have the same attribute. Each group of nodes with the same attribute in the converted static intermediate representation is correspondingly fused into the same kernel function, so that the intermediate results do not need to be saved back to the global memory, the migration of intermediate data is reduced, and frequent back and forth reading and writing are avoided. Alternatively, the data layout conversion can be performed on the converted static intermediate representation. Tensor operations are basic operators of the computational graph. Tensors may have different layout requirements in different operators. The layout requirements corresponding to the tensor operations of each node in the converted static intermediate representation are obtained, and the static tensor data of the converted static intermediate representation is converted according to the layout requirements to optimize the local memory access efficiency. For example, a node in the converted static intermediate representation may use a 4x4 tensor operation, so the original 2x2 static tensor data needs to be converted into a 4x4 block for storage to optimize the local memory access efficiency.

[0136] As an optional embodiment, the optimizing the converted static intermediate representation includes:

[0137] Fusing the nodes contained in the converted static intermediate representation, so as to fuse multiple nodes having the same attribute in the converted static intermediate representation into the same kernel function, and obtain at least one fused node;

[0138] Obtaining a layout requirement corresponding to the tensor operation of each of the fusion nodes;

[0139] The data layout of the static tensor data of the fused static intermediate representation is converted according to the layout requirement.

[0140] Specifically, perform operator fusion and data layout conversion on the transformed static intermediate representation, group all nodes contained in the transformed static intermediate representation according to node attributes, obtain at least one group of nodes, all nodes in each group of nodes have the same attribute, and fuse each group of nodes with the same attribute in the transformed static intermediate representation into the same kernel function to obtain at least one fused node. Obtain the layout requirements corresponding to the tensor operation of each fused node, and perform data layout conversion on the static tensor data of the fused transformed static intermediate representation according to the layout requirements. For example, a fused node in the transformed static intermediate representation may use a 4x4 tensor operation, so the original 2x2 static tensor data needs to be converted into 4x4 blocks for storage to optimize local memory access efficiency.

[0141] This embodiment optimizes the operator fusion and data layout conversion of the converted static intermediate representation, so that the intermediate results do not need to be saved back to the global memory, the migration of intermediate data is reduced, frequent back-and-forth reading and writing is avoided, and the local memory access efficiency is optimized.

[0142] As an optional embodiment, the step of screening the at least one computation subgraph according to the runtime corresponding to each of the at least one computation subgraphs to obtain at least one computation subgraph to be optimized includes:

[0143] For the changed computation subgraph, obtaining the running time of the changed computation subgraph executed on the backend hardware device;

[0144] When the running time is greater than the first threshold, the computation subgraph corresponding to the running time is used as the computation subgraph to be optimized.

[0145] Exemplarily, the dynamic intermediate representation includes at least one computational subgraph, and the computational subgraph changes dynamically. For each computational subgraph that changes, the running time of the changed computational subgraph executed on the backend hardware device is obtained. The running time is compared with a preset first time threshold, and the case where the running time is greater than the preset first time threshold is screened. If the running time is greater than the preset first time threshold, the computational subgraph corresponding to the running time is used as the computational subgraph to be optimized.

[0146] This embodiment optimizes the computation subgraph with a long running time during the running process to obtain the optimal computation subgraph, and then obtains the optimized dynamic intermediate representation.

[0147] As an optional embodiment, the step of screening the at least one computation subgraph according to the runtime corresponding to each of the at least one computation subgraphs to obtain at least one computation subgraph to be optimized includes:

[0148] For the changed computational subgraph, the running time of the changed computational subgraph executed on the backend hardware device is obtained to obtain a ranking result; wherein, in the ranking result, the longer the running time, the higher the ranking;

[0149] The computation subgraphs corresponding to the running times ranked in the top L in the sorting result are used as the computation subgraphs to be optimized, where L>0.

[0150] Exemplarily, the first dynamic intermediate representation includes at least one computational subgraph, and the computational subgraph is dynamically changing. For each computational subgraph that changes, the runtime of the changed computational subgraph executed on the backend hardware device is obtained. All the acquired runtimes are sorted in order from large to small to obtain a sorting result. The computational subgraphs corresponding to the runtimes ranked in the top L in the sorting result are used as the computational subgraphs to be optimized. Wherein, L is a positive integer greater than 0.

[0151] This embodiment optimizes certain computational subgraphs with a long running time during the running process, thereby obtaining an optimized dynamic intermediate representation.

[0152] As an optional embodiment, the step of screening the at least one computation subgraph according to the runtime corresponding to each of the at least one computation subgraphs to obtain at least one computation subgraph to be optimized includes:

[0153] For the same computational subgraph that has changed, obtaining a first running time of the computational subgraph after the change being executed on the backend hardware device, and a second running time of the computational subgraph before the change being executed on the backend hardware device;

[0154] When the difference between the first running time and the second running time is greater than a second threshold, the computation subgraph corresponding to the first running time is used as the computation subgraph to be optimized.

[0155] Exemplarily, the first dynamic intermediate representation includes at least one computational subgraph, and the computational subgraph changes dynamically. For the same computational subgraph that changes, the first running time of the computational subgraph after the change on the backend hardware device and the second running time of the computational subgraph before the change on the backend hardware device are obtained. The difference between the first running time and the second running time is calculated, and the difference is compared with a preset second time threshold. The case where the difference is greater than the preset second time threshold is screened. If the difference is greater than the second threshold, the computational subgraph corresponding to the first running time is used as the computational subgraph to be optimized.

[0156] This embodiment optimizes the computation subgraphs with a large difference in running time before and after the change during the running process to obtain the optimal computation subgraph, and then obtains the optimized dynamic intermediate representation.

[0157] In practical applications, the present invention can be optimized only for the static intermediate representation, or only for the dynamic intermediate representation, or both for the static intermediate representation and the dynamic intermediate representation according to actual needs to achieve different optimization purposes.

[0158] In specific applications, this embodiment takes image processing as an example. Figure 4 , Figure 4 The following is a flow chart of a data processing method provided by the present invention applied to image processing. The data processing method comprises:

[0159] S401, obtaining a target code and an image to be processed;

[0160] S402, processing the image to be processed according to the target code to obtain an image processing result; wherein the target code is obtained by the data processing method described in any of the above embodiments of the present invention.

[0161] Specifically, this embodiment can be executed by a graphics processing unit (GPU). After obtaining the target code and the image to be processed, the image processor processes the image to be processed according to the target code, and obtains the image processing result accordingly. The target code can be obtained by optimizing the data processing method described in any of the above embodiments, and the image processing can include but is not limited to any one of convolution processing, classification processing and grayscale processing. By optimizing the intermediate representation and then compiling to obtain the target code, the running efficiency of the target code can be improved, thereby improving the image processing efficiency.

[0162] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should know that the present disclosure is not limited by the order of the actions described, because according to the present disclosure, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily required by the present disclosure.

[0163] It should be further explained that Figure 1 For example, although Figure 1 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 1 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0164] Correspondingly, the present invention also provides a data processing device capable of implementing all processes of the data processing method in the above embodiments.

[0165] See also Figure 5 , Figure 5 5 is a schematic diagram of a data processing device according to an embodiment of the present invention. The data processing device comprises a receiving unit 501 and a processing unit 502;

[0166] The receiving unit 501 is used to obtain a first dynamic intermediate representation of a deep learning model; wherein the first dynamic intermediate representation includes at least one computational subgraph, the computational subgraph includes N nodes and directed edges connecting different nodes, and the directed edges carry dynamic tensor data, N>1;

[0167] The processing unit 502 is used to screen the at least one computational subgraph according to the running time corresponding to each of the at least one computational subgraphs to obtain at least one computational subgraph to be optimized; optimize the at least one computational subgraph to be optimized to obtain a second dynamic intermediate representation; wherein the second dynamic intermediate representation at least includes the optimized computational subgraph; compile the second dynamic intermediate representation to obtain a target code executable by a backend hardware device, so that the backend hardware device executes the target code to implement the function of the target code.

[0168] As an optional embodiment, the processing unit 502 screens the at least one computation subgraph according to the runtime corresponding to each of the at least one computation subgraphs to obtain at least one computation subgraph to be optimized, specifically including:

[0169] For the changed computation subgraph, obtaining the running time of the changed computation subgraph executed on the backend hardware device;

[0170] When the running time is greater than the first threshold, the computation subgraph corresponding to the running time is used as the computation subgraph to be optimized.

[0171] As an optional embodiment, the processing unit 502 screens the at least one computation subgraph according to the runtime corresponding to each of the at least one computation subgraphs to obtain at least one computation subgraph to be optimized, specifically including:

[0172] For the changed computational subgraph, the running time of the changed computational subgraph executed on the backend hardware device is obtained to obtain a ranking result; wherein, in the ranking result, the longer the running time, the higher the ranking;

[0173] The computation subgraphs corresponding to the running times ranked in the top L in the sorting result are used as the computation subgraphs to be optimized, where L>0.

[0174] As an optional embodiment, the processing unit 502 screens the at least one computation subgraph according to the runtime corresponding to each of the at least one computation subgraphs to obtain at least one computation subgraph to be optimized, specifically including:

[0175] For the same computational subgraph that has changed, obtaining a first running time of the computational subgraph after the change being executed on the backend hardware device, and a second running time of the computational subgraph before the change being executed on the backend hardware device;

[0176] When the difference between the first running time and the second running time is greater than a second threshold, the computation subgraph corresponding to the first running time is used as the computation subgraph to be optimized.

[0177] In specific applications, this embodiment takes image processing as an example. Figure 6 , Figure 6 60 is a schematic diagram of another embodiment of a data processing device provided by the present invention. The data processing device includes a receiving unit 601 and a processing unit 602;

[0178] The receiving unit 601 is used to obtain the target code and the image to be processed;

[0179] The processing unit 602 is used to process the image to be processed according to the target code to obtain an image processing result; wherein the target code is obtained by any of the data processing methods described above.

[0180] In the specific implementation, the working principle, control flow and technical effect achieved by the data processing device provided in the embodiment of the present invention are the same as those of the data processing method in the above embodiment, and will not be repeated here.

[0181] See also Figure 7 , Figure 7 7 is a schematic diagram of a structure of an embodiment of a data processing device provided by the present invention. The data processing device includes a memory 701 and a processor 702, the memory 701 is used to store programs, and the processor 702 executes the programs stored in the memory 701. When the programs stored in the memory 701 are executed, the processor 702 is used to execute the data processing method described in any of the above embodiments.

[0182] As an optional embodiment, the computer program may be divided into one or more modules / units (such as computer program 1, computer program 2, ...), and the one or more modules / units are stored in the memory 701 and executed by the processor 702 to complete the present invention. The one or more modules / units may be a series of computer program instruction segments capable of completing specific functions, and the instruction segments are used to describe the execution process of the computer program in the data processing device.

[0183] The processor 702 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor, or the processor 702 may be any conventional processor. The processor 702 is the control center of the data processing device, and various parts of the data processing device are connected using various interfaces and lines.

[0184] The memory 701 mainly includes a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function, etc., and the data storage area can store related data, etc. In addition, the memory 701 can be a high-speed random access memory, or a non-volatile memory, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, and a flash card (Flash Card), etc., or the memory 701 can also be other volatile solid-state storage devices.

[0185] It should be noted that the above data processing device may include, but is not limited to, a processor and a memory. Those skilled in the art will understand that Figure 7 The structural diagram is only an example of the above-mentioned data processing device and does not constitute a limitation of the above-mentioned data processing device. It may include more or less components than shown in the figure, or combine certain components, or different components.

[0186] An embodiment of the present invention further provides a computer-readable storage medium, which includes a stored computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute the data processing method described in any of the above embodiments.

[0187] If the various component modules of the above-mentioned device are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer product is stored in a computer-readable storage medium.

[0188] The computer-readable storage medium may be an internal storage unit of the device of the aforementioned embodiment, such as a hard disk or a memory. The computer-readable storage medium may also be an external storage device of the device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. Further, the computer-readable storage medium may also include both an internal storage unit of the device and an external storage device. The computer-readable storage medium is used to store the computer program and other programs and data required by the device. The computer-readable storage medium may also be used to temporarily store data that has been output or is to be output.

[0189] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing related hardware through a computer program, and the computer program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. The aforementioned storage medium includes: ROM, RAM, magnetic disk or optical disk and other media that can store program codes.

[0190] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0191] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0192] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0193] The above-mentioned integrated unit implemented in the form of a software functional unit can be stored in a computer-readable storage medium. The above-mentioned software functional unit is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor (English: processor) to perform some steps of the methods of each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (English: Read-Only Memory, abbreviated: ROM), random access memory (English: Random Access Memory, abbreviated: RAM), disk or optical disk and other media that can store program codes.

[0194] In summary, the embodiments of the present invention provide a data processing method, apparatus, device and computer storage medium, by obtaining a first dynamic intermediate representation of a deep learning model; wherein the first dynamic intermediate representation includes at least one computational subgraph, the computational subgraph includes N nodes and directed edges connecting different nodes, and the directed edges carry dynamic tensor data; the at least one computational subgraph is screened according to the corresponding running time of each of the at least one computational subgraphs to obtain at least one computational subgraph to be optimized; the at least one computational subgraph to be optimized is optimized to obtain a second dynamic intermediate representation; wherein the second dynamic intermediate representation includes at least the optimized computational subgraph; the second dynamic intermediate representation is compiled to obtain a target code executable by a backend hardware device, so that the backend hardware device executes the target code and realizes the function of the target code. The embodiments of the present invention can achieve front-end framework independence and back-end platform independence by optimizing the dynamic intermediate representation, thereby improving the portability of the intermediate representation to be applicable to a variety of hardware platforms.

[0195] The above is a specific implementation of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A data processing method, characterized in that: include: Obtain a first dynamic intermediate representation of a deep learning model; wherein the first dynamic intermediate representation includes at least one computational subgraph, the computational subgraph includes N nodes and directed edges connecting different nodes, the directed edges carry dynamic tensor data, and N>1; Filter the at least one computation subgraph according to the runtime corresponding to each of the at least one computation subgraphs to obtain at least one computation subgraph to be optimized; Optimizing the at least one computational subgraph to be optimized to obtain a second dynamic intermediate representation; wherein the second dynamic intermediate representation at least includes the optimized computational subgraph; The second dynamic intermediate representation is compiled to obtain a target code executable by a backend hardware device, so that the backend hardware device executes the target code to implement the function of the target code.

2. The data processing method according to claim 1, characterized in that: The step of screening the at least one computation subgraph according to the runtime corresponding to each of the at least one computation subgraphs to obtain at least one computation subgraph to be optimized includes: For the changed computation subgraph, obtaining the running time of the changed computation subgraph executed on the backend hardware device; When the running time is greater than the first threshold, the computation subgraph corresponding to the running time is used as the computation subgraph to be optimized.

3. The data processing method according to claim 1, characterized in that: The step of screening the at least one computation subgraph according to the runtime corresponding to each of the at least one computation subgraphs to obtain at least one computation subgraph to be optimized includes: For the changed computational subgraph, the running time of the changed computational subgraph executed on the backend hardware device is obtained to obtain a ranking result; wherein, in the ranking result, the longer the running time, the higher the ranking; The computation subgraphs corresponding to the running times ranked in the top L in the sorting result are used as the computation subgraphs to be optimized, where L>0.

4. The data processing method according to claim 1, characterized in that: The step of screening the at least one computation subgraph according to the runtime corresponding to each of the at least one computation subgraphs to obtain at least one computation subgraph to be optimized includes: For the same computational subgraph that has changed, obtaining a first running time of the computational subgraph after the change being executed on the backend hardware device, and a second running time of the computational subgraph before the change being executed on the backend hardware device; When the difference between the first running time and the second running time is greater than a second threshold, the computation subgraph corresponding to the first running time is used as the computation subgraph to be optimized.

5. A data processing method, characterized in that: include: Get the target code and the image to be processed; The image to be processed is processed according to the target code to obtain an image processing result; wherein the target code is obtained by the data processing method according to any one of claims 1 to 4.

6. A data processing device, characterized in that: Including receiving unit and processing unit: The receiving unit is used to obtain a first dynamic intermediate representation of a deep learning model; wherein the first dynamic intermediate representation includes at least one computational subgraph, the computational subgraph includes N nodes and directed edges connecting different nodes, and the directed edges carry dynamic tensor data, N>1; The processing unit is used to screen the at least one computational subgraph according to the running time corresponding to each of the at least one computational subgraphs to obtain at least one computational subgraph to be optimized; optimize the at least one computational subgraph to be optimized to obtain a second dynamic intermediate representation; wherein the second dynamic intermediate representation at least includes the optimized computational subgraph; compile the second dynamic intermediate representation to obtain a target code executable by a backend hardware device, so that the backend hardware device executes the target code to implement the function of the target code.

7. The data processing device according to claim 6, characterized in that The processing unit screens the at least one computation subgraph according to the runtime corresponding to each of the at least one computation subgraphs to obtain at least one computation subgraph to be optimized, specifically including: For the changed computation subgraph, obtaining the running time of the changed computation subgraph executed on the backend hardware device; When the running time is greater than the first threshold, the computation subgraph corresponding to the running time is used as the computation subgraph to be optimized.

8. The data processing device according to claim 6, characterized in that The processing unit screens the at least one computation subgraph according to the runtime corresponding to each of the at least one computation subgraphs to obtain at least one computation subgraph to be optimized, specifically including: For the changed computational subgraph, the running time of the changed computational subgraph executed on the backend hardware device is obtained to obtain a ranking result; wherein, in the ranking result, the longer the running time, the higher the ranking; The computation subgraphs corresponding to the running times ranked in the top L in the sorting result are used as the computation subgraphs to be optimized, where L>0.

9. The data processing device according to claim 6, characterized in that The processing unit screens the at least one computation subgraph according to the runtime corresponding to each of the at least one computation subgraphs to obtain at least one computation subgraph to be optimized, specifically including: For the same computational subgraph that has changed, obtaining a first running time of the computational subgraph after the change being executed on the backend hardware device, and a second running time of the computational subgraph before the change being executed on the backend hardware device; When the difference between the first running time and the second running time is greater than a second threshold, the computation subgraph corresponding to the first running time is used as the computation subgraph to be optimized.

10. A data processing device, characterized in that: Including receiving unit and processing unit: The receiving unit is used to obtain the target code and the image to be processed; The processing unit is used to process the image to be processed according to the target code to obtain an image processing result; wherein the target code is obtained by the data processing method according to any one of claims 1 to 4.

11. A data processing device, characterized in that: include: A memory and a processor, wherein the memory is used to store a program, and the processor executes the program stored in the memory. When the program stored in the memory is executed, the processor is used to execute the data processing method according to any one of claims 1 to 4 or the data processing method according to claim 5.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored computer program, wherein when the computer program is executed, the device where the computer-readable storage medium is located is controlled to execute the data processing method according to any one of claims 1 to 4 or the data processing method according to claim 5.

Images