A data processing method, device, equipment and computer storage medium

Abstract

The present invention discloses a data processing method, device, equipment and computer storage medium, the method comprising: obtaining a first intermediate representation of a deep learning model, the first intermediate representation comprising a computational subgraph, the computational subgraph comprising an operator and a directed edge; the computational subgraph corresponds to a functional attribute; the operator comprises a functional function; according to the functional attribute of the computational subgraph, the data structure and variables of the functional function in the target operator are converted; based on the converted data structure and variables, the execution order of the target operator and the scalar operation contained in the target operator are optimized to obtain a second intermediate representation; the second intermediate representation is compiled to obtain a target code executable by a back-end hardware device, so that the back-end hardware device executes the target code and realizes the function of the target code. The present invention can achieve front-end framework independence and back-end platform independence by optimizing the intermediate representation, and improves the portability of the intermediate representation to be applicable to a variety of hardware platforms.

Description

Technical Field

[0001] The present invention relates to the field of computer software technology, and in particular to a data processing method, device, equipment and computer storage medium. Background Art

[0002] The new wave of AI enthusiasm represented by machine learning and deep learning has been heating up for several years. Deep learning has been a research hotspot in the field of AI in recent years and has achieved breakthrough progress in many fields. There are many deep learning frameworks and hardware platforms that support multiple deep learning frameworks. The diversity of deep learning frameworks and hardware brings great benefits to users and is crucial to maintaining the healthy development of the AI ​​ecosystem, but supporting multiple frameworks and hardware requires a huge workload, which also brings considerable challenges to AI developers.

[0003] As deep learning is increasingly used, people are increasingly concerned about the efficiency of deep learning algorithm training and reasoning on different hardware architectures. Since deep learning has many different front-ends and back-ends, a bridge is needed to effectively optimize and map them. IR (Intermediate Representation) is the medium for translating between source code and target code during program compilation. The design of IR is very critical for compilers. IR should consider the completeness of compilation from source code to target code, the ease of use and performance of compilation optimization. Therefore, the competition of intermediate representation will be an important part of the future framework competition. Referring to the experience of traditional compiler design, NNVM (Neural Network Virtual Machine), TVM (Tensor Virtual Machine) and XLA (Accelerated Linear Algebra) have all made good attempts. They are all optimization and compilation tools built around specific intermediate representations. However, they have not been tested and optimized for adaptation on multiple hardware platforms and artificial intelligence development frameworks, resulting in low portability of intermediate representations, and they need to be re-optimized for different hardware platforms. Moreover, in actual applications, the types of intermediate representations vary greatly, and the difficulty of optimizing intermediate representations increases exponentially with the growth of the number of expressions in the intermediate representations. Based on this, how to optimize the intermediate representation is an urgent problem to be solved. Summary of the invention

[0004] The technical problem to be solved by the embodiments of the present invention is to provide a data processing method, device, equipment and computer storage medium, which can achieve front-end framework independence and back-end platform independence by optimizing the intermediate representation, thereby improving the portability of the intermediate representation and being suitable for a variety of hardware platforms.

[0005] In a first aspect, an embodiment of the present invention provides a data processing method, including:

[0006] Obtaining a first intermediate representation of a deep learning model; wherein the deep learning model is a model built based on a front-end model framework, the first intermediate representation includes at least one computational subgraph, each computational subgraph includes at least one group of operators and directed edges related to the at least one group of operators, and the directed edges carry tensor data; each computational subgraph corresponds to a functional attribute; and each operator includes at least one functional function;

[0007] In the at least one computation subgraph, data structures and variables of a functional function included in at least one target operator are converted according to functional attributes corresponding to each computation subgraph; wherein the converted data structures and variables can be executed by the functional function;

[0008] Based on the converted data structure and variables, optimizing the execution order of the at least one target operator and the scalar operations included in the at least one target operator to obtain a second intermediate representation;

[0009] The second intermediate representation is compiled to obtain a target code executable by a backend hardware device, so that the backend hardware device executes the target code to implement the function of the target code.

[0010] This embodiment optimizes the intermediate representation to achieve front-end framework independence and back-end platform independence, thereby improving the portability of the intermediate representation to be applicable to a variety of hardware platforms.

[0011] In a possible implementation manner of the first aspect, optimizing, based on the converted data structure and variables, the execution order of the at least one target operator and the scalar operation included in the at least one target operator to obtain a second intermediate representation includes:

[0012] Based on the converted data structure and variables, the execution order of the at least one target operator is optimized, and the scalar operation contained in the at least one target operator after the execution order optimization is converted into a vector operation to obtain the second intermediate representation; wherein the scalar operation is used to represent the processing of scalar data in the target operator; and the vector operation is used to represent the processing of vector data in the target operator;

[0013] or,

[0014] Based on the converted data structure and variables, the scalar operation included in the at least one target operator is converted into a vector operation, and the execution order of the at least one target operator after the conversion operation is optimized to obtain the second intermediate representation.

[0015] This embodiment optimizes the operator execution order and the scalar operation inside the operator of the first intermediate representation obtained from the deep learning model, so that the optimized second intermediate representation can express the information of the source program more accurately.

[0016] In a possible implementation manner of the first aspect, if the number of the target operators is greater than or equal to 2, the execution order of the target operators is optimized, including:

[0017] Acquire an irrelevant operator; wherein the irrelevant operator is used to indicate that there is no directed edge between one operator and another operator in the target operator;

[0018] The execution order of the unrelated operators is converted into a parallel execution order.

[0019] This embodiment optimizes the execution order of the target operators to achieve the specific functions of the intermediate representation with better performance.

[0020] Optionally, in the embodiment of the present invention, converting the scalar operation included in the at least one target operator into a vector operation may be implemented in the following possible ways:

[0021] In a first possible implementation manner, converting a scalar operation included in at least one of the target operators into a vector operation includes:

[0022] For each of the target operators, obtaining scalar operations in the target operator that have the same functions and are unrelated;

[0023] Converts functionally identical and unrelated scalar operations to vector operations.

[0024] This embodiment simplifies the intermediate representation before inputting it into the vectorizer, uses concise statements to describe complex operations, reduces the complexity of the vectorizer input, and improves the vectorization mapping capability.

[0025] In a second possible implementation manner, converting the scalar operation included in the at least one target operator into a vector operation includes:

[0026] For each of the target operators, when the scalar operation included in the target operator is a stride memory access, each scalar operation corresponding to the stride memory access is converted into multiple vector operations.

[0027] This embodiment converts a scalar operation into multiple vector operations, so that the target operator can achieve more functions, thereby making the computation subgraph have more functional attributes.

[0028] In a second aspect, an embodiment of the present invention further provides a data processing method, including:

[0029] Get the target code and the image to be processed;

[0030] The image to be processed is processed according to the target code to obtain an image processing result; wherein the target code is obtained by the data processing method described in any one of the first aspects above.

[0031] This embodiment optimizes the intermediate representation and then compiles it to obtain the target code, which can improve the running efficiency of the target code and thus improve the image processing efficiency.

[0032] In a third aspect, an embodiment of the present invention further provides a data processing device, including a receiving unit and a processing unit:

[0033] The receiving unit is used to obtain a first intermediate representation of a deep learning model; wherein the deep learning model is a model built based on a front-end model framework, the first intermediate representation includes at least one computational subgraph, each computational subgraph includes at least one group of operators and directed edges related to the at least one group of operators, and the directed edges carry tensor data; each computational subgraph corresponds to a functional attribute; each operator includes at least one functional function;

[0034] The processing unit is used to convert the data structure and variables of the functional function contained in at least one target operator in the at least one computational subgraph according to the functional attributes corresponding to each computational subgraph; wherein the converted data structure and variables can be executed by the functional function; based on the converted data structure and variables, the execution order of the at least one target operator and the scalar operations contained in the at least one target operator are optimized to obtain a second intermediate representation; the second intermediate representation is compiled to obtain a target code executable by a backend hardware device, so that the backend hardware device executes the target code to realize the function of the target code.

[0035] In a possible implementation manner of the third aspect, the processing unit optimizes, based on the converted data structure and variables, the execution order of the at least one target operator and the scalar operation included in the at least one target operator to obtain a second intermediate representation, including:

[0036] Based on the converted data structure and variables, the execution order of the at least one target operator is optimized, and the scalar operation contained in the at least one target operator after the execution order optimization is converted into a vector operation to obtain the second intermediate representation; wherein the scalar operation is used to represent the processing of scalar data in the target operator; and the vector operation is used to represent the processing of vector data in the target operator;

[0037] or,

[0038] Based on the converted data structure and variables, the scalar operation included in the at least one target operator is converted into a vector operation, and the execution order of the at least one target operator after the conversion operation is optimized to obtain the second intermediate representation.

[0039] In a possible implementation manner of the first aspect, if the number of the target operators is greater than or equal to 2, the processing unit optimizes the execution order of the target operators, including:

[0040] Acquire an irrelevant operator; wherein the irrelevant operator is used to indicate that there is no directed edge between one operator and another operator in the target operator;

[0041] The execution order of the unrelated operators is converted into a parallel execution order.

[0042] Optionally, in the embodiment of the present invention, the processing unit converts the scalar operation included in the at least one target operator into a vector operation in the following possible implementations:

[0043] In a first possible implementation manner, the processing unit converts the scalar operation included in the at least one target operator into a vector operation, including:

[0044] For each of the target operators, obtaining scalar operations in the target operator that have the same functions and are unrelated;

[0045] Converts functionally identical and unrelated scalar operations to vector operations.

[0046] In a second possible implementation manner, the processing unit converts the scalar operation included in the at least one target operator into a vector operation, including:

[0047] For each of the target operators, when the scalar operation included in the target operator is a stride memory access, each scalar operation corresponding to the stride memory access is converted into multiple vector operations.

[0048] It should be noted that the relevant specific implementation schemes and beneficial effects of each embodiment of a data processing device provided in the third aspect of an embodiment of the present invention are the same as the specific implementation schemes and beneficial effects of each embodiment of a data processing method provided in the first aspect of an embodiment of the present invention, and will not be repeated here.

[0049] In a fourth aspect, an embodiment of the present invention further provides a data processing device, including a receiving unit and a processing unit:

[0050] The receiving unit is used to obtain the target code and the image to be processed;

[0051] The processing unit is used to process the image to be processed according to the target code to obtain an image processing result; wherein the target code is obtained by the data processing method described in any one of the first aspects above.

[0052] In a fifth aspect, an embodiment of the present invention further provides a data processing device, comprising: a memory and a processor, wherein the memory is used to store programs, and the processor executes the programs stored in the memory. When the programs stored in the memory are executed, the processor is used to execute the data processing method described in any one of the first aspect above, or the data processing method described in the second aspect above.

[0053] In the sixth aspect, an embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium includes a stored computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute the data processing method described in any one of the first aspect above, or the data processing method described in the second aspect above.

[0054] Compared with the prior art, the beneficial effects of a data processing method, apparatus, device and computer storage medium provided by an embodiment of the present invention are: by obtaining a first intermediate representation of a deep learning model; wherein the deep learning model is a model built based on a front-end model framework, the first intermediate representation includes at least one computational subgraph, each computational subgraph includes at least one group of operators and directed edges related to the at least one group of operators, and the directed edges carry tensor data; each computational subgraph corresponds to a functional attribute; each operator includes at least one functional function; in the at least one computational subgraph, the data structure and variables of the functional function contained in at least one target operator are converted according to the functional attributes corresponding to each computational subgraph; wherein the converted data structure and variables can be executed by the functional function; based on the converted data structure and variables, the execution order of the at least one target operator and the scalar operations contained in the at least one target operator are optimized to obtain a second intermediate representation; the second intermediate representation is compiled to obtain a target code executable by a back-end hardware device, so that the back-end hardware device executes the target code and realizes the function of the target code. The embodiment of the present invention can achieve front-end framework independence and back-end platform independence by optimizing the intermediate representation, thereby improving the portability of the intermediate representation to be applicable to a variety of hardware platforms. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 is a flow chart of an embodiment of a data processing method provided by the first aspect of the present invention;

[0056] Figure 2 is a flow chart of an embodiment of a data processing method provided by the second aspect of the present invention;

[0057] Figure 3 is a structural schematic diagram of an embodiment of a data processing device provided by the third aspect of the present invention;

[0058] Figure 4 is a structural schematic diagram of an embodiment of a data processing device provided by the fourth aspect of the present invention;

[0059] Figure 5 It is a structural diagram of an embodiment of a data processing device provided by the fifth aspect of the present invention. DETAILED DESCRIPTION

[0060] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0061] In a first aspect, an embodiment of the present invention provides a data processing method, see Figure 1 , Figure 1 1 is a flow chart of an embodiment of a data processing method provided by the first aspect of the present invention. The data processing method comprises:

[0062] S101, obtaining a first intermediate representation of a deep learning model; wherein the deep learning model is a model built based on a front-end model framework, the first intermediate representation includes at least one computational subgraph, each computational subgraph includes at least one group of operators and directed edges related to the at least one group of operators, and the directed edges carry tensor data; each computational subgraph corresponds to a functional attribute; and each operator includes at least one functional function;

[0063] Specifically, in the initial stage of deep learning, each deep learning researcher needs to write a lot of repetitive code. In order to improve work efficiency, these researchers wrote these codes into a framework and put it online for all researchers to use. Then, different frameworks appeared on the Internet. Over time, the most useful frameworks were widely used and became popular. At present, the most popular deep learning frameworks in the world are PaddlePaddle, Tensorflow, Caffe, Theano, MXNet, Torch and PyTorch. A deep learning framework is an interface, library or tool that enables us to build deep learning models more easily and quickly without having to deeply understand the details of the underlying algorithms. The deep learning framework defines the model using a pre-built and optimized set of components, providing a clear and concise method for building deep learning models. Since deep learning has many different front-ends and back-ends, IR (Intermediate Representation) acts as an intermediary for translation between source code and target code, which can express the data structure of the source program and related algorithms.

[0064] Exemplarily, the first intermediate representation of the deep learning model obtained in this embodiment includes at least one computational subgraph, wherein each computational subgraph includes at least one group of operators and directed edges related to at least one group of operators, and the directed edges carry tensor data. Each computational subgraph corresponds to a corresponding functional attribute, and each operator includes at least one functional function. Exemplarily, taking the first intermediate representation corresponding to the convolutional neural network as an example, the first intermediate representation includes three computational subgraphs, wherein the computational subgraphs Figure 1 Used to implement the convolution function, the calculation sub Figure 2 Used to implement pooling function, calculation sub Figure 3 Used to realize the classification function. In practical applications, the convolution function can be realized by the convolution layer in the convolutional neural network, the pooling function can be realized by the pooling layer in the convolutional neural network, and the classification layer function can be realized by the neural network layer in the convolutional neural network.

[0065] In an embodiment of the present application, the convolution layer may include a plurality of convolution operators, which are also called kernels. The convolution operator is equivalent to a filter that extracts specific information from an input image matrix in image processing. The convolution operator can essentially be a weight matrix, which is usually predefined. In the process of performing a convolution operation on an image, the weight matrix is ​​usually processed one pixel after another (or two pixels after two pixels, depending on the value of the step length stride) in the horizontal direction on the input image, thereby completing the work of extracting specific features from the image. The size of the weight matrix should be related to the size of the image. It should be noted that the depth dimension of the weight matrix is ​​the same as the depth dimension of the input image. In the process of performing a convolution operation, the weight matrix extends to the entire depth of the input image. Therefore, convolution with a single weight matrix will produce a convolution output with a single depth dimension, but in most cases a single weight matrix is ​​not used, but multiple weight matrices of the same size (row × column), i.e., multiple isotype matrices, are applied. The output of each weight matrix is ​​stacked to form the depth dimension of the convolution image, and the dimension here can be understood as being determined by the "multiple" described above. Different weight matrices can be used to extract different features in the image, for example, one weight matrix is ​​used to extract image edge information, another weight matrix is ​​used to extract specific colors of the image, and another weight matrix is ​​used to blur unnecessary noise in the image, etc. The multiple weight matrices have the same size (rows × columns), and the feature maps extracted by the multiple weight matrices of the same size are also the same size. The extracted multiple feature maps of the same size are then merged to form the output of the convolution operation.

[0066] The weight values ​​in these weight matrices need to be obtained through a lot of training in practical applications. The weight matrices formed by the weight values ​​obtained through training can be used to extract information from the input image, so that the convolutional neural network can make correct predictions.

[0067] The pooling layer is used to sample the data and reduce the amount of data. For example, taking the data as image data, in the process of image processing, the spatial size of the image can be reduced through the pooling layer. In general, the pooling layer may include an average pooling operator and / or a maximum pooling operator to sample the input image to obtain an image of smaller size. The average pooling operator can calculate the pixel values ​​in the image within a specific range to generate an average value as the result of average pooling. The maximum pooling operator can take the pixel with the largest value in the range as the result of maximum pooling within a specific range. In addition, just as the size of the weight matrix used in the convolution layer should be related to the image size, the operator in the pooling layer should also be related to the size of the image. The size of the image output after processing by the pooling layer can be smaller than the size of the image input to the pooling layer. Each pixel in the image output by the pooling layer represents the average value or maximum value of the corresponding sub-region of the image input to the pooling layer.

[0068] After being processed by the convolution layer / pooling layer, the convolution neural network is not sufficient to output the required output information. Because as mentioned above, the convolution layer / pooling layer will only extract features and reduce the parameters brought by the input image. However, in order to generate the final output information (the required class information or other related information), the convolution neural network needs to use the neural network layer to generate one or a group of outputs of the required number of classes. Therefore, the neural network layer may include multiple hidden layers and an output layer, and the parameters contained in the multiple hidden layers can be pre-trained according to the relevant training data of the specific task type. For example, the task type may include image recognition, image classification, image super-resolution reconstruction, etc.

[0069] After the multiple hidden layers in the neural network layer, the last layer of the entire convolutional neural network is the output layer. The output layer has a loss function similar to the classification cross entropy, which is specifically used to calculate the prediction error. Once the forward propagation of the entire convolutional neural network is completed, the back propagation will begin to update the weight values ​​and biases of the previously mentioned layers to reduce the loss of the convolutional neural network and the error between the result output by the convolutional neural network through the output layer and the ideal result.

[0070] S102, in the at least one computation subgraph, converting the data structure and variables of the functional function included in at least one target operator according to the functional attributes corresponding to each computation subgraph; wherein the converted data structure and variables can be executed by the functional function;

[0071] Specifically, for at least one computational subgraph, the data structure and variables of the functional function contained in at least one target operator are converted according to the functional attributes corresponding to each computational subgraph, so that the converted data structure and variables can be executed by the corresponding functional function contained in the target operator. Among them, the target operator is at least one operator contained in at least one computational subgraph, that is, there can be multiple target operators. The data structure and variables of the functional function contained in each target operator need to be converted according to the functional attributes corresponding to the computational subgraph, so that the converted data structure and variables can be executed by the corresponding functional function contained in the target operator.

[0072] S103, based on the converted data structure and variables, optimizing the execution order of the at least one target operator and the scalar operations included in the at least one target operator to obtain a second intermediate representation;

[0073] Specifically, in the process of converting between intermediate languages, adding new operations can control the conversion process between different intermediate representations. Therefore, based on the converted data structure and variables, the execution order of the target operator can be optimized. For example, if there is only one target operator, the execution order can be optimized within the target operator. If there are multiple target operators, the execution order can be optimized within the target operator and between target operators to complete the specific functions of the intermediate representation with better performance. At the same time, the scalar operations contained in each target operator can also be optimized to obtain the second intermediate representation.

[0074] It should be noted that the intermediate representation, as an intermediary for translating between source code and target code, can express the data structure and related algorithms of the source program. By compiling the intermediate representation, the target code executable by the back-end hardware device can be obtained. A good intermediate representation must have the ability to accurately express the source program information and be easy to transform. During the transformation process, the intermediate representation will undergo multiple operations, making it very complicated. Therefore, in order to improve the running performance of the compiled target code, it is necessary to optimize the first intermediate representation obtained from the deep learning model.

[0075] S104, compile the second intermediate representation to obtain a target code executable by a backend hardware device, so that the backend hardware device executes the target code to implement the function of the target code.

[0076] Specifically, the optimized second intermediate representation is compiled to obtain a target code executable by the back-end hardware device. The back-end hardware device executes the target code to implement the function of the target code.

[0077] This embodiment optimizes the intermediate representation to achieve front-end framework independence and back-end platform independence, thereby improving the portability of the intermediate representation to be applicable to a variety of hardware platforms.

[0078] As an optional embodiment, the S103, based on the converted data structure and variables, optimizing the execution order of the at least one target operator and the scalar operation included in the at least one target operator to obtain a second intermediate representation, includes:

[0079] Based on the converted data structure and variables, the execution order of the at least one target operator is optimized, and the scalar operation contained in the at least one target operator after the execution order optimization is converted into a vector operation to obtain the second intermediate representation; wherein the scalar operation is used to represent the processing of scalar data in the target operator; and the vector operation is used to represent the processing of vector data in the target operator;

[0080] or,

[0081] Based on the converted data structure and variables, the scalar operation included in the at least one target operator is converted into a vector operation, and the execution order of the at least one target operator after the conversion operation is optimized to obtain the second intermediate representation.

[0082] Specifically, when optimizing the execution order of the target operator and the scalar operations contained in the target operator, the execution order of the target operator can be optimized first, and then the scalar operations contained in the target operator after the execution order optimization can be converted into vector operations; or the scalar operations contained in the target operator can be converted into vector operations first, and then the execution order of the target operator after the conversion operation can be optimized.

[0083] This embodiment optimizes the operator execution order and the scalar operation inside the operator of the first intermediate representation obtained from the deep learning model, so that the optimized second intermediate representation can express the information of the source program more accurately.

[0084] As an optional embodiment, if the number of the target operators is greater than or equal to 2, the execution order of the target operators is optimized, including:

[0085] Acquire an irrelevant operator; wherein the irrelevant operator is used to indicate that there is no directed edge between one operator and another operator in the target operator;

[0086] The execution order of the unrelated operators is converted into a parallel execution order.

[0087] Specifically, in this embodiment, each computation subgraph includes at least one group of operators, and therefore, the number of target operators is greater than or equal to 2. When optimizing the execution order of the target operator, an irrelevant operator is obtained, and the irrelevant operator refers to an operator in the target operator that has no directed edge between one operator and another operator. Since there is no directed edge between the two operators, the execution order of the two operators does not affect each other, and therefore, the execution order of the irrelevant operators is converted into a parallel execution order.

[0088] It should be noted that when optimizing the execution order of the target operator, if there is only one target operator, the execution order can be optimized within the target operator. If there are multiple target operators, the execution order can be optimized within the target operator and between target operators. When optimizing the execution order within the target operator, irrelevant function functions that do not have dependencies in the target operator are obtained, and the execution order of the irrelevant function functions is converted into a parallel execution order.

[0089] This embodiment optimizes the execution order of the target operators to achieve the specific functions of the intermediate representation with better performance.

[0090] As an optional embodiment, converting the scalar operation included in the at least one target operator into a vector operation includes:

[0091] For each of the target operators, obtaining scalar operations in the target operator that have the same functions and are unrelated;

[0092] Converts functionally identical and unrelated scalar operations to vector operations.

[0093] Specifically, when converting the scalar operations included in the target operator into vector operations, for each target operator, first obtain the scalar operations with the same function and unrelated in the target operator, and convert these scalar operations with the same function and unrelated into vector operations. For example, if the scalar operations in the target operator are all addition operations, and each addition operation is unrelated, then these unrelated addition operations can be converted into vector operations.

[0094] It should be noted that for simple operations, such as addition, subtraction, logical operations, bit operations and other regular operations, scalar operations can be replaced one by one with vector operations; for complex operations, such as array assignment, conditional execution, etc., multiple scalar operations can be converted into one vector operation.

[0095] This embodiment simplifies the intermediate representation before inputting it into the vectorizer, uses concise statements to describe complex operations, reduces the complexity of the vectorizer input, and improves the vectorization mapping capability.

[0096] As an optional embodiment, converting a scalar operation included in at least one of the target operators into a vector operation includes:

[0097] For each of the target operators, when the scalar operation included in the target operator is a stride memory access, each scalar operation corresponding to the stride memory access is converted into multiple vector operations.

[0098] Specifically, when this embodiment converts the scalar operation included in the target operator into a vector operation, for each target operator, when the scalar operation included in the target operator is a stride memory access, each scalar operation corresponding to the stride memory access is converted into multiple vector operations.

[0099] This embodiment converts a scalar operation into multiple vector operations, so that the target operator can achieve more functions, thereby making the computation subgraph have more functional attributes.

[0100] In a second aspect, the embodiment of the present invention further provides a data processing method. Taking image processing as an example, see Figure 2 , Figure 2 1 is a flow chart of an embodiment of a data processing method provided by the second aspect of the present invention. The data processing method comprises:

[0101] S201, obtaining a target code and an image to be processed;

[0102] S202, processing the image to be processed according to the target code to obtain an image processing result; wherein the target code is obtained by the data processing method described in any embodiment of the first aspect of the present invention.

[0103] Exemplarily, this embodiment can be executed by a graphics processing unit (GPU). After obtaining the target code and the image to be processed, the image processor processes the image to be processed according to the target code, and obtains the image processing result accordingly. The target code can be obtained by optimizing the data processing method described in any of the above embodiments, and the image processing can include but is not limited to any one of convolution processing, classification processing and grayscale processing. By optimizing the intermediate representation and then compiling to obtain the target code, the running efficiency of the target code can be improved, thereby improving the image processing efficiency.

[0104] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should know that the present disclosure is not limited by the order of the actions described, because according to the present disclosure, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily required by the present disclosure.

[0105] It should be further explained that Figure 1 For example, although Figure 1 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 1 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0106] In a third aspect, an embodiment of the present invention further provides a data processing device that can implement all the processes of the data processing method in any embodiment of the first aspect. Figure 3 , Figure 3 301 is a schematic diagram of a data processing device according to an embodiment of the present invention. The data processing device comprises a receiving unit 301 and a processing unit 302:

[0107] The receiving unit 301 is used to obtain a first intermediate representation of a deep learning model; wherein the deep learning model is a model built based on a front-end model framework, the first intermediate representation includes at least one computational subgraph, each computational subgraph includes at least one group of operators and directed edges related to the at least one group of operators, and the directed edges carry tensor data; each computational subgraph corresponds to a functional attribute; each operator includes at least one functional function;

[0108] The processing unit 302 is used to convert the data structure and variables of the functional function contained in at least one target operator in the at least one computational subgraph according to the functional attributes corresponding to each computational subgraph; wherein the converted data structure and variables can be executed by the functional function; based on the converted data structure and variables, the execution order of the at least one target operator and the scalar operations contained in the at least one target operator are optimized to obtain a second intermediate representation; the second intermediate representation is compiled to obtain a target code executable by a backend hardware device, so that the backend hardware device executes the target code to realize the function of the target code.

[0109] As an optional embodiment, the processing unit 302 optimizes the execution order of the at least one target operator and the scalar operation included in the at least one target operator based on the converted data structure and variables to obtain a second intermediate representation, including:

[0110] Based on the converted data structure and variables, the execution order of the at least one target operator is optimized, and the scalar operation contained in the at least one target operator after the execution order optimization is converted into a vector operation to obtain the second intermediate representation; wherein the scalar operation is used to represent the processing of scalar data in the target operator; and the vector operation is used to represent the processing of vector data in the target operator;

[0111] or,

[0112] Based on the converted data structure and variables, the scalar operation included in the at least one target operator is converted into a vector operation, and the execution order of the at least one target operator after the conversion operation is optimized to obtain the second intermediate representation.

[0113] As an optional embodiment, if the number of the target operators is greater than or equal to 2, the processing unit optimizes the execution order of the target operators, including:

[0114] Acquire an irrelevant operator; wherein the irrelevant operator is used to indicate that there is no directed edge between one operator and another operator in the target operator;

[0115] The execution order of the unrelated operators is converted into a parallel execution order.

[0116] As an optional embodiment, the processing unit converts the scalar operation included in the at least one target operator into a vector operation, including:

[0117] For each of the target operators, obtaining scalar operations in the target operator that have the same functions and are unrelated;

[0118] Converts functionally identical and unrelated scalar operations to vector operations.

[0119] As an optional embodiment, the processing unit converts the scalar operation included in the at least one target operator into a vector operation, including:

[0120] For each of the target operators, when the scalar operation included in the target operator is a stride memory access, each scalar operation corresponding to the stride memory access is converted into multiple vector operations.

[0121] It should be noted that the relevant specific implementation schemes and beneficial effects of each embodiment of a data processing device provided in the third aspect of an embodiment of the present invention are the same as the specific implementation schemes and beneficial effects of each embodiment of a data processing method provided in the first aspect of an embodiment of the present invention, and will not be repeated here.

[0122] In a fourth aspect, an embodiment of the present invention further provides a data processing device. Taking image processing as an example, please refer to Figure 4 , Figure 4 401 is a schematic diagram of a data processing device according to an embodiment of the present invention. The data processing device comprises a receiving unit 401 and a processing unit 402:

[0123] The receiving unit 401 is used to obtain the target code and the image to be processed;

[0124] The processing unit 402 is used to process the image to be processed according to the target code to obtain an image processing result; wherein the target code is obtained by the data processing method described in any embodiment of the first aspect above.

[0125] In a fifth aspect, an embodiment of the present invention further provides a data processing device, see Figure 5 , Figure 5 It is a structural diagram of an embodiment of a data processing device provided by the fifth aspect of the present invention. The data processing device includes a memory 501 and a processor 502, the memory is used to store programs, and the processor executes the programs stored in the memory. When the programs stored in the memory are executed, the processor is used to execute the data processing method described in any embodiment of the first aspect above, or the data processing method described in the embodiment of the second aspect above.

[0126] As an optional embodiment, the computer program may be divided into one or more modules / units (such as computer program 1, computer program 2, ...), and the one or more modules / units are stored in the memory 501 and executed by the processor 502 to complete the present invention. The one or more modules / units may be a series of computer program instruction segments capable of completing specific functions, and the instruction segments are used to describe the execution process of the computer program in the data processing device.

[0127] The processor 502 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor, or the processor 502 may be any conventional processor. The processor 502 is the control center of the data processing device, and various parts of the data processing device are connected using various interfaces and lines.

[0128] The memory 501 mainly includes a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function, etc., and the data storage area can store related data, etc. In addition, the memory 501 can be a high-speed random access memory, or a non-volatile memory, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, and a flash card (Flash Card), etc., or the memory 501 can also be other volatile solid-state storage devices.

[0129] It should be noted that the above data processing device may include, but is not limited to, a processor and a memory. Those skilled in the art will understand that Figure 5 The structural diagram is only an example of the above-mentioned data processing device and does not constitute a limitation of the above-mentioned data processing device. It may include more or less components than shown in the figure, or combine certain components, or different components.

[0130] In the sixth aspect, an embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium includes a stored computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute the data processing method described in any embodiment of the first aspect above, or the data processing method described in the embodiment of the second aspect above.

[0131] If the various component modules of the above-mentioned device are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer product is stored in a computer-readable storage medium.

[0132] The computer-readable storage medium may be an internal storage unit of the device of the aforementioned embodiment, such as a hard disk or a memory. The computer-readable storage medium may also be an external storage device of the device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. Further, the computer-readable storage medium may also include both an internal storage unit of the device and an external storage device. The computer-readable storage medium is used to store the computer program and other programs and data required by the device. The computer-readable storage medium may also be used to temporarily store data that has been output or is to be output.

[0133] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing related hardware through a computer program, and the computer program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. The aforementioned storage medium includes: ROM, RAM, magnetic disk or optical disk and other media that can store program codes.

[0134] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0135] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0136] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0137] The above-mentioned integrated unit implemented in the form of a software functional unit can be stored in a computer-readable storage medium. The above-mentioned software functional unit is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor (English: processor) to perform some steps of the methods of each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (English: Read-Only Memory, abbreviated: ROM), random access memory (English: Random Access Memory, abbreviated: RAM), disk or optical disk and other media that can store program codes.

[0138] In summary, the embodiments of the present invention provide a data processing method, apparatus, device and computer storage medium, by obtaining a first intermediate representation of a deep learning model; wherein the deep learning model is a model built based on a front-end model framework, the first intermediate representation includes at least one computational subgraph, each computational subgraph includes at least one group of operators and directed edges related to the at least one group of operators, and the directed edges carry tensor data; each computational subgraph corresponds to a functional attribute; each operator includes at least one functional function; in the at least one computational subgraph, the data structure and variables of the functional function contained in at least one target operator are converted according to the functional attributes corresponding to each computational subgraph; wherein the converted data structure and variables can be executed by the functional function; based on the converted data structure and variables, the execution order of the at least one target operator and the scalar operations contained in the at least one target operator are optimized to obtain a second intermediate representation; the second intermediate representation is compiled to obtain a target code executable by a back-end hardware device, so that the back-end hardware device executes the target code and realizes the function of the target code. The embodiment of the present invention can achieve front-end framework independence and back-end platform independence by optimizing the intermediate representation, thereby improving the portability of the intermediate representation to be applicable to a variety of hardware platforms.

[0139] The above is a specific implementation of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A data processing method, characterized in that: include: Obtaining a first intermediate representation of a deep learning model; wherein the deep learning model is a model built based on a front-end model framework, the first intermediate representation includes at least one computational subgraph, each computational subgraph includes at least one group of operators and directed edges related to the at least one group of operators, and the directed edges carry tensor data; each computational subgraph corresponds to a functional attribute; and each operator includes at least one functional function; In the at least one computation subgraph, data structures and variables of a functional function included in at least one target operator are converted according to functional attributes corresponding to each computation subgraph; wherein the converted data structures and variables can be executed by the functional function; Based on the converted data structure and variables, optimizing the execution order of the at least one target operator and the scalar operations included in the at least one target operator to obtain a second intermediate representation; Compiling the second intermediate representation to obtain a target code executable by a backend hardware device, so that the backend hardware device executes the target code to implement a function of the target code; The step of optimizing the execution order of the at least one target operator and the scalar operations included in the at least one target operator based on the converted data structure and variables to obtain a second intermediate representation includes: Based on the converted data structure and variables, the execution order of the at least one target operator is optimized, and the scalar operation contained in the at least one target operator after the execution order optimization is converted into a vector operation to obtain the second intermediate representation; wherein the scalar operation is used to represent the processing of scalar data in the target operator; and the vector operation is used to represent the processing of vector data in the target operator; or, Based on the converted data structure and variables, the scalar operation included in the at least one target operator is converted into a vector operation, and the execution order of the at least one target operator after the conversion operation is optimized to obtain the second intermediate representation.

2. The data processing method according to claim 1, characterized in that: If the number of the target operators is greater than or equal to 2, the execution order of the target operators is optimized, including: Acquire an irrelevant operator; wherein the irrelevant operator is used to indicate that there is no directed edge between one operator and another operator in the target operator; The execution order of the unrelated operators is converted into a parallel execution order.

3. The data processing method according to claim 1, characterized in that: The converting the scalar operation included in the at least one target operator into a vector operation comprises: For each of the target operators, obtaining scalar operations in the target operator that have the same functions and are unrelated; Converts functionally identical and unrelated scalar operations to vector operations.

4. The data processing method according to claim 3, characterized in that: The converting the scalar operation included in the at least one target operator into a vector operation comprises: For each of the target operators, when the scalar operation included in the target operator is a stride memory access, each scalar operation corresponding to the stride memory access is converted into multiple vector operations.

5. A data processing method, characterized in that: include: Get the target code and the image to be processed; The image to be processed is processed according to the target code to obtain an image processing result; wherein the target code is obtained by the data processing method according to any one of claims 1 to 4.

6. A data processing device, characterized in that: Including receiving unit and processing unit: The receiving unit is used to obtain a first intermediate representation of a deep learning model; wherein the deep learning model is a model built based on a front-end model framework, the first intermediate representation includes at least one computational subgraph, each computational subgraph includes at least one group of operators and directed edges related to the at least one group of operators, and the directed edges carry tensor data; each computational subgraph corresponds to a functional attribute; each operator includes at least one functional function; The processing unit is used to convert the data structure and variables of the functional function contained in at least one target operator in the at least one computation subgraph according to the functional attributes corresponding to each computation subgraph; wherein the converted data structure and variables can be executed by the functional function; based on the converted data structure and variables, the execution order of the at least one target operator and the scalar operation contained in the at least one target operator are optimized to obtain a second intermediate representation; the second intermediate representation is compiled to obtain a target code executable by a backend hardware device, so that the backend hardware device executes the target code to realize the function of the target code; The processing unit optimizes the execution order of the at least one target operator and the scalar operation included in the at least one target operator based on the converted data structure and the variable to obtain a second intermediate representation, including: Based on the converted data structure and variables, the execution order of the at least one target operator is optimized, and the scalar operation contained in the at least one target operator after the execution order optimization is converted into a vector operation to obtain the second intermediate representation; wherein the scalar operation is used to represent the processing of scalar data in the target operator; and the vector operation is used to represent the processing of vector data in the target operator; or, Based on the converted data structure and variables, the scalar operation included in the at least one target operator is converted into a vector operation, and the execution order of the at least one target operator after the conversion operation is optimized to obtain the second intermediate representation.

7. The data processing device according to claim 6, characterized in that If the number of the target operators is greater than or equal to 2, the processing unit optimizes the execution order of the target operators, including: Acquire an irrelevant operator; wherein the irrelevant operator is used to indicate that there is no directed edge between one operator and another operator in the target operator; The execution order of the unrelated operators is converted into a parallel execution order.

8. The data processing device according to claim 6, characterized in that The processing unit converts the scalar operation included in the at least one target operator into a vector operation, including: For each of the target operators, obtaining scalar operations in the target operator that have the same functions and are unrelated; Converts functionally identical and unrelated scalar operations to vector operations.

9. The data processing device according to claim 8, characterized in that The processing unit converts the scalar operation included in the at least one target operator into a vector operation, including: For each of the target operators, when the scalar operation included in the target operator is a stride memory access, each scalar operation corresponding to the stride memory access is converted into multiple vector operations.

10. A data processing device, characterized in that: Including receiving unit and processing unit: The receiving unit is used to obtain the target code and the image to be processed; The processing unit is used to process the image to be processed according to the target code to obtain an image processing result; wherein the target code is obtained by the data processing method according to any one of claims 1 to 4.

11. A data processing device, characterized in that: include: A memory and a processor, wherein the memory is used to store a program, and the processor executes the program stored in the memory. When the program stored in the memory is executed, the processor is used to execute the data processing method according to any one of claims 1 to 4 or the data processing method according to claim 5.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored computer program, wherein when the computer program is executed, the device where the computer-readable storage medium is located is controlled to execute the data processing method according to any one of claims 1 to 4 or the data processing method according to claim 5.

Images