Method, device, storage medium and board for convolution of image data

Abstract

The present invention relates to electronic devices, devices, methods, and readable storage media for performing convolution on image data based on a graph optimization mechanism. The processing device of the present invention is included in an integrated circuit device, which includes a universal interconnect interface and a computing device. The computing device interacts with the processing device to jointly complete user-specified computing operations. The integrated circuit device may also include a storage device, which is connected to the computing device and the processing device, respectively, for storing data in the computing device and the processing device.

Description

Technical Field

[0001] The present invention generally relates to the field of computers, and more particularly to a method, an integrated circuit device, a storage medium, and a board for performing convolution on image data based on a graph optimization mechanism. Background Art

[0002] Deep learning has been applied to scenarios such as image recognition, speech recognition, natural language processing, and machine translation, achieving excellent results in industrial applications. Several deep learning frameworks have been developed, including TensorFlow, Caffe, Theano, TorCh, and MXNet, all of which support models such as deep neural networks, convolutional neural networks, deep belief networks, and recurrent neural networks. TensorFlow, one of the most popular deep learning frameworks, is an open-source computing framework developed by Google. TensorFlow is also a highly flexible neural network learning platform, utilizing a graph computing model and supporting high-level APIs such as Python, C++, Go, and Java. This framework conveniently integrates various deep learning algorithms.

[0003] With the continuous development of technology, various deep learning algorithms are also being continuously optimized to achieve the effect of accelerated operation. Based on the native graph optimization mechanism, how to add support for optimization operators in TensorFlow has become an urgent problem to be solved.

[0004] Therefore, the present invention proposes a solution for convolving image data based on a graph optimization mechanism. Summary of the Invention

[0005] In order to at least partially solve the technical problems mentioned in the background technology, the solution of the present invention provides a method, an integrated circuit device, a storage medium and a board card for convolving image data according to a graph optimization mechanism.

[0006] In one aspect, the present invention discloses a method for convolving image data according to a graph optimization mechanism, the method comprising: obtaining the image data; replacing conventional convolution nodes with optimized convolution nodes in the graph optimization mechanism; based on the graph optimization mechanism, determining whether the image data needs to be padded in dimension; and if so, inserting a padded node before the optimized convolution node.

[0007] In another aspect, the present invention discloses a computer-readable storage medium having computer program instructions stored thereon. When the computer program instructions are executed by a processing device, the aforementioned method is executed.

[0008] In another aspect, the present invention discloses an integrated circuit device comprising a processing device and a DRAM for storing executable instructions. The processing device is configured to call the instructions stored in the DRAM to execute the above method.

[0009] In another aspect, the present invention discloses a board comprising the integrated circuit device described above, for executing the aforementioned method.

[0010] The present invention replaces conventional convolution nodes with optimized convolution nodes in the graph optimization mechanism, and determines whether the image data needs to be padded according to the graph optimization mechanism. It can support the optimization of convolution operators at the framework level and improve computing performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The above and other objects, features and advantages of the exemplary embodiments of the present disclosure will become readily understood by reading the detailed description below with reference to the accompanying drawings. In the accompanying drawings, several embodiments of the present disclosure are shown in an exemplary and non-limiting manner, and the same or corresponding reference numerals represent the same or corresponding parts.

[0012] Figure 1 1 is a schematic diagram showing the structure of a board according to an embodiment of the present invention;

[0013] Figure 2 is a structural diagram showing an integrated circuit device according to an embodiment of the present invention;

[0014] Figure 3 Shows the software and hardware architecture design diagram of data flow programming in an embodiment of the present invention;

[0015] Figure 4 is a flow chart illustrating a method for performing convolution on image data in an embodiment of the present invention;

[0016] Figure 5 is a schematic diagram illustrating node replacement in an embodiment of the present invention;

[0017] Figure 6 is a schematic diagram illustrating inserting a completion node according to an embodiment of the present invention;

[0018] Figure 7 is a flow chart illustrating a method in an embodiment of the present invention;

[0019] Figure 8 is a flow chart illustrating another method in an embodiment of the present invention;

[0020] Figure 9 is a schematic diagram illustrating device designation in an embodiment of the present invention;

[0021] Figure 10 is a schematic diagram illustrating adding a conversion node in an embodiment of the present invention;

[0022] Figure 11 is a schematic diagram showing another additional conversion node in an embodiment of the present invention;

[0023] Figure 12 is another schematic diagram showing an embodiment of the present invention; and

[0024] Figure 13 Schematic diagram of an optimized convolution device in an embodiment of the present invention. DETAILED DESCRIPTION

[0025] The following will be combined with the accompanying drawings in the embodiments of the present disclosure to clearly and completely describe the technical solutions in the embodiments of the present disclosure. Obviously, the embodiments described are part of the embodiments of the present disclosure, not all of them. Based on the embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without making any creative efforts shall fall within the scope of protection of the present disclosure.

[0026] It should be understood that the terms "first," "second," "third," and "fourth," etc. in the claims, specification, and drawings of the present disclosure are used to distinguish different objects rather than to describe a specific order. The terms "include" and "comprising" used in the specification and claims of the present disclosure indicate the presence of the described features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or combinations thereof.

[0027] It should also be understood that the terms used in this disclosure are for the purpose of describing specific embodiments only and are not intended to limit the present disclosure. As used in this disclosure and the claims, the singular forms "a," "an," and "the" are intended to include the plural forms unless the context clearly indicates otherwise. It should further be understood that the term "and / or" as used in this disclosure and the claims refers to any and all possible combinations of one or more of the associated listed items, including and including these combinations.

[0028] As used in this specification and claims, the term "if" may be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting," depending on the context.

[0029] The specific embodiments of the present disclosure are described in detail below with reference to the accompanying drawings.

[0030] The term "operator" mentioned in the embodiments of the present invention is used from the perspective of computer computing (or from the perspective of software or algorithm), while the term "node" is a more vivid term (from the perspective of graphics or a more intuitive level). In terms of what they refer to, the terms "operator" and "node" actually refer to the same thing. That is, in the embodiments of the present invention, the terms "operator" and "node" can be considered to have the same meaning and can be used interchangeably, but are described from different perspectives.

[0031] Figure 1 FIG1 shows an application scenario of the present invention, which is a structural diagram of a board 10. Figure 1 As shown, board 10 includes chip 101, which is a system-on-chip (SoC), or system-on-chip, integrated with one or more combined processing devices. The combined processing device is an artificial intelligence computing unit that supports various deep learning and machine learning algorithms, such as image segmentation algorithms, video encoding and decoding algorithms, etc., to meet the intelligent processing needs in complex scenarios in fields such as computer vision, speech, natural language processing, and data mining. In particular, deep learning technology is widely used in the field of cloud intelligence. A notable feature of cloud intelligence applications is the large amount of input data, which places high demands on the platform's storage and computing capabilities. Board 10 of this embodiment is suitable for cloud intelligence applications and has extensive off-chip storage, on-chip storage, and powerful computing capabilities.

[0032] Chip 101 is connected to external device 103 via external interface device 102. External device 103 may be, for example, a server, computer, camera, monitor, mouse, keyboard, network card, or Wi-Fi interface. Data to be processed can be transmitted from external device 103 to chip 101 via external interface device 102. Calculation results from chip 101 can be transmitted back to external device 103 via external interface device 102. Depending on the application scenario, external interface device 102 may have different interface formats, such as a PCIe interface.

[0033] Board 10 also includes a memory device 104 for storing data, such as video sequences and image data, and includes one or more storage units 105. Memory device 104 is connected to a control device 106 and chip 101 via a bus for data transmission. Control device 106 in board 10 is configured to control the state of chip 101. To this end, in one application scenario, control device 106 may include a microcontroller (MCU).

[0034] Figure 2 FIG. 1 is a block diagram showing the combined processing device in the chip 101 of this embodiment. Figure 2As shown in , the combined processing device 20 includes a computing device 201 , an interface device 202 , a processing device 203 and a DRAM 204 .

[0035] The computing device 201 is configured to perform user-specified operations and is mainly implemented as a single-core intelligent processor or a multi-core intelligent processor to perform deep learning or machine learning calculations. It can interact with the processing device 203 through the interface device 202 to jointly complete the user-specified operations.

[0036] Interface device 202 is used to transmit data and control instructions between computing device 201 and processing device 203. For example, computing device 201 can obtain input video image data from processing device 203 via interface device 202 and write it to the on-chip storage device of computing device 201. Furthermore, computing device 201 can obtain control instructions from processing device 203 via interface device 202 and write them to the on-chip control cache of computing device 201. Alternatively or optionally, interface device 202 can also read data from the storage device of computing device 201 and transmit it to processing device 203.

[0037] The processing device 203 is a general processing device that performs basic controls including but not limited to data handling, starting and / or stopping the computing device 201. Depending on the implementation, the processing device 203 can be a central processing unit (CPU), a graphics processing unit (GPU), or one or more types of processors in other general and / or special processors, such as a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., and the number of such processors can be determined according to actual needs. As mentioned above, with respect to the computing device 201 of the present invention, it can be regarded as having a single-core structure or a homogeneous multi-core structure. However, when the computing device 201 and the processing device 203 are integrated and considered together, the two are regarded as forming a heterogeneous multi-core structure.

[0038] The DRAM 204 is used to store data to be processed. It is a DDR memory, typically 16G or larger, and is used to store data of the computing device 201 and / or the processing device 203 .

[0039] There are many deep learning programming frameworks available today, such as TensorFlow, Caffe, Caffe2, MXNet, and PyTorCh. Deep learning frameworks serve as the primary programming interface for building corresponding machine learning / deep learning networks for various machine learning algorithm models.

[0040] Figure 3 A diagram illustrating the hardware and software architecture design for data flow programming in one embodiment of the present invention includes an AI processor 301, a driver and operating system 302, a compiler and programming language 303, a library 304, a framework layer 305, and an application layer 306.

[0041] The AI ​​processor 301 takes into account both computational optimization and data handling optimization in its hardware design, uses customized computing units to accelerate computations, and uses on-chip storage to accelerate data handling, thereby achieving extremely high performance and energy efficiency. In order to support various algorithm optimizations, customized computing units and instruction sets are required, and the instruction set provides computing instructions of different granularities (scalar, vector, matrix). At the same time, taking into account various factors such as algorithm memory access characteristics, hardware cost, and verification difficulty, on-chip storage is used to optimize data handling. AI processors (such as DaDianNao released by the Institute of Computing Technology, Chinese Academy of Sciences) can achieve speeds more than 20 times that of mainstream GPUs (Nvidia K20M).

[0042] The driver and operating system 302 is primarily responsible for scheduling tasks on the AI ​​processor 301, including allocating and releasing device memory, scheduling tasks based on priority, and enabling communication and synchronization between multiple devices. Compiled programs, through the operating system and driver, schedule tasks for execution on specific processors: allocating and releasing device memory, enabling data transfer between devices, maintaining task queues, scheduling tasks based on priority, and enabling synchronization and collaboration between multiple devices.

[0043] The compiler and programming language 303 has developed an assembly language for the instruction set of the AI ​​processor 301, translating the deep learning operators developed for the AI ​​processor 301 into processor instruction combinations, making it easier to call the AI ​​processor 301 and use it efficiently.

[0044] The library 304 includes a runtime library 314 and a machine learning library 324. The runtime library 314 is a set of high-performance operator libraries specially developed for the AI ​​processor 301. The library 304 uses the instruction set of the AI ​​processor 301 and is partially optimized according to the instruction set of the AI ​​processor 301 to improve the running speed of the operator. The runtime library 314 is also used to complete the interaction between the general-purpose processor and the artificial intelligence processor. The runtime library 314 also provides a set of interfaces for the artificial intelligence processor. The machine learning library 324 is also used to accelerate various machine learning or deep learning algorithms on the artificial intelligence processor. The machine learning library 324 provides a set of efficient, general, flexible and scalable programming interfaces. The upper-level machine learning applications can directly use the programming interfaces of various programming frameworks (such as TensorFlow, Caffe, MXNet, etc.), or can be directly programmed using the interfaces provided by the machine learning library 324. The machine learning library 324 facilitates the calling of the hardware platform, and the runtime library 314 implements some basic commonly used operators, such as convolution, pooling and other operations.

[0045] The framework layer 305 adds encapsulation for operators developed for AI processors, primarily for operators in the runtime library 314. Furthermore, the framework layer 305 modifies related aspects such as task scheduling and memory management. The framework layer 305 can adopt the TensorFlow architecture.

[0046] The application layer 306 provides an application platform for deep learning algorithm developers, and based on the native framework 305, it expands the support for calling the AI ​​processor 301 during model runtime.

[0047] Among them, the framework layer 305 implements the encapsulation and support of operators in the high-performance operator library in the runtime library 314. The framework layer 305 mainly uses the data flow graph to build the calculation process of the deep learning model according to the graph optimization mechanism. The graph optimization mechanism refers to the use of the interface defined in the deep learning framework, the user designs the graph optimizer by himself, and then registers it to the deep learning framework for operation. The deep learning framework can convert the constructed network into a graph-level intermediate representation (IR), referred to as an IR graph or a computational graph. The computational graph contains nodes and edges. Each node of the computational graph represents an operation or operator, and the edge is used to represent the flow of data. Graph optimization is to modify the computational graph according to certain rules to achieve the effect of accelerated operation, such as constant folding, operator fusion, graph substitution and other optimization methods.

[0048] A neural network is composed of an input layer, a convolutional layer, an activation function, a pooling layer, and a fully connected layer. There are at least a few layers and at most hundreds of layers. Each layer executes an operator. For example, a convolutional layer executes a convolution operator. There are as many operators as there are layers. In this disclosure, when a specific layer is mentioned, it refers to the operator corresponding to that layer. When performing calculations at each layer of deep learning, a large amount of off-chip or on-chip access is required, especially reading input data from DRAM 204 to the computing device 201, and then storing the calculation results of the computing device 201 to DRAM 204. This frequent access consumes a lot of hardware resources.

[0049] Operator fusion addresses the problem of frequent accesses. It combines multiple operators into a single fused operator, using the inputs of the multiple operators as the input of the fused operator, and the output of the fused operator as the output of the multiple operators. This allows intermediate results between operators to be cached in on-chip memory as much as possible, reducing the number of DRAM accesses, thereby reducing the amount of model calculations and the number of kernel startups. This significantly reduces off-chip data transmission.

[0050] The Grappler is a TensorFlow graph optimization module that accelerates execution through graph simplification and other advanced optimizations. Think of a graph optimizer as a processing plant: it receives a graph, performs some optimizations on it, and returns a new graph. For example, the pruning graph optimizer is a native TensorFlow graph optimizer that removes redundant nodes from the graph, reducing the number of nodes requiring computation and speeding up execution.

[0051] In this embodiment of the present invention, image data is used as an example. The image data format includes the dimensions N, H, W, and C, where N represents the batch, H represents the height, W represents the width, and C represents the channel. N represents the total number of images in the batch, H represents the number of vertical pixels in the image, W represents the number of horizontal pixels, and C represents the number of channels (for example, the number of channels C for a black and white image is 1, while the number of channels C for an RGB color image is 3).

[0052] The input data type of the conventional convolution operator only supports float16 or float32 data. However, some cases only require eight-bit input data, such as uint8 type data. If float16 or float32 input data is used, a large amount of space will be wasted.

[0053] In the input data of conventional convolution, the C dimension needs to be padded to 16 numbers or 32 numbers (depending on the data type). However, in some scenarios, the input C dimension of the input data is relatively small, and the C dimension only needs to be padded to 4 numbers. Some folding optimization can be performed, and the performance is already good enough.

[0054] For image classification and object detection networks, the input data is generally images, and the most common first-layer operator is the convolution operator. In order to match the AI ​​processor 301, conventional convolution is usually converted into a convolution that is more suitable for a specific hardware structure to optimize performance. To this end, during the operator development process of the runtime library 314, the present invention targets Figure 1 and Figure 2 The hardware of the proposed convolution operator is optimized to replace the conventional convolution operator to improve performance. In the framework layer 305, a graph optimizer is developed to support this optimized convolution operator.

[0055] The FirstConv operator graph optimizer is a graph optimizer proposed in this paper. It replaces conventional convolution with FirstConv to achieve the effect of running faster. The FirstConv graph optimizer can be regarded as a factory. The function of this factory is to replace the first convolution layer in the computation graph with FirstConv. Both before and after the replacement are convolution operations, but the optimized convolution is more suitable. Figure 1 and Figure 2 The hardware structure of the replacement is called graph optimization, and the tool that completes this project is called a graph optimizer.

[0056] The present invention proposes a method for convolving image data based on a graph optimization mechanism, which supports optimized convolution operations based on the original convolution calculation graph to achieve the purpose of improving performance.

[0057] Figure 4 A flow chart of a method for convolving image data according to an embodiment of the present invention is shown. The method is applied to a single or multiple processor cores in the board 10 or the combined device 20 described above.

[0058] Step 401: Obtain image data. The input of the optimized convolution is image data. The image value is an integer between 0 and 255. It can be an original image data or an image optimized by another image optimizer as the input image data.

[0059] Step 402: Replace the regular convolution nodes with the optimized convolution nodes in the graph optimization mechanism. As mentioned earlier, the graph optimization mechanism is a set of interfaces provided by TensorFlow that allows users to define some graph optimizers and then register them in TensorFlow. First, all custom graph optimizers must inherit the CustomGraphOptimizer class (a class is a collection of related attributes and behaviors. It can be regarded as a template for a class of things, using the attribute characteristics and behavioral characteristics of the things to describe the class of things. For example, for birds, color is its attribute and singing is its behavior) and implement the Init and Optimize functions.

[0060] The Init function is typically called only once, primarily to perform some initialization work. Because a custom graph optimizer typically calls a parameterless constructor when it is created, initialization is primarily performed by the Init function. However, the convolutional graph optimizer has no variables that require initialization and does not accept external parameters, so the Init function is typically implemented as an empty function.

[0061] The Optimize function is responsible for optimizing the interaction between the convolutional graph optimizer and other modules. It is also the entry point for the convolutional graph optimizer, and all optimization logic begins here. It accepts the original image data or a graph optimized by another graph optimizer as input, then performs the relevant optimization operations. If the optimization fails, it returns the original input image; otherwise, it returns the optimized image. The Optimize function may be called multiple times.

[0062] Figure 5 A schematic diagram showing node replacement in an embodiment of the present invention is shown. Figure 5 a is the calculation diagram of conventional convolution, which inputs the input data and weights into the conventional convolution, performs the conventional convolution operation, and finally outputs the result of the operation. Figure 5 b. Replace the conventional convolution with the computational graph after the optimized convolution node, input the input data and weights into the optimized convolution, perform the optimized convolution operation, and finally output the result of the operation. In this embodiment, the optimized convolution is specifically designed for Figure 1 and Figure 2 The hardware was debugged and the diagram was modified according to certain rules. Figure 1 and Figure 2 When running on hardware, the acceleration effect can be achieved. The process from Figure 5a to Figure 5b is the process of graph optimization.

[0063] Step 403: Based on the graph optimization mechanism, determine whether the image data needs to be padded.

[0064] In this embodiment, the dimension of the optimized convolution is smaller than that of the conventional convolution. The optimized convolution only needs to pad the input C dimension to 4 numbers or less, while the input C dimension of the conventional convolution needs to be padded to 16 numbers or 32 numbers. When supporting the optimized convolution operator at the framework level, it is necessary to determine whether to perform a padding operation on the C dimension of the input image data based on the overall performance. Among them, the overall performance refers to the time consumed to complete the convolution operation. When the overall performance of the padding operation is greater than the non-padding operation, the dimension is padded. At this time, step 404 is executed to insert a padding node before the optimized convolution node. When the overall performance of the padding operation is less than or equal to the non-padding operation, the dimension is not padded, and there is no need to insert a padding node. At this time, step 405 is executed to stop inserting the padding node before the optimized convolution node. Among them, the padding node pads the channel dimension of the image data.

[0065] Figure 6 A schematic diagram of inserting padding nodes in an embodiment of the present invention is shown. When image data needs to be padded, a padding node is inserted before the optimized convolution node in the computation graph. Figure 6 a is a schematic diagram showing that no padding nodes are inserted. The input data and weights are input into the optimized convolution, and the convolution result is finally output. Figure 6 Figure b shows a schematic diagram of inserting padding nodes between the optimized convolution and the input. After the input data is padded by the padding nodes, it is fed into the optimized convolution along with the weights, and the convolution result is output. The input data for the optimized convolution is no longer the original input data, but rather the new input data obtained after the C channel has been padded.

[0066] The above embodiment is optimized on the basis of the conventional convolution operator to obtain an optimized convolution operator and improve performance. Therefore, in order to support the optimized convolution operator at the framework level, a graph optimization operation needs to be performed. When optimizing the graph, it is first necessary to find an optimized convolution node that meets the conditions. If there is no optimized convolution node, the graph optimization process is directly terminated. If there is an optimized convolution node, it is necessary to further determine whether the performance of the optimized convolution node is better than that of the conventional convolution node at the framework level. In this embodiment, performance refers to the length of time it takes to execute the node. The shorter the time used, the better the performance, and vice versa.

[0067] Figure 7 Another embodiment of the present invention for performing convolution on image data is shown.

[0068] Step 701: Get image data. Figure 4 In step 401, the input of the optimized convolution is image data, and the image value is an integer between 0 and 255. It can be an original image data or an image optimized by another image optimizer as the input image data.

[0069] Step 702: Calculate the first time period for executing the unpadding optimized convolution node. Under the premise of not padding the optimized convolution, calculate the time for executing only the optimized convolution.

[0070] Step 703: Calculate the second time period for executing the regular convolution node. Calculate the time for executing only the regular convolution.

[0071] Step 704: Determine whether the first time period is less than the second time period. If the first time period is less than the second time period, it means that executing the optimized convolution node is indeed faster than executing the conventional convolution node, and then execute step 705. This step is the same as Figure 4 Step 402.

[0072] Step 705: Replace the regular convolution node with the optimized convolution node in the graph optimization mechanism.

[0073] Step 706: This step is the same as Figure 4 In step 403, based on the graph optimization mechanism, it is determined whether the image data needs to be padded. If the overall performance of the padded operation is better than the non-padded operation, the padded dimension is performed, and step 707 is executed.

[0074] Step 707: Insert a padding node before the optimized convolution node. Figure 4 In step 404, the padding node pads the channel dimension of the image data.

[0075] When the overall performance of the padding operation is less than or equal to that of the non-padding operation, the dimension is not padded, and there is no need to insert a padding node. In this case, step 708 is executed.

[0076] Step 708: Stop inserting padding nodes before optimizing convolution nodes. Figure 4 Step 405.

[0077] Returning to step 704 , if it is determined in step 704 that the first time period is greater than or equal to the second time period, it indicates that replacing the conventional convolution node with the optimized convolution node does not generate any benefit, and then step 709 is executed.

[0078] Step 709: No optimization is performed, and the optimization process ends.

[0079] Figure 8 A flow chart of another method in an embodiment of the present invention is shown.

[0080] Step 801: This step is the same as Figure 4 In step 401, image data is obtained.

[0081] Step 802: This step is the same as Figure 4 In step 402, the conventional convolution nodes are replaced with optimized convolution nodes in the graph optimization mechanism.

[0082] Step 803: Calculate a first time period for executing the incomplete optimized convolution nodes.

[0083] Step 804: Calculate the third time period required to execute the padding optimization convolution node.

[0084] Step 805: Calculate the fourth time period required to complete the node.

[0085] Step 806: Determine whether the first time period is less than the sum of the third time period and the fourth time period.

[0086] If the first time period is not less than the sum of the third time period and the fourth time period, it indicates that it is beneficial to perform the supplementation, so step 807 is executed.

[0087] Step 807: insert the fill-in node. Figure 4 In step 404, if the time required to calculate the non-padded optimized convolution node is greater than or equal to the time required to calculate the padded optimized convolution node, it indicates that the non-padded optimized convolution node consumes relatively more time and has poor performance. Therefore, a padding node is inserted to pad the dimension. The padding node pads the channel dimension of the image data.

[0088] If the first time period is less than the sum of the third time period and the fourth time period, it means that the benefit of optimizing the convolution node without inserting the padding node is higher, so step 808 is executed.

[0089] Step 808: Stop inserting the padding node. If the time required to calculate the optimized convolution node without padding is less than the time required to calculate the optimized convolution node without padding and the time required to calculate the padding node, it indicates that the time required to calculate the optimized convolution node without padding is relatively less and the performance is better, so the padding node is not inserted.

[0090] in, Figure 8 Steps 803 to 806 are equivalent to Figure 4 Step 403 in the above process is based on a graph optimization mechanism to determine whether the image data needs to be padded with dimensions.

[0091] Generally speaking, the dimension of the optimized convolution is smaller than that of the conventional convolution. The optimized convolution only needs to pad the input C dimension to 4 numbers or smaller, while the input C dimension of the conventional convolution needs to be padded to 16 numbers or 32 numbers (determined by the data type). When supporting the optimized convolution operator at the framework level, it is necessary to determine whether to perform the padding operation on the C dimension of the input image data based on the overall performance. Among them, the overall performance in this embodiment refers to the time consumed to complete the convolution operation.

[0092] In an optional embodiment, when determining whether to fill the dimension (this step is the same as Figure 4 Step 403 or Figure 7 Step 706 or Figure 8 In step 803-step 805), this embodiment can also be implemented by setting an environment variable, and the value of the environment variable reflects the convolution padding strategy. The environment variable includes multiple values, each value represents a padding strategy, and different values ​​represent different padding strategies. Further, when the padding operation is performed on the input of the optimized convolution, the environment variable value can also specify the execution device. The padding strategy includes not performing a padding operation on the input of the optimized convolution, performing a padding operation on the input of the optimized convolution and the padding operation is performed on the first device, performing a padding operation on the input of the optimized convolution and the padding operation is performed on the second device, etc. Selecting the convolution padding strategy according to the environment variable specifically includes:

[0093] When the environment variable value is the first value, it means that no padding operation is performed on the input of the optimized convolution, and the placement of the padding node is stopped;

[0094] When the value of the environment variable is the second value, indicating that a padding operation is performed on the input of the optimized convolution and the padding operation is performed on the first device, a padding node is placed, and the placing step is performed by the first device, wherein the first device is used to obtain the image data and weights;

[0095] When the environment variable value is a third value, indicating that a padding operation is performed on the input of the optimized convolution and the padding operation is performed on the second device, a padding node is inserted, and the insertion step is executed by the second device, wherein the second device performs a convolution operation on the image data and the weight.

[0096] Specifically, the first value, the second value, and the third value are any representations that can distinguish three values. For example, the first value, the second value, and the third value are all numbers, such as 0, 1, and 2, or letters, such as a, b, and c. They can also include both numbers and letters, such as the first value is 1, the second value is a, and the third value is b. The present invention does not impose any limitation on this. The first device can be Figure 2 The processing device 203 in the second device may be Figure 2 The computing device 201 in.

[0097] The present invention also provides a solution for offline models. An offline model is a training model relative to an online model. Machine learning can be divided into two simple stages to solve business problems: offline and online (offline refers to before the model is deployed to the production environment, and online refers to after the model is deployed to the production environment). In the offline stage, you first need to train a model, and then perform an offline evaluation on the trained model to understand the performance of the model. The model trained in the offline stage is the offline model. The offline model includes compiled binary machine instructions that can be run directly on the artificial intelligence processor.

[0098] In the present invention, the runtime library 314 provides a set of interfaces for artificial intelligence processors, including an interface for loading offline models. Calling this interface can load offline models into the artificial intelligence processor. It should be noted that the runtime library 314 can be separated from the machine learning library 324 and use offline model files alone to complete the calculation of the neural network. Taking the mobile phone in the artificial intelligence processing device as an example, the mobile phone only includes the runtime library 314. When the mobile phone downloads the artificial intelligence application, the offline model included in the artificial intelligence application is run through the runtime library 314 in the mobile phone. The offline model in the artificial intelligence application is generated by the runtime library 314 and the machine learning library 324 in the artificial intelligence processing device on the development side, and then the offline model is packaged into the artificial intelligence application through the runtime library 314.

[0099] The offline model may include model structure information, weight data, input and output data. The offline model may also include model information such as version information of the offline model and version information of the machine learning processing instructions, which is not limited here. Furthermore, the model structure information may include the layer structure corresponding to the neural network model. For example, the offline model contains convolutional layers, normalization layers, scaling layers, and fully connected layers. The weight data includes the weights corresponding to each layer. The input and output data may include the scale of input and output data, for example, the input size of the image data is 50mm*50mm, and the pixel value range is (-1024, 3071). The input and output data may also include input and output quantity information, that is, defining several input data, several output data, etc. The offline model can be obtained through Figure 3 The machine learning library 324 and runtime library 314 are generated. Specifically, the machine learning library and runtime library package a series of data such as data and instructions used to execute neural network model calculations to generate an offline model.

[0100] In another embodiment of the present invention, the step of determining whether the image data requires dimensional padding can also be performed by determining whether the graph optimization mechanism uses an offline model. Because offline models can be run directly on an artificial intelligence processor, if the graph optimization mechanism uses an offline model, the embedding step is performed by the first device. If the graph optimization mechanism does not use an offline model, the embedding step is performed by the second device.

[0101] Figure 9 Assuming that the environment variable is a third value, the placing step is performed by the second device that performs a convolution operation on the image data and the weights. Figure 9 a shows a schematic diagram of replacing regular nodes with optimized convolution nodes. Figure 9 b is in Figure 9 Schematic diagram of padding optimization based on a. Depending on the value of the environment variable, the padding operation is performed on the second device. Unshaded nodes are running on the second device, while shaded nodes are running on the first device.

[0102] Furthermore, the embodiment of performing optimized convolution further includes: determining whether the data type of the image data is consistent with the data type supported by the graph optimization mechanism; when the data type of the image data is inconsistent with the data type supported by the graph optimization mechanism, adding a conversion node before the optimized convolution node to convert the data type of the image data to the data type supported by the graph optimization mechanism. When the data type of the image data is consistent with the data type supported by the graph optimization mechanism, stopping adding the conversion node before the optimized convolution node.

[0103] Specifically, if the optimized convolution supports eight-bit or smaller data types, such as unit8, while the regular convolution supports sixteen-bit or thirty-two-bit data types, such as float16 or float32, the data types supported by the optimized and regular convolutions are inconsistent. The graph optimization mechanism's interface is configured based on the optimized convolution nodes, so it's necessary to compare the image data type with the data types supported by the graph optimization mechanism to determine whether data type conversion is necessary. Conversion nodes are used to convert the data type of the input image data.

[0104] Figure 10 A schematic diagram showing adding a conversion node according to an embodiment of the present invention is shown. Figure 10 a shows a schematic diagram of conventional convolution, Figure 10 b shows a schematic diagram of replacing the regular convolution node with the optimized convolution node and adding a conversion node before the optimized convolution node.

[0105] Figure 11 A schematic diagram showing another example of adding a conversion node according to an embodiment of the present invention is shown. Figure 11a is also a schematic diagram of conventional convolution. When the conversion node and the padding node are inserted at the same time, the resulting calculation diagram is as follows Figure 11 b. The order of converting nodes and completing nodes is not limited.

[0106] Optionally, when data conversion is required, the aforementioned environment variable can also reflect the device performing the data conversion step. In the aforementioned embodiment, the device performing the data conversion is the same as the device performing the C-dimensional padding operation on the input image data to avoid copying data between different devices. If the two devices are inconsistent, an additional copy operation will be performed between them, copying data from the first device to the second device or from the second device to the first device, which will reduce performance.

[0107] Figure 12 A schematic diagram of another embodiment of the present invention is shown. When both padding nodes and conversion nodes are present, the device performing the data conversion is the same as the device performing the C-dimensional padding operation on the input image data. This embodiment uses the first device as an example. Optionally, the device performing the data conversion and the C-dimensional padding operation on the input image data can both be performed on the second device, which is not limited in this invention.

[0108] Figure 13 Another embodiment of the present invention is shown as a schematic diagram of a convolution optimization device. The device 1300 includes an acquisition module 1310, a replacement module 1320, a determination module 1330, an insertion module 1340, and an addition module 1350.

[0109] The acquisition module 1310 is used to acquire image data. The input of the optimized convolution is image data. The image value is an integer between 0 and 255. It can be an original image data or an image optimized by another image optimizer as the input image data.

[0110] The judgment module 1330 is used to determine whether to use the optimized convolution node to optimize the convolution. When the device 1300 performs graph optimization, the judgment module 1330 first needs to determine whether there is an optimized convolution node that meets the conditions. If there is no optimized convolution node, the device 1300 directly ends the graph optimization process. If there is an optimized convolution node, the judgment module 1330 needs to further determine whether the performance of the optimized convolution node is better than the performance of the conventional convolution node at the framework level. When the performance of the optimized convolution node is better than the performance of the conventional convolution node, the replacement module 1320 is used to replace the conventional convolution node with the optimized convolution node in the graph optimization mechanism. The judgment module 1330 includes a calculation module 1331 and a comparison module 1332. The specific judgment process of the judgment module 1330 is as follows.

[0111] Calculation module 1331 is used to calculate a first time period for executing the incomplete optimized convolution node. Calculation module 1331 is also used to calculate a second time period for executing the regular convolution node. Comparison module 1332 is used to compare whether the first time period is less than the second time period. If the first time period is less than the second time period, indicating that executing the optimized convolution node is indeed faster than executing the regular convolution node, replacement module 1320 is used to replace the regular convolution node with the optimized convolution node in the graph optimization mechanism.

[0112] Furthermore, after determining that the convolution is optimized using the optimized convolution node, determining whether the image data requires dimensionality padding based on the graph optimization mechanism is further configured to determine if the image data requires dimensionality padding. If the image data requires dimensionality padding, inserting module 1340 is configured to insert the padding node before the optimized convolution node. If the image data does not require dimensionality padding, inserting module 1340 stops inserting the padding node before the optimized convolution node.

[0113] More specifically, the calculation module 1331 is further configured to calculate a third time period required to complete the optimized convolution node and a fourth time period required to complete the completed node. The comparison module 1332 is further configured to compare whether the first time period is less than the third time period plus the fourth time period. If the first time period is less than or equal to the third time period plus the fourth time period, i.e., the time required to complete the optimized convolution node without completing the optimization is less than or equal to the time required to complete the optimized convolution node and the time required to complete the node, indicating that the optimized node with the completed node is not optimized better than the optimized node without the completed node, the placement module 1340 stops placing the completed node. If the first time period is greater than the third time period plus the fourth time period, i.e., the time required to complete the optimized convolution node without completing the optimization is greater than the time required to complete the optimized convolution node and the time required to complete the node, indicating that the optimized node with the completed node is faster than the optimized node without the completed node, the placement module 1340 places the completed node.

[0114] When an environment variable is provided in the device 1300, the judgment module 1330 can also be used to judge the environment variable value, which reflects the convolution padding strategy. Specifically, when the environment variable value is the first value, it indicates that the convolution padding strategy is not to insert the padding node in the optimized convolution node, and the insertion module 1340 stops inserting the padding node. When the environment variable value is the second value, it indicates that the convolution padding strategy is to insert the padding node in the optimized convolution node, and the insertion step is performed by the first device, and the first device is used to obtain the image data and weights, and the insertion module 1340 inserts the padding node. When the environment variable value is the third value, it indicates that the convolution padding strategy is to insert the padding node in the optimized convolution node, and the insertion step is performed by the second device, wherein the second device performs a convolution operation on the image data and weights, and the insertion module 1340 inserts the padding node. Wherein, the above-mentioned first device can be Figure 2 The processing device 203 in the second device may be Figure 2 The placement module 1340 is located in the first device or the second device, and the location of the placement module 1340 is consistent with the execution device of the placement step.

[0115] The determination module 1330 is further configured to determine whether the data type of the image data is consistent with the data type supported by the graph optimization mechanism. If the data type of the image data is inconsistent with the data type supported by the graph optimization mechanism, the addition module 1350 is configured to add a conversion node before the optimization convolution node to convert the data type of the image data to a data type supported by the graph optimization mechanism.

[0116] Another embodiment of the present invention is a computer-readable storage medium having stored thereon computer program instructions for performing graph optimization. When the computer program instructions are executed by a server, the server includes a processor and a memory, the memory stores the aforementioned computer program instructions, and the processor runs the computer program instructions in the memory. In some implementation scenarios, the above-mentioned integrated unit can be implemented in the form of a software program module. If implemented in the form of a software program module and sold or used as an independent product, the integrated unit can be stored in a computer-readable memory. Based on this, when the solution of the present invention is embodied in the form of a software product (such as a computer-readable storage medium), the software product can be stored in a memory, which may include several instructions for causing a computer device (such as a personal computer, a server, or a network device, etc.) to perform some or all of the steps of the method described in the embodiment of the present invention. When the solution of the present invention is embodied in the form of an electronic device, the device includes a processor and a memory, the memory being used to store processor-executable instructions, and the processor being configured to call the instructions stored in the memory to execute the above-mentioned method for encoding and decoding video images. The aforementioned memory may include, but is not limited to, various media that can store program codes, such as a USB flash drive, a flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard drive, a magnetic disk, or an optical disk.

[0117] The present invention replaces conventional convolution nodes with optimized convolution nodes in the graph optimization mechanism, and determines whether the image data needs to be padded in dimension according to the graph optimization mechanism, so as to support optimized convolution operators at the framework level and improve computing performance. At the same time, the present invention can also convert the data type of the input data by adding a conversion node when the data type of the input data is different from the data type supported by the graph optimization mechanism, and can support convolution operations on data of multiple data types. Finally, the present invention also specifies the devices for dimension padding operations and data type conversion operations by setting environment variables, which can reduce the copying operations of data between different devices and further improve performance.

[0118] Depending on the application scenario, the electronic device or device of the present invention may include a server, a cloud server, a server cluster, a data processing device, a robot, a computer, a printer, a scanner, a tablet computer, a smart terminal, a PC device, an Internet of Things terminal, a mobile terminal, a mobile phone, a driving recorder, a navigator, a sensor, a camera, a camera, a video camera, a projector, a watch, a headset, a mobile storage, a wearable device, a visual terminal, an automatic driving terminal, a vehicle, a household appliance, and / or a medical device. The vehicle includes an airplane, a ship and / or a vehicle; the household appliance includes a television, an air conditioner, a microwave oven, a refrigerator, an electric rice cooker, a humidifier, a washing machine, an electric light, a gas stove, and a range hood; the medical device includes an MRI, an ultrasound machine and / or an electrocardiograph. The electronic device or device of the present invention may also be applied to the Internet, the Internet of Things, data centers, energy, transportation, public administration, manufacturing, education, power grids, telecommunications, finance, retail, construction sites, medical care and other fields. Furthermore, the electronic device or device of the present invention may also be used in application scenarios related to artificial intelligence, big data and / or cloud computing, such as the cloud, edge, and terminal. In one or more embodiments, electronic devices or apparatuses with high computing power according to the solution of the present invention can be applied to cloud devices (such as cloud servers), while electronic devices or apparatuses with low power consumption can be applied to terminal devices and / or edge devices (such as smartphones or cameras). In one or more embodiments, the hardware information of the cloud device and the hardware information of the terminal device and / or edge device are compatible with each other, so that according to the hardware information of the terminal device and / or edge device, appropriate hardware resources can be matched from the hardware resources of the cloud device to simulate the hardware resources of the terminal device and / or edge device, so as to complete the unified management, scheduling and collaborative work of end-to-end or cloud-edge-to-end.

[0119] It should be noted that, for the purpose of simplicity, the present invention describes some methods and embodiments thereof as a series of actions and combinations thereof, but those skilled in the art will understand that the scheme of the present invention is not limited by the order of the described actions. Therefore, based on the disclosure or teachings of the present invention, those skilled in the art will understand that some of the steps therein can be performed in other orders or simultaneously. Further, those skilled in the art will understand that the embodiments described in the present invention can be regarded as optional embodiments, that is, the actions or modules involved therein are not necessarily necessary for the implementation of one or more schemes of the present invention. In addition, depending on the different schemes, the present invention also has different emphases on the description of some embodiments. In view of this, those skilled in the art will understand that the parts that are not described in detail in a certain embodiment of the present invention may also refer to the relevant descriptions of other embodiments.

[0120] In terms of specific implementation, based on the disclosure and teachings of the present invention, those skilled in the art can understand that several embodiments disclosed in the present invention can also be implemented in other ways not disclosed herein. For example, with respect to the various units in the electronic device or device embodiments described above, this article splits them based on the consideration of logical functions, and there may be other ways of splitting them in actual implementation. For another example, multiple units or components can be combined or integrated into another system, or some features or functions in the units or components can be selectively disabled. With respect to the connection relationship between different units or components, the connection discussed above in conjunction with the accompanying drawings can be a direct or indirect coupling between units or components. In some scenarios, the aforementioned direct or indirect coupling involves a communication connection using an interface, wherein the communication interface can support electrical, optical, acoustic, magnetic or other forms of signal transmission.

[0121] In the present invention, the units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units. The aforementioned components or units may be located in the same location or distributed across multiple network elements. In addition, according to actual needs, some or all of the units may be selected to achieve the purpose of the solutions described in the embodiments of the present invention. In addition, in some scenarios, multiple units in the embodiments of the present invention may be integrated into a single unit or each unit may exist physically separately.

[0122] In other implementation scenarios, the above-mentioned integrated units may also be implemented in the form of hardware, that is, as specific hardware circuits, which may include digital circuits and / or analog circuits. The physical implementation of the hardware structure of the circuit may include, but is not limited to, physical devices, which may include, but are not limited to, devices such as transistors or memristors. In view of this, the various devices described herein (such as computing devices or other processing devices) may be implemented using appropriate hardware processors, such as central processing units, GPUs, FPGAs, DSPs, and ASICs. Furthermore, the aforementioned storage unit or storage device can be any appropriate storage medium (including magnetic storage medium or magneto-optical storage medium, etc.), which can be, for example, resistive random access memory (RRAM), dynamic random access memory (DRAM), static random access memory (SRAM), enhanced dynamic random access memory (EDRAM), high bandwidth memory (HBM), hybrid memory cube (HMC), ROM and RAM, etc.

[0123] The foregoing content can be better understood in accordance with the following terms:

[0124] Clause A1. A method for convolving image data according to a graph optimization mechanism, the method comprising: obtaining the image data; replacing conventional convolution nodes with optimized convolution nodes in the graph optimization mechanism; based on the graph optimization mechanism, determining whether the image data needs to be padded in dimension; and if so, inserting a padded node before the optimized convolution node.

[0125] Clause A2. The method according to Clause A1 further includes: calculating a first time period for executing the unfilled optimized convolution node; calculating a second time period for executing the conventional convolution node; determining whether the first time period is less than the second time period; and if so, executing a replacement step.

[0126] Clause A3. According to the method described in Clause A2, the judgment step also includes: calculating the third time period required to execute the padding of the optimized convolution node; calculating the fourth time period required to execute the padding node; and judging whether the first time period is less than the third time period plus the fourth time period; wherein, the method also includes: if the first time period is less than the third time period plus the fourth time period, stopping placing the padding node.

[0127] Clause A4. According to the method described in Clause A3, the judgment step includes: judging the value of the environment variable, and the value of the environment variable reflects the convolution padding strategy; wherein, the method also includes: when the value of the environment variable is the first value, stopping placing the padding node.

[0128] Item A5. The method according to Item A4, wherein when the value of the environment variable is a second value, the placing step is performed by a first device, and the first device is used to obtain the image data and weight.

[0129] Item A6. The method according to Item A4, wherein when the value of the environment variable is a third value, the placing step is performed by a second device, and the second device performs a convolution operation on the image data and the weight.

[0130] Item A7. The method according to Item A5, wherein the step of determining whether the image data needs to be dimensionally supplemented further comprises: determining whether the graph optimization mechanism uses an offline model; and if so, the embedding step is performed by the first device.

[0131] Item A8. According to the method described in A1, the padding node pads the channel dimension of the image data.

[0132] Clause A9. The method according to any one of items A4-A8 further includes: determining whether the data type of the image data is consistent with the data type supported by the graph optimization mechanism; when inconsistent, adding a conversion node before the optimized convolution node to convert the data type of the image data into the data type supported by the graph optimization mechanism.

[0133] Item A10. The method according to Item A9, wherein the environment variable is further used to reflect the device that performs the adding step.

[0134] Item A11. The method according to Item 10, wherein the placing step is performed by the same device as that performing the adding step.

[0135] Item A12. A computer-readable storage medium having computer program instructions stored thereon, wherein the computer program instructions, when executed by a processor, implement the method described in any one of Items A1 to A11.

[0136] Item A13. An integrated circuit device, characterized in that it includes: a processing device; and a DRAM for storing executable instructions; wherein the processing device is configured to call the instructions stored in the DRAM to execute the method described in any one of items A1 to A11.

[0137] Item A14. A board comprising the integrated circuit device of Item A13.

[0138] Item A15. A convolution optimization device, comprising an acquisition module, a replacement module, a judgment module, and an insertion module, wherein the acquisition module is used to acquire the image data; the replacement module is used to replace the conventional convolution node with the optimized convolution node in the graph optimization mechanism; the judgment module is used to determine whether the image data needs to be padded in dimension based on the graph optimization mechanism; and if the image data needs to be padded in dimension, the insertion module is used to insert the padded node before the optimized convolution node.

[0139] Item A16. An apparatus according to Item A15, wherein the judgment module includes a calculation module and a comparison module, the calculation module is used to calculate a first time period for executing the unfilled optimized convolution node; and to calculate a second time period for executing the conventional convolution node; the comparison module is used to compare whether the first time period is smaller than the second time period; if so, the replacement module performs a replacement step.

[0140] Item A17. According to the device described in Item A16, the calculation module is also used to calculate the third time period required to complete the optimized convolution node and the fourth time period required to calculate the completion node, and the comparison module is also used to compare whether the first time period is less than the third time period plus the fourth time period; if the first time period is less than the third time period plus the fourth time period, the placement module stops placing the completion node.

[0141] Clause A18. According to the device described in Clause A17, the judgment device is also used to judge the environment variable value, and the environment variable value reflects the convolution padding strategy; wherein, the device also includes: when the environment variable value is a first value, the placement module stops placing the padding node.

[0142] Item A19. An apparatus according to Item A18, wherein when the environment variable value is a second value, the placement module places the completion node, and the placement step is performed by a first device, which is used to obtain the image data and weight.

[0143] Item A20. An apparatus according to Item A19, wherein when the environment variable value is a third value, the placement module places the padding node, and the placement step is performed by a second device, which performs a convolution operation on the image data and weights.

[0144] Item A21. According to the device described in Item A20, the judgment module is also used to determine whether the image data needs to be padded in dimensions, and the operation also includes: determining whether the graph optimization mechanism uses an offline model; and if so, the insertion module inserts the padded node, and the insertion step is performed by the first device.

[0145] Item A22. According to the apparatus described in Item A21, the padding node pads the channel dimension of the image data.

[0146] Clause A23. According to the device described in clauses A18-A22, the judgment module is also used to judge whether the data type of the image data is consistent with the data type supported by the graph optimization mechanism, and the device also includes an adding module. When inconsistent, the adding module is used to add a conversion node before the optimized convolution node to convert the data type of the image data into the data type supported by the graph optimization mechanism.

[0147] Item A24. The apparatus according to Item A23, wherein the environment variable is further used to reflect the device that performs the adding step.

[0148] Item A25. The apparatus according to Item A24, wherein the placing step is performed by the same device as that performing the adding step.

[0149] The embodiments of the present invention are described in detail above. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core ideas. At the same time, for those skilled in the art, according to the ideas of the present invention, there may be changes in the specific implementation methods and application scopes. In summary, the contents of this specification should not be understood as limiting the present invention.

Claims

1. A method for convolution of image data based on a graph optimization mechanism, characterized in that: The method comprises: Acquire the image data; Replacing a regular convolution node with an optimized convolution node in the graph optimization mechanism; wherein the performance of the optimized convolution node is better than the performance of the regular convolution node, where the performance refers to the length of time it takes to execute the node; and the dimension of the optimized convolution is smaller than the dimension of the regular convolution; Based on the graph optimization mechanism, determining whether the image data needs to be padded according to overall performance or by setting an environment variable; wherein the value of the environment variable reflects the convolution padding strategy; overall performance refers to the time consumed to complete the convolution operation; and If so, a padding node is placed before the optimized convolution node; the padding node pads the channel dimension of the image data.

2. The method according to claim 1, characterized in that The method further comprises: Calculating and executing a first time period in which the optimized convolution node is not completed; Calculating and executing a second time period of the conventional convolution node; Determining whether the first time period is less than the second time period; If so, the replacement step is performed.

3. The method according to claim 2, characterized in that The judging step further comprises: Calculating a third time period required to complete the optimized convolution node; Calculating a fourth time period required to execute the completed node; and Determining whether the first time period is less than the sum of the third time period and the fourth time period; The method further includes: if the first time period is less than the sum of the third time period and the fourth time period, stopping inserting the filling node.

4. The method according to claim 1, wherein The method further includes: when the value of the environment variable is a first value, stopping placing the completion node.

5. The method according to claim 4, characterized in that When the environment variable value is the second value, the placing step is performed by the first device, and the first device is used to obtain the image data and the weight.

6. The method according to claim 4, characterized in that When the environment variable value is a third value, the placing step is performed by a second device, and the second device performs a convolution operation on the image data and the weight.

7. The method according to claim 5, characterized in that The step of determining whether the image data needs to be padded in dimension further includes: Determining whether the graph optimization mechanism uses an offline model; and If so, the placing step is performed by the first device.

8. The method according to any one of claims 4 to 7, characterized in that: The method further comprises: Determine whether the data type of the image data is consistent with the data type supported by the graph optimization mechanism; When there is inconsistency, a conversion node is added before the optimized convolution node to convert the data type of the image data into a data type supported by the graph optimization mechanism.

9. The method according to claim 8, characterized in that The environment variable is also used to reflect the device performing the adding step.

10. The method according to claim 9, characterized in that The placing step is performed using the same device as that used to perform the adding step.

11. A computer-readable storage medium having stored thereon computer program instructions for a graph optimization mechanism, characterized in that: When the computer program instructions are executed by the server, the method according to any one of claims 1 to 10 is implemented.

12. An integrated circuit device, characterized in that: include: processing device; DRAM for storing executable instructions; The processing device is configured to call the instructions stored in the DRAM to execute the method according to any one of claims 1 to 10.

13. A board comprising the integrated circuit device according to claim 12.

Images