Operator insertion method and apparatus, electronic device, chip and medium

Abstract

The application provides an operator insertion method and device, electronic equipment, a chip and a medium, and relates to the field of artificial intelligence acceleration. The method comprises the following steps: acquiring a second insertion data set of a second operator connected with a first operator in a model calculation graph, and determining a first insertion data set of the first operator based on the second insertion data set and the data format of the required input data of the first operator; in the case that the first operator is located at the end operator position in the model calculation graph, determining the minimum conversion operator insertion amount of the first operator based on the first insertion data set, so as to insert a conversion operator according to the minimum conversion operator insertion amount. When the conversion operator is inserted to perform data format conversion, the insertion amount of the conversion operator is effectively reduced, the memory access overhead is reduced, and the overall calculation efficiency of the model is improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence acceleration, and more particularly to an operator insertion method, apparatus, electronic device, chip, and medium. Background Technology

[0002] In recent years, with the widespread application of deep learning technology in fields such as computer vision and natural language processing, the demands for computational efficiency in deep learning technology have been increasing. The Neural Processing Unit (NPU), as hardware specifically designed to accelerate deep learning model inference, has become a core computing unit due to its highly efficient parallel computing capabilities. However, to maximize computational efficiency, the internal architecture of NPU hardware typically imposes strict and specific requirements on the data formats of the input and output data of its operators.

[0003] Currently, in order to adapt to NPU, related technologies constrain the input and output data formats of some key operators (such as convolution operators and general matrix multiplication operators). They insert layout transformation operators before and after the operators that require format conversion to convert the data format to the format required by the NPU hardware. Finally, they use graph compilation to eliminate redundant layout transformation operators.

[0004] However, for models with complex topologies (such as those containing multiple branches, multiple input operators, or multiple data formats), graph compilation cannot accurately identify the format conversion requirements of different data paths and has difficulty coordinating the conversion relationships between various data paths. This results in redundant conversion operators not being effectively eliminated, leading to excessive insertion of layout conversion operators, wasting computational resources, increasing memory access latency, and ultimately affecting the inference speed of the entire model. Summary of the Invention

[0005] This application provides an operator insertion method, apparatus, electronic device, chip, and medium to solve problems in related technologies. By utilizing the dependencies between operators in the model computation graph, the minimum number of transformation operators required for the operators in the model computation graph is determined. Transformation operators are inserted according to the minimum number of transformation operators, which can effectively avoid excessive insertion of transformation operators, reduce waste of computing resources, reduce memory access overhead, and thus improve the inference speed of the entire model.

[0006] A first aspect of this application proposes an operator insertion method, comprising: obtaining a second insertion data set of a second operator connected to a first operator in a model computation graph, the second insertion data set including the conversion operator insertion amount when the output data of the second operator is in different data formats, the first operator being any operator in the model computation graph; determining a first insertion data set of the first operator based on the second insertion data set and the data format of the input data required by the first operator, the first insertion data set including the conversion operator insertion amount when the output data of the first operator is in different data formats; and, when the first operator is at the end operator position in the model computation graph, determining the minimum conversion operator insertion amount of the first operator based on the first insertion data set, so as to insert a conversion operator according to the minimum conversion operator insertion amount.

[0007] In some embodiments of this application, before obtaining the second insertion data set of the second operator connected to the first operator in the model computation graph, the method includes: determining the operator position of the second operator in the model computation graph; if the second operator is at the first operator position in the model computation graph, determining the data format of the input data of the model computation graph; and determining the second insertion data set of the second operator based on the data format of the input data of the model computation graph and the data format of the input data required by the second operator.

[0008] In some embodiments of this application, obtaining a second set of inserted data for a second operator connected to a first operator in a model computation graph includes: determining the operator position of the first operator in the model computation graph; and obtaining a second set of inserted data when the first operator is not at the first operator position in the model computation graph.

[0009] In some embodiments of this application, determining the first insertion data set of the first operator based on the second insertion data set and the data format of the input data required by the first operator includes: obtaining the insertion amount of candidate transformation operators in the second insertion data set that matches the data format of the input data required by the first operator; determining the smallest candidate transformation operator insertion amount among the candidate transformation operator insertion amounts as the target insertion amount; and determining the first insertion data set based on the target insertion amount.

[0010] In some embodiments of this application, when the first operator is at the end of the model computation graph, determining the minimum transformation operator insertion amount of the first operator based on the first insertion data set includes: when the first operator is at the end of the model computation graph, determining the data format of the output data of the model computation graph; when the data format of the output data of the model computation graph is inconsistent with the data format of the output data of the first operator, adjusting the transformation operator insertion amount of the first operator to obtain an adjusted updated insertion data set, wherein the data format of the output data of the first operator in the updated insertion data set is the same as the data format of the output data of the model computation graph; and determining the minimum transformation operator insertion amount of the first operator based on the updated insertion data set.

[0011] In some embodiments of this application, inserting a conversion operator based on the minimum conversion operator insertion amount includes: determining the conversion operator insertion position corresponding to the minimum conversion operator insertion amount; and inserting the conversion operator based on the conversion operator insertion position.

[0012] A second aspect of this application provides an operator insertion apparatus, the apparatus comprising:

[0013] The acquisition unit is used to acquire the second insertion data set of the second operator connected to the first operator in the model computation graph. The second insertion data set includes the number of conversion operators inserted when the output data of the second operator is in different data formats. The first operator is any operator in the model computation graph. The determining unit is used to determine the first insertion data set of the first operator based on the second insertion data set and the data format of the input data required by the first operator. The first insertion data set includes the conversion operator insertion amount when the output data of the first operator is in different data formats. An insertion unit is used to determine the minimum transformation operator insertion amount of the first operator based on a first insertion data set when the first operator is at the end operator position in the model computation graph, so as to insert the transformation operator according to the minimum transformation operator insertion amount.

[0014] A third aspect of this application provides an electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the methods described in the first aspect of this application.

[0015] A fourth aspect of this application provides a chip including one or more interface circuits and one or more processors; the interface circuits are configured to receive signals from the memory of an electronic device and send signals to the processors, the signals including computer instructions stored in the memory, which, when executed by the processors, cause the electronic device to perform the methods described in the first aspect of this application.

[0016] A fifth aspect of this application provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause a computer to perform the methods described in the first aspect of this application.

[0017] In summary, the operator insertion method proposed in this application obtains a second insertion data set for the second operator connected to the first operator in the model computation graph. This second insertion data set includes the insertion amount of the conversion operator when the output data of the second operator has different data formats. The first operator is any operator in the model computation graph. Based on the second insertion data set and the data format of the input data required by the first operator, a first insertion data set for the first operator is determined. This first insertion data set includes the insertion amount of the conversion operator when the output data of the first operator has different data formats. When the first operator is located at the end of the model computation graph, the minimum insertion amount of the conversion operator for the first operator is determined based on the first insertion data set, and the conversion operator is inserted according to this minimum insertion amount. This method effectively reduces the insertion amount of the conversion operator, lowers memory access overhead, and improves the overall computational efficiency of the model when data format conversion requires insertion of a conversion operator.

[0018] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0019] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application, and do not constitute an undue limitation of this application.

[0020] Figure 1 A schematic diagram of a simple model insertion transformation operator provided in an embodiment of this application; Figure 2 A schematic diagram of a complex model insertion transformation operator provided in an embodiment of this application; Figure 3 A flowchart illustrating an operator insertion method provided in an embodiment of this application; Figure 4 A flowchart illustrating the second operator insertion method provided in the embodiments of this application; Figure 5A flowchart illustrating the third operator insertion method provided in this application embodiment; Figure 6 A flowchart illustrating the fourth operator insertion method provided in this application embodiment; Figure 7 A flowchart illustrating the fifth operator insertion method provided in this application embodiment; Figure 8 A schematic diagram of a model calculation graph provided in an embodiment of this application; Figure 9 A flowchart illustrating an operator insertion scheme provided in an embodiment of this application; Figure 10 This is a schematic diagram of the structure of an operator insertion device provided in an embodiment of this application; Figure 11 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application; Figure 12 This is a schematic diagram of the structure of a chip provided in an embodiment of this application. Detailed Implementation

[0021] The embodiments of this application are described in detail below. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.

[0022] In recent years, with the widespread application of deep learning technology in fields such as computer vision and natural language processing, the demand for computational efficiency has been increasing. As hardware specifically designed to accelerate deep learning model inference, the NPU has become a key computing unit due to its highly efficient parallel computing capabilities. However, to maximize computational efficiency, the internal architecture of NPU hardware typically imposes strict and specific requirements on the data formats of input and output data.

[0023] To ensure that operators can execute correctly and efficiently on the NPU, related techniques typically employ a strategy of local insertion and late elimination, such as... Figure 1 The diagram illustrates a simplified model with inserted transformation operators. Specifically, in the model's computation graph, when an operator requiring a special data format (such as a convolution operator or a general matrix multiplication operator) is encountered, a layout transformation operator is inserted before the operator to convert its input from a general flat format to the hardware-required special format. Correspondingly, after the operator, another layout transformation operator is inserted to convert its output from the special format back to the flat format, ensuring the normal input of subsequent operators. Finally, relying on the optimization capabilities of the graph compiler, adjacent and inverse layout transformation operators in the computation graph are eliminated to reduce redundant transformation operations in the final computation graph.

[0024] However, the relevant technologies have obvious drawbacks, such as... Figure 2 The diagram illustrates a complex model insertion transformation operator. The local insertion and late elimination strategies of related technologies are only effective for network models with simple topologies. When faced with complex model structures, or scenarios with multiple input operators (each input requires a different memory layout) or more special memory layout requirements, graph compilers cannot eliminate redundant layout transformation operators. This not only wastes computing resources (layout transformation, as an additional computational operation, consumes computing units and memory bandwidth, occupying resources that could be used for actual inference computation), but also increases the number and amount of data transmissions in memory, leading to increased memory access latency and thus affecting the inference speed of the model.

[0025] To address the aforementioned issues, this application proposes an operator insertion method. This method involves obtaining a second insertion data set of operators connected to the first operator in the model computation graph. Based on the second insertion data set and the data format of the input data required by the first operator, a first insertion data set for the first operator is determined. When the first operator is located at the end of the model computation graph, the minimum number of transformation operators to be inserted is determined based on the first insertion data set. The transformation operator is then inserted according to this minimum insertion amount. This method effectively reduces the number of transformation operators to be inserted when data format conversion requires insertion, thereby lowering memory access overhead and improving the overall computational efficiency of the model. It is applicable to various AI inference scenarios such as AIoT, autonomous driving, and intelligent security.

[0026] The operator insertion method provided in this application will be described in detail below with reference to the accompanying drawings.

[0027] Figure 3 This is a flowchart illustrating an operator insertion method provided in an embodiment of this application. Figure 3 As shown, the operator insertion method includes steps 101-103.

[0028] Step 101: Obtain the second insertion data set of the second operator connected to the first operator in the model computation graph. The second insertion data set includes the conversion operator insertion amount when the output data of the second operator is in different data formats. The first operator is any operator in the model computation graph.

[0029] In this embodiment, the second operator precedes and is connected to the first operator; that is, the output of the second operator serves as the input of the first operator. There can be one or more second operators. The number of conversion operators inserted can be the number of conversion operators inserted, the number of memory accesses (bytes) for the inserted conversion operators, or the number of computation cycles for the conversion operators. The required data format for an operator refers to the layout of its input and output data in memory, which must conform to the specific constraints of the NPU hardware. The data format can include multiple formats such as general formats, special formats, or other formats.

[0030] Before obtaining the minimum number of transformation operators required for the entire computation graph, this application needs to traverse all operators in a one-way direction from input to output according to the dependencies between operators in the computation graph, so as to obtain the relevant data for the insertion of transformation operators for each operator under different combinations of input and output data formats.

[0031] Specifically, during the traversal process, it is necessary to consider the hardware constraints of each operator (i.e., the specific requirements of the NPU hardware for the data formats of input and output data during computation for each operator), and comprehensively refer to the various data formats that the input data (output data of the second operator) of each operator (the first operator) may present (i.e., different memory layout types). The system analyzes the compatibility of each input data format with the input data format required by the hardware constraints of the first operator to determine whether a conversion operator needs to be inserted on the input side of the first operator. Simultaneously, considering the potential conversion requirements for subsequent connected operators when the output data of the first operator uses different data formats, the system collects relevant data on the insertion of conversion operators for each operator (such as insertion position and insertion amount).

[0032] It is worth noting that the different operator layout constraints (i.e., the data formats of the input and output data of the operators required by the NPU hardware) and the conversion operator coefficients in this application embodiment are stored in the hardware constraint library. The conversion operator coefficients refer to the memory access volume (number of bytes) or computation cycles corresponding to different conversion operators. For example, the memory access volume required to convert from a normal format to a special format is 2.5MB, and the memory access volume required to convert from a special format to a normal format is 1.8MB. 2.5MB and 1.8MB are the conversion operator coefficients corresponding to different conversion operators.

[0033] Step 102: Based on the second insertion data set and the data format of the input data required by the first operator, determine the first insertion data set of the first operator. The first insertion data set includes the conversion operator insertion amount when the output data of the first operator is in different data formats.

[0034] In the embodiments of this application, the data format of the input data required by the first operator refers to the specific requirements of the NPU hardware for the memory layout of the input data during the computation of the first operator. For example, the NPU hardware may require the input data of the first operator to be in a general format, a special format, or other formats, etc., which is predetermined by the underlying computing kernel or hardware architecture of the NPU (such as a systolic array, dedicated computing unit, etc.). Only when the data format of the input data of the first operator strictly matches the hardware requirements can the NPU efficiently execute the computation of the first operator; otherwise, it may lead to a significant decrease in computing performance or computational errors.

[0035] The first operator may be connected to one or more operators (i.e., the second operator). The output of the second operator may support multiple data formats, and the insertion amount of the conversion operator corresponding to each data format is recorded in the second inserted data set. To determine the first inserted data set of the first operator, it is necessary to comprehensively consider the possible combinations of data formats of the output of the second operator.

[0036] Specifically, for each candidate output data format of the first operator, all possible combinations of input data (i.e., output data of the second operator) are enumerated. For each combination, it is first determined whether it meets the hardware input requirements of the first operator. For input data that does not meet the requirements, a conversion operator is inserted to obtain a candidate total insertion quantity. Finally, the minimum value among the candidate total insertion quantities corresponding to all input combinations is taken as the cumulative conversion operator insertion quantity of the first operator on the input side under that candidate output data format. At the same time, according to the hardware output constraints of the first operator, it is determined whether the data format to be output needs to be converted. If so, the cumulative conversion operator insertion quantity on the input side is added to the conversion operator insertion quantity on the output side to obtain the total conversion operator insertion quantity of the first operator under that output data format, and it is recorded in the first inserted data set.

[0037] Step 103: When the first operator is at the end of the model computation graph, determine the minimum transformation operator insertion amount of the first operator based on the first insertion data set, so as to insert the transformation operator according to the minimum transformation operator insertion amount.

[0038] In the embodiments of this application, all operators are traversed unidirectionally from input to output according to the dependencies between operators in the computation graph and the topological sorting order. When the last operator is reached, it means that the insertion data sets of all preceding operators have been calculated. At this point, a final decision needs to be made on the first insertion data set of the last operator, taking into account the data format required by the output data of the computation graph, in order to determine the globally optimal layout transformation path.

[0039] Specifically, first, the data format that the output data of the computation graph must follow (i.e., global output layout constraints) is determined. Then, the first set of inserted data for the final operator is evaluated based on the global output layout constraints: if the format of a certain output data of the final operator is consistent with the global output layout constraints, the insertion amount of the transformation operator corresponding to that format is directly used as a candidate value. If the format of a certain output data of the final operator is inconsistent with the global output layout constraints, an additional transformation operator is added to the insertion amount of the transformation operator corresponding to that format to transform the output data to the global output layout constraints, thus obtaining an adjusted candidate value. Finally, all candidate values ​​are compared, and the minimum value is determined as the global minimum transformation operator insertion amount. This minimum value identifies the globally optimal layout transformation path in the computation graph; inserting transformation operators based on this path minimizes the insertion amount of the transformation operators.

[0040] In summary, according to the operator insertion method proposed in this application, a second insertion data set of the second operator connected to the first operator is obtained in the model computation graph. This second insertion data set includes the insertion amount of the conversion operator when the output data of the second operator has different data formats. The first operator is any operator in the model computation graph. Based on the second insertion data set and the data format of the input data required by the first operator, a first insertion data set of the first operator is determined. This first insertion data set includes the insertion amount of the conversion operator when the output data of the first operator has different data formats. When the first operator is at the end of the model computation graph, the minimum insertion amount of the conversion operator for the first operator is determined based on the first insertion data set, and the conversion operator is inserted according to the minimum insertion amount. This method effectively reduces the insertion amount of the conversion operator, lowers memory access overhead, and improves the overall computational efficiency of the model when data format conversion requires insertion of a conversion operator.

[0041] As one possible implementation method, Figure 4 A flowchart of the second operator insertion method provided in the embodiments of this application is shown. Figure 4 based on Figure 3 The illustrated embodiment further defines step 101. Figure 4 In the illustrated embodiment, steps 201, 202, and 203 precede step 101. For example... Figure 4 As shown, the method includes the following steps: Step 201: Determine the position of the second operator in the model computation graph.

[0042] In the embodiments of this application, the insertion amount of the transformation operator of the first operator depends on the insertion amount of the transformation operator of the second operator, and the insertion amount of the transformation operator of the second operator needs to be further analyzed and determined based on its position in the computation graph, specifically in two cases: If the second operator is also an intermediate operator in the computation graph (i.e., neither the first nor the last operator), then the method for determining the number of transformation operators inserted by the second operator is exactly the same as that of the first operator. That is, the second operator is treated as a new first operator, and the calculation is performed recursively based on the set of inserted data of its predecessor operators.

[0043] If the second operator is the first operator in the computation graph, then its transformation operator insertion amount does not depend on any predecessor operator, but is determined based on the data format of the input data of the computation graph (i.e., global input layout constraints) and the hardware constraints of the second operator itself.

[0044] Step 202: If the second operator is located at the first operator position in the model computation graph, determine the data format of the input data of the model computation graph.

[0045] In the embodiments of this application, when the second operator is identified as the first operator in the computation graph, its transformation operator insertion amount does not depend on the computation results of other operators, but is determined by the data format of the input data of the model computation graph (i.e., global input layout constraints) and the hardware constraints of the second operator itself.

[0046] Among them, the global input layout constraint specifies the memory layout format (e.g., normal format or special format) of the input data (or input tensor) when it enters the computation graph. This constraint may be determined by hardware interface specifications, upstream processing flow or user configuration.

[0047] Step 203: Based on the data format of the input data of the model computation graph and the data format of the input data required by the second operator, determine the second insertion data set of the second operator.

[0048] In the embodiments of this application, the data format of the input data required by the second operator is the hard requirement of the NPU hardware for the input memory layout of the second operator (e.g., the hardware requires the input data format of the second operator to be in a special format). By comparing the input data format of the model computation graph with the input format requirement of the second operator, the initial transformation operator insertion amount of the second operator (i.e., the transformation operator insertion amount on the input side of the second operator) can be determined, specifically in the following two cases: If the input data format of the computation graph meets the input format requirements of the second operator, then there is no need to insert a transformation operator on the input side of the second operator. In this case, the initial number of transformation operators inserted by the second operator is 0. If the input data format of the computation graph does not meet the hardware input format requirements of the second operator, a conversion operator needs to be inserted on the input side of the second operator. In this case, the initial conversion operator insertion amount of the second operator is 1.

[0049] It should be noted that, for ease of understanding, the conversion operator insertion quantity in this embodiment refers to the number of conversion operators inserted. In practical applications, the conversion operator insertion quantity can also be a more precise measure such as the number of data accesses or the number of calculation cycles.

[0050] After obtaining the initial number of conversion operator insertions for the second operator, a complete set of second insertion data needs to be constructed based on the hardware output format requirements of the second operator. Specifically, for all possible candidate output data formats of the second operator, the cumulative number of conversion operator insertions is analyzed. That is, if the candidate output data format is consistent with the hardware output format requirements of the second operator, the total number of conversion operator insertions for the second operator is the initial number of conversion operator insertions (including only the number of conversion operator insertions on the input side); if the candidate output data format is inconsistent with the hardware output format requirements of the second operator, the total number of insertions is the initial number of conversion operator insertions plus the number of conversion operators inserted on the output side.

[0051] Ultimately, the second insertion data set contains the total number of transformation operator insertions corresponding to all candidate output data formats of the second operator.

[0052] In summary, this application first clarifies the initial conditions through dynamic programming, and then systematically constructs the second set of insertion data for the first operator in the computation graph, providing basic data support for the calculation of the insertion amount of the transformation operator for other operators in the future.

[0053] As one possible implementation method, Figure 5 A flowchart of the third operator insertion method provided in the embodiments of this application is shown. Figure 5 based on Figure 3 The illustrated embodiment further defines step 101. Figure 5 In the illustrated embodiment, step 101 includes steps 301 and 302. For example... Figure 5 As shown, the method includes the following steps: Step 301: Determine the position of the first operator in the model computation graph.

[0054] In the embodiments of this application, this step is a preliminary decision condition for performing the calculation of the insertion amount of the transformation operator. Its purpose is to determine whether the first operator to be processed depends on the predecessor operator, thereby determining the calculation path of the first insertion data set of the first operator.

[0055] If the first operator is in the first operator position, the first inserted data of the first operator is directly determined by the global input layout constraints of the model computation graph and the hardware layout constraints of the first operator itself. The first inserted data set of the first operator can be determined through steps 201 to 203.

[0056] If the first operator is not in the first operator position (i.e., the first operator is an intermediate operator or the last operator), then its inserted data set must be solved based on the calculation result of its predecessor operator (the second operator).

[0057] Step 302: If the first operator is not located at the first operator position in the model computation graph, obtain the second insertion data set.

[0058] In embodiments of this application, when the first operator is identified as an intermediate or final operator, it means that the first operator has one or more predecessor operators (i.e., second operators). In this case, it is necessary to combine the second insertion data set, which has already been computed by the second operators, to calculate the cumulative transformation operator insertion amount of the first operator. The second insertion data set contains the minimum cumulative transformation operator insertion amount satisfying all constraints from the start of the computation graph to the output of each second operator.

[0059] Understandably, the second set of inserted data is a necessary input for the first operator to compute its own global minimum transformation operator insertion amount.

[0060] In summary, this application utilizes the dependencies between operators to obtain the complete insertion data set of the predecessor operator (second operator) of the first operator, ensuring that global layout optimization can be systematically carried out along the topological order of the computation graph, laying a data foundation for subsequent propagation calculations based on the minimum number of transformation operator insertions.

[0061] As one possible implementation method, Figure 6 A flowchart of the fourth operator insertion method provided in the embodiments of this application is shown. Figure 6 based on Figure 3 The illustrated embodiment further defines step 102. Figure 6 In the illustrated embodiment, step 102 includes steps 401, 402, and 403. For example... Figure 6 As shown, the method includes the following steps: Step 401: Obtain the number of candidate transformation operators that match the data format of the input data required by the first operator in the second insertion data set.

[0062] In embodiments of this application, the second inserted data set includes the conversion operator insertion amount when the output data of the second operator is in different data formats. The second operator includes one or more predecessor operators connected to the first operator. It is understood that if the first operator contains multiple input data, due to hardware limitations, the first operator may have different requirements for the data format of different input data (for example, the first operator requires the first input to be in a normal format and the second input to be in a special format).

[0063] Specifically, the second set of inserted data contains the number of conversion operators inserted for each predecessor operator under multiple output data formats. First, for each predecessor operator, if its output data format is inconsistent with the data format of the input data required by the first operator, a conversion operator needs to be added; if they are consistent, no conversion operator needs to be added. This ensures that the output data format of each predecessor operator matches the data format of the input data required by the first operator.

[0064] Then, all possible combinations of the output data formats of the predecessor operators are enumerated to obtain multiple candidate conversion operator insertion quantities. For example, if the output data of predecessor operator 1 contains both ordinary and special data formats, and the output data of predecessor operator 2 also contains both ordinary and special data formats, then there are four combinations: predecessor operator 1 outputs ordinary format and predecessor operator 2 outputs ordinary format, predecessor operator 1 outputs ordinary format and predecessor operator 2 outputs special format, predecessor operator 1 outputs special format and predecessor operator 2 outputs ordinary format, and predecessor operator 1 outputs special format and predecessor operator 2 outputs special format. The conversion operator insertion quantity corresponding to each combination is taken as the candidate conversion operator insertion quantity.

[0065] The insertion amount of each candidate transformation operator consists of two parts: the insertion amount of the transformation operator of the predecessor operator and the cumulative insertion amount of the transformation operator when the output data format of the predecessor operator does not meet the data format of the input data required by the first operator.

[0066] Step 402: Determine the smallest candidate transformation operator insertion quantity among the candidate transformation operator insertion quantities as the target insertion quantity.

[0067] In the embodiments of this application, the target insertion amount is the conversion operator insertion amount accumulated on the input side of the first operator.

[0068] The insertion amounts of multiple candidate transformation operators are compared, and the minimum value of the insertion amounts of multiple candidate transformation operators is taken as the minimum number of transformation operator insertions required on the input side of the first operator.

[0069] Step 403: Determine the first set of data to be inserted based on the target insertion amount.

[0070] In embodiments of this application, the first inserted data set includes the number of conversion operators inserted when the output data of the first operator is in different data formats.

[0071] Specifically, after obtaining the input-side conversion operator insertion amount (i.e., the target insertion amount), based on the specific hardware layout constraints of the first operator (i.e., the NPU hardware requires the first operator's output data to be in a specific data format), when outputting different data formats to the first operator, the required conversion operator insertion amount is accumulated based on the first operator's target insertion amount. That is, the conversion operator insertion amount corresponding to each output data format of the first operator is obtained by accumulating the input-side and output-side conversion operator insertion amounts. It can be understood that the required conversion operator insertion amount for the output may be 0 or 1, depending on whether the actual output data format of the first operator matches the output data format required by its own hardware.

[0072] In summary, this application utilizes the dependencies between operators and considers all possible combinations of input layouts to calculate the minimum cumulative transformation operator insertion amount for each operator. This systematically allocates the minimum number of transformation operators to each operator, effectively avoiding wasted computational resources, reducing system access overhead, and improving the overall inference speed of the model.

[0073] As one possible implementation method, Figure 7 A flowchart of the fifth operator insertion method provided in the embodiments of this application is shown. Figure 7 based on Figure 3 The illustrated embodiment further defines step 103. Figure 7 In the illustrated embodiment, step 103 includes steps 501, 502, 503, 504, and 505. For example... Figure 7 As shown, the method includes the following steps: Step 501: If the first operator is located at the end operator position in the model computation graph, determine the data format of the output data of the model computation graph.

[0074] In the embodiments of this application, the data format of the output data of the model computation graph, namely the global output layout constraint, is the memory layout requirement that the final output result of the model must meet. This requirement may be determined by the hardware interface specification, downstream processing flow or user configuration, and is the core basis for determining whether the output of the first operator needs additional conversion.

[0075] Specifically, after completing the forward traversal of the entire computation graph, the number of transformation operator insertions for the final operator (first operator) under various output data formats can be obtained. To obtain the minimum number of transformation operator insertions required for the entire computation graph, the data format requirements of the final output data of the model computation graph need to be further considered.

[0076] Step 502: If the data format of the output data of the model calculation graph is inconsistent with the data format of the output data of the first operator, adjust the insertion amount of the transformation operator of the first operator to obtain the adjusted updated insertion data set.

[0077] In the embodiments of this application, if the output data format of the first operator is inconsistent with the global output data format requirements of the model computation graph, a conversion operator needs to be inserted on the output side of the first operator to convert its output data format into the globally required output data format.

[0078] Specifically, since the first inserted data set contains the output data of the first operator, which are the insertion amounts of conversion operators corresponding to different data formats, for the first operator's output data format that does not meet the global output data format requirements, a conversion operator needs to be added to the original insertion amount of the conversion operator corresponding to that data format. However, for the first operator's output data format that meets the global output requirements, its corresponding insertion amount remains unchanged. After the above adjustments, the updated inserted data set is obtained.

[0079] It is understandable that the various data formats of the output data of the first operator in the updated insertion dataset already meet the data format requirements of the output data of the model computation graph.

[0080] Step 503: Determine the minimum number of transformation operators to insert for the first operator based on the updated insertion data set.

[0081] In the embodiments of this application, updating the insertion data set ensures that all output data formats of the first operator conform to the global output data format requirements. Therefore, it is only necessary to select the value with the smallest conversion operator insertion amount from the set to obtain the minimum conversion operator insertion amount of the first operator. Since the conversion operator insertion amount of the first operator is obtained by summing the minimum conversion operator insertion amount of the second operator, the minimum conversion operator insertion amount of the first operator is the minimum conversion operator insertion amount required for the entire computation graph.

[0082] Step 504: Determine the insertion position of the transformation operator corresponding to the minimum transformation operator insertion amount.

[0083] In the embodiments of this application, this step is a backtracking process, that is, after determining the minimum number of transformation operators inserted into the computation graph, it is necessary to trace back to find out which specific layout decision paths jointly led to this optimal result.

[0084] Specifically, starting from the last operator, the computation graph is traversed backwards based on the input layout combination that results in the minimum insertion amount. On the backtracking path, whenever an operator's actual input layout is found to be not directly derived from the output layout of its predecessor operator, but rather after a transformation, the input position of that operator (or the output position of the predecessor operator) is the insertion position of a transformation operator. This process is repeated sequentially to determine the insertion position of each transformation operator until the first operator node of the computation graph is reached.

[0085] Step 505: Insert the transformation operator according to the insertion position of the transformation operator.

[0086] In the embodiments of this application, according to the insertion position of each transformation operator marked by the reverse traversal of the computation graph in step 504, the corresponding transformation operator is inserted into the original computation graph.

[0087] The resulting computational graph after insertion ensures that the input and output data formats of all operators meet their hardware layout constraints, and the total overhead of data format conversion for the entire model is minimized.

[0088] To facilitate better understanding, this application provides a specific example, such as Figure 8 As shown, Figure 8 This diagram illustrates a model calculation graph provided in an embodiment of this application. The number of transformation operator insertions in this application is taken as an example.

[0089] Reference Figure 8 The model computation graph contains three operators, and the dependencies between the operators are as follows: Figure 8 As shown, assume that the entire computation graph has two data formats for input and output data: a normal format and a special format. Due to hardware limitations, the input and output data of the convolution operators (convolution 0 and convolution 1) only support the special format; one input of the special addition operator (from the output of convolution 1) must be in the special format, and the other input (from the output of convolution 0) must be in the normal format. The output data of the special addition operator is in the special format. The input and output data of the computation graph are in the normal format.

[0090] An array table of [output_num][layout_count] is created for each operator node to represent the cost (corresponding to the number of transformation operator insertions in this application). "output_num" represents the number of output data (output tensors) of the operator node (usually 1). This dimension is used as the index of the output tensor. For example, [0] represents the first output tensor and [1] represents the second output tensor. "layout_count" represents the number of data formats (i.e., the number of memory layout types supported by the system). For example, "node0[0][normal format]" represents the minimum cost when the first output data of node 0 is in normal format.

[0091] First, consider the convolution 0 node (corresponding to the case where the second operator is the first operator in this application). Since the input and output data of convolution 0 only support special formats, and the input data format of the computation graph is a normal format, if convolution 0 needs to output a normal format, then two transformation operators need to be inserted (i.e., one at each of the input and output positions of convolution 0). Therefore, the cost corresponding to the output data of convolution 0 being in a normal format is: node0[0][normal format] = 2 If convolution 0 needs to output a special format, then only one transformation operator needs to be inserted at the input position of convolution 0. Therefore, the cost corresponding to the output data of convolution 0 being in normal format is: node0[0][Special Format] = 1 Next, following the topological sorting order, we continue to examine the convolution 1 node. The input of convolution 1 is the output of convolution 0, and the input data of convolution 1 only supports a special format. Based on the insertion data set of convolution 0, if the output of convolution 0 is in a normal format, a conversion operator needs to be inserted at the input position of convolution 1 to convert the normal format to a special format; if the output of convolution 0 is in a special format, no insertion is needed. For these two cases, the minimum value is taken to obtain the initial conversion operator insertion amount of convolution 1, i.e., min(node0[0][normal format]+1, node0[0][special format]). Since the output data of convolution 1 only supports a special format, if convolution 1 needs to output in a normal format, a conversion operator needs to be inserted at its output position; if convolution 1 needs to output in a special format, no insertion is needed. Therefore, the insertion data set corresponding to convolution 1 is: node1[0][normal format] = min(node0[0][normal format] + 1, node0[0][special format]) + 1 node1[0][Special format] = min(node0[0][Normal format] + 1, node0[0][Special format]) Here, "+1" indicates the insertion of a transformation operator.

[0092] Finally, we examine the special addition node (i.e., the terminal operator). Due to hardware limitations, one input (from the output of convolution 1) of the special addition operator must be in a special format, while the other input (from the output of convolution 0) must be in a normal format. Furthermore, the output data of the special addition operator must also be in a special format. Based on this requirement, since each of the two inputs to the special addition node has two possible data formats, we need to consider four cases for the output layout: If convolution 1 outputs in a normal format, one conversion operator needs to be inserted; if it outputs in a special format, no operator is needed. If convolution 0 outputs in a special format, one conversion operator needs to be inserted; if it outputs in a normal format, no operator is needed. Combining these four cases and taking the minimum value yields the initial number of conversion operators to be inserted for the special addition node. Then, based on the output data format requirements of the special addition node, if the node needs to output in a normal format, one conversion operator needs to be inserted; otherwise, no operator is needed. The final set of inserted data for the special addition node is as follows: node2[0][normal format] = min(node1[0][normal format] + 1 + node0[0][normal format], node1[0][normal format] + 1 + node0[0][special format] + 1, node1[0][special format] + node0[0][normal format], node1[0][special format] + node0[0][special format] + 1) + 1 node2[0][Special format] = min(node1[0][Normal format] + 1 + node0[0][Normal format], node1[0][Normal format] + 1 + node0[0][Special format] + 1, node1[0][Special format] + node0[0][Normal format], node1[0][Special format] + node0[0][Special format] + 1) Since the output data of the computation graph is required to be in a normal format, if the special addition output is in a special format, a transformation operator needs to be inserted at its output position; if the output is in a normal format, no insertion is required. The minimum value of these two cases is taken to obtain the final output cost (corresponding to the minimum number of transformation operators inserted by the first operator in this application), i.e., min(node2[0][special format]+ 1, node2[0][normal format]).

[0093] After the final solution is obtained, the goal of inserting the fewest transformation operators can be achieved while meeting the hardware computing requirements. It can also adapt to various complex operator layout requirements and multiple memory layouts. Moreover, the computation time only increases linearly with the number of operators in the model, which can greatly reduce the total overhead caused by inserting all necessary transformation operators in the computation graph.

[0094] To further explain the entire process of the insertion transformation operator, please refer to [further details]. Figure 9 , Figure 9 This is a flowchart illustrating an operator insertion scheme provided in an embodiment of this application.

[0095] Reference Figure 9 First, before the model computation graph is loaded into the NPU to execute model inference, the driver creates a cost table for each operator with "[number of output tensors][supported layout types]" (corresponding to the insertion data set in this application), and fills in the initial values ​​(usually 0) according to the input and output data format requirements of each operator to generate the initial cost table.

[0096] Next, each operator is traversed unidirectionally from input to output according to the topological sorting order of the computation graph. When traversing to the current operator, the matching of each input data (from the predecessor operator connected to the current operator) with the hardware requirements of the predecessor operator is analyzed in conjunction with the input data format required by the current operator's hardware. If the format does not match, a conversion operator needs to be added to the corresponding input conversion operator insertion amount; if it matches, no addition is needed. Subsequently, the cumulative memory layout cost (corresponding to the conversion operator insertion amount of this application) for each possible memory layout combination of all inputs of the current operator is calculated, and the minimum value is taken as the cumulative target insertion amount on the input side of the operator.

[0097] Then, based on the specific requirements of the NPU hardware for the operator output memory layout, if the actual output layout of the operator is inconsistent with the requirements, the output-side conversion cost (corresponding to the output-side conversion operator insertion amount in this application) needs to be added to its cost, and finally the cumulative conversion cost of each operator is obtained.

[0098] Finally, when traversing to the last operator of the model computation graph, the output cost of the last operator is adjusted according to the format requirements of the final output data of the model computation graph: if the output data format of a certain last operator does not match the final output data format of the computation graph, the output transformation cost needs to be added to the cumulative transformation cost corresponding to the last operator; if they match, no addition is needed. After adjustment, the minimum value of the cumulative transformation cost corresponding to all output data formats of the last operator is taken to obtain the globally optimal output cost scheme. Inserting transformation operators according to the globally optimal output cost scheme can ensure that the entire computation graph satisfies all constraints while minimizing transformation costs.

[0099] In summary, this application calculates the minimum cumulative number of transformation operators required for each operator by sequentially traversing the operators based on the dependencies of each operator in the model computation graph. This achieves automatic generation of a memory layout scheme with the minimum number of global transformation operators inserted, while satisfying NPU hardware layout constraints and model output layout requirements. This solves the problem of excessive transformation operator insertion, effectively avoids wasting computational resources, reduces system memory access overhead, and improves the computational speed of model inference.

[0100] Figure 10 This is a schematic diagram of the structure of an operator insertion device 1000 provided in an embodiment of this application. Figure 10 As shown, the operator insertion device includes: The acquisition unit 1010 is used to acquire the second insertion data set of the second operator connected to the first operator in the model calculation graph. The second insertion data set includes the number of conversion operators inserted when the output data of the second operator is in different data formats. The first operator is any operator in the model calculation graph. The determining unit 1020 is used to determine the first insertion data set of the first operator based on the second insertion data set and the data format of the input data required by the first operator. The first insertion data set includes the conversion operator insertion amount when the output data of the first operator is in different data formats. The insertion unit 1030 is used to determine the minimum transformation operator insertion amount of the first operator based on the first insertion data set when the first operator is at the end operator position in the model calculation graph, so as to insert the transformation operator according to the minimum transformation operator insertion amount.

[0101] In some embodiments of this application, the acquisition unit 1010 is used to: determine the operator position of the second operator in the model computation graph; when the second operator is at the first operator position in the model computation graph, determine the data format of the input data of the model computation graph; and determine the second insertion data set of the second operator based on the data format of the input data of the model computation graph and the data format of the input data required by the second operator.

[0102] In some embodiments of this application, the acquisition unit 1010 is used to: determine the operator position of the first operator in the model computation graph; and acquire the second insertion data set when the first operator is not in the first operator position in the model computation graph.

[0103] In some embodiments of this application, the determining unit 1020 is configured to: obtain the insertion amount of candidate transformation operators that match the data format of the input data required by the first operator in the second insertion data set; determine the minimum insertion amount of the candidate transformation operators among the candidate insertion amounts as the target insertion amount; and determine the first insertion data set based on the target insertion amount.

[0104] In some embodiments of this application, the insertion unit 1030 is configured to: determine the data format of the output data of the model computation graph when the first operator is at the end operator position in the model computation graph; adjust the insertion amount of the transformation operator of the first operator to obtain an adjusted updated insertion data set, wherein the data format of the output data of the first operator in the updated insertion data set is the same as the data format of the output data of the model computation graph; and determine the minimum insertion amount of the transformation operator of the first operator based on the updated insertion data set.

[0105] In some embodiments of this application, the insertion unit 1030 is used to: determine the insertion position of the conversion operator corresponding to the minimum conversion operator insertion amount; and insert the conversion operator according to the insertion position of the conversion operator.

[0106] Since the apparatus provided in this application corresponds to the methods provided in the above-mentioned embodiments, the implementation of the methods is also applicable to the apparatus provided in this embodiment, and will not be described in detail in this embodiment.

[0107] The methods and apparatus provided in the embodiments of this application have been described above. To implement the functions of the methods provided in the embodiments of this application, the electronic device may include a hardware structure and software modules, and may implement the above functions in the form of a hardware structure, software modules, or a hardware structure plus software modules. One of the above functions may be executed in the form of a hardware structure, software modules, or a hardware structure plus software modules.

[0108] Figure 11 This is a block diagram illustrating an electronic device for implementing the above-described operator insertion method according to an exemplary embodiment. For example, the electronic device may be an Artificial Intelligence (AI) server, a training and promotion integrated machine, etc.

[0109] Reference Figure 11 The electronic device may include one or more of the following components: an AI-accelerated computing module, a central processing unit (CPU) module, a power supply module, a hard drive module, and a fan module. Each module works in conjunction with the bus system through a standardized hardware interface, as detailed in the following architecture: The AI-accelerated computing module comprises multiple AI accelerator cards deployed in parallel. Each AI accelerator card integrates at least one AI acceleration chip (such as a Resistive Processing Unit (RPU) chip, a GPU chip, or a Coarse-Grained Reconfigurable Array (CGRA) chip). The AI ​​accelerator cards communicate with each other via a high-speed card-to-card (C2C) interconnect structure, supporting low-latency, high-bandwidth horizontal scaling. The AI ​​acceleration chip is dedicated to performing AI computing tasks such as high-density matrix operations, neural network model training, and / or inference, providing the main computing power support.

[0110] The CPU module includes at least one CPU board, on which a CPU chip and matching CPU memory (such as fourth / fifth generation double data rate (DDR4 / DDR5) or random access memory (RAM) are mounted. The CPU chip, as the system control center, is responsible for task scheduling, resource allocation, input / output (I / O) management, and coordinating the parallel operation of the AI ​​acceleration computing module, while also handling non-accelerated general-purpose computing tasks.

[0111] The power module is equipped with redundant power supply units to provide stable power distribution and management for the AI ​​acceleration computing module, CPU module and other modules.

[0112] The hard drive module integrates a high-speed solid-state drive (SSD) and / or a large-capacity hard disk drive (HDD), connected to the system bus via a backplane. The hard drive stores the operating system, AI training datasets, model parameters, and computation results, providing high-throughput data read / write channels and supporting data preprocessing and persistence.

[0113] The fan module uses a multi-zone independent speed-controlled fan array, which is configured in key heat source areas (such as AI accelerator cards and CPU heat dissipation areas) to achieve system heat dissipation through forced air cooling and ensure the stable operation of high-efficiency computing components.

[0114] The CPU module is connected to the AI ​​acceleration computing module via the PCIe bus to enable task distribution, result collection, and memory coordination.

[0115] The CPU module manages the data access of the hard drive module through SATA / SAS / NVMe interfaces.

[0116] The power module provides tiered power to all functional modules through the power distribution backplane.

[0117] The fan module adjusts the fan speed based on temperature monitoring signals from the CPU board and AI accelerator card.

[0118] In an exemplary embodiment, the electronic device may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), general-purpose processors, controllers, micro controller units (MCUs), microprocessors, or other electronic components to perform the aforementioned method.

[0119] Embodiments of this application also propose a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause a computer to execute the operator insertion method described in the above embodiments of this application.

[0120] Embodiments of this application also propose a computer program product, including a computer program that is executed by a processor using the operator insertion method described in the above embodiments of this application.

[0121] Embodiments of this application also propose a chip, such as Figure 12 As shown, the chip includes one or more interface circuits 1201 and one or more processors 1202; the interface circuits are used to receive signals from the memory of the electronic device and send signals to the processors, the signals including computer instructions stored in the memory, and when the processor executes the computer instructions, it causes the electronic device to perform the operator insertion method described in the above embodiments of this application.

[0122] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0123] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "illustrative embodiment," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with an embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0124] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process, and the scope of the preferred embodiments of the invention includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as will be understood by those skilled in the art to which embodiments of the invention pertain.

[0125] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a system including a processing module, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (control method), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic device, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which programs can be printed, because programs can be obtained electronically, for example, by optically scanning the paper or other media, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0126] It should be understood that various parts of the embodiments of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0127] Those skilled in the art will understand that all or part of the steps of the methods described in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0128] Furthermore, the functional units in the various embodiments of the present invention can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc.

[0129] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. An operator insertion method, characterized in that, The method includes: Obtain the second insertion data set of the second operator connected to the first operator in the model computation graph. The second insertion data set includes the number of conversion operators inserted when the output data of the second operator is in different data formats. The first operator is any operator in the model computation graph. Based on the second insertion data set and the data format of the input data required by the first operator, a first insertion data set for the first operator is determined. The first insertion data set includes the number of conversion operators inserted when the output data of the first operator is in different data formats. When the first operator is at the end of the model computation graph, the minimum transformation operator insertion amount of the first operator is determined based on the first insertion data set, so as to insert the transformation operator according to the minimum transformation operator insertion amount.

2. The method according to claim 1, characterized in that, Before obtaining the second insertion data set of the second operator connected to the first operator in the model computation graph, the method includes: Determine the operator position of the second operator in the model computation graph; When the second operator is located at the first operator position in the model computation graph, the data format of the input data of the model computation graph is determined; Based on the data format of the input data of the model computation graph and the data format of the input data required by the second operator, the second set of inserted data for the second operator is determined.

3. The method according to claim 1, characterized in that, The second insertion data set of the second operator connected to the first operator in the model computation graph includes: Determine the operator position of the first operator in the model computation graph; If the first operator is not located at the first operator position in the model computation graph, the second inserted data set is obtained.

4. The method according to claim 1, characterized in that, The process of determining the first insertion data set of the first operator based on the data format of the second insertion data set and the input data required by the first operator includes: Obtain the number of candidate transformation operators that match the data format of the input data required by the first operator from the second set of inserted data; The minimum candidate transformation operator insertion value among the candidate transformation operator insertion values ​​is determined as the target insertion value; The first set of inserted data is determined based on the target insertion amount.

5. The method according to claim 1, characterized in that, When the first operator is located at the end of the model computation graph, determining the minimum transformation operator insertion amount for the first operator based on the first inserted data set includes: When the first operator is located at the end operator position in the model computation graph, determine the data format of the output data of the model computation graph; If the data format of the output data of the model computation graph is inconsistent with the data format of the output data of the first operator, the insertion amount of the transformation operator of the first operator is adjusted to obtain an adjusted updated insertion data set, wherein the data format of the output data of the first operator in the updated insertion data set is the same as the data format of the output data of the model computation graph. Based on the updated insertion data set, determine the minimum number of transformation operators to be inserted for the first operator.

6. The method according to claim 1, characterized in that, The insertion transformation operator based on the minimum transformation operator includes: Determine the insertion position of the transformation operator corresponding to the minimum transformation operator insertion amount; Insert the transformation operator according to the insertion position of the transformation operator.

7. An operator insertion device, characterized in that, The device includes: The acquisition unit is used to acquire a second insertion data set of a second operator connected to the first operator in the model computation graph. The second insertion data set includes the number of conversion operators inserted when the output data of the second operator is in different data formats. The first operator is any operator in the model computation graph. The determining unit is configured to determine a first insertion data set for the first operator based on the second insertion data set and the data format of the input data required by the first operator. The first insertion data set includes the conversion operator insertion amount when the output data of the first operator is in different data formats. An insertion unit is configured to determine the minimum transformation operator insertion amount of the first operator based on the first insertion data set when the first operator is at the end operator position in the model computation graph, so as to insert a transformation operator according to the minimum transformation operator insertion amount.

8. An electronic device, characterized in that, include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-6.

9. A chip, characterized in that, The device includes one or more interface circuits and one or more processors; the interface circuits are configured to receive signals from the memory of the electronic device and send the signals to the processors, the signals including computer instructions stored in the memory, which, when executed by the processors, cause the electronic device to perform the method of any one of claims 1-6.

10. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-6.

Images