Operator execution method and apparatus, electronic device, and storage medium

Abstract

The present disclosure provides an operator execution method and device, electronic equipment and storage medium, relates to the technical field of artificial intelligence, in particular to the technical field of operators, matrix operations, vector operations and the like. The specific implementation scheme is as follows: determining a computation graph and a backend constraint corresponding to a target operator to be compiled; determining at least one target switching point chain of the target operator based on the computation graph and the backend constraint, the target switching point chain including at least one switching point, the switching point being between two different types of computation units; performing constraint solving on the target switching point chain to obtain a candidate operator execution strategy; determining a target operator execution strategy from the candidate operator execution strategy, and compiling the target operator based on the target operator execution strategy to obtain execution information of the target operator; wherein the execution information includes executable code and execution configuration information; and transmitting the execution information to a heterogeneous accelerator to execute the target operator.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, specifically to the fields of operators, matrix operations, vector operations, etc., and particularly to an operator execution method, apparatus, electronic device, and storage medium. Background Technology

[0002] For complex fusion operators in large models, the computation path typically involves alternating matrix and vector computations. In heterogeneous accelerators with explicit queues or explicit data handling, each data handover across computational units usually requires scheduling using explicit queues, explicit synchronization primitives, and local caches.

[0003] Currently, the generation of execution strategies for complex fusion operators is prone to problems where they are feasible in some parts but not in the entire chain. This makes it difficult to stably generate operator execution strategies that balance correctness, resource constraints, and execution performance when deploying large-model complex fusion operators in heterogeneous accelerators with explicit queues or explicit data transport. Summary of the Invention

[0004] This disclosure provides a method, apparatus, electronic device, and storage medium for operator execution.

[0005] According to one aspect of this disclosure, an operator execution method is provided, comprising: determining a computation graph and backend constraints corresponding to a target operator to be compiled; determining at least one target switching point chain of the target operator based on the computation graph and backend constraints, the target switching point chain including at least one switching point located between two different types of computational units; performing constraint solving on the target switching point chain to obtain candidate operator execution strategies; determining a target operator execution strategy from the candidate operator execution strategies, and compiling the target operator based on the target operator execution strategy to obtain execution information of the target operator; wherein the execution information includes executable code and execution configuration information; and transmitting the execution information to a heterogeneous accelerator to execute the target operator.

[0006] According to another aspect of this disclosure, an operator execution apparatus is provided, comprising: a first determining module, configured to determine a computation graph and backend constraints corresponding to a target operator to be compiled; a second determining module, configured to determine at least one target switching point chain of the target operator based on the computation graph and backend constraints, the target switching point chain including at least one switching point located between two different types of computational units; a constraint solving module, configured to perform constraint solving on the target switching point chain to obtain candidate operator execution strategies; a third determining module, configured to determine a target operator execution strategy from the candidate operator execution strategies, and compile the target operator based on the target operator execution strategy to obtain execution information of the target operator; wherein the execution information includes executable code and execution configuration information; and an execution module, configured to transmit the execution information to a heterogeneous accelerator to execute the target operator.

[0007] According to another aspect of this disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the operator execution method described in one aspect of the above-described embodiment.

[0008] According to another aspect of this disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein a computer program / instruction is stored thereon, the computer instructions being used to cause the computer to perform the operator execution method described in the above-described embodiment.

[0009] According to another aspect of this disclosure, a computer program product is provided, including a computer program / instructions that, when executed by a processor, implement the operator execution method described in one aspect of the embodiments above.

[0010] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0011] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0012] Figure 1 A flowchart illustrating an operator execution method provided in an embodiment of this disclosure; Figure 2 A flowchart illustrating another operator execution method provided in this embodiment of the disclosure; Figure 3A schematic diagram illustrating the process of constructing queue producer-consumer pairs provided in an embodiment of this disclosure; Figure 4 A flowchart illustrating the adaptation operation provided in the embodiments of this disclosure; Figure 5 A flowchart illustrating another operator execution method provided in this embodiment of the disclosure; Figure 6 A flowchart illustrating another operator execution method provided in this embodiment of the disclosure; Figure 7 A flowchart illustrating the process of determining the target operator execution strategy provided in this embodiment of the disclosure; Figure 8 This is a schematic diagram of the structure of an operator execution device provided in an embodiment of the present disclosure; Figure 9 This is a block diagram of an electronic device used to implement the operator execution method of the embodiments of this disclosure. Detailed Implementation

[0013] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0014] The operator execution method, apparatus, electronic device, and storage medium of the present disclosure are described below with reference to the accompanying drawings.

[0015] Artificial intelligence (AI) is the study of enabling computers to simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, and planning). It involves both hardware and software technologies. AI hardware technologies generally include computer vision, speech recognition, natural language processing, and related technologies such as deep learning, big data processing, and knowledge graphs.

[0016] In the context of artificial intelligence, operators can be the basic computational units in neural networks, such as matrix multiplication, convolution, activation, and normalization. Operators include matrix operators, vector operators, etc., and the operators in this disclosure can be complex fusion operators that combine matrix operators and vector operators.

[0017] Figure 1 This is a flowchart illustrating an operator execution method provided in an embodiment of the present disclosure.

[0018] like Figure 1As shown, the operator execution method may include: S101, determine the computation graph and backend constraints corresponding to the target operator to be compiled.

[0019] It should be noted that the executing entity of the operator execution method in this embodiment can be a hardware device with data processing capabilities and / or the necessary software to drive the hardware device. Optionally, the executing entity may include a server, a user terminal, and other smart devices. Optionally, the user terminal includes, but is not limited to, mobile phones, computers, smart voice interaction devices, etc. Optionally, the server includes, but is not limited to, a network server, an application server, or a server of a distributed system, or a server combined with blockchain, etc. This embodiment does not impose specific limitations.

[0020] It should be noted that an operator is a basic unit of computation. It encapsulates a certain mathematical operation logic (such as matrix multiplication, convolution, addition, etc.) and defines the shape, data type, and properties of the input or output tensor.

[0021] An operator ultimately corresponds to a sequence of instructions that runs on an accelerator. Different hardware may provide multiple implementations of the same operator, i.e., operator execution strategies.

[0022] In some embodiments, the target operator can be an operator to be compiled into an explicit queue heterogeneous accelerator. That is, the operator needs to be converted into executable code in the heterogeneous accelerator so that the operator can be executed in the heterogeneous accelerator. The executable code can be generated based on the operator's execution strategy.

[0023] In this embodiment of the disclosure, the target operator includes at least one of the following: a fusion attention operator, a fusion multilayer perceptron operator, a normalization fusion operator, or an expert fusion operator in an expert hybrid model.

[0024] In some embodiments, the target operator can be obtained by acquiring a list of operators for a large model and then retrieving the operator to be compiled from the list of operators.

[0025] In some embodiments, the target operator can also be an operator to be optimized or an operator for which an execution strategy is to be generated. Here, an operator to be optimized refers to an operator that already has a corresponding execution strategy, but this strategy does not meet performance requirements, and a better-performing execution strategy can be determined to achieve optimization.

[0026] In this embodiment of the disclosure, the target operator is an operator with a matrix-vector or vector-matrix multi-level switching relationship. That is, the calculation of the target operator switches back and forth between the batch intensive operation processed by the matrix calculation unit and the point-by-point or simple vector operation processed by the vector calculation unit. The target operator can be an overall calculation task that alternately executes a series of matrix operations and vector operations.

[0027] In some embodiments, the target operator can be decomposed into a directed graph composed of basic mathematical operations, serving as a computation graph. The computation graph consists of at least one node, and each node can be a node of a different type.

[0028] For example, nodes in a computation graph can include: matrix nodes, vector nodes, and control / transfer / adaptation nodes. Among them, matrix nodes are nodes mapped to matrix computation units; vector nodes are nodes that can be mapped to vector computation units; and control / transfer / adaptation nodes are nodes used for layout transformation, copying, synchronization, reshaping, transposition, type conversion, filling, or defilling.

[0029] Among them, matrix-type nodes include: General Matrix Multiply (GEMM), batched General Matrix Multiply (batched GEMM), convolution, and Query-Key Transpose (QK). At least one of the following: probability-value multiplication (PV) and tensor shrinkage operation.

[0030] Vector-type nodes include at least one of the following: elementwise operation, reduction, normalization, activation, mask, softmax, bias add, and residual add.

[0031] In some embodiments, the backend constraints of a target operator refer to a set of interface conventions, memory layouts, calling rules, and hardware characteristic commitments that the target operator and the hardware backend must adhere to. Corresponding backend constraints can be pre-configured for each operator.

[0032] S102, based on the computation graph and backend constraints, determine at least one target switching point chain for the target operator.

[0033] In some embodiments, the nodes in the computation graph have different node types. In this disclosure, the target operator is to perform matrix operations and vector operations alternately. Therefore, the node types may include matrix nodes and vector nodes, and matrix nodes and vector nodes can be mapped to corresponding computation units respectively.

[0034] In some embodiments, at least one initial switching point chain corresponding to the target operator can be determined first based on the computation graph, and the initial switching point chain can be adapted based on backend constraints to obtain at least one target switching point chain.

[0035] In some embodiments, the target switching point chain includes at least one switching point located between two different types of computing units. That is, the initial switching point chain also includes at least one switching point located between two different types of computing units.

[0036] In this embodiment of the disclosure, different types of computing units are matrix computing units and vector computing units, which correspond to matrix-type nodes and vector-type nodes. Switching points can be set between matrix-type nodes and vector-type nodes to generate an initial switching point chain.

[0037] In some embodiments, the output of the matrix calculation unit is not compatible with the input of the vector calculation unit, and the output of the vector calculation unit is not compatible with the input of the matrix calculation unit. That is, the information output by the matrix calculation unit cannot be directly input into the vector calculation unit, and the information output by the vector calculation unit cannot be directly input into the matrix calculation unit.

[0038] In some embodiments, the initial switching point chain is adapted based on backend constraints, that is, the outputs and inputs of different computing units are adapted so that the information output by the matrix computing unit can be directly input into the vector computing unit, and the information output by the vector computing unit can be directly input into the matrix computing unit.

[0039] In some embodiments, backend constraints include at least the following fields: explicit queue mechanism, queue depth limit, local storage capacity, matrix computation unit capability, vector computation unit capability, synchronization primitives, data layout, alignment units, and supported data types.

[0040] In some embodiments, in the initial switching point chain, the matrix calculation unit can be a producer that inputs output information to the vector calculation unit, or a consumer that receives output information from the vector calculation unit; the vector calculation unit can be a producer that inputs output information to the matrix calculation unit, or a consumer that receives output information from the matrix calculation unit.

[0041] In other words, based on backend constraints, queue producer-consumer pairs corresponding to the initial switching point chain can be constructed. Then, according to the input-output relationship of different computing units indicated by the queue producer-consumer pairs, the output and input can be adapted so that the output information of any computing unit can be directly input into another computing unit of a different type, thereby obtaining the target switching point chain.

[0042] S103, perform constraint solving on the target switching point chain to obtain candidate operator execution strategies.

[0043] In some embodiments, different constraints can be obtained, and the target switching point chain can be constrained and solved based on the different constraints to obtain one or more configuration parameter combinations corresponding to the target switching point chain when all constraints are satisfied, and candidate operator execution strategies can be generated based on the configuration parameter combinations.

[0044] In some embodiments, constraints can limit the range of different parameters. Under the combined effect of multiple constraints, a combination of configuration parameters that simultaneously satisfy the constraints can be obtained to generate candidate operator execution strategies.

[0045] In some embodiments, the constraints may be at least one of the following: local storage capacity constraints, queue depth constraints, synchronization resource constraints, alignment constraints, cache lifecycle constraints, and adaptation feasibility constraints.

[0046] In other words, it is possible to solve for the combination of configuration parameters when the target switching point chain satisfies at least one constraint, and generate candidate operator execution strategies based on the combination of configuration parameters.

[0047] In some embodiments, the configuration parameter combination may include, but is not limited to: the queued on-chip direct transfer strategy of the switching point, the external relay strategy, the queue position, the queue depth, the number of buffers, the synchronization term strategy, the synchronization primitive, the release strategy, the adaptation strategy, the address alignment requirement, the cache time, etc.

[0048] It should be noted that constraints can limit the range of possible values ​​for different parameters. In other words, the same parameter can have multiple values ​​when all constraints are satisfied. For example, for parameter 1, if the constraint limits the range of possible values ​​for parameter A to between 0.4 and 5, then parameter A can be any value within that range, such as 1, 2, 0.5, etc.

[0049] By solving for at least one value of different parameters when at least one constraint condition is met simultaneously, at least one combination of configuration parameters can be determined based on this value, thereby generating at least one candidate operator execution strategy based on at least one combination of configuration parameters.

[0050] In some embodiments, interval scheduling, graph coloring, enumeration pruning, or heuristic search methods can be used to solve for the combination of configuration parameters when the target switching point chain satisfies the constraints.

[0051] In some embodiments, candidate operator execution strategies can be generated through rule generation. This involves pre-setting generation rules and combining configuration parameters according to those rules to generate candidate operator execution strategies.

[0052] For example, generating candidate operator execution strategies based on rule templates.

[0053] In some embodiments, the agent can also generate an operator execution strategy based on a combination of configuration parameters. The operator execution strategy can also be generated by a compiler.

[0054] In some embodiments, a candidate generator can be constructed based on generation rules, an agent, and a compiler, and the candidate generator can generate an operator execution strategy based on a combination of configuration parameters.

[0055] It should be noted that the operator execution strategy refers to the final executable implementation of the target operator on specific explicit queues or explicit data handling hardware, such as heterogeneous accelerators.

[0056] In this embodiment of the disclosure, the candidate operator execution strategy includes at least one of the following: external relay baseline candidate, partially queued on-chip direct transfer candidate, fully queued on-chip direct transfer candidate, conservative backoff candidate, and local computation path rewriting candidate.

[0057] S104, determine the target operator execution strategy from the candidate operator execution strategies, and compile the target operator based on the target operator execution strategy to obtain the execution information of the target operator.

[0058] In some embodiments, candidate operator execution strategies can be screened to determine the target operator execution strategy. For example, candidate operator execution strategies can be verified, and the verified strategies can be used as the target operator execution strategy.

[0059] In some embodiments, if there is a verified operator execution strategy, it can be used as the target operator execution strategy. If there are multiple verified operator execution strategies, performance tests can be performed on these strategies, and the best-performing operator execution strategy can be selected as the target operator execution strategy based on the test results.

[0060] In some embodiments, verifying the operator execution strategy can be done by verifying the correctness of the operator execution strategy, thereby filtering out strategies that can be executed correctly and avoiding situations where the determined target operator execution strategy cannot be executed.

[0061] In some embodiments, after determining the target operator execution strategy, the target operator can be compiled based on the target operator execution strategy to convert the target operator execution strategy into a sequence of instructions, such as code, that the target operator can execute. That is, the execution information of the target operator may include executable code.

[0062] In some embodiments, the combination of configuration parameters corresponding to the execution strategy of the target operator can also be used as execution configuration information, so that the target operator can execute executable code according to the execution configuration information. That is, the execution information includes executable code and execution configuration information.

[0063] S105, the execution information is transmitted to the heterogeneous accelerator to execute the target operator.

[0064] It should be noted that heterogeneous accelerators refer to dedicated artificial intelligence (AI) processors that differ from general-purpose processor architectures, contain multiple types of computing units such as matrices and vectors, and communicate using explicit queues and data handling instructions.

[0065] The heterogeneous accelerator in this disclosure can be a heterogeneous accelerator having matrix computation units, vector computation units, scalar control units, explicit local storage, explicit queues, and explicit synchronization primitives.

[0066] In some embodiments, by transmitting execution information to a heterogeneous accelerator, the execution information can be executed in the heterogeneous accelerator to realize the execution target operator. Executable code and execution configuration information can be obtained from the execution information, and the executable code can be executed according to the execution configuration information to realize the execution target operator.

[0067] For example, a heterogeneous accelerator can launch vector computing units and matrix computing units based on executable code, and ultimately complete the target operator by autonomously moving data, performing calculations, and passing intermediate results through queues.

[0068] According to the operator execution method provided in this disclosure, by determining the computation graph and backend constraints corresponding to the target operator to be compiled, and based on the computation graph and backend constraints, at least one target switching point chain of the target operator is determined, thereby enabling constraint solving of the target switching point chain to obtain candidate operator execution strategies. By determining the target operator execution strategy from the candidate operator execution strategies, and compiling the target operator based on the target operator execution strategy, the execution information of the target operator is obtained, and the execution information is transmitted to the heterogeneous accelerator for execution of the target operator. Therefore, this solution generates operator execution strategies through computation graphs and backend constraints, enabling automated generation of operator execution strategies. During the generation of operator execution strategies, the identification of switching point chains clarifies data transmission relationships, and through joint constraint solving, the operator execution strategy maximizes direct transmission paths, reduces off-chip write-back of intermediate results, and ensures the correctness of the execution strategy while guaranteeing hardware resource requirements. Furthermore, during the execution of the operator execution strategy, the execution efficiency and resource utilization of the target operator on the heterogeneous accelerator are improved.

[0069] Figure 2 This is a flowchart illustrating another operator execution method provided in an embodiment of the present disclosure.

[0070] like Figure 2 As shown, the operator execution method may include: S201, determine the computation graph and backend constraints corresponding to the target operator to be compiled.

[0071] The details of step S201 can be found in the above embodiments and will not be repeated here.

[0072] S202, Generate the initial switching point chain corresponding to the target operator based on the computation graph.

[0073] In some embodiments, the computation graph is a directed graph composed of nodes and edges. A node in the computation graph can be understood as a computational operation or sub-operator, wherein the computational operation or sub-operator can be mapped to a matrix computation unit, a vector computation unit, or a control / transfer / adaptation unit. The nodes included in the computation graph are identified, and their node types are determined. The node type can indicate the type of computation unit, which includes matrix computation units and vector computation units.

[0074] In some embodiments, the data handover boundary between adjacent nodes of different node types can be determined according to the topological order of the computation graph, and a switching point can be set at the data handover boundary to obtain an initial switching point chain.

[0075] In some embodiments, matrix calculations and vector calculations alternate linearly, such as matrix A - vector operation 1 - matrix B - vector operation 2. By setting switching points between matrix A and vector operation 1, between vector operation 1 and matrix B, and between matrix B and vector operation 2, an initial switching point chain can be obtained as: matrix A - switching point 1 - vector operation 1 - switching point 2 - matrix B - switching point 3 - vector operation 2.

[0076] Because the output of one computational unit in a computation graph can correspond to the input of one or more computational units, the computation graph can have branching, parallel, or multi-path structures. For example, the output of a matrix operation can be split into two paths, each fed into a different vector operation.

[0077] By constructing a corresponding switching point chain for each branch, parallel, or multi-path structure, multiple initial switching point chains corresponding to the target operator can be obtained.

[0078] In this disclosure, by identifying switching point chains, it is possible to explicitly model the matrix and vector computation units in the target operator, accurately characterize the data dependencies and switching boundaries between computation stages, and improve the scheduling capability of the operator.

[0079] S203, based on backend constraints, construct queue producer-consumer pairs corresponding to the initial switching point chain.

[0080] In some embodiments, for each switching point in the initial switching point chain, a queue producer-consumer pair can be constructed based on backend constraints. The producer and consumer in the queue producer-consumer pair correspond to different computational units.

[0081] In some embodiments, the queue producer-consumer pair includes the producer-consumer pair at each switching point, so the producer and consumer corresponding to each switching point can be determined first. Based on the switching points in the initial switching point chain, the first computing unit upstream of the switching point can be determined as the producer, and the second computing unit downstream of the switching point can be determined as the consumer, to obtain the producer-consumer pair.

[0082] For example, suppose the initial switching point chain is matrix A - switching point 1 - vector operation 1 - switching point 2 - matrix B - switching point 3 - vector operation 2. For switching point 1, its corresponding first calculation unit is matrix A, then the producer is matrix calculation unit A, and its corresponding second calculation unit is vector operation 1, then the consumer is vector calculation unit 1.

[0083] For switching point 2, its corresponding first calculation unit is vector operation 1, so the producer is vector calculation unit 1, and its corresponding second calculation unit is matrix B, so the consumer is matrix calculation unit B.

[0084] By combining the first and second computing units, producer-consumer pairs can be obtained. For example, the producer-consumer pair at switching point 1 is matrix computing unit A - vector computing unit 1; the producer-consumer pair at switching point 2 is vector computing unit 1 - matrix computing unit B.

[0085] In some embodiments, after determining the producer-consumer pair at each switching point, the abstract producer-consumer relationship can be materialized into hardware queue instances. Adjustable parameters can be determined from backend constraints and adjusted based on producer attribute information and consumer expectation information to obtain the adjusted parameters.

[0086] The producer's attribute information may include: the shape, data type, layout of the output data, the number of bytes per data block, and the production rate. The consumer's expected information may include: the required shape, data type, layout, alignment boundaries, and the amount of data consumed each time.

[0087] In some embodiments, backend constraints include at least the following fields: explicit queuing mechanism, queue depth limit, local storage capacity, matrix computation unit capability, vector computation unit capability, synchronization primitives, data layout, alignment units, and supported data types. Adjustable parameters may include: storage capacity, queue depth, synchronization primitives, etc.

[0088] For example, if the producer produces one data block at a time and the consumer's processing speed is slow, the queue depth can be adjusted from a minimum of 2 to 4 to avoid producer blocking.

[0089] In some embodiments, adjusting adjustable parameters based on producer attribute information and consumer expectation information can enable the producer's output to be consumed by the consumer without violating hardware constraints.

[0090] In some embodiments, queue instances can be determined based on adjusted parameters, and each producer-consumer pair can be combined into a queue instance to obtain queue producer-consumer pairs. Queue instances that satisfy the adjusted parameters can be determined based on the adjusted parameters to solidify the adjusted parameters into a hardware-recognizable queue configuration.

[0091] Figure 3 This is a schematic diagram of the process for constructing queue producer-consumer pairs provided in an embodiment of this disclosure. Figure 3 (a) in the text is the process of constructing queue producer-consumer pairs.

[0092] For the computation graph of the target operator, different node types, such as matrix and vector, can be identified from the computation graph. Based on the identified nodes, the data handover boundary between matrix and vector nodes can be determined. Switching points can then be inserted at the boundaries of adjacent matrix-vector or vector-matrix nodes to obtain the initial switching point chain.

[0093] Furthermore, queue producer-consumer pairs are constructed based on the initial switching point chain. For example, the producer-consumer pair at each switching point is defined, and the adjustable parameters in the backend constraints are adjusted according to the producer's attribute information and the consumer's expectation information. Based on the adjusted parameters, the queue instance is determined, and each producer-consumer pair is combined into the queue instance to obtain the queue producer-consumer pair.

[0094] Figure 3 (b) is an example of a queue producer-consumer pair: vector normalization - switch point 1 - matrix multiplication 1 - switch point 2 - vector activation - switch point 3 - matrix multiplication 2 - switch point 4 - vector bias addition and residual connection.

[0095] S204, based on the queue producer-consumer pair, performs an adaptation operation on the initial switching point chain to obtain the target switching point chain.

[0096] In some embodiments, the queue producer-consumer pair may indicate each producer-consumer pair in the initial switching point chain, i.e., matrix computation unit-vector computation unit, or vector computation unit-matrix computation unit.

[0097] The data transmission process involves inputting information from the producer to the consumer. However, since the information output by the producer cannot necessarily be directly input to the consumer—that is, the information output by the matrix calculation unit cannot be directly input to the vector calculation unit, and the information output by the vector calculation unit cannot be directly input to the matrix calculation unit—an adaptation operation can be performed on the initial switching point chain. This allows the producers in the initial switching point chain to directly input their output information to the consumer, reducing the risk of cross-computation unit data adaptation errors.

[0098] In some embodiments, adaptation operations can be performed based on the difference information of producer-consumer pairs. This is achieved by determining the difference information of each producer-consumer pair in the queue. This difference information may include: shape differences, layout differences, data type differences, accumulation boundaries, padding differences, alignment requirements, etc., and the specific representation of the difference information is shown below.

[0099] For example, regarding shape differences, it could be whether the data blocks output by the matrix are consistent with the one-dimensional or two-dimensional data blocks consumed by the vector; regarding layout differences, it could be whether the row-major order, column-major order, block layout, internal padding, and continuity requirements are consistent; regarding data type differences, it could be that the matrix calculation unit uses a high-precision accumulator, while the vector calculation unit uses a low-precision accumulator.

[0100] Regarding accumulation boundaries, if vector calculation requires a complete accumulation result, the switching point must be located after the matrix accumulation is completed. Regarding padding differences, there may be padding data in the data block output by the producer, but the consumer only wants to process valid data, so padding needs to be removed. Regarding alignment requirements, there may be whether the producer output address, consumer read address, transport length, or data block byte count meets the backend alignment unit.

[0101] In some embodiments, once the difference information is determined, an adaptation operator can be determined based on the difference information, and an adaptation operation can be performed on the initial switching point chain based on the adaptation operator to obtain the target switching point chain.

[0102] The adaptation operator is used to adapt the producer's output information to the input information required by the consumer.

[0103] In some embodiments, the adaptation operator may include at least one of transpose, reshape, layout rearrangement, data type conversion, padding, depadding, masking, and vector expectation partitioning.

[0104] For example, if the difference information includes data type differences, then data type conversion can be selected as the adaptation operator to adapt the initial switching point chain.

[0105] In some embodiments, the adaptation operation on the initial switching point chain can be based on the adaptation operator, which adapts the producer in the queue producer-consumer pair corresponding to the initial switching point chain so that the producer can output the data expected by the consumer to the consumer.

[0106] Figure 4 This is a flowchart illustrating the adaptation operation provided in the embodiments of this disclosure. Figure 4 The diagram illustrates the adaptation operations for the producer-consumer pair, from matrix computation unit to vector computation unit. For a data block generated by the matrix computation unit, a series of optional adaptation operations can be performed, such as transpose / reshape, layout rearrangement, data type conversion, padding / depadding / masking, and vector expectation segmentation, to obtain the input data format that the vector computation unit expects to receive.

[0107] S205, perform constraint solving on the target switching point chain to obtain candidate operator execution strategies.

[0108] S206, determine the target operator execution strategy from the candidate operator execution strategies, and compile the target operator based on the target operator execution strategy to obtain the execution information of the target operator.

[0109] S207, the execution information is transmitted to the heterogeneous accelerator to execute the target operator.

[0110] The details of steps S206-S207 can be found in the above embodiments and will not be repeated here.

[0111] According to the operator execution method provided in this disclosure, an initial switching point chain corresponding to the target operator can be generated based on the computation graph. Based on backend constraints, queue producer-consumer pairs corresponding to the initial switching point chain are constructed. Then, based on the queue producer-consumer pairs, an adaptation operation is performed on the initial switching point chain to obtain the target switching point chain. By adapting the difference information between matrix computation units and vector computation units, the differences between the outputs and inputs of different computation units can be eliminated, thereby reducing the risk of cross-computation unit data adaptation errors.

[0112] Figure 5 This is a flowchart illustrating another operator execution method provided in an embodiment of the present disclosure.

[0113] like Figure 5 As shown, the operator execution method may include: S501, determine the computation graph and backend constraints corresponding to the target operator to be compiled.

[0114] S502, based on the computation graph and backend constraints, determines at least one target switching point chain for the target operator.

[0115] The details of steps S501-S502 can be found in the above embodiments and will not be repeated here.

[0116] S503, determine at least one constraint condition.

[0117] In some embodiments, at least one constraint may be predetermined, such as one sent by the client. Alternatively, it may be generated by a large model based on historical experience information.

[0118] In some embodiments, at least one constraint may be generated based on backend constraints, so that the configuration information obtained by solving does not exceed the upper limit specified by the backend constraints.

[0119] In some embodiments, at least one constraint includes: local storage capacity constraint, queue depth constraint, synchronization resource constraint, alignment constraint, cache lifecycle constraint, and adaptation feasibility constraint.

[0120] S504. Based on at least one constraint condition, perform constraint solution on the target switching point chain to obtain candidate configuration information that satisfies at least one constraint condition.

[0121] In some embodiments, a joint constraint model can be established based on at least one constraint condition, and the target switching point chain can be input into the joint constraint model for constraint solving to obtain candidate configuration information that satisfies at least one constraint condition.

[0122] In some embodiments, the candidate configuration information includes combinations of configuration parameters for each switching point in the target switching point chain. The parameters in these configuration parameter combinations are those that satisfy at least one constraint condition for each switching point in the target switching point chain.

[0123] It is understandable that constraints can limit the range of possible parameters. In other words, when the same parameter satisfies all constraints, it can take multiple values, resulting in multiple combinations of configuration parameters and multiple candidate configuration information.

[0124] In some embodiments, the candidate configuration information includes at least one of the following: a queued on-chip direct transfer strategy for the switching point, an external relay strategy, a queue position, a queue depth, a number of buffers, a synchronization term strategy, a synchronization primitive, a release strategy, an adaptation strategy, an address alignment requirement, and a cache timing.

[0125] In other words, by solving the joint constraints, at least one of the following can be determined as parameters in the configuration parameter combination: the queued on-chip direct transmission strategy, external relay strategy, queue position, queue depth, number of buffers, synchronization term strategy, synchronization primitive, release strategy, adaptation strategy, address alignment requirements, and cache timing of the switching point, so as to obtain candidate configuration information.

[0126] In this disclosure, by solving the joint constraints of at least one constraint, the combination of configuration parameters that satisfies all constraints can be automatically determined, which can reduce the cost of operator development and porting, and improve the efficiency of operator tuning.

[0127] In this embodiment of the disclosure, the solution of constraints can be explained for local storage capacity constraints, queue depth constraints, synchronization resource constraints, alignment constraints, cache lifecycle constraints, and adaptation feasibility constraints.

[0128] For example, based on local storage capacity constraints, the target switching point chain is constrained and the candidate configuration information that satisfies the local storage capacity constraints is obtained. This can be achieved by: obtaining the storage capacity occupied by the target switching point chain when it is active at the target time, and solving for the first configuration parameter combination when the storage capacity is less than the product of the local storage capacity and the safety factor, as the candidate configuration information.

[0129] The first configuration parameter combination includes at least the producer buffer, consumer buffer, number of buffers, and release strategy at the switching point when the local storage capacity constraint is met.

[0130] In some embodiments, the storage capacity occupied when the target switching point chain is active at the target time can be obtained by acquiring the first storage capacity of the switching point's buffer when it is active at the target time, the second storage capacity of the queue item associated with the switching point when it is active at the target time, and the third storage capacity of the adaptation work area associated with the switching point when it is active at the target time, and summing the first, second, and third storage capacities.

[0131] The target time can be the scheduling time, which is used to represent the order or time point of different operations (such as data writing, queue enqueueing, dequeueing, adaptation calculation, etc.) during the execution of the operator.

[0132] Local storage capacity constraints can be expressed as: (1) Where i represents the switching point i, Indicates whether the target is active at any given time. Indicates a buffer. Represents a queue item. This indicates that the workspace is being adapted. Indicates local storage capacity. This represents the safety factor.

[0133] In some embodiments, the core of the local storage capacity constraint is to ensure that the on-chip local storage occupied by the active local storage at the same time does not exceed the local storage capacity limit.

[0134] Based on queue depth constraints, the target switching point chain is constrained and solved to obtain candidate configuration information that satisfies the queue depth constraints. This can be achieved in the following way: determine the queue depth of the queue producer-consumer pair corresponding to the target switching point chain, and solve for the second configuration parameter combination when the queue depth is less than or equal to the depth threshold, as the candidate configuration information.

[0135] The second configuration parameter combination includes at least the queued on-chip direct transmission strategy, external relay strategy, queue depth, and queue position of the switching point when the queue depth constraint is satisfied.

[0136] In some embodiments, the second configuration parameter combination may also include the number of queue items, candidate values ​​for queue depth, etc.

[0137] In some embodiments, the queued on-chip direct transfer strategy can indicate whether the switching point uses queued on-chip direct transfer; the external relay strategy can indicate whether the switching point uses external relay.

[0138] In some embodiments, the queue depth can be determined based on the producer's attribute information and the consumer's expectation information when generating queue producer-consumer pairs.

[0139] The queue depth constraint can be expressed as: (2) in, Indicates the queue depth. This represents the depth threshold.

[0140] Based on synchronization resource constraints, the target switching point chain is constrained and solved to obtain candidate configuration information that satisfies the synchronization resource constraints. This can be achieved in the following way: obtain the number of synchronization vocabularies that are active at the same time in the target switching point chain at the target time, and solve for the third configuration parameter combination when the number of synchronization vocabularies is less than or equal to the maximum number of vocabularies, as the candidate configuration information.

[0141] The third configuration parameter combination includes at least the synchronization term strategy and synchronization primitive of the switching point when the synchronization resource constraints are met.

[0142] In some embodiments, the third configuration parameter combination may also include lexical allocation method, waiting position, trigger position, release strategy, and synchronization safety order between switching points.

[0143] In some embodiments, the number of simultaneously active synchronization tokens can be determined based on the queue depth and the number of buffers. The number of simultaneously active synchronization tokens is usually equal to the queue depth, since each queue slot requires one token to ensure synchronization.

[0144] Synchronization resource constraints can be expressed as: (3) in, Indicates the number of synchronization words. Indicates the maximum number of lexical units.

[0145] In some embodiments, the core of synchronization resource constraints is to ensure that the number of synchronization resources active at any given time does not exceed the upper limit specified by the backend constraints, and to avoid the problem of consumers consuming resources before the producers have finished.

[0146] Based on alignment constraints, the target switching point chain is constrained to obtain candidate configuration information that satisfies the alignment constraints. This can be achieved in the following way: obtain at least one of the memory address, byte step size, number of bytes, and number of transported bytes associated with the switching point in the target switching point chain; solve for at least one of the memory address, byte step size, number of bytes, and number of transported bytes associated with the switching point; and use the fourth configuration parameter combination when it is an integer multiple of the alignment unit as candidate configuration information.

[0147] The fourth configuration parameter combination includes at least the address alignment requirements of the switching point when the alignment constraints are met.

[0148] In some embodiments, the fourth configuration parameter combination may also include: data block byte count, data block shape, padding / depadding size, and data layout, etc.

[0149] In some embodiments, the memory address associated with the switching point can be the producer output address and the consumer read address, the byte step size can be determined based on the byte distance between the starting addresses of adjacent data blocks, the number of bytes is the total number of bytes occupied by a single data block, and the number of bytes moved is the number of data bytes moved in one queue transfer operation.

[0150] Based on cache lifecycle constraints, the target switching point chain is constrained and the candidate configuration information that satisfies the cache lifecycle constraints is obtained. This can be achieved by obtaining the cache lifecycles of at least two switching points in the target switching point chain and solving for the fifth configuration parameter combination when their respective cache lifecycles do not overlap, which is then used as the candidate configuration information.

[0151] The fifth configuration parameter combination includes at least one of the following when the cache lifecycle constraint is satisfied: the cache timing of the switching point, the number of buffers, the cache reuse strategy, and the release strategy.

[0152] In some embodiments, when the cache lifetimes of two different switching points do not overlap, local storage can be reused as a cache reuse strategy; when the lifetimes overlap, an independent buffer must be allocated or other feasible strategies must be adopted as an independent buffer allocation strategy.

[0153] In some embodiments, cache lifecycle constraints can be solved using interval scheduling, graph coloring, lifecycle analysis, or heuristic allocation methods.

[0154] Based on the adaptation feasibility constraint, the target switching point chain is constrained and solved to obtain candidate configuration information that satisfies the adaptation feasibility constraint. This can be achieved in the following way: obtain the adaptation operator of the switching point in the target switching point chain, and solve for the sixth configuration parameter combination when the adaptation operator can convert the producer's data attributes into the data attributes expected by the consumer, as the candidate configuration information.

[0155] The sixth configuration parameter combination includes at least the adaptation strategy for the switching point when the adaptation feasibility constraint is met.

[0156] In some embodiments, the adaptation operator of the switching point in the target switching point chain refers to the adaptation operator used when performing an adaptation operation on the initial switching point chain to obtain the target switching point chain. That is, the adaptation operator is determined based on the difference information of each producer-consumer pair in the queue producer-consumer pair.

[0157] In some embodiments, interval scheduling, graph coloring, enumeration pruning, or heuristic search methods can be used to solve for the candidate configuration information corresponding to the target switching point chain for each constraint.

[0158] In this embodiment of the disclosure, by solving joint constraints, constraints such as local storage capacity, queue depth, synchronization resources, and alignment requirements can be checked, configuration information that does not meet the backend constraints can be eliminated, hardware anomalies caused by resource overruns or synchronization deadlocks can be avoided, and the buffer lifecycle of each switching point can be ensured to be free from conflict and the adaptation operator can be feasible, thereby ensuring the correctness of the switching point chain in data transmission and transformation.

[0159] In some embodiments, when the vector computation unit in the target switching point chain includes row reduction, softmax, normalization or masking, at least one constraint, i.e., the joint constraint model, may further include a reduction semantic integrity constraint to ensure that the range of data blocks participating in the reduction satisfies the corresponding vector computation semantics.

[0160] When softmax is implemented using integrity reduction, the producer data block width is constrained to cover the reduction width of softmax; when softmax is implemented using block-based online softmax, the joint constraint model can further constrain the lifecycle of block statistics and cumulative states.

[0161] In some embodiments, after obtaining candidate configuration information, when the candidates for full-queue on-chip direct transmission do not meet the joint constraint model, the queue depth can be automatically reduced, the number of buffers reduced, the data block size reduced, the adapter workspace reduced, or some switching points can be changed to external relay paths to generate new candidate configuration information.

[0162] S505, retrieve the set generation rules.

[0163] S506 generates candidate operator execution strategies from candidate configuration information through generation rules.

[0164] In some embodiments, the set generation rules can be sent by the client, and a candidate generator can be built based on the generation rules so that the candidate configuration information can be generated into candidate operator execution strategies by the candidate generator.

[0165] In some embodiments, the agent can also generate an operator execution strategy based on a combination of configuration parameters. The operator execution strategy can also be generated by a compiler.

[0166] In some embodiments, a candidate generator can be constructed based on generation rules, an agent, and a compiler, thereby generating an operator execution strategy based on the candidate configuration information.

[0167] For example, candidate operator execution strategies may include: 1. External relay baseline candidate: All switching points write intermediate results back to external storage, and then read them back in subsequent stages; 2. Partially queued on-chip direct transfer candidates: Some switching points use explicit queues and local buffers for on-chip direct transfer, while other switching points use external relays; 3. Full-queue on-chip direct transfer candidate: All switching points use explicit queues and local buffers for on-chip direct transfer; 4. Conservative rollback candidate: When the queued on-chip direct transmission configuration does not meet the constraints, external relay, segmented execution, or existing backend library path are adopted; 5. Candidate Local Computation Path Rewriting: Under the conditions of semantic equivalence, backend contract permission, and acceptable cost, rewrite part of the vector segment or matrix segment into a form that is more suitable for execution by the target computation unit.

[0168] In some embodiments, candidate operator execution strategies are generated hierarchically using a subset of queued on-chip direct transmission at switching points, and at least one fully external relay candidate is reserved as a safety baseline.

[0169] In some embodiments, conservative fallback candidates include one or more of the following: external transfer, segmented execution, existing backend library path, or backend pre-built operator implementation.

[0170] In some embodiments, candidate operator execution strategies may be generated based on one or more of latency, local storage peak, queue event utilization, transport volume, adapter overhead, synchronization term usage, or coverage shape range.

[0171] S507, determine the target operator execution strategy from the candidate operator execution strategies, and compile the target operator based on the target operator execution strategy to obtain the execution information of the target operator.

[0172] S508 transmits execution information to the heterogeneous accelerator to execute the target operator.

[0173] The relevant content of steps S507-S508 can be found in the above embodiments, and will not be repeated here.

[0174] According to the operator execution method provided in this disclosure, by determining at least one constraint condition and solving the constraint on the target switching point chain based on the at least one constraint condition, candidate configuration information satisfying the at least one constraint condition is obtained. This allows the acquisition of predefined generation rules, and the candidate configuration information is then used to generate candidate operator execution strategies. By jointly solving the constraints, the cost of manual tuning can be reduced, and by generating operator execution strategies based on the candidate configuration information, the operator execution strategies can satisfy backend constraints, thereby improving the execution efficiency of the operators.

[0175] Figure 6 This is a flowchart illustrating another operator execution method provided in an embodiment of the present disclosure.

[0176] like Figure 6 As shown, the operator execution method may include: S601, determine the computation graph and backend constraints corresponding to the target operator to be compiled.

[0177] S602, based on the computation graph and backend constraints, determines at least one target switching point chain for the target operator.

[0178] S603, perform constraint solving on the target switching point chain to obtain candidate operator execution strategies.

[0179] The details of steps S601-S603 can be found in the above embodiments and will not be repeated here.

[0180] S604, verify the correctness of the candidate operator execution strategy and obtain the operator execution strategy that passes the correctness verification.

[0181] In some embodiments, a candidate operator execution strategy can be compared with a reference operator execution strategy to verify whether the candidate operator execution strategy is correct, wherein the reference operator execution strategy is a standard, reliable, and correct strategy.

[0182] In some embodiments, element-by-element comparison and overall comparison can be performed to determine whether the candidate operator execution strategy is correct, and the operator execution strategy that passes the element-by-element comparison or the overall comparison is taken as the operator execution strategy that passes the correctness verification.

[0183] In some embodiments, synchronous security verification can be performed on the candidate operator execution strategy to verify the security of the candidate operator execution strategy, and backend constraint verification can be performed on the candidate operator execution strategy to verify whether the candidate operator execution strategy conforms to the backend constraints.

[0184] S605 tests the execution strategies of operators that have passed the correctness verification, and obtains the test performance of each execution strategy of the operators that have passed the correctness verification.

[0185] In some embodiments, if there is only one operator execution strategy that passes the correctness verification, it can be used as the target operator execution strategy; if there are multiple operator execution strategies that pass the correctness verification, performance tests can be performed on the operator execution strategies that pass the correctness verification to determine their respective test performance.

[0186] In some embodiments, performance testing may include tests such as latency, local storage peak, queue event utilization, throughput, and adapter overhead to determine test performance. These include parameters such as latency duration, utilization rate, throughput, and adapter overhead.

[0187] In some embodiments, the operator execution strategy that has passed the correctness verification can be tested in the key tensor shapes, where the key tensor shapes refer to a set of tensor shapes that have a decisive impact on the performance or correctness of the operator and that occur frequently in actual loads or contribute significantly to the overall performance.

[0188] S606 determines the target operator execution strategy from the operator execution strategies that have passed the correctness verification based on test performance.

[0189] In some embodiments, the target operator execution strategy can be determined based on the test performance of each operator execution strategy that passes the correctness verification. For example, the operator execution strategy with the best test performance can be selected as the target operator execution strategy.

[0190] In other words, the execution strategies of operators that have passed the correctness verification can be ranked based on test performance. For example, the execution strategies of operators that have passed the correctness verification can be ranked in descending order of test performance. The execution strategy of the operator ranked first is the execution strategy of the operator with the best test performance.

[0191] In some embodiments, in response to the ranking result indicating the existence of an operator execution strategy with optimal test performance, the operator execution strategy with optimal test performance is used as the target operator execution strategy.

[0192] In some embodiments, in response to the sorting result indicating the existence of multiple operator execution strategies with optimal test performance, a target operator execution strategy is determined from the multiple operator execution strategies with optimal test performance based on performance selection rules.

[0193] In some embodiments, the performance selection rule can be a shape distribution rule, which can select the target operator execution strategy based on the shape of the operator runtime input tensor. That is, the operator execution strategy is tested on the tensor shape, so the target operator execution strategy can be determined based on the performance of the operator execution strategy on different tensor shapes, i.e., the test performance.

[0194] For example, it can be determined whether the input shape belongs to a predefined shape region A. If it does, the operator execution strategy with the best performance in shape region A is determined as the target operator execution strategy. Otherwise, it is further determined whether the input shape belongs to a predefined shape region B. If it does, the operator execution strategy with the best performance in shape region B is determined as the target operator execution strategy.

[0195] If none of the operations belong to any predefined shape region, then a general operator execution strategy can be used as the target operator execution strategy. This general operator execution strategy can be a pre-defined operator execution strategy that performs at the baseline test performance level.

[0196] In some embodiments, the shape distribution rules include one or more of the following: batch size, sequence length, hidden size, head number, data type, or layout.

[0197] In this disclosure, the target operator execution strategy is determined from the operator execution strategies that have passed the correctness verification by testing the performance. This can ensure that the target operator execution strategy has the best performance and enhance the performance stability under different tensor shapes.

[0198] S607, compiles the target operator based on the target operator execution strategy to obtain the target operator's execution information.

[0199] S608 transmits execution information to the heterogeneous accelerator to execute the target operator.

[0200] The details of steps S607-S608 can be found in the above embodiments and will not be repeated here.

[0201] According to the operator execution method provided in this disclosure, candidate operator execution strategies are verified for correctness, and those strategies that pass the verification are obtained. These verified strategies are then tested to obtain their respective test performance. Based on the test performance, a target operator execution strategy is determined from the verified strategies. Correctness verification filters logically correct operator execution strategies, and performance testing on tensor shapes allows for the selection of target operator execution strategies based on test performance. This enables the selection of different target operator execution strategies for different tensor shapes, enhancing performance stability.

[0202] Figure 7 This is a flowchart illustrating the process of determining the target operator execution strategy provided in an embodiment of this disclosure. Figure 7 The joint constraint solution is based on the target switching point chain to determine at least one candidate operator execution strategy. Figure 7 In the solution, a solver can be used to solve the constraints C1-C6. C1 is the local storage capacity constraint, C2 is the queue depth constraint, C3 is the synchronization resource constraint, C4 is the alignment constraint, C5 is the cache lifetime constraint, and C6 is the adaptation feasibility constraint.

[0203] For details on the specific implementation of the candidate operator execution strategy, please refer to the above embodiments, which will not be repeated here.

[0204] By verifying the correctness and testing the performance of candidate operator execution strategies, the correct and optimal target operator execution strategy can be determined. Specific implementation methods for determining the target operator execution strategy can be found in the above embodiments, and will not be repeated here.

[0205] Corresponding to the operator execution methods provided in the above embodiments, one embodiment of this disclosure also provides an operator execution device. Since the operator execution device provided in this disclosure corresponds to the operator execution methods provided in the above embodiments, the implementation methods of the above operator execution methods are also applicable to the operator execution device provided in this disclosure, and will not be described in detail in the following embodiments.

[0206] Figure 8 This is a schematic diagram of the structure of an operator execution device provided in an embodiment of the present disclosure.

[0207] like Figure 8 As shown, the operator execution device 800 of this embodiment includes a first determining module 801, a second determining module 802, a constraint solving module 803, a third determining module 804, and an execution module 805.

[0208] The first determining module 801 is used to determine the computation graph and backend constraints corresponding to the target operator to be compiled; The second determining module 802 is used to determine at least one target switching point chain of the target operator based on the computation graph and back-end constraints. The target switching point chain includes at least one switching point, which is located between two different types of computation units. The constraint solving module 803 is used to solve the constraints of the target switching point chain to obtain candidate operator execution strategies; The third determining module 804 is used to determine the target operator execution strategy from the candidate operator execution strategies, and compile the target operator based on the target operator execution strategy to obtain the execution information of the target operator; wherein, the execution information includes executable code and execution configuration information; The execution module 805 is used to transmit the execution information to the heterogeneous accelerator to execute the target operator.

[0209] In one embodiment of this disclosure, the second determining module 802 is further configured to: generate an initial switching point chain corresponding to the target operator based on the computation graph; construct a queue producer-consumer pair corresponding to the initial switching point chain based on the backend constraints; wherein the computation units corresponding to the producers and consumers in the queue producer-consumer pair are different; and perform an adaptation operation on the initial switching point chain based on the queue producer-consumer pair to obtain the target switching point chain.

[0210] In one embodiment of this disclosure, the second determining module 802 is further configured to: identify the nodes included in the computation graph and determine the node type of the nodes; determine the data handover boundary between adjacent nodes of different node types, and set a switching point at the data handover boundary to obtain the initial switching point chain.

[0211] In one embodiment of this disclosure, the second determining module 802 is further configured to: determine a first computing unit upstream of the switching point as the producer and a second computing unit downstream of the switching point as the consumer, based on the switching point in the initial switching point chain, to obtain a producer-consumer pair; determine an adjustable parameter from the backend constraints, and adjust the adjustable parameter based on the attribute information of the producer and the expectation information of the consumer, to obtain the adjusted parameter; determine a queue instance based on the adjusted parameter, and combine each producer-consumer pair into the queue instance to obtain the queue producer-consumer pair.

[0212] In one embodiment of this disclosure, the second determining module 802 is further configured to: determine the difference information of each producer-consumer pair in the queue producer-consumer pair; determine an adaptation operator based on the difference information; and perform an adaptation operation on the initial switching point chain based on the adaptation operator to obtain the target switching point chain; wherein the adaptation operator is used to adapt the output information of the producer to the input information required by the consumer.

[0213] In one embodiment of this disclosure, the constraint solving module 803 is further configured to: determine at least one constraint condition; perform constraint solving on the target switching point chain based on the at least one constraint condition to obtain candidate configuration information that satisfies the at least one constraint condition; wherein, the candidate configuration information includes a combination of configuration parameters for each switching point in the target switching point chain; obtain a set generation rule; and generate the candidate configuration information as the candidate operator execution strategy through the generation rule.

[0214] In one embodiment of this disclosure, the at least one constraint includes: local storage capacity constraint, queue depth constraint, synchronization resource constraint, alignment constraint, cache lifecycle constraint, and adaptation feasibility constraint; the candidate configuration information includes at least one of the following: queued on-chip direct transmission strategy, external relay strategy, queue position, queue depth, number of buffers, synchronization term strategy, synchronization primitive, release strategy, adaptation strategy, address alignment requirement, cache reuse strategy, and cache timing.

[0215] In one embodiment of this disclosure, the constraint solving module 803 is further configured to: obtain the storage capacity occupied by the target switching point chain when it is active at the target time; solve for a first configuration parameter combination when the storage capacity is less than the product of the local storage capacity and the safety factor, as the candidate configuration information; wherein the first configuration parameter combination includes at least the producer buffer, consumer buffer, number of buffers, and release strategy of the switching point when the local storage capacity constraint is satisfied.

[0216] In one embodiment of this disclosure, the constraint solving module 803 is further configured to: determine the queue depth of the queue producer-consumer pair corresponding to the target switching point chain; solve for a second configuration parameter combination when the queue depth is less than or equal to a depth threshold, as the candidate configuration information; wherein the second configuration parameter combination includes at least the queued on-chip direct transmission strategy, external relay strategy, queue depth, and queue position of the switching point when the queue depth constraint is satisfied.

[0217] In one embodiment of this disclosure, the constraint solving module 803 is further configured to: obtain the number of synchronous lexicals that are simultaneously active in the target switching point chain at the target time; solve for a third configuration parameter combination when the number of synchronous lexicals is less than or equal to the maximum number of lexicals, as the candidate configuration information; wherein the third configuration parameter combination includes at least the synchronization lexical strategy and synchronization primitive of the switching point when the synchronization resource constraint is satisfied.

[0218] In one embodiment of this disclosure, the constraint solving module 803 is further configured to: obtain at least one of the memory address, byte step size, number of bytes, and number of transported bytes associated with the switching point in the target switching point chain; solve for a fourth configuration parameter combination when at least one of the memory address, byte step size, number of bytes, and number of transported bytes associated with the switching point is an integer multiple of the alignment unit, and use it as the candidate configuration information; wherein, the fourth configuration parameter combination includes at least the address alignment requirement of the switching point when the alignment constraint is satisfied.

[0219] In one embodiment of this disclosure, the constraint solving module 803 is further configured to: obtain the cache lifetime of each of at least two switching points in the target switching point chain; solve for a fifth configuration parameter combination when the respective cache lifetimes do not overlap, as the candidate configuration information; wherein the fifth configuration parameter combination includes at least one of the following when the cache lifetime constraint is satisfied: cache time, buffer quantity, cache reuse strategy, and release strategy of the switching point.

[0220] In one embodiment of this disclosure, the constraint solving module 803 is further configured to: obtain the adaptation operator of the switching point in the target switching point chain; solve for a sixth configuration parameter combination when the adaptation operator can convert the producer's data attributes into the data attributes expected by the consumer, as the candidate configuration information; wherein the sixth configuration parameter combination includes at least the adaptation strategy of the switching point when the adaptation feasibility constraint is satisfied.

[0221] In one embodiment of this disclosure, the third determining module 804 is further configured to: perform correctness verification on the candidate operator execution strategies and obtain operator execution strategies that pass the correctness verification; test the operator execution strategies that pass the correctness verification to obtain the test performance of each of the operator execution strategies that pass the correctness verification; and determine the target operator execution strategy from the operator execution strategies that pass the correctness verification based on the test performance.

[0222] In one embodiment of this disclosure, the third determining module 804 is further configured to: sort the operator execution strategies that have passed the correctness verification based on the test performance; in response to the sorting result indicating the existence of an operator execution strategy with optimal test performance, select the operator execution strategy with optimal test performance as the target operator execution strategy; and in response to the sorting result indicating the existence of multiple operator execution strategies with optimal test performance, determine the target operator execution strategy from the multiple operator execution strategies with optimal test performance based on performance selection rules.

[0223] According to the operator execution apparatus provided in this disclosure, by determining the computation graph and backend constraints corresponding to the target operator to be compiled, and based on the computation graph and backend constraints, at least one target switching point chain of the target operator is determined. This allows for constraint solving of the target switching point chain to obtain candidate operator execution strategies. By determining the target operator execution strategy from the candidate strategies and compiling the target operator based on the target execution strategy, execution information of the target operator is obtained. This execution information is then transmitted to a heterogeneous accelerator for execution of the target operator. Therefore, this solution generates operator execution strategies through computation graphs and backend constraints, enabling automated generation of operator execution strategies. During the generation of operator execution strategies, the identification of switching point chains clarifies data transmission relationships. Through joint constraint solving, direct transmission paths can be maximized while ensuring hardware resource availability, reducing off-chip write-back of intermediate results, and improving the execution efficiency and resource utilization of the target operator on the heterogeneous accelerator.

[0224] The acquisition, storage, and application of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0225] The examples in this disclosure may involve user data, data acquisition, and / or use. All of these aspects comply with relevant laws, regulations, and provisions. In the examples of this disclosure, all data collection, acquisition, processing, manipulation, forwarding, and use are conducted with the user's knowledge and confirmation. Accordingly, when implementing each example, the type, scope of use, and usage scenarios of any data or information that may be involved should be communicated to the user and their authorization obtained through appropriate means in accordance with relevant laws and regulations. The specific methods of notification and / or authorization can vary depending on the actual situation and application scenario, and the scope of the solution is not limited in this regard.

[0226] In this manual and the sample solutions, any processing of personal information will be conducted only under legal grounds (such as obtaining the consent of the data subject or being necessary for the performance of a contract) and will only be carried out within the scope stipulated or agreed upon. A user's refusal to process personal information beyond what is necessary for basic functions will not affect the user's use of basic functions.

[0227] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0228] Figure 9 A schematic block diagram of an example electronic device 900 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0229] like Figure 9 As shown, device 900 includes a computing unit 901, which can perform various appropriate actions and processes based on computer programs / instructions stored in read-only memory (ROM) 902 or loaded from storage unit 908 into random access memory (RAM) 903. RAM 903 may also store various programs and data required for the operation of device 900. The computing unit 901, ROM 902, and RAM 903 are interconnected via bus 904. Input / output (I / O) interface 905 is also connected to bus 904.

[0230] Multiple components in device 900 are connected to I / O interface 905, including: input units 906 such as keyboard, mouse, etc.; output units 907 such as various types of displays, speakers, etc.; storage units 908 such as disks, optical disks, etc.; and communication units 909 such as network interface cards, modems, wireless transceivers, etc. Communication unit 909 allows device 900 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0231] The computing unit 901 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 901 performs the various methods and processes described above, such as operator execution methods. For example, in some embodiments, the operator execution method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 908. In some embodiments, part or all of the computer program / instructions may be loaded and / or installed on device 900 via ROM 902 and / or communication unit 909. When the computer program / instructions are loaded into RAM 903 and executed by the computing unit 901, one or more steps of the operator execution method described above may be performed. Alternatively, in other embodiments, the computing unit 901 may be configured to execute operator execution methods by any other suitable means (e.g., by means of firmware).

[0232] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include: implementations in one or more computer programs / instructions that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transferring data and instructions to the storage system, the at least one input device, and the at least one output device.

[0233] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0234] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0235] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0236] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), the Internet, and blockchain networks.

[0237] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. The client-server relationship is created by computer programs / instructions running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0238] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0239] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. An operator execution method, wherein, The method includes: Determine the computation graph and backend constraints corresponding to the target operator to be compiled; Based on the computation graph and backend constraints, at least one target switching point chain of the target operator is determined, the target switching point chain including at least one switching point, the switching point being located between two different types of computation units; Constraints are solved on the target switching point chain to obtain candidate operator execution strategies; The target operator execution strategy is determined from the candidate operator execution strategies, and the target operator is compiled based on the target operator execution strategy to obtain the execution information of the target operator; wherein, the execution information includes executable code and execution configuration information; The execution information is transmitted to a heterogeneous accelerator to execute the target operator.

2. The method according to claim 1, wherein, The step of determining at least one target switching point chain for the target operator based on the computation graph and backend constraints includes: Generate the initial switching point chain corresponding to the target operator based on the computation graph; Based on the backend constraints, a queue producer-consumer pair corresponding to the initial switching point chain is constructed; wherein, the producer and consumer in the queue producer-consumer pair correspond to different computing units; Based on the queue producer-consumer pair, the initial switching point chain is adapted to obtain the target switching point chain.

3. The method according to claim 2, wherein, The step of generating the initial switching point chain corresponding to the target operator based on the computation graph includes: Identify the nodes included in the computation graph and determine the node type of the nodes; Determine the data handover boundary between adjacent nodes of different node types, and set a switching point at the data handover boundary to obtain the initial switching point chain.

4. The method according to claim 2, wherein, The step of constructing the queue producer-consumer pair corresponding to the initial switching point chain based on the backend constraints includes: Based on the switching points in the initial switching point chain, a first computing unit upstream of the switching point is determined as the producer, and a second computing unit downstream of the switching point is determined as the consumer, so as to obtain a producer-consumer pair; Adjustable parameters are determined from the backend constraints, and adjusted based on the producer's attribute information and the consumer's expectation information to obtain the adjusted parameters. Based on the adjusted parameters, a queue instance is determined, and each producer-consumer pair is combined into the queue instance to obtain the queue producer-consumer pair.

5. The method according to claim 4, wherein, The process of adapting the initial switching point chain based on the queue producer-consumer pair to obtain the target switching point chain includes: Determine the difference information of each producer-consumer pair in the queue producer-consumer pair; Based on the difference information, an adaptation operator is determined, and based on the adaptation operator, an adaptation operation is performed on the initial switching point chain to obtain the target switching point chain; wherein, the adaptation operator is used to adapt the producer's output information to the consumer's required input information.

6. The method according to claim 1, wherein, The constraint solving of the target switching point chain to obtain candidate operator execution strategies includes: Identify at least one constraint; Based on the at least one constraint condition, the target switching point chain is constrained and solved to obtain candidate configuration information that satisfies the at least one constraint condition; wherein, the candidate configuration information includes the configuration parameter combination of each switching point in the target switching point chain; Get the set generation rules; The candidate configuration information is generated into the candidate operator execution strategy using the generation rules.

7. The method according to claim 6, wherein, The at least one constraint includes: local storage capacity constraint, queue depth constraint, synchronization resource constraint, alignment constraint, cache lifecycle constraint, and adaptation feasibility constraint. The candidate configuration information includes at least one of the following: the queued on-chip direct transmission strategy, external relay strategy, queue position, queue depth, number of buffers, synchronization term strategy, synchronization primitive, release strategy, adaptation strategy, address alignment requirements, cache reuse strategy, and cache timing.

8. The method according to claim 7, wherein, Based on the local storage capacity constraint, the target switching point chain is constrained to obtain candidate configuration information that satisfies the local storage capacity constraint, including: Obtain the storage capacity occupied by the target switching point chain when it is active at the target time; The first combination of configuration parameters when the storage capacity is less than the product of the local storage capacity and the security factor is used as the candidate configuration information; The first configuration parameter combination includes at least the producer buffer, consumer buffer, number of buffers, and release strategy of the switching point when the local storage capacity constraint is satisfied.

9. The method according to claim 7, wherein, Based on the queue depth constraint, the target switching point chain is constrained to obtain candidate configuration information that satisfies the queue depth constraint, including: Determine the queue depth of the producer-consumer pair corresponding to the target switching point chain; The second combination of configuration parameters when the queue depth is less than or equal to the depth threshold is determined and used as the candidate configuration information. The second configuration parameter combination includes at least the queued on-chip direct transmission strategy, external relay strategy, queue depth, and queue position of the switching point when the queue depth constraint is satisfied.

10. The method according to claim 7, wherein, Based on the synchronization resource constraints, the target switching point chain is constrained to obtain candidate configuration information that satisfies the synchronization resource constraints, including: Obtain the number of synchronous morphemes that are simultaneously active in the target switching point chain at the target time; The third combination of configuration parameters is determined when the number of synchronization lexical units is less than or equal to the maximum number of lexical units, and is used as the candidate configuration information; The third configuration parameter combination includes at least the synchronization term strategy and synchronization primitive of the switching point when the synchronization resource constraints are satisfied.

11. The method according to claim 7, wherein, Based on the alignment constraints, the target switching point chain is constrained to obtain candidate configuration information that satisfies the alignment constraints, including: Obtain at least one of the following in the target switching point chain: memory address, byte step size, number of bytes, and number of bytes moved; The fourth configuration parameter combination, which is an integer multiple of the memory address, byte step size, number of bytes, and number of bytes to be moved associated with the switching point, is used as the candidate configuration information. The fourth configuration parameter combination includes at least the address alignment requirement of the switching point when the alignment constraint is satisfied.

12. The method according to claim 7, wherein, Based on the cache lifecycle constraints, the target switching point chain is constrained to obtain candidate configuration information that satisfies the cache lifecycle constraints, including: Obtain the cache lifetime of at least two switching points in the target switching point chain; The fifth configuration parameter combination when the respective cache lifetimes do not overlap is determined and used as the candidate configuration information; The fifth configuration parameter combination includes at least one of the following when the cache lifecycle constraint is satisfied: the cache timing of the switching point, the number of buffers, the cache reuse strategy, and the release strategy.

13. The method according to claim 7, wherein, Based on the aforementioned adaptation feasibility constraints, the target switching point chain is constrained and solved to obtain candidate configuration information that satisfies the adaptation feasibility constraints, including: Obtain the adaptation operator for the switching points in the target switching point chain; The sixth configuration parameter combination that the adaptation operator can convert the producer's data attributes into the consumer's expected data attributes is used as the candidate configuration information. The sixth configuration parameter combination includes at least the adaptation strategy of the switching point when the adaptation feasibility constraint is satisfied.

14. The method according to any one of claims 1-6, wherein, Determining the target operator execution strategy from the candidate operator execution strategies includes: The correctness of the candidate operator execution strategies is verified, and the operator execution strategies that pass the correctness verification are obtained. The execution strategies of the operators that have passed the correctness verification are tested to obtain the test performance of each of the execution strategies of the operators that have passed the correctness verification. Based on the test performance, the target operator execution strategy is determined from the operator execution strategies that have passed the correctness verification.

15. The method according to claim 14, wherein, The step of determining the target operator execution strategy from the operator execution strategies that have passed the correctness verification based on the test performance includes: Based on the test performance, the operators that passed the correctness verification are ranked according to the execution strategy. In response to the sorting result indicating the existence of an operator execution strategy with optimal test performance, the operator execution strategy with optimal test performance is taken as the target operator execution strategy; In response to the sorting result indicating the existence of multiple operator execution strategies with optimal test performance, the target operator execution strategy is determined from the multiple operator execution strategies with optimal test performance based on performance selection rules.

16. An operator execution device, wherein, The device includes: The first determining module is used to determine the computation graph and backend constraints corresponding to the target operator to be compiled; The second determining module is used to determine at least one target switching point chain of the target operator based on the computation graph and back-end constraints. The target switching point chain includes at least one switching point, which is located between two different types of computation units. The constraint solving module is used to solve the constraints of the target switching point chain to obtain candidate operator execution strategies; The third determining module is used to determine the target operator execution strategy from the candidate operator execution strategies, and compile the target operator based on the target operator execution strategy to obtain the execution information of the target operator; wherein, the execution information includes executable code and execution configuration information; An execution module is used to transmit the execution information to a heterogeneous accelerator to execute the target operator.

17. An electronic device comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method as described in any one of claims 1-15.

18. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-15.

19. A computer program product comprising a computer program / instructions, wherein, When the computer program / instructions are executed by the processor, they implement the method of any one of claims 1-15.

Images