Instruction sequence generation apparatus, method, storage medium, and electronic device

Abstract

An instruction sequence generation apparatus, method, storage medium and electronic device are disclosed. The apparatus comprises a processor configured to: obtain a computation graph; determine a set of splitting strategies corresponding to the computation graph; in response to a first type of splitting strategy in the set of splitting strategies not satisfying a preset hardware memory limit condition, determine a first target splitting manner satisfying the preset hardware memory limit condition based on a second type of splitting strategy in the set of splitting strategies and the first type of splitting strategy; the first type of splitting strategy is used for splitting the entire computation graph, and the second type of splitting strategy is used for splitting part of the computation graph; and generate a first target instruction sequence for splitting the computation graph according to the first target splitting manner. The present disclosure can achieve splitting or blocking of the computation graph through combination of different splitting strategies, thereby reducing the access frequency of the computation graph to the main memory during computation, reducing the access delay of the computation graph during computation, and further shortening the computation time and improving the computation efficiency.

Description

Technical Field

[0001] This disclosure relates to compiler optimization techniques, and in particular to an instruction sequence generation apparatus, method, storage medium, and electronic device. Background Technology

[0002] Tiling is a compiler optimization technique. Its core idea is to break down intensive computational tasks for large datasets into several small data block (tile) level subtasks that are adapted to high-speed memory. This allows for full reuse of data in high-speed memory during computation, thereby reducing the frequency of access to main memory and improving computational efficiency.

[0003] When applying Tiling to the field of deep learning model inference, it is often necessary to split the computation graph and generate an instruction sequence based on the splitting result. By executing the instruction sequence, the computation task corresponding to the computation graph can be completed by making full use of the data in high-speed memory.

[0004] However, in some cases, the computation graph may not be able to be split, thus making it impossible to reduce the frequency of access to main memory and improve computational efficiency based on the splitting of the computation graph. Summary of the Invention

[0005] To address the aforementioned technical problems, this disclosure provides an instruction sequence generation apparatus, method, storage medium, and electronic device.

[0006] According to one aspect of the present disclosure, an instruction sequence generation apparatus is provided, including a processor configured to: Obtain the computation graph; Determine the set of splitting strategies corresponding to the computation graph; In response to the first type of splitting strategy in the splitting strategy set not meeting the preset hardware memory limit, a first target splitting method that meets the preset hardware memory limit is determined based on the second type of splitting strategy in the splitting strategy set and the first type of splitting strategy; wherein, the first type of splitting strategy is used to split the entire computation graph, and the second type of splitting strategy is used to split a portion of the computation graph; Generate a first target instruction sequence to split the computation graph according to the first target splitting method.

[0007] According to another aspect of the present disclosure, a method for generating an instruction sequence is provided, comprising: Obtain the computation graph; Determine the set of splitting strategies corresponding to the computation graph; In response to the first type of splitting strategy in the splitting strategy set not meeting the preset hardware memory limit, a first target splitting method that meets the preset hardware memory limit is determined based on the second type of splitting strategy in the splitting strategy set and the first type of splitting strategy; wherein, the first type of splitting strategy is used to split the entire computation graph, and the second type of splitting strategy is used to split a portion of the computation graph; Generate a first target instruction sequence to split the computation graph according to the first target splitting method.

[0008] According to another aspect of the present disclosure, a computer-readable storage medium is provided, the storage medium storing a computer program that is executed by a processor to implement the above-described instruction sequence generation method.

[0009] According to another aspect of the present disclosure, an electronic device is provided, including the above-described instruction sequence generation apparatus.

[0010] According to another aspect of the present disclosure, a computer program product is provided, which, when instructions in the computer program product are executed by a processor, performs the above-described instruction sequence generation method.

[0011] Based on the instruction sequence generation apparatus, method, storage medium, electronic device, and program product provided in the above embodiments of this disclosure, if the first type of splitting strategy in the splitting strategy set does not meet the preset hardware memory limit, a first target splitting method that meets the preset hardware memory limit is determined based on the second type of splitting strategy and the first type of splitting strategy in the splitting strategy set, and a first target instruction sequence is generated to split the computation graph according to the first target splitting method. In related technologies, a splitting strategy is usually determined for the computation graph. If the splitting strategy does not meet the preset hardware memory limit, the computation graph is usually not split. This results in global, coarse-grained memory access and computation dependencies during the computation process, leading to high access frequency to main memory and low computational efficiency. However, in the embodiments of this disclosure, a splitting strategy set corresponding to the computation graph can be determined. If the first type of splitting strategy in the splitting strategy set does not meet the preset hardware memory limit, a first target splitting method that meets the preset hardware memory limit can be determined based on the second type of splitting strategy and the first type of splitting strategy in the splitting strategy set. The computation graph is then split using the first target splitting method, so that the split computation graph meets the preset hardware memory limit, thereby achieving the purpose of optimizing the computation graph. Therefore, in the embodiments of this disclosure, the instruction sequence generation device can split or block the computation graph by combining different splitting strategies, so that the computation process conforms to local, fine-grained tile-level dependencies. This allows for full reuse of data in high-speed memory, thereby effectively reducing the frequency of main memory access during computation, reducing access latency during computation, and thus effectively shortening computation time and improving computation efficiency. Attached Figure Description

[0012] Figure 1 This is a schematic diagram of an exemplary application scenario of this disclosure.

[0013] Figure 2 This is a schematic diagram of the structure of an instruction sequence generation apparatus provided in some exemplary embodiments of this disclosure.

[0014] Figure 3-1 This is a schematic diagram of a computation graph in some exemplary embodiments of this disclosure.

[0015] Figure 3-2 This is a schematic diagram illustrating the dimensional mapping relationship between the output data and input data of a reshaping operator in some exemplary embodiments of this disclosure.

[0016] Figure 3-3 This is a schematic diagram of the computation graph and the first split dimension transmission relationship graph in some exemplary embodiments of this disclosure.

[0017] Figure 4 This is a schematic diagram of a computation graph in some other exemplary embodiments of this disclosure.

[0018] Figure 5 This is a schematic diagram of a computation graph and a second computation subgraph in some exemplary embodiments of this disclosure.

[0019] Figure 6-1 This is a schematic diagram of a fourth computational subgraph in some exemplary embodiments of this disclosure.

[0020] Figure 6-2 This is a schematic diagram of a computation graph and a fourth computation subgraph in some exemplary embodiments of this disclosure.

[0021] Figure 7 This is one of the flowcharts illustrating an instruction sequence generation method provided in some exemplary embodiments of this disclosure.

[0022] Figure 8-1 This is a second flowchart illustrating an instruction sequence generation method provided in some exemplary embodiments of this disclosure.

[0023] Figure 8-2 This is the third flowchart illustrating an instruction sequence generation method provided in some exemplary embodiments of this disclosure.

[0024] Figure 9 This is the fourth flowchart illustrating a method for generating instruction sequences provided in some exemplary embodiments of this disclosure.

[0025] Figure 10 This is the fifth flowchart illustrating an instruction sequence generation method provided in some exemplary embodiments of this disclosure.

[0026] Figure 11 This is the sixth flowchart illustrating an instruction sequence generation method provided in some exemplary embodiments of this disclosure.

[0027] Figure 12 This is the seventh flowchart illustrating a method for generating instruction sequences provided in some exemplary embodiments of this disclosure.

[0028] Figure 13 This is a schematic diagram of the structure of a chip provided by some exemplary embodiments of this disclosure.

[0029] Figure 14 This is a schematic diagram of the structure of an electronic device provided by some exemplary embodiments of this disclosure. Detailed Implementation

[0030] To explain this disclosure, exemplary embodiments of the disclosure will now be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the disclosure, and not all of them. It should be understood that the disclosure is not limited to exemplary embodiments.

[0031] It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of this disclosure.

[0032] Application Overview A computation graph consists of the basic data structure tensor and the basic computational unit operator. Nodes are typically used to represent operators in a computation graph, and directed edges between nodes represent tensor states and describe the dependencies between computations.

[0033] When applying Tiling to deep learning model inference, it is often necessary to decompose the computation graph and compile the results to generate instruction sequences. Execution of these instructions allows for the full reuse of data in high-speed memory, completing the computational tasks corresponding to the computation graph. It's important to note that decomposing the computation graph allows for the reconfiguration of the execution mode of the computational tasks. For example, without decomposition, subsequent operators in the computation graph must wait for the preceding operator to complete the computation of all its input data before starting execution. However, with decomposition, once the preceding operator completes the computation of a tile and obtains the corresponding output data, which is stored in high-speed memory, subsequent operators can perform computations based on this output data. Therefore, by decomposing the computation graph, global, coarse-grained memory access and computational dependencies can be transformed into local, fine-grained tile-level dependencies, which is beneficial for both meeting hardware memory space limitations and facilitating instruction scheduling.

[0034] In the process of developing this disclosure, the inventors discovered that in some cases, the computation graph cannot be split. For example, if splitting the computation graph results in each tile exceeding the hardware memory limit, the computation graph cannot be split, thus preventing the reduction of main memory access frequency and the improvement of computational efficiency based on computation graph splitting.

[0035] Exemplary System Figure 1 An exemplary application scenario of this disclosure is shown, including instruction sequence generation apparatus 110 and chip 120.

[0036] Optionally, the instruction sequence generation device 110 is an electronic device capable of compiling neural network models. This electronic device includes, but is not limited to, terminal devices, computer systems, servers, etc. Terminal devices include, but are not limited to, personal computers, tablets, etc. Examples of well-known terminal devices, computing systems, environments, and / or configurations suitable for use with terminal devices, computer systems, servers, etc., include, but are not limited to: personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, network personal computers, minicomputer systems, mainframe computer systems, and distributed cloud computing environments including any of the above systems. Terminal devices may also include, in vehicle scenarios, in-vehicle intelligent terminals, in-vehicle infotainment terminals, in-vehicle sensing terminals, in-vehicle computing terminals; or in robotic scenarios, robot body control terminals, robot sensing terminals, robot interaction terminals, robot execution terminals, etc.

[0037] Optionally, chip 120 is a semiconductor chip capable of executing a compiled instruction sequence, such as, but not limited to, intelligent driving chips, intelligent cockpit chips, image processing chips, etc. In some embodiments, chip 120 may include a neural network accelerator to execute the compiled instruction sequence. Neural network accelerators include, but are not limited to, neural processing units (NPUs) and vision processing units (VPUs). It should be noted that chip 120 and instruction sequence generation device 110 may be located in different electronic devices or in the same electronic device; this disclosure does not limit this.

[0038] In the embodiments of this disclosure, during the compilation stage, the instruction sequence generation device 110 can execute the instruction sequence generation method in the embodiments of this disclosure to obtain a target instruction sequence for deployment on the chip 120 based on the splitting of the computation graph. Specifically, the instruction sequence generation device 110 can acquire the computation graph; determine the splitting strategy set corresponding to the computation graph; in response to a first type of splitting strategy in the splitting strategy set not satisfying a preset hardware memory limit, determine a first target splitting method corresponding to the computation graph that satisfies the preset hardware memory limit based on the second type of splitting strategy and the first type of splitting strategy in the splitting strategy set; wherein, the first type of splitting strategy is used to split the entire computation graph, and the second type of splitting strategy is used to split a portion of the computation graph; and generate a first target instruction sequence that splits the computation graph according to the first target splitting method. During the runtime stage, the chip 120 is used to execute the target instruction sequence to complete the corresponding neural network computation task. The neural network computation task is, for example, but not limited to, semantic segmentation tasks, instance segmentation tasks, object detection tasks, environmental perception tasks, or intelligent driving tasks.

[0039] Exemplary device Figure 2 This is a schematic diagram of the structure of an instruction sequence generation apparatus 110 provided in some exemplary embodiments of this disclosure. For example... Figure 2 As shown, the instruction sequence generation device 110 includes a processor 210, which is configured to: acquire a computation graph; determine a set of splitting strategies corresponding to the computation graph; in response to a first type of splitting strategy in the splitting strategy set not satisfying a preset hardware memory limit, determine a first target splitting method corresponding to the computation graph that satisfies the preset hardware memory limit based on a second type of splitting strategy and the first type of splitting strategy in the splitting strategy set; wherein, the first type of splitting strategy is used to split the entire computation graph, and the second type of splitting strategy is used to split a portion of the computation graph; and generate a first target instruction sequence that splits the computation graph according to the first target splitting method.

[0040] Optionally, processor 210 is a circuit in instruction sequence generation device 110 that compiles the model by executing a computer program to generate instructions, execute instructions, process data, and coordinate and control the operation of other components. Processor 210 may be, for example, but is not limited to, a Central Processing Unit (CPU) or a Microcontroller Unit (MCU). The number of processors 210 may be one or more. For ease of understanding, the following description will use the case of one processor 210 as an example.

[0041] Optionally, the computation graph acquired by the processor 210 may be the computation graph corresponding to the neural network model to be compiled. The neural network model to be compiled may include, but is not limited to, semantic segmentation models, instance segmentation models, object detection models, environment perception models, end-to-end models, and Transformer models (which are neural network models based on a self-attention mechanism). The computation graph acquired by the processor 210 may include multiple operators, such as, but not limited to, convolution operators, pooling operators, store operators, elementwise operators, rectified linear unit (ReLU) operators, reshape operators, transpose operators, and linear operators.

[0042] Optionally, after acquiring the computation graph, the processor 210 can determine the set of splitting strategies corresponding to the computation graph based on brute-force search. The set of splitting strategies includes two types: a first type and a second type. The first type of splitting strategy is used to split the entire computation graph. For example, the first type of splitting strategy can indicate the splitting dimension corresponding to the output data of each operator in the computation graph, so that each operator in the computation graph is split along the splitting dimension corresponding to that operator. The first type of splitting strategy can also be called a complete Tiling strategy. The second type of splitting strategy is used to split a portion of the computation graph. For example, the second type of splitting strategy can indicate the splitting dimension corresponding to the output data of some operators in the computation graph, so that some operators are split along the splitting dimension corresponding to that operator. Here, the partial operators can be operators in the computation subgraph obtained by splitting the computation graph according to the first type of splitting strategy, or partial operators can be partial operators in the computation subgraph obtained by splitting the computation graph according to the first type of splitting strategy. Of course, partial operators can also be partial operators corresponding to the computation graph itself, without specific limitations here. The second type of splitting strategy can also be called a non-complete Tiling strategy.

[0043] It should be noted that the partitioning strategy set corresponding to the computation graph can be a set of partitioning strategies consisting of one or more first-type partitioning strategies and one or more second-type partitioning strategies. If the partitioning strategy set includes multiple first-type partitioning strategies, for the same operator, the partitioning dimension of the output data of the operator can be different under different first-type partitioning strategies. If the partitioning strategy set includes multiple second-type partitioning strategies, for the same operator, the partitioning dimension of the output data of the operator can be different under different second-type partitioning strategies.

[0044] Optionally, the preset hardware memory constraints represent the limitations imposed by the hardware memory of the hardware deploying the neural network model (e.g., chip 120) on the computation graph splitting. For example, the preset hardware memory constraints could represent the limitations imposed by the physical capacity of the high-speed memory of the hardware deploying the neural network model (e.g., chip 120) on the tile size. High-speed memory includes, but is not limited to, static random-access memory (SRAM), L1 cache (L1M), etc.

[0045] In the embodiments of this disclosure, the processor 210 can determine whether a first type of splitting strategy in the splitting strategy set corresponding to the computation graph satisfies a preset hardware memory limit. If the first type of splitting strategy in the splitting strategy set does not satisfy the preset hardware memory limit, the processor 210 can determine a first target splitting method that satisfies the preset hardware memory limit based on the second type of splitting strategy and the first type of splitting strategy in the splitting strategy set. Optionally, the processor 210 can form several combined splitting strategies by combining different splitting strategies of the second type of splitting strategy and the first type of splitting strategy in the splitting strategy set, and determine the first target splitting method based on the several combined splitting strategies and the preset hardware memory limit. For example, the processor 210 can select a combined splitting strategy that satisfies the preset hardware memory limit from several combined splitting strategies, and determine the first target splitting method based on the selected combined splitting strategy. The first target splitting method is a specific implementation method of computation graph splitting. For example, the first target splitting method can indicate information such as the number of levels of splitting the computation graph, the splitting dimension of the operator at each level, and the number of splits at each level.

[0046] Optionally, the number of levels for splitting the computation graph refers to the hierarchical depth of the split. The number of levels for splitting the computation graph can be two or more. For example, if the number of levels for splitting the computation graph is two, the computation graph can be used as the splitting object first, and several computational subgraphs can be obtained through the first level of splitting. Then, each computational subgraph obtained through the first level of splitting can be used as the splitting object, and several new computational subgraphs can be obtained through the second level of splitting.

[0047] Optionally, the splitting dimension of the operator at each level of splitting refers to the splitting dimension corresponding to the output data of each operator at that level of splitting, or the splitting dimension corresponding to the input data of each operator at that level of splitting. Specifically, the splitting dimension corresponding to the output data of any operator at that level of splitting can be understood as: that level of splitting is used to cut the output data of the operator into several parts along that splitting dimension; the splitting dimension corresponding to the input data of any operator at that level of splitting can be understood as: that level of splitting is used to cut the input data of the operator into several parts along that splitting dimension. For example, if the computation graph is split into two levels, then the splitting dimension of the operator at the first level of splitting refers to the splitting dimension corresponding to the output data (or input data) of each operator in the computation graph, and the splitting dimension of the operator at the second level of splitting refers to the splitting dimension corresponding to the output data (or input data) of each operator in each computation subgraph obtained after the first level of splitting.

[0048] Optionally, the number of partitions at each level refers to the number of parts into which the input or output data of each operator is partitioned when that level of partitioning is performed. For example, if the computation graph is partitioned at two levels, and the number of partitions at the first level is denoted as U1 (U1 can be an integer greater than or equal to 2), and the number of partitions at the second level is denoted as U2 (U2 can be an integer greater than or equal to 2), then through the first level of partitioning, the input and output data of each operator in the computation graph can be partitioned into U1 parts, and through the second level of partitioning, the input and output data of each operator in each computational subgraph obtained after the first level of partitioning can be partitioned into U2 parts.

[0049] According to the information indicated by the first target partitioning method, the processor 210 can partition the computation graph at the corresponding number of levels. The partitioning dimension of the operator and the number of partitions at each level are adapted to this information, thus obtaining multiple computational subgraphs. The processor 210 can also compile each computational subgraph separately, generating binary machine instructions corresponding to each subgraph, and sorting the generated machine instructions to generate a corresponding instruction sequence. This instruction sequence can then serve as the first target instruction sequence for partitioning the computation graph according to the first target partitioning method. Here, when sorting the generated machine instructions, they can be sorted according to the rule of executing machine instructions with data dependencies sequentially, and executing machine instructions without data dependencies concurrently as much as possible, in order to shorten the total execution time.

[0050] It should be noted that in related technologies, a splitting strategy is typically determined for the computation graph. If this splitting strategy does not meet the preset hardware memory limit, the computation graph is usually not split. This results in global, coarse-grained memory access and computational dependencies during the computation process, leading to high access frequency to main memory and low computational efficiency. However, in the embodiments of this disclosure, a set of splitting strategies corresponding to the computation graph can be determined. If the first type of splitting strategy in the splitting strategy set does not meet the preset hardware memory limit, a first target splitting method that meets the preset hardware memory limit can be determined based on the second type of splitting strategy and the first type of splitting strategy in the splitting strategy set. The computation graph is then split using the first target splitting method, so that the split computation graph meets the preset hardware memory limit, thereby achieving the purpose of optimizing the computation graph. Therefore, in the embodiments of this disclosure, the instruction sequence generation device 110 can split or block the computation graph by combining different splitting strategies, so that the computation process conforms to local, fine-grained tile-level dependencies. This allows for full reuse of data in high-speed memory, thereby effectively reducing the frequency of access to main memory by the computation graph during the computation process, reducing the access latency of the computation graph during the computation process, and thus effectively shortening the computation time and improving computation efficiency.

[0051] In some optional examples, for any first-type splitting strategy, the processor 210 determines the first-type splitting strategy in the splitting strategy set, specifically configured to: determine the first splitting dimension corresponding to the output data of the first operator in the computation graph; based on the first position information of the first operator in the computation graph and the dimension mapping relationship between the output data of each operator in the computation graph, propagate the first splitting dimension along the computation graph to determine the first splitting dimension propagation relationship graph corresponding to the computation graph; and determine the first-type splitting strategy based on the first splitting dimension propagation relationship graph.

[0052] Optionally, the first operator can be any operator in the computation graph. The output data of the first operator is multidimensional data, such as a multidimensional tensor, and the first splitting dimension can be any dimension in the output data of the first operator. As an example, the output data of the first operator includes four dimensions, represented as dimension 0, dimension 1, dimension 2, and dimension 3, respectively. Dimension 0 can specifically be the batch processing dimension (N dimension, also known as the batch dimension), dimension 1 can specifically be the height dimension (H dimension), dimension 2 can specifically be the width dimension (W dimension), and dimension 3 can specifically be the channel dimension (C dimension). If the output data of the first operator has a size greater than 1 in the batch processing dimension, width dimension, height dimension, and channel dimension, then for the output data of the first operator, any one of the four dimensions can be determined as the first splitting dimension.

[0053] Optionally, the position information of each operator in the computation graph may include a start position, an intermediate position, and an end position; wherein the start position is the position of the first operator in the computation graph, the end position is the position of the last operator in the computation graph, and the intermediate position is the position of all operators in the computation graph except for the first and last operators. The first position information can characterize the position of the first operator in the computation graph; for example, the first position information can be any one of the start position, intermediate position, and end position. The first position information can be represented by an identifier or other form. For example, it can be in the form of an ID.

[0054] As an example, such as Figure 3-1As shown, the computation graph includes operators S1, S2, S3, and S4 in sequence. The IDs of operators S1, S2, S3, and S4 can be 0, 1, 2, and 3, respectively. If the first operator is operator S1, the first position information is 0, indicating that the first operator's position in the computation graph is the starting position; if the first operator is operator S2, the first position information is 1, indicating that the first operator's position in the computation graph is the middle position; if the first operator is operator S3, the first position information is 2, indicating that the first operator's position in the computation graph is the middle position; and if the first operator is operator S3, the first position information is 3, indicating that the first operator's position in the computation graph is the ending position.

[0055] Optionally, the dimensional mapping relationship between the output data of each operator in the computation graph can be determined based on the operator type. Generally, based on the operator type, the dimensions of the input data and the dimensions of the output data may have the same or different dimensional arrangement orders. For example, the modified linear unit operator is used to operate on each element value in the input data individually. The operation is performed independently between each element position, without involving information interaction across element positions or dimensions, so that the dimensions of the output data and the dimensions of the input data have the same dimensional arrangement order. As another example, the transpose operator is used to exchange the order of two or more specified dimensions in the input data to change the data memory layout, so that the dimensions of the output data and the dimensions of the input data have different dimensional arrangement orders. It can be seen that for any operator, based on the operator type, there is a specific dimensional mapping relationship between the operator's output data and the input data. To further determine the dimensional mapping relationship between the output data of each operator, it can be inferred based on the data dependencies between the operators. For example, for any two adjacent operators in a computation graph, if the later operator depends on the earlier operator and the output data of the earlier operator serves as the input data of the later operator, then based on the dimensional mapping relationship between the output data of the later operator and the input data of the later operator, the dimensional mapping relationship between the output data of the earlier operator and the output data of the later operator can be deduced.

[0056] by Figure 3-1For example, if operator S2 is of type ReLU (i.e., operator S2 is a ReLU operator), the operation logic of the ReLU operator can be: for each element value in the input data, if the element value is greater than 0, leave the element value unchanged; if the element value is less than or equal to 0, update the element value to 0. Since the operation logic of the ReLU operator operates on each element value in the input data individually and does not involve data rearrangement, the batch processing dimensions (height, width, and channel dimensions) of the output data of operator S2 are mapped one-to-one with the batch processing dimensions (height, width, and channel dimensions) of the input data of operator S2. Furthermore, since the output data of operator S1 serves as the input data of operator S2, it can be determined that the batch processing dimensions (height, width, and channel dimensions) of the output data of operator S2 are mapped one-to-one with the batch processing dimensions (height, width, and channel dimensions) of the output data of operator S1.

[0057] Still with Figure 3-1 For example, if operator S2 is of type Transpose (i.e., operator S2 is a Transpose operator), the operation logic of the Transpose operator can be: to swap the order of the width and height dimensions of the input data. Since the operation logic of the Transpose operator involves data rearrangement, the batch processing dimensions, height dimensions, width dimensions, and channel dimensions of the output data of operator S2 are mapped one-to-one with the batch processing dimensions, width dimensions, height dimensions, and channel dimensions of the input data of operator S2. Furthermore, since the output data of operator S1 serves as the input data of operator S2, it can be determined that the batch processing dimensions, height dimensions, width dimensions, and channel dimensions of the output data of operator S2 are mapped one-to-one with the batch processing dimensions, width dimensions, height dimensions, and channel dimensions of the output data of operator S1.

[0058] In other embodiments, the type of operator S2 is Reshape (i.e., operator S2 is a Reshape operator), and the dimensionality mapping relationship between the output data and the input data of operator S2 can be referred to Figure 3-2 That is, the input data of operator S2 includes two dimensions, represented as dimension 0 and dimension 1 respectively, and the output data of operator S2 includes four dimensions, represented as dimension 0, dimension 1, dimension 2, and dimension 3 respectively. Therefore, the dimensions 0 and 1 of the output data of operator S2 are mapped to the dimension 0 of the input data of operator S2, and the dimensions 2 and 3 of the output data of operator S2 are mapped to the dimension 1 of the input data of operator S2. Based on this, the dimensional mapping relationship between the output data of operator S2 and the output data of operator S1 can also be deduced.

[0059] Optionally, the processor 210 can propagate the first split dimension along the computation graph based on the first position information of the first operator in the computation graph and the dimensional mapping relationship between the output data of each operator in the computation graph, to determine the split dimensions corresponding to the output data of other operators in the computation graph besides the first operator, thereby further determining the first split dimension propagation relationship graph corresponding to the computation graph. Specifically, based on the above-mentioned method of determining the dimensional mapping relationship, the position of the first operator can be used as the starting point for split dimension propagation. According to the computational dependencies of the computation graph, the split dimensions corresponding to the output data of other operators in the computation graph besides the first operator can be determined step by step, thereby determining the first split dimension propagation relationship graph. The processor 210 can also determine a first type of splitting strategy based on the first split dimension propagation relationship graph. For example, if the first split dimension propagation relationship graph includes the split dimensions corresponding to the output data of each operator in the computation graph, then a first type of splitting strategy including the split dimensions corresponding to the output data of each operator in the computation graph can be determined.

[0060] In related technologies, when applying Tiling to deep learning model inference, if the computation graph needs to be split, a unified splitting dimension is usually used to represent the entire computation graph split. However, in some cases, it may not be possible to represent the entire computation graph split using a single unified splitting dimension. For example, if the computation graph includes operators for data rearrangement such as Reshape and Transpose operators, then a unified splitting dimension cannot be used to represent the entire computation graph split. In such cases, usually only some operators in the computation graph are split, such as only operators other than those used for data rearrangement. The splitting of the computation graph is not thorough enough, resulting in a still high frequency of access to main memory and failing to effectively improve computational efficiency.

[0061] In view of this, in the embodiments of this disclosure, a first splitting dimension corresponding to the output data of the first operator in the computation graph can be determined. Based on the first position information of the first operator in the computation graph and the dimension mapping relationship between the output data of each operator in the computation graph, the first splitting dimension is propagated along the computation graph to determine the first splitting dimension propagation relationship graph corresponding to the computation graph, which is used to determine the first type of splitting strategy. That is, in the embodiments of this disclosure, a unified splitting dimension is not directly used to represent the splitting of the entire computation graph. Instead, the splitting dimension is propagated based on the dimension mapping relationship between the output data of each operator in the computation graph to determine the splitting dimension of each operator in the computation graph. Since the first target splitting method is determined based on the second type of splitting strategy and the first type of splitting strategy in the splitting strategy set, when splitting the computation graph, each operator in it can be split, and the splitting of the computation graph is more thorough. This avoids the need for frequent access to main memory due to some operators not being split or some operators not being split thoroughly. This effectively reduces the access frequency of the computation graph to main memory during the computation process, reduces the access latency of the computation graph during the computation process, and thus effectively shortens the computation time and improves the computation efficiency.

[0062] It should be noted that the above embodiments describe the case where the first splitting dimension is a dimension in the output data of the first operator. In other embodiments, the first splitting dimension can be a dimension in the input data of the first operator. In this case, the corresponding splitting dimension propagation relationship graph can be determined based on the splitting dimension propagation, and the computation graph can be split on this basis.

[0063] In some optional examples, the processor 210 propagates the first split dimension along the computation graph based on the first position information of the first operator in the computation graph and the dimensional mapping relationship between the output data of each operator in the computation graph. Specifically, it is configured to: in response to the first position information indicating that the first operator belongs to the starting operator, propagate the first split dimension backward along the computation graph based on the dimensional mapping relationship between the output data of each operator in the computation graph; or, in response to the first position information indicating that the first operator belongs to the intermediate operator, propagate the first split dimension forward and backward along the computation graph based on the dimensional mapping relationship between the output data of each operator in the computation graph; or, in response to the first position information indicating that the first operator belongs to the ending operator, propagate the first split dimension forward along the computation graph based on the dimensional mapping relationship between the output data of each operator in the computation graph.

[0064] Optionally, the computation graph may include a start operator, intermediate operators, and an end operator; wherein the start operator is the first operator in the computation graph, the end operator is the last operator in the computation graph, and intermediate operators are operators located between the start operator and the end operator in the computation graph. Figure 3-1 For example, operator S1 can be used as the starting operator, operator S2 and operator S3 can be used as intermediate operators respectively, and operator S4 can be used as the ending operator.

[0065] As discussed above, the first position information can characterize the position of the first operator in the computation graph. If the position represented by the first position information is the starting position, then the first position information can characterize the first operator as the starting operator; if the position represented by the first position information is the middle position, then the first position information can characterize the first operator as the middle operator; if the position represented by the first position information is the ending position, then the first position information can characterize the first operator as the ending operator.

[0066] If the first position information indicates that the first operator belongs to the initial operator (for example, the first operator is...), then... Figure 3-1 If operator S1 is involved, the first splitting dimension is propagated backward along the computation graph, i.e., sequentially to operators S2, S3, and S4. When propagating to operator S2, the dimension corresponding to the output data of operator S2 and the first splitting dimension can be determined according to the dimension mapping relationship between the output data of each operator in the computation graph. This determined dimension can be used as the splitting dimension corresponding to the output data of operator S2. The splitting dimensions corresponding to the output data of operators S3 and S4 are determined sequentially in the same way.

[0067] If the first position information indicates that the first operator belongs to the end operator (for example, the first operator is...), then... Figure 3-1 If operator S4 is involved, the first splitting dimension is forward-propagated along the computation graph, i.e., sequentially propagated to operators S3, S2, and S1. When propagating to operator S3, the dimension corresponding to the output data of operator S3 and the first splitting dimension can be determined according to the dimension mapping relationship between the output data of each operator in the computation graph. This determined dimension can be used as the splitting dimension corresponding to the output data of operator S3. The splitting dimensions corresponding to the output data of operators S2 and S1 are determined sequentially in the same way.

[0068] If the first position information indicates that the first operator belongs to the intermediate operator (for example, the first operator is...), then... Figure 3-1 If operator S2 is used, then the first split dimension is propagated forward and backward along the computation graph, that is, forward propagation to operator S1, and backward propagation to operator S3 and operator S4 in sequence. Following the same method, the split dimension corresponding to the output data of operator S1 is determined, and the split dimensions corresponding to the output data of operator S3 and operator S4 are determined in sequence.

[0069] After determining the split dimensions corresponding to the output data of all operators except the first operator in the computation graph, a first split dimension propagation graph can be constructed based on the split dimensions corresponding to the output data of the other operators and the first split dimension corresponding to the output data of the first operator. The first split dimension propagation graph can include the split dimensions corresponding to the output data of each operator in the computation graph. Assume the first operator is... Figure 3-1 In the example, operator S4's output data corresponds to the width dimension in its splitting dimension. Based on the forward propagation of the splitting dimension, the splitting dimensions corresponding to operators S3, S2, and S1 are sequentially determined to be the height dimension, channel dimension, and width dimension, respectively. This allows the construction of... Figure 3-3 The diagram shows the transmission relationship of the first split dimension.

[0070] In the embodiments of this disclosure, a suitable propagation direction can be determined based on the first position information of the first operator in the computation graph. Specifically, if the first position information indicates that the first operator belongs to the starting operator, the determined propagation direction can be backward; if the first position information indicates that the first operator belongs to the intermediate operator, the determined propagation direction can be forward and backward; if the first position information indicates that the first operator belongs to the ending operator, the determined propagation direction can be forward. Taking the position of the first operator as the starting point of the split dimension propagation, based on the dimension mapping relationship between the output data of each operator in the computation graph, the split dimension propagation is performed according to the determined propagation direction. This allows the split dimensions corresponding to the output data of other operators in the computation graph besides the first operator to be determined. Based on this, a first split dimension propagation relationship graph can be constructed. Since the first split dimension transmission relationship graph includes the split dimensions corresponding to each operator in the computation graph, each operator in the computation graph can be split based on the first split dimension transmission relationship graph, making the split of the computation graph more thorough. This can effectively reduce the frequency of access to main memory by the computation graph during the computation process, reduce the access latency of the computation graph during the computation process, and thus effectively shorten the computation time and improve the computation efficiency.

[0071] It should be noted that the above description describes the construction of the first split-dimensional transmission relationship graph through split-dimensional transmission. In a practical implementation, user input operations can also be used to specify the split-dimensional corresponding to the output data of each operator in the computation graph, thereby determining the first split-dimensional transmission relationship graph that includes the split-dimensional corresponding to the output data of each operator in the computation graph. Here, user input operations include, but are not limited to, voice input operations, touch input operations, and keyboard input operations.

[0072] In some optional examples, for any second-type splitting strategy, the processor 210 determines the second-type splitting strategy in the splitting strategy set, specifically configured as follows: Based on the dependencies of each operator in the computation graph, a second operator is determined; wherein the input data of the second operator is the output data of at least two third operators in the computation graph; based on the second position information of the second operator in the computation graph, the computation graph is split to obtain multiple first computation subgraphs; for each first computation subgraph, a second splitting dimension corresponding to the output data of the fourth operator in the first computation subgraph is determined; based on the third position information of the fourth operator in the first computation subgraph and the dimension mapping relationship between the output data of each operator in the first computation subgraph, the second splitting dimension is propagated along the first computation subgraph to determine the second splitting dimension propagation relationship graph corresponding to the first computation subgraph; based on the second splitting dimension propagation relationship graph corresponding to each first computation subgraph, the second-type splitting strategy is determined.

[0073] As described above, computational graphs can represent computational dependencies through edges. Based on the computational dependencies represented by the edges of the computational graph, a second operator can be determined. The second operator satisfies the following conditions: the input data of the second operator is the output data of at least two third operators in the computational graph (i.e., the second operator depends on at least two third operators), and the second operator begins computation only after at least one of the third operators has calculated its complete output data. For example, if the second operator is a convolution operator, then the second operator depends on two third operators, one of which outputs a weight tensor, and the other outputs a feature tensor. The second operator only begins computation after the other third operator has calculated its complete feature tensor (i.e., computation can begin without requiring the complete weight tensor to be calculated).

[0074] Optionally, the second position information can characterize the position of the second operator in the computation graph, and the second position information can be in the form of an ID, for example. Based on the second position information, the processor 210 can divide the computation graph into multiple segments, where each segment can serve as a first computation subgraph.

[0075] As an example, such as Figure 4As shown, the computation graph includes operators R1, R2, R3, R4, R5, and R6. Operator R5 depends on operators R3 and R4, and is used to begin computation after operator R3 has obtained complete output data. Therefore, operators R3 and R4 can each be considered as a third operator, and operator R5 as a second operator. Based on the second position information of operator R5 in the computation graph, the computation graph can be divided into two segments: one segment includes operators R1, R2, and R3, and the other segment includes operators R4, R5, and R6. These two segments can each be considered as a first computation subgraph. For the first computation subgraph including operators R1, R2, and R3, the processor 210 can construct a corresponding second split dimension transmission relationship graph according to the first split dimension transmission relationship graph construction method described above. For the first computational subgraph including operators R4, R5, and R6, processor 210 can also construct a corresponding second splitting dimension transmission graph according to the first splitting dimension transmission graph construction method described above. In this way, processor 210 can construct two second splitting dimension transmission graphs to determine the second type of splitting strategy. For example, each of the two second splitting dimension transmission graphs can include the splitting dimensions corresponding to the output data of each operator in the corresponding first computational subgraph, thus determining the second type of splitting strategy including the splitting dimensions corresponding to the output data of each operator in each of the two first computational subgraphs.

[0076] It should be noted that under the first type of splitting strategy, a splitting dimension is assigned to the output data of an operator in the computation graph, and the splitting dimensions corresponding to the output data of other operators in the computation graph are determined based on the propagation of the splitting dimension along the computation graph. In contrast, under the second type of splitting strategy, the computation graph is divided into multiple sub-segments, and for each sub-segment, a splitting dimension is assigned to the output data of an operator. Based on the propagation of the splitting dimension along the sub-segment, the splitting dimensions corresponding to the output data of other operators in that sub-segment are determined. That is, the propagation of splitting dimensions can be interrupted at some operators, and the splitting dimension can be reassigned and propagated at the interrupted operator location. Thus, based on the first type of splitting strategy, if the split computation graph cannot meet the preset hardware memory limits, the second type of splitting strategy can be used to further split the computation graph to meet the preset hardware memory limits, thereby optimizing the computation graph more effectively. Furthermore, this can more effectively reduce the frequency of main memory access during computation, reduce access latency, and thus more effectively shorten computation time and improve computational efficiency.

[0077] In some optional examples, the splitting strategy set includes multiple second-type splitting strategies and multiple first-type splitting strategies. The processor 210 determines the first target splitting method corresponding to the computation graph that meets the preset hardware memory limit based on the second-type splitting strategies and the first-type splitting strategies in the splitting strategy set. Specifically, it is configured to: combine one or more second-type splitting strategies with one or more first-type splitting strategies to obtain one or more first combined splitting strategies; and in response to any first combined splitting strategy meeting the preset hardware memory limit, use the first combined splitting strategy as the first target splitting method.

[0078] It should be noted that using a single splitting strategy may not completely split the computation graph, and the resulting graph may not meet the preset hardware memory limitations. Therefore, the processor 210 can combine different splitting strategies. Specifically, the processor 210 can combine one or more second-type splitting strategies with one or more first-type splitting strategies to obtain one or more first-combination splitting strategies. For example, the processor 210 can combine one second-type splitting strategy with one first-type splitting strategy to obtain a first-combination splitting strategy. As another example, the processor 210 can combine one second-type splitting strategy with each of the multiple first-type splitting strategies to obtain multiple first-combination splitting strategies. Yet another example, the processor 210 can combine multiple second-type splitting strategies with each of the multiple first-type splitting strategies to obtain multiple first-combination splitting strategies. These are just some examples of combinations; in actual implementation, the processor 210 can determine all combinations through brute-force search.

[0079] In some optional embodiments of this disclosure, the processor 210 combines one or more second-type splitting strategies with one or more first-type splitting strategies to obtain one or more first-combined splitting strategies. Specifically, it is configured to: sequentially arrange one or more first-type splitting strategies and one or more second-type splitting strategies to obtain one or more splitting strategy sequences, the splitting strategy sequence including splitting strategies that first split the computation graph according to the first-type splitting strategy and then split the computation graph according to the second-type splitting strategy; and determine one or more splitting strategy sequences as one or more first-combined splitting strategies.

[0080] Taking the case of combining a second-type splitting strategy with a first-type splitting strategy to obtain a first-combined splitting strategy as an example, assuming that the first-type splitting strategy is represented by C1 and the second-type splitting strategy is represented by C2, then C1 and C2 can be arranged in order to obtain the splitting strategy sequence L1, and L1 can be used as a first-combined splitting strategy.

[0081] Taking the example of combining a second-type splitting strategy with each of several first-type splitting strategies to obtain multiple first-combination splitting strategies, let's assume the multiple first-type splitting strategies are three strategies, denoted as D1, D2, and D3, and the second-type splitting strategy is denoted as D4. Then, we can arrange D1 and D4 sequentially to obtain the splitting strategy sequence L2, which can be used as one first-combination splitting strategy. Similarly, we can arrange D2 and D4 sequentially to obtain the splitting strategy sequence L3, which can be used as another first-combination splitting strategy. Likewise, we can arrange D3 and D4 sequentially to obtain the splitting strategy sequence L4, which can be used as yet another first-combination splitting strategy. That is, in this example, a total of three first-combination splitting strategies can be obtained.

[0082] Thus, by arranging different splitting strategies in sequence, it is possible to combine different splitting strategies to obtain the corresponding first combined splitting strategy.

[0083] In response to obtaining one or more first combination splitting strategies, the processor 210 can determine whether each first combination splitting strategy satisfies a preset hardware memory limit. If only one first combination splitting strategy satisfies the preset hardware memory limit, this first combination splitting strategy can be used as the first target splitting method. If multiple first combination splitting strategies satisfy the preset hardware memory limit, one first combination splitting strategy can be selected as the first target splitting method from among them. This selection can be random, or it can be achieved by using an algorithm to estimate the instruction sequence execution performance under each of the multiple first combination splitting strategies satisfying the preset hardware memory limit, and then selecting the first combination splitting strategy with the best corresponding execution performance.

[0084] In the embodiments of this disclosure, a first combination splitting strategy that meets the preset hardware memory limit conditions can be searched by combining different splitting strategies as the first target splitting method. This is beneficial to split the computation graph more thoroughly by combining different splitting strategies, thereby more effectively reducing the access frequency of the computation graph to main memory during the computation process, reducing the access latency of the computation graph during the computation process, and thus more effectively shortening the computation time and improving the computation efficiency.

[0085] In some optional examples, the first target splitting method includes a first type of splitting strategy, a first target splitting number corresponding to the first type of splitting strategy, a second type of splitting strategy, and a second target splitting number corresponding to the second type of splitting strategy. The target splitting number refers to the number of parts into which the input data or output data of the operator is split. The processor 210 generates a first target instruction sequence to split the computation graph according to the first target splitting method, specifically configured as follows: splitting the computation graph according to the first type of splitting strategy and the first target splitting number to obtain multiple second computation subgraphs; splitting each second computation subgraph according to the second type of splitting strategy and the second target splitting number to obtain a third computation subgraph; determining the fourth position information in the third computation subgraph that satisfies the preset aggregation condition based on the type of each operator in each third computation subgraph; inserting aggregation operators into each third computation subgraph according to the fourth position information to obtain the fourth computation subgraph corresponding to the second computation subgraph; and generating the first target instruction sequence based on each fourth computation subgraph.

[0086] Optionally, after determining the first type of splitting strategy, the processor 210 may determine at least one reference number of splits for the first type of splitting strategy, and determine the first target number of splits from the at least one reference number of splits. As described above, the first type of splitting strategy may include the splitting dimensions corresponding to the output data of each operator in the computation graph. Then, the processor 210 may determine the minimum data size from the data sizes corresponding to each splitting dimension in the first type of splitting strategy; and determine at least one reference number of splits based on the minimum data size.

[0087] In some embodiments, the computation graph is as follows Figure 3-1 As shown, in the first type of splitting strategy, the splitting dimension corresponding to the output data of operators S1, S2, S3, and S4 is the batch size dimension. Assuming that the output data of operators S1, S2, S3, and S4 all have a data size of 3 in the batch size dimension, then when determining the minimum data size from these data sizes, the determined minimum data size is 3. The processor 210 can determine multiple divisors of the minimum data size and, based on the divisors other than 1, determine at least one reference splitting number. Here, the minimum data size is 3, and the divisors of 3 are only 1 and 3, so only 3 can be determined as the reference splitting number, that is, only one reference splitting number is determined.

[0088] In other embodiments, the computation graph is as follows: Figure 3-1As shown, in the first type of splitting strategy, the splitting dimension corresponding to the output data of operators S1, S2, S3, and S4 is the channel dimension. Assuming that the data size of the output data of operators S1, S2, S3, and S4 in the channel dimension is 12, then the minimum data size determined from these data sizes is 12. The processor 210 can determine multiple divisors of the minimum data size and, based on divisors other than 1, determine at least one reference splitting number. Here, the minimum data size is 12, and the divisors of 12 are 1, 2, 3, 4, 6, and 12. Therefore, 2, 3, 4, 6, and 12 can each be determined as a reference splitting number, thus determining five reference splitting numbers.

[0089] If there is only one type of reference split, the processor 210 can determine this reference split as the first target split. If there are multiple types of reference splits, the processor 210 can select one of the multiple reference splits as the first target split. The selection method can be random selection, or it can be that an algorithm is used to predict the execution performance of the instruction sequence under various reference splits and select the reference split with the best corresponding execution performance as the first target split.

[0090] Similarly, after determining the second type of splitting strategy, the processor 210 can determine at least one reference number of splits for the second type of splitting strategy, and determine the second target number of splits from the at least one reference number of splits. The specific method for determining the second target number of splits is the same as the description of the specific method for determining the first target number of splits above, and will not be repeated here. In this way, the processor 210 can determine a first target splitting method that includes the first type of splitting strategy, the first target number of splits, the second type of splitting strategy, and the second target number of splits.

[0091] In embodiments of this disclosure, the processor 210 can first split the computation graph according to a first type of splitting strategy and a first target number of splits to obtain multiple second computational subgraphs. Assuming the first target number of splits is represented by P, the computation graph can be split into P second computational subgraphs according to the splitting dimensions in the first type of splitting strategy. The operator structure of each second computational subgraph is consistent with that of the computation graph; that is, each second computational subgraph and the computation graph have the same graph structure, and the types of operators at corresponding positions in each second computational subgraph and the computation graph are the same. Furthermore, the data size of the output data of each operator in each second computational subgraph is smaller than the data size of the output data of the operator at the corresponding position in the computation graph. For example, as... Figure 5As shown in the left-hand side, the computation graph includes the element-wise operators Eltwise1, Eltwise2, and Conv. The output data of Eltwise1, Eltwise2, and Conv all have a size of 3 in the batch dimension. In the first type of splitting strategy, the splitting dimension corresponding to the output data of Eltwise1, Eltwise2, and Conv is the batch dimension. The first target number of splits P is 3. Therefore, by... Figure 5 In the computation graph on the left, the output data of each operator is split in the batch processing dimension, and the data in the batch processing dimension is split into three parts, so that the output data of each operator in the three split second sub-computation graphs has a data size of 1 in the batch processing dimension. Each second sub-computation graph is as follows. Figure 5 As shown in the right-hand side.

[0092] Then, the processor 210 can split each second computational subgraph according to the second type of splitting strategy and the second target number of splits to obtain a third computational subgraph; wherein, there can be a one-to-one correspondence between the third computational subgraph and the second computational subgraph. Assuming that the value of the second target number of splits is represented by Q, then according to each splitting dimension in the second type of splitting strategy, each operator in the second computational subgraph can be split into Q operators to obtain a third computational subgraph including the Q operators split from each operator in the second computational subgraph. The processor 210 can also determine the fourth position information in the third computational subgraph that satisfies the preset aggregation conditions based on the type of each operator in each third computational subgraph. For example, a second operator can be determined in the third computational subgraph. Assuming that the second operator is used to start calculation after the complete output data is calculated by one of the at least two third operators it depends on (hereinafter referred to as the target third operator), then the information representing the position between the second operator and the target third operator can be determined as the fourth position information in the third computational subgraph that satisfies the preset aggregation conditions. According to the fourth position information, the processor 210 can insert an aggregation operator between the second operator and the target third operator in the third computational subgraph. Furthermore, the processor 210 can also determine the information representing the position after the end operator in the third computational subgraph as the fourth position information. Thus, the processor 210 can also insert an aggregation operator after the end operator of the third computational subgraph to obtain the fourth computational subgraph. That is, the difference between the fourth and third computational subgraphs is that the fourth computational subgraph adds some aggregation operators.

[0093] In some embodiments, if the second target number of partitions Q is 2, then the partitions can be divided according to the partitioning dimensions in the second type of partitioning strategy. Figure 5 The second computational subgraph shown in the diagram is split into subgraphs such that... Figure 5In the second computational subgraph illustrated, Eltwise1, Eltwise2, and Conv are each split into two operators to obtain a third computational subgraph containing two Eltwise1s, two Eltwise2s, and two Convs. Furthermore, for the third computational subgraph, based on the type of each operator, Conv can be identified as the second operator, Eltwise1 as the target third operator, and the information representing the position between Conv and Eltwise1 can be identified as the fourth positional information, as can the information representing the position following Conv. Therefore, an aggregation operator compose1 can be inserted between Eltwise1 and Conv in the third computational subgraph. Additionally, an aggregation operator compose2 can be inserted after Conv in the third computational subgraph to update the third computational subgraph into a fourth computational subgraph. The fourth computational subgraph can be found in [reference needed]. Figure 6-1 .

[0094] Referring to the example above, P second computational subgraphs can be three second computational subgraphs. For each of these second computational subgraphs, the corresponding fourth computational subgraph can be obtained using the method described above. Thus, a total of three fourth computational subgraphs can be obtained. For example, see [link to example]. Figure 6-2 The diagram illustrates three fourth computation subgraphs. The processor 210 can then compile each fourth computation subgraph to generate corresponding binary machine instructions, and sort these machine instructions to generate a first target instruction sequence. Here, when sorting the generated machine instructions, the sorting can follow the rule of executing machine instructions with data dependencies sequentially, and executing machine instructions without data dependencies concurrently as much as possible.

[0095] In the embodiments of this disclosure, based on the first target splitting method, multiple fourth computational subgraphs can be obtained through two-level splitting and the insertion operation of aggregation operators, and a first target instruction sequence can be generated based on each fourth computational subgraph. The first-level splitting can be performed according to a first type of splitting strategy and a first target splitting number, while the second-level splitting can be performed according to a second type of splitting strategy and a second target splitting number, splitting the second computational subgraph obtained from the first-level splitting. This allows for a more thorough splitting of the computational graph, ensuring that the split computational graph meets preset hardware memory limitations, and more effectively reducing the frequency of main memory access during computation, thus reducing access latency and shortening computation time, thereby improving computational efficiency. Furthermore, through the insertion operation of the aggregation operator, the aggregation operation at the aggregation operator (such as the aggregation operation at compose1) can ensure that the complete input data required by the second operator (i.e. the complete output data of the target third operator) is obtained and provided to the second operator for calculation, so as to conform to the calculation characteristics of the second operator and enable the second operator to perform the calculation correctly. It can also ensure the equivalence between the final output results obtained before and after the split through the aggregation operation at the aggregation operator (such as the aggregation operation at compose2).

[0096] It should be noted that the embodiments of this disclosure can employ a multi-level splitting scheme. In this case, the number of splitting levels is not limited to two levels; for example, it can be more than two levels. In some embodiments, in the first target splitting method, the number of first-type splitting strategies and the number of first-target splitting parts are both two, and the two first-target splitting parts correspond one-to-one with the two first-type splitting strategies. Therefore, the computation graph can be first split according to one of the first-type splitting strategies and the corresponding first-target splitting parts to obtain multiple computation subgraphs. Then, according to another first-type splitting strategy and the corresponding first-target splitting parts, the computation subgraphs obtained from the first-level splitting are split into a second-level splitting. Afterward, according to the second-type splitting strategy and the second-target splitting parts, the computation subgraphs obtained from the second-level splitting are split into a third-level splitting. The case with three or more splitting levels follows the same logic and will not be elaborated further here. When using a multi-level splitting scheme, multiple computation subgraphs can be obtained after the last level of splitting. Then, each computation subgraph obtained after the last level of splitting can be compiled to obtain corresponding binary machine instructions. By sorting these machine instructions, a first target instruction sequence can be formed. By using a multi-level splitting scheme, the computation graph can be split more thoroughly, thereby ensuring that the split computation graph meets the preset hardware memory limits.

[0097] In some optional examples, the splitting strategy set includes multiple first-type splitting strategies and multiple second-type splitting strategies; In response to the first type of splitting strategy in the splitting strategy set not meeting the preset hardware memory limit, the processor 210 is specifically configured to: in response to the absence of a splitting strategy that meets the preset hardware memory limit among the multiple first type of splitting strategies in the splitting strategy set, combine at least two first type of splitting strategies in the splitting strategy set to obtain multiple second combined splitting strategies; in response to the absence of a second combined splitting strategy that meets the preset hardware memory limit among the multiple second combined splitting strategies, determine that the first type of splitting strategy in the splitting strategy set does not meet the preset hardware memory limit.

[0098] In the embodiments of this disclosure, when determining whether a first type of splitting strategy in the splitting strategy set satisfies a preset hardware memory limit, the processor 210 can first individually determine whether each type of first-class splitting strategy in the splitting strategy set satisfies the preset hardware memory limit. If none of the first-class splitting strategies in the splitting strategy set satisfy the preset hardware memory limit, then different types of first-class splitting strategies in the splitting strategy set can be combined. For example, referring to the method for obtaining the first combined splitting strategy described above, different types of first-class splitting strategies in the splitting strategy set can be arranged sequentially to obtain a splitting strategy sequence, and the obtained splitting strategy sequence can be determined as the second combined splitting strategy. Optionally, for different types of first-class splitting strategies in the splitting strategy set, the processor 210 can use brute force search to determine various combination cases to obtain multiple second combined splitting strategies. The processor 210 can determine whether there is a second combined splitting strategy among the multiple second combined splitting strategies that satisfies the preset hardware memory limit. If none of the multiple second-combination splitting strategies satisfies the preset hardware memory limit, then no matter how the different first-type splitting strategies in the splitting strategy set are combined, a splitting strategy that satisfies the preset hardware memory limit cannot be obtained. Therefore, it can be determined that the first-type splitting strategies in the splitting strategy set do not satisfy the preset hardware memory limit. In this case, the processor 210 then determines the first target splitting method that satisfies the preset hardware memory limit based on the first-type and second-type splitting strategies in the splitting strategy set.

[0099] As can be seen, in the embodiments of this disclosure, a first type of splitting strategy or a combination of different first type splitting strategies is preferentially used to split the computation graph. If the splitting cannot meet the preset hardware memory limit, a second type of splitting strategy is introduced to further split the computation graph in combination with the second type of splitting strategy to meet the preset hardware memory limit, thereby optimizing the computation graph more thoroughly. On this basis, the frequency of access to main memory by the computation graph during the computation process can be reduced more effectively, the access latency of the computation graph during the computation process can be reduced, and the computation time can be shortened more effectively, thereby improving the computation efficiency.

[0100] In some optional examples, processor 210 is also configured as follows: In response to at least one first type of splitting strategy in the splitting strategy set satisfying a preset hardware memory limit, the first type of splitting strategy is determined as the second target splitting method corresponding to the computation graph that satisfies the preset hardware memory limit; or, in response to at least one second combined splitting strategy satisfying the preset hardware memory limit, the second combined splitting strategy is determined as the second target splitting method corresponding to the computation graph that satisfies the preset hardware memory limit. Generate a second objective instruction sequence that splits the computation graph according to the second objective splitting method.

[0101] In the embodiments of this disclosure, if only one first-type splitting strategy in the splitting strategy set satisfies the preset hardware memory limit, then this first-type splitting strategy can be determined as the second target splitting method. If multiple first-type splitting strategies in the splitting strategy set satisfy the preset hardware memory limit, then one first-type splitting strategy can be selected as the second target splitting method from among the multiple first-type splitting strategies that satisfy the preset hardware memory limit. Here, the selection method can be random selection, or it can be that the instruction sequence execution performance under the multiple first-type splitting strategies that satisfy the preset hardware memory limit is estimated by an algorithm, and the first-type splitting strategy with the best corresponding execution performance is selected.

[0102] Alternatively, if any of the first-type splitting strategies in the splitting strategy set cannot meet the preset hardware memory limit, different first-type splitting strategies can be combined to form multiple second-type splitting strategies. If a set of second-type splitting strategies that meets the preset hardware memory limit is obtained by combining different first-type splitting strategies, this second-type splitting strategy can be determined as the second target splitting method. If multiple sets of second-type splitting strategies that meet the preset hardware memory limit are obtained by combining different first-type splitting strategies, one set of second-type splitting strategies that meets the preset hardware memory limit can be selected as the second target splitting method. The selection method here is the same as described in the previous paragraph and will not be repeated here.

[0103] After determining the second target splitting method, the processor 210 can split the computation graph according to the second target splitting method to obtain multiple computation subgraphs. The processor 210 can compile each of the multiple computation subgraphs to generate binary machine instructions corresponding to each of the multiple computation subgraphs. The processor 210 can also sort the machine instructions corresponding to each of the multiple computation subgraphs to generate a corresponding instruction sequence, which can then be used as the second target instruction sequence.

[0104] As can be seen, the embodiments of this disclosure can employ a first-type splitting strategy or a combination of different first-type splitting strategies to split the computation graph. This can effectively optimize the computation graph, thereby effectively reducing the frequency of main memory access during computation and reducing access latency, thus effectively shortening computation time and improving computation efficiency.

[0105] In some optional examples, processor 210 determines that the first type of splitting strategy meets preset hardware memory limits, specifically configured as follows: In response to the fact that, under the first type of splitting strategy, in each computational subgraph obtained after splitting the computational graph, the sum of the input data and output data of each operator is less than or equal to the available memory space capacity corresponding to that operator, it is determined that the first type of splitting strategy satisfies the preset hardware memory limit condition.

[0106] Optionally, chip 120 may include several computing units, such as, but not limited to, tensor computing units, vector computing units, and vector computation units. Chip 120 may include high-speed memory corresponding to each computing unit. The available memory space capacity corresponding to any operator can be understood as: the total capacity of the high-speed memory corresponding to the computing units used to run that operator.

[0107] As described above, after determining the first type of splitting strategy, the processor 210 can determine one or more reference splitting numbers for the first type of splitting strategy.

[0108] If only one reference number of partitions is determined here, combining this reference number of partitions with the first type of partitioning strategy can yield a partitioning method. Corresponding to this partitioning method, the processor 210 can determine the sum of the input and output data of each operator in each computational subgraph obtained after partitioning. If the sum of the data corresponding to each operator is less than or equal to the available memory space capacity corresponding to that operator, then the input and output data of that operator can be completely stored in the corresponding high-speed memory. Therefore, it can be determined that the first type of partitioning strategy satisfies the preset hardware memory limit conditions, and this reference number of partitions can also be determined as the first target number of partitions corresponding to the first type of partitioning strategy mentioned above.

[0109] If multiple reference splitting numbers are determined here, combining each of these reference splitting numbers with the first type of splitting strategy can yield multiple splitting methods. Corresponding to each of these splitting methods, the processor 210 can determine the sum of the input and output data of each operator in each computational subgraph obtained after splitting. If, under at least one of the multiple splitting methods, the sum of the data corresponding to each operator is less than or equal to the available memory space capacity corresponding to that operator, then the input and output data of that operator can be completely stored in the corresponding high-speed memory. Therefore, it can be determined that the first type of splitting strategy satisfies the preset hardware memory limit condition, and a reference splitting number can be selected as the first target splitting number from at least one reference splitting number corresponding one-to-one with the at least one splitting method.

[0110] In the embodiments of this disclosure, by comparing the sum of the input and output data of each operator in each subgraph of the computation graph obtained after splitting the computation graph with the available memory space capacity corresponding to that operator, it is possible to efficiently and reliably determine whether the first type of splitting strategy meets the preset hardware memory limit. A similar method can be used to determine whether the second type of splitting strategy, the first combined splitting strategy, and the second combined splitting strategy meet the preset hardware memory limit, and this disclosure will not elaborate further. This helps ensure that the split computation graph meets the preset hardware memory limit, thereby effectively optimizing the computation graph, effectively reducing the frequency of main memory access during computation, reducing access latency during computation, and thus effectively shortening computation time and improving computational efficiency.

[0111] In some embodiments, a high-speed memory may be shared by two or more computing units. In this case, other methods can be used to determine whether the first type of partitioning strategy meets the preset hardware memory limit. For example, if, when partitioning the computation graph using the first type of partitioning strategy, operator 1 in one computation subgraph and operator 2 in another computation subgraph are executed concurrently, the sum of the input and output data of operator 1 is represented as the first data volume, and the sum of the input and output data of operator 2 is represented as the second data volume. Furthermore, the computation graphs corresponding to operators 1 and 2 share the same high-speed memory. If the sum of the first and second data volumes is greater than the total capacity of the same high-speed memory, it can be determined that the first type of partitioning strategy does not meet the preset hardware memory limit; if the sum of the first and second data volumes is less than or equal to the total capacity of the same high-speed memory, it can be determined that the first type of partitioning strategy meets the preset hardware memory limit.

[0112] In some optional examples, processor 210 can determine the set of splitting strategies corresponding to the computation graph. Next, processor 210 can determine whether each of the various first-type splitting strategies included in the splitting strategy set satisfies a preset hardware memory limit. If a first-type splitting strategy that satisfies the preset hardware memory limit exists in the splitting strategy set, the computation graph is split according to that first-type splitting strategy, and a corresponding instruction sequence is obtained. If no first-type splitting strategy that satisfies the preset hardware memory limit exists in the splitting strategy set, different first-type splitting strategies in the splitting strategy set are combined to obtain one or more second-combination splitting strategies. If one or more second-combination splitting strategies contain a second-combination splitting strategy that satisfies the preset hardware memory limit, the computation graph is split according to that second-combination splitting strategy, and a corresponding instruction sequence is obtained. If no second-combination splitting strategy that satisfies the preset hardware memory limit exists in one or more second-type splitting strategies, any one or more second-type splitting strategies in the splitting strategy set are combined with any one or more first-type splitting strategies to obtain one or more first-combination splitting strategies. If one or more of the first combination splitting strategies meet the preset hardware memory limit, the computation graph is split according to that first combination splitting strategy, and the corresponding instruction sequence is obtained accordingly. If no first combination splitting strategy meets the preset hardware memory limit, the computation graph is not split.

[0113] Exemplary methods Figure 7 This is a flowchart illustrating an instruction sequence generation method provided by some exemplary embodiments of this disclosure. The instruction sequence generation method can be applied to the instruction sequence generation apparatus in the above embodiments. Figure 7 As shown, the instruction sequence generation method includes: Step 710, Obtain the computation graph; Step 720: Determine the set of splitting strategies corresponding to the computation graph; Step 730: In response to the fact that the first type of splitting strategy in the splitting strategy set does not meet the preset hardware memory limit, based on the second type of splitting strategy and the first type of splitting strategy in the splitting strategy set, determine the first target splitting method that meets the preset hardware memory limit for the computation graph; wherein, the first type of splitting strategy is used to split the entire computation graph, and the second type of splitting strategy is used to split a part of the computation graph. Step 740: Generate a first target instruction sequence to split the computation graph according to the first target splitting method.

[0114] In some optional examples, such as Figure 8-1 As shown, step 720 includes: Step 810: Determine the first split dimension corresponding to the output data of the first operator in the computation graph; Step 820: Based on the first position information of the first operator in the computation graph and the dimension mapping relationship between the output data of each operator in the computation graph, the first split dimension is propagated along the computation graph to determine the first split dimension propagation relationship graph corresponding to the computation graph; Step 830: Determine the first type of splitting strategy based on the first splitting dimension transmission relationship graph.

[0115] In some optional examples, step 820 includes: In response to the first position information indicating that the first operator belongs to the starting operator, based on the dimensional mapping relationship between the output data of each operator in the computation graph, the first splitting dimension is backward propagated along the computation graph; or, In response to the first position information indicating that the first operator belongs to an intermediate operator, based on the dimensional mapping relationship between the output data of each operator in the computation graph, the first split dimension is forward-propagated and backward-propagated along the computation graph; or, In response to the first position information indicating that the first operator belongs to the end operator, the first split dimension is forward-propagated along the computation graph based on the dimensional mapping relationship between the output data of each operator in the computation graph.

[0116] In some optional examples, such as Figure 8-2 As shown, step 720 includes: Step 840: Determine the second operator based on the dependencies between operators in the computation graph; wherein the input data of the second operator is the output data of at least two third operators in the computation graph; Step 850: Based on the second position information of the second operator in the computation graph, the computation graph is split to obtain multiple first computation subgraphs; Step 860: For each first computational subgraph, determine the second splitting dimension corresponding to the output data of the fourth operator in the first computational subgraph; based on the third position information of the fourth operator in the first computational subgraph and the dimension mapping relationship between the output data of each operator in the first computational subgraph, propagate the second splitting dimension along the first computational subgraph to determine the second splitting dimension propagation relationship graph corresponding to the first computational subgraph. Step 870: Determine the second type of splitting strategy based on the second splitting dimension transmission relationship graph corresponding to each first computational subgraph.

[0117] In some optional examples, the splitting strategy set includes multiple second-class splitting strategies and multiple first-class splitting strategies, such as Figure 9 As shown, step 730 includes: Step 910: Combine one or more second-type splitting strategies with one or more first-type splitting strategies to obtain one or more first-combined splitting strategies; Step 920: In response to any of the first combination splitting strategies satisfying the preset hardware memory limit conditions, determine the first combination splitting strategy as the first target splitting method.

[0118] In some optional examples, step 910 includes: One or more first-type splitting strategies and one or more second-type splitting strategies are arranged in sequence to obtain one or more splitting strategy sequences. The splitting strategy sequence includes splitting strategies that first split the computation graph according to the first-type splitting strategy and then split the computation graph according to the second-type splitting strategy. One or more splitting strategy sequences are determined as one or more first combined splitting strategies.

[0119] In some optional examples, the first target splitting method includes a first type of splitting strategy, a first target splitting number corresponding to the first type of splitting strategy, a second type of splitting strategy, and a second target splitting number corresponding to the second type of splitting strategy. The target splitting number refers to the number of parts into which the input data or output data of the operator is split. like Figure 10 As shown, step 740 includes: Step 1010: According to the first type of splitting strategy and the first target number of splits, the computation graph is split to obtain multiple second computation subgraphs; Step 1020: According to the second type of splitting strategy and the second target splitting number, split each second computational subgraph to obtain the third computational subgraph; Step 1030: Based on the type of each operator in each third computational subgraph, determine the fourth position information in the third computational subgraph that satisfies the preset aggregation conditions; Step 1040: According to the fourth position information, insert the aggregation operator into each third computation subgraph to obtain the fourth computation subgraph corresponding to the second computation subgraph; Step 1050: Generate the first target instruction sequence based on each of the fourth computational subgraphs.

[0120] In some optional examples, the splitting strategy set includes multiple first-type splitting strategies and multiple second-type splitting strategies; like Figure 11 As shown, step 730 includes: Step 1110: In response to the fact that there is no splitting strategy that satisfies the preset hardware memory limit among the multiple first-type splitting strategies in the splitting strategy set, at least two first-type splitting strategies in the splitting strategy set are combined to obtain multiple second-combined splitting strategies. Step 1120: In response to the absence of a splitting strategy that satisfies the preset hardware memory limit among the various second combination splitting strategies, determine that the first type of splitting strategy in the splitting strategy set does not satisfy the preset hardware memory limit.

[0121] In some optional examples, such as Figure 12 As shown, the instruction sequence generation method also includes: Step 1210: In response to at least one first type of splitting strategy in the splitting strategy set satisfying the preset hardware memory limit, determine the first type of splitting strategy as the second target splitting method corresponding to the computation graph that satisfies the preset hardware memory limit; or, in response to at least one second combined splitting strategy satisfying the preset hardware memory limit, determine the second combined splitting strategy as the second target splitting method corresponding to the computation graph that satisfies the preset hardware memory limit. Step 1220: Generate a second objective instruction sequence to split the computation graph according to the second objective splitting method.

[0122] In some optional examples, the instruction sequence generation method also includes: In response to the fact that, under the first type of splitting strategy, in each computational subgraph obtained after splitting the computational graph, the sum of the input data and output data of each operator is less than or equal to the available memory space capacity corresponding to that operator, it is determined that the first type of splitting strategy satisfies the preset hardware memory limit condition.

[0123] In the methods disclosed herein, the various optional embodiments, optional implementation methods and optional examples disclosed in the above exemplary device section can be flexibly selected and combined as needed to achieve the corresponding functions and effects, and this disclosure does not list them all.

[0124] The beneficial technical effects corresponding to the exemplary embodiments of this method can be found in the corresponding beneficial technical effects of the exemplary device section above, and will not be repeated here.

[0125] Exemplary chip Figure 13 This is a schematic diagram of the structure of chip 120 provided in some exemplary embodiments of this disclosure. For example... Figure 13 As shown, chip 120 includes: The processor 1310 is configured to read a target instruction sequence from the memory 1320 and, based on the data to be calculated, control the computing unit 1330 to execute the target instruction sequence to perform calculations on the data to be calculated; wherein the target instruction sequence is generated by the instruction sequence generation device 110. The computing unit 1330 is used to execute the target instruction sequence based on the control of the processor 1310 to perform calculations on the data to be calculated and obtain the calculation results.

[0126] Optionally, memory 1320 is hardware within chip 120 used for storing data, instructions, etc. For example, memory 1320 can be used to store a target instruction sequence, which may be the first target instruction sequence mentioned above, or it may be the second target instruction sequence mentioned above. If chip 120 includes a neural network accelerator, memory 1320 may be located inside or outside the neural network accelerator.

[0127] Optionally, the computing unit 1330 is hardware within the chip 120 used to perform computational operations. There can be multiple computing units 1330, such as, but not limited to, tensor computing units, vector computing units, and vector computation units. If the chip 120 includes a neural network accelerator, the computing unit 1330 can be located within the neural network accelerator.

[0128] Optionally, the processor 1310 is the hardware in the chip 120 used to perform data read operations and to coordinate and control the operation of other components. The processor 1310 can be electrically connected to the memory 1320 and the computing unit 1330, respectively. In this way, the processor 1310 can read the required data, instructions, etc. from the memory 1320, and the processor 1310 can also coordinate and control the computing unit 1330.

[0129] Optionally, the data to be calculated is data used to input the computation graph for computation, such as a tensor used to input the computation graph for computation. The data to be calculated can be stored in memory 1320.

[0130] In some embodiments, in an intelligent driving scenario, chip 120 can be an intelligent driving chip located in the onboard computing platform of the intelligent driving vehicle, the computation task corresponding to the computation graph can be a target detection task, and the data to be computed can be environmental images of the vehicle's surroundings captured by onboard cameras installed in the intelligent driving vehicle. Thus, by executing a target instruction sequence through chip 120, the data to be computed can be calculated to obtain the target detection results of the environmental images, thereby completing the target detection task.

[0131] In some embodiments, memory 1320 may include main memory and high-speed memory. Main memory can be used to store program instructions, model parameters, and data to be computed, while high-speed memory can be used to cache instructions, data, and / or intermediate results generated during computation. Multiple computing units 1330 may share the data stored in the high-speed memory. Generally, compared to main memory, high-speed memory has a smaller capacity, faster access speed, and is located closer to the computing units 1330, reducing memory access bottlenecks in chip 120.

[0132] In one possible implementation, if chip 120 includes a neural network accelerator, the main memory may be located outside the neural network accelerator, and the high-speed memory may be located inside the neural network accelerator. Therefore, the high-speed memory is closer to the computing unit 1330 in the neural network accelerator, and the computing unit 1330 accesses the high-speed memory much faster than it accesses the main memory.

[0133] During the runtime phase, the first target instruction sequence is executed by chip 120 to complete the corresponding neural network computation task. Because the computation graph is decomposed during the compilation phase, and this decomposition is quite thorough—for example, each operator in the computation graph is decomposed—the input and output data of each operator in each computation subgraph can be stored in the high-speed memory of chip 120 during runtime. This allows for more efficient reuse of data in the high-speed memory of chip 120 during computation, avoiding frequent accesses to the main memory of chip 120 due to incomplete or inadequate decomposition of the computation graph. For example, if the computation graph is not decomposed, and the input and output data of each operator in the computation graph are large, they cannot be stored in high-speed memory and must be stored in main memory. In this case, frequent accesses to main memory are required to perform the computations related to each operator in the computation graph. For example, if some operators in the computation graph are not split, and the input and output data of these unsplit operators are large, they cannot be stored in high-speed memory and must be stored in main memory. In this case, frequent access to main memory is required to perform the computation related to that operator. The embodiments of this disclosure, however, can obviously avoid the situation of frequent access to the main memory of chip 120 due to the computation graph not being split or not being split thoroughly, by executing the first target instruction sequence corresponding to the thoroughly split computation graph. Therefore, the embodiments of this disclosure can complete the corresponding neural network computation task while minimizing access to the main memory of chip 120, thereby effectively reducing access latency, shortening computation time, and improving computational efficiency.

[0134] Exemplary electronic devices Embodiments of this disclosure also provide an electronic device, including the above-described instruction sequence generation apparatus 110.

[0135] In one example, the electronic device may also include input devices and output devices, which are interconnected via a bus system and / or other forms of connection mechanism.

[0136] The input device may also include, for example, a keyboard, a mouse, etc.

[0137] The output device can output various information to the outside, including, for example, a display, a speaker, a printer, and a communication network and its connected remote output devices, etc.

[0138] The implementation of the instruction sequence generation device 110 can be referred to the relevant description above, and will not be repeated here. Since the instruction sequence generation device 110 can achieve the above-mentioned technical effects, electronic devices including the instruction sequence generation device 110 can also achieve the corresponding technical effects.

[0139] Embodiments of this disclosure also provide another electronic device, such as Figure 14 The electronic device 1400 shown includes one or more processors 1410 and memory 1420.

[0140] The processor 1410 may be a central processing unit (CPU) or other form of processing unit with data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 1400 to perform desired functions.

[0141] The memory 1420 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 1410 may execute one or more computer program instructions to implement the instruction sequence generation methods of the various embodiments of this disclosure described above and / or other desired functions.

[0142] In one example, the electronic device 1400 may also include an input device 1430 and an output device 1440, which are interconnected via a bus system and / or other forms of connection mechanism (not shown).

[0143] The input device 1430 may also include, for example, a keyboard, a mouse, etc.

[0144] The output device 1440 can output various information to the outside, including, for example, a display, a speaker, a printer, and a communication network and its connected remote output devices, etc.

[0145] Of course, for the sake of simplicity, Figure 14 Only some of the components of the electronic device 1400 relevant to this disclosure are shown, omitting components such as buses, input / output interfaces, etc. In addition, the electronic device 1400 may include any other suitable components depending on the specific application.

[0146] Exemplary computer program products and computer-readable storage media In addition to the methods and apparatus described above, embodiments of this disclosure may also be computer program products comprising computer program instructions that, when executed by a processor, cause the processor to perform the steps in the instruction sequence generation methods according to various embodiments of this disclosure as described in the "Exemplary Methods" section of this specification.

[0147] Computer program products can be written in any combination of one or more programming languages ​​to perform the operations of embodiments of this disclosure. These programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on a user's computing device, partially on a user's computing device, as a standalone software package, partially on a user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0148] Furthermore, embodiments of this disclosure may also be computer-readable storage media storing computer program instructions that, when executed by a processor, cause the processor to perform the steps in the instruction sequence generation method according to various embodiments of this disclosure as described in the "Exemplary Methods" section above.

[0149] The computer-readable storage medium may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof.

[0150] The basic principles of this disclosure have been described above with reference to specific embodiments. However, the advantages, benefits, and effects mentioned in this disclosure are merely examples and not limitations, and should not be considered as essential features of each embodiment of this disclosure. The specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the scope of this disclosure to the necessity of employing the specific details described above.

[0151] Various modifications and variations can be made to this disclosure without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this disclosure and their equivalents, this disclosure is also intended to include such modifications and variations.

Claims

1. An instruction sequence generation apparatus, comprising a processor, the processor being configured to: Obtain the computation graph; Determine the set of splitting strategies corresponding to the computation graph; In response to the first type of splitting strategy in the splitting strategy set not meeting the preset hardware memory limit, a first target splitting method that meets the preset hardware memory limit is determined based on the second type of splitting strategy and the first type of splitting strategy in the splitting strategy set; wherein... The first type of splitting strategy is used to split the entire computation graph, and the second type of splitting strategy is used to split a portion of the computation graph; Generate a first target instruction sequence to split the computation graph according to the first target splitting method.

2. The apparatus according to claim 1, wherein, The processor determines the first type of splitting strategy in the splitting strategy set, specifically configured as follows: Determine the first split dimension corresponding to the output data of the first operator in the computation graph; Based on the first position information of the first operator in the computation graph and the dimension mapping relationship between the output data of each operator in the computation graph, the first split dimension is propagated along the computation graph to determine the first split dimension propagation relationship graph corresponding to the computation graph; Based on the first splitting dimension transmission relationship graph, the first type of splitting strategy is determined.

3. The apparatus according to claim 2, wherein, The processor, based on the first position information of the first operator in the computation graph and the dimensional mapping relationship between the output data of each operator in the computation graph, propagates the first split dimension along the computation graph as a split dimension, specifically configured as follows: In response to the first position information indicating that the first operator belongs to the starting operator, the first split dimension is backward propagated along the computation graph based on the dimension mapping relationship; or, In response to the first position information indicating that the first operator belongs to the intermediate operator, the first split dimension is forward-propagated and backward-propagated along the computation graph based on the dimension mapping relationship; or, In response to the first position information indicating that the first operator belongs to the end operator, the first split dimension is forward-propagated along the computation graph based on the dimension mapping relationship.

4. The apparatus according to claim 1, wherein, The processor determines the second type of splitting strategy in the splitting strategy set, specifically configured as follows: Based on the dependencies between operators in the computation graph, a second operator is determined; wherein the input data of the second operator is the output data of at least two third operators in the computation graph; Based on the second position information of the second operator in the computation graph, the computation graph is split to obtain multiple first computation subgraphs; For each of the first computational subgraphs, a second splitting dimension corresponding to the output data of the fourth operator in the first computational subgraph is determined; based on the third position information of the fourth operator in the first computational subgraph and the dimension mapping relationship between the output data of each operator in the first computational subgraph, the second splitting dimension is propagated along the first computational subgraph to determine the second splitting dimension propagation relationship graph corresponding to the first computational subgraph. Based on the second splitting dimension transmission relationship graph corresponding to each of the first computational subgraphs, the second type of splitting strategy is determined.

5. The apparatus according to claim 1, wherein, The splitting strategy set includes multiple second-type splitting strategies and multiple first-type splitting strategies. The processor, based on the second-type splitting strategies and the first-type splitting strategies in the splitting strategy set, determines a first target splitting method corresponding to the computation graph that satisfies the preset hardware memory limit conditions. Specifically, this is configured as follows: One or more of the second type of splitting strategies are combined with one or more of the first type of splitting strategies to obtain one or more first combined splitting strategies; In response to any of the first combined splitting strategies satisfying the preset hardware memory limit condition, the first combined splitting strategy is determined as the first target splitting method.

6. The apparatus according to claim 5, wherein, The processor combines one or more of the second type of splitting strategies with one or more of the first type of splitting strategies to obtain one or more first combined splitting strategies, specifically configured as follows: One or more of the first type of splitting strategies and one or more of the second type of splitting strategies are arranged in sequence to obtain one or more splitting strategy sequences, wherein the splitting strategy sequence includes a splitting strategy that first splits the computation graph according to the first type of splitting strategy and then splits the computation graph according to the second type of splitting strategy; One or more of the splitting strategy sequences are determined as one or more of the first combined splitting strategies.

7. The apparatus according to claim 1, wherein, The first target splitting method includes the first type of splitting strategy, the first target splitting number corresponding to the first type of splitting strategy, the second type of splitting strategy, and the second target splitting number corresponding to the second type of splitting strategy. The target splitting number refers to the number of parts into which the input data or output data of the operator is split. The processor generates a first target instruction sequence that splits the computation graph according to the first target splitting method, specifically configured as follows: According to the first type of splitting strategy and the first target number of splits, the computation graph is split to obtain multiple second computation subgraphs; According to the second type of splitting strategy and the second target number of splits, each of the second computational subgraphs is split to obtain the third computational subgraph; Based on the type of each operator in each of the third computational subgraphs, the third position information in the third computational subgraph that satisfies the preset aggregation conditions is determined; According to the third position information, an aggregation operator is inserted into each of the third computational subgraphs to obtain the fourth computational subgraph corresponding to the second computational subgraph. Based on each of the fourth computational subgraphs, the first target instruction sequence is generated.

8. The apparatus according to claim 1, wherein, The splitting strategy set includes multiple first-type splitting strategies and multiple second-type splitting strategies; The processor is specifically configured to respond to the first type of splitting strategy in the splitting strategy set not meeting the preset hardware memory limit condition as follows: In response to the absence of a splitting strategy that satisfies the preset hardware memory limit among the multiple first-type splitting strategies in the splitting strategy set, at least two first-type splitting strategies in the splitting strategy set are combined to obtain multiple second combined splitting strategies; In response to the absence of a second combined splitting strategy that satisfies the preset hardware memory limit among the various second combined splitting strategies, it is determined that the first type of splitting strategy in the splitting strategy set does not satisfy the preset hardware memory limit.

9. The apparatus according to claim 8, wherein, The processor is also configured to: In response to at least one of the first type of splitting strategies in the splitting strategy set satisfying the preset hardware memory limit, the first type of splitting strategy is determined as the second target splitting method corresponding to the computation graph that satisfies the preset hardware memory limit. Alternatively, in response to at least one of the second combined splitting strategies satisfying the preset hardware memory limit, the second combined splitting strategy is used as the second target splitting method corresponding to the computation graph that satisfies the preset hardware memory limit; and a second target instruction sequence is generated to split the computation graph according to the second target splitting method.

10. The apparatus according to claim 1, wherein, The processor determines that the first type of splitting strategy satisfies the preset hardware memory limit condition, and is specifically configured as follows: In response to the fact that, under the first type of splitting strategy, in each computational subgraph obtained by splitting the computational graph, the sum of the input data and output data of each operator is less than or equal to the available memory space capacity corresponding to that operator, it is determined that the first type of splitting strategy satisfies the preset hardware memory limit condition.

11. A method for generating an instruction sequence, comprising: Obtain the computation graph; Determine the set of splitting strategies corresponding to the computation graph; In response to the first type of splitting strategy in the splitting strategy set not meeting the preset hardware memory limit, a first target splitting method that meets the preset hardware memory limit is determined based on the second type of splitting strategy in the splitting strategy set and the first type of splitting strategy; wherein, the first type of splitting strategy is used to split the entire computation graph, and the second type of splitting strategy is used to split a part of the computation graph; Generate a first target instruction sequence to split the computation graph according to the first target splitting method.

12. The method according to claim 11, wherein, Determining the first type of splitting strategy in the splitting strategy set corresponding to the computation graph includes: Determine the first split dimension corresponding to the output data of the first operator in the computation graph; Based on the first position information of the first operator in the computation graph and the dimension mapping relationship between the output data of each operator in the computation graph, the first split dimension is propagated along the computation graph to determine the first split dimension propagation relationship graph corresponding to the computation graph; Based on the first splitting dimension transmission relationship graph, the first type of splitting strategy in the splitting strategy set is determined.

13. The method according to claim 11, wherein, Determining the second type of splitting strategy in the splitting strategy set corresponding to the computation graph includes: Based on the dependencies between operators in the computation graph, a second operator is determined; wherein the input data of the second operator is the output data of at least two third operators in the computation graph; Based on the second position information of the second operator in the computation graph, the computation graph is split to obtain multiple first computation subgraphs; For each of the first computational subgraphs, a second splitting dimension corresponding to the output data of the fourth operator in the first computational subgraph is determined; based on the third position information of the fourth operator in the first computational subgraph and the dimension mapping relationship between the output data of each operator in the first computational subgraph, the second splitting dimension is propagated along the first computational subgraph to determine the second splitting dimension propagation relationship graph corresponding to the first computational subgraph. Based on the second splitting dimension transmission relationship graph corresponding to each of the first computational subgraphs, the second type of splitting strategy is determined.

14. The method according to claim 11, wherein, The splitting strategy set includes multiple second-type splitting strategies and multiple first-type splitting strategies. The step of determining the first target splitting method corresponding to the computation graph that satisfies the preset hardware memory limit based on the second-type splitting strategies and the first-type splitting strategies in the splitting strategy set includes: One or more of the second type of splitting strategies are combined with one or more of the first type of splitting strategies to obtain one or more first combined splitting strategies; In response to any of the first combined splitting strategies satisfying the preset hardware memory limit condition, the first combined splitting strategy is determined as the first target splitting method.

15. A computer-readable storage medium storing a computer program that is executed by a processor to perform the instruction sequence generation method according to any one of claims 11-14.

16. An electronic device comprising an instruction sequence generation apparatus as described in any one of claims 1-10.

Images