Compilation method and device of convolutional neural network, equipment, medium and program product

Abstract

Provided are a method, device, equipment, medium and program product for compiling a convolutional neural network. The method comprises: obtaining a convolutional neural network, an input size range and configuration information of a processing unit supporting static graph execution, the configuration information comprising at least a tensor alignment granularity; selecting a plurality of candidate input sizes within the range based on the granularity, and determining the compilation structure characteristics of the network under each candidate input size; determining at least one size boundary based on the characteristic differences and dividing a plurality of size regions, so that the candidate input sizes in the same region correspond to the same static graph structure; determining a representative input size from each region and statically compiling accordingly to obtain a plurality of static subgraphs executable by the processing unit and corresponding to each region respectively; and generating a compilation result comprising each static subgraph and a size region table, the size region table recording the mapping relationship between each region and each static subgraph. In this way, the processing overhead of compiling a convolutional neural network with different input sizes can be reduced.

Description

Technical Field

[0001] The examples in this article generally relate to the field of computer science, and in particular to compilation methods, apparatus, devices, media, and program products for convolutional neural networks. Background Technology

[0002] Convolutional neural networks (CNNs) are widely used in image enhancement, object detection, image classification, and image segmentation. These networks typically consist of multiple two-dimensional convolution operators. In practical applications, the same CNN may need to process images of different resolutions; therefore, the height and width of the input activation tensor may dynamically change within a certain range.

[0003] To improve the inference efficiency of neural networks, some deep learning accelerators and other devices employ static graph execution. In this approach, the compiler, based on a defined input size, optimizes the neural network execution graph, performs operator transformations, memory planning, data movement planning, and instruction generation during the compilation phase, ultimately generating a static graph for execution by the deep learning accelerator. Summary of the Invention

[0004] In a first aspect, this paper provides a method for compiling a convolutional neural network (CNN). The method includes: obtaining a CNN to be compiled, the input size range of the CNN, and configuration information of a processing unit, wherein the processing unit supports the execution of static graphs, and the configuration information includes at least tensor alignment granularity; selecting multiple candidate input sizes within the input size range based on the tensor alignment granularity, and determining the compilation structure features of the CNN under each of the multiple candidate input sizes; determining at least one size boundary based on the differences between the compilation structure features corresponding to the multiple candidate input sizes, and dividing the input size range into multiple size regions using the at least one size boundary, such that candidate input sizes within the same size region correspond to the same static graph structure; determining a representative input size from each size region, and performing static compilation of the CNN according to the representative input size to obtain multiple static subgraphs that support execution by the processing unit, wherein the multiple static subgraphs correspond to the multiple size regions respectively; and generating a compilation result of the CNN, the compilation result including multiple static subgraphs and a size region table, wherein the size region table records the mapping relationship between the multiple size regions and the multiple static subgraphs.

[0005] In a second aspect of this paper, a compilation apparatus for convolutional neural networks is provided. The device includes: an acquisition module configured to acquire a convolutional neural network to be compiled, the input size range of the convolutional neural network, and configuration information of a processing unit, wherein the processing unit supports the execution of static graphs, and the configuration information includes at least tensor alignment granularity; a feature determination module configured to select multiple candidate input sizes within the input size range based on the tensor alignment granularity, and determine the compiled structural features of the convolutional neural network under the multiple candidate input sizes respectively; a boundary determination module configured to determine at least one size boundary based on the differences between the compiled structural features corresponding to the multiple candidate input sizes, and divide the input size range into multiple size regions using the at least one size boundary, such that candidate input sizes within the same size region correspond to the same static graph structure; a static compilation module configured to determine a representative input size from each size region, and perform static compilation of the convolutional neural network according to the representative input size to obtain multiple static subgraphs that support execution by the processing unit, wherein the multiple static subgraphs correspond to the multiple size regions respectively; and a network compilation module configured to generate the compilation result of the convolutional neural network, the compilation result including multiple static subgraphs and a size region table, wherein the size region table records the mapping relationship between the multiple size regions and the multiple static subgraphs.

[0006] In a third aspect of this document, an electronic device is provided. The electronic device includes at least one processing unit; and at least one memory coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions causing the electronic device to perform the method according to the first aspect when executed by the at least one processing unit.

[0007] In a fourth aspect of this document, a computer-readable storage medium is provided. This computer-readable storage medium stores a computer program that can be executed by a processor to implement the method according to the first aspect.

[0008] In a fifth aspect of this document, a computer program product is provided. This computer program product is tangibly stored in a computer storage medium and includes computer-executable instructions that, when executed by a device, cause the device to perform the method according to the first aspect.

[0009] In this way, a common size region can be established for candidate input sizes with the same static graph structure within the input size range, and static compilation results oriented towards different input sizes can be formed through the mapping relationship between this size region and the executable static subgraph.

[0010] It should be understood that the description in the Summary of the Invention section is not intended to limit the key or essential features of the examples in this disclosure, nor is it intended to restrict the scope of the solutions. Other features will become readily apparent from the following description. Attached Figure Description

[0011] The above and other features, advantages, and aspects of the various examples in this disclosure will become more apparent from the accompanying drawings and the following detailed description. In the drawings, the same or similar reference numerals denote the same or similar elements, wherein: Figure 1 A schematic diagram of an example environment according to some embodiments of the present disclosure is shown; Figure 2 A flowchart illustrating an example process for compiling a convolutional neural network according to some embodiments of the present disclosure is shown; Figure 3 A schematic diagram illustrating an example process of compilation and execution according to some embodiments of the present disclosure is shown; Figure 4 A flowchart illustrating an example process for generating sized regions according to some embodiments of this disclosure is shown; Figure 5 A flowchart illustrating an example process for searching a segmentation strategy according to some embodiments of this disclosure is shown; Figure 6 A schematic diagram illustrating an example process of block computation according to some embodiments of the present disclosure is shown; Figure 7 A schematic diagram illustrating a model package and an example process of operation according to some embodiments of the present disclosure is shown; Figure 8 A schematic diagram illustrating an example process of channel segmentation according to some embodiments of the present disclosure is shown; Figure 9 A schematic structural block diagram of an example apparatus for compiling a convolutional neural network according to some embodiments of the present disclosure is shown; and Figure 10 A block diagram of an electronic device capable of implementing embodiments of the present disclosure is shown. Detailed Implementation

[0012] The examples in this document will now be described in more detail with reference to the accompanying drawings. While some examples are shown in the drawings, it should be understood that solutions can be implemented in various forms and should not be construed as limited to the examples presented herein. Rather, these examples are provided to provide a more thorough and complete understanding of the solutions. It should be understood that the drawings and examples in this document are for illustrative purposes only and are not intended to limit the scope of protection of the solutions.

[0013] It should be noted that the headings of any section / subsection provided in this disclosure are not restrictive. Various examples are described throughout this document, and examples of any type may be included under any section / subsection. Furthermore, examples described in any section / subsection may be combined in any way with any other examples described in the same section / subsection and / or different sections / subsections.

[0014] In the description of examples in this disclosure, the term "comprising" and similar terms should be understood as open inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "an example" or "the example" should be understood as "at least one example". The term "some examples" should be understood as "at least some examples". Other explicit and implicit definitions may also be included below. The terms "first", "second", etc., may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0015] The examples disclosed herein may involve user data, data acquisition, and / or use. All of these aspects comply with applicable laws, regulations, and relevant provisions. In the examples disclosed herein, all data collection, acquisition, processing, manipulation, forwarding, and use are conducted with the user's knowledge and confirmation. Accordingly, in implementing each example, the type, scope of use, and usage scenarios of any data or information that may be involved should be communicated to the user and their authorization obtained through appropriate means in accordance with relevant laws and regulations. The specific methods of notification and / or authorization may vary depending on the actual situation and application scenario, and the scope of the solution is not limited in this regard.

[0016] In this manual and its examples, any processing of personal information will be conducted only on a legal basis (e.g., with the consent of the data subject, or as necessary for the performance of a contract) and will only be carried out within the scope stipulated or agreed upon. A user's refusal to process personal information beyond what is necessary for basic functions will not affect their use of basic functions.

[0017] The term "convolutional neural network" as used in this article refers to a neural network that includes convolution operations. This network can extract and process features of input data through one or more convolutional layers, and may include pooling, activation, and other operators.

[0018] The term "input size range" as used in this article refers to the range of values ​​for the input tensor size supported by the network to be compiled. This range can include the height and width ranges of the input activation tensor, and can be represented by allowed combinations of height and width.

[0019] As used in this paper, the term "processing unit" refers to a hardware computing component capable of executing compiled computational instructions. Processing units support static graph execution and may have corresponding data storage, data movement, and tensor alignment constraints.

[0020] As used in this article, the term "static graph" refers to a computation graph whose computational structure and corresponding execution arrangements are determined before deployment and execution. A static graph may contain pre-defined operator connections, branching, storage arrangements, and execution instructions.

[0021] The term "configuration information" as used in this document refers to information describing the computational and data organization constraints of the target processing unit. Configuration information includes at least tensor alignment granularity and may further include on-chip cache limits, tensor layout, and channel splitting constraints.

[0022] The term "tensor alignment granularity" as used in this article refers to the granularity specified by the hardware for each dimension of a tensor or its corresponding storage layout. The physical size of a tensor can be expanded at the corresponding granularity while retaining the effective size corresponding to the original data.

[0023] As used in this article, the term "candidate input size" refers to an input size selected from the range of input sizes for analyzing compilation results or attempting static compilation. A candidate input size may include corresponding height and width values.

[0024] The term "compilational structural features" as used in this paper refers to information obtained from the compilation analysis of convolutional neural networks that distinguishes the static graph structure corresponding to different input sizes. This information may involve splitting, graph partitioning, output shape, pooling boundaries, pruning rules, and caching strategies.

[0025] The term "size boundary" as used in this article refers to the boundary used to distinguish sets of input dimensions with different compilation structure characteristics. Size boundaries can correspond to the division between height values, width values, or combinations of two-dimensional input dimensions.

[0026] The term "size region" as used in this paper refers to the set of input dimensions formed based on size boundaries. Candidate input dimensions within the same size region correspond to the same static graph structure, and the size region can be represented by continuous intervals or combinations of discrete dimensions.

[0027] As used in this paper, the term "representative input size" refers to the input size determined from a size region for generating the corresponding static subgraph of that size region. The representative input size belongs to the corresponding size region and meets the compilation conditions required to generate an executable static subgraph.

[0028] The term "static subgraph" as used in this paper refers to the static graph execution unit corresponding to the corresponding size region in the compilation result. The static subgraph can contain the computation and output processing of the network under the corresponding segmentation and storage arrangement, and can be executed by the processing unit.

[0029] The term "size region table" used in this paper refers to a data structure that records the mapping relationship between size regions and static subgraphs. The size region table can be further associated with spatial partitioning strategies, channel partitioning strategies, and output processing information that works in conjunction with the corresponding static subgraphs.

[0030] The term "input activation tensor" as used in this paper refers to the multidimensional data set of the input network or an operator in the input network. The output activation tensor is the multidimensional data set generated by the corresponding operation, and the output activation tensor of the previous operator can become the input activation tensor of the next operator.

[0031] The term "input size grid" as used in this paper refers to a set of combined height and width values ​​formed according to tensor alignment granularity. This grid is used to organize the selection of candidate input sizes and structural boundary searches within a finite range of input sizes.

[0032] As used in this article, "analytical compilation" refers to the compilation analysis process performed to extract compiler structure features. This process generates intermediate compilation results needed for the search, but does not generate static subgraphs for deployment.

[0033] The term "graph partitioning" as used in this paper refers to the process of organizing network computation into several computational parts according to computational and resource constraints. Graph partitioning can be combined with the partitioning of tensor height, width, or channels, and can cover one or more connected operators.

[0034] The term "input channel partitioning" used in this paper refers to the method of partitioning the input channels corresponding to the input activations and weights. Each branch calculates the contribution of the corresponding input channel to the output, and the complete contribution is formed by summing them element by element.

[0035] The term "output channel segmentation method" used in this article refers to the method of dividing the output channels into weights and calculating the subsets of output channels separately. The subsets of output channels can be concatenated along the output channel direction to form a complete output.

[0036] The term "pooling boundary" as used in this paper refers to the boundary of the window coverage and output value of the pooling operation, which is related to the input size. This boundary can affect the effective computational range within a graph partition and the connectivity with adjacent operators.

[0037] The term "on-chip cache usage strategy" as used in this article refers to the way inputs, outputs, weights, and intermediate data reside and reused in the on-chip cache of a processing unit. The on-chip cache limit is the maximum cache capacity allowed by the corresponding compilation configuration.

[0038] As used in this paper, a “segmentation block” refers to a portion of computational data that is divided along one or more dimensions of a tensor. A candidate segmentation block may contain logically valid regions, overlapping regions read for neighborhood operations, and alignment regions added for hardware layout.

[0039] The term "overlap" as used in this paper refers to the extent to which the computational dependencies across the segmentation boundary are extended into adjacent regions. The overlap can be determined separately in the height and width directions and can be affected by the computational parameters of the connecting operators.

[0040] The term "alignment correction" as used in this article refers to the sizing correction made to satisfy tensor alignment constraints when determining the physical size or boundaries of the segmentation blocks. The data range covered by the alignment correction may differ from the overlap required by the convolution itself.

[0041] The term "output pruning rule" as used in this article refers to the rules that determine which redundant output regions should be removed from the computation results. Redundant output regions can originate from convolution overlap, alignment padding, or data shifting granularity.

[0042] The term "output splicing" used in this article refers to the process of combining multiple branch outputs into a complete output according to their spatial location or channel order. Spatial splicing and output channel splicing maintain a logical correspondence in their respective dimensions.

[0043] The term "kernel size" as used in this paper refers to the spatial range covered by the convolution kernel. The stride is the input sampling interval corresponding to adjacent convolution output positions, and the dilation rate is the expansion interval between adjacent sampling points of the convolution kernel.

[0044] The term "padding parameter" as used in this paper refers to parameters that specify the method and amount of expansion of the convolution input boundary. Boundary padding in convolution semantics and padding added to satisfy hardware alignment can be determined separately.

[0045] The term "channel partitioning granularity" as used in this article refers to the hardware scale or alignment constraints that channel partitioning must satisfy. The channel partitioning granularity can be matched with the spatial size of the partition blocks.

[0046] The term "bias" as used in this paper refers to an additional parameter applied to the convolution summation result. After the contributions from the input channel splitting are combined, a bias can be applied to the corresponding complete output.

[0047] As used in this article, the term "dynamic model package" refers to a deployment data set that contains multiple static subgraphs and information for selecting static subgraphs based on input dimensions. This data set may also include shared parameters, hardware configuration identifiers, and version information.

[0048] The term "shared index" as used in this article refers to information that establishes reference relationships between multiple static subgraphs and the same set of weight data. A shared index allows different subgraphs to locate commonly stored weight content.

[0049] The term "target code" as used in this article refers to statically compiled code that is loaded and executed by the target execution environment. The hardware configuration identifier indicates the target hardware configuration corresponding to this code and model package.

[0050] The term "tensor layout" as used in this article refers to the arrangement of tensor data elements in storage. Layout transformations can change the storage location of elements while preserving their position and meaning within the logical tensor.

[0051] The term "two-dimensional convolution operator" as used in this paper refers to an operator that performs a convolution operation on input data and a convolution kernel along two spatial dimensions. A two-dimensional convolution operator can accumulate contributions across input channels and produce one or more output channels.

[0052] The term "neural network inference" as used in this paper refers to the process of performing operations on input data using a predetermined network structure and parameters to obtain an output. The compilation of a convolutional neural network generates an executable representation for this operation without requiring retraining of the network parameters during the compilation process.

[0053] As mentioned above, convolutional neural networks (CNNs) are widely used in image enhancement, object detection, image classification, and image segmentation. These neural networks typically consist of multiple two-dimensional convolution operators. In practical applications, the same CNN may need to process images of different resolutions; therefore, the height and width of the input activation tensor may dynamically change within a certain range.

[0054] To improve the inference efficiency of neural networks, some deep learning accelerators and other devices employ static graph execution. In this approach, the compiler, based on a defined input size, optimizes the neural network execution graph, performs operator transformations, memory planning, data movement planning, and instruction generation during the compilation phase, ultimately generating a static graph for execution by the deep learning accelerator.

[0055] A compilation scheme for convolutional neural networks (CNNs) is proposed. The scheme includes: obtaining the CNN to be compiled, the input size range of the CNN, and the configuration information of the processing unit, wherein the processing unit supports the execution of static graphs, and the configuration information includes at least tensor alignment granularity; based on the tensor alignment granularity, selecting multiple candidate input sizes within the input size range, and determining the compilation structure features of the CNN under each of the multiple candidate input sizes; based on the differences between the compilation structure features corresponding to the multiple candidate input sizes, determining at least one size boundary, and using the at least one size boundary to divide the input size range into multiple size regions, such that candidate input sizes within the same size region correspond to the same static graph structure; determining a representative input size from each size region, and performing static compilation of the CNN according to the representative input size to obtain multiple static subgraphs that support execution by the processing unit, wherein the multiple static subgraphs correspond to the multiple size regions; and generating the compilation result of the CNN, which includes multiple static subgraphs and a size region table, wherein the size region table records the mapping relationship between the multiple size regions and the multiple static subgraphs.

[0056] Through the above processing, different input sizes of the network can be mapped to static compilation results according to their respective regions. When the same region contains multiple input sizes that can reuse the same structure, the electronic device can use static compilation of the region to represent the size instead of performing complete deployment compilation for each size individually. The processing unit still executes the pre-formed static graph, so there is no need to add a dynamic graph execution mechanism to obtain the compilation result.

[0057] The following describes various examples of this scheme in further detail with reference to the accompanying drawings.

[0058] Figure 1 A schematic diagram of example environment 100 is shown. Example environment 100 includes processing device 110, convolutional neural network 120, and compilation result 130. Processing device 110 acquires convolutional neural network 120 and performs compilation processing to form compilation result 130.

[0059] Processing device 110 may include computing devices for running compilers, which may include, for example, a terminal and / or a server. The terminal may be any type of mobile terminal, fixed terminal, or portable terminal, including mobile phones, desktop computers, laptop computers, notebook computers, netbook computers, tablet computers, media computers, multimedia tablets, handheld computers, portable gaming terminals, virtual reality (VR) / augmented reality (AR) devices, personal communication system (PCS) devices, personal navigation devices, personal digital assistants (PDAs), audio / video players, digital cameras / camcorders, positioning devices, television receivers, radio receivers, e-book devices, gaming devices, or any combination thereof, including accessories and peripherals of these devices or any combination thereof. In some cases, the terminal may also support any type of user-facing interface (such as "wearable" circuitry).

[0060] A server can be a single physical server, a server cluster or a distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, and content delivery networks. Servers can also include computing systems / servers, such as mainframes, edge computing nodes, and computing devices in cloud environments, etc.

[0061] A communication connection can be established between the server and the terminal. This connection can be established via wired or wireless means. The communication connection can include, but is not limited to, Bluetooth, mobile network, Universal Serial Bus (USB), and Wireless Fidelity (WiFi) connections. In some cases, the server and terminal can exchange signaling information through this communication connection.

[0062] It should be understood that the structure and function of the various elements in the example environment 100 are described for illustrative purposes only and do not imply any limitation on the scope of the scheme.

[0063] The following description of the example will continue with reference to the accompanying drawings.

[0064] Figure 2 A flowchart illustrating an example process 200 for compiling a convolutional neural network according to some embodiments of the present disclosure is shown. Process 200 can be implemented at processing device 110. Reference is made below. Figure 2 To describe process 200.

[0065] In step 210, the processing device 110 acquires the convolutional neural network to be compiled, the input size range of the convolutional neural network, and the configuration information of the processing unit. The processing unit supports executing static graphs, and the configuration information includes at least tensor alignment granularity.

[0066] The input size range here constrains the inputs covered by this compilation, while the configuration information constrains the computation and storage arrangements that the static graph can use. The processing device 110 can read in the network structure and weights along with the ranges of dynamic height and dynamic width. The processing device 110 can also obtain hardware target information, compilation configuration, and the output path of the compilation results.

[0067] For example, processing device 110 can receive input ranges where both height and width vary between 320 and 1920. If the hardware requires height, width, and channels to be aligned to 8, 8, and 4 respectively, processing device 110 records the alignment requirements for each dimension separately. Input activation and output activation must both meet the corresponding hardware requirements, rather than just performing an alignment once on the image at the very front of the network.

[0068] By taking both the size range and the target hardware constraints as input, the processing device 110 can generate the compilation result of the corresponding target processing unit within a defined search space. This process provides a consistent configuration basis for subsequent identification of structural changes, avoiding misinterpreting compilation differences under different hardware configurations as size region relationships under the same configuration.

[0069] To further illustrate how the data obtained in step 210 is used in subsequent compilation and execution, Figure 3 A schematic diagram of an example process 300 for compiling and executing according to some embodiments of the present disclosure is shown. Figure 3 The compilation device 320 can perform the compilation function of the processing device 110, and the execution device 340 is used to load and execute the formed static subgraph. The compilation input 310 includes the neural network model and weights 311, dynamic height and width range 312, cache and alignment constraints 313, and compilation configuration 314.

[0070] The compilation device 320 receives compilation input 310 through the compilation entry point 321. Structural boundary search 322 analyzes structural differences based on input size variations, and graph partitioning strategy search 323 further analyzes the partitioning arrangement to satisfy hardware constraints such as caching. Overlap and alignment analysis 324 determines the neighborhood reading range and alignment correction for the partitioned blocks, and static subgraph construction 325 generates corresponding static execution results based on this.

[0071] After the static subgraph is formed, shared parameters are established (326) to identify commonly referenced weights. The dynamic model package generation (327) organizes the static subgraph, size region table, shared parameters, and related configurations into a dynamic model package (330). The dynamic model package (330) includes the size region table and subgraph mapping, the static subgraph and target code, shared parameters, and version information. The data dependencies in generating the dynamic model package (330) ensure that the region partitioning results correspond to the final executable subgraph and its parameters.

[0072] The execution device 340 receives the dynamic model package 330, as well as the actual input and dimensions 350. Dimension lookup table assignment 341 selects the target sub-graph, and static graph execution 342 completes the corresponding calculations. Output clipping and transformation 343 then processes the execution result into a logical output corresponding to the actual input.

[0073] Figure 3 The various functions can be implemented as functional parts of the compiler and corresponding execution program run by the processing device 110. The compilation device 320 and the execution device 340 can also respectively undertake compilation and deployment execution, with the execution device 340 obtaining the data required for static execution by receiving the dynamic model package 330. This approach concentrates dimensional analysis and segmentation search during the compilation process, while the execution phase selects static results based on existing mappings.

[0074] Back Figure 2 In step 220, the processing device 110 selects multiple candidate input sizes within the input size range based on tensor alignment granularity, and determines the compilation structure features of the convolutional neural network under the multiple candidate input sizes respectively.

[0075] In some embodiments, the input size range includes the height range and width range of the input activation tensor. The processing device 110 can discretize the height range and width range into a combination of multiple height values ​​and multiple width values ​​based on the tensor alignment granularity to obtain an aligned input size mesh.

[0076] Each position in the input dimension grid represents a combination of a height value and a width value. The processing device 110 can determine values ​​from the height range that satisfy the height alignment condition and values ​​from the width range that satisfy the width alignment condition. The allowed values ​​in both directions together constitute the candidate search space, rather than simply considering the diagonal dimensions where the height equals the width.

[0077] In the example with dimensions from 320 to 1920 and aligned to 8, the height can increase by 8 increments from 320 to 1920, and the width can be determined in the same way. For inputs with different heights and widths, the processing device 110 still records their positions using the corresponding two-dimensional coordinates. For example, 1280×736 and 1920×1080 can both be used as combinations of dimensions in a two-dimensional grid.

[0078] Figure 4 A flowchart of an example process 400 for generating a sized region according to some embodiments of the present disclosure is shown. Process 400 proceeds from a starting node 402 to step 404, where processing device 110 receives dynamic height and width ranges and hardware constraints, and forms a search mesh at an alignment granularity. This mesh provides a locatable and finite set of candidates for subsequent analytical compilation.

[0079] By forming a mesh based on hardware alignment granularity, the processing device 110 is able to perform structural analysis on combinations of dimensions that meet configuration requirements. This processing reduces the distinction between repetitive dimensional states that do not conform to the alignment representation and ensures that subsequent boundaries are consistent with the physical dimensions used in the actual static diagram.

[0080] After forming the search grid, in some embodiments, the processing device 110 may select multiple candidate input sizes from the input size grid and perform analytical compilation on the convolutional neural network with each candidate input size. Analytical compilation is used to generate compilation results for extracting compilation structure features, rather than generating static subgraphs for deployment. The processing device 110 can extract corresponding compilation structure features from the compilation results of each analytical compilation.

[0081] Specifically, in Figure 4 In step 406, processing device 110 selects candidate sizes and performs analytical compilation. For example, processing device 110 sets the current candidate height and width values ​​as the network input sizes. Processing device 110 performs compilation analysis based on the same network weights and hardware configuration to obtain intermediate results such as graph optimization, partitioning, memory allocation, and other parameters. In step 408, processing device 110 extracts compilation structure features: partitioning, memory planning, and pruning rules. Thus, processing device 110 can extract information reflecting the static structure from these results.

[0082] Analytical compilation and full static compilation, which generates deployment subgraphs, have different outputs. The former preserves structural information for the search, while the latter also needs to generate target code and execution arrangements that processing units can load. Therefore, even if the search process analyzes multiple grid locations, it does not mean that a set of deployable static graphs is generated and saved for each location.

[0083] For grid locations where the search is not yet complete, the processing device 110 can continue to select the next candidate and repeat the analysis. For each analysis, the processing device 110 retains the candidate height, candidate width, and corresponding compilation structure features. This corresponding record is used for... Figure 4 Subsequent nodes determine whether the current region can be expanded.

[0084] Through analytical compilation, processing device 110 can obtain the structural basis required for dividing dimensional regions before generating deployment code. Reusable candidate input dimensions within the region then share a representative static subgraph, thereby reducing the work of generating deployment code per candidate and repeatedly saving deployment results.

[0085] To ensure that the aforementioned structural comparison reflects the actual static execution arrangement, in some embodiments, the compilation structure features may include at least two of the following: the number of height-direction segments, the number of width-direction segments, the input channel segmentation method, the output channel segmentation method, the number of graph partitions, the shape of the output tensor, the pooling boundary, the output pruning rules, and the on-chip cache usage strategy.

[0086] The number of segments in the height direction and the number of segments in the width direction describe the branch scale of the spatial blocks, respectively. The input channel segmentation method records which input channels and their corresponding weights enter the same convolutional branch, and the output channel segmentation method records which output channels are generated by the same branch. When comparing these features, the processing device 110 needs to distinguish between two different branch merging structures: element-wise summation and channel-wise concatenation.

[0087] The number of graph partitions reflects how many computational parts the computation graph is organized into. The output tensor shape reflects the output representation used, and the pooling boundary reflects the effective coverage of the pooling window at the corresponding size. The output pruning rules and on-chip cache usage strategy reflect how redundant results are removed and how computational data is retained, respectively.

[0088] Processing device 110 can record these features according to network operators and compilation stages, and compare the corresponding records item by item. Two dimensions may have the same number of spatial partitions, but if the channel branches, pooling boundaries, or pruning structures are different, they may still correspond to different static graph structures. Conversely, different logical input dimensions can use the same static subgraph when the compared structures and their associated constraints can maintain the same static execution arrangement.

[0089] If the actual output shape itself is used as a structural comparison item, different actual output shapes can lead to further subdivision of size regions. If a static subgraph already supports a common physical output layout and rules for determining clipping based on the effective range, the processing device 110 can make comparisons based on the output representations supported by that static structure. This comparison is always conditional on the ability to reuse the same static graph structure and cannot be merged simply because the input dimensions are similar.

[0090] By comprehensively considering multiple structural dimensions, the processing device 110 enables the region partitioning to correspond to the actual branching, storage, and output processing. This processing reduces the possibility of misclassifying incompatible input dimensions into the same static subgraph and provides a structural basis for maintaining the correct output of the partitioning operation.

[0091] In step 230, the processing device 110 determines at least one size boundary based on the differences between the compiled structure features corresponding to the multiple candidate input sizes. The processing device 110 uses the at least one size boundary to divide the input size range into multiple size regions, such that candidate input sizes within the same size region correspond to the same static graph structure.

[0092] In some embodiments, the processing device 110 can group candidate input sizes according to the differences between compilation structure features to obtain at least one size region. Candidate input sizes within the same size region correspond to the same compilation structure features, while candidate input sizes in different size regions correspond to different compilation structure features.

[0093] Size boundaries indicate locations where the compilation structure changes and are not pre-defined as fixed resolution brackets. Increasing the size may cause a convolutional block to exceed the cache limit or alter the hardware adaptation of an operator. The processing device 110 identifies these changes based on the analysis results and separates sizes that cannot share the same structure before and after the change.

[0094] exist Figure 4 In step 410, the processing device 110 determines whether the structural features of the current candidate size are the same as or compatible with those of the current size region. Thus, the processing device 110 can determine whether the structural features of the current candidate size are the same as or compatible with the structural features of the current size region, allowing for the reuse of the same static graph structure. The comparison is based on features such as segmentation, partitioning, memory planning, and pruning rules extracted in step 408. Here, compatibility means the ability to reuse the same static structure, rather than allowing arbitrary changes to the computation graph within the same region.

[0095] If the current candidate input size has the same structural features as the current region or can reuse the same static graph structure, then process 400 proceeds to step 412. In step 412, processing device 110 expands the current size region. For example, processing device 110 adds the candidate size to the current size region, expanding the set of input sizes covered by the region.

[0096] If the current candidate input size differs from the structural features of the current region and the same static graph structure cannot be reused, process 400 proceeds to step 414. In step 414, processing device 110 records the boundary and establishes a new region. For example, processing device 110 records the corresponding size boundary and establishes a new size region, so that the new structural arrangement has an independent region record.

[0097] The processing steps 412 and 414 are combined into step 416. In step 416, processing device 110 determines whether the search space has been traversed. For example, processing device 110 determines whether the search space has been traversed based on the positions that have been processed and the positions that have not yet been processed in the search grid. If there are still untraversed candidate input sizes in the search space, process 400 proceeds to step 420.

[0098] In step 420, processing device 110 selects the next candidate input size and returns to step 406 for analytical compilation. If the search space has been traversed, process 400 proceeds to step 418. Thus, Figure 4 The loop continues to complete candidate analysis and region expansion until regional information is formed for subsequent boundary merging.

[0099] By dividing regions according to structural differences, processing device 110 can group dimensional changes that do not alter the static graph structure into the same reuse scope. This grouping reduces redundant deployments and compilations caused by different input dimension values, while allowing locations of structural changes to maintain independent execution schedules.

[0100] The aforementioned candidate groups can be influenced by multiple operators and multiple optimization stages in the network. In some embodiments, the processing device 110 can take the union of the size boundaries obtained by different operators or different compilation optimization stages, and sort the obtained size boundaries according to their height and width values ​​to determine at least one size boundary. This at least one size boundary is the boundary of the corresponding size region.

[0101] Specifically, in Figure 4 In step 418, the processing device 110 merges the boundaries of each operator and optimization stage, and generates a sized region. For example, the processing device 110 can collect the boundaries obtained from convolution splitting, graph partitioning, memory planning, and other optimization stages. When different stages generate boundaries at the same location, the processing device 110 can represent them as a common boundary. Boundaries at different locations are retained as the basis for region division and organized into queryable ordered records according to their corresponding height and width.

[0102] A cache constraint for one operator may change at a height value, while a pooling or hardware specification constraint for another operator may change at other height or width values. Processing device 110 merges these boundaries and determines the region based on the combination of dimensions commonly defined by these boundaries. Sorting is used to organize boundary records, but it does not mean that discontinuities in the two-dimensional region can be ignored.

[0103] By merging the boundaries of various operators and compilation stages, the processing device 110 enables the final region to simultaneously reflect the structural requirements of different parts of the network. This processing avoids directly using the grouping results that only satisfy a single operator for the entire network, thereby reducing erroneous static graph selection caused by structural incompatibility in other stages.

[0104] After boundary merging, the representation of the size region still needs to reflect the relationship between the changes in height and width. In some embodiments, the processing device 110 can classify candidate input dimensions that can reuse the same static subgraph into the same size region. The region can be a regular two-dimensional range, or it can be composed of irregular or discontinuous combinations of height and width.

[0105] For example, within an input range of 320×320 to 1920×1920, a search might reveal structural reuse ranges corresponding to segments of 320 to 512, 512 to 1024, 1024 to 1536, and 1536 to 1920. If the corresponding two-dimensional input combinations have been verified to share the same structure, the processing device 110 can generate a representative static subgraph for each of the four regions. 800×800 can fall into the second region, and 1700×1700 can fall into the fourth region.

[0106] The common endpoints of these segments need to have a defined affiliation in the size region table. For example, processing device 110 can assign the candidate size corresponding to the endpoint to one side region based on the actual compilation structure at the boundary. For dimensions whose height and width span different segments respectively, processing device 110 determines the affiliation according to the actual two-dimensional region record.

[0107] In other embodiments, the set of dimensions corresponding to the same static subgraph may form an irregular shape in the height and width coordinates, or contain mutually separate combinations of dimensions. The processing device 110 may represent this set using multiple range records or allow size combination records. The input dimension is selected from actual candidate points within the set; the maximum height and maximum width from different candidate points cannot be directly combined to form a dimension outside the region.

[0108] The logical size of the actual input and its aligned physical size can be stored separately. For a 1700×1700 actual input, aligning it upwards in both directions by 8 will yield a physical size of 1704×1704. The processing device 110 processes this type of input according to a pre-determined coverage relationship within the region and retains 1700×1700 as the actual valid range.

[0109] By preserving the actual shape and boundary assignments of the two-dimensional region, the processing device 110 can cover inputs with different aspect ratios. During the runtime phase, a table is consulted to select the static subgraph, which reduces matching ambiguities caused by roughly classifying based on a single maximum side length.

[0110] In step 240, the processing device 110 determines a representative input size from each size region and statically compiles the convolutional neural network according to the representative input size to obtain multiple static subgraphs that support execution by the processing unit. Each static subgraph corresponds to a different size region.

[0111] In some embodiments, the processing device 110 may sort candidate input sizes within a corresponding size region according to the amount of input activation data corresponding to multiple candidate input sizes. Candidate input sizes ranked higher are greater than those ranked lower. The processing device 110 may perform static compilation on the multiple candidate input sizes based on the ranking result, and sequentially select the next candidate input size if no static subgraph executable by the processing unit is generated, until a representative input size capable of completing static compilation is determined.

[0112] The amount of input active data here reflects the actual input storage size corresponding to the candidate size. When the number of input channels, data types, and batch configurations are the same, this amount of data varies with the product of height and width. If different storage alignment arrangements exist, the processing device 110 can also compare candidate sizes based on the amount of data under the corresponding layout, rather than just comparing height or just width.

[0113] exist Figure 4 In step 422, processing device 110 selects a compilable representative size for each region and falls back to a smaller size if the selection fails. For example, processing device 110 can attempt full static compilation starting with the largest candidate size ranked higher. A larger size can represent the corresponding data scale and can cover a larger effective range in inputs that require more spatial information. However, structural or hardware constraints of certain operators in the network may prevent static compilation from being completed at this size.

[0114] If the current candidate input size does not generate an executable static subgraph, the processing device 110 can select the next candidate input size from the sorting results and perform static compilation. If the next candidate is also unsuccessful, the processing device 110 continues to backtrack sequentially. This process means that the input size is determined by the actual successfully compiled candidates, rather than simply by the maximum value of the region.

[0115] When using the representative input size obtained through backtracking, the corresponding static subgraph still needs to meet the coverage conditions of the input within that size area. The processing device 110 confirms the correspondence between the region and the subgraph by combining the pre-obtained structure, physical input range, and clipping arrangement. The smaller candidate can be compiled on its own, but this does not replace the confirmation that the static subgraph can be reused for other sizes within the region.

[0116] After the representative input dimensions are determined, in step 424, the processing device 110 fully compiles the representative dimensions and establishes the mapping, and outputs a static sub-graph and a dimension region table. Thus, the processing device 110 can complete the complete static compilation and mapping establishment of the representative dimensions. The corresponding static sub-graph and dimension region table are then used... Figure 2 The compilation results in step 250 are generated. Only the actually generated executable subgraphs are used as the static execution results for the corresponding regions.

[0117] By sorting by data volume and rolling back sequentially when compilation fails, the processing device 110 is able to find a representative executable size within a region. This process avoids the inability to generate candidate static results for the entire region due to network or hardware constraints caused by a single large candidate, while retaining the basis for confirming executability through actual compilation.

[0118] The static compilation in step 240 further requires that the storage conditions of the target processing unit be met. In some embodiments, the configuration information may also include an on-chip cache limit. For any representative input size, the processing device 110 can obtain the overlap and alignment correction amount of the convolutional layer when it is segmented in the height and width directions.

[0119] Processing device 110 can determine the on-chip cache occupancy of the complete graph or candidate slices of the convolutional neural network. This occupancy includes the storage space occupied by at least one of input activations, output activations, weights, intermediate buffers, and alignment padding. If the occupancy exceeds the on-chip cache limit, processing device 110 determines the input size of each candidate slice based on the number of slices in the height direction, the number of slices in the width direction, the overlap amount, and the alignment correction amount.

[0120] The on-chip cache occupancy reflects the actual data that needs to reside in a computation phase, not just the height and width of the original image. Input activations and weights need to be entered into the on-chip cache for the computation unit to read, and output activations and intermediate results required by subsequent operators may also occupy this cache. The processing device 110 can determine the occupancy of each type of data according to the physical size of the tensor and the storage size of the data elements.

[0121] For inputs, outputs, weights, and intermediate buffers that need to be retained simultaneously, processing device 110 can count their corresponding occupancy based on the compile-time storage arrangement. For storage space that has already been reused according to the arrangement, processing device 110 estimates based on the actual occupancy relationship. When alignment padding is already included in the tensor physical size, the same padding is not counted again.

[0122] For example, when the height and width are aligned to 8 and the channels are aligned to 4, the storage range of the input block may be larger than its logical effective range. The processing device 110 also needs to determine the physical size and alignment range of the convolution output and take into account the intermediate buffers of the computation branches. Overlapping data of adjacent input blocks also occupy the storage of the corresponding blocks, and the final cache result cannot be simply obtained by dividing the total graph occupancy by the number of segments.

[0123] Figure 5 A flowchart of an example process 500 for searching a segmentation strategy according to some embodiments of the present disclosure is shown. Process 500 proceeds from start step 502 to step 504, where processing device 110 obtains the size to be searched, the amount of overlap, and the alignment correction granularity of height and width. In step 506, processing device 110 estimates the on-chip cache usage of the complete subgraph.

[0124] In step 508, the processing device 110 determines whether space partitioning needs to be increased due to cache capacity by comparing the total occupancy of the complete subgraph with the on-chip cache limit. If the total occupancy of the complete subgraph exceeds the on-chip cache limit, process 500 proceeds to step 510. If the total occupancy of the complete subgraph does not exceed the on-chip cache limit, process 500 proceeds to step 522.

[0125] Paths that do not exceed the cache limit can record strategies that do not require further spatial partitioning. For convolutions still constrained by channel hardware specifications, processing device 110 can combine subsequent channel partitioning processing to form a complete strategy record. This path indicates that the cache capacity does not require additional spatial blocks, but does not exclude the adaptation of other hardware constraints.

[0126] By incorporating overlapping regions and aligned physical tensors into occupancy estimates, processing device 110 is able to generate segmented inputs corresponding to the actual resident data. This processing enables subsequent cache comparisons to be performed based on actual computed blocks and reduces on-chip memory overruns caused by underestimating edge-extended data.

[0127] After confirming that the complete graph or the current computation portion requires spatial partitioning, in some embodiments, the processing device 110 may determine the initial partitioning quantity based on the ratio of the occupancy to the on-chip cache limit. The processing device 110 may allocate the initial partitioning quantity into height-direction partitioning quantity and width-direction partitioning quantity based on the ratio between the height and width values ​​representing the input dimensions.

[0128] The processing device 110 can calculate the candidate segment block with the largest on-chip cache usage under the current number of segments in the height and width directions. If the on-chip cache usage of this candidate segment block exceeds the on-chip cache limit, or if the channel segmentation granularity does not meet the configuration information, the processing device 110 can increase at least one of the number of segments in the height and width directions. The processing device 110 thus determines the number of segments in the height and width directions.

[0129] exist Figure 5 In step 510, the processing device 110 can use the ratio of the total subgraph occupancy to the cache limit as an initial estimate of the required number of blocks and convert it into an integer number of usable blocks. This initial estimate is used to initiate the search, and subsequent calculations of overlap, alignment, and channel constraints will still be taken into account for the actual block occupancy. In step 512, the processing device 110 allocates the initial number of blocks to the height and width directions.

[0130] During allocation, the processing device 110 can balance the input aspect ratio with the overlap cost introduced by segmentation. When the input is longer in the height direction, the number of segments in the height direction can be increased first. When the input is longer in the width direction, the number of segments in the width direction can be increased first to make the data size of each block more balanced.

[0131] The processing device 110 determines the logical location and basic size of each block according to the current number of partitions, and then adds the required overlap based on the location of the block. Inner blocks may require expansion on both sides in a certain direction, while boundary blocks only need to expand towards the adjacent block in that direction. Subsequently, the processing device 110 determines the actual read and store range based on the alignment correction amount.

[0132] In step 514, the processing device 110 calculates candidate blocks containing double-sided overlap and aligns them upwards at a hardware granularity. In step 516, the processing device 110 estimates the cache occupancy of each candidate block and determines the candidate block with the largest occupancy. Inner blocks are typically larger due to double-sided overlap, but the processing device 110 still determines the worst-case occupancy based on the actual shape of each block and the alignment results, avoiding ignoring the impact of edge blocks or output layout.

[0133] Processing device 110 also verifies channel splitting constraints. The hardware-supported channel size may be related to the block size, so meeting the cache capacity requirements does not necessarily mean that channel constraints are met. In some configurations, larger blocks require smaller channel granularity, and further reducing the block size can make channel calculations conform to hardware specifications.

[0134] In step 518, the processing device 110 can determine whether the cache and channel constraints are simultaneously satisfied. If the cache usage of the worst-case candidate partition block does not exceed the on-chip cache limit, and the channel partitioning granularity meets the configuration information, then process 500 proceeds to step 522. If the cache usage of the worst-case candidate partition block exceeds the on-chip cache limit, or the channel partitioning granularity does not meet the configuration information, then process 500 proceeds to step 520.

[0135] In step 520, processing device 110 increases the number of candidate blocks and returns to step 512 to reallocate the number of segments in the height and width directions. The increase can be applied to the height direction, the width direction, or both. After reallocation, processing device 110 re-determines the overlapping and aligned block ranges and performs cache and channel constraint comparisons.

[0136] In step 522, the processing device 110 records the final number of height segments, the number of width segments, the channel segmentation method, and the clipping rules. In step 524, the processing device 110 categorizes input dimensions that have the same segmentation strategy and can reuse the same static structure. This categorization can be used as... Figure 4 Structural boundary search and Figure 2 The basis for dividing the size regions is to ensure that the regions correspond to the actual hardware executable strategies.

[0137] For example, for certain input sizes, a strategy of dividing the height and width into two parts can satisfy the cache constraint, forming a 2×2 spatial block. For a 1920×1080 image enhancement input, if this strategy still does not meet the resource conditions, the processing device 110 can continue searching. In an example where dividing the height into three parts and the width into two parts can satisfy the conditions, the processing device 110 forms a 3×2 segmented static subgraph for it.

[0138] 3×2 represents three parts in the height direction and two parts in the width direction; it does not indicate the order in which the input dimensions are written. The processing device 110 divides the data according to the height and width axes already determined in the model input description. For different input layouts, the positions of the corresponding axes can change, but the number of divisions recorded always corresponds to their respective spatial dimensions.

[0139] By initializing the occupancy ratio and performing cyclic checks on the worst-case block, the processing device 110 can obtain a space partitioning strategy that meets the requirements of cache and channels. This process converts computations that cannot reside in the complete graph into resident block computations, and allows sizes with the same strategy to share a static structure. The processing unit can read on-chip data for computation based on the compilation results, reducing the workload of re-searching for partitioning arrangements during runtime.

[0140] In determining the number of candidate segments, the actual input range of each segment also depends on the computational dependencies of convolution and adjacent operators. In some embodiments, the processing device 110 can determine the input region corresponding to each segment based on the convolution kernel size, stride, dilation rate, padding parameters, the operational parameters of the preceding and following operators connected to the convolutional layer, and the tensor alignment granularity.

[0141] Along each segmentation direction in both the height and width directions, the processing device 110 can extend the overlap of the input region of a segmented block located at the boundary of that direction to the side where the adjacent segmented block is located. The processing device 110 can also extend the overlap of the input region of a segmented block located at a non-boundary position in that direction to the opposite sides respectively.

[0142] Overlapping reads are used to supplement neighboring data across block convolutions. Taking a 3×3 convolution as an example, the output position near the split boundary may depend on the input on both sides of the boundary. If only the data within the split block is read, the output position will lack a portion of the convolution sample values. Therefore, the processing device 110 extends the input read range to adjacent regions according to the actual convolution parameters.

[0143] Specifically, the processing device 110 can deduce the input position from the output position that needs to be retained. For an output position, the starting point of its input sampling is determined by the output position, stride, and padding parameters, and the other sampling points in the convolution kernel are then expanded according to the dilation rate. After determining the required sampling range for the beginning and end positions of an output region, the processing device 110 obtains the input region required to cover the output region.

[0144] Processing device 110 performs this range derivation along both the height and width. The portion of the required input region that extends beyond the current logical segmentation block but remains within the original valid input constitutes the data that needs to be read from the region containing the adjacent block. The outer boundary of the original tensor is handled according to the network's own padding parameters; the inner segmentation boundary cannot be directly treated as the boundary of the original tensor.

[0145] After determining the read range required for convolution, the processing device 110 further adjusts the block size or read boundary at the hardware granularity of height and width. For example, if the theoretically required neighborhood has not yet reached the hardware alignment length, the processing device 110 can extend upwards to a physical range that conforms to the granularity. The range of additional reads or padding due to alignment is recorded separately for effective positions so that its influence can be removed during output.

[0146] A block located on the outermost side of a certain cutting direction only expands and overlaps with its adjacent block inside that direction. A block located inside that direction needs to expand in both adjacent directions. A two-dimensional block may be a boundary block in the height direction and an interior block in the width direction; the processing device 110 determines the two directions separately.

[0147] By inferring the input read region from the required output neighborhood, the processing device 110 can preserve sufficient input dependencies for boundary convolutions. Combined with alignment correction, the segmentation blocks can satisfy hardware read requirements while preserving the data required for complete convolutions, avoiding changes to the logical output of block edges due to missing neighborhoods.

[0148] The above-described input range derivation can also be used for graph partitioning across multiple operators. In this embodiment, the processing device 110 can combine the operational parameters of the preceding and following operators connected to the convolutional layer to determine the input range and effective output range of the entire computation section. The required overlap between layers is not limited to the coverage width of a single convolutional kernel.

[0149] For multiple interconnected convolutional layers, the processing device 110 can start from the end output region to be retained and determine the region that the previous layer needs to provide layer by layer along the operator connection relationship. The stride, dilation rate, and padding parameters of each layer act on the range transformation of that layer, and the transformed input requirements continue to become the output requirements of the previous layer. The windows and boundaries of adjacent operations such as pooling also participate in the range determination at the corresponding positions.

[0150] As the number of consecutively segmented layers increases, invalid edges generated by interlayer padding may continue to propagate, expanding the area that needs to be removed at the end. Therefore, the processing device 110 calculates the overlap based on the interlayer relationships of the entire partition, rather than independently trimming each layer to the same fixed width. Subsequent trimming is performed based on the effective output range obtained from the propagation.

[0151] For cache usage, processing device 110 can organize the computation of the same spatial block in connected operators as sequential processing. This arrangement allows intermediate features to be generated in blocks and saved according to subsequent usage relationships. If only the input of a certain layer is split while the output of the complete layer still resides, the output residency size will not automatically decrease due to input splitting, and processing device 110 needs to include this situation in the occupancy estimation of the complete graph or partition.

[0152] By incorporating multi-layer dependencies and intermediate result residency relationships into the segmentation, the processing device 110 is able to use block execution while preserving end-to-end convolutional semantics. This processing avoids missing invalid edges that accumulate layer by layer and ensures that arrangements that reduce cache footprint are implemented in actual intermediate tensors, rather than just in the partitioning of the input shape.

[0153] After determining the input region and the corresponding valid output range, in some embodiments, the processing device 110 may trim redundant output regions generated by at least one of convolution overlap, alignment padding, and data shifting granularity before or after output splicing.

[0154] Output pruning is based on which positions within each block correspond to the final logical output, rather than simply deleting all outputs that have read overlapping data. Processing device 110 can determine the valid output range to retain for each block, and the position of that range within the complete output. The retained ranges from each block collectively cover the final logical output, avoiding duplicate writing of the same output position to the concatenated result.

[0155] Figure 6 A schematic diagram of an example process 600 for block computation according to some embodiments of the present disclosure is shown. Figure 6 Figure (a) shows the X11 input block 612, X12 input block 614, X21 input block 616, and X22 input block 618 in input 610, located at four positions in the two-dimensional partition. The shaded areas in the figure represent the overlapping regions that need to be read for adjacent block computations. Block convolution 640 is formed based on these inputs. Figure 6 The block output of (b) is 620.

[0156] Block output 620 includes Y11 output block 622, Y12 output block 624, Y21 output block 626, and Y22 output block 628. Due to overlapping read ranges among the blocks, some output positions may appear repeatedly. Alignment padding and data shifting granularity may also result in additional areas in the physical output. Figure 6 The cropping region icon in the diagram represents the invalid portion that needs to be removed.

[0157] The trimming and splicing process 650 processes the blocks according to their effective range, and outputs 620 to form a block. Figure 6 The valid output 630 in (c) includes Y11' output block 632, Y12' output block 634, Y21' output block 636, and Y22' output block 638. Y11' output block 632, Y12' output block 634, Y21' output block 636, and Y22' output block 638 are arranged according to their original two-dimensional positions, each providing the corresponding valid portion of the final output. Figure 6 The 2×2 spatial division shown is used to illustrate the relationship between reading, calculation, and clipping; the same range processing can be applied to... Figure 5 Other number of segments determined in the process.

[0158] In the implementation of cropping before stitching, the processing device 110 can remove invalid edges and duplicate outputs from each branch before writing the valid results to the corresponding positions in the complete output. This method avoids moving regions that will be deleted as valid stitching data. Overlapping inputs that have already participated in convolution before cropping retain their function for correctly calculating edge outputs.

[0159] Alternatively, in the implementation of splicing followed by cropping, the processing device 110 can first form an output representation with redundant regions according to the positional relationships determined at compile time, and then remove the redundant parts according to the valid output positions. For repeated regions located between blocks, the cropping rules need to preserve the corresponding internal positional relationships and cannot simply crop the outer edges of the entire output. Both orders are based on forming the same logical output range.

[0160] In the 1920×1080 example using a 3×2 partition, the static subgraph can pre-contain the reading and computation arrangements for six spatial blocks. Each block undergoes single-sided or double-sided overlapping reading according to its location, preserving the hardware-aligned storage range. After execution, the static subgraph trims the output of each block according to predetermined rules and concatenates them to form the complete result corresponding to the actual input.

[0161] By distinguishing between overlapping inputs used for convolution dependencies and redundant outputs that should not be retained, the processing device 110 is able to recover the original network's logical output after block computation. Effective range pruning also ensures that subsequent readings and passes are only directed to the desired output regions, avoiding the continued processing of alignment padding and repeated edges as valid results.

[0162] After the spatial partitioning is determined, static compilation can further adapt to the hardware constraints of the channel dimension. Input channel partitioning and output channel partitioning can employ at least one of these methods. In some embodiments, the processing device 110 can divide the input activations and weights into multiple channel sub-intervals according to the same input channel interval, perform convolution calculations for each channel sub-interval separately, and perform element-wise summation on the outputs of each convolution calculation. When the convolution calculation includes bias calculation, the bias is not repeatedly applied during the convolution calculation of each channel sub-interval, and a bias is applied only after the element-wise summation.

[0163] Input channel splitting decomposes the convolution summation at the same output position into several partial contributions. Therefore, each branch needs to read the input activation corresponding to the same input channel interval, as well as the portion of the weights corresponding to that input channel interval. The outputs of each branch have an element-wise additive relationship; simple concatenation along the channels cannot replace addition.

[0164] exist Figure 5 In the strategy recording and classification in steps 522 and 524, processing device 110 compares not only the number of height and width segments, but also the segmentation methods of input and output channels. If the network does not require channel segmentation, it can be grouped according to the corresponding spatial structure and other selected structural features. If the spatial segmentation is the same but the channel branch structure is different, processing device 110 retains the corresponding structural differences.

[0165] By using both spatial and channel strategies as the basis for reuse, the processing device 110 can enable the selected static subgraph to simultaneously satisfy multiple hardware constraints. Branch addition, channel splicing, and spatial pruning splicing are applied to their respective dimensions, ensuring that the overall computational relationship of the network is maintained after the combined segmentation.

[0166] After the segmentation methods and output processing arrangements are determined, the processing device 110 can re-determine the on-chip cache usage of each candidate segmentation block. If the usage of each candidate segmentation block does not exceed the on-chip cache limit, the processing device 110 can generate a static subgraph that includes segmentation block calculation, output pruning, and output splicing.

[0167] A complete static compilation organizes the predetermined operator connections, branch counts, data movement, memory allocation, and pruning / concatenation order into executable code for the target processing unit. Overlapping reads and alignment are incorporated into the input arrangement of corresponding blocks, and the addition or concatenation corresponding to channel splitting are also incorporated into the static operation structure. This static subgraph is then associated with its representative input size and size region.

[0168] By pre-programming the resource-verified blocks and result aggregation into a static subgraph, the processing unit can execute according to a defined computation and storage arrangement. During runtime, these sub-branches do not need to be regenerated, and the block outputs can be restored to complete logical outputs according to existing rules.

[0169] In step 250, the processing device 110 generates the compilation result of the convolutional neural network. The compilation result includes multiple static subgraphs and a size region table, which records the mapping relationship between multiple size regions and multiple static subgraphs.

[0170] In some embodiments, the compilation result can be a dynamic model package. The processing device 110 can write the target code, size region table, shared parameter file, and hardware configuration identifier of each static subgraph into the dynamic model package. The processing device 110 can also save the corresponding version information in conjunction with the target hardware and runtime interface.

[0171] The dynamic model package organizes size selection information together with executable static results. A size region table determines which static subgraph should be used for a given input, the target code executes that static subgraph, and a shared parameter file provides the corresponding weights. Hardware configuration identifiers and version information describe the execution environment to which this code and data are adapted, preventing the understanding of the model package from being detached from its target configuration.

[0172] Figure 3 The dynamic model package 330 is formed by generating the static subgraph 327 and establishing the shared parameters 326. Figure 7A schematic diagram of a process 700 for a model package and an example of its operation according to some embodiments of the present disclosure is shown. Figure 7 In this model package, the size region table 702, the static subgraph set 704, the shared parameters 706, and the mapping information 708 constitute the interconnected contents of the model package.

[0173] By saving the static subgraph and its selection criteria, the processing device 110 generates a compiled result that can be deployed. The execution device 340 can determine the execution object based on the mapping already established in the model package, reducing the work of re-analyzing the network structure and reconstructing the relationship between size and code at runtime.

[0174] In organizing the aforementioned dynamic model package, in some embodiments, the processing device 110 can identify weight data with identical content in multiple static subgraphs. The processing device 110 can store a copy of the identical weight data in the dynamic model package and establish a shared index between the multiple static subgraphs and the stored weight data.

[0175] The "same content" here refers to the identical weight data that can be used by related static subgraphs, not just the same weight names. Static subgraphs of different sizes may use the same network parameters, but their segmentation and referencing ranges may differ. Processing device 110 uses this to distinguish between the shared storage of data content and the usage relationship of corresponding data by each subgraph.

[0176] Processing device 110 can compare the required weight content and data representation of each subgraph to determine the overlapping weights. For a set of weight data that can be referenced together, processing device 110 retains the corresponding data in a shared parameter file and records its locatability information. The parameter references of each static subgraph are then associated with this data through a shared index.

[0177] If a subgraph only uses the corresponding channel portion of the weights, the parameter references must maintain their correspondence with the channel ranges required by its convolutional branch. When different subgraphs have different weight contents or different representations that cannot be directly shared, the processing device 110 retains the corresponding contents respectively. Shared processing does not come at the cost of deleting the parameter differences required for the subgraphs.

[0178] exist Figure 7 In the process, shared parameter 706 can contain several parameter data items, and mapping information 708 establishes the relationship between the size region, static subgraphs, and related parameter data items. Multiple subgraphs in the static subgraph set 704 can reference the same parameter data item. The running program parses the corresponding index based on the selected subgraph, allowing the use of the already saved weights.

[0179] By sharing and indexing repeated weights, the dynamic model package can support multiple static subgraphs without needing to save the complete repeated weights for each subgraph. With shared parameters already loaded, switching subgraphs can continue using existing parameter mappings, thus reducing the storage and loading overhead of repeated weights.

[0180] After the static subgraphs and shared parameters have been established, the size region table further associates the actual input dimensions with this execution data. The processing device 110 can record the height and width range or allowed size combinations for each region, and record the corresponding static subgraph identifier. Spatial segmentation, channel segmentation, and related clipping rules can be saved along with the region records or subgraph association information.

[0181] Figure 7 Table 702 shows the size ranges, height and width ranges, and sub-areas. Figure 3 This mapping is illustrated. Processing device 110 can generate actual entries according to the determined endpoint affiliation, enabling a valid input to determine the corresponding target region.

[0182] For irregular or discontinuous size regions, entries can contain multiple ranges or allowed combinations of sizes associated with the static subgraph. For inputs where height and width fall into different segments, the query needs to compare both dimensions simultaneously. The process of selecting a subgraph only by the largest side length is only applicable if the corresponding region has been verified to cover the input and the same structure can be reused.

[0183] By retaining complete two-dimensional matching conditions and subgraph associations in the dimension region table, the execution device 340 can use real-dimensional positioning of static execution results. This table passes the structural analysis results from compile time to the runtime phase, keeping the drawing selection behavior within the already confirmed reusability range.

[0184] In generation Figure 2 After the compilation result of step 250, the dynamic model package can also be used for actual network operation. In some embodiments, the processing device 110 can load the dynamic model package, receive the input size to be processed of the convolutional neural network, and determine the target size region to which the input size to be processed belongs based on the size region table.

[0185] The processing device 110 can select a target static subgraph corresponding to a target size region and align the input activations according to the tensor alignment granularity indicated by the configuration information. The processing device 110 also arranges the data elements of the input activations according to the tensor layout indicated by the configuration information and executes the target static subgraph using the processing unit. Based on the output size corresponding to the input size to be processed, the processing device 110 clips the aligned fill region and overlapping output region in the execution result.

[0186] The above-mentioned operating functions can be provided by Figure 3and Figure 7 The execution device 340 in the middle undertakes this task. Figure 7 In the process, the actual input 710 includes the input tensor and its height and width, and the actual dimensions are provided to both the subgraph selection 712 and the input processing 714. The subgraph selection 712 reads the size region table 702, determines the region, and selects the target subgraph from the static subgraph set 704.

[0187] The program can load the size region table, static subplots, and weight data when loading the dynamic model package. With the corresponding code and weights already loaded, the plot selection process can determine the execution target from the existing subplots and determine its parameter references based on mapping information 708. Alternatively, the required subplots can be loaded after the target subplot has been determined, provided that the corresponding target code and parameter mappings have been provided by the model package.

[0188] Input processing 714 aligns the actual input according to hardware requirements and converts it to the target tensor layout. The alignment process preserves the actual height and width and fills in the necessary space for physical storage or computation. The layout transformation arranges the same data elements according to the target dimension order and channel organization, without changing the correspondence between these elements in the logical input.

[0189] The static graph execution unit 716 receives the target subgraph, the processed input, and the corresponding shared parameters, and executes them using a processing unit that supports static graphs. Spatial block reading, convolutional branching, channel summation or concatenation, and output pruning and concatenation can be performed according to the pre-compiled schedule. The execution device 340 does not need to perform structural boundary searches or generate deployment static graphs every time an input arrives.

[0190] Output processing 718 determines the logical range to be retained based on the output size corresponding to the actual input. This output size is jointly determined by the operation parameters of each operator in the network and the actual input, and does not necessarily have to be equal to the height and width of the input. Execution device 340 removes the alignment padding region and overlapping output region accordingly to obtain the logical output 720.

[0191] If the output layout of the processing unit differs from the required external layout, the output processing 718 can also perform format conversion. Format conversion arranges the valid results into the required logical output form while preserving the spatial and channel positional relationships. Output clipping and format conversion are linked by pre-defined range and layout information and cannot simply return based on the full length of the physical output buffer.

[0192] By using table lookups, alignment, static execution, and pruning according to the actual output range, execution device 340 can return corresponding logical results for varying input sizes. Size-related structure search and deployment compilation have been moved forward to the compilation stage, thereby reducing the need for complex graph scheduling at runtime to adapt to input sizes.

[0193] For example, when the actual input is 1280×736, the execution device 340 can first read the values ​​of these two dimensions and query the size region table 702. After determining the target region to which the size belongs, the execution device 340 selects the corresponding static subgraph and the associated shared weights.

[0194] Both 1280 and 736 can be aligned to 8, so the overall input height and width do not need to be increased at this alignment granularity. If the segmentation blocks, output tensors, or channel dimensions still need to be filled with alignment positions, the execution device 340 processes them according to the layout arrangement of the target static subgraph. After the static graph is executed, the execution device 340 trims, aligns, fills, and overlaps the output according to the actual output size, and then transforms the output layout as needed.

[0195] In an example where the actual input is 1920×1080 and the corresponding region is divided into 3×2 segments, the execution device 340 selects a static subgraph from the size region table that already contains the segmentation strategy. The processing unit performs overlapping reading and block operation of six spatial blocks according to a preset arrangement. Subsequently, the processing unit or the corresponding runtime process completes the output clipping and splicing according to the static graph specifications, returning the complete output corresponding to the actual input.

[0196] For the 800×800 example, the execution device 340 can select the second static subgraph from the aforementioned verified four-region example. For the 1700×1700 example, the execution device 340 can select the fourth static subgraph and record both the actual size and the physical size after upward alignment. The clipping uses the output range corresponding to the actual input, avoiding returning the alignment extension position as actual data.

[0197] For inputs such as 800×1700 with height and width spanning different segments, the execution device 340 determines the corresponding subgraph based on a two-dimensional dimension region table. In embodiments where a static subgraph representing a larger dimension already covers the input, the execution device 340 may choose to cover a subgraph with a larger dimension. This selection relies on established structural and range correspondences, rather than solely on the value of 1700.

[0198] In embodiments where the target hardware supports computation controlled based on the effective input range, the processing unit can skip computations exceeding the actual effective range within the structure of the selected static subgraph. If the execution still involves a large physical range, the output processing can remove parts that do not belong to the actual result according to predetermined rules. This effective range processing does not change the predetermined subgraph branches and connection relationships.

[0199] By recording the actual size, physical alignment range, and coverage area of ​​the target subgraph separately, the execution device 340 can process square and non-square inputs under static execution arrangements. Clipping and layout transformations ensure that the returned data corresponds to the actual input, and hardware with effective range control capabilities can also reduce invalid calculations that exceed the actual range.

[0200] Figure 8 A schematic diagram illustrating an example process of channel segmentation according to some embodiments of the present disclosure is shown. Figure 8 In input channel segmentation and summation (a), the first segmented data 802, the second segmented data 804, and the third segmented data 806, based on the input and weights, are fed into the first convolution branch 812, the second convolution branch 814, and the third convolution branch 816, respectively. The convolution branches generate the first part output 822, the second part output 824, and the third part output 826, respectively. Summation 832 combines these part outputs into a complete output 834.

[0201] For example, the input channels can be divided into two, three, or more parts. The processing device 110 determines the same input and weight channel index interval for each part, ensuring that different sub-intervals jointly cover the input channels that need to participate in the convolution. Each branch calculates its contribution to the input channels it is responsible for for the same output position.

[0202] For convolutions that do not include bias, Figure 8 The summation relationship in the expression can be represented by the following formula. (Full output follows) It means that the first The partial output generated by each input channel sub-interval is indexed. of This indicates that the outputs of each part are summed element-wise at the same position, and the summation range is the sum of each input channel branch that participated in the calculation of that output.

[0203] When bias is included, the outputs of the aforementioned parts first represent the convolutional contributions that have not yet been repeatedly biased. The processing device 110 may not apply bias in each branch, and apply bias only once after the summation 832 forms the complete contribution. Thus, the number of times bias is applied does not increase with the number of input channel segments.

[0204] Even if the space block meets the on-chip cache requirements, the number of channels may still exceed the channel specification of the processing unit. The processing device 110 can handle this situation by splitting the input channels and allocating storage for some results and accumulation during compilation. The branch granularity is determined based on the hardware configuration and is not limited to... Figure 8 The indicated number of branches.

[0205] By synchronously splitting the input and weights and accumulating the output at the same position, the processing device 110 can complete the original convolution using multiple branches that comply with the channel limit. The bias is applied only once to maintain the bias meaning of the original convolution in the output, avoiding the introduction of duplicate biases due to changes in the number of branches.

[0206] In addition to adapting to input channel limitations, static compilation can also form other branches based on the output channel size. In some embodiments, the processing device 110 can divide the weights into multiple weight sub-intervals according to the output channels, perform convolution calculations for each weight sub-interval respectively, and concatenate the outputs of each convolution calculation along the output channel direction.

[0207] Output channel splitting assigns different branches to different sets of output channels. Each branch can share the corresponding input activation, but reads the weights of its respective output channel. Each branch forms a channel portion of the complete output, and thus the results of each branch are concatenated in the order of the output channels.

[0208] Figure 8 Figure (b) illustrates this process by showing the output channel segmentation and splicing. The first combination 842, the second combination 844, and the third combination 846 of the input and weights use a common input and different output channel weight sub-intervals, respectively, and are fed into the fourth convolution branch 852, the fifth convolution branch 854, and the sixth convolution branch 856. The first output 862, the second output 864, and the third output 866 correspond to each output channel portion and are combined along the output channel direction to form a complete output 870.

[0209] Processing device 110 can record the output channel position of each weight sub-interval, ensuring that the generated result maintains the same output channel order as the original network. For output channel branches that require alignment, the physical channel range can include the alignment position, and the final result retains valid channels based on the actual channel range. The results of different output channels are not mixed by element-wise summation.

[0210] By calculating and concatenating the output channels separately, the processing device 110 can organize the calculation of larger output channels into branches that conform to hardware limitations. This process preserves the independent meaning of each output channel and avoids requiring the convolution of all output channels at once to generate different output channels.

[0211] The aforementioned channel segmentation can also be combined with Figure 5 and Figure 6 The processing device 110 may first determine the height and width partitioning based on cache occupancy, overlap, and alignment requirements, and then determine the input or output channel partitioning based on the hardware constraints corresponding to each space block. The processing device 110 may record these strategies together as a static execution arrangement for the region.

[0212] Input channel segmentation and output channel segmentation can be used separately or in combination with height and width segmentation. For a spatial block that uses both input and output channel segmentation, the processing device 110 can aggregate the contributions of each input channel sub-interval within the corresponding output channel set. After completing the corresponding offset processing, the output channel sets are then spliced ​​together according to their original channel positions.

[0213] The cache base, already determined by spatial partitioning, still needs to be coordinated with the actual channel branches. Processing device 110 can verify the actual residency requirements resulting from partial output, accumulation buffering, and channel alignment, ensuring that the final strategy simultaneously satisfies both cache capacity and channel granularity. This sequence avoids determining the space block size solely based on the full channel size, ignoring the relationship between channel constraints and space block size.

[0214] Through the complete process described above, the processing device 110 can form size regions based on structural changes within the dynamic input size range and generate an executable representative static subgraph for each region. Multiple input sizes with the same static structure are associated with the same region, thus reducing the number of complete static compilations and the repeated generation of deployment results when these sizes can be reused.

[0215] Spatial partitioning search incorporates overlapping reads, hardware alignment, and actual cache usage into the computation, while channel partitioning further adapts to the constraints of input and output channels. Various partitions preserve the network's logical output through corresponding summation, pruning, and concatenation. Thus, larger inputs or intermediate activations can be executed in computational blocks that meet on-chip resource requirements.

[0216] The dynamic model package stores the mapping between regions and subgraphs, and reuses the same weights through a shared index. The execution device 340 selects the static subgraph based on the actual input, and completes the runtime processing according to the layout and effective output range. This technology chain supports multiple input sizes while maintaining the static graph execution method, and reduces the work of storing repetitive parameters and recompiling and structure searching during the runtime phase.

[0217] In this way, the processing device 110 can establish a correspondence between size regions and executable static subgraphs for different input sizes. Multiple candidate input sizes in the same region can reuse the same static structure, thereby reducing the work of generating deployment static subgraphs for each size. The processing unit still executes according to the pre-formed static graph, and there is no need to re-search for the compilation structure to obtain the corresponding static graph during the runtime phase.

[0218] A corresponding apparatus for implementing the above methods or processes is also provided. Figure 9A schematic structural block diagram of an apparatus 900 for compiling an example of a convolutional neural network according to some embodiments of the present disclosure is shown. The apparatus 900 may be implemented as or included in the processing device 110. The various modules / components in the apparatus 900 may be implemented by hardware, software, firmware, or any combination thereof.

[0219] like Figure 9 As shown, the apparatus 900 includes: an acquisition module 910 configured to acquire a convolutional neural network to be compiled, the input size range of the convolutional neural network, and configuration information of a processing unit, wherein the processing unit supports the execution of static graphs, and the configuration information includes at least tensor alignment granularity; a feature determination module 920 configured to select multiple candidate input sizes within the input size range based on the tensor alignment granularity, and determine the compiled structural features of the convolutional neural network under the multiple candidate input sizes respectively; a boundary determination module 930 configured to determine at least one size boundary based on the differences between the compiled structural features corresponding to the multiple candidate input sizes, and divide the input size range into multiple size regions using the at least one size boundary. This division causes candidate input sizes within the same size region to correspond to the same static graph structure; and a static compilation module 940 configured to determine a representative input size from each size region, and perform static compilation of the convolutional neural network according to the representative input size to obtain multiple static subgraphs that support execution by the processing unit. Multiple static subgraphs correspond to multiple size regions respectively; the network compilation module 950 is configured to generate the compilation result of the convolutional neural network, which includes multiple static subgraphs and a size region table, and the size region table records the mapping relationship between multiple size regions and multiple static subgraphs.

[0220] In some embodiments, the input size range may include the height and width range of the input activation tensor. The feature determination module 920 may also be configured to discretize the height and width ranges into combinations of multiple height and width values ​​based on tensor alignment granularity, obtaining an aligned input size grid. The feature determination module 920 may also be configured to select multiple candidate input sizes from the input size grid and perform analytical compilation on the convolutional neural network using each candidate input size. Analytical compilation generates compilation results for extracting compilation structure features, but does not generate static subgraphs for deployment. The feature determination module 920 may also be configured to extract corresponding compilation structure features from the compilation results of each analytical compilation.

[0221] In some embodiments, the boundary determination module 930 may further be configured to group candidate input dimensions based on the differences between the compilation structure features corresponding to multiple candidate input dimensions, thereby obtaining at least one size region. Candidate input dimensions within the same size region correspond to the same compilation structure features, while candidate input dimensions in different size regions correspond to different compilation structure features. The boundary determination module 930 may further be configured to take the union of size boundaries obtained from different operators or different compilation optimization stages, and sort the obtained size boundaries according to their height and width values, thereby determining at least one size boundary. This at least one size boundary is the boundary of the corresponding size region.

[0222] In some embodiments, the compilation structure features may include at least two of the following: the number of height-direction segments of the convolution operator; the number of width-direction segments; the input channel segmentation method; the output channel segmentation method; the number of graph partitions; the shape of the output tensor; the pooling boundary; the output pruning rule; and the on-chip cache usage strategy.

[0223] In some embodiments, the static compilation module 940 can also be configured to sort candidate input sizes within a corresponding size region according to the amount of input activation data corresponding to multiple candidate input sizes. The candidate input size ranked higher is greater than the candidate input size ranked lower. The static compilation module 940 can also be configured to perform static compilation on multiple candidate input sizes based on the sorting result. And if no static subgraph executable by the processing unit is generated, the static compilation module 940 can sequentially select the next candidate input size in the sorting result to perform static compilation until a representative input size capable of completing static compilation is determined.

[0224] In some embodiments, the configuration information may further include an on-chip cache limit. The static compilation module 940 may also be configured to, for any representative input size, obtain the overlap and alignment correction amount of the convolutional layer when segmenting in the height and width directions. The static compilation module 940 may also be configured to determine the on-chip cache occupancy of the complete graph or candidate segmentation blocks of the convolutional neural network, the occupancy including the storage space occupied by at least one of input activation, output activation, weights, intermediate buffers, and alignment padding. The static compilation module 940 may also be configured to, if the occupancy exceeds the on-chip cache limit, determine the input size of each candidate segmentation block based on the number of segments in the height direction, the number of segments in the width direction, the overlap, and the alignment correction amount. The static compilation module 940 may also be configured to determine the on-chip cache occupancy of each candidate segmentation block. The static compilation module 940 may also be configured to, if the on-chip cache occupancy of each candidate segmentation block does not exceed the on-chip cache limit, generate a static subgraph including segmentation block computation, output pruning, and output concatenation.

[0225] In some embodiments, the static compilation module 940 can also be configured to determine the initial number of segments based on the ratio of the occupied portion to the on-chip cache limit. The static compilation module 940 can also be configured to allocate the initial number of segments into a height-direction segmentation number and a width-direction segmentation number based on the ratio between the height and width values ​​representing the input dimensions. The static compilation module 940 can also be configured to calculate the candidate segmentation block with the largest on-chip cache occupancy under the current height-direction and width-direction segmentation numbers. The static compilation module 940 can also be configured to increase at least one of the height-direction and width-direction segmentation numbers to determine the height-direction and width-direction segmentation numbers if the on-chip cache occupancy of a candidate segmentation block exceeds the on-chip cache limit or the channel segmentation granularity does not meet the configuration information.

[0226] In some embodiments, the static compilation module 940 can also be configured to determine the input region corresponding to each segmentation block based on the convolution kernel size, stride, dilation rate, padding parameters, operational parameters of the previous and next operators connected to the convolutional layer, and tensor alignment granularity. The static compilation module 940 can also be configured to, along each segmentation direction in the height and width directions, extend the overlap of the input region of the segmentation block located at the boundary of that segmentation direction to the side where the adjacent segmentation block is located, and extend the overlap of the input region of the segmentation block located at a non-boundary position in that segmentation direction to the opposite sides respectively. The static compilation module 940 can also be configured to, before or after output concatenation, trim redundant output regions generated by at least one of convolution overlap, alignment padding, and data shifting granularity.

[0227] In some embodiments, the static compilation module 940 may also be configured to further include at least one of the following: The static compilation module 940 may also be configured to divide the input activations and weights into multiple channel sub-intervals according to the same input channel interval, perform convolution calculations for each channel sub-interval respectively, and perform element-wise summation on the outputs of each convolution calculation. Where the convolution calculation includes bias calculation, the bias is not applied repeatedly during the convolution calculation of each channel sub-interval, and a bias is applied only once after element-wise summation. The static compilation module 940 may also be configured to divide the weights into multiple weight sub-intervals according to the output channel, perform convolution calculations for each weight sub-interval respectively, and concatenate the outputs of each convolution calculation along the output channel direction.

[0228] In some embodiments, the compilation result can be a dynamic model package. The static compilation module 940 can also be configured to identify identical weight data in multiple static subgraphs and establish a shared index between the multiple static subgraphs and the stored weight data. The network compilation module 950 can also be configured to store a copy of the identical weight data in the dynamic model package. The network compilation module 950 can also be configured to write the target code, size region table, shared parameter file, and hardware configuration identifier of each static subgraph into the dynamic model package.

[0229] In some embodiments, the apparatus 900 may further be configured to load a dynamic model package, receive the input size to be processed from the convolutional neural network, and determine the target size region to which the input size to be processed belongs based on a size region table. The apparatus 900 may also be configured to select a target static subgraph corresponding to the target size region, align the input activations according to the tensor alignment granularity indicated by the configuration information, arrange the data elements of the input activations according to the tensor layout indicated by the configuration information, and execute the target static subgraph using the processing unit. The apparatus 900 may also be configured to clip the aligned padding region and overlapping output region in the execution result based on the output size corresponding to the input size to be processed.

[0230] The modules included in device 900 can be implemented in various ways, including software, hardware, firmware, or any combination thereof. In some embodiments, one or more modules can be implemented using software and / or firmware, such as machine-executable instructions stored on a storage medium. In addition to or as an alternative to machine-executable instructions, some or all of the units in device 900 can be implemented at least partially by one or more hardware logic components. By way of example and not limitation, exemplary types of hardware logic components that can be used include field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard parts (ASSPs), systems on a chip (SOCs), complex programmable logic devices (CPLDs), and so on.

[0231] Figure 10 A block diagram of an electronic device 1000 in which one or more examples can be implemented is shown. It should be understood that... Figure 10 The electronic device 1000 shown is merely exemplary and should not be construed as limiting the functionality and scope of the examples described in this disclosure. Figure 10The electronic device 1000 shown can be used to implement the processing device 110 discussed above.

[0232] like Figure 10 As shown, electronic device 1000 is in the form of a general-purpose electronic device. Components of electronic device 1000 may include, but are not limited to, one or more processing units or processors 1010, memory 1020, storage device 1030, one or more communication units 1040, one or more input devices 1050, and one or more output devices 1060. Processor 1010 may be a physical or virtual processor and is capable of performing various processes according to programs stored in memory 1020. In a multiprocessor system, multiple processors execute computer-executable instructions in parallel to improve the parallel processing capability of electronic device 1000.

[0233] Electronic device 1000 typically includes multiple computer storage media. Such media can be any accessible media that is accessible to electronic device 1000, including but not limited to volatile and non-volatile media, removable and non-removable media. Memory 1020 can be volatile memory (e.g., registers, cache, random access memory (RAM)), non-volatile memory (e.g., read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof). Storage device 1030 can be removable or non-removable media and may include machine-readable media, such as flash drives, disks, or any other media capable of storing information and / or data and accessible within electronic device 1000.

[0234] Electronic device 1000 may further include additional removable / non-removable, volatile / non-volatile storage media. Although not explicitly stated... Figure 10 As shown, disk drives for reading from or writing to removable, non-volatile disks (e.g., "floppy disks") and optical disk drives for reading from or writing to removable, non-volatile optical disks can be provided. In these cases, each drive can be connected to a bus (not shown) via one or more data media interfaces. Memory 1020 may include computer program product 1025 having one or more program modules configured to perform various methods or actions of various examples.

[0235] The communication unit 1040 enables communication with other electronic devices via a communication medium. Additionally, the functionality of the components of the electronic device 1000 can be implemented using a single computing cluster or multiple computers capable of communicating via communication connections. Therefore, the electronic device 1000 can operate in a networked environment using logical connections to one or more other servers, networked personal computers, or another network node.

[0236] Input device 1050 can be one or more input devices, such as a mouse, keyboard, trackball, etc. Output device 1060 can be one or more output devices, such as a monitor, speaker, printer, etc. Electronic device 1000 can also communicate with one or more external devices (not shown) via communication unit 1040 as needed. External devices may include storage devices, display devices, devices for enabling user interaction with electronic device 1000, or devices for enabling electronic device 1000 to communicate with one or more other electronic devices (e.g., network card, modem, etc.). Such communication can be performed via input / output (I / O) interface (not shown).

[0237] A computer-readable storage medium is provided, having stored thereon computer-executable instructions, wherein the computer-executable instructions are executed by a processor to implement the method described above. The computer-readable storage medium can be disposed in an electronic device or communicatively connected to an electronic device. The electronic device may include a mobile phone, tablet computer, personal computer, image editing device, server, or other device with image processing capabilities.

[0238] A computer program product is also provided, comprising computer-executable instructions. When executed by a processor, the computer-executable instructions cause the processor to implement the methods or processes described above. The computer program product can run on or be implemented in an electronic device, or it can be run collaboratively by the electronic device and a server. The electronic device can implement the methods described above, either alone or in collaboration with a server, through the computer program product. The computer program product may include, for example, applications running on the device, functional modules, application plugins, software installation packages, firmware upgrade packages, client programs, or cloud programs. For example, as described above, the computer program product may include a compiler application implementing a convolutional neural network, a network processing application, a model deployment application, a network processing functional module, an application plugin, a client program, or a compiler for a cloud-based convolutional neural network. The electronic device can implement the methods or processes described in the embodiments of this disclosure, for example, by running the computer program product. For example, by running a computer program product, an electronic device can obtain the configuration information of the convolutional neural network to be compiled, the input size range, and the processing unit that supports static graph execution, and determine the candidate input size and corresponding compilation structure features based on tensor alignment granularity; determine the size boundary and divide the size region based on the differences in the compilation structure, determine the representative input size from each region and generate the corresponding static subgraph; and generate a compilation result containing multiple static subgraphs and a size region table.

[0239] A computer program product is tangibly stored on a non-transient computer-readable medium and includes computer-executable instructions, and is installed, loaded, or invoked into an electronic device, wherein the computer-executable instructions are executed by the processor of the electronic device to implement the methods described above.

[0240] The flowcharts and / or block diagrams of the methods, apparatus, devices, and computer program products referred to herein describe various aspects. It should be understood that each block of the flowcharts and / or block diagrams, as well as combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer-readable program instructions.

[0241] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0242] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions that execute on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0243] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to some embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which contains one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0244] Various examples have been described above. The foregoing description is exemplary and not exhaustive, nor is it limited to the disclosed implementations. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described implementations. The terminology used in this disclosure is chosen to best explain the principles, practical applications, or improvements to technology in the market, or to enable others skilled in the art to understand the various implementations of this disclosure.

Claims

1. A compilation method for a convolutional neural network, characterized in that, include: Obtain the convolutional neural network to be compiled, the input size range of the convolutional neural network, and the configuration information of the processing unit. The processing unit supports the execution of static graphs, and the configuration information includes at least tensor alignment granularity. Based on the tensor alignment granularity, multiple candidate input sizes are selected within the input size range, and the compilation structure features of the convolutional neural network under the multiple candidate input sizes are determined respectively. Based on the differences between the compilation structure features corresponding to the multiple candidate input sizes, at least one size boundary is determined, and the input size range is divided into multiple size regions using the at least one size boundary, so that the candidate input sizes in the same size region correspond to the same static graph structure. A representative input size is determined from each size region, and the convolutional neural network is statically compiled according to the representative input size to obtain multiple static subgraphs that support execution by the processing unit, wherein the multiple static subgraphs correspond to the multiple size regions respectively; and The compilation result of the convolutional neural network is generated. The compilation result includes the plurality of static subgraphs and a size region table. The size region table records the mapping relationship between the plurality of size regions and the plurality of static subgraphs.

2. The method according to claim 1, characterized in that, The input size range includes the height and width range of the input activation tensor, and the step of selecting multiple candidate input sizes within the input size range based on the tensor alignment granularity, and determining the compilation structure features of the convolutional neural network under the multiple candidate input sizes respectively includes: Based on the tensor alignment granularity, the height range and the width range are discretized into a combination of multiple height values ​​and multiple width values ​​to obtain the aligned input size mesh; The plurality of candidate input sizes are selected from the input size grid, and the convolutional neural network is analytically compiled using each candidate input size. This analytical compilation generates compilation results for extracting the compiled structural features, but does not generate static subgraphs for deployment. Extract the corresponding compilation structure features from the compilation results of each analytical compilation.

3. The method according to claim 2, characterized in that, Determining at least one size boundary based on the differences between the compilation structure features corresponding to the multiple candidate input sizes includes: Based on the differences in the compilation structure features corresponding to the multiple candidate input sizes, the candidate input sizes are grouped to obtain at least one size region, wherein the compilation structure features corresponding to candidate input sizes within the same size region are the same, and the compilation structure features corresponding to candidate input sizes in different size regions are different; and The union of the size boundaries obtained by different operators or different compilation optimization stages is taken, and the obtained size boundaries are sorted according to the height and width values ​​to determine the at least one size boundary, wherein the at least one size boundary is the boundary of the corresponding size region.

4. The method according to any one of claims 1 to 3, characterized in that, The compilation structure features include at least two of the following: the number of height-direction segments of the convolution operator; the number of width-direction segments; the input channel segmentation method; the output channel segmentation method; the number of graph partitions; the shape of the output tensor; the pooling boundary; the output pruning rules; and the on-chip cache usage strategy.

5. The method according to claim 1, characterized in that, The step of determining the representative input dimension from each dimension region includes: Based on the amount of input activation data corresponding to the plurality of candidate input sizes, the candidate input sizes within the corresponding size region are sorted, wherein the candidate input size sorted first is greater than the candidate input size sorted later; and Based on the sorting results, static compilation is performed on the multiple candidate input sizes. If no static subgraph that can be executed by the processing unit is generated, the next candidate input size in the sorting results is selected sequentially for static compilation until a representative input size that can be statically compiled is determined.

6. The method according to claim 1, characterized in that, The configuration information also includes an on-chip cache limit, and the static compilation of the convolutional neural network according to the representative input size includes: For any representative input size, obtain the overlap and alignment correction amount of the convolutional layer when it is segmented in the height and width directions; Determine the on-chip cache usage of the complete graph or candidate segment of the convolutional neural network, the usage including the storage space occupied by at least one of input activation, output activation, weights, intermediate buffers and alignment padding; If the occupied amount exceeds the on-chip cache limit, determine the number of segments in the height direction and the number of segments in the width direction, and determine the input size of each candidate segment based on the number of segments in the height direction, the number of segments in the width direction, the overlap amount, and the alignment correction amount; Determine the on-chip cache usage of each candidate partition block; and If the on-chip cache usage of each candidate segmentation block does not exceed the on-chip cache limit, a static subgraph is generated, which includes segmentation block calculation, output pruning, and output splicing.

7. The method according to claim 6, characterized in that, The configuration information also includes channel segmentation constraints, and determining the number of segments in the height direction and the number of segments in the width direction includes: The initial number of segments is determined based on the ratio of the occupied amount to the upper limit of the on-chip cache; Based on the ratio between the height and width values ​​representing the input dimensions, the initial number of segments is allocated as a number of segments in the height direction and a number of segments in the width direction; Calculate the candidate partition block with the largest on-chip cache usage given the current number of partitions in the height and width directions; and If the on-chip cache usage of the candidate segmentation block exceeds the on-chip cache limit or the channel segmentation granularity does not meet the channel segmentation constraint, at least one of the height segmentation number and the width segmentation number is increased to determine the height segmentation number and the width segmentation number.

8. The method according to claim 6, characterized in that, Also includes: Based on the kernel size, stride, dilation rate, padding parameters, the operation parameters of the previous and next operators connected to the convolutional layer, and the tensor alignment granularity, the input region corresponding to each segment is determined. Along each segmentation direction in both the height and width directions, the input region of the segmentation block located at the boundary of that segmentation direction is extended by the overlap amount to one side of the adjacent segmentation block, and the input region of the segmentation block located at a non-boundary position in that segmentation direction is extended by the overlap amount to the opposite sides respectively; and Before or after output concatenation, redundant output regions generated by at least one of convolution overlap, alignment padding, and data shifting granularity are trimmed.

9. The method according to claim 1, characterized in that, The static compilation of the convolutional neural network according to the representative input size further includes at least one of the following: The input activation and weights are divided into multiple channel sub-intervals according to the same input channel interval. Convolution calculations are performed on each channel sub-interval, and the outputs of each convolution calculation are summed element by element. In the case where the convolution calculation includes bias calculation, the bias is not applied repeatedly in the convolution calculation of each channel sub-interval, and a bias is applied once after the element-by-element summation. The weights are divided into multiple weight sub-intervals according to the output channel. Convolution calculations are performed on each weight sub-interval, and the outputs of each convolution calculation are concatenated along the output channel direction.

10. The method according to claim 1, characterized in that, The compilation result is a dynamic model package, and generating the compilation result includes: Identify weight data with identical content in the multiple static subgraphs; A copy of the weight data with the same content is stored in the dynamic model package, and a shared index is established between the multiple static subgraphs and the stored weight data; and Write the target code of each static subgraph, the size region table, the shared parameter file, and the hardware configuration identifier into the dynamic model package.

11. The method according to claim 10, characterized in that, The configuration information also includes a tensor layout, and the method further includes: Load the dynamic model package, receive the input size to be processed from the convolutional neural network, and determine the target size region to which the input size to be processed belongs based on the size region table; Select a target static subgraph corresponding to the target size region, align the input activations according to the tensor alignment granularity indicated by the configuration information, arrange the data elements of the input activations according to the tensor layout, and execute the target static subgraph using the processing unit; and Based on the output size corresponding to the input size to be processed, the aligned padding area and overlapping output area in the execution result are cropped.

12. A compilation apparatus for convolutional neural networks, characterized in that, include: The acquisition module is configured to acquire the convolutional neural network to be compiled, the input size range of the convolutional neural network, and the configuration information of the processing unit, wherein the processing unit supports the execution of static graphs, and the configuration information includes at least tensor alignment granularity. The feature determination module is configured to select multiple candidate input sizes within the input size range based on the tensor alignment granularity, and to determine the compilation structure features of the convolutional neural network under the multiple candidate input sizes respectively; The boundary determination module is configured to determine at least one size boundary based on the differences between the compilation structure features corresponding to the multiple candidate input sizes, and to divide the input size range into multiple size regions using the at least one size boundary, so that the candidate input sizes in the same size region correspond to the same static graph structure. A static compilation module is configured to determine a representative input size from each size region and statically compile the convolutional neural network according to the representative input size to obtain a plurality of static subgraphs that support execution by the processing unit, the plurality of static subgraphs corresponding to the plurality of size regions respectively; as well as The network compilation module is configured to generate the compilation result of the convolutional neural network. The compilation result includes the plurality of static subgraphs and a size region table, wherein the size region table records the mapping relationship between the plurality of size regions and the plurality of static subgraphs.

13. An electronic device, characterized in that, include: At least one processing unit; as well as At least one memory, coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, which, when executed by the at least one processing unit, cause the electronic device to perform the method according to any one of claims 1 to 11.

14. A computer-readable storage medium, characterized in that, It stores a computer program that can be executed by a processor to implement the method according to any one of claims 1 to 11.

15. A computer program product, characterized in that, The computer program product is tangibly stored in a computer storage medium and includes computer-executable instructions that, when executed by a device, cause the device to perform the method according to any one of claims 1 to 11.

Images