A gpu progressive optimization method for neural operator backbone network inference

Abstract

The application relates to a GPU progressive optimization method for neural operator backbone network reasoning and belongs to the electronic information technical field.The method comprises the following steps: step S1, initializing constraints and tolerances;step S2, obtaining baseline execution time and reference results through baseline measurement;step S3, obtaining an estimated split value based on a working set model and expanding;step S4, performing KSweep evaluation and consistency checking, and preferably obtaining; a process of scanning and measuring each candidate set is called KSweep; step S5, fixing, adding fusion reasoning optimization, and performing consistency verification, if the consistency verification is passed, outputting and saving the optimal configuration, and directly loading and executing in the running stage; otherwise, backtracking or adjusting and recalibrating, and returning to step S3. The application shortens the reuse distance between adjacent layers, reduces the round-trip overhead of intermediate activation between the video memory and the cache, and thus improves the reasoning efficiency.

Description

Technical Field

[0001] This invention relates to a GPU progressive optimization method for inference in neural operator backbone networks, belonging to the field of electronic information technology. Background Technology

[0002] As deep learning continues to be applied in scientific computing and engineering simulation, neural operator networks, as a model for learning function-to-function-space mappings, have gained widespread attention in problems such as partial differential equation solving, fluid dynamics calculations, and materials and multiphysics modeling. These models typically employ a trunk-branch structure, where the trunk network is often based on a multilayer perceptron. It performs layer-by-layer feature transformations on the input, which consists of a large number of discrete sampling points, and outputs the target physical quantity or intermediate representation. In large-scale inference scenarios, trunk networks often exhibit structural characteristics of "large number of sampling points, medium hidden dimension, and deep network layers," causing the activation data size to grow linearly with the number of sampling points. The activation working sets of a single layer and across layers often exceed the on-chip cache capacity of the GPU. If the traditional layer-by-layer approach is still used to compute all sampling points layer by layer, large-scale activations need to be repeatedly read and written between adjacent layers, the cross-layer reuse interval is lengthened, and intermediate activations are difficult to reside in the on-chip cache. This results in frequent data round trips to external storage, making inference performance more susceptible to limitations imposed by external storage bandwidth and access latency.

[0003] On the other hand, each layer of a trunk network typically includes post-processing operations such as bias stacking and activation functions in addition to linear transformations. If the linear transformation and post-processing are executed separately, intermediate results need to be written back and read in again, increasing the number of operator startups and scheduling, further amplifying external memory access and execution overhead. Existing GPU inference optimization techniques mostly focus on the throughput or latency metrics of general models, usually assuming that single-layer or a few-layer operators can be well reused at the on-chip cache level. For trunk network inference scenarios where the activation working set is significantly amplified by the number of sampling points, leading to a significant increase in cross-layer reuse distance, there is still a lack of a unified process for execution organization and parameter optimization oriented towards cache capacity constraints. As a result, problems such as insufficient cache utilization, frequent external memory access, and high overhead of separating operators are likely to occur.

[0004] In summary, the trunk network of a neural operator network often needs to perform forward computation on a large set of sampling points simultaneously during inference, resulting in a large number of sampling points. Increasing the size of the activation working set will lead to a significant expansion. If the traditional layer-by-layer approach is still used to calculate all sampled points layer by layer, intermediate activations will be difficult to reside in the on-chip cache between adjacent layers, resulting in frequent external memory accesses and limiting inference performance to external memory bandwidth and access latency. Furthermore, if post-processing such as linear transformations, bias superposition, and activation functions are performed separately within a single layer, intermediate results will generate additional read / write and scheduling overhead, further exacerbating memory access pressure and reducing throughput. Existing methods lack a granularity determination for on-chip cache capacity constraints and a reproducible optimization process, and also lack independent quantification and numerical consistency verification mechanisms for the contributions of partitioning scheduling and operator fusion. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides a GPU progressive optimization method for inference in neural operator backbone networks;

[0006] This invention proposes a progressive optimization method for inference in neural operator trunk networks. Without altering the semantics and accuracy constraints of the inference, it optimizes the method along the dimension of the number of sampling points. The input sampling point set is segmented, and a working set model is established based on the on-chip cache capacity constraint. The center estimates of the segmentation granularity and the number of segments are obtained, and a candidate set is constructed. The optimal number of segments is determined by KSweep measurement. .exist The method employs a tile-centric scheduling approach and uses double-buffered alternating writes to ensure correct data flow between layers, thereby shortening the cross-layer reuse distance and improving on-chip hit probability. Furthermore, within each layer of the trunk network, linear transformation and post-processing are organized into an integrated computational process as much as possible, reducing external storage round trips for intermediate results and suppressing scheduling overhead that may be amplified after segmentation. The method also introduces a baseline-segmentation-fusion progressive evaluation and consistency verification / rollback mechanism to ensure that the optimization results are stable and reproducible under the required accuracy.

[0007] The technical solution of this invention is as follows:

[0008] This invention addresses large-scale inference scenarios using neural operator networks (trunk networks). Without altering the semantics and accuracy constraints of network inference, it improves performance along the dimension of the number of sampling points. The input sampling point set is segmented, and the segmentation granularity and the number of candidate segments are determined based on the on-chip cache capacity constraint. The optimal number of segments is obtained by KSweep measurement. .exist The system employs a tile-centric scheduling approach and uses double-buffered alternating writes to ensure correct data flow between layers, thereby shortening cross-layer reuse distances, improving on-chip cache hits, and reducing external storage round trips. Furthermore, within each layer, linear transformations and post-processing are organized into an integrated computational process to reduce intermediate result read / write operations and suppress the amplification of scheduling overhead after partitioning. The optimal process is best suited for the calibration phase during initial deployment or when the input shape / hardware changes. After calibration, the optimal parameters are saved and directly loaded and executed during the runtime phase.

[0009] A GPU progressive optimization method for inference in neural operator backbone networks includes:

[0010] Step S1: Initialize constraints and tolerances;

[0011] Step S2: Baseline measurement to obtain baseline execution time Compared with reference results ;

[0012] Step S3: Obtain the estimated segmentation value based on the working set model And expand;

[0013] Step S4: KSweep evaluates the execution time of the program for the number of segments K. And perform a consistency check to select the best option. The process of scanning and measuring each candidate in the set is called KSweep.

[0014] Step S5: Fix Add fusion inference optimization and perform consistency verification. If the consistency verification passes, output and save the optimal configuration, and load and execute it directly during the runtime phase; otherwise, roll back or adjust and recalibrate, and return to step S3.

[0015] According to a preferred embodiment of the present invention, the specific implementation process of step S1 includes:

[0016] In the inference process of the backbone network of the neural operator network, the input includes a large number of sampling points, and the inference performs layer-by-layer forward computation on all sampling points; the backbone network of the neural operator network is defined with the following variables:

[0017] This represents the total number of sampling points that need to be processed in one inference operation. Represents the hidden dimensions of the backbone network; Indicates the number of layers in the backbone network; This indicates the capacity of the graphics processor's on-chip L2 cache; This indicates the L2 cache availability scaling factor. This indicates the number of bytes stored per element for activation and weight data.

[0018] According to a preferred embodiment of the present invention, the specific implementation process of step S2 includes:

[0019] A complete inference is performed using a baseline execution method without sampling point dimension splitting, i.e., for all... Each sampling point is completed sequentially according to the layer order. Perform layer-by-layer forward computation and record baseline execution time. Meanwhile, the output of this baseline execution is recorded as the reference result. .

[0020] According to a preferred embodiment of the present invention, the specific implementation process of step S3 includes:

[0021] After the variables and constraints are determined in steps S1 and S2, when When it is large, let the threshold be ,when Time is defined as Larger; the size of the activation matrix of each layer of the backbone network varies with... Linear growth;

[0022] Without altering the semantics of a single inference task, the set of sampling points is divided into several tiles, and the number of sampling points contained in a single tile is used as the basis for classification. As the granularity of segmentation, tile represents the subset of sampling points obtained by dividing along the dimension of the number of sampling points; Indicates the number of sampling points contained in a single tile; by the number of segments. Let the number of tiles be:

[0023] (1);

[0024] in, This refers to the total number of sampling points that need to be processed in a single reasoning task;

[0025] The following conditions must be met simultaneously for the segmentation to be successful:

[0026] First, the size of the critical working set corresponding to a single tile should not exceed the actual available on-chip cache capacity, so that the intermediate activation and necessary data access of the tile reside in the on-chip cache as much as possible;

[0027] Second, the number of tiles does not increase excessively after segmentation;

[0028] Constructing a working set model based on L2 capacity constraints: The intermediate activation residency requirement, which is closely related to inter-layer data flow during the forward computation of a single tile, is denoted as... The weighted residency requirement is recorded as ,in, This represents the intermediate activation dwell space required for a single tile to achieve inter-layer data flow during the forward computation process; Describe the weight data space that the backbone network needs to reside in during inference; and require the following:

[0029] (2);

[0030] in, This refers to the physical capacity of the GPU's on-chip L2 cache; It is a usable scaling factor;

[0031] (3);

[0032] in, This represents the hidden dimension of the backbone network, which is the length of the feature vector corresponding to each sampling point in a certain layer. Indicates the number of bytes occupied by a single numeric element;

[0033] For the hidden layer structure of the square matrix, the weight scale is used Approximate representation;

[0034] (4);

[0035] in, Indicates the number of layers in the backbone network;

[0036] From equations (2) to (4), we obtain Feasible upper bound:

[0037] (5);

[0038] definition This represents the upper bound on the number of sampling points that can be taken in a single tile under the capacity constraint condition of equation (2); let and take no more than The largest integer as This is to ensure that the capacity constraint is met;

[0039] (6);

[0040] In addition, a minimum tile threshold is introduced. And order:

[0041] (7);

[0042] Among them, if Then update according to formula (7); if This indicates that the lower bound constraint is not feasible under the current hardware, and the capacity constraint should be used as the standard. The constraint should be updated according to equation (6), letting... And subsequently, KSweep was used to weigh the actual performance, and the final calculated results were used to determine the final performance. Record ;

[0043] The updated Substituting into equation (1), we obtain the center value of the number of segments estimated by the working set model. :

[0044] (8);

[0045] Then perform the reverse recalculation and use the result obtained from the reverse recalculation. Recorded as :

[0046] (9);

[0047] when Cannot be When divisible by integers, boundary tiles will appear after segmentation;

[0048] Definition of the first Effective size of a tile for:

[0049] (10);

[0050] It stipulates that each tile should maintain consistency in scheduling order and buffered data flow rules, but when performing any layer of computation and its write-back, it should follow the... As an effective size parameter for a tile, data is only read, calculated, and written back within the effective range;

[0051] Retain center value and with Construct a candidate set for the center : Preset downward expansion index With upward expansion index Generate candidate values ​​using the following formula :

[0052] (11);

[0053] in, This indicates rounding to the nearest integer; duplicate candidate values ​​are removed from the generated candidate values, and the values ​​are pruned to the valid range to obtain the final candidate values ​​used for selection. .

[0054] Further preferred, the legal range refers to: And it does not exceed the preset maximum number of segments.

[0055] According to a preferred embodiment of the present invention, the specific implementation process of step S4 includes:

[0056] right Number of candidate segments Perform a segmentation inference measurement once in sequence and record the execution time of the program for the number of segments K. and with reference output The consistency check results are used; the segmentation inference measurement adopts the tile-first scheduling and double-buffered data flow method after segmentation; the process of scanning and measuring the candidate set one by one is called KSweep; after completing the evaluation of all candidates, the optimal number of segments is selected. :

[0057] (12);

[0058] And record .

[0059] Further preferably, the segmented inference measurement is performed using a segmented tile-first scheduling and double-buffered data flow method, specifically including:

[0060] Tile-first scheduling uses tiles as the basic processing unit, given... and the corresponding granularity of the segmentation The sampling point set is then divided into... The tile, and for the first tile Each tile must be completed consecutively first. Perform forward calculations on the current tile, then switch to the next tile;

[0061] Under tile-first scheduling, a double-buffered data transfer mechanism is introduced, as follows:

[0062] Two fixed intermediate activation buffers are set up and used alternately as input and output buffers between adjacent layers; that is, for a certain tile, the first activation buffer is used as the second activation buffer. Each layer of computation reads input from one intermediate active buffer and writes output to another. The next layer then swaps the read and write roles of these two intermediate active buffers, and this process continues until the entire tile is completed. Layer computing;

[0063] when Cannot be When a boundary tile exists due to divisibility, its effective size and processing rules are determined in step S3. The actual calculation process of the boundary tile is executed according to these processing rules in step S4 and subsequent running phases; the first definition... The effective size of a tile is It stipulates that boundary tiles maintain the same tile-first advancement order and double-buffered data flow rules as regular tiles, with the only difference being that boundary tiles, when performing calculations and write-backs at each layer, follow the same order. As an effective size parameter, data is only read, calculated, and written back within the effective range.

[0064] According to a preferred embodiment of the present invention, the specific implementation process of step S5 includes:

[0065] Introducing operator fusion computation method:

[0066] The first of the backbone network Layer, with the weight matrix defined as follows The bias vector is Input activation is Output activation is The activation function is Then the backbone network's first The layer is calculated as follows:

[0067] (13);

[0068] in, This refers to the activation function;

[0069] Operator fusion calculation is used to complete the calculation of all tiles. After layer forward computation, record the total execution time of the fusion process. And record the execution output as ;

[0070] right Compared with reference results To perform consistency verification, a criterion that includes both absolute and relative tolerance is used, as shown below:

[0071] (14);

[0072] rtol refers to absolute tolerance, which allows for a fixed absolute error; rtol refers to relative tolerance, which allows for an error proportional to the range of the reference value.

[0073] If the consistency verification fails (i.e., does not conform to equation (14), a rollback mechanism is triggered: rollback to the execution path that satisfies correctness; re-evaluate the candidate range and related constraint parameters during the calibration phase until the consistency constraints are met or rollback to the baseline execution path; if the consistency verification passes (i.e., conforms to equation (14), the final adopted path is output. And the corresponding performance and accuracy indicators; the performance indicators refer to the running time and speedup ratio, and the accuracy indicators are the consistency verification results.

[0074] An automated interface RTIO is provided for implementing the aforementioned GPU progressive optimization method for inference in neural operator backbone networks.

[0075] A computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps of the above-described GPU progressive optimization method for inference on neural operator backbone networks.

[0076] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-described GPU progressive optimization method for inference on neural operator backbone networks.

[0077] The beneficial effects of this invention are as follows:

[0078] 1. This invention addresses the problems of long cache reuse distance and frequent external storage access in large-scale sampling point inference scenarios of neural operator backbone networks. By segmenting along the sampling point dimension and adopting a tile-first execution organization method, the reuse distance between adjacent layers is shortened, the round-trip overhead of intermediate activations between GPU memory and cache is reduced, thereby improving inference efficiency.

[0079] 2. This invention constructs a working set model based on the on-chip L2 cache capacity constraint of the GPU, introduces an effective available scaling factor to characterize runtime interference, and obtains the estimated value of the number center of the segment and the candidate range accordingly, thereby reducing the blind search space.

[0080] 3. The present invention employs a double buffering mechanism during the splitting inference process, which alternates between adjacent layers as input and output buffers. This avoids the risk of "the output of the upper layer covering the input of the lower layer". At the same time, it restricts the intermediate activation residence between layers to two fixed buffers, suppresses working set diffusion and cache pollution, and enhances cache access locality and execution stability.

[0081] 4. This invention uses KSweep to perform experimental scanning and evaluation of candidate segmentation parameters, enabling the selection of the optimal number of segments under the target hardware and actual operating environment. This enables adaptive optimization under different sampling scales and network parameter conditions, thereby improving the stability and generalization of the optimization effect.

[0082] 5. This invention further introduces operator fusion optimization, which integrates linear transformation with post-processing operations such as bias superposition and activation functions, reducing the overhead of writing back intermediate results and operator startup and scheduling, and further improving inference performance on the basis of split scheduling optimization. Attached Figure Description

[0083] Figure 1 This is a flowchart illustrating a GPU progressive optimization method for inference in neural operator backbone networks according to the present invention.

[0084] Figure 2 This is a schematic diagram comparing the execution of the computational task before and after it is segmented in this invention.

[0085] Figure 3 This is a schematic diagram comparing the double buffering with single and multiple buffering methods of the present invention. Detailed Implementation

[0086] The present invention will be further defined below with reference to the accompanying drawings and embodiments, but is not limited thereto.

[0087] Example 1

[0088] A GPU progressive optimization method for inference in neural operator backbone networks, such as... Figure 1 As shown, it includes:

[0089] Step S1: Initialize constraints and tolerances;

[0090] Step S2: Baseline measurement (Level 0) obtains the baseline execution time. Compared with reference results ;

[0091] Step S3: Obtain the estimated segmentation value based on the working set model And expand;

[0092] Step S4: KSweep evaluation (Level 1) to obtain the program execution time for the number of segments K. And perform a consistency check to select the best option. The process of scanning and measuring each candidate in the set is called KSweep.

[0093] Step S5: Fix Add a fusion inference optimization (Level 2) and perform consistency verification. If the consistency verification passes, output and save the optimal configuration, and load and execute it directly during the runtime phase; otherwise, roll back or adjust and recalibrate, and return to step S3.

[0094] Example 2

[0095] The difference between this method and the GPU progressive optimization method for neural operator backbone network inference described in Example 1 is that:

[0096] The specific implementation process of step S1 includes:

[0097] During the inference process of the neural operator network's trunk, the input includes a large number of sampling points, and the inference performs layer-by-layer forward computation on all sampling points. To facilitate a unified description of subsequent segmentation and scheduling parameters, the trunk of the neural operator network is defined with the following variables:

[0098] This represents the total number of sampling points that need to be processed in one inference operation. Represents the hidden dimensions of the backbone network; Indicates the number of layers in the backbone network; This indicates the capacity of the graphics processing unit's (GPU) on-chip secondary cache (L2 cache); This represents the L2 cache availability scaling factor, used to characterize the effective available capacity under runtime system occupancy and interference conditions. This indicates the number of bytes stored per element for activation and weight data.

[0099] The specific implementation process of step S2 includes:

[0100] After the variables and constraints are determined, a complete inference is performed using the baseline execution method without sampling point dimension splitting, that is, for all... Each sampling point is completed sequentially according to the layer order. Perform layer-by-layer forward computation and record baseline execution time. Meanwhile, the output of this baseline execution is recorded as the reference result. The aforementioned and It is used to measure the effectiveness of subsequent split scheduling optimization and operator fusion computation optimization, and to determine numerical consistency constraints.

[0101] Neural operator networks typically consist of two parts: a branch network and a trunk network. The branch network extracts and encodes a global feature representation from the input function, boundary conditions, or other global conditions. The trunk network performs point-by-point feature mapping on discrete sampling points (e.g., spatial or spatiotemporal coordinates) and, combined with the global features provided by the branch network, outputs the prediction result or intermediate feature representation for each sampling point. This invention focuses on the computational process of the trunk network during the inference phase. Because inference often requires processing a large number of sampling points, repeatedly performing the same forward computation on all sampling points by the trunk network easily leads to significant computational and storage access overhead.

[0102] The backbone network structure is usually not complex, generally a multilayer perceptron (MLP) structure. Its connection relationship is generally a standard feedforward cascade structure: the output of the previous layer serves as the input of the next layer, and forward propagation is completed layer by layer in sequence. Each layer typically contains basic components such as linear transformations (e.g., matrix multiplication or fully connected mapping), bias stacking, and activation functions. Linear transformations are used to perform feature space mapping and channel mixing, and are the main source of computational cost; biases and activation functions are used to enhance the network's expressive power and introduce nonlinear features.

[0103] The "constraints" here, in conjunction with the above, mainly refer to two things: first, the L2 cache capacity constraint, corresponding to the constraints in S1. and That is, subsequent splitting and scheduling should be based on The "effective available L2 capacity" is set as an upper limit to avoid excessively large critical data sizes in a single partition block. Secondly, there are numerical consistency constraints, corresponding to the reference output in S2. That is, any subsequent splitting / fusion optimization output needs to be consistent with... Consistency must be maintained within the allowable tolerance; otherwise, the consistency determination will fail.

[0104] The specific implementation process of step S3 includes:

[0105] After the variables and constraints are determined in steps S1 and S2, when When it is large, let the threshold be ,when Time is defined as Larger; the size of the activation matrix of each layer of the backbone network varies with... Linear growth;

[0106] If we still use "for all" The baseline organization method of "incrementing the sampling points layer by layer" requires read-write alternation for large-scale activations between adjacent layers. Since this activation working set typically significantly exceeds... Activation data between adjacent layers is difficult to reside continuously in the on-chip cache, resulting in a large amount of activation data frequently traveling between external memory and on-chip cache between layers, making the overall performance of forward computation more susceptible to external memory access bandwidth and access latency.

[0107] To shorten the reuse distance between adjacent layers and control the size of the on-chip working set, this step does not change the semantics of a single inference task (still affecting all layers). Each sampling point was completed. Under the premise of layer-wise forward computation, the set of sampling points is divided into several tiles, and the number of sampling points contained in a single tile is used as the basis for calculation. As the granularity of segmentation, tile represents the subset of sampling points obtained by dividing along the dimension of the number of sampling points; Indicates the number of sampling points contained in a single tile; by the number of segments. Let the number of tiles be:

[0108] (1);

[0109] in, This refers to the total number of sampling points that need to be processed in a single inference task; that is, without segmentation, the backbone network will process these... All sampling points have been completed. The number of points in the entire batch calculated in the forward layer.

[0110] Finer slicing is not always better. A proper slice needs to satisfy the following conditions simultaneously:

[0111] First, the size of the critical working set corresponding to a single tile should not exceed the actual available on-chip cache capacity (e.g., not exceeding...). The effective available capacity (represented by the chip) allows intermediate activations and necessary data accesses of the tile to reside in the on-chip cache as much as possible;

[0112] Second, the number of tiles after segmentation should not increase excessively, in order to avoid amplifying the startup and scheduling overhead of execution units.

[0113] Constructing a working set model based on L2 capacity constraints: The intermediate activation residency requirement, which is closely related to inter-layer data flow during the forward computation of a single tile, is denoted as... The weighted residency requirement is recorded as ,in, This represents the intermediate activation dwell space required for a single tile to achieve inter-layer data flow during the forward computation process; Describe the weight data space that the backbone network needs to reside in during inference; and require the following:

[0114] (2);

[0115] in, and These are two quantities used to characterize "how much of the on-chip cache can actually be used"; It refers to the physical capacity of the GPU's on-chip L2 cache, which is the total size given by the hardware; It is an available scaling factor; used to indicate that in actual runtime, not all L2 cache can be stably occupied by the current inference process, because it is also affected by runtime system, concurrent tasks, or other cache usage. Therefore, it is used... This represents "the effective L2 capacity that can be stably used in this inference process". Equation (2) requires: the total amount of critical data that needs to reside during the calculation of a single tile (intermediate activation residency). Plus weighted residency Do not exceed this effective available cache capacity.

[0116] Sure The specific numerical method is as follows: Under the target GPU and a fixed operating environment, the total number of sampling points for a single inference iteration is selected as... And fix the backbone network structure parameters , and the number of bytes per element corresponding to data precision Then, calibration tests were performed under these fixed conditions: only the granularity of the cutting was changed. (equivalently change the number of segments) ), for each candidate Perform a tile-first inference and measure the runtime. Select the largest feasible granularity that achieves stable performance under the calibration conditions, denoted as . .Should This can be considered as the granularity of the segmentation when the single-tile working set is close to the upper limit of the available L2 capacity on the chip under the current hardware and operating environment.

[0117] In obtaining Then, the working set model was used to substitute it and reverse the process. According to the definition of this method, the intermediate activation persistence requirement for a single tile is: The weighted residency requirement is approximately Under the calibration critical conditions, the size of the critical work set is considered to meet the following requirements. Thus obtain

[0118] Under the same hardware and operating environment, subsequent steps S3 and S4 both use this fixed... Parameter calculations and KSweep testing are performed to select the optimal parameters, instead of... Adjustments will be made.

[0119] in, The decision is based on the double-buffered residency required for secure data transfer between layers (see step 3). Since double buffering requires maintaining two copies of the intermediate active buffer simultaneously to prevent read / write overwriting, therefore:

[0120] (3);

[0121] in, and It is used to express the exact size of the activation data of a tile in bytes. This represents the hidden dimension of the backbone network, which is the length of the feature vector corresponding to each sampling point in a certain layer. This indicates the number of bytes occupied by a single numeric element; for example, it is typically 2 bytes in FP16 and 4 bytes in FP32. Because a tile contains... There are 1 sampling point, and each sampling point has 1 sampling point in the hidden layer. There are one activation value, so the size of one activation buffer is approximately Bytes; and because double buffering requires maintaining two active buffers simultaneously to avoid overwriting during reads and writes, therefore ;

[0122] For common square matrix hidden layer structures, the weight size is used Approximate representation;

[0123] (4);

[0124] In this context, the square matrix hidden layer structure refers to the fact that the hidden layers of the backbone network typically maintain the same input and output dimensions, meaning that the weight matrix of each linear transformation layer has an approximate shape. In other words, the hidden layer does not expand or compress dimensions, but rather maintains a fixed hidden dimension. Feature transformation is performed layer by layer, therefore the weight parameters of each layer can be approximated by... Based on this assumption, equation (4) uses... To approximate the weighted residency scale, This represents the number of layers in the backbone network; that is, the number of linear layers (and their corresponding weights) that need to be executed sequentially in the forward computation; therefore, the total weight size can be approximated as the "weight size per layer". Multiply by the number of floors Multiply by the number of bytes per element Obtain the storage amount in bytes.

[0125] From equations (2) to (4), we obtain Feasible upper bound:

[0126] (5);

[0127] definition This represents the upper bound on the number of sampling points that can be taken in a single tile under the capacity constraint condition of equation (2); let and take no more than The largest integer as This is to ensure that the capacity constraint is met;

[0128] (6);

[0129] In addition, to avoid Too small a threshold leads to increased scheduling overhead, so a minimum tile threshold is introduced. , This is used to avoid excessive tile count due to overly fine segmentation, which would then dominate startup and scheduling overhead. This lower bound constraint is used to narrow down the candidate range and does not replace subsequent KSweep test-based selection. And let:

[0130] (7);

[0131] Among them, if Then update according to formula (7); if This indicates that the lower bound constraint is not feasible under the current hardware, and the capacity constraint should be used as the standard. The constraint should be updated according to equation (6), letting... And subsequently, KSweep was used to weigh the actual performance, and the final calculated results were used to determine the final performance. Record ;

[0132] Formula (6) first put Select no more than The largest integer is used to ensure that the capacity constraint is met; Equation (7) further introduces a lower bound of "don't cut too small". This avoids excessive tile counts leading to startup and scheduling overhead becoming dominant. The real potential conflict arises when the upper bound calculated by equation (5) is very small, making... This means that "the maximum tile size allowed by the capacity constraint cannot reach the desired minimum tile threshold." If, at this point, equation (7) is forcibly applied... Upgraded to This would conversely violate capacity constraints. Therefore, the above operations were added to avoid this contradictory situation. The following are practical examples of both scenarios.

[0133] Assuming the GPU's , (Approximately Valid and Available) Hidden Dimensions The data precision is FP16, therefore Bytes, number of layers First calculate the weighted residency. And then consider the upper boundary. Equation (6) yields Assuming a minimum threshold is set... Then, after updating equation (7) This will not change the outcome, and therefore there is no contradiction. At this point, if... Substituting into equation (1) the central value .

[0134] Assumption , (Approximately Valid and Available) ), , (FP16) At this time The remaining capacity available for activation is approximately ,so Equation (6) yields If still set Then, according to formula (7), it is forcibly updated to It will directly exceed the upper limit of capacity (because) This is the "point of contradiction": the lower bound constraint... Increase it to a value that no longer satisfies the capacity constraint of equation (2). The solution is to explicitly state that equation (7) only applies when... If it takes effect immediately, otherwise the result of equation (6) shall be considered the final feasible result. In this example, the final choice should be... Therefore, if ,but This also clearly shows that too many tiles will bring scheduling pressure, thus explaining why KSweep needs to conduct further testing to weigh the trade-offs.

[0135] The updated Substituting into equation (1), we obtain the center value of the number of segments estimated by the working set model. :

[0136] (8);

[0137] To ensure strict coverage of the segmentation and and Self-consistent, then perform reverse recalculation, and use the results of the reverse recalculation Recorded as :

[0138] (9);

[0139] when Cannot be When divisible by integers, boundary tiles will appear after segmentation;

[0140] To ensure complete coverage of the segmentation and avoid shape mismatches or out-of-bounds access, the first... Effective size of a tile for:

[0141] (10);

[0142] It stipulates that each tile should maintain consistency in scheduling order and buffered data flow rules, but when performing any layer of computation and its write-back, it should follow the... As an effective size parameter for tiles, data is read, calculated, and written back only within the effective range; thus, the size of boundary tiles is adaptive, and the inference semantics and numerical results are consistent with the unsegmented case.

[0143] The above working set model is used to provide a reasonable range for the splitting parameters and the center estimate. Instead of replacing all actual measurements with a single calculation, the model estimates are retained to minimize the discrepancy between the model estimates and the actual hardware performance. and with Construct a candidate set for the center : Preset downward expansion index With upward expansion index Generate candidate values ​​using the following formula :

[0144] (11);

[0145] in, This indicates rounding to the nearest integer; duplicate candidate values ​​are removed from the generated candidate values, and the values ​​are pruned to the valid range to obtain the final candidate values ​​used for selection. .

[0146] The legal scope refers to: And it does not exceed the preset maximum number of segments.

[0147] The specific implementation process of step S4 includes:

[0148] right Number of candidate segments Perform a segmentation inference measurement once in sequence and record the execution time of the program for the number of segments K. and with reference output The consistency check results are used; the segmentation inference measurement adopts the tile-first scheduling and double-buffered data flow method after segmentation; the process of scanning and measuring the candidate set one by one is called KSweep; after completing the evaluation of all candidates, the optimal number of segments is selected. :

[0149] (12);

[0150] And record Through "capacity constraint estimation—" The closed loop of "candidate expansion centered on KSweep test optimization" can make parameter selection more robust to specific hardware and input scale while maintaining the interpretability and reproducibility of the selection of segmentation parameters.

[0151] The segmented inference measurement is performed using a segmented tile-first scheduling and double-buffered data flow method, specifically including:

[0152] Inference execution preserves computational semantics (still for all). Each sampling point was completed. Under the premise of layer-by-layer computation, the execution organization method is changed from "layer-first" to "tile-first" when not segmented; it should be noted that in step 2, KSweep performs candidate... The segmentation inference measurement is performed using the tile-first scheduling and double-buffered data flow method described in this step. After determining... Then, this step is in The execution organization is fixed to complete the segmentation and inference process, and to provide a basis for the superposition of subsequent operator fusion calculation optimization.

[0153] Unsplit layer-first scheduling applies the full amount at each layer. The computation is performed at each sampling point, causing the single-layer activation working set to follow... The increased size significantly exceeds the on-chip cache capacity, leading to long cross-layer reuse distances and unstable on-chip residency. Tile-first scheduling, on the other hand, uses tiles as the basic processing unit, and within a given... and the corresponding granularity of the segmentation The sampling point set is then divided into... The tile, and for the first tile Each tile must be completed consecutively first. The process proceeds by performing forward computation on each tile before switching to the next tile. Because the active data of the same tile undergoes continuous computation across multiple adjacent tiles within a short time window, tile-first scheduling significantly shortens the cross-layer reuse distance, increases on-chip hit probability, and reduces external memory round trips. Its benefit mechanism can be explained by comparing "unsegmented vs. properly segmented" (see [link to diagram]). Figure 2 ).

[0154] Under tile-first scheduling, inter-layer data transfer must simultaneously meet the following constraints: first, avoid data corruption caused by read / write overwriting; second, control the intermediate dwell range to prevent working set expansion due to increased buffer size. To meet these constraints, this step introduces a double-buffered data transfer mechanism, as follows:

[0155] Set up two fixed intermediate activation buffers (e.g., buffer A and buffer B), and use them alternately as input and output buffers between adjacent layers; that is: for a certain tile, the first... Each layer of computation reads input from one intermediate active buffer and writes output to another. The next layer then swaps the read and write roles of these two intermediate active buffers, and this process continues until the entire tile is completed. Layered computation; this double-buffering mechanism avoids the risk of "the output of the upper layer overwriting the input of the lower layer" compared to single-buffering; compared to multi-buffering, it avoids the risk of increased resident area and cache pollution caused by the growth of intermediate buffers. Its core lies in restricting intermediate activations between layers to alternate between two fixed address regions, keeping intermediate resident demand at a constant size of two buffers, thus aligning with the working set constraint model, and suppressing working set diffusion while ensuring correctness, forming a relatively stable cache access locality. The comparison between the double-buffering mechanism and single / multi-buffering mechanisms can be understood with reference to the diagram (see...). Figure 3 ).

[0156] when Cannot be When boundary tiles exist due to divisibility, the effective size and processing rules of the boundary tiles are determined in step S3. The actual calculation process of the boundary tiles is executed according to these processing rules in step S4 and subsequent running stages; that is, step S3 is responsible for deriving the parameters and rules related to the segmentation (e.g., obtaining...). ,Sure And give the effective size of the boundary tile. The actual reading, layer-by-layer calculation, and writing back of the boundary tiles occur during the "tile-first advancement and double-buffered alternating read / write" process, which is the KSweep evaluation of S4 for each candidate tile. When running inference and timing / consistency checks, the last tile that is not a complete block will be... Execute as an effective size. Define the first... The effective size of a tile is It stipulates that boundary tiles maintain the same tile-first advancement order and double-buffered data flow rules as regular tiles, with the only difference being that boundary tiles, when performing calculations and write-backs at each layer, follow the same order. As an effective size parameter, data is read, computed, and written back only within its effective range. This ensures complete coverage, avoids shape mismatches or out-of-bounds access, and maintains overall semantic consistency in inference.

[0157] The specific implementation process of step S5 includes:

[0158] Segmentation and tile-first scheduling primarily improve cross-layer locality between adjacent layers within the same tile. However, segmentation also increases the number of tiles, thereby amplifying the scheduling frequency of the splitting operator within a single layer and the external read / write overhead of intermediate results. To further reduce external memory round trips within a single layer and suppress scheduling overhead, this step maintains tile-first scheduling and Under the premise of not changing, an operator fusion calculation method is introduced: the linear transformation and post-processing such as bias superposition and activation are organized into a calculation process that is completed in one go, so that the intermediate results are completed in the accumulation path of matrix operation and written back to the output at once, thereby reducing the extra round trip of "write back - read in again - write back again".

[0159] The first of the backbone network Layer, with the weight matrix defined as follows The bias vector is Input activation is Output activation is The activation function is Then the backbone network's first The layer is calculated as follows:

[0160] (13);

[0161] in, This refers to the activation function; that is, a function that takes the result of a linear transformation with a bias and then applies a nonlinear transformation. Its purpose is to introduce nonlinear expressive power into the network; otherwise, multiple layers of linear transformations would still be equivalent to a single linear transformation, limiting the model's capabilities.

[0162] Operator fusion calculation is used to complete the calculation of all tiles. After layer forward computation, record the total execution time of the fusion process. And record the execution output as ;

[0163] To ensure the optimization is reproducible under numerical precision constraints, this step involves... Compared with reference results To perform consistency verification, a criterion that includes both absolute and relative tolerance is used, as shown below:

[0164] (14);

[0165] Here, atol and rtol are tolerance parameters for "consistency verification," used to determine the optimized output. and baseline reference output Whether they are consistent within the allowable numerical error range. atol (absolute tolerance) refers to absolute tolerance, used to allow a fixed absolute error; rtol (relative tolerance) refers to relative tolerance, used to allow an error proportional to the reference value. Therefore, the right side... Will follow As the magnitude increases, the allowable error range is correspondingly widened. The above-mentioned atol and rtol are tolerance thresholds preset by the user (or system), which can be selected and adjusted according to the allowable range of numerical error, data precision type, and precision requirements of the actual application.

[0166] "Consistency verification failed" means that the criterion (14) is not satisfied. More specifically, it means that when performing element-by-element comparison on the output, there exists at least some elements that satisfy the condition. Greater than the upper limit of the allowable error on the right. If the result is negative, it is considered a failure; if it is a success, it means that all the elements to be compared fall within the tolerance range.

[0167] The core purpose of this criterion is to use a standard that is neither too strict nor too lenient to judge whether the results before and after optimization are consistent. Using only absolute tolerance will be too strict when the value is very large (because the allowable error does not change with the order of magnitude), and using only relative tolerance will fail when the reference value is close to 0 (because the relative error benchmark is too small, even a very small numerical fluctuation will be judged as inconsistent). Therefore, combining the two is equivalent to saying that "the upper limit of the allowable error = a fixed bottom line (atol) + a part that changes with the order of magnitude of the reference value (rtol·|Y_ref|)", which can cover both the case close to 0 and the case of large values.

[0168] For a concrete example, suppose we set... , If a certain position references the output Then the upper limit of the allowable error is If the output is optimized at this point... The error is The value is less than 1.00001, so this point is considered passed; if If the error is 2.5, which is greater than 1.00001, then this point is considered a failure. It can be seen that when the numerical magnitude is large, the standard is mainly determined by the relative tolerance, and the allowable error will be reasonably relaxed as the value increases.

[0169] However, for cases where the reference output is close to 0, under the same tolerance settings, if at a certain position... The upper limit of the allowable error is If at this time ,error , through; if ,error The result was not passed. It can be seen from this that when the reference value is very small or 0, the relative tolerance term has almost no effect. The judgment mainly relies on the absolute tolerance to provide a "minimum allowable fluctuation range". Otherwise, using only the relative error will become very unstable. Overall, the consistency verification "passes" when all the compared elements satisfy the inequality of equation (14); "failes" when there is any element that does not satisfy the inequality.

[0170] If the consistency verification fails (i.e., does not conform to equation (14), a rollback mechanism is triggered: rollback to the execution path that meets the correctness requirement (e.g., keep tile-first and double buffering unchanged, but switch operator fusion calculation to separate execution mode); re-evaluate the candidate range and related constraint parameters during the calibration phase until the consistency constraints are met or rollback to the baseline execution path; if the consistency verification passes (i.e., conforms to equation (14), the final adopted path is output. And corresponding performance and accuracy metrics; the performance and accuracy metrics output after successful consistency verification include two types of information: performance metrics refer to runtime and speedup ratio, for example, corresponding Execution time (Time after splitting and scheduling) and time after merging and further provide , , These three metrics represent benefits; the accuracy metric is the consistency verification result. That is, given... , The verification passed, indicating that the optimized output and the reference output numerically meet the preset accuracy constraints. These two pieces of information show that the output is not only... It also includes "how fast the corresponding response is, and under what tolerance standard is it considered consistent." The revenue from split scheduling and the revenue from operator fusion are respectively comprised of... and Characterization, overall return is determined by Characterization.

[0171] Finally, it should be noted that the above process is preferably used for the parameter calibration stage during initial deployment or when the input shape or hardware environment changes. After calibration is completed, the runtime phase can directly load the optimal parameters obtained from calibration and execute them through a fixed single path. If the consistency determination fails, a fallback path will be triggered, thereby avoiding the additional overhead of repeatedly traversing candidate parameters in each run.

[0172] Example 3

[0173] An automated interface RTIO (Resolve Trunk Inference Optimization) is provided to implement the GPU progressive optimization method for neural operator backbone network inference described in Embodiment 1 or 2.

[0174] This invention designs and implements an automated interface RTIO (Resolve TrunkInference Optimization) based on CUDA C++. This interface aims to address the performance degradation issues caused by frequent external storage round trips and amplified scheduling overhead after partitioning in neural operator network trunks during large-scale sampling point inference scenarios. The implementation of this interface reduces programming complexity, allowing programmers to directly call it to complete parameter optimization during the calibration phase and inference execution during the runtime phase. Interface specifications are shown in Table 1.

[0175] Table 1. RTIO Interface Description;

[0176]

[0177] The following is an example of host-side CUDA C++ calling the RTIO interface:

[0178] / / Calling the RTIO interface (CUDA C++) from the host (CPU) side

[0179] #include "rtio.h"

[0180] int main(){

[0181] / / ... Model / data preparation omitted (weights and inputs / outputs are located on the device).

[0182] RTIO_Init( / *device_id=* / 0);

[0183] / / Calibration (warm-up): Called upon initial deployment or when inputting shape / hardware changes.

[0184] RTIO_Calibrate(N_total, H, L, alpha, N_min, a, b, atol, rtol);

[0185] / / Run: Repeated inference directly reuses the optimal configuration obtained from calibration

[0186] for (int iter = 0; iter <NUM_INFER; ++iter) {

[0187] RTIO_Run(X_dev, Y_dev);

[0188] }

[0189] RTIO_Finalize();

[0190] return 0;

[0191] }

[0192] Through the implementation of the above method, the incremental optimization method of the present invention was tested in a large-scale inference scenario of a neural operator network trunk network. The test platform was a GPU environment, the trunk structure was fixed with hidden dimension H=256 and number of layers L=6, and comparisons were made under different sampling point numbers N_total. After the necessary warm-up was completed, the test data of each group were measured multiple times and the average value was taken to reduce the impact of random fluctuations on the results. Each group was configured to warm up 10 times first, then iterate 50 times and take the average value. At the same time, in order to verify the consistency of the results before and after optimization, this test adopted a combination of absolute tolerance and relative tolerance criteria to compare the output results with the reference results item by item: when the item-by-item error satisfies the above formula (14), the consistency verification is determined to be passed. Where atol is the absolute tolerance and rtol is the relative tolerance. In this test, rtol=1×10 -4 atol=1×10 -4 All consistency verifications passed; the maximum absolute error of the fusion path is approximately 10. -6 To verify the necessity of "optimizing the granularity of the segmentation" in Level 1 (sampling point dimension segmentation and scheduling), Table 2 shows the results of KSweep scanning for different segmentation numbers K under the condition of fixed N_total=1000000.

[0193] Table 2 Comparison of Level 1 time for different number of segments K under a fixed number of sampling points (unit: ms);

[0194]

[0195] The above results indicate that when K is too small (insufficient segmentation), the working set of a single tile is too large, and the cross-layer reuse effect is not significant; when K is too large (excessive segmentation), the increased number of tiles amplifies the startup and scheduling overhead, thus offsetting the benefits of segmentation; only when K is reasonable... Stable performance improvements can be achieved in the vicinity. Therefore, in Level 1, "the center value is obtained based on the working set model." and around The process of "constructing a candidate set and selecting the best candidate through actual testing" is necessary.

[0196] To verify the generality of the method, while keeping all other parameters the same, only the number of sampling points was changed. Table 3 shows the comparison results of Level 1 and Level 2 relative to the baseline execution method under different sampling scales, which illustrates that the method of the present invention can obtain stable benefits under different input scales and can achieve acceleration while maintaining numerical consistency constraints.

[0197] Table 3 Comparison of incremental optimization time for Level 1 and Level 2 with different numbers of sampling points (unit: ms);

[0198]

[0199] Example 4

[0200] A computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps of the GPU progressive optimization method for neural operator backbone network inference as described in Embodiment 1 or 2.

[0201] Example 5

[0202] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the GPU progressive optimization method for inference of neural operator backbone networks as described in Embodiment 1 or 2.

Claims

1. A GPU progressive optimization method for inference in neural operator backbone networks, characterized in that, include: Step S1: Initialize constraints and tolerances; Step S2: Baseline measurement to obtain baseline execution time Compared with reference results ; Step S3: Obtain the estimated segmentation value based on the working set model And expand; Step S4: KSweep evaluates the execution time of the program for the number of segments K. A consistency check is performed to determine the optimal number of segments. The process of scanning and measuring each candidate in the set is called KSweep. Step S5: Fix the optimal number of segments Add fusion inference optimization and perform consistency verification. If the consistency verification passes, output and save the optimal configuration, and load and execute it directly during the runtime phase; otherwise, roll back or adjust and recalibrate, and return to step S3. The specific implementation process of step S3 includes: After the variables and constraints are determined in steps S1 and S2, when When it is large, let the threshold be ,when Time is defined as Larger; the size of the activation matrix of each layer of the backbone network varies with... Linear growth; Without altering the semantics of a single inference task, the set of sampling points is divided into several tiles, and the number of sampling points contained in a single tile is used as the basis for classification. As the granularity of segmentation, tile represents the subset of sampling points obtained by dividing along the dimension of the number of sampling points; Indicates the number of sampling points contained in a single tile; by the number of segments. Let the number of tiles be: （1）； in, This refers to the total number of sampling points that need to be processed in a single reasoning task; The segmentation needs to satisfy the following conditions simultaneously: First, the size of the critical working set corresponding to a single tile should not exceed the actual available on-chip cache capacity, so that the intermediate activation and necessary data access of the tile reside in the on-chip cache as much as possible; Second, the number of tiles does not increase excessively after segmentation; Constructing a working set model based on L2 capacity constraints: The intermediate activation residency requirement, which is closely related to inter-layer data flow during the forward computation of a single tile, is denoted as... The weighted residency requirement is recorded as ,in, This represents the intermediate activation dwell space required for a single tile to achieve inter-layer data flow during the forward computation process; Describe the weight data space that the backbone network needs to reside in during inference; and require the following: （2）； in, This refers to the physical capacity of the GPU's on-chip L2 cache; It is a usable scaling factor; （3）； in, This represents the hidden dimension of the backbone network, which is the length of the feature vector corresponding to each sampling point in a certain layer. Indicates the number of bytes occupied by a single numeric element; For the hidden layer structure of the square matrix, the weight scale is used To make an approximate representation; （4）； in, Indicates the number of layers in the backbone network; From equations (2) to (4), we obtain Feasible upper bound: （5）； definition This represents the upper bound on the number of sampling points that can be taken in a single tile under the capacity constraint condition of equation (2); let and take no more than The largest integer as This is to ensure that the capacity constraint is met; （6）； In addition, a minimum tile threshold is introduced. And order: （7）； Among them, if Then update according to formula (7); if This indicates that the lower bound constraint is not feasible under the current hardware, and the capacity constraint should be used as the standard. The constraint should be updated according to equation (6), letting... And subsequently, KSweep was used to weigh the actual performance, and the final calculated results were used to determine the final performance. Record ; The updated Substituting into equation (1), we obtain the center value of the number of segments estimated by the working set model. : （8）； Then perform the reverse recalculation and use the result obtained from the reverse recalculation. Recorded as : （9）； when Cannot be When divisible by integers, boundary tiles will appear after segmentation; Definition of the first Effective size of a tile for: （10）； It stipulates that each tile should maintain consistency in scheduling order and buffered data flow rules, but when performing any layer of computation and its write-back, it should follow the... As an effective size parameter for a tile, data is only read, calculated, and written back within the effective range; Retain center value and with Construct a candidate set for the center : Preset downward expansion index With upward expansion index Generate candidate values ​​using the following formula : （11）； in, This indicates rounding to the nearest integer; duplicate candidate values ​​are removed from the generated candidate values, and the values ​​are pruned to the valid range to obtain the final candidate values ​​used for selection. ; The legal scope refers to: And it does not exceed the preset maximum number of segments.

2. The GPU progressive optimization method for inference in neural operator backbone networks according to claim 1, characterized in that, The specific implementation process of step S1 includes: In the inference process of the backbone network of the neural operator network, the input includes a large number of sampling points, and the inference performs layer-by-layer forward computation on all sampling points; the backbone network of the neural operator network is defined with the following variables: This represents the total number of sampling points that need to be processed in one inference operation. Represents the hidden dimensions of the backbone network; Indicates the number of layers in the backbone network; This indicates the capacity of the graphics processor's on-chip L2 cache; This indicates the L2 cache availability scaling factor. This indicates the number of bytes stored per element for activation and weight data.

3. The GPU progressive optimization method for inference in neural operator backbone networks according to claim 1, characterized in that, The specific implementation process of step S2 includes: A complete inference is performed using a baseline execution method without sampling point dimension splitting, i.e., for all... Each sampling point is completed sequentially according to the layer order. Perform layer-by-layer forward computation and record baseline execution time. Meanwhile, the output of this baseline execution is recorded as the reference result. .

4. The GPU progressive optimization method for inference in neural operator backbone networks according to claim 1, characterized in that, The specific implementation process of step S4 includes: right Number of candidate segments Perform a segmentation inference measurement once in sequence and record the execution time of the program for the number of segments K. and with reference output The consistency check results are used; the segmentation inference measurement adopts the tile-first scheduling and double-buffered data flow method after segmentation; the process of scanning and measuring the candidate set one by one is called KSweep; after completing the evaluation of all candidates, the optimal number of segments is selected. : （12）； And record .

5. The GPU progressive optimization method for inference in neural operator backbone networks according to claim 1, characterized in that, The segmented inference measurement is performed using a segmented tile-first scheduling and double-buffered data flow method, specifically including: Tile-first scheduling uses tiles as the basic processing unit, given... and the corresponding granularity of the segmentation The sampling point set is then divided into... The tile, and for the first tile Each tile must be completed consecutively first. Perform forward calculations on the current tile, then switch to the next tile; Under tile-first scheduling, a double-buffered data transfer mechanism is introduced, as follows: Two fixed intermediate activation buffers are set up and used alternately as input and output buffers between adjacent layers; that is, for a certain tile, the first activation buffer is used as the second activation buffer. Each layer of computation reads input from one intermediate active buffer and writes output to another. The next layer then swaps the read and write roles of these two intermediate active buffers, and this process continues until the entire tile is completed. Layer computing; when Cannot be When a boundary tile exists due to divisibility, its effective size and processing rules are determined in step S3. The actual calculation process of the boundary tile is executed according to these processing rules in step S4 and subsequent running phases; the first definition... The effective size of a tile is It stipulates that boundary tiles maintain the same tile-first advancement order and double-buffered data flow rules as regular tiles, with the only difference being that boundary tiles, when performing calculations and write-backs at each layer, follow the same order. As an effective size parameter, data is only read, calculated, and written back within the effective range.

6. A GPU progressive optimization method for inference in neural operator backbone networks according to any one of claims 1-5, characterized in that, The specific implementation process of step S5 includes: Introducing operator fusion computation method: The first of the backbone network Layer, with the weight matrix defined as follows The bias vector is Input activation is Output activation is The activation function is Then the backbone network's first The layer is calculated as follows: （13）； in, This refers to the activation function; Operator fusion calculation is used to complete the calculation of all tiles. After layer forward computation, record the total execution time of the fusion process. And record the execution output as ; right Compared with reference results To perform consistency verification, a criterion that includes both absolute and relative tolerance is used, as shown below: （14）； rtol refers to absolute tolerance, which allows for a fixed absolute error; rtol refers to relative tolerance, which allows for an error proportional to the range of the reference value. If the consistency verification fails (i.e., does not conform to equation (14), a rollback mechanism is triggered: rollback to the execution path that satisfies correctness; re-evaluate the candidate range and related constraint parameters during the calibration phase until the consistency constraints are met or rollback to the baseline execution path; if the consistency verification passes (i.e., conforms to equation (14), the final adopted path is output. And the corresponding performance and accuracy indicators; the performance indicators refer to the running time and speedup ratio, and the accuracy indicators are the consistency verification results.

7. An automated interface RTIO, characterized in that, This is used to implement a GPU progressive optimization method for neural operator backbone network inference as described in any one of claims 1-6.

Images