A flexible operator replacement experiment platform

Abstract

This invention discloses a flexible operator replacement experimental platform, relating to the field of GPU operator replacement experimental technology. It includes a rule parsing module, a sandbox construction module, a communication planning module, a device adaptation module, an operator switching module, an experimental monitoring module, and a strategy solidification module. The rule parsing module receives operator replacement rule files, parses the target module matching conditions, candidate operator implementations, traffic allocation strategies, and runtime switching conditions, and generates structured configuration information. The sandbox construction module, during the model loading phase, locates the target modules in the model based on the matching conditions in the structured configuration information and creates a runtime sandbox container for each target module. The sandbox container integrates the original module and multiple candidate operator implementations, supporting traffic distribution based on request characteristics. The communication planning module analyzes the tensor dependencies between sandbox containers to determine cross-device transmission paths.

Description

Technical Field

[0001] This invention relates to the field of GPU operator replacement experimental technology, and in particular to a flexible operator replacement experimental platform. Background Technology

[0002] GPU operator replacement experimentation technology refers to a technical system that provides multiple implementation versions of a specific neural network operator in a deep learning inference system, and conducts parallel testing, performance comparison, and optimal selection of these versions at runtime through sandboxing, traffic distribution, performance monitoring, and dynamic decision-making mechanisms. Therefore, how to utilize advanced technologies to improve the intelligence and security of GPU operator replacement experiments has become one of the urgent problems to be solved.

[0003] In the field of GPU operator replacement experiments, existing technologies typically employ static compilation or manual replacement methods. Once the replacement is completed, the replacement is fixed and cannot be used to conduct parallel testing and performance comparison of multiple candidate operators at runtime. This results in a lack of data support for optimization decisions, high experimental costs, and long cycles. Furthermore, traditional solutions struggle to integrate multiple operator implementations into the same model and distribute traffic according to request characteristics, making it impossible to conduct A / B testing or canary releases. This limits the ability to securely verify new operators in production environments. Summary of the Invention

[0004] In view of the aforementioned existing problems, the present invention is proposed.

[0005] Therefore, this invention provides a flexible operator replacement experimental platform to solve the problems of existing technologies that typically use static compilation or manual replacement methods, which are fixed once the replacement is completed. This makes it impossible to conduct parallel testing and performance comparison of multiple candidate operators at runtime, resulting in a lack of data support for optimization decisions, high experimental costs and long cycles. Furthermore, traditional solutions are difficult to integrate multiple operator implementations in the same model and distribute traffic according to request characteristics, making it impossible to achieve A / B testing or canary release, thus limiting the ability to securely verify new operators in a production environment.

[0006] To solve the above-mentioned technical problems, the present invention provides the following technical solution:

[0007] In a first aspect, the present invention provides a flexible operator substitution experimental platform, comprising:

[0008] The system includes a rule parsing module, a sandbox construction module, a communication planning module, a device adaptation module, an operator switching module, an experiment monitoring module, and a strategy solidification module.

[0009] The rule parsing module is used to receive the operator replacement rule file, parse the target module matching conditions, candidate operator implementations, traffic allocation strategies and runtime switching conditions in it, and generate structured configuration information.

[0010] The sandbox construction module is used to locate the target module in the model according to the matching conditions in the structured configuration information during the model loading phase, and create a runtime sandbox container for each target module. The sandbox container integrates the original module and multiple candidate operators to support traffic distribution based on request characteristics.

[0011] The communication planning module is used to analyze the tensor dependencies between sandbox containers, determine cross-device transmission paths, and generate a data flow execution plan that includes transmission timing and memory layout requirements.

[0012] The device adaptation module is used to pre-allocate a fixed memory space on the host side according to the data flow execution plan, configure an independent execution flow for the computing device, and assign data transfer operations and computing tasks to different execution flows for parallel execution.

[0013] The operator switching module is used to monitor the operating status of each sandbox container during the inference process. When the operating status meets the switching conditions in the structured configuration information, the currently active operator is replaced with the specified candidate operator in real time to maintain the continuity of interface calls.

[0014] The experimental monitoring module is used to collect latency, memory usage and throughput performance data of each operator according to the traffic distribution path during the parallel operation of multiple candidate operators, and form a performance record with identification by combining request context information.

[0015] The strategy solidification module is used to perform performance comparison analysis based on the identified performance records, determine the optimal operator implementation, set the implementation as the unique active instance of the corresponding sandbox container, and output a model deployment description containing the final configuration.

[0016] As a preferred embodiment of the flexible operator replacement experimental platform described in this invention, the steps of receiving the operator replacement rule file, parsing the target module matching conditions, candidate operator implementations, traffic allocation strategies, and runtime switching conditions therein, and generating structured configuration information are as follows:

[0017] Perform lexical and syntactic analysis on the rule file to extract the name pattern or type identifier of the target module;

[0018] Establish matching criteria based on name patterns or type identifiers;

[0019] Read the implementation type of the candidate operators associated with the target module and their deployment device information;

[0020] Retrieve the request distribution weights or routing rules defined in the traffic allocation strategy;

[0021] Identify the performance threshold parameters and input feature variables involved in runtime switching conditions;

[0022] The above information is integrated into structured configuration data that includes matching criteria, candidate implementation list, distribution weights, and switching criteria.

[0023] As a preferred embodiment of the flexible operator replacement experimental platform described in this invention, the following steps are taken: During the model loading phase, the target module in the model is located based on the matching conditions in the structured configuration information, and a runtime sandbox container is created for each target module. The sandbox container integrates the original module with multiple candidate operators, supporting traffic distribution based on request characteristics.

[0024] Iterate through all submodules in the model and obtain the name and type of the current submodule;

[0025] Compare the name of the current submodule with the name pattern in the structured configuration information, or determine whether its type matches the type identifier;

[0026] When the comparison result meets the matching conditions, the submodule is retained as the original module instance;

[0027] Based on the candidate operator implementation type specified in the structured configuration information, initialize the alternative module instance on the corresponding computing device;

[0028] Register the original module and all candidate operator implementation instances into the same sandbox container;

[0029] Set the default active initial operator;

[0030] Configure traffic distribution logic so that subsequent inference requests can select different operator paths for execution based on request identifier, input length, or user tag.

[0031] As a preferred embodiment of the flexible operator replacement experimental platform described in this invention, the steps for analyzing the tensor dependencies between sandbox containers, determining cross-device transmission paths, and generating a data flow execution plan that includes transmission timing and memory layout requirements are as follows:

[0032] Based on the data flow direction during the model's forward propagation, an input-output connection diagram is established between each sandbox container.

[0033] Identify tensor transport edges in a connected graph that span different computing devices;

[0034] Label the data volume for each cross-device transmission edge. With transmission frequency ,in, Derived from the dimension and data type of this tensor, This number originates from the number of times this path is called during the model inference process;

[0035] Calculate the data bandwidth requirement per unit time for each transmission path based on the labeled information. ;

[0036] based on The size of the transmission path is used to prioritize the transmission paths;

[0037] By combining the execution order of candidate operators, the insertion timing and synchronization point of each transmission operation in the overall execution flow are determined;

[0038] A data flow execution plan is formed, which includes memory allocation locations, transmission start and end time windows, and execution flow divisions.

[0039] As a preferred embodiment of the flexible operator replacement experimental platform described in this invention, the steps of pre-allocating a fixed memory space on the host side according to the data flow execution plan, configuring an independent execution flow for the computing device, and allocating data transfer operations and computing tasks to different execution flows for parallel execution are as follows:

[0040] Reserve a fixed address region in the host memory for tensors that frequently participate in cross-device transfers, and ensure that this region supports zero-copy access.

[0041] Create at least two independent execution flows on the computing device side. and ,in, Specifically designed for performing data moving tasks. Used to perform core computing tasks;

[0042] According to the transmission timing defined in the data flow execution plan, Arrange asynchronous data copy instructions in the middle;

[0043] At the same time The forward computation kernel that schedules the current operator;

[0044] Introducing an event synchronization mechanism ,exist Triggered after data transfer is completed ,notify Start the computation task that depends on this data;

[0045] pass and Parallel scheduling enables time overlap between computation and communication.

[0046] As a preferred embodiment of the flexible operator replacement experimental platform described in this invention, the following steps are taken: During the inference process, the operating status of each sandbox container is monitored. When the operating status meets the switching conditions in the structured configuration information, the currently active operator is replaced in real time with a specified candidate operator, maintaining the continuity of interface calls.

[0047] The monitoring unit collects the context length of the currently processed request. Video memory usage and the time consumption of single-step reasoning ,in The number of tokens derived from the input sequence. Originating from the device memory monitoring interface, Derived from a high-precision timer;

[0048] The collected context length Video memory usage and the time consumption of single-step reasoning The switching threshold preset in the structured configuration information Compare;

[0049] When satisfied or or When the switching condition is triggered, determine the switching condition.

[0050] Suspend new requests entering the current sandbox container;

[0051] After completing the request being executed, the function call pointer is changed from the original activation operator to the target candidate operator within the sandbox.

[0052] As a preferred embodiment of the flexible operator replacement experimental platform described in this invention, the following steps are taken: During the parallel operation of multiple candidate operators, latency, memory usage, and throughput performance data of each operator are collected according to the traffic distribution path, and combined with request context information to form an identified performance record.

[0053] Each time a request enters the sandbox container, the specific operator path identifier assigned to that request is recorded. ;

[0054] Start a high-precision timer and record the end-to-end response time after the operator completes the forward computation. ;

[0055] Obtain the peak video memory consumption of the computing device during this calculation process from the system monitoring interface. ;

[0056] Statistical unit time Number of requests successfully processed Calculate the actual throughput. ;

[0057] Will , , , Together with the corresponding request context features, they are written into the log storage area to form a complete performance record.

[0058] As a preferred embodiment of the flexible operator replacement experimental platform described in this invention, the steps of performing performance comparison analysis based on identified performance records, determining the optimal operator implementation, setting this implementation as the unique active instance of the corresponding sandbox container, and outputting a model deployment description containing the final configuration are as follows:

[0059] Read the performance record set of all candidate operators under a target module from the log storage area;

[0060] Delay for each candidate operator Video memory usage and throughput Calculate the statistical values ​​separately, including the mean. ;

[0061] Based on preset performance evaluation weights The indicators are normalized and weighted summed to obtain a comprehensive score. Its expression is:

[0062] ;

[0063] in, The minimum average delay among all candidate operators. To minimize average video memory usage, This represents the maximum average throughput.

[0064] Select comprehensive score The highest-ranking candidate operator is selected as the optimal implementation.

[0065] Update the activation status of the sandbox container corresponding to the target module so that it retains only the optimal operator as the only available instance;

[0066] A new configuration file is generated, which contains all optimized target modules and their final selected operator types, device locations, and execution parameters.

[0067] In a second aspect, the present invention provides a computer device including a memory and a processor, wherein the memory stores a computer program, wherein: when the computer program is executed by the processor, it implements any step of the flexible operator substitution experimental platform as described in the first aspect of the present invention.

[0068] Thirdly, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein: when the computer program is executed by a processor, it implements any step of the flexible operator substitution experimental platform as described in the first aspect of the present invention.

[0069] The beneficial effects of this invention are as follows: by constructing an operator replacement experimental architecture that integrates rule parsing, sandbox encapsulation, communication optimization and dynamic decision-making, it enables parallel testing, performance comparison and adaptive switching of multiple operator implementations without interrupting the inference service. It solves the problems of irreversible operator replacement, high experimental cost, large communication overhead of heterogeneous deployment and lack of runtime awareness in traditional methods, and improves the flexibility, resource utilization and optimization iteration efficiency of deep learning inference systems. Attached Figure Description

[0070] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0071] Figure 1 This is a system schematic diagram of the flexible operator replacement experimental platform in Example 1. Detailed Implementation

[0072] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0073] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0074] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it a single or selective embodiment that is mutually exclusive with other embodiments.

[0075] Example 1, referring to Figure 1 This embodiment of the invention provides a flexible operator substitution experimental platform, comprising:

[0076] The system includes a rule parsing module, a sandbox construction module, a communication planning module, a device adaptation module, an operator switching module, an experiment monitoring module, and a strategy solidification module.

[0077] The rule parsing module is used to receive the operator replacement rule file, parse the target module matching conditions, candidate operator implementations, traffic allocation strategies and runtime switching conditions, and generate structured configuration information.

[0078] Furthermore, lexical and syntactic analysis is performed on the rule file to extract the name pattern or type identifier of the target module;

[0079] Establish matching criteria based on name patterns or type identifiers;

[0080] Read the implementation type of the candidate operators associated with the target module and their deployment device information;

[0081] Retrieve the request distribution weights or routing rules defined in the traffic allocation strategy;

[0082] Identify the performance threshold parameters and input feature variables involved in runtime switching conditions;

[0083] The above information is integrated into structured configuration data that includes matching criteria, candidate implementation list, distribution weight, and switching criteria;

[0084] It should be noted that by extracting target module matching conditions and generating structured configuration information through lexical and syntactic analysis, standardized parsing and unified management of operator replacement rules are achieved, avoiding manual configuration errors, improving the reliability and maintainability of rule loading, and providing a clear and executable data foundation for subsequent automated processing.

[0085] The sandbox building module is used to locate the target module in the model based on the matching conditions in the structured configuration information during the model loading phase, and create a runtime sandbox container for each target module. The sandbox container integrates the original module with multiple candidate operator implementations and supports traffic distribution based on request characteristics.

[0086] Furthermore, iterate through all submodules in the model and obtain the name and type of the current submodule;

[0087] Compare the name of the current submodule with the name pattern in the structured configuration information, or determine whether its type matches the type identifier;

[0088] When the comparison result meets the matching conditions, the submodule is retained as the original module instance;

[0089] Based on the candidate operator implementation type specified in the structured configuration information, initialize the replacement module instance on the corresponding computing device;

[0090] Register the original module and all candidate operator implementation instances into the same sandbox container;

[0091] Set the default active initial operator;

[0092] Configure traffic distribution logic so that subsequent inference requests can select different operator paths for execution based on request identifier, input length, or user tag;

[0093] It should be noted that by traversing the model and locating the target module based on name patterns or type identifiers, combined with the sandbox container's ability to encapsulate the original module and multiple candidate operators, transparent deployment and dynamic runtime routing of operator replacement are achieved. This maintains the integrity of the overall model structure and supports parallel experiments of multiple versions, enhancing the flexibility and scalability of the inference system.

[0094] The communication planning module is used to analyze the tensor dependencies between sandbox containers, determine cross-device transmission paths, and generate a data flow execution plan that includes transmission timing and memory layout requirements.

[0095] Furthermore, based on the data flow direction during the model's forward propagation, an input-output connection diagram is established between each sandbox container;

[0096] Identify tensor transport edges in a connected graph that span different computing devices;

[0097] Label the data volume for each cross-device transmission edge. With transmission frequency ,in, Derived from the dimension and data type of this tensor, This number originates from the number of times this path is called during the model inference process;

[0098] Calculate the data bandwidth requirement per unit time for each transmission path based on the labeled information. ;

[0099] based on The size of the transmission path is used to prioritize the transmission paths;

[0100] By combining the execution order of candidate operators, the insertion timing and synchronization point of each transmission operation in the overall execution flow are determined;

[0101] Generate a data flow execution plan that includes memory allocation locations, transmission start and end time windows, and execution flow divisions;

[0102] It should be noted that by marking the data volume and transmission frequency of the cross-device transmission path, and calculating the bandwidth requirements and generating the data flow execution plan accordingly, communication bottlenecks can be accurately identified, memory layout and transmission timing can be planned in advance, effectively reducing latency jitter caused by tensor transport, and improving the overall execution efficiency in heterogeneous environments.

[0103] The device adaptation module is used to pre-allocate a fixed memory space on the host side according to the data flow execution plan, configure an independent execution flow for the computing device, and distribute data transfer operations and computing tasks to different execution flows for parallel execution;

[0104] Furthermore, a fixed address region is reserved in the host memory for tensors that frequently participate in cross-device transfers, ensuring that this region supports zero-copy access;

[0105] Create at least two independent execution flows on the computing device side. and ,in, Specifically designed for performing data moving tasks. Used to perform core computing tasks;

[0106] According to the transmission timing defined in the data flow execution plan, Arrange asynchronous data copy instructions in the middle;

[0107] At the same time The forward computation kernel that schedules the current operator;

[0108] Introducing an event synchronization mechanism ,exist Triggered after data transfer is completed ,notify Start the computation task that depends on this data;

[0109] pass and Parallel scheduling enables time overlap between computation and communication;

[0110] It should be noted that by pre-allocating a fixed memory space on the host side and configuring an independent execution flow for the computing device, data transfer and core computing tasks are separated into different flows and executed concurrently. Combined with an event synchronization mechanism, the correctness of the execution order is ensured, and the multi-stream parallel capabilities of the GPU hardware are fully utilized to achieve efficient overlap of computing and communication, reducing idle waiting time.

[0111] The operator switching module is used to monitor the running status of each sandbox container during inference. When the running status meets the switching conditions in the structured configuration information, the currently active operator is replaced with the specified candidate operator in real time to maintain the continuity of interface calls.

[0112] Furthermore, the monitoring unit collects the context length of the currently processed request. Video memory usage and the time consumption of single-step reasoning ,in The number of tokens derived from the input sequence. Originating from the device memory monitoring interface, Derived from a high-precision timer;

[0113] The collected context length Video memory usage and the time consumption of single-step reasoning With the preset switching threshold in the structured configuration information Compare;

[0114] When satisfied or or When the switching condition is triggered, determine the switching condition.

[0115] Suspend new requests entering the current sandbox container;

[0116] After completing the request being executed, the function call pointer is changed from the original activation operator to the target candidate operator within the sandbox;

[0117] It should be noted that by collecting runtime status parameters such as context length, video memory usage, and inference time in real time, and triggering operator switching when preset thresholds are met, adaptive optimization capabilities based on actual load characteristics are achieved. This enables the system to automatically select the optimal operator path under different input conditions, thereby improving resource utilization and service stability.

[0118] The experimental monitoring module is used to collect latency, memory usage, and throughput performance data of each operator according to the traffic distribution path during the parallel operation of multiple candidate operators, and form a labeled performance record by combining the request context information.

[0119] Furthermore, each time a request enters the sandbox container, the specific operator path identifier assigned to that request is recorded. ;

[0120] Start a high-precision timer and record the end-to-end response time after the operator completes the forward computation. ;

[0121] Obtain the peak video memory consumption of the computing device during this calculation process from the system monitoring interface. ;

[0122] Statistical unit time Number of requests successfully processed Calculate the actual throughput. ;

[0123] Will , , , The corresponding request context features are written together into the log storage area to form a complete performance record;

[0124] It should be noted that by recording the latency, memory usage, and throughput performance data of each candidate operator according to the traffic distribution path, and combining this with the request context to form tagged performance records, a fine-grained and traceable experimental evaluation system was constructed. This ensures that the performance comparison results truly reflect the performance differences of different operators in specific scenarios, providing a reliable basis for scientific decision-making.

[0125] The strategy solidification module is used to perform performance comparison analysis based on the identified performance records, determine the optimal operator implementation, set the implementation as the unique active instance of the corresponding sandbox container, and output a model deployment description containing the final configuration.

[0126] Furthermore, the performance record set of all candidate operators under a certain target module is read from the log storage area;

[0127] Delay for each candidate operator Video memory usage and throughput Calculate the statistical values ​​separately, including the mean. ;

[0128] Based on preset performance evaluation weights The indicators are normalized and weighted summed to obtain a comprehensive score. Its expression is:

[0129] ;

[0130] in, The minimum average delay among all candidate operators. To minimize average video memory usage, This represents the maximum average throughput.

[0131] Select comprehensive score The highest-ranking candidate operator is selected as the optimal implementation.

[0132] Update the activation status of the sandbox container corresponding to the target module so that it retains only the optimal operator as the only available instance;

[0133] Generate a new configuration file containing all optimized target modules and their final selected operator types, device locations, and execution parameters;

[0134] It should be noted that by using a normalized weighted scoring model to comprehensively evaluate multi-dimensional performance indicators, the optimal operator is automatically identified and the strategy is solidified and output. This achieves a closed-loop transformation from experimental data to the final deployment plan, reduces manual intervention, and improves the automation level of the operator optimization process and the efficiency of engineering implementation.

[0135] This embodiment also provides a computer device suitable for a flexible operator replacement experimental platform, including: a memory and a processor; the memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions to realize the flexible operator replacement experimental platform proposed in the above embodiment.

[0136] The computer device can be a terminal, comprising a processor, memory, communication interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, carrier networks, NFC (Near Field Communication), or other technologies. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad on the computer device's casing, or an external keyboard, touchpad, or mouse.

[0137] This embodiment also provides a storage medium on which a computer program is stored. When executed by a processor, the program implements the flexible operator substitution experimental platform proposed in the above embodiments. The storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read Only Memory (EPROM), Programmable Red-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0138] In summary, this invention constructs an operator replacement experimental architecture that integrates rule parsing, sandbox encapsulation, communication optimization, and dynamic decision-making. This enables parallel testing, performance comparison, and adaptive switching of multiple operator implementations without interrupting inference services. It solves the problems of irreversible operator replacement, high experimental costs, large communication overhead in heterogeneous deployment, and lack of runtime awareness in traditional methods, thereby improving the flexibility, resource utilization, and optimization iteration efficiency of deep learning inference systems.

[0139] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A flexible operator substitution experimental platform, characterized in that: include: The system includes a rule parsing module, a sandbox construction module, a communication planning module, a device adaptation module, an operator switching module, an experiment monitoring module, and a strategy solidification module. The rule parsing module is used to receive the operator replacement rule file, parse the target module matching conditions, candidate operator implementations, traffic allocation strategies and runtime switching conditions in it, and generate structured configuration information. The sandbox construction module is used to locate the target module in the model according to the matching conditions in the structured configuration information during the model loading phase, and create a runtime sandbox container for each target module. The sandbox container integrates the original module and multiple candidate operators to support traffic distribution based on request characteristics. The communication planning module is used to analyze the tensor dependencies between sandbox containers, determine cross-device transmission paths, and generate a data flow execution plan that includes transmission timing and memory layout requirements. The device adaptation module is used to pre-allocate a fixed memory space on the host side according to the data flow execution plan, configure an independent execution flow for the computing device, and assign data transfer operations and computing tasks to different execution flows for parallel execution. The operator switching module is used to monitor the operating status of each sandbox container during the inference process. When the operating status meets the switching conditions in the structured configuration information, the currently active operator is replaced with the specified candidate operator in real time to maintain the continuity of interface calls. The experimental monitoring module is used to collect latency, memory usage and throughput performance data of each operator according to the traffic distribution path during the parallel operation of multiple candidate operators, and form a performance record with identification by combining request context information. The strategy solidification module is used to perform performance comparison analysis based on the identified performance records, determine the optimal operator implementation, set the implementation as the unique active instance of the corresponding sandbox container, and output a model deployment description containing the final configuration.

2. The flexible operator substitution experimental platform as described in claim 1, characterized in that: The receiving operator replacement rule file is parsed to extract target module matching conditions, candidate operator implementations, traffic allocation strategies, and runtime switching conditions, generating structured configuration information. The specific steps are as follows: Perform lexical and syntactic analysis on the rule file to extract the name pattern or type identifier of the target module; Establish matching criteria based on name patterns or type identifiers; Read the implementation type of the candidate operators associated with the target module and their deployment device information; Retrieve the request distribution weights or routing rules defined in the traffic allocation strategy; Identify the performance threshold parameters and input feature variables involved in runtime switching conditions; The target module's name pattern or type identifier, candidate operator implementation types and their deployment device information, request distribution weights or routing rules, and performance threshold parameters are integrated with input feature variables into structured configuration data that includes matching criteria, candidate implementation list, distribution weights, and switching criteria.

3. The flexible operator substitution experimental platform as described in claim 2, characterized in that: During the model loading phase, the target module in the model is located based on the matching conditions in the structured configuration information, and a runtime sandbox container is created for each target module. The sandbox container integrates the original module with multiple candidate operators and supports traffic distribution based on request characteristics. The specific steps are as follows: Iterate through all submodules in the model and obtain the name and type of the current submodule; Compare the name of the current submodule with the name pattern in the structured configuration information, or determine whether its type matches the type identifier; When the comparison result meets the matching conditions, the submodule is retained as the original module instance; Based on the candidate operator implementation type specified in the structured configuration information, initialize the alternative module instance on the corresponding computing device; Register the original module and all candidate operator implementation instances into the same sandbox container; Set the default active initial operator; Configure traffic distribution logic so that subsequent inference requests can select different operator paths for execution based on request identifier, input length, or user tag.

4. The flexible operator substitution experimental platform as described in claim 3, characterized in that: The analysis of tensor dependencies between sandbox containers determines cross-device transmission paths and generates a data flow execution plan that includes transmission timing and memory layout requirements. The specific steps are as follows: Based on the data flow direction during the model's forward propagation, an input-output connection diagram is established between each sandbox container. Identify tensor transport edges in a connected graph that span different computing devices; Label the data volume for each cross-device transmission edge. With transmission frequency ,in, Derived from the dimension and data type of this tensor, This number originates from the number of times this path is called during the model inference process; Calculate the data bandwidth requirement per unit time for each transmission path based on the labeled information. ; based on The size of the transmission path is used to prioritize the transmission paths; By combining the execution order of candidate operators, the insertion timing and synchronization point of each transmission operation in the overall execution flow are determined; A data flow execution plan is formed, which includes memory allocation locations, transmission start and end time windows, and execution flow divisions.

5. The flexible operator substitution experimental platform as described in claim 4, characterized in that: Based on the data flow execution plan, a fixed memory space is pre-allocated on the host side, an independent execution flow is configured for the computing device, and data transfer operations and computing tasks are assigned to different execution flows for parallel execution. The specific steps are as follows: Reserve a fixed address region in the host memory for tensors that frequently participate in cross-device transfers, and ensure that this region supports zero-copy access. Create at least two independent execution flows on the computing device side. and ,in, Specifically designed for performing data moving tasks. Used to perform core computing tasks; According to the transmission timing defined in the data flow execution plan, Arrange asynchronous data copy instructions in the middle; At the same time The forward computation kernel that schedules the current operator; Introducing an event synchronization mechanism ,exist Triggered after data transfer is completed ,notify Start the computation task that depends on this data; pass and Parallel scheduling enables time overlap between computation and communication.

6. The flexible operator substitution experimental platform as described in claim 5, characterized in that: The process involves monitoring the operational status of each sandbox container during inference. When the operational status meets the switching conditions in the structured configuration information, the currently active operator is replaced in real time with a specified candidate operator to maintain the continuity of interface calls. The specific steps are as follows: The monitoring unit collects the context length of the currently processed request. Video memory usage and the time consumption of single-step reasoning ,in The number of tokens derived from the input sequence. Originating from the device memory monitoring interface, Derived from a high-precision timer; The collected context length Video memory usage and the time consumption of single-step reasoning The switching threshold preset in the structured configuration information Compare; When satisfied or or When the switching condition is triggered, determine the switching condition. Suspend new requests entering the current sandbox container; After completing the request being executed, the function call pointer is changed from the original activation operator to the target candidate operator within the sandbox.

7. The flexible operator substitution experimental platform as described in claim 6, characterized in that: During the parallel operation of multiple candidate operators, latency, memory usage, and throughput performance data of each operator are collected according to the traffic distribution path, and combined with request context information to form an identified performance record. The specific steps are as follows: Each time a request enters the sandbox container, the specific operator path identifier assigned to that request is recorded. ; Start a high-precision timer and record the end-to-end response time after the operator completes the forward computation. ; Obtain the video memory usage of the computing device during this calculation from the system monitoring interface. ; Statistical unit time Number of requests successfully processed Calculate throughput ; Will , , , Together with the corresponding request context features, they are written into the log storage area to form a complete performance record.

8. The flexible operator substitution experimental platform as described in claim 7, characterized in that: The steps are as follows: Based on the identified performance records, a performance comparison analysis is performed to determine the optimal operator implementation. This implementation is then set as the unique active instance of the corresponding sandbox container, and a model deployment description containing the final configuration is output. Read the performance record set of all candidate operators under a target module from the log storage area; Delay for each candidate operator Video memory usage and throughput Calculate the statistical values ​​separately, including the mean. ; Based on preset performance evaluation weights The indicators are normalized and weighted summed to obtain a comprehensive score. Its expression is: ； in, The minimum average delay among all candidate operators. To minimize average video memory usage, This represents the maximum average throughput. Select comprehensive score The highest-ranking candidate operator is selected as the optimal implementation. Update the activation status of the sandbox container corresponding to the target module so that it retains only the optimal operator as the only available instance; A new configuration file is generated, which contains all optimized target modules and their final selected operator types, device locations, and execution parameters.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, it implements the flexible operator substitution experimental platform as described in any one of claims 1 to 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by the processor, it implements the flexible operator substitution experimental platform as described in any one of claims 1 to 8.

Images