Flexible operator replacement experiment platform

Abstract

The invention discloses a flexible operator replacement experiment platform, which relates to the technical field of GPU operator replacement experiments and comprises a rule analysis module, a sandbox construction module, a communication planning module, an equipment adaptation module, an operator switching module, an experiment monitoring module and a strategy solidification module. The rule analysis module is used for receiving an operator replacement rule file, analyzing a target module matching condition, candidate operator implementation, a flow distribution strategy and a runtime switching condition in the operator replacement rule file, and generating structured configuration information; the sandbox construction module is used for positioning target modules in a model in a model loading stage according to matching conditions in the structured configuration information and creating a runtime sandbox container for each target module, and the sandbox container integrates an original module and a plurality of candidate operator implementation and supports flow distribution based on request features; and the communication planning module is used for analyzing a tensor dependency relationship between the sandbox containers and determining a cross-device transmission path.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of GPU operator replacement experiments, and in particular to a flexible operator replacement experiment platform. BACKGROUND

[0002] The GPU operator replacement experiment technology refers to a technology system that provides multiple implementation versions for a specific neural network operator in a deep learning inference system, and performs parallel testing, performance comparison and optimal selection of these versions through sandbox encapsulation, traffic distribution, performance monitoring and dynamic decision mechanism at runtime. Therefore, how to use advanced technical means to improve the intelligent level and security of GPU operator replacement experiments has become one of the current problems to be solved.

[0003] In the field of GPU operator replacement experiments, the existing technology usually adopts a static compilation or manual replacement method, which is fixed once the replacement is completed, and cannot perform parallel testing and performance comparison of multiple candidate operators at runtime, resulting in a lack of data support for optimization decisions, high experimental cost and long cycle, and the traditional scheme is difficult to integrate multiple operator implementations in the same model and distribute traffic according to the requested features, cannot realize A / B testing or gray release, and limits the ability to safely verify new operators in a production environment. SUMMARY

[0004] In view of the above existing problems, the present application is proposed.

[0005] Therefore, the present application provides a flexible operator replacement experiment platform to solve the problem that the existing technology usually adopts a static compilation or manual replacement method, which is fixed once the replacement is completed, and cannot perform parallel testing and performance comparison of multiple candidate operators at runtime, resulting in a lack of data support for optimization decisions, high experimental cost and long cycle, and the traditional scheme is difficult to integrate multiple operator implementations in the same model and distribute traffic according to the requested features, cannot realize A / B testing or gray release, and limits the ability to safely verify new operators in a production environment.

[0006] To solve the above technical problems, the present application provides the following technical solutions: In a first aspect, the present application provides a flexible operator replacement experiment platform, which comprises: a rule analysis module, a sandbox construction module, a communication planning module, a device adaptation module, an operator switching module, an experiment monitoring module and a strategy solidification module; The rule analysis module is configured to receive an operator replacement rule file, analyze the target module matching conditions, candidate operator implementations, traffic distribution strategies and runtime switching conditions therein, and generate structured configuration information. The sandbox construction module is configured to locate target modules in the model according to matching conditions in the structured configuration information in a model loading stage, and create a runtime sandbox container for each target module, wherein the sandbox container integrates an original module and a plurality of candidate operator implementations, and supports traffic distribution based on a request feature. The communication planning module is configured to analyze tensor dependency relationships between sandbox containers, determine a cross-device transmission path, and generate a data flow execution plan containing transmission timing and memory layout requirements. The device adaptation module is configured to pre-allocate a fixed memory space on a host side according to the data flow execution plan, configure an independent execution flow for a computing device, and allocate data carrying operations and computing tasks to different execution flows for parallel execution. The operator switching module is configured to monitor the running state of each sandbox container during inference, and replace a currently activated operator implementation with a specified candidate operator in real time when the running state meets switching conditions in the structured configuration information, while maintaining interface call continuity. The experiment monitoring module is configured to collect delay, video memory occupation, and throughput performance data of each operator according to a traffic distribution path during parallel operation of multiple candidate operators, and form an identified performance record in combination with request context information. The strategy solidification module is configured to perform performance comparison and analysis based on the identified performance record, determine an optimal operator implementation, set the implementation as the only activated instance of the corresponding sandbox container, and output a model deployment description containing a final configuration.

[0007] As a preferred scheme of the flexible operator replacement experiment platform, the operator replacement rule file is received, and target module matching conditions, candidate operator implementations, traffic distribution strategies, and runtime switching conditions in the rule file are parsed to generate structured configuration information, and the specific steps are as follows: The rule file is subjected to lexical and syntactic analysis, and the name pattern or type identifier of the target module is extracted; The name pattern or type identifier is used to establish a matching judgment basis; The candidate operator implementation type and deployment device information associated with the target module are read; The request distribution weight or routing rule defined in the traffic distribution strategy is obtained; The performance threshold parameter and input feature variable involved in the runtime switching condition are identified; The above information is integrated into structured configuration data containing a matching basis, a candidate implementation list, a distribution weight, and a switching criterion.

[0008] As a preferred scheme of the flexible operator replacement experiment platform, wherein: in the model loading stage, the target module in the model is located according to the matching condition in the structured configuration information, and a runtime sandbox container is created for each target module, the sandbox container integrates the original module and a plurality of candidate operator implementations, supports traffic distribution based on request characteristics, and the specific steps are as follows: Traverse all sub-modules in the model to obtain the name and type of the current sub-module; Compare the name of the current sub-module with the name pattern in the structured configuration information, or determine whether the type meets the type identifier; When the comparison result meets the matching condition, the sub-module is retained as an original module instance; According to the candidate operator implementation type specified in the structured configuration information, initialize a replacement module instance on the corresponding computing device; Register the original module and all candidate operator implementation instances in the same sandbox container; Set the initial operator activated by default; Configure traffic distribution logic to make subsequent inference requests select different operator paths for execution according to request identifiers, input lengths or user labels.

[0009] As a preferred scheme of the flexible operator replacement experiment platform, wherein: the tensor dependency relationship between the analysis sandbox containers is analyzed, the cross-device transmission path is determined, and a data flow execution plan containing transmission timing and memory layout requirements is generated, and the specific steps are as follows: Based on the data flow direction in the model forward propagation process, an input-output connection graph between the sandbox containers is established; Identify the tensor transmission edges across different computing devices in the connection graph; Label the data volume of each cross-device transmission edge And the transmission frequency , wherein, The dimension and data type derived from the tensor, The number of calls of the path in the model inference process; According to the labeled information, calculate the unit time data bandwidth requirement of each transmission path ; Based on The size of the transmission path is prioritized; Combined with the execution order of the candidate operator, determine the insertion time and synchronization point of each transmission operation in the overall execution flow; Form a data flow execution plan containing memory allocation location, transmission start and end time window and execution flow division.

[0010] As a preferred embodiment of the flexible operator replacement experimental platform described in this invention, the steps of pre-allocating a fixed memory space on the host side according to the data flow execution plan, configuring an independent execution flow for the computing device, and allocating data transfer operations and computing tasks to different execution flows for parallel execution are as follows: Reserve a fixed address region in the host memory for tensors that frequently participate in cross-device transfers, and ensure that this region supports zero-copy access. Create at least two independent execution flows on the computing device side. and ,in, Specifically designed for performing data moving tasks. Used to perform core computing tasks; According to the transmission timing defined in the data flow execution plan, Arrange asynchronous data copy instructions in the middle; At the same time The forward computation kernel that schedules the current operator; Introducing an event synchronization mechanism ,exist Triggered after data transfer is completed ,notify Start the computation task that depends on this data; pass and Parallel scheduling enables time overlap between computation and communication.

[0011] As a preferred embodiment of the flexible operator replacement experimental platform described in this invention, the following steps are taken: During the inference process, the operating status of each sandbox container is monitored. When the operating status meets the switching conditions in the structured configuration information, the currently active operator is replaced in real time with a specified candidate operator, maintaining the continuity of interface calls. The monitoring unit collects the context length of the currently processed request. Video memory usage and the time consumption of single-step reasoning ,in The number of tokens derived from the input sequence. Originating from the device memory monitoring interface, Derived from a high-precision timer; The collected context length Video memory usage and the time consumption of single-step reasoning The switching threshold preset in the structured configuration information Compare; When satisfied or or When the switching condition is triggered, determine the switching condition. Suspend new requests entering the current sandbox container; After completing the request being executed, the function call pointer is changed from the original activation operator to the target candidate operator within the sandbox.

[0012] As a preferred embodiment of the flexible operator replacement experimental platform described in this invention, the following steps are taken: During the parallel operation of multiple candidate operators, latency, memory usage, and throughput performance data of each operator are collected according to the traffic distribution path, and combined with request context information to form an identified performance record. Each time a request enters the sandbox container, the specific operator path identifier assigned to that request is recorded. ; Start a high-precision timer and record the end-to-end response time after the operator completes the forward computation. ; Obtain the peak video memory consumption of the computing device during this calculation process from the system monitoring interface. ; Statistical unit time Number of requests successfully processed Calculate the actual throughput. ; Will , , , Together with the corresponding request context features, they are written into the log storage area to form a complete performance record.

[0013] As a preferred embodiment of the flexible operator replacement experimental platform described in this invention, the steps of performing performance comparison analysis based on identified performance records, determining the optimal operator implementation, setting this implementation as the unique active instance of the corresponding sandbox container, and outputting a model deployment description containing the final configuration are as follows: Read the performance record set of all candidate operators under a target module from the log storage area; Delay for each candidate operator Video memory usage and throughput Calculate the statistical values ​​separately, including the mean. ; Based on preset performance evaluation weights The indicators are normalized and weighted summed to obtain a comprehensive score. Its expression is: ; in, The minimum average delay among all candidate operators, To minimize average video memory usage, for the maximum average throughput rate; selecting the comprehensive score the highest candidate operator as the optimal implementation; updating the activation state of the sandbox container corresponding to the target module, so that it only retains the optimal operator as the only available instance; generating a new configuration file containing all the optimized target modules and their final selected operator types, device locations and execution parameters.

[0014] In a second aspect, the present application provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and wherein the computer program, when executed by the processor, implements any step of the flexible operator replacement experiment platform according to the first aspect of the present application.

[0015] In a third aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements any step of the flexible operator replacement experiment platform according to the first aspect of the present application.

[0016] The present application has the following beneficial effects: by constructing an operator replacement experiment architecture integrating rule analysis, sandbox encapsulation, communication optimization and dynamic decision-making, parallel testing, performance comparison and adaptive switching of multiple operator implementations are realized without interrupting inference services, solving the problems of irreversible operator replacement, high experimental cost, large communication overhead of heterogeneous deployment and lack of runtime sensing capability in traditional methods, and improving the flexibility, resource utilization and optimization iteration efficiency of deep learning inference systems. BRIEF DESCRIPTION OF DRAWINGS

[0017] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiment description. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained from these drawings without creative labor.

[0018] Figure 1 The system schematic diagram of the flexible operator replacement experiment platform in embodiment 1. DETAILED DESCRIPTION

[0019] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the specific embodiments of the present application will be described in detail below with reference to the drawings of the specification.

[0020] In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present application. However, it will be apparent to one skilled in the art that the present application can be practiced without the specific details set forth in this description in other implementations. Therefore, the scope of the present application understood by reference to the description and should not be limited by the specific examples disclosed herein.

[0021] Secondly, the "one embodiment" or "embodiment" referred to herein means a specific feature, structure, or characteristic that can be included in at least one implementation of the present application. "In one embodiment" appearing in different places in the specification does not mean the same embodiment, nor is it an independent or alternative embodiment mutually exclusive with other embodiments.

[0022] Embodiment 1, reference Figure 1 As an embodiment of the present application, the embodiment provides a flexible operator replacement experiment platform, comprising: a rule analysis module, a sandbox construction module, a communication planning module, a device adaptation module, an operator switching module, an experiment monitoring module, and a policy solidification module; The rule analysis module is configured to receive an operator replacement rule file, analyze the target module matching condition, candidate operator implementation, traffic distribution strategy, and runtime switching condition in the rule file, and generate structured configuration information. Furthermore, the rule file is subjected to lexical and syntactic analysis to extract the name pattern or type identifier of the target module. The name pattern or type identifier is used to establish a matching judgment basis. The candidate operator implementation type and its deployment device information associated with the target module are read. The request distribution weight or routing rule defined in the traffic distribution strategy is obtained. The performance threshold parameter and input feature variable involved in the runtime switching condition are identified. The above information is integrated into structured configuration data containing the matching basis, candidate implementation list, distribution weight, and switching criterion. It should be noted that the target module matching condition is extracted through lexical and syntactic analysis, and structured configuration information is generated, which realizes standardized analysis and unified management of operator replacement rules, avoids manual configuration errors, improves the reliability and maintainability of rule loading, and provides a clear and executable data basis for subsequent automated processing.

[0023] The sandbox construction module is configured to locate the target module in the model according to the matching condition in the structured configuration information during the model loading stage, and create a runtime sandbox container for each target module. The sandbox container integrates the original module and multiple candidate operator implementations, and supports traffic distribution based on request features. Furthermore, the name and type of the current sub-module are obtained by traversing all sub-modules in the model. Compare the name of the current submodule with the name pattern in the structured configuration information, or determine whether its type meets the type identifier; When the comparison result meets the matching condition, retain the submodule as the original module instance; According to the candidate operator implementation type specified in the structured configuration information, initialize the replacement module instance on the corresponding computing device; Register the original module and all candidate operator implementation instances in the same sandbox container; Set the initial operator activated by default; Configure the traffic distribution logic, so that subsequent inference requests select different operator paths for execution according to request identifiers, input lengths, or user tags; It should be noted that by traversing the model and locating the target module based on the name pattern or type identifier, and combining the encapsulation capabilities of the sandbox container for the original module and multiple candidate operators, transparent deployment and runtime dynamic routing of operator replacement are achieved, which not only maintains the integrity of the overall model structure, but also supports multiple version parallel experiments, enhancing the flexibility and scalability of the inference system.

[0024] The communication planning module is used to analyze the tensor dependency relationship between sandbox containers, determine the cross-device transmission path, and generate a data flow execution plan containing transmission timing and memory layout requirements; Further, based on the data flow direction in the model forward propagation process, an input-output connection graph is established between the sandbox containers; Identify the tensor transmission edges in the connection graph that span different computing devices; Label the data volume for each cross-device transmission edge And the transmission frequency , wherein The dimensions and data types derived from the tensor, The number of calls of this path in the model inference process; According to the labeled information, calculate the unit time data bandwidth requirement of each transmission path ; Based on The size of the transmission path is prioritized; Combined with the execution order of the candidate operators, determine the insertion time and synchronization point of each transmission operation in the overall execution flow; Form a data flow execution plan containing memory allocation location, transmission start and end time window, and execution flow division; It should be noted that by marking the data volume and transmission frequency of the cross-device transmission path, and calculating the bandwidth requirements and generating the data flow execution plan accordingly, communication bottlenecks can be accurately identified, memory layout and transmission timing can be planned in advance, effectively reducing latency jitter caused by tensor transport, and improving the overall execution efficiency in heterogeneous environments.

[0025] The device adaptation module is used to pre-allocate a fixed memory space on the host side according to the data flow execution plan, configure an independent execution flow for the computing device, and distribute data transfer operations and computing tasks to different execution flows for parallel execution; Furthermore, a fixed address region is reserved in the host memory for tensors that frequently participate in cross-device transfers, ensuring that this region supports zero-copy access; Create at least two independent execution flows on the computing device side. and ,in, Specifically designed for performing data moving tasks. Used to perform core computing tasks; According to the transmission timing defined in the data flow execution plan, Arrange asynchronous data copy instructions in the middle; At the same time The forward computation kernel that schedules the current operator; Introducing an event synchronization mechanism ,exist Triggered after data transfer is completed ,notify Start the computation task that depends on this data; pass and Parallel scheduling enables time overlap between computation and communication; It should be noted that by pre-allocating a fixed memory space on the host side and configuring an independent execution flow for the computing device, data transfer and core computing tasks are separated into different flows and executed concurrently. Combined with an event synchronization mechanism, the correctness of the execution order is ensured, and the multi-stream parallel capabilities of the GPU hardware are fully utilized to achieve efficient overlap of computing and communication, reducing idle waiting time.

[0026] The operator switching module is used to monitor the running status of each sandbox container during inference. When the running status meets the switching conditions in the structured configuration information, the currently active operator is replaced with the specified candidate operator in real time to maintain the continuity of interface calls. Furthermore, the monitoring unit collects the context length of the currently processed request. Video memory usage and the time consumption of single-step reasoning ,in The number of tokens derived from the input sequence. derived from the memory monitoring interface of the device, derived from the high-precision timer; the collected context length , the memory usage and the single-step reasoning time consumption are compared with the preset switching threshold in the structured configuration information; when or or is met, it is determined that the switching condition is triggered; pause new requests from entering the current sandbox container; after completing the request being executed, the function call pointer in the sandbox is pointed to the target candidate operator from the original active operator; It should be noted that by collecting real-time running state parameters such as context length, memory usage and reasoning time consumption, and triggering operator switching when the preset threshold is met, the adaptive optimization capability based on actual load characteristics is realized, so that the system can automatically select the optimal operator path under different input conditions, and the resource utilization rate and service stability are improved.

[0027] The experimental monitoring module is used to collect the delay, memory usage and throughput performance data of each operator according to the traffic distribution path during the parallel running of multiple candidate operators, and form performance records with identifiers in combination with the request context information; Furthermore, when each request enters the sandbox container, the specific operator path identifier assigned to the request is recorded; start the high-precision timer and record the end-to-end response time after the operator completes the forward calculation ; obtain the peak memory consumption of the computing device in this calculation process from the system monitoring interface ; count the number of successfully processed requests per unit time , and calculate the actual throughput rate ; , , , , and the corresponding request context features are written into the log storage area together to form complete performance records; It should be noted that by recording the delay, memory and throughput performance data of each candidate operator according to the traffic distribution path, and forming performance records with identifiers in combination with the request context, a fine-grained and traceable experimental evaluation system is constructed, which ensures that the performance comparison results truly reflect the performance differences of different operators in specific scenarios, and provides a reliable basis for scientific decision-making.​​

[0028] a strategy solidification module configured to perform performance comparison analysis according to the performance records with labels, determine an optimal operator implementation, set the implementation as the only active instance of the corresponding sandbox container, and output a model deployment description containing a final configuration; Further, the strategy solidification module reads a performance record set of all candidate operators under a target module from a log storage area; calculates statistical values including mean values for the delay , memory usage , and throughput of each candidate operator, respectively; ; According to a preset performance evaluation weight , the indicators are normalized and weighted summed to obtain a comprehensive score , and the expression is: ; wherein, is the minimum average delay among all candidate operators, is the minimum average memory usage, and is the maximum average throughput; The candidate operator with the highest comprehensive score is selected as the optimal implementation; The active state of the sandbox container corresponding to the target module is updated so that it only retains the optimal operator as the only available instance; A new configuration file is generated, which contains all the target modules that have completed optimization and the final selected operator type, device location, and execution parameters thereof; It should be noted that the multi-dimensional performance indicators are comprehensively evaluated by the normalized weighted score model, the optimal operator implementation is automatically identified, and the strategy solidification output is completed, realizing the closed-loop conversion from experimental data to the final deployment scheme, reducing manual intervention, and improving the automation level and engineering landing efficiency of the operator optimization process.

[0029] The embodiment also provides a computer device suitable for the flexible operator replacement experiment platform, including a memory and a processor; the memory is configured to store computer executable instructions, and the processor is configured to execute the computer executable instructions to implement the flexible operator replacement experiment platform proposed in the above embodiment.

[0030] The computer device can be a terminal, and the computer device includes a processor, a memory, a communication interface, a display screen and an input device connected by a system bus. The processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for running the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is configured to perform wired or wireless communication with an external terminal. The wireless communication can be achieved by WIFI, an operator network, NFC (Near Field Communication) or other technologies. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer overlaid on the display screen, or a key, trackball or touchpad arranged on the shell of the computer device, or an external keyboard, touchpad or mouse, etc.

[0031] The embodiment also provides a storage medium having a computer program stored thereon, and the computer program is executed by a processor to implement the flexible operator replacement experiment platform provided by the above embodiment. The storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as a static random access memory (SRAM), an electrically erasable programmable read-only memory (EEPROM), an erasable programmable read-only memory (EPROM), a programmable read-only memory (PROM), a read-only memory (ROM), a magnetic storage, a flash memory, a magnetic disk or an optical disk.

[0032] To sum up, the application constructs an operator replacement experiment architecture integrating rule analysis, sandbox encapsulation, communication optimization and dynamic decision making, realizes parallel testing, performance comparison and adaptive switching of various operator implementations without interrupting inference services, solves the problems of irreversible operator replacement, high experiment cost, large communication overhead of heterogeneous deployment and lack of runtime sensing capability in the traditional method, and improves the flexibility, resource utilization and optimization iteration efficiency of the deep learning inference system.

[0033] It should be noted that the above examples are only used to illustrate the technical solutions of the present application but not limit the present application. Although the present application is described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present application can be modified or equivalently replaced, without departing from the spirit and scope of the technical solutions of the present application, which should be covered in the scope of the claims of the present application.

Claims

1. A flexible operator substitution experimental platform, characterized in that: include: The system includes a rule parsing module, a sandbox construction module, a communication planning module, a device adaptation module, an operator switching module, an experiment monitoring module, and a strategy solidification module. The rule parsing module is used to receive the operator replacement rule file, parse the target module matching conditions, candidate operator implementations, traffic allocation strategies and runtime switching conditions in it, and generate structured configuration information. The sandbox construction module is used to locate the target module in the model according to the matching conditions in the structured configuration information during the model loading phase, and create a runtime sandbox container for each target module. The sandbox container integrates the original module and multiple candidate operators to support traffic distribution based on request characteristics. The communication planning module is used to analyze the tensor dependencies between sandbox containers, determine cross-device transmission paths, and generate a data flow execution plan that includes transmission timing and memory layout requirements. The device adaptation module is used to pre-allocate a fixed memory space on the host side according to the data flow execution plan, configure an independent execution flow for the computing device, and assign data transfer operations and computing tasks to different execution flows for parallel execution. The operator switching module is used to monitor the operating status of each sandbox container during the inference process. When the operating status meets the switching conditions in the structured configuration information, the currently active operator is replaced with the specified candidate operator in real time to maintain the continuity of interface calls. The experimental monitoring module is used to collect latency, memory usage and throughput performance data of each operator according to the traffic distribution path during the parallel operation of multiple candidate operators, and form a performance record with identification by combining request context information. The strategy solidification module is used to perform performance comparison analysis based on the identified performance records, determine the optimal operator implementation, set the implementation as the unique active instance of the corresponding sandbox container, and output a model deployment description containing the final configuration.

2. The flexible operator substitution experimental platform as described in claim 1, characterized in that: The receiving operator replacement rule file is parsed to extract target module matching conditions, candidate operator implementations, traffic allocation strategies, and runtime switching conditions, generating structured configuration information. The specific steps are as follows: Perform lexical and syntactic analysis on the rule file to extract the name pattern or type identifier of the target module; Establish matching criteria based on name patterns or type identifiers; Read the implementation type of the candidate operators associated with the target module and their deployment device information; Retrieve the request distribution weights or routing rules defined in the traffic allocation strategy; Identify the performance threshold parameters and input feature variables involved in runtime switching conditions; The above information is integrated into structured configuration data that includes matching criteria, candidate implementation list, distribution weights, and switching criteria.

3. The flexible operator substitution experimental platform as described in claim 2, characterized in that: During the model loading phase, the target module in the model is located based on the matching conditions in the structured configuration information, and a runtime sandbox container is created for each target module. The sandbox container integrates the original module with multiple candidate operators and supports traffic distribution based on request characteristics. The specific steps are as follows: Iterate through all submodules in the model and obtain the name and type of the current submodule; Compare the name of the current submodule with the name pattern in the structured configuration information, or determine whether its type matches the type identifier; When the comparison result meets the matching conditions, the submodule is retained as the original module instance; Based on the candidate operator implementation type specified in the structured configuration information, initialize the alternative module instance on the corresponding computing device; Register the original module and all candidate operator implementation instances into the same sandbox container; Set the default active initial operator; Configure traffic distribution logic so that subsequent inference requests can select different operator paths for execution based on request identifier, input length, or user tag.

4. The flexible operator substitution experimental platform as described in claim 3, characterized in that: The analysis of tensor dependencies between sandbox containers determines cross-device transmission paths and generates a data flow execution plan that includes transmission timing and memory layout requirements. The specific steps are as follows: Based on the data flow direction during the model's forward propagation, an input-output connection diagram is established between each sandbox container. Identify tensor transport edges in a connected graph that span different computing devices; Label the data volume for each cross-device transmission edge. With transmission frequency ,in, Derived from the dimension and data type of this tensor, This number originates from the number of times this path is called during the model inference process; Calculate the data bandwidth requirement per unit time for each transmission path based on the labeled information. ; based on The size of the transmission path is used to prioritize the transmission paths; By combining the execution order of candidate operators, the insertion timing and synchronization point of each transmission operation in the overall execution flow are determined; A data flow execution plan is formed, which includes memory allocation locations, transmission start and end time windows, and execution flow divisions.

5. The flexible operator substitution experimental platform as described in claim 4, characterized in that: Based on the data flow execution plan, a fixed memory space is pre-allocated on the host side, an independent execution flow is configured for the computing device, and data transfer operations and computing tasks are assigned to different execution flows for parallel execution. The specific steps are as follows: Reserve a fixed address region in the host memory for tensors that frequently participate in cross-device transfers, and ensure that this region supports zero-copy access. Create at least two independent execution flows on the computing device side. and ,in, Specifically designed for performing data moving tasks. Used to perform core computing tasks; According to the transmission timing defined in the data flow execution plan, Arrange asynchronous data copy instructions in the middle; At the same time The forward computation kernel that schedules the current operator; Introducing an event synchronization mechanism ,exist Triggered after data transfer is completed ,notify Start the computation task that depends on this data; pass and Parallel scheduling enables time overlap between computation and communication.

6. The flexible operator substitution experimental platform as described in claim 5, characterized in that: The process involves monitoring the operational status of each sandbox container during inference. When the operational status meets the switching conditions in the structured configuration information, the currently active operator is replaced in real time with a specified candidate operator to maintain the continuity of interface calls. The specific steps are as follows: The monitoring unit collects the context length of the currently processed request. Video memory usage and the time consumption of single-step reasoning ,in The number of tokens derived from the input sequence. Originating from the device memory monitoring interface, Derived from a high-precision timer; The collected context length Video memory usage and the time consumption of single-step reasoning The switching threshold preset in the structured configuration information Compare; When satisfied or or When the switching condition is triggered, determine the switching condition. Suspend new requests entering the current sandbox container; After completing the request being executed, the function call pointer is changed from the original activation operator to the target candidate operator within the sandbox.

7. The flexible operator substitution experimental platform as described in claim 6, characterized in that: During the parallel operation of multiple candidate operators, latency, memory usage, and throughput performance data of each operator are collected according to the traffic distribution path, and combined with request context information to form an identified performance record. The specific steps are as follows: Each time a request enters the sandbox container, the specific operator path identifier assigned to that request is recorded. ; Start a high-precision timer and record the end-to-end response time after the operator completes the forward computation. ; Obtain the peak video memory consumption of the computing device during this calculation process from the system monitoring interface. ; Statistical unit time Number of requests successfully processed Calculate the actual throughput. ; Will , , , Together with the corresponding request context features, they are written into the log storage area to form a complete performance record.

8. The flexible operator substitution experimental platform as described in claim 7, characterized in that: The steps are as follows: Based on the identified performance records, a performance comparison analysis is performed to determine the optimal operator implementation. This implementation is then set as the unique active instance of the corresponding sandbox container, and a model deployment description containing the final configuration is output. Read the performance record set of all candidate operators under a target module from the log storage area; Delay for each candidate operator Video memory usage and throughput Calculate the statistical values ​​separately, including the mean. ; Based on preset performance evaluation weights The indicators are normalized and weighted summed to obtain a comprehensive score. Its expression is: ； in, The minimum average delay among all candidate operators. To minimize average video memory usage, This represents the maximum average throughput. Select comprehensive score The highest-ranking candidate operator is selected as the optimal implementation. Update the activation status of the sandbox container corresponding to the target module so that it retains only the optimal operator as the only available instance; A new configuration file is generated, which contains all optimized target modules and their final selected operator types, device locations, and execution parameters.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, it implements the steps of the flexible operator replacement experimental platform as described in any one of claims 1 to 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by the processor, it implements the steps of the flexible operator replacement experimental platform as described in any one of claims 1 to 8.

Images