Operator call request processing method and electronic device

Abstract

The application discloses an operator calling request processing method and electronic equipment, and relates to the technical field of deep learning framework, and comprises the following steps: obtaining an operator calling request and context information of the operator calling request; if a first operator cannot process a to-be-processed task, obtaining an information mapping rule corresponding to the operator calling request; mapping the context information by using the information mapping rule, determining a target function of a second hardware according to a mapping result; calling the second operator in the second hardware according to the target function, and processing the to-be-processed task by using the second operator. The problem that an adaptation layer needs to be separately developed to make a deep learning framework support different artificial intelligence chip operators and the adaptation cost is high can be solved. The method is suitable for various deep learning frameworks, and makes the deep learning framework support different artificial intelligence chip operators. An adaptation layer does not need to be separately developed, new hardware can be supported by adjusting the information mapping rule and the target function, and the adaptation cost is reduced.

Description

Technical Field

[0001] This invention relates to the field of deep learning framework technology, specifically to an operator call request processing method and an electronic device. Background Technology

[0002] Operators are the computational units of artificial intelligence (AI) models. Currently, operator computation is implemented using dedicated AI chips, and a single AI chip can perform computations on multiple operators. Operators with the same function can be implemented completely differently on different AI chips. Chip manufacturers provide an operator library containing the implementation logic of each operator and its corresponding interface description. When a deep learning framework connects to a specific chip, it essentially loads the chip's operator library and calls the chip's operators.

[0003] To enable deep learning frameworks to call different chip operators without altering upper-layer code, they provide an adapter layer and a registration mechanism. The adapter layer encapsulates the chip's operators, with each operator corresponding to an interface in the adapter layer. The registration mechanism assigns the adapter layer's interface function pointers to variables reserved by the framework, allowing the framework to indirectly call the chip's operators by accessing its own variables. For a deep learning framework to support operators from different AI chips, it's necessary to implement adapter layers for operators with the same functionality on different AI chips and register these adapter layer functions with the framework. Furthermore, since different deep learning frameworks use different adapter layers, chip manufacturers must develop a separate adapter layer for each framework, resulting in high adaptation costs. Summary of the Invention

[0004] This invention provides an operator call request processing method and an electronic device to solve the problem that in order to enable deep learning frameworks to support different artificial intelligence chip operators, it is necessary to develop an adaptation layer separately, which results in high adaptation costs.

[0005] Firstly, this application provides an operator invocation request processing method, the method comprising:

[0006] Obtain the operator call request, wherein the operator call request contains context information, which is used to determine the first operator called in the first hardware and the task to be processed corresponding to the operator call request;

[0007] If the first operator cannot process the task to be processed, obtain the information mapping rule corresponding to the operator call request, wherein the information mapping rule is used to map the context information to the information corresponding to the second operator in the second hardware;

[0008] Based on the information mapping rules and context information, the mapping result is obtained, and the objective function of the second hardware processing task is determined based on the mapping result.

[0009] The second operator in the second hardware is called according to the objective function, and the task to be processed is handled by the second operator.

[0010] Secondly, this application provides an operator invocation request processing apparatus, the apparatus comprising:

[0011] The request acquisition module is used to acquire operator call requests, wherein the operator call request contains context information, which is used to determine the first operator called in the first hardware and the task to be processed corresponding to the operator call request;

[0012] The rule acquisition module is used to acquire the information mapping rule corresponding to the operator call request when the first operator cannot process the task to be processed. The information mapping rule is used to map the context information to the information corresponding to the second operator in the second hardware.

[0013] The function determination module is used to obtain the mapping result based on the information mapping rules and context information, and to determine the target function of the second hardware processing task based on the mapping result;

[0014] The operator invocation module is used to invoke the second operator in the second hardware according to the target function, and to use the second operator to process the task to be processed.

[0015] Thirdly, this application provides an electronic device, including: a memory and a processor, which are communicatively connected to each other. The memory stores computer instructions, and the processor executes the computer instructions to perform the operator call request processing method of the first aspect or any corresponding embodiment described above.

[0016] Fourthly, this application provides a computer-readable storage medium storing computer instructions for causing a computer to execute the operator call request processing method of the first aspect or any corresponding embodiment described above.

[0017] Fifthly, this application provides a computer program product, including computer instructions for causing a computer to execute the operator call request processing method of the first aspect or any corresponding embodiment described above.

[0018] This application addresses the problem of high adaptation costs associated with developing separate adaptation layers to support different AI chip operators in deep learning frameworks. This method obtains an operator invocation request and the context information contained within it; when the first operator cannot handle the task, it obtains the information mapping rule corresponding to the operator invocation request; it maps the context information using the information mapping rule and determines the objective function of the second hardware based on the mapping result; and it calls the second operator in the second hardware according to the objective function to process the task. This method is applicable to multiple deep learning frameworks, enabling them to support different AI chip operators. It eliminates the need for separate adaptation layer development, reducing development effort. Furthermore, using the objective function of the second hardware improves task processing performance, and by adjusting the information mapping rule and objective function, it supports new hardware, further reducing adaptation costs. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the specific embodiments or related technologies of this application, the drawings used in the description of the specific embodiments or related technologies will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0020] Figure 1 This is a flowchart illustrating the operator invocation request processing method according to an embodiment of this application;

[0021] Figure 2 This is a schematic diagram of an operator invocation request processing system according to an embodiment of this application;

[0022] Figure 3 This is a schematic diagram of the operator registration method invocation flow according to an embodiment of this application;

[0023] Figure 4 This is a structural block diagram of an operator invocation request processing apparatus according to an embodiment of this application;

[0024] Figure 5 This is a schematic diagram of the hardware structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0025] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.

[0026] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., in this application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0027] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0028] Deep learning frameworks serve deep learning tasks. Deep learning relies on an AI model, which is trained to perform specific tasks such as speech recognition and image classification. An AI model consists of many neural network layers, each with a different function. Commonly used layers include convolutional layers, fully connected layers, activation layers, and pooling layers. A deep learning framework is software that abstracts away the complexities and cumbersome operation of the underlying hardware, providing simple application functions to allow users to better utilize machine resources and complete their customized tasks. Currently, common deep learning frameworks include PyTorch, TensorFlow, and PaddlePaddle.

[0029] Hardware operators are the computational units of AI models, and they can be used to implement the functions of neural network layers. A simple neural network layer corresponds to one operator, while a complex neural network corresponds to multiple operators. Initially, when models were simple, the computation of operators using a central processing unit (CPU) was sufficient to meet the developers' needs. However, as the complexity of the models increased, CPU computation became too slow; training a complex model typically took half a month or even a month. Therefore, dedicated AI chips were developed to implement operator computation. Commonly used AI chips include GPUs, NPUs, and MLUs.

[0030] The implementation of operators with the same function varies significantly across different AI chips. Typically, chip manufacturers provide a C++ operator library containing methods for calling each operator and corresponding interface descriptions. For a deep learning framework to connect to a specific chip, it essentially loads the chip's operator library and calls its operators. To enable deep learning frameworks to call different chip operators without altering the upper-layer code, they employ two methods: 1. Providing an adapter layer that encapsulates the chip's operators, with each operator corresponding to an interface in the adapter layer, shielding it from hardware-specific operations. 2. Providing a registration mechanism that assigns the adapter layer's interface function pointers to variables reserved by the framework. The framework then indirectly calls the chip's operators by accessing these variables. Therefore, for a deep learning framework to support different hardware operators, from a development perspective, it requires reimplementing the adapter layer in code for operators with the same function for different hardware, and registering the adapter layer's functions with the framework. This significantly increases the workload for AI algorithm developers, who also need to learn the framework's operator writing standards.

[0031] Currently, the two commonly used chip access technologies can be divided into Built-in and Plugin solutions. Built-in solutions are intrusive hardware access methods. Operator libraries and other components are embedded into the deep learning framework, compiled and packaged together with the framework, and the development of operator code must adhere to the framework's development specifications. Plugin solutions are decoupled plug-in hardware access methods. The deep learning framework defines a standard interface for operators, allowing hardware manufacturers to access their respective adapter layers. During the adaptation process, developers do not need to concern themselves with the underlying framework concepts; they only need to complete the adapter layer writing. However, for the Built-in solution, the framework's internal scheduling logic and memory management need to be modified, and it must be compiled and packaged together with the deep learning framework, making it highly intrusive. Currently popular frameworks all use the Plugin solution to register operators, but the Plugin solution requires compiling the adapter layer and the framework together. The framework must be pre-compiled to support the target hardware and cannot be dynamically extended at runtime; the number of parameters in the adapter layer interface is fixed and does not support variable-length parameters. Different frameworks have different adapter layers, requiring chip manufacturers to develop a separate adapter layer for each framework, resulting in high adaptation costs.

[0032] Based on the above, this application provides an operator call request processing method. A deep learning framework initiates an operator call request, and a hijacking layer is set up to intercept this request. The hijacking layer parses tensor attributes, passes the input tensors TensorA and TensorB to the operator mapper, and parses their metadata (including data type dtype). The operator mapper calls the device simulator and uses the device manager to manage the target device's memory and streams, ensuring data runs on the device. The operator mapper calls the hardware library, matches the target hardware library's interface based on the tensor attributes, and calls the corresponding function in the target hardware. The target hardware performs operator computation. This method hijacks the training framework's calls to the native hardware library, dynamically redirecting operator requests that should be executed by the native hardware to the operator library of an unfamiliar chip. Through dynamic library hijacking technology, it intercepts the training framework's calls to the standard hardware library at the binary level and transparently redirects them to the target hardware operator library. The entire process is non-intrusive, does not modify the framework source code, does not recompile the framework, supports dynamic loading at runtime, does not change the behavior of the original Application Programming Interface (API), and supports parameters of variable length.

[0033] According to an embodiment of this application, an embodiment of an operator call request processing method is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0034] This embodiment provides an operator invocation request processing method. Figure 1 This is a flowchart of an operator invocation request processing method according to an embodiment of this application, such as... Figure 1 As shown, the process includes the following steps:

[0035] Step S101: Obtain operator call request, wherein the operator call request contains context information, and the context information is used to determine the first operator called by the operator call request in the first hardware and the task to be processed corresponding to the operator call request.

[0036] Specifically, this embodiment hijacks the deep learning framework's calls to native hardware libraries (such as libcuda.so), dynamically redirecting tasks that should be executed by the native hardware to operator libraries on other hardware. This embodiment utilizes an operator call request processing system to execute operator call request processing methods. Such an operator call request processing system... Figure 2As shown, the system includes: a deep learning framework, a hijacking layer, an operator mapper, a hardware simulator, and the target hardware. The dynamic library hijacking layer is used to forge a dynamic library with the same name as the native library (e.g., libcuda.so) to intercept the framework's API calls. The operator mapper is used to convert hijacked operator call requests (e.g., cudaLaunchKernel) into equivalent calls for an unfamiliar chip (e.g., npukernelLaunch). The hardware simulator is used to masquerade the hardware, completely transparently deceiving the deep learning framework so that it runs on other hardware without modification. It manages memory, ensures data runs on the device, and manages streams to guarantee asynchronous computation.

[0037] Obtain an operator invocation request, such as invoking the addition operator (add). The operator invocation request includes context information, such as the framework operator name, parameters, memory address, framework data type, and framework parameter names. This context information is used to determine the first operator invoked in the first hardware and the corresponding task to be processed.

[0038] Step S102: If the first operator cannot process the task to be processed, obtain the information mapping rule corresponding to the operator call request. The information mapping rule is used to map the context information to the information corresponding to the second operator in the second hardware.

[0039] Specifically, if the first operator cannot handle the task, for example, if the first operator does not exist or is occupied, the second operator on the second hardware can be used to handle the operator call request. The first hardware and the second hardware are different hardware, for example, the first hardware is a GPU and the second hardware is an NPU.

[0040] Obtain the information mapping rules corresponding to the operator call request, such as: the first mapping rule between framework operator names and hardware operator names, the second mapping rule between framework parameter names and hardware parameter names, and the third mapping rule between framework data types and hardware data types. Using these information mapping rules, context information can be mapped to the information corresponding to the second operator in the second hardware.

[0041] Step S103: Obtain the mapping result according to the information mapping rules and context information, and determine the objective function of the second hardware processing task based on the mapping result.

[0042] Specifically, information mapping rules can be used to map context information to information corresponding to the second operator in the second hardware. For example, the frame operator name in the context information can be mapped to the hardware operator name, the frame parameter name can be mapped to the hardware parameter name, and the frame data type can be mapped to the hardware data type to obtain the mapping result.

[0043] The mapping result includes the hardware operator name, hardware parameter name, and hardware data type. Based on the mapping result, the objective function for the second hardware to process the task to be processed is determined. The objective function may be, for example, the memory allocation function, memory copy function, stream creation function, stream synchronization function, event creation function, etc. of the second hardware.

[0044] Step S104: Call the second operator in the second hardware according to the objective function, and use the second operator to process the task to be processed.

[0045] Specifically, the objective function described above is used to call the second operator in the second hardware, and the second operator is used to process the task to be processed.

[0046] The above content is as follows Figure 3 As shown, the deep learning framework, hijacking layer, operator mapper, loading operator configuration information, calling device simulator, managing the memory and streams of the target device, driver for the target hardware, and operator computation are all included.

[0047] The operator invocation request processing method provided in this embodiment obtains the operator invocation request and the context information contained in the request; when the first operator cannot process the task to be processed, it obtains the information mapping rule corresponding to the request; it maps the context information using the information mapping rule, and determines the objective function of the second hardware based on the mapping result; it calls the second operator in the second hardware according to the objective function, and uses the second operator to process the task. This method is applicable to various deep learning frameworks, enabling them to support different AI chip operators. It eliminates the need for a separate adaptation layer, reducing development effort. Furthermore, using the objective function of the second hardware improves task processing performance, and by adjusting the information mapping rule and objective function, it supports new hardware, reducing adaptation costs. This solves the problem of high adaptation costs associated with developing separate adaptation layers to enable deep learning frameworks to support different AI chip operators.

[0048] As an optional embodiment, before determining the objective function of the second hardware processing task based on the mapping result, the method further includes:

[0049] A loading order rule for hardware libraries is generated using a dynamic linker. This loading order rule is used to prioritize loading the hardware library of the second hardware, which includes the target function.

[0050] The hardware library for the second hardware is loaded using environment variables and loading order rules.

[0051] Specifically, dynamic linkers, such as ld.so, can generate hardware library loading order rules. By prioritizing the loading order rules, the hardware library of the second hardware will be loaded first, so that the program will call the user-defined function instead of the original function of the first hardware, thus achieving SO library hijacking.

[0052] Environment variables, such as LD_PRELOAD, are Linux environment variables used to preferentially load user-specified dynamic libraries (.so files) at runtime, overriding functions in the default library. By utilizing environment variables and load order rules, hardware libraries for secondary hardware can be loaded. For example, in Linux, LD_PRELOAD can be used to preferentially load a fake libcuda.so file according to the load order rules.

[0053] The target function is, for example, a function that hijacks cudaMalloc.

[0054] / / Hijacking cudaMalloc

[0055] cudaError_t cudaMalloc(void devPtr, size_t size) {

[0056] printf("Disguise: Allocate NPU memory\n");

[0057] devPtr = npu_alloc(size); / / Call the NPU for memory allocation

[0058] return cudaSuccess;

[0059] In this embodiment, loading rules are defined by the dynamic linker and combined with environment variables to prioritize loading the second hardware forgery library, thereby achieving SO library hijacking. This allows the program to prioritize calling the second hardware function, replacing the execution logic of the first hardware without modifying the source code.

[0060] As an optional embodiment, after obtaining the operator invocation request, the method further includes:

[0061] Based on the operator call request, obtain the initial context information;

[0062] Using preset commands, obtain the symbol table corresponding to the hardware library of the first hardware.

[0063] If the symbol table contains information corresponding to the initial context information, the initial context information will be used as the context information.

[0064] Specifically, all API calls, i.e., operator call requests, are intercepted, and the context information of the operator call requests (such as operator name, parameters, memory address, etc.) is recorded. The operator call requests are parsed, and the context information contained in the operator call requests is used as the initial context information. Initial context information includes, for example: framework operator name, parameters, memory address, framework data type, framework parameter name, etc.

[0065] Preset commands include, for example, the `nm` command. Using a preset command, you can retrieve the symbol table corresponding to the hardware library of the first hardware component. For example, to retrieve the symbol table from the original `libcuda.so`, the command would be: `nm -gC your_lib.so`.

[0066] This embodiment needs to ensure that the hijacked symbol is exactly the same as the symbol in the symbol table. Therefore, it is determined whether the symbol table contains information corresponding to the initial context information. If it does, it is determined that the hijacked symbol must be exactly the same as the symbol in the symbol table, and the initial context information is used as the context information.

[0067] In this embodiment, the initial context is obtained by intercepting operator call requests, the first hardware library symbol table is extracted with the help of preset commands, and the context information is matched and confirmed to ensure that the hijacked symbol is completely consistent with the original library, accurately identify the API to be hijacked, and provide accurate input for subsequent operator mapping without intruding on the adaptation hardware.

[0068] As an optional embodiment, before obtaining the information mapping rule corresponding to the operator invocation request, the method further includes:

[0069] Load information mapping rules in a preset file. The information mapping rules include one or more of the following: a first mapping rule between framework operator names and hardware operator names, a second mapping rule between framework parameter names and hardware parameter names, and a third mapping rule between framework data types and hardware data types.

[0070] The information mapping rules are registered to a global hash table, which is used to provide information mapping rules when retrieving them.

[0071] Specifically, the preset file is, for example, a YAML file. At runtime, information mapping rules are loaded from the preset file. These rules include one or more of the following: a first mapping rule between framework operator names and hardware operator names; a second mapping rule between framework parameter names and hardware parameter names; and a third mapping rule between framework data types and hardware data types. For example, the following code loads the information mapping rules.

[0072] void load_operator_mapping(const char yaml_path) {

[0073] / / Parse YAML files

[0074] OperatorMapping mapping = parse_yaml(yaml_path);

[0075] Register the information mapping rules to the global hash table. The global hash table is used to provide information mapping rules when retrieving them. For example, the following code registers the information mapping rules to the global hash table: `g_operator_map[hash_string(mapping->framework_func)] = mapping; / / Register to the global hash table`

[0076] A global hash table is used to provide information mapping rules when retrieving information mapping rules. For example, the following code retrieves information mapping rules from a global hash table.

[0077] / / Retrieve the pointer to the corresponding mapping structure from the global operator mapping table based on the hash value of the framework operator name.

[0078] OperatorMapping mapping=g_operator_map[hash_string(framework_func_name)];

[0079] / / Determine if no matching operator mapping rule was found

[0080] if (!mapping)

[0081] / / If not found, return the CUDA configuration invalid error code.

[0082] return cudaErrorInvalidConfiguration;

[0083] Alternatively, you can configure information mapping rules using the following code.

[0084] # Framework of the Addition Operator - Hardware Mapping Configuration File

[0085] mappings: # List of operator mapping rules

[0086] - framework_func: "elementwise_add" # The name of the framework-side addition operator (PyTorch)

[0087] hardware_func: "npuElementwiseAdd" # Target hardware-side addition operator name (NPU)

[0088] param_map: # Framework-hardware parameter name mapping rules

[0089] - src: "input0" # PyTorch parameter name # The name of the first input parameter on the framework side

[0090] dst: "src0"# NPU parameter name# The first input parameter name on the hardware side

[0091] - src: "input1" # The name of the second input parameter on the frame side

[0092] dst: "src1"# The name of the second input parameter on the hardware side

[0093] - src: "output"# Frame-side output parameter name

[0094] dst: "dst"# Hardware-side output parameter name

[0095] dtype_map: # Framework-hardware data type mapping rules

[0096] - float32: "fp32" # The float32 frame corresponds to the NPU's fp32.

[0097] - float16: "fp16" # The framework's float16 corresponds to the NPU's fp16.

[0098] - CUDA_R_32F: "NPU_FLOAT32" # CUDA type corresponds to NPU_FLOAT32

[0099] The information mapping rules can be compiled and injected using the following code.

[0100] export LD_PRELOAD= / path / to / libfake_cuda.so

[0101] python train.py # PyTorch will call the disguised CUDA functions

[0102] In this embodiment, operators, parameters, and data type mapping rules are loaded from a YAML file and registered to a global hash table. When an operator is invoked, a matching rule is quickly retrieved using the hash; if no match is found, an invalid CUDA configuration error code is returned, enabling efficient retrieval and reuse of mapping rules and supporting transparent operator redirection.

[0103] As an optional embodiment, the mapping result is obtained based on the information mapping rules and context information, including:

[0104] Based on the context information, we can obtain the frame operator name, frame parameter name, and frame data type;

[0105] Based on the information mapping rules, the first mapping rule, the second mapping rule, and the third mapping rule are obtained.

[0106] The framework operator name is mapped to the hardware operator name according to the first mapping rule, the framework parameter name is mapped to the hardware parameter name according to the second mapping rule, and the framework data type is mapped to the hardware data type according to the third mapping rule, thus obtaining the mapping result, which includes the hardware operator name, hardware parameter name, and hardware data type.

[0107] Specifically, the frame operator name, frame parameter name, and frame data type contained in the context information are obtained from the context information.

[0108] Based on the information mapping rules, the first mapping rule, the second mapping rule, and the third mapping rule are obtained. For example, when the system starts, the mapping rules are loaded from the YAML file and converted into OperatorMapping in the C language. The mapping rules are shown in the following code.

[0109] typedef struct {

[0110] / / First mapping rule (frame operator name → hardware operator name)

[0111] char framework_func; / / Frame operator name (e.g., "elementwise_add")

[0112] char hardware_func; / / Hardware operator name (e.g., "npuElementwiseAdd")

[0113] / / Parameter mapping rules (framework parameter name → hardware parameter name)

[0114] ParamMap param_map; / / Array to store multiple parameter pairs

[0115] int param_count;

[0116] / / Data type mapping (framework data type → hardware data type)

[0117] DTypeMap dtype_map; / / such as {"float32": "fp32", "float16": "fp16"}

[0118] int dtype_count;

[0119] OperatorMapping;

[0120] The first mapping rule maps frame operator names to hardware operator names, the second mapping rule maps frame parameter names to hardware parameter names, and the third mapping rule maps frame data types to hardware data types, thus obtaining the mapping results. For example, the following code performs information conversion.

[0121] / / Call the type conversion function to convert the framework data type to the NPU data type, and assign the result to npu_dtype

[0122] const char npu_dtype = convert_dtype(mapping, framework_args[0].dtype);

[0123] / / Data type conversion function definition

[0124] / / Receives mapping rules and framework data types, returns hardware data types.

[0125] const char convert_dtype(OperatorMapping mapping, const char framework_dtype) {

[0126] / / Traverse all data type mapping items in the mapping rules

[0127] for (int i = 0; i <mapping->dtype_count; i++) {

[0128] / / Match frame data types with frame types in the mapping items

[0129] if (strcmp(mapping->dtype_map[i].framework_type, framework_dtype) ==0) {

[0130] / / Return the corresponding hardware data type if a match is successful

[0131] return mapping->dtype_map[i].hardware_type;

[0132] }

[0133] In this embodiment, the framework operator name, parameter name, and data type are parsed from the context information, matched with three types of pre-loaded mapping rules, and converted into corresponding names / types on the hardware side to generate mapping results, providing a precise adaptation basis for redirecting operators to target hardware.

[0134] As an optional embodiment, obtaining the operator invocation request includes:

[0135] Hijacking the initial operator call request of a deep learning framework;

[0136] Obtain the type of request to be hijacked, which includes one or more of the following: device attribute request, memory management request, stream management request, and event management request;

[0137] If the request type of the initial operator call request is included in the request type to be hijacked, the initial operator call request will be treated as the operator call request.

[0138] Specifically, this embodiment masquerades the second hardware to completely and transparently deceive the deep learning framework, enabling it to run on the second hardware without modification, and to implement device management and memory management.

[0139] Hijacking the initial operator call request of a deep learning framework. For example, PyTorch's addition operation ultimately calls the elementwiseKernel of the underlying library (such as CUDA). It's necessary to hijack the CUDA kernel function startup API, where the initial operator call request is cudaLaunchKernel. The following code hijacks cudaLaunchKernel.

[0140] / / Hijacking cudaLaunchKernel

[0141] cudaError_tcudaLaunchKernel(

[0142] const char framework_func_name, / / such as "elementwise_add"

[0143] void framework_args, / / An array of arguments passed to the framework

[0144] int num_framework_args / / Number of arguments )

[0146] Without modifying the deep learning framework code, this embodiment hijacks operator call requests such as the CUDA API of the deep learning framework, misleading it into believing it is operating on the first hardware, when in fact it is calling other hardware for computation. This embodiment categorizes the operator call requests to be hijacked into the following types: device attribute requests, memory management requests, stream management requests, and event management requests. Therefore, the request types to be hijacked include one or more of the following: device attribute requests, memory management requests, stream management requests, and event management requests.

[0147] Device property requests, such as cudaGetDeviceProperties, cudaGetDeviceCount, and cudaSetDevice, are used to trick the framework into recognizing the first hardware.

[0148] Memory management requests, such as cudaMalloc, cudaFree, and cudaMemcpy, are used to masquerade device memory and hide the actual hardware's memory management logic.

[0149] Stream management requests, such as cudaStreamCreate and cudaStreamDestroy, are used to simulate asynchronous computation behavior.

[0150] Event management requests, such as cudaEventCreate and cudaEventRecord, are used to disguise computational events for performance statistics.

[0151] If the request type of the initial operator call request is included in the request type to be hijacked, the initial operator call request will be treated as the operator call request. For example, if the request type of the initial operator call request is a memory management request, which is included in the request type to be hijacked, the initial operator call request will be treated as the operator call request.

[0152] In this embodiment, the initial operator call request of the deep learning framework is hijacked, and device attribute, memory / stream / event management requests are selected as target requests. Without modifying the framework source code, the first hardware feature is disguised, so that the framework can call the second hardware to complete the calculation without being aware of it, thus achieving transparent hardware replacement.

[0153] As an optional embodiment, the target function includes a masquerade function and an operator calling function, and calls a second operator in the second hardware according to the target function, including:

[0154] Adjust the preset hardware parameters of the second hardware according to the spoofing function;

[0155] The operator call function calls the second operator in the second hardware.

[0156] Specifically, the preset hardware parameters of the second hardware are adjusted according to the masquerading function. These preset hardware parameters include, for example, device attributes, host memory management data, stream handles, and performance parameters. The preset hardware parameters of the second hardware are then disguised as those of the first hardware, allowing the framework to seamlessly invoke the second hardware to complete the computation, thus achieving transparent hardware replacement.

[0157] The second operator in the second hardware is invoked according to the operator call function, and the task to be processed is handled by the second operator.

[0158] As an optional embodiment, before adjusting the preset hardware parameters of the second hardware according to the masquerade function, the method further includes:

[0159] Get the request type of the operator invocation request;

[0160] Based on the request type, obtain the spoofing function from the hardware library of the second hardware. The hardware library of the second hardware includes one or more of the following: device attribute spoofing function, memory management spoofing function, stream management spoofing function, and performance statistics spoofing function.

[0161] Specifically, obtain the request type of the operator call request, such as: device attribute request, memory management request, stream management request, event management request.

[0162] The hardware library for the second hardware includes one or more of the following: device attribute spoofing functions, memory management spoofing functions, stream management spoofing functions, and performance statistics spoofing functions.

[0163] Depending on the request type, retrieve the disguised function from the hardware library of the second hardware. For example: if the request type is a device attribute request, retrieve the device attribute disguised function from the hardware library of the second hardware; if the request type is a memory management request, retrieve the memory management disguised function from the hardware library of the second hardware; if the request type is a stream management request, retrieve the stream management disguised function from the hardware library of the second hardware; if the request type is an event management request, since event management requests are used for performance statistics, retrieve the performance statistics disguised function from the hardware library of the second hardware.

[0164] As an optional embodiment, adjusting the preset hardware parameters of the second hardware according to the masquerade function includes:

[0165] If the masquerading function includes a device attribute masquerading function, initialize the device attribute structure of the second hardware;

[0166] Write the hardware identifier of the first hardware into the name field of the device attribute structure;

[0167] Call the device attribute acquisition function of the second hardware to determine the hardware attributes of the second hardware;

[0168] Alternatively, call the device quantity acquisition function of the second hardware, and set the device quantity of the second hardware according to the device quantity acquisition function;

[0169] Based on the number of devices, obtain the function's execution result and generate a second return message;

[0170] If the masquerading function includes a memory management masquerading function, the memory allocation function or memory copy function of the second hardware is called, and the execution result of the memory allocation function or memory copy function is determined, wherein the execution result includes preset hardware parameters.

[0171] Specifically, device property spoofing functions include functions for hijacking cudaGetDeviceProperties and functions for hijacking cudaGetDeviceCount. The function that hijacks cudaGetDeviceProperties can be defined using the following code, returning a CUDA error code and receiving a pointer to a device property structure and a device index.

[0172] cudaError_t cudaGetDeviceProperties(cudaDeviceProp prop, int device){

[0173] / / Set the device properties structure memory to 0, and initialize the structure.

[0174] memset(prop, 0, sizeof(cudaDeviceProp));

[0175] / / Copy the fake GPU model to the device properties structure name field

[0176] strcpy(prop->name, "NVIDIA A100-PCIE-40GB");

[0177] / / Call the NPU's device property retrieval function and populate the actual hardware properties.

[0178] npuGetDeviceProperties(prop,device);

[0179] }

[0180] The following code defines a function to hijack cudaGetDeviceCount, which always returns 1 "GPU".

[0181] / / Define the hijacked cudaGetDeviceCount function, which returns a CUDA error code and receives a pointer to the number of devices.

[0182] cudaError_t cudaGetDeviceCount(int count) {

[0183] / / Call the NPU's device count get function to set the device count.

[0184] npuGetDeviceCount(count);

[0185] / / Returns CUDA success code, fake query successful

[0186] return cudaSuccess;

[0187] }

[0188] If the masquerading function includes the aforementioned device attribute masquerading function, initialize the device attribute structure of the second hardware. For example, set the memory of the device attribute structure to 0 to initialize the device attribute structure of the second hardware. Write the hardware identifier of the first hardware into the name field of the device attribute structure, for example: strcpy(prop->name, "NVIDIA A100-PCIE-40GB"). Call the device attribute acquisition function of the second hardware, such as the device attribute acquisition function of the NPU, to determine the hardware attributes of the second hardware. Alternatively, call the device quantity acquisition function of the second hardware, such as the device quantity acquisition function of the NPU, and set the device quantity of the second hardware according to the device quantity acquisition function; generate the second return information based on the execution result of the device quantity acquisition function, such as the CUDA success code.

[0189] Memory management disguised functions include functions used to hijack cudaMalloc and functions used to hijack cudaMemcpy.

[0190] The following code defines a function to hijack cudaMalloc, which allocates host memory but marks it as device memory.

[0191] / / Define the hijacked cudaMalloc function, which returns a CUDA error code and receives the memory pointer and allocation size.

[0192] cudaError_t cudaMalloc(void devPtr, size_t size) {

[0193] / / Call the NPU memory allocation function and return its execution result.

[0194] return npuMalloc(devPtr,size);

[0195] }

[0196] The following code defines a function for hijacking cudaMemcpy. This function returns a CUDA error code and receives the target address, source address, copy size, and copy type.

[0197] cudaError_t cudaMemcpy(void dst, const void src, size_t count, enumcudaMemcpyKind kind) {

[0198] / / Call the NPU memory copy function and return its execution result.

[0199] return npuMemcpy(dst,src,count,kind);

[0200] }

[0201] In cases where the dummy function contains a memory management dummy function, the memory allocation function or memory copy function of the second hardware is called, and the execution result of the memory allocation function or memory copy function is determined, for example: npuMalloc(devPtr,size) or npuMemcpy(dst,src,count,kind).

[0202] As an optional embodiment, adjusting the preset hardware parameters of the second hardware according to the masquerade function includes:

[0203] If the masquerading function includes a stream management masquerading function, call the stream creation function or stream synchronization function of the second hardware, and determine the execution result of the stream creation function or stream synchronization function;

[0204] If the spoofing function includes a performance statistics spoofing function, the event creation function of the second hardware is called to determine the execution result of the event creation function.

[0205] Specifically, stream management spoofing functions include, for example, functions used to hijack cudaStreamCreate and functions used to hijack cudaStreamSynchronize.

[0206] The following code defines a function to hijack cudaStreamCreate, which returns a CUDA error code and receives a pointer to the stream handle.

[0207] cudaError_t cudaStreamCreate(cudaStream_t stream) {

[0208] / / Call the NPU's stream creation function and return the execution result.

[0209] return npuStreamCreate(stream);

[0210] }

[0211] The following code defines a function to hijack cudaStreamSynchronize, which returns a CUDA error code and receives a stream handle.

[0212] cudaError_t cudaStreamSynchronize(cudaStream_t stream) {

[0213] / / Call the NPU's stream synchronization function (assuming NPU computation is synchronous), and return the execution result.

[0214] return npuStreamSynchronize(stream); / / Assuming NPU computation is synchronous.

[0215] }

[0216] If the masquerading function contains a stream management masquerading function, call the stream creation function or stream synchronization function of the second hardware, and determine the execution result of the stream creation function or stream synchronization function, for example: npuStreamCreate(stream) or npuStreamSynchronize(stream).

[0217] Performance statistics spoofing functions, for example: functions used to hijack cudaEventCreate.

[0218] The following code defines a function to hijack cudaEventCreate, which returns a fictitious timeout, a CUDA error code, and receives an event handle pointer.

[0219] cudaError_t cudaEventCreate(cudaEvent_t event) {

[0220] / / Call the NPU's event creation function and return its execution result.

[0221] return npuEventCreate(event);

[0222] }

[0223] If the spoofing function includes a performance statistics spoofing function, the event creation function of the second hardware is called to determine the execution result of the event creation function, for example: npuEventCreate(event).

[0224] As an optional embodiment, invoking a second operator in the second hardware according to an operator invocation function includes:

[0225] Retrieve hardware parameters and hardware operator names from the mapping results;

[0226] If the hardware operator name is a preset operator name, call the second operator corresponding to the hardware operator name and input the hardware parameters into the second operator;

[0227] If the hardware operator name is not the default operator name, then the second operator call is determined to have failed.

[0228] Obtain the hardware execution status of the second operator and generate status information based on the hardware execution status of the second operator.

[0229] Specifically, the hardware parameters and hardware operator names are obtained from the mapping results. For example, the hardware operator name is npuElementwiseAdd. The hardware parameters are npu_args[0], npu_args[1], npu_args[2], ...

[0230] The default operator name is, for example, npuElementwiseAdd. If the hardware operator name is the default operator name, the second operator corresponding to the hardware operator name is called, and the hardware parameters are input into the second operator, for example: status = npu_add_impl(npu_args[0], npu_args[1], npu_args[2], ...), where status is a defined variable that stores the execution status of the hardware operator.

[0231] If the hardware operator name is not the default operator name, then the second operator call is considered to have failed, for example: status=npu_FAILURE.

[0232] Get the hardware execution status of the second operator and generate status information based on the hardware execution status of the second operator. For example, the status information is: return (status == nuisances) cudaSuccess : cudaErrorUnknown.

[0233] The above steps can be achieved using the following code.

[0234] / / Define variables to store the execution state of hardware operators

[0235] npuStatus_t status;

[0236] / / Check if the hardware operator name in the mapping rule is npuElementwiseAdd

[0237] if (strcmp(mapping->hardware_func, "npuElementwiseAdd") == 0)

[0238] / / If a match is found, the addition operator implementation function is called, passing in the parameters from the hardware parameter array.

[0239] status = npu_add_impl(npu_args[0], npu_args[1], npu_args[2], ...);

[0240] else

[0241] / / If there is no match, set the operator call failure status.

[0242] status=npu_FAILURE;

[0243] / / Return a CUDA-compatible error code based on the hardware execution state (Note: nuisances should be a typo, it should be NPU_SUCCESS)

[0244] return (status == nuisances) cudaSuccess : cudaErrorUnknown;

[0245] }

[0246] / / The outer function returns NULL (a fallback return when there is no matching operator).

[0247] return NULL;

[0248] }

[0249] In this embodiment, hardware parameters and operator names are extracted from the mapping results. If a preset operator name is matched, the corresponding second operator is called and the parameters are passed in. If no match is found, the call is marked as failed. The execution status of the second operator is obtained and converted into compatibility status information and returned, so as to achieve accurate operator calling and unified status feedback.

[0250] As an optional embodiment, the method further includes, before invoking the second operator in the second hardware according to the objective function:

[0251] Obtain the input tensor from the mapping result;

[0252] Write the input tensor to the preset device memory address;

[0253] Obtain the device memory address of the input tensor and create a streaming task;

[0254] Invoke the second operator, and determine the execution status of the second operator based on the second operator, the stream task, the preset device memory address, and the mapping result;

[0255] Generate a preset error code based on the execution status.

[0256] Specifically, the input tensors, such as Tensor A and Tensor B, are obtained from the mapping result. Additionally, the mapping result also includes the output tensor, such as Tensor A and Tensor B. C.

[0257] Write the input tensor to a preset device memory address. For example, the preset device memory address of Tensor A is device_addrA, the preset device memory address of Tensor B is device_addrB, and the preset device memory address of Tensor C is device_addrC.

[0258] The writing process adjusts the data based on whether the input tensor (e.g., Tensor A) is in the device (NPU) memory. If the input tensor is not in the NPU memory, it is first copied from the host memory to the device memory, and then the tensor data is copied to the corresponding preset device memory address, for example: device_addrA, A->data, A->size, device_addrB, B->data, B->size, device_addrC, C->data, C->size.

[0259] If the input tensor is in NPU memory, directly assign the device memory address of the input tensor to the corresponding preset device memory address, for example: device_addrA=A->data, device_addrB=B->data, device_addrC=C->data.

[0260] Obtain the device memory address of the input tensor and create a streaming task. For example, use `cudaStreamCreate(stream)` to create a CUDA stream, which is then hijacked as an NPU stream. In high-performance computing, a stream is a core concept that allows multiple operations to be executed concurrently on a CUDA-enabled GPU. Different computational tasks can be placed into different "streams," and the GPU will process them as simultaneously as possible to greatly improve efficiency.

[0261] The second operator is invoked. Based on the second operator, the stream task, the preset device memory address, and the mapping result, the execution status of the second operator is determined. For example, if the second operator is the NPU native addition operator, the second operator is invoked, receiving the stream, input / output address, number of elements, and data type, and the execution status is returned, as shown in the following code.

[0262] npuStatus_t status = npuElementwiseAdd(

[0263] stream, / / Pass in the handle of the stream you created

[0264] device_addrA, / / Input device memory address of tensor A

[0265] device_addrB, / / Input device memory address of tensor B

[0266] device_addrC, / / Outputs the device memory address of tensor C

[0267] C->numel(), / / Total number of elements

[0268] C->dtype / / Data type conversion

[0269] Generate a preset error code based on the execution status. For example, if the preset error code is a CUDA-compatible error code, convert it to a CUDA error code based on the NPU execution status and return it as a return (status == NPU_SUCCESS) cudasuccess :cudaerrorunknown.

[0270] In this embodiment, the input tensor is obtained and written to a preset device memory address (adaptive tensor memory location), and a stream task is created; the second operator is called, and the execution status is determined by combining the stream task, memory address, etc., and the status is converted into a preset compatible error code and returned to ensure that the operator executes the adaptation framework logic.

[0271] As an optional embodiment, the specific process of step S104 "processing the task to be processed using the second operator" may also include steps A1 to A5.

[0272] In this embodiment, the second hardware is an NPU. The operator call request is a request to call the addition operator (add). After the operator call request is intercepted, the pending tasks of the operator call request are processed based on the NPU.

[0273] Step A1: Verify the validity of the NPU operator call parameters.

[0274] Specifically, the operator mapper, based on the NPU hardware specification, verifies the core information in the `npu_args` parameter array: it checks whether `device_addrA`, `device_addrB`, and `device_addrC` are valid NPU device memory addresses; it verifies whether the total number of output tensor elements `C->numel()` matches the shape of the input tensor (e.g., if TensorA / TensorB is (32, 64), then `C->numel()` must equal 2048 to avoid shape mismatch errors); and it verifies whether the data type `C->dtype` is an NPU-supported type, such as `fp32` / `fp16`, rejecting non-standard types incompatible with the NPU. If parameter verification fails, the NPU execution status is directly set to `NPU_INVALID_ARGUMENT`; if verification passes, the operator call phase begins.

[0275] Step A2: Invoke the NPU's native addition operator and trigger hardware computation.

[0276] Specifically, the operator mapper calls the native addition operator interface npuElementwiseAdd provided by the NPU driver, passing in standardized parameters such as: NPU Stream handle (for asynchronous scheduling), the verified NPU memory address of TensorA, the verified NPU memory address of TensorB, the NPU memory address of the output Tensor (reserved space), the total number of verified output tensor elements, and the data type identifier of the NPU compatible format.

[0277] The NPU driver adds the addition calculation task to the task queue of the specified Stream; the NPU hardware scheduler allocates computing cores, reads the data of device_addrA and device_addrB element by element, performs the operation A[i]+B[i], and writes the result to device_addrC in real time.

[0278] Step A3: Wait for the NPU operator computation task to complete.

[0279] Specifically, the waiting logic can be handled in Stream mode, which can be divided into synchronous mode and asynchronous mode.

[0280] Synchronous mode, for example: calling a fake `cudaStreamSynchronize` (actually `npuStreamSynchronize`), blocks the thread until the NPU completes the addition calculation within the stream. Asynchronous mode, for example: the NPU driver marks the task as "in progress", the current thread returns directly, and the device simulator fakes the result when the framework queries the status later.

[0281] After the calculation is complete, the NPU driver updates the status: NPU_SUCCESS if successful, and the corresponding error code if it fails.

[0282] Step A4: Change the NPU execution status to a CUDA-compatible error code.

[0283] Specifically, set conversion rules, such as: NPU_SUCCESS→cudaSuccess (successful calculation), NPU_INVALID_ARGUMENT→cudaErrorInvalidValue (invalid parameter), NPU_OUT_OF_MEMORY→cudaErrorMemoryAllocation (insufficient memory), other errors→cudaErrorUnknown (unknown error), etc.

[0284] The converted CUDA error codes are temporarily stored in the hijacking layer context variables.

[0285] Step A5: Return the results layer by layer to the deep learning framework.

[0286] Specifically, the error code is passed from the operator mapper to the dynamic library hijacking layer, which confirms that the format is CUDA standard; the hijacking layer returns the error code to the framework's cudaLaunchKernel call, which the framework mistakenly interprets as a return value from the native CUDA library.

[0287] In this embodiment, the validity of parameters is verified to avoid anomalies such as illegal addresses and mismatched context information; the native computing power of the NPU is triggered to perform addition calculations to ensure computational efficiency; synchronous and asynchronous modes are adapted to ensure the complete completion of the computation task; error codes are compatible with the CUDA standard and can be correctly identified and processed by the framework; the framework source code is not modified throughout the process, realizing transparent redirection of the addition operator to the NPU.

[0288] This embodiment also provides an operator call request processing apparatus for implementing the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that implements a predetermined function. Although the apparatus described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0289] This embodiment provides an operator invocation request processing device, such as... Figure 4 As shown, it includes:

[0290] The request acquisition module 401 is used to acquire operator call requests, wherein the operator call request contains context information, and the context information is used to determine the first operator called in the first hardware and the task to be processed corresponding to the operator call request;

[0291] The rule acquisition module 402 is used to acquire the information mapping rule corresponding to the operator call request when the first operator cannot process the task to be processed. The information mapping rule is used to map the context information to the information corresponding to the second operator in the second hardware.

[0292] The function determination module 403 is used to obtain the mapping result based on the information mapping rules and context information, and to determine the target function of the second hardware processing task based on the mapping result;

[0293] Operator calling module 404 is used to call the second operator in the second hardware according to the target function, and use the second operator to process the task to be processed.

[0294] Further functional descriptions of the above modules and units are the same as those in the corresponding embodiments described above, and will not be repeated here.

[0295] In this embodiment, the operator call request processing device is presented in the form of a functional unit. Here, a unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that execute one or more software or fixed programs, and / or other devices that can provide the above functions.

[0296] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention.

[0297] The following is a detailed reference. Figure 5 The diagram illustrates a structural schematic suitable for implementing an electronic device according to embodiments of the present invention. The electronic device may include a processor (e.g., a central processing unit, graphics processor, etc.) 501, which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 502 or a program loaded from memory 508 into random access memory (RAM) 503. The RAM 503 also stores various programs and data required for the operation of the electronic device. The processor 501, ROM 502, and RAM 503 are interconnected via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0298] Typically, the following devices can be connected to I / O interface 505: input devices 506 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 507 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; memory devices 508 including, for example, magnetic tapes, hard disks, etc.; and communication devices 509. Communication device 509 allows electronic devices to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 5 Electronic devices with various devices are shown, but it should be understood that it is not required to implement or have all of the devices shown, and more or fewer devices may be implemented or have instead.

[0299] In particular, according to embodiments of the present invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of the present invention include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 509, or installed from a memory 508, or installed from a ROM 502. When the computer program is executed by the processor 501, it performs the functions defined in the operator invocation request processing method of the embodiments of the present invention.

[0300] Figure 5 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments of the present invention.

[0301] This invention also provides a computer-readable storage medium. The methods described above according to embodiments of the invention can be implemented in hardware or firmware, or implemented as computer code that can be recorded on a storage medium, or implemented as computer code downloaded via a network and originally stored on a remote storage medium or a non-transitory machine-readable storage medium and then stored on a local storage medium. Thus, the methods described herein can be processed by software stored on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. The storage medium can be a magnetic disk, optical disk, read-only memory, random access memory, flash memory, hard disk, or solid-state drive, etc.; further, the storage medium can also include combinations of the above types of memory. It is understood that computers, processors, microprocessor controllers, or programmable hardware include storage components capable of storing or receiving software or computer code. When the software or computer code is accessed and executed by the computer, processor, or hardware, the operator call request processing method shown in the above embodiments is implemented.

[0302] A portion of this invention can be applied as a computer program product, such as computer program instructions, which, when executed by a computer, can invoke or provide the methods and / or technical solutions according to the invention through the operation of the computer. Those skilled in the art will understand that the forms in which computer program instructions exist in a computer-readable medium include, but are not limited to, source files, executable files, installation package files, etc. Correspondingly, the ways in which computer program instructions are executed by a computer include, but are not limited to: the computer directly executing the instructions, or the computer compiling the instructions and then executing the corresponding compiled program, or the computer reading and executing the instructions, or the computer reading and installing the instructions and then executing the corresponding installed program. Here, the computer-readable medium can be any available computer-readable storage medium or communication medium accessible to a computer.

[0303] Although embodiments of the invention have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the invention, and all such modifications and variations fall within the scope defined by the appended claims.

Claims

1. A method for processing operator invocation requests, characterized in that, The method includes: Obtain an operator invocation request, wherein the operator invocation request includes context information, the context information being used to determine the first operator invoked by the operator invocation request in the first hardware and the task to be processed corresponding to the operator invocation request; If the first operator cannot process the task to be processed, the information mapping rule corresponding to the operator call request is obtained, wherein the information mapping rule is used to map the context information to the information corresponding to the second operator in the second hardware; Based on the information mapping rules and the context information, a mapping result is obtained, and the objective function for the second hardware to process the task to be processed is determined based on the mapping result. The second operator in the second hardware is invoked according to the target function, and the task to be processed is processed using the second operator.

2. The method according to claim 1, characterized in that, Before determining the objective function for the second hardware to process the task based on the mapping result, the method further includes: A loading order rule for a hardware library is generated using a dynamic linker, wherein the loading order rule is used to preferentially load the hardware library of the second hardware, the hardware library of the second hardware including the target function; The hardware library of the second hardware is loaded using environment variables and the loading order rules.

3. The method according to claim 1, characterized in that, After obtaining the operator invocation request, the method further includes: Based on the operator call request, the initial context information is obtained; Using a preset command, obtain the symbol table corresponding to the hardware library of the first hardware; If the symbol table contains information corresponding to the initial context information, the initial context information shall be used as the context information.

4. The method according to claim 1, characterized in that, Before obtaining the information mapping rule corresponding to the operator invocation request, the method further includes: The information mapping rules are loaded in a preset file, wherein the information mapping rules include one or more of the following: a first mapping rule between framework operator names and hardware operator names, a second mapping rule between framework parameter names and hardware parameter names, and a third mapping rule between framework data types and hardware data types. The information mapping rule is registered to a global hash table, wherein the global hash table is used to provide the information mapping rule when the information mapping rule is retrieved.

5. The method according to claim 4, characterized in that, The step of obtaining the mapping result based on the information mapping rule and the context information includes: Based on the context information, the frame operator name, the frame parameter name, and the frame data type are obtained; Based on the information mapping rules, the first mapping rule, the second mapping rule, and the third mapping rule are obtained; The framework operator name is mapped to the hardware operator name according to the first mapping rule, the framework parameter name is mapped to the hardware parameter name according to the second mapping rule, and the framework data type is mapped to the hardware data type according to the third mapping rule, thereby obtaining the mapping result, wherein the mapping result includes the hardware operator name, the hardware parameter name, and the hardware data type.

6. The method according to claim 1, characterized in that, The process of obtaining the operator invocation request includes: Hijacking the initial operator call request of a deep learning framework; Obtain the type of request to be hijacked, wherein the type of request to be hijacked includes one or more of the following: device attribute request, memory management request, stream management request, and event management request; If the request type of the initial operator call request is included in the hijacking request type, the initial operator call request shall be used as the operator call request.

7. The method according to claim 6, characterized in that, The target function includes a dummy function and an operator calling function. The step of calling the second operator in the second hardware according to the target function includes: The preset hardware parameters of the second hardware are adjusted according to the spoofing function; The operator call function calls the second operator in the second hardware.

8. The method according to claim 7, characterized in that, Before adjusting the preset hardware parameters of the second hardware according to the spoofing function, the method further includes: Obtain the request type of the operator invocation request; According to the request type, the spoofing function is obtained from the hardware library of the second hardware, wherein the hardware library of the second hardware includes one or more of the following: device attribute spoofing function, memory management spoofing function, stream management spoofing function, and performance statistics spoofing function.

9. The method according to claim 8, characterized in that, The step of adjusting the preset hardware parameters of the second hardware according to the spoofing function includes: If the spoofing function includes the device attribute spoofing function, initialize the device attribute structure of the second hardware; Write the hardware identifier of the first hardware into the name field of the device attribute structure; Call the device attribute acquisition function of the second hardware to determine the hardware attributes of the second hardware; Alternatively, the device quantity acquisition function of the second hardware can be called, and the device quantity of the second hardware can be set according to the device quantity acquisition function; Based on the number of devices, obtain the execution result of the function and generate the second return information; If the spoofing function includes the memory management spoofing function, the memory allocation function or memory copy function of the second hardware is called, and the execution result of the memory allocation function or the memory copy function is determined, wherein the execution result includes the preset hardware parameters.

10. The method according to claim 9, characterized in that, The step of adjusting the preset hardware parameters of the second hardware according to the spoofing function includes: If the spoofing function includes the stream management spoofing function, the stream creation function or stream synchronization function of the second hardware is called, and the execution result of the stream creation function or the stream synchronization function is determined; If the spoofing function includes the performance statistics spoofing function, the event creation function of the second hardware is called to determine the execution result of the event creation function.

11. The method according to claim 7, characterized in that, The step of calling the second operator in the second hardware according to the operator calling function includes: Obtain the hardware parameters and hardware operator names from the mapping results; If the hardware operator name is a preset operator name, call the second operator corresponding to the hardware operator name and input the hardware parameters into the second operator; If the hardware operator name is not the preset operator name, then the second operator call is determined to have failed; Obtain the hardware execution status of the second operator, and generate status information based on the hardware execution status of the second operator.

12. The method according to claim 1, characterized in that, Before invoking the second operator in the second hardware according to the objective function, the method further includes: The input tensor is obtained from the mapping result; Write the input tensor to a preset device memory address; Obtain the device memory address of the input tensor and create a streaming task; The second operator is invoked, and the execution status of the second operator is determined based on the second operator, the stream task, the preset device memory address, and the mapping result. Based on the execution status, a preset error code is generated.

13. An electronic device, characterized in that, include: A memory and a processor are communicatively connected, the memory storing computer instructions, and the processor executing the computer instructions to perform the operator invocation request processing method according to any one of claims 1 to 12.

Images