Model dynamic loading and processing method and device

Abstract

The invention discloses a model dynamic loading and processing method and device, and the method comprises the steps: receiving a first call request which corresponds to a first programming language; converting the first calling request into a second calling request; the second calling request corresponds to a second programming language; selecting a target object from a plurality of preset objects based on the second calling request; the plurality of preset objects are created based on a plurality of models; the plurality of preset objects are used for supporting the plurality of models to perform reasoning; obtaining a target model file based on the second calling request; the target model file comprises a model network structure and weight parameters; and reasoning based on the target model file and the target object.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, and particularly relates to a model dynamic loading and processing method and device. BACKGROUND

[0002] In the field of artificial intelligence application, mainstream model inference frameworks are mostly developed based on C / C++ language, and have limited native support for Java language. Therefore, in actual engineering implementation, JNI (Java Native Interface) technology is usually used as a cross-language interaction bridge to realize the calling of C / C++ native inference library by Java layer through JNI encapsulation. However, the current way of calling native inference library through JNI has the problem of poor flexibility. SUMMARY

[0003] The technical scheme provided by the present application is as follows:

[0004] The first aspect of the present application provides a model dynamic loading and processing method, comprising:

[0005] receiving a first calling request, the first calling request corresponding to a first programming language;

[0006] converting the first calling request into a second calling request; the second calling request corresponding to a second programming language;

[0007] selecting a target object from a plurality of preset objects based on the second calling request; the plurality of preset objects being created based on a plurality of models; the plurality of preset objects being used to support the plurality of models to perform inference;

[0008] obtaining a target model file based on the second calling request; the target model file including a model network structure and a weight parameter;

[0009] performing inference based on the target model file and the target object.

[0010] In a possible implementation, the plurality of preset objects comprises:

[0011] at least one first preset object and a plurality of second preset objects; the first preset object being reusable by the plurality of models; the second preset object being exclusively owned by one of the plurality of models.

[0012] In a possible implementation, the second preset object comprises: an independent inference logic saved through a dynamic link library file; the independent inference logic being exclusively owned by a single model in the plurality of models for inference.

[0013] In a possible implementation, the first calling request includes a plurality of first calling sub-requests, which are respectively generated by calling a first interface and a second interface; the first interface and the second interface are both in a non-coupling relationship with the plurality of models.

[0014] The first interface is configured to trigger selection of an object from a plurality of preset objects and inference initialization; and the second interface is configured to trigger loading of a model file.

[0015] In a possible implementation, the inference based on the target model file and the target object includes:

[0016] selecting a target hardware unit from a plurality of hardware units;

[0017] selecting a target inference dependency library matched with the target hardware unit from preset inference dependency libraries, the target inference dependency library being configured to drive the target hardware unit to perform inference;

[0018] performing inference on the target hardware unit based on the target model file, the target object and the target inference dependency library.

[0019] In a possible implementation, the selecting of the target hardware unit from the plurality of hardware units includes any of the following:

[0020] obtaining a hardware unit selection instruction from the calling request, and selecting a target hardware unit from the plurality of hardware units based on the hardware unit selection instruction;

[0021] obtaining a hardware resource parameter, and selecting a target hardware unit from the plurality of hardware units based on the hardware resource parameter; the hardware resource parameter representing a running state of the plurality of hardware units.

[0022] In a possible implementation, the selecting of the target object from the plurality of preset objects based on the second calling request includes:

[0023] obtaining first data from the second calling request; the first data being configured to identify a target model;

[0024] determining second data based on the second calling request; the second data being configured to represent an inference attribute of the target model;

[0025] matching a target object storage path in a configuration file based on at least one of the first data and the second data; the configuration file containing a mapping relationship between configuration information and preset object storage paths; the configuration information including at least one of first configuration information and second configuration information; the first configuration information being configured to identify a model; and the second configuration information being configured to represent an inference attribute of the model;

[0026] load a target object based on the target object storage path;

[0027] The target model file is obtained based on the second invocation request, comprising:

[0028] The first data is obtained from the second invocation request;

[0029] The second data is determined based on the second invocation request;

[0030] The target model file storage path is matched in a configuration file based on at least one of the first data and the second data; the configuration file contains a mapping relationship between the configuration information and the model file storage path;

[0031] The target model file is loaded based on the target model file storage path.

[0032] In a possible implementation, the target object is selected from a plurality of preset objects based on the second invocation request, comprising:

[0033] The first to-be-used model is determined based on the second invocation request;

[0034] The second to-be-used model is determined based on the first to-be-used model; the first to-be-used model and the second to-be-used model have a dependency relationship;

[0035] The first target object and the second target object corresponding to the first to-be-used model and the second to-be-used model respectively are selected from a plurality of preset objects;

[0036] The target model file is obtained based on the second invocation request, comprising:

[0037] The first to-be-used model is determined based on the second invocation request;

[0038] The second to-be-used model is determined based on the first to-be-used model; the first to-be-used model and the second to-be-used model have a dependency relationship;

[0039] The first target model file and the second target model file corresponding to the first to-be-used model and the second to-be-used model respectively are obtained.

[0040] In a possible implementation, the first to-be-used model and the second to-be-used model correspond to the same process; the inference based on the target model file and the target object comprises:

[0041] In the same process, a shared resource is applied for the first to-be-used model; the shared resource includes at least one of a target buffer and a target resource; the target resource is used for inference initialization and inference execution scheduling;

[0042] In the same process, after inference is performed based on the first target object, the first target model file and the shared resource, the shared resource is allocated to the second to-be-used model;

[0043] Inference is performed based on the second target object, the second target model file and the shared resource.

[0044] In another aspect of the present application, a model dynamic loading and processing device is provided, comprising:

[0045] A receiving module is configured to receive a first calling request, the first calling request corresponding to a first programming language;

[0046] A conversion module is configured to convert the first calling request into a second calling request; the second calling request corresponding to a second programming language;

[0047] A selection module is configured to select a target object from a plurality of preset objects based on the second calling request; the plurality of preset objects are created based on a plurality of models; the plurality of preset objects are used to support the plurality of models to perform inference;

[0048] An acquisition module is configured to acquire a target model file based on the second calling request; the target model file includes a model network structure and a weight parameter;

[0049] An inference module is configured to perform inference based on the target model file and the target object. BRIEF DESCRIPTION OF DRAWINGS

[0050] The above and other features, advantages and aspects of the present disclosure will become more apparent by describing in detail some embodiments thereof with reference to the attached drawings in which:

[0051] Figure 1 A flowchart of a model dynamic loading and processing method provided by Embodiment 1 of the present application is shown;

[0052] Figure 2 A flowchart of a model dynamic loading and processing method provided by Embodiment 5 of the present application is shown;

[0053] Figure 3 An interaction diagram of a three-level architecture of a JNI layer, an intermediate bridging library and a dynamic loading framework provided by the present application is shown.

[0054] Figure 4 A flowchart of a model dynamic loading and processing method provided for Embodiment 7 of the present application is shown in FIG. 7.

[0055] Figure 5 A flowchart of a model dynamic loading and processing method provided for Embodiment 8 of the present application is shown in FIG. 8.

[0056] Figure 6 A flowchart of a model dynamic loading and processing method provided for Embodiment 9 of the present application is shown in FIG. 9.

[0057] Figure 7 A structural diagram of a model dynamic loading and processing device provided for the present application is shown in FIG. 10. DETAILED DESCRIPTION

[0058] The embodiments of the present application are described below in conjunction with the accompanying drawings. The terms used in the embodiment part of the present application are only used to explain the specific embodiments of the present application, and are not intended to limit the present application.

[0059] The embodiments of the present application are described below in conjunction with the accompanying drawings. It is known to those of ordinary skill in the art that, as technology develops and new scenarios appear, the technical solutions provided by the embodiments of the present application are also applicable to similar technical problems.

[0060] The terms "first", "second", and the like in the present application are used to distinguish similar objects, and do not necessarily indicate a specific order or sequence. It should be understood that the terms used in this way can be interchanged under appropriate circumstances, which is only a distinguishing way adopted in the description of the embodiments of the present application for the objects with the same attribute. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, so that the processes, methods, systems, products or devices containing a series of units do not have to be limited to those units, but can include other units not clearly listed or inherent to these processes, methods, products or devices.

[0061] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the present application will be further described in detail below in conjunction with the accompanying drawings and specific embodiments.

[0062] Reference Figure 1 A flowchart of a model dynamic loading and processing method provided for Embodiment 1 of the present application is shown in FIG. 1, which can include but is not limited to the following steps: Figure 1

[0063] Step S101, receiving a first calling request, the first calling request corresponding to a first programming language.

[0064] ​In this embodiment, a second function written in a second programming language in the intermediate bridging library can be encapsulated into a first function corresponding to the first programming language. For example, the second function written in C or C++ (i.e., an implementation of the second programming language) in the intermediate bridging library can be encapsulated into a first function corresponding to Java or Kotlin (i.e., an implementation of the first programming language) using JNI technology.

[0065] The call entry point encapsulated in the first function can be exposed to the application layer. The application layer can then invoke this call entry point, based on actual inference needs (such as image classification, text semantic analysis, or speech recognition), to initiate a first call request. This first call request can include parameters required for model invocation, such as model identifiers and inference task identifiers.

[0066] In this embodiment, the JNI layer can receive a first call request and, based on the call entry point encapsulated by the first function called by the application layer, obtain a second function in the intermediate bridging library. The JNI layer can then call the second function, passing the first call request to the execution context of the second function in the intermediate bridging library.

[0067] Step S102: Convert the first call request into a second call request; the second call request corresponds to a second programming language.

[0068] In this embodiment, after the first call request is passed to the execution context of the second function of the intermediate bridging library, by executing the second function, the first parameter of the first programming language in the first call request can be converted into the second parameter of the second programming language, and then the second parameter can be organized into the second call request according to the specification of the second programming language.

[0069] The conversion logic of the intermediate bridging library can be completely decoupled from the model type and inference task type. Regardless of whether the first call request is for image classification, NLP (Natural Language Processing), or speech recognition model, the same set of data format mapping rules and request protocol structure are used. When adding a new model or inference task, there is no need to modify the conversion code of the intermediate bridging library. It is only necessary to ensure that the parameters of the first call request initiated by the application layer conform to the general specification to achieve seamless conversion, which greatly improves the scalability of the architecture.

[0070] Step S103: Based on the second invocation request, select a target object from multiple preset objects; the multiple preset objects are created based on multiple models; the multiple preset objects are used to support the multiple models inference.

[0071] In this embodiment, the second call request, converted by the intermediate bridging library, can be passed to the dynamic loading framework corresponding to the second programming language.

[0072] The dynamic loading framework can obtain the model identifier from the second calling request, and select a target object corresponding to the model identifier from a plurality of preset objects.

[0073] The preset object can be used to assist the inference framework (such as ONNX Runtime, Lllama3, OpenVINO, etc.) to perform inference, for example, triggering the initialization of the inference framework, and converting the input data into a format recognizable by the inference framework before inference.

[0074] During the inference process, the preset object can provide model-specific logic to the inference framework, so that when the inference framework encounters a model-defined operation that cannot be recognized, the model-specific logic in the preset object can be directly called to complete the operation analysis and execution of this step, ensuring the integrity of the core inference.

[0075] After the inference framework obtains the output result, the output result of the inference framework can be converted. For example, the probability value output by the inference framework is converted into a category label.

[0076] In step S104, a target model file is obtained based on the second calling request; the target model file includes a model network structure and a weight parameter.

[0077] The dynamic loading framework can obtain the model identifier from the second calling request, determine the storage path of the target model file according to the model identifier, and read the target model file from the corresponding storage location according to the storage path.

[0078] The model network structure can include but is not limited to: the hierarchical architecture of the model (such as the number, order and connection method of convolution layers, fully connected layers and attention layers), the operation rules of each layer (such as the size of the convolution kernel, the step, the type of activation function), the dimensions and data types of input and output tensors, and other topological information. The model network structure can determine the inference process and functional boundary of the model.

[0079] The weight parameter can include the core parameter obtained after a large number of sample training of the model, such as the convolution kernel weight, the bias value, the attention weight matrix, and the fully connected layer weight. The weight parameter can determine the accuracy and effect of the model inference. The better the weight parameter of the same model is, the better the inference effect is.

[0080] In step S105, inference is performed based on the target model file and the target object.

[0081] In this embodiment, the dynamic loading framework can pass the target model file and the target object to the inference framework.

[0082] The inference framework can be initialized based on the target object, and the input data can be converted into target data in a recognizable format.

[0083] Then, the inference framework can read the target model file and, based on the target model file and target data, perform core tensor operations such as convolution and matrix multiplication.

[0084] During the core tensor operations, the inference framework can call the model-specific logic in the target object to perform custom model operations.

[0085] After the inference framework obtains the output (such as a probability tensor), the output can be converted into a business result that the application layer can recognize (such as a "cat / dog" category label) based on the target object.

[0086] In this embodiment, by encapsulating a second function written in a second programming language (e.g., C / C++) in the intermediate bridging library into a unified first function and exposing it to the application layer (e.g., Java / Kotlin), the application layer only needs to call this unified entry point to initiate call requests for different models (e.g., AI models). Subsequently, the JNI layer can pass the first call request to the execution context of the second function in the intermediate bridging library. By executing the second function in the intermediate bridging library, the first call request is converted into a second call request, thus shielding the differences between different operating system platforms and the implementation differences of different models, achieving complete decoupling between the application layer and the specific model.

[0087] Based on this, the second call request is passed to the dynamic loading framework, which then matches the corresponding target object and target model file on demand based on the model identifier in the request. This enables cross-language dynamic loading and inference of the model, effectively avoiding the tedious problem of having to define an independent interface for each model and repeatedly compile and package it in JNI calls, and greatly simplifying the development and maintenance costs of the application layer.

[0088] Furthermore, when adding or iterating a model, only the corresponding preset object and model file need to be added, without modifying the core code in the three-level architecture of the JNI layer, intermediate bridging library and dynamic loading framework, which further improves the system's flexibility, scalability and maintainability.

[0089] As another optional embodiment of this application, this embodiment provides a method for dynamic loading and processing of models in embodiment 2. This embodiment is mainly an implementation of multiple preset objects in embodiment 1. The multiple preset objects may include, but are not limited to:

[0090] There is at least one first preset object and multiple second preset objects; the first preset object can be reused by the multiple models; the second preset object belongs exclusively to one of the multiple models.

[0091] In this embodiment, the first preset object may include, but is not limited to, at least one of the following:

[0092] General data format conversion logic: Convert input data (e.g., ...) Image byte streams of type int[] and audio sample data of type int[] are uniformly converted into tensor format that can be recognized by the inference framework (ONNX Runtime / OpenVINO / Llama3).

[0093] General basic operation logic: Basic computational operations shared by multiple models, such as tensor dimension verification, general data normalization template (excluding model-specific normalization parameters), and basic logic for invalid value filtering.

[0094] The general initialization logic of the inference framework is as follows: it uniformly triggers the startup of the inference framework, configures the global runtime environment (such as specifying GPU / CPU devices, setting memory cache size, configuring the operation precision level), and creates a general inference session.

[0095] The general management logic for inference framework resources is as follows: It unifies the release of session resources, tensor memory, and hardware device handles of the inference framework to avoid memory leaks; it unifies the monitoring of the running status of the inference framework (such as resource utilization and operation exception alarms) and provides a unified fault tolerance processing entry point.

[0096] The second preset object may include, but is not limited to, input processing logic specific to the model. For example, the input processing logic of an image model may include: specific image size scaling (e.g., a fixed 224×224 pixels for a cat / dog classification model, and a fixed 112×112 pixels for a face recognition model), specific normalization parameters (e.g., the mean and variance of the R / G / B channels for a specific model), and specific image enhancement logic (e.g., rotation and cropping rules for a specific model).

[0097] The input processing logic of NLP-like models can include: exclusive word segmentation rules (such as word segmentation logic of domain-specific vocabularies), exclusive text embedding rules (such as custom word vector mapping relationships), and exclusive text length truncation / padding rules.

[0098] The input processing logic of speech models may include: dedicated audio sampling rate conversion (such as a fixed 16kHz sampling rate for a specific model), dedicated audio frame division rules, and dedicated noise suppression parameters.

[0099] Custom layer operation logic: Implementation of proprietary network layers that the inference framework cannot recognize, such as the operation logic of custom convolutional layers for specific image models, proprietary attention layers for NLP models, and proprietary encoding layers for speech models.

[0100] Dedicated algorithm optimization logic: Personalized optimization of the core computation process of a specific model, such as parallelization optimization of convolution operations and attention weight pruning logic for a certain model.

[0101] Model-specific post-processing logic: For example, the post-processing logic of a classification model may include: a mapping relationship between a specific tensor index and a category label (e.g., in a cat and dog classification model, index 0 corresponds to "cat" and index 1 corresponds to "dog"; in a flower classification model, index 0 corresponds to "rose") and specific probability threshold filtering rules (e.g., a model sets a probability ≥ 0.8 to output the category result).

[0102] The post-processing logic of recognition models may include: a matching algorithm between the face recognition model's exclusive feature vector and the identity database, a decoding logic for converting the exclusive audio feature matrix of the speech recognition model into a text sequence, and exclusive character concatenation and error correction rules for the OCR model.

[0103] In this embodiment, step S103 may include, but is not limited to:

[0104] Step S11: If there is only one first preset object, pass it directly as the first target object to the inference framework.

[0105] Step S12: If there are multiple first preset objects, select the first target object from the multiple first preset objects based on the second call request.

[0106] Step S13: Based on the second call request, select a second target object from multiple second preset objects.

[0107] The inference framework can perform inference based on a first target object and a second target object. For example, the inference framework can complete general initialization and general data format conversion based on the first target object, and perform model-specific input preprocessing (such as specific size scaling and normalization) based on the second target object, ultimately obtaining target data that the inference framework can directly use.

[0108] The inference framework primarily performs core tensor operations such as convolution and matrix multiplication. When it encounters a custom operation that cannot be recognized, it calls the model-specific custom operation logic in the second target object to ensure the integrity of the inference. The first target object provides general support such as thread safety control and resource usage monitoring.

[0109] After the inference framework outputs the original results (such as probability tensors), model-specific post-processing (such as specific label mapping and probability filtering) can be performed based on the second target object to obtain business results.

[0110] In this embodiment, the first preset object can be reused by multiple models, avoiding redundant operations of repeatedly developing and initializing the inference framework for each model, reducing the system's memory usage and CPU overhead. At the same time, maintaining the first preset object that can be reused by multiple models can also reduce the risk of conflicts during code iteration.

[0111] Furthermore, when adding or iterating a model, only the corresponding second preset object needs to be developed or modified, without changing the first preset object and the third-level architecture code. This effectively simplifies the development and maintenance costs in multi-model scenarios and improves the flexibility of model inference, making it better suited to the inference needs of multiple models.

[0112] As another optional embodiment of this application, this embodiment provides a model dynamic loading and processing method in embodiment 3 of this application. This embodiment is mainly an implementation of the second preset object in embodiment 2. The second preset object may include, but is not limited to, independent reasoning logic saved through a dynamic link library file; the independent reasoning logic is dedicated to reasoning for a single model among the multiple models.

[0113] In this embodiment, to adapt to different operating systems, the dynamic link library file corresponding to the second preset object can adopt the standard format of the corresponding platform to ensure portability. For example, for Windows systems, the format of the dynamic link library file may include: .dll files (such as cat_dog_classify.dll, corresponding to the second preset object of the cat and dog classification model); or, for Linux / Unix systems, the format of the dynamic link library file may include: .so files (such as face_recognize.so, corresponding to the second preset object of the face recognition model).

[0114] In this embodiment, the explanation of the second preset object in Embodiment 2 can be used to understand the independent reasoning logic saved through the dynamic link library file, which will not be repeated here.

[0115] Dynamic link library files can be stored in a specified directory, and a mapping relationship between model identifiers and dynamic link library file paths can be established for the dynamic loading framework to quickly query and load them.

[0116] In this embodiment, the first preset object may, but is not limited to, be encapsulated as a function or class, embedded in the source code of the dynamic loading framework, initialized together with the dynamic loading framework when the system starts, and resides in memory for reuse by multiple models.

[0117] By encapsulating the first predefined object as a function or class, this approach is suitable for scenarios with relatively stable general logic, no need for frequent iterations, and a single system deployment environment (such as a small, privately deployed multi-model inference system). This method offers advantages such as high call efficiency (no need to load interfaces across files), high integration, and no need for additional management of dynamic link library files.

[0118] In this embodiment, the first preset object may also include: general inference logic stored in a dynamic link library file. The dynamic link file can be dynamically loaded and the general inference logic initialized upon system startup.

[0119] The general inference logic stored in dynamic link library files can be applied to scenarios where the general logic needs to be frequently iterated, the system deployment environment is complex (multiple servers, multiple architectures), and the general logic needs to be uniformly distributed (such as large cloud service inference platforms).

[0120] In this embodiment, after the general inference logic saved through the dynamic link library file is modified, only the dynamic link library file needs to be recompiled, without recompiling the entire dynamic loading framework, thus shortening the iteration cycle.

[0121] In this embodiment, when a single model undergoes logical iteration or anomaly repair, only the corresponding dynamic link library file needs to be compiled and replaced, without modifying the system core code or other model files, which greatly reduces the development and maintenance costs in multi-model scenarios.

[0122] Furthermore, dynamic link library files can be dynamically loaded and unloaded during system operation, supporting hot deployment and seamless iteration of independent inference logic without restarting system services, effectively improving system availability and continuity.

[0123] As another optional embodiment of this application, this embodiment provides a method for dynamic loading and processing of a model in embodiment 4. This embodiment is mainly an implementation of the first call request in embodiment 1. The first call request may include, but is not limited to, multiple first call sub-requests, which are generated by calling the first interface and the second interface respectively.

[0124] Both the first interface and the second interface are uncoupled from the plurality of models. In this embodiment, the uncoupled relationship between the first interface and the plurality of models can be understood from the following aspects:

[0125] In terms of code implementation, the declaration (definition) of the first interface is written in the application layer code (e.g., code in the first programming language such as Java / Kotlin), which is the interaction entry point for application layer developers to directly define and call. However, the implementation code of the first interface (within the application layer) does not directly reference or call specific methods or properties of the model, nor does it contain any model-specific business logic. It only interacts indirectly with model-related resources through the JNI layer and intermediate bridging libraries.

[0126] Even if the model-related resources (preset objects, model files) are iteratively updated (such as replacing the weight file of the cat and dog classification model, or updating the dedicated preprocessing logic of the face recognition model), or if models are added or deleted (such as adding an OCR model or deleting an outdated speech recognition model), the code (declaration + implementation) of the first interface does not need to be modified. As long as the model-related resources follow the unified specifications (such as the preset objects conforming to the recognition standards of the dynamic loading framework, and the model file path mapping relationship being complete), the first interface can still work normally.

[0127] In this embodiment, the understanding that the second interface and multiple models are uncoupled can be referred to the relevant introduction of the first interface, which will not be repeated here.

[0128] The first interface and the second interface can be encapsulated based on the first function in Embodiment 1, respectively.

[0129] The first interface is used to trigger the selection of an object from multiple preset objects and perform inference initialization; the second interface is used to trigger the loading of the model file.

[0130] In this embodiment, the application layer can call the first interface to initiate the first first call sub-request, and the JNI layer can map the first interface to the second function in the intermediate bridging library. The JNI layer can then call the second function, passing the first first call sub-request to the execution context of the second function in the intermediate bridging library.

[0131] After passing the first first call subrequest to the execution context of the second function of the intermediate bridging library, by executing the second function, the first parameter of the first programming language in the first first call subrequest can be converted into the second parameter of the second programming language, and then the second parameter can be organized into the first second call subrequest according to the specification of the second programming language.

[0132] The first second call sub-request can be passed to the dynamic loading framework, which can select the target object from multiple preset objects and perform inference initialization.

[0133] After calling the first interface, the application layer can call the second interface, initiating a second first call sub-request. The JNI layer can then map the second interface to a second function in the intermediate bridging library. The JNI layer can then call the second function, passing the second first call sub-request to the execution context of the second function in the intermediate bridging library.

[0134] After passing the second first call sub-request to the execution context of the second function of the intermediate bridging library, by executing the second function, the first parameter of the first programming language in the second first call sub-request can be converted into the second parameter of the second programming language, and then the second parameter can be organized into the second second call sub-request according to the specification of the second programming language.

[0135] The second sub-request can be passed to the dynamic loading framework, which can then load the target model file.

[0136] In this embodiment, both the first interface and the second interface are uncoupled from the multiple models and do not involve any model-specific code. When adding, iterating, or deleting a model, it is only necessary to maintain the mapping relationship between the model identifier and the preset object and model file. There is no need to modify the interface code, which reduces the development and maintenance cost of the application layer in multi-model scenarios. When making JNI calls, it is not necessary to define an independent interface for each model.

[0137] Furthermore, the step-by-step execution of initialization and model loading allows inference initialization to be reused for multiple model inference operations, avoiding the waste of system resources such as CPU and memory caused by repeated initialization inference, and improving resource utilization and execution efficiency.

[0138] As another optional embodiment of this application, refer to Figure 2 This is a flowchart illustrating a model dynamic loading and processing method provided in Embodiment 5 of this application. This embodiment mainly describes one implementation of step S105 above. Figure 2 As shown, step S105 may include, but is not limited to, the following steps:

[0139] Step S21: Select the target hardware unit from multiple hardware units.

[0140] In this embodiment, a target hardware unit that meets at least one of the following conditions can be selected from multiple hardware units: optimal performance, lowest current load, and meets the model inference resource requirements.

[0141] Step S22: Select a target inference dependency library that matches the target hardware unit from the preset inference dependency library. The target inference dependency library is used to drive the target hardware unit to perform inference.

[0142] In this embodiment, a mapping relationship between hardware index information (e.g., hardware type and / or model) and a preset inference dependency library can be pre-built.

[0143] In this embodiment, the target inference dependency library corresponding to the type and / or model of the target hardware unit can be found from the mapping relationship.

[0144] If no matching inference dependency library is found in the mapping relationship, a general inference dependency library (such as the ONNXRuntime CPU version) can be selected as the target inference dependency library to ensure that the inference process is not interrupted.

[0145] Step S23: Perform inference on the target hardware unit based on the target model file, the target object, and the target inference dependency library.

[0146] In this embodiment, the general initialization of the inference framework can be completed based on the target object, and the input data can be converted into a format recognizable by the target hardware unit and the inference dependency library.

[0147] The inference framework can drive the target hardware unit through the target inference dependency library, load the target model file, and perform core tensor operations such as convolution, matrix multiplication, and attention calculation. During the operation, if the inference framework encounters custom logic that it cannot recognize, it calls the model-specific custom operation logic in the second target object to ensure the integrity of the inference.

[0148] After inference is complete, the inference results (such as probability tensors) can be converted into application-layer recognizable business results (such as category labels and text content) based on the target object. Finally, the business results are returned to the application layer after being reverse-converted by an intermediate bridging library and the JNI layer.

[0149] In this embodiment, by selecting a target hardware unit from multiple hardware units and a target inference dependency library that matches the target hardware unit from a preset inference dependency library, compatibility issues caused by strong binding between hardware and dependency libraries can be effectively avoided, enabling adaptive inference across hardware platforms. Furthermore, by adapting the target inference dependency library to the target hardware unit, the computing power advantages of hardware units such as GPUs and dedicated AI accelerators can be fully utilized, improving the speed of model inference compared to fixed CPU inference or mismatched dependency libraries.

[0150] In this embodiment, combined with Figure 3 This section explains the functionality of the three-tier architecture: the JNI layer, the intermediate bridging library, and the dynamically loaded framework. For example... Figure 3 As shown, the Java / Kotlin application layer can call the JNI interface (i.e., the call entry point encapsulated by the first function, which corresponds to the Java or Kotlin language) to initiate the first call request.

[0151] The JNI layer can receive the first call request and, based on the call entry point encapsulated in the first function from the application layer, map it to the second function in the intermediate bridge library. The JNI layer can then call the second function, passing the first call request to the execution context of the second function in the intermediate bridge library.

[0152] The intermediate bridging library, by executing a second function, can first convert the first parameter in the first programming language of the first call request into a second parameter in the second programming language, and then organize the second parameter into a second call request according to the specifications of the second programming language. The intermediate bridging library can pass the second call request to the dynamically loaded framework by dynamically loading the C Interface / ExportedFunctions exposed by the framework.

[0153] The dynamic loading framework can select a target object from multiple preset objects. Preset objects can include independent inference logic stored in dynamic link library files, such as me5s / clip / intents / ocr / nlp, etc.

[0154] The dynamic loading framework can select matching target inference dependency libraries for target hardware units (e.g., Intel / AMD / NVIDIA etc.) based on the LLM Platform Loader. The LLM Platform Loader can be stored as a dynamic link library file.

[0155] Alternatively, the dynamic loading framework can select matching target inference dependency libraries for the target hardware unit (e.g., Intel / AMD / NVIDIA etc.) based on the SD3 Platform Loader. The LSD3 Platform Loader can be stored as a dynamic link library file.

[0156] The dynamic loading framework can perform inference on the target hardware unit based on the target model file, the target object, and the target inference dependency library.

[0157] As another optional embodiment of this application, the model dynamic loading and processing method provided in Embodiment 6 of this application is mainly an implementation of the above-mentioned step S21, which may include, but is not limited to, any of the following:

[0158] Step S211: Obtain a hardware unit selection instruction from the call request, and select a target hardware unit from multiple hardware units based on the hardware unit selection instruction;

[0159] When initiating a call request, the application layer can add hardware unit selection instructions to the request based on actual needs, business scenarios, or user specifications. For example, in a real-time image recognition scenario with extremely high inference speed requirements, if the application layer wants to use a powerful GPU for inference, it can explicitly specify the use of the GPU hardware unit in the call request.

[0160] After receiving the call request, the dynamic loading framework parses the hardware unit selection instruction. Based on the hardware unit type specified in the instruction, it filters among multiple available hardware units in the system. For example, if the instruction specifies using a GPU, the dynamic loading framework will select one from all available GPU devices in the system as the target hardware unit. If multiple hardware units that meet the instruction requirements exist in the system (such as multiple GPUs), the selection can be further refined based on other conditions (such as load conditions, performance parameters, etc.), or one can be selected according to default rules (such as by device number order).

[0161] In this embodiment, the hardware unit selection instruction obtained from the call request is used to directly specify the target hardware, which can meet the precise hardware requirements of specific business scenarios and improve the user experience.

[0162] Step S212: Obtain hardware resource parameters, and select a target hardware unit from multiple hardware units based on the hardware resource parameters; the hardware resource parameters characterize the operating status of the multiple hardware units.

[0163] Before inference, the dynamic loading framework automatically scans all available hardware units in the current system (such as NVIDIA CUDA, AMD ROCm, Intel or other accelerators) and collects hardware resource parameters (hardware type, model, computing power, memory capacity, current load rate, whether it supports deep learning inference, etc.).

[0164] In this embodiment, selecting a target hardware unit from multiple hardware units based on the hardware resource parameters may include, but is not limited to, at least one of the following:

[0165] Step S31: Evaluate the performance of each hardware unit based on the computing power, memory capacity and other indicators in the hardware resource parameters.

[0166] Choose the most powerful hardware unit as the target hardware unit. For example, when comparing multiple GPUs (Graphics Processing Units), select the GPU with the highest number of floating-point operations per second and the largest video memory to ensure that it can quickly process large-scale model inference tasks.

[0167] Step S32: Consider the current load rate of the hardware unit and select the hardware unit with the lowest load.

[0168] This avoids inference latency or performance degradation caused by hardware unit overload. For example, if the system has two GPUs with similar performance, one with a load rate of 30% and the other with a load rate of 70%, the GPU with the 30% load rate is selected as the target hardware unit to ensure the smoothness of the inference process.

[0169] Step S33: Select hardware units that can meet the resource requirements of the target model (such as model size, computational complexity, etc.).

[0170] For example, a large deep learning model requires significant GPU memory and high computing power to run smoothly. The dynamic loading framework will select hardware units with sufficient GPU memory and computing power as the target hardware unit from the hardware units. If multiple conditions are met simultaneously (optimal performance, lowest current load, and meeting the model inference resource requirements), the selection will be carried out in a certain priority order. For example, the model inference resource requirements will be met first, and then the hardware unit with the best performance and lowest load will be selected based on meeting this condition.

[0171] In this embodiment, the target hardware unit is automatically selected based on hardware resource parameters, which effectively avoids hardware resource waste and inference delay, ensures efficient and smooth inference process, and can adapt to the inference needs of models of different sizes, thereby improving hardware resource utilization and the flexibility and adaptability of model inference.

[0172] As another optional embodiment of this application, refer to Figure 4 This is a flowchart illustrating a model dynamic loading and processing method provided in Embodiment 7 of this application. This embodiment mainly describes one implementation of step S103 above. Figure 4 As shown, step S103 may include, but is not limited to, the following steps:

[0173] Step S41: Obtain first data from the second call request; the first data is used to identify the target model.

[0174] In this embodiment, the first data can be passed in by the application layer when initiating the first call request, and after being converted by the JNI layer and the intermediate bridging library, it can be carried in the second call request.

[0175] The first data may include, but is not limited to, at least one of the following: model name and model version.

[0176] Step S42: Based on the second call request, determine the second data; the second data is used to characterize the inference attributes of the target model.

[0177] In this embodiment, the dynamic loading framework can determine the current inference task based on the second invocation request.

[0178] The dynamic loading framework can determine at least one of the following based on historical data and user behavior analysis: the required model accuracy (e.g., high or low accuracy), task category, and task priority for the current inference task. The task category characterizes the business type of the current inference task, reflecting the core functional requirements of the task for the model. The task priority characterizes the urgency or importance of the current inference task.

[0179] Different inference properties determine the performance and effectiveness of a model when processing a task. High-precision models typically have more complex structures and more parameters, providing more accurate inference results, but they require more computation and have a relatively slower inference speed; low-precision models, on the other hand, have a faster inference speed, but their accuracy may be slightly lower.

[0180] Step S43: Based on at least one of the first data and the second data, match the target object storage path in the configuration file; the configuration file contains a mapping relationship between configuration information and preset object storage paths; the configuration information includes at least one of first configuration information and second configuration information; the first configuration information is used to identify the model; the second configuration information is used to characterize the inference attributes of the model.

[0181] In this embodiment, the configuration file can adopt a hierarchical structure, with each layer corresponding to a type of configuration information, ultimately mapped to a preset object storage path. For example, the first layer of the configuration file can correspond to the model name (i.e., one implementation of the first configuration information), under the model name it can contain the model version (i.e., one implementation of the first configuration information), under the model version it can contain inference attributes (i.e., one implementation of the second configuration information), and the inference attributes correspond to a preset object storage path.

[0182] In this embodiment, the model name in the first data can be used as the first-level index to select all corresponding target version branches from the configuration file.

[0183] Select the target version from the target version branch using the model version in the first data as a secondary index.

[0184] Using the inference attributes in the second data as a third-level index, match the corresponding target object storage path under the target version branch.

[0185] Step S44: Load the target object based on the target object storage path.

[0186] In this embodiment, the loaded target object can be passed to the inference framework.

[0187] In this embodiment, step S104 may include, but is not limited to, the following steps:

[0188] Step S51: Obtain the first data from the second call request.

[0189] Step S52: Determine the second data based on the second call request.

[0190] Step S53: Based on at least one of the first data and the second data, match the target model file storage path in the configuration file; the configuration file contains the mapping relationship between the configuration information and the model file storage path;

[0191] Step S54: Load the target model file based on the target model file storage path.

[0192] In this embodiment, the logic of steps S51-S54 is similar to that of steps S41-S42. To avoid redundancy, the specific implementation details of steps S51-S54 in this embodiment can be found in the detailed description of steps S41-S44, and will not be repeated here.

[0193] In this embodiment, by matching the target object storage path and the target model file storage path in the configuration file based on at least one of the first data and the second data, inference errors caused by version or precision mismatch can be avoided, and the flexibility, accuracy and scalability of dynamic model loading can be improved.

[0194] Furthermore, the mapping relationship of the configuration file supports dynamic expansion. When adding a new model branch or version, there is no need to modify the core code; only the configuration file needs to be updated to complete the adaptation.

[0195] As another optional embodiment of this application, refer to Figure 5 This is a flowchart illustrating a model dynamic loading and processing method provided in Embodiment 8 of this application. This embodiment mainly describes one implementation of step S103 above. Figure 5 As shown, step S103 may include, but is not limited to, the following steps:

[0196] Step S61: Based on the second call request, determine the first model to be used.

[0197] In this embodiment, if the second call request carries a model identifier, the model identifier can be read directly, and the first model to be used can be determined based on the model representation.

[0198] If the second call request does not carry a model identifier, the corresponding first model to be used can be matched based on the business task type (such as "intelligent question answering") corresponding to the second call request, combined with the preset mapping relationship between task types and core models.

[0199] Step S62: Based on the first model to be used, determine the second model to be used; the first model to be used and the second model to be used have a dependency relationship.

[0200] In this embodiment, multiple mapping relationships can be pre-established in the configuration file. Each mapping relationship can include: a model identifier (used to identify any level of first model) and a list of models that have a direct dependency relationship with the first model. The model list can include the model identifier and dependency type (e.g., sequential execution or parallel execution) of the second model that has a direct dependency relationship with the first model. The model identifier of the second model can be used as an independent model identifier, corresponding to its own list of directly dependent models in the configuration file. This allows for the support of multi-level dependencies formed by three or more models (e.g., a three-level dependency of A→B→C, or a parallel dependency between A→B and A→C) through the association of multiple mapping relationships.

[0201] For example, the configuration file can contain two independent mapping relationships: "Model A identifier - Model list B (corresponding to model B (e.g., CLIP model) that has a direct dependency on model A (e.g., Embedding model))" and "Model B identifier - Model list C (corresponding to model C (e.g., LLM model) that has a direct dependency on model B). Through the association of these two mapping relationships, a multi-level dependency of A→B→C can be formed, where model B is the second model to be used for model A and the first model to be used for model C.

[0202] In this embodiment, based on the model identifier of the first model to be used, the corresponding model list in the configuration file can be matched to extract all second model identifiers and dependency types that directly depend on the first model to be used. Then, for each extracted second model identifier, the configuration file is checked again to see if there is a mapping relationship with that identifier as the model identifier. If there is, the next level directly dependent model identifier and dependency type are extracted. This layer-by-layer matching and loop traversal method is used until a model with no subsequent dependencies is traversed.

[0203] If the target dependency model list is not matched in the configuration file, inference can be performed based solely on the first model without loading the second model to be used.

[0204] Step S63: Select the first target object and the second target object corresponding to the first model to be used and the second model to be used respectively from multiple preset objects.

[0205] In this embodiment, the method of selecting the target object in Embodiment 7 can be used to understand step S63, which will not be repeated here.

[0206] In this embodiment, step S104 may include, but is not limited to:

[0207] Step S71: Based on the second call request, determine the first model to be used.

[0208] Step S72: Based on the first model to be used, determine the second model to be used; the first model to be used and the second model to be used have a dependency relationship.

[0209] Step S73: Obtain the first target model file and the second target model file corresponding to the first model to be used and the second model to be used, respectively.

[0210] In this embodiment, the logic of steps S71-S73 is similar to that of steps S61-S63. To avoid redundancy, the specific implementation details of steps S71-S73 in this embodiment can be found in the detailed description of steps S61-S63, and will not be repeated here.

[0211] In this embodiment, the dynamic loading framework can automatically identify all directly or indirectly dependent models, eliminating the need for the application layer to manually write multi-model calling logic. A single call request is all that is needed to trigger the complete collaborative process, reducing the application layer development and maintenance costs in complex inference scenarios.

[0212] As another optional embodiment of this application, refer to Figure 6 This is a flowchart illustrating a model dynamic loading and processing method provided in Embodiment 9 of this application. This embodiment is mainly an implementation of step S105 in Embodiment 8 above. In this embodiment, the first model to be used and the second model to be used can correspond to the same process, such as the process of dynamically loading the framework.

[0213] like Figure 6 As shown, step S105 may include, but is not limited to, the following steps:

[0214] Step S81: Within the same process, request shared resources for the first model to be used; the shared resources include at least one of a target buffer and a target resource; the target resource is used for inference initialization and inference execution scheduling.

[0215] In this embodiment, the process of the dynamic loading framework can apply for a corresponding size of video memory space (i.e., an implementation of the target buffer) according to the inference requirements of the first model to be used (e.g., tensor operation scale, input and output data volume), and this video memory space belongs to the process management of the dynamic loading framework.

[0216] Target resources may include, but are not limited to: inference sessions (e.g., ONNX Session), inference framework configuration information, thread scheduling pools, etc.

[0217] Step S82: Within the same process, after reasoning based on the first target object, the first target model file, and the shared resources, the shared resources are allocated to the second model to be used.

[0218] After reasoning based on the first target object, the first target model file, and the shared resources, the process of dynamically loading the framework may not release the target buffer it occupies, nor destroy target resources such as ONNXSession. Instead, these resources can be marked as idle and retained in the process's resource pool.

[0219] Step S83: Perform inference based on the second target object, the second target model file, and the shared resources.

[0220] The target buffer can be used to hold tensor operation data of the second model to be used. Target resources such as ONNX Session can directly support the initialization and execution scheduling of the second model to be used without additional configuration.

[0221] In this embodiment, a target buffer and target resources are requested for the first model to be used within the same process. After inference is completed, the resources are not released but marked as idle and directly allocated to the second model to be used. This can avoid repeated resource requests and releases, reduce frequent memory allocation and cross-language data copying, and improve inference efficiency.

[0222] The following section introduces the model dynamic loading and processing apparatus provided in this application. The model dynamic loading and processing apparatus described below can be referred to in correspondence with the model dynamic loading and processing method described above.

[0223] Reference Figure 7 The model dynamic loading and processing device includes: a receiving module 100, a conversion module 200, a selection module 300, an acquisition module 400, and an inference module 500.

[0224] The receiving module 100 is used to receive a first call request, the first call request corresponding to a first programming language.

[0225] The conversion module 200 is used to convert the first call request into a second call request; the second call request corresponds to a second programming language.

[0226] Selection module 300 is used to select a target object from multiple preset objects based on the second invocation request; the multiple preset objects are created based on multiple models; the multiple preset objects are used to support the multiple models to perform inference.

[0227] The acquisition module 400 is used to acquire the target model file based on the second call request; the target model file includes the model network structure and weight parameters.

[0228] The reasoning module 500 is used to perform reasoning based on the target model file and the target object.

[0229] In this embodiment, the plurality of preset objects may include:

[0230] There is at least one first preset object and multiple second preset objects; the first preset object can be reused by the multiple models; the second preset object belongs exclusively to one of the multiple models.

[0231] In this embodiment, the second preset object may include: independent inference logic saved through a dynamic link library file; the independent inference logic is dedicated to a single model among the multiple models for inference.

[0232] In this embodiment, the first invocation request may include multiple first invocation sub-requests, which are generated by invoking a first interface and a second interface, respectively; the first interface and the second interface are both uncoupled from the multiple models;

[0233] The first interface is used to trigger the selection of an object from multiple preset objects and perform inference initialization; the second interface is used to trigger the loading of the model file.

[0234] In this embodiment, the inference module 500 can be specifically used for:

[0235] Select the target hardware unit from multiple hardware units;

[0236] Select a target inference dependency library that matches the target hardware unit from the preset inference dependency library. The target inference dependency library is used to drive the target hardware unit to perform inference.

[0237] Inference is performed on the target hardware unit based on the target model file, the target object, and the target inference dependency library.

[0238] In this embodiment, the inference module 500 selects a target hardware unit from a plurality of hardware units, which may include any of the following:

[0239] Obtain a hardware unit selection instruction from the call request, and select a target hardware unit from multiple hardware units based on the hardware unit selection instruction;

[0240] Obtain hardware resource parameters, and select a target hardware unit from multiple hardware units based on the hardware resource parameters; the hardware resource parameters characterize the operating status of the multiple hardware units.

[0241] In this embodiment, the selection module 300 can be specifically used for:

[0242] Obtain first data from the second call request; the first data is used to identify the target model;

[0243] Based on the second invocation request, second data is determined; the second data is used to characterize the inference attributes of the target model;

[0244] Based on at least one of the first data and the second data, a target object storage path is matched in the configuration file; the configuration file contains a mapping relationship between configuration information and preset object storage paths; the configuration information includes at least one of first configuration information and second configuration information; the first configuration information is used to identify the model; the second configuration information is used to characterize the inference attributes of the model;

[0245] Load the target object based on the target object's storage path.

[0246] Module 400 can be used specifically for:

[0247] Obtain the first data from the second call request;

[0248] Based on the second call request, the second data is determined;

[0249] Based on at least one of the first data and the second data, the target model file storage path is matched in the configuration file; the configuration file contains the mapping relationship between the configuration information and the model file storage path;

[0250] Load the target model file based on the target model file storage path.

[0251] In this embodiment, the selection module 300 can be specifically used for:

[0252] Based on the second call request, the first model to be used is determined;

[0253] Based on the first model to be used, a second model to be used is determined; the first model to be used and the second model to be used have a dependency relationship;

[0254] Select the first target object and the second target object corresponding to the first model to be used and the second model to be used, respectively, from a plurality of preset objects.

[0255] In this embodiment, the acquisition module 400 can be specifically used for:

[0256] Based on the second call request, the first model to be used is determined;

[0257] Based on the first model to be used, a second model to be used is determined; the first model to be used and the second model to be used have a dependency relationship;

[0258] Obtain the first target model file and the second target model file corresponding to the first model to be used and the second model to be used, respectively.

[0259] In this embodiment, the first model to be used and the second model to be used correspond to the same process; the inference module 500 can specifically be used for:

[0260] Within the same process, shared resources are requested for the first model to be used; the shared resources include at least one of a target buffer and a target resource; the target resource is used for inference initialization and inference execution scheduling.

[0261] Within the same process, after reasoning based on the first target object, the first target model file, and the shared resources, the shared resources are allocated to the second model to be used;

[0262] Reasoning is performed based on the second target object, the second target model file, and the shared resources.

[0263] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the device embodiment drawings provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0264] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, training equipment, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0265] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0266] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, training device, or data center to another website, computer, training device, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)).

Claims

1. A model dynamic loading and processing method, comprising: receiving a first calling request, the first calling request corresponding to a first programming language; converting the first calling request into a second calling request; the second calling request corresponding to a second programming language; selecting a target object from a plurality of preset objects based on the second calling request; the plurality of preset objects being created based on a plurality of models; the plurality of preset objects being used to support the plurality of models to perform inference; obtaining a target model file based on the second calling request; the target model file including a model network structure and weight parameters; performing inference based on the target model file and the target object.

2. The model dynamic loading and processing method of claim 1, wherein the plurality of preset objects comprises: at least one first preset object and a plurality of second preset objects; the first preset object being reusable by the plurality of models; the second preset object being exclusive to one of the plurality of models.

3. The model dynamic loading and processing method of claim 2, wherein the second preset object comprises: an independent inference logic saved through a dynamic link library file; the independent inference logic being exclusive to a single model of the plurality of models for performing inference.

4. The model dynamic loading and processing method of claim 1, wherein the first calling request comprises a plurality of first calling sub-requests, each of which is generated by calling a first interface and a second interface; the first interface and the second interface are in a non-coupling relationship with the plurality of models; the first interface is used to trigger selection of an object from a plurality of preset objects and initialization of inference; and the second interface is used to trigger loading of a model file.

5. The model dynamic loading and processing method of claim 1, wherein performing inference based on the target model file and the target object comprises: selecting a target hardware unit from a plurality of hardware units; selecting a target inference dependency library matching the target hardware unit from a preset inference dependency library, the target inference dependency library being used to drive the target hardware unit to perform inference; and performing inference on the target hardware unit based on the target model file, the target object, and the target inference dependency library.

6. The model dynamic loading and processing method of claim 5, wherein selecting a target hardware unit from a plurality of hardware units comprises any one of: obtaining a hardware unit selection instruction from the calling request, and selecting a target hardware unit from a plurality of hardware units based on the hardware unit selection instruction; obtaining a hardware resource parameter, and selecting a target hardware unit from a plurality of hardware units based on the hardware resource parameter; the hardware resource parameter representing a running state of the plurality of hardware units.

7. The model dynamic loading and processing method of claim 1, wherein selecting a target object from a plurality of preset objects based on the second calling request comprises: obtaining first data from the second calling request; the first data being used to identify a target model; determining second data based on the second calling request; the second data being used to represent an inference attribute of the target model; and match, in a configuration file, a target object storage path based on at least one of the first data and the second data; the configuration file contains a mapping relationship between configuration information and a preset object storage path; the configuration information includes at least one of first configuration information and second configuration information; the first configuration information is used to identify a model; and the second configuration information is used to represent inference attributes of the model; load a target object based on the target object storage path; the obtaining, based on the second invocation request, of a target model file includes: obtaining the first data from the second invocation request; determining the second data based on the second invocation request; matching, in a configuration file, a target model file storage path based on at least one of the first data and the second data; the configuration file contains a mapping relationship between the configuration information and a model file storage path; loading a target model file based on the target model file storage path.

8. The model dynamic loading and processing method according to claim 1, wherein the selecting, based on the second invocation request, of a target object from a plurality of preset objects includes: determining a first to-be-used model based on the second invocation request; determining a second to-be-used model based on the first to-be-used model; the first to-be-used model and the second to-be-used model have a dependency relationship; selecting, from the plurality of preset objects, a first target object and a second target object corresponding to the first to-be-used model and the second to-be-used model respectively; the obtaining, based on the second invocation request, of a target model file includes: determining a first to-be-used model based on the second invocation request; determining a second to-be-used model based on the first to-be-used model; the first to-be-used model and the second to-be-used model have a dependency relationship; obtaining a first target model file and a second target model file corresponding to the first to-be-used model and the second to-be-used model respectively.

9. The model dynamic loading and processing method according to claim 8, wherein the first to-be-used model and the second to-be-used model correspond to a same process; and the performing inference based on the target model file and the target object includes: applying, in the same process, a shared resource for the first to-be-used model; the shared resource contains at least one of a target buffer and a target resource; the target resource is used for inference initialization and inference execution scheduling; after performing inference based on the first target object, the first target model file and the shared resource in the same process, allocating the shared resource to the second to-be-used model; performing inference based on the second target object, the second target model file and the shared resource.

10. A model dynamic loading and processing apparatus, comprising: a receiving module configured to receive a first invocation request, the first invocation request corresponding to a first programming language; a conversion module configured to convert the first invocation request into a second invocation request; the second invocation request corresponding to a second programming language; a selection module configured to select, based on the second invocation request, a target object from a plurality of preset objects; ​ The plurality of preset objects are created based on a plurality of models; The plurality of preset objects are used to support the plurality of models to perform reasoning; An acquisition module is configured to acquire a target model file based on the second invocation request; The target model file includes a model network structure and weight parameters; An inference module is configured to perform inference based on the target model file and the target object.

Images