Model migration method and device across inference engines, computer device and storage medium

Abstract

The application discloses a model migration method and device across reasoning engines, a computer device and a storage medium. The method comprises the following steps: loading a model to a new reasoning engine; obtaining meta information of input and output nodes based on the new reasoning engine scanning the model meta information; obtaining new output node information under the engine migration by taking preset probe pseudo data as input data and performing first inference detection based on the new reasoning engine; adapting the input data based on the input meta information; and outputting in the output node information mode based on the new reasoning engine deeply inferring the adapted input data. The application does not attempt to change the model when loading, but automatically scans the meta information of the interface under the new engine as soon as the model enters the new engine. The processing strategy is derived in real time from the adapted meta information, and no matter how the specification of the new engine is, the system can be self-adapted, and the adaptation effect of adapting the model across the reasoning engines by using artificial hard coding is improved.

Description

Technical Field

[0001] This application relates to the field of deep learning inference optimization technology, and in particular to methods, apparatus, computer devices and storage media for model transfer across inference engines. Background Technology

[0002] The process of training and deploying deep learning models typically involves exporting an intermediate format, such as ONNX, from training frameworks like PyTorch and TensorFlow, and then loading and executing it through different inference engines such as ONNX Runtime, TensorRT, and OpenVINO. In multi-model cascaded systems, such as OCR systems containing multiple models including detection, recognition, and KIE, the input and output node names, dimension arrangements, and preprocessing requirements of each model may differ depending on the engine used.

[0003] The core challenge of cross-engine migration lies in the interface adaptation for each model, rather than the model accuracy itself. A common approach in multi-model cascade systems is manual hard-coding of engine adaptation. This involves hard-coding input node names (e.g., "input", "images"), output dimension transpose logic, and preprocessing parameters such as fixed normalization and fixed NCHW layout in the inference code of each model. When switching inference engines, developers must compare the node names, dimension order, and preprocessing differences for each model under the old and new engines, and modify the inference code accordingly.

[0004] However, in the above stages, the input-output mapping relies on manual hard coding. Different engines may export different model node names, numbers, and orders, especially multi-input model networks with multiple input nodes. Manually maintaining the name mapping is labor-intensive and error-prone, and changes to node names require simultaneous modifications to the business code. Furthermore, differences in output dimensions necessitate manual comparison and hard-coded transposition, and preprocessing strategies cannot automatically adapt to engine differences. Therefore, existing technologies suffer from the technical problem of poor adaptation performance when using manual hard-coding for deep learning model cross-inference engine adaptation. Summary of the Invention

[0005] This application provides a method, apparatus, computer device, and storage medium for model migration across inference engines, which can solve the technical problem in the prior art where the adaptation effect of deep learning models using manual hard coding is poor when adapting across inference engines.

[0006] In a first aspect, embodiments of this application provide a model transfer method across inference engines, comprising:

[0007] In response to the model migration instruction, the target model is loaded onto a new local inference engine. The new inference engine is different from the original inference engine of the target model. The target model is used for information recognition and extraction in document understanding.

[0008] Based on the new inference engine, the newly loaded target model is scanned for metadata to obtain the metadata of the input and output nodes. The metadata includes the node name, node dimension, and data type.

[0009] Using the preset probe pseudo data as input data, the target model is subjected to the first inference detection based on the new inference engine to obtain the new output node information under the engine migration.

[0010] Based on the input metadata of the target model input nodes, the preset input data is adapted. The adaptation process includes at least one of dimensional layout transformation, normalization, channel order adjustment, and scaling.

[0011] The new inference engine performs deep inference on the adapted input data and outputs the inference results in the manner indicated by the output node information.

[0012] In some embodiments, a metadata scan is performed on the newly loaded target model based on the new inference engine to obtain the metadata of the input and output nodes, including:

[0013] Meta-information scanning is performed on the newly loaded target model based on the new inference engine;

[0014] If the scan determines that the target model is a multi-input model, the original input data obtained from the scan is processed to obtain multiple tensors, including relation matrix tensors, text encoding tensors, and identity matrix tensors.

[0015] Based on the relation matrix tensor, the text encoding tensor, and the identity matrix tensor, an ordered external input list is constructed, which represents the business processing logic.

[0016] The scanned input node names are mapped to tensors in the external input list to construct a mapping dictionary. The mapping dictionary includes meta-information of input and output nodes, including node name, node dimension, and data type.

[0017] In some embodiments, preset probe pseudo-data is used as input data, and the target model is subjected to initial inference detection based on the new inference engine to obtain new output node information under engine migration, including:

[0018] Generate image pseudo-tensors that meet the input requirements of the target model;

[0019] The image pseudo tensor is input as probe pseudo data into the new inference engine, and a forced forward graph calculation is performed to obtain the output tensor.

[0020] The output tensor is parsed to obtain new output node information under engine migration. The new output node information includes the number of outputs, dimension axis semantics, and dynamic sequence length.

[0021] In some embodiments, based on the input metadata of the target model input nodes, the preset input data is adapted, including:

[0022] Using the input metadata of the target model's input nodes as a reference, the layout dimensions of the preset input data are transformed;

[0023] Normalize the input data after layout transformation;

[0024] Scaling the normalized input data;

[0025] The channel order is automatically matched and adjusted to obtain input data that is compatible with the target model.

[0026] In some embodiments, deep inference is performed on the adapted input data based on a new inference engine, and the inference result is output in the output manner indicated by the output node information, including:

[0027] The new inference engine is used to batch-divide the adapted input data.

[0028] If the last batch is insufficient during partitioning, it will be filled with zero values;

[0029] The input data from the multiple batches is subjected to deep inference batch by batch to obtain effective inference results, wherein the inference results of the last batch are removed from the inference results of the batch containing padding data.

[0030] The valid reasoning results are concatenated and output in their original order to obtain reasoning results that conform to the output method indicated by the output node information.

[0031] In some embodiments, deep inference is performed on the adapted input data based on the new inference engine, and the inference result is output in the output manner indicated by the output node information, further comprising:

[0032] The new inference engine is invoked to perform deep inference on the adapted input data.

[0033] If the inference engine being invoked meets the preset engine disabling conditions, it will be downgraded to the next backup engine based on the preset priority ranking table for engines. The engine disabling conditions include at least one of engine failure, engine timeout, resource exhaustion, and cost overrun.

[0034] Periodically detect the primary engine of the backup engine. If the primary engine meets the preset engine recovery conditions, switch back to the primary engine.

[0035] The adapted input data is re-inferred based on the superior priority engine, and the inference results are output in the manner indicated by the output node information.

[0036] In some embodiments, batch partitioning of the adapted input data based on the new inference engine includes:

[0037] The batch size is extracted from the first input dimension based on the new inference engine;

[0038] The adapted input data is divided into batches based on the batch size.

[0039] Secondly, embodiments of this application also provide a model migration apparatus across inference engines, comprising:

[0040] The loading model unit is used to load the target model onto a new local inference engine in response to a model migration instruction. The new inference engine is different from the original inference engine of the target model. The target model is used for information recognition and extraction in document understanding.

[0041] The meta-information scanning unit is used to scan the newly loaded target model based on the new inference engine to obtain the meta-information of the input and output nodes. The meta-information includes the node name, node dimension, and data type.

[0042] The initial inference detection unit is used to take the preset probe pseudo data as input data and perform the initial inference detection on the target model based on the new inference engine to obtain the new output node information under the engine migration.

[0043] An input adaptation unit is used to adapt preset input data based on the input metadata of the target model input nodes. The adaptation process includes at least one of dimensional layout transformation, normalization, channel order adjustment, and scaling.

[0044] The deep inference unit is used to perform deep inference on the adapted input data based on the new inference engine, and output the inference results in the output manner indicated by the output node information.

[0045] Thirdly, embodiments of this application also provide a cross-inference engine model migration computer device, which includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the above-described method.

[0046] Fourthly, embodiments of this application also provide a computer-readable storage medium storing a computer program, the computer program including program instructions that, when executed by a processor, can implement the above-described method.

[0047] This application provides a method, apparatus, computer device, and storage medium for model migration across inference engines. The method includes: in response to a model migration instruction, loading a target model onto a new local inference engine, the new inference engine being different from the original inference engine of the target model, the target model being used for information recognition and extraction in document understanding; performing a metadata scan on the newly loaded target model based on the new inference engine to obtain metadata of input and output nodes, the metadata including node name, node dimension, and data type; using preset probe pseudo-data as input data, performing an initial inference detection on the target model based on the new inference engine to obtain new output node information under engine migration; performing adaptation processing on preset input data based on the input metadata of the target model's input nodes, the adaptation processing including at least one of dimension layout transformation, normalization, channel order adjustment, and scaling; performing deep inference on the adapted input data based on the new inference engine, and outputting the inference result in the output mode indicated by the output node information. This application does not employ manual hard-coding for engine adaptation, nor does it attempt to modify the model during loading. Instead, as soon as the model enters a new engine, the system automatically and thoroughly scans the metadata of the interfaces under the new engine, passively receiving the input metadata returned by the new engine. All processing strategies are not written into the code logic but are calculated and derived in real time from the scanned metadata. Regardless of the specifications of the new engine, the system can adapt adaptively, demonstrating extremely strong compatibility. This solves the technical problem in existing technologies where manual hard-coding for deep learning model adaptation across inference engines yields poor adaptation results. Attached Figure Description

[0048] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0049] Figure 1 A flowchart illustrating the model migration method across inference engines provided in this application embodiment;

[0050] Figure 2 Schematic block diagram of a model migration apparatus across inference engines provided in embodiments of this application;

[0051] Figure 3 A schematic block diagram of a computer device provided in an embodiment of this application. Detailed Implementation

[0052] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0053] It should be understood that, when used in this specification and the appended claims, the terms "comprising" and "including" indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0054] It should also be understood that the terminology used in this application specification is for the purpose of describing particular embodiments only and is not intended to limit the application. As used in this application specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0055] It should also be further understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0056] The process of training and deploying deep learning models typically involves exporting an intermediate format, such as ONNX, from training frameworks like PyTorch and TensorFlow, and then loading and executing it through different inference engines like ONNX Runtime, TensorRT, and OpenVINO. In multi-model cascade systems, such as OCR systems containing multiple models for detection, recognition, and KIE, the input / output node names, dimension arrangements, and preprocessing requirements of each model may differ depending on the engine. Multi-model cascade systems represent a cutting-edge AI deployment paradigm. By abandoning reliance on a single, all-powerful large model, they instead use an intelligent scheduling layer to collaboratively call multiple models, each with its own strengths, to complete complex tasks with better cost and efficiency. There are various collaboration modes for multiple models, including cascading, which is the classic pipeline model. Requests are first processed by lightweight smaller models; if the smaller model lacks confidence in the results, the task is then escalated to a more powerful larger model. The goal is to achieve a balance between cost and quality by using low-cost smaller models as much as possible while ensuring quality. In practical implementations, a separate model gateway layer is typically introduced to encapsulate the aforementioned complexity. The business layer only needs to interact with the gateway and does not need to know which models are being called in the background.

[0057] The core challenge of cross-engine migration lies in the interface adaptation for each model, rather than the model accuracy itself. A common approach in multi-model cascade systems is manual hard-coding of engine adaptation. This involves hard-coding input node names (e.g., "input", "images"), output dimension transpose logic, and preprocessing parameters such as fixed normalization and fixed NCHW layout in the inference code of each model. When switching inference engines, developers must compare the node names, dimension order, and preprocessing differences for each model under the old and new engines, and modify the inference code accordingly.

[0058] However, in the above stages, the input-output mapping relies on manual hard coding. Different engines may export different model node names, numbers, and orders, especially multi-input model networks with multiple input nodes. Manually maintaining the name mapping is labor-intensive and error-prone, and changes to node names require simultaneous modifications to the business code. Furthermore, differences in output dimensions necessitate manual comparison and hard-coded transposition, and preprocessing strategies cannot automatically adapt to engine differences. Therefore, existing technologies suffer from the technical problem of poor adaptation performance when using manual hard-coding for deep learning model cross-inference engine adaptation.

[0059] In some embodiments, the cross-inference engine model migration method is applied to a cross-inference engine model migration apparatus, which is deployed on a cross-inference engine model migration device. The device is a computer device, and the cross-inference engine model migration computer device can be a terminal or a server. The terminal can be a smartphone, tablet computer, PDA, or laptop computer, etc.

[0060] Figure 1 This is a flowchart illustrating the model migration method across inference engines provided in an embodiment of this application. Figure 1 As shown, the method includes the following steps S110-S150:

[0061] S110. In response to the model migration instruction, the target model is loaded onto the local new inference engine. The new inference engine is different from the original inference engine of the target model. The target model is used for information recognition and extraction in document understanding.

[0062] The target model of this application is mainly used in the field of intelligent document understanding. Specifically, it is a multi-model cascade system that serves OCR (Optical Character Recognition) and KIE (Key Information Extraction).

[0063] New and old engines, such as TensorRT and ONNX Runtime, have different specifications for memory layout (NCHW vs NHWC) and batch dimension processing. When the model is used across inference engines, it does not attempt to change the model at loading time, but passively receives the meta-information returned by the new engine. This ensures that the system can adapt regardless of the specifications of the new engine, demonstrating strong compatibility.

[0064] In response to model migration instructions, the model object is instantiated, and the target model file, such as a .onnx file, is loaded into the new inference engine according to the migration instructions. As an example, when switching from the original inference engine ONNX Runtime to the new inference engine TensorRT runtime environment, an InferenceSession object is constructed; the old and new engines use different versions.

[0065] In existing technologies, when developers switch engines, they need to manually check the documentation to see what the nodes are called, for example, the old engine calls them "input" and the new engine calls them "images", and then hardcode the code to modify them.

[0066] The innovation of this application lies in its loading and identification binding. As soon as the model enters the new engine, the system automatically and thoroughly scans the metadata of the interfaces under the new engine, eliminating the need for manual verification and modification of name strings in the code, thus providing the prerequisite for subsequent automatic mapping.

[0067] This application's loading process features automatic reconnaissance and fault-tolerant escape capabilities. It doesn't just read the file into memory; instead, it automatically identifies the interface of the new environment through reflection and ensures successful loading even in the event of hardware resource anomalies, thus transforming all subsequent adaptation work from manual coding to automatic configuration.

[0068] S120. Based on the new inference engine, perform meta-information scanning on the newly loaded target model to obtain the meta-information of the input and output nodes. The meta-information includes node name, node dimension, and data type.

[0069] Traditional manual hard-coding involves developers looking up node names for a new engine; any change to a node name necessitates a recompilation and re-release of the code. This solution eliminates the need for manual maintenance of name mappings, its breakthrough lying in decoupling names from assignments. It doesn't rely on specific node name strings, but rather on maintaining synchronization between the output order of business tensors and the definition order of model input nodes. As long as the business side ensures a fixed order—relationships first, then text, then units—and the model side defines nodes in the same order during export, the system can automatically pair nodes by index. Even if node names change, as long as the order remains the same, the mapping logic in the mapping dictionary doesn't need modification.

[0070] In some embodiments, S120 includes S1201-S1204:

[0071] S1201. Perform meta-information scanning on the newly loaded target model based on the new inference engine;

[0072] Scanning the basic metadata is a technical prerequisite. The system automatically obtains the original interface description of the model in the new environment, i.e., the list of input node names, through the runtime reflection interface of the new inference engine, such as session.get_inputs(). As an example, the list of input node names is such as ["relations", "texts", "identity"].

[0073] The issue of new node names under the new engine has been resolved, providing an official standard for subsequent pairing.

[0074] S1202. If the scan determines that the target model is a multi-input model, the original input data obtained from the scan is processed to obtain multiple tensors, including relation matrix tensors, text encoding tensors and identity matrix tensors.

[0075] This step completes the material transformation from raw business data to the underlying mathematical input of the model. The raw input data is processed into specific tensors. If the scan determines that the target model is a multi-input model, such as a complex multi-input model like KIE, after confirmation, the raw images or text are not directly input. Instead, the heterogeneous data produced by upstream detection must be processed into three specific mathematical tensors that the model can handle. These three specific mathematical tensors are the relation matrix tensor, the text encoding tensor, and the identity matrix tensor.

[0076] The relation matrix tensor is generated through geometric calculations from the text box coordinates output by the detection model, encoding the spatial topological relationships between text elements. The text encoding tensor is generated from the string output by the recognition model, transformed into a numerical matrix through character dictionary mapping and fixed-length padding. The identity matrix tensor is dynamically constructed based on code and used by graph neural networks to compute the features of individual nodes.

[0077] S1203. Based on the relation matrix tensor, the text encoding tensor, and the identity matrix tensor, construct an ordered external input list, wherein the external input list represents the business processing logic;

[0078] An ordered external input list is constructed. Multiple processed tensors are assembled into an ordered NumPy tensor array list strictly according to a fixed order in the business preprocessing logic. As an example, the external input list might look like `[relations_tensor, texts_tensor, eyes_tensor]`. This order is determined by the business code, not the model. The external input list serves as a crucial bridge connecting the business side and the model side.

[0079] S1204. Map the scanned input node names to the tensors in the external input list to construct a mapping dictionary. The mapping dictionary includes metadata about the input and output nodes, including node name, node dimension, and data type.

[0080] A mapping dictionary is constructed based on the index position. The list of node names obtained by scanning on the model side is matched one by one with the list of external inputs on the business side through the index position to construct the final input dictionary.

[0081] As an example, the model-side scan yields input_names = ["relations", "texts", "identity"] (index 0, 1, 2), the business-side list is: [relations_tensor, texts_tensor, eyes_tensor] (index 0, 1, 2), and the system automatically pairs them as: {"relations": relations_tensor, "texts": texts_tensor, "identity": eyes_tensor}.

[0082] S130. Using the preset probe pseudo data as input data, perform the first inference detection on the target model based on the new inference engine to obtain the new output node information under the engine migration.

[0083] Although S120 has obtained the static metadata of the output node through reflection, such as the officially defined output shape as (Batch, Seq, Vocab), different inference engines may arbitrarily compress or adjust the dimensions at runtime in order to optimize performance.

[0084] As an example, when Batch=1, the TensorRT engine might compress (1, 512, 30522) into (512, 30522), directly changing the batch dimension. If post-processing code relies solely on the static metadata of S120, inconsistent recognition dimensions will cause decoding to crash. Therefore, this application does not completely depend on the configuration documentation, but instead verifies the truth through practice, selecting a real test run and adopting the actual results.

[0085] In some embodiments, S130 includes S1301-S1303:

[0086] S1301. Generate an image pseudo-tensor that meets the input requirements of the target model;

[0087] Generating probe pseudo-data: Based on the input node metadata scanned by S120, a set of pseudo-data conforming to the interface specification is automatically constructed. This probe pseudo-data is typically a tensor of all zeros or random numbers. The probe pseudo-data only cares about whether the shape and type match, not whether the numerical value is correct.

[0088] S1302. Input the image pseudo tensor as probe pseudo data into the new inference engine, perform a forced forward graph calculation, and obtain the output tensor.

[0089] The pseudo tensor is formally input into the new inference engine, and the forward computation graph of the model is executed once.

[0090] In some embodiments, certain models, such as dynamic graph export, may trigger memory allocation or kernel function compilation when they are executed for the first time. This step can expose problems such as engine incompatibility and full video memory in advance, and intercept failures before formal business requests.

[0091] S1303. The output tensor is parsed to obtain new output node information under engine migration. The new output node information includes the number of outputs, dimension axis semantics, and dynamic sequence length.

[0092] The output tensor is parsed, the actual configuration is read, and the new engine returns the real output tensor object after running. It doesn't care about the values ​​within the real output tensor object; in fact, all of this data is invalid pseudo-data. Only the underlying tensor's metadata is read, which includes three types of information: the number of outputs, the semantics of the dimension axes, and the length of the dynamic sequence.

[0093] The number of outputs refers to how many output heads the detection model has. Some outputs have two outputs, logits and hidden_states, while others only output one. Reading the number of outputs can prevent logical breaks caused by only taking the first one later.

[0094] Dimensional axis semantics refers to automatically inferring the physical meaning of each dimension in the current output by combining the scanned original dimensional information. For example, if the current output is three-dimensional (512, 30522), the system will identify that the batch dimension is missing and automatically record that it needs to be padded with 1 at index 0 in the future; if the output is (512, 512, 30522), the system will compare the vocabulary size (30522) to determine that the last dimension is the category axis.

[0095] Dynamic sequence length refers to recording the sequence length dimension value in the current output and marking whether the dimension is allowed to be dynamically scaled in subsequent real inference.

[0096] Under this caching mechanism, after detection is complete, the actual output dimension arrangement rules under the new engine are written into the memory cache. This means that the extra overhead of tens of milliseconds is only incurred during the first inference; from the second inference onwards, post-processing code, such as Softmax, Argmax, and CTC decoding, will directly read the cached transpose rules and automatically complete the dimension alignment, completely unaware of the business code. Combined, the model's input and output interfaces under the new engine are automatically registered by the system, achieving zero-manual coding adaptation across the entire process.

[0097] S140. Based on the input metadata of the target model input nodes, perform adaptation processing on the preset input data. The adaptation processing includes at least one of dimensional layout transformation, normalization, channel order adjustment, and scaling.

[0098] This solution separates preprocessing logic from business logic, allowing for backend switching without modifying the business code and avoiding omissions from manual alignment. All preprocessing strategy switches (whether to transpose, normalize, scale to, or flip channels) are not written into the code logic but are calculated and derived in real-time from the scanned metadata. This means that when the model migrates to a new engine, although the metadata of the input nodes changes, the general processing code does not need to be modified; the system will automatically adjust and adapt.

[0099] After these four steps, the output is a tensor that fully conforms to the input specifications of the new inference engine. This tensor is then passed to the mapping dictionary, where the new engine performs the actual inference.

[0100] S140 includes S1401-S1404:

[0101] S1401. Using the input metadata of the target model input node as a reference, transform the layout dimension of the preset input data;

[0102] Dimensional layout transformation solves the problem of data placement order. Image data is placed in memory in two main ways: NCHW and NHWC. PyTorch commonly uses the NCHW layout, which is set according to batch, channel, height, and width; TensorRT commonly uses the NHWC layout, which is set according to batch, height, width, and channel, and is often used to optimize memory access settings.

[0103] The scanned input metadata explicitly specifies whether the dimension layout is NCHW or NHWC, and automatically performs a transpose operation on the original input data.

[0104] S1402. Normalize the input data after layout conversion;

[0105] The normalization process automatically determines whether to divide by 255. Some model files have a built-in normalization layer in the computation graph, while others require external inputs to be floating-point numbers between 0 and 1. If external normalization is performed repeatedly, it will cause the numerical range to be incorrect (becoming 0~0.003), and the model will directly fail. The system does not blindly perform normalization, but rather judges based on the scanned input data type and value range metadata: if the metadata indicates that the input requirement is float32 and the expected range is 0~1, then normalization processing by dividing by 255 is performed; if the metadata indicates that the input requirement is uint8 or the model already has a normalization node, then this step is skipped.

[0106] It solves the pain point that preprocessing strategies cannot automatically adapt to engine differences and avoids omissions during manual alignment.

[0107] S1403. Scale the normalized input data.

[0108] Based on the input dimensions in the metadata (such as height and width in [1, 3, 224, 224]), the original image is scaled to the target size. For example, it is scaled to 224x224.

[0109] It should be noted that if the scanned dimensions are dynamic, such as [-1, 3, -1, -1], the system will use the preset default values ​​or keep the original image size without forcibly scaling it, leaving room for flexibility.

[0110] S1404. Automatically match and adjust the channel order to obtain input data that matches the target model.

[0111] Automatic matching and adjustment of channel order resolves the RGB / BGR color difference trap problem.

[0112] OpenCV reads images in BGR order by default, while model training is usually based on RGB. This is often overlooked when writing code manually, leading to an inexplicable decrease in inference accuracy. The system automatically performs channel flipping based on the model's metadata and the source of the current input data. If the source data and model requirements are consistent, no operation is performed.

[0113] S150. Perform deep reasoning on the adapted input data based on the new inference engine, and output the reasoning result in the output mode indicated by the output node information.

[0114] Existing technologies use line-by-line inference, which is simple but has poor throughput and extremely low GPU utilization; processing 10 boxes requires running the model 10 times. If the input is insufficient, it is directly padded before inference, but no false positives are removed, resulting in the output containing a large amount of garbage data filled in. Post-processing requires manual judgment to determine which results are valid.

[0115] This solution decouples dynamic logical batches from fixed physical batches. Without modifying the model structure, the model retains a fixed B input, allowing it to seamlessly process any number of variable-length inputs. Validation data shows a 3-5x increase in throughput.

[0116] This solution addresses the inefficiency of variable-length input processing and the low GPU utilization of sequential inference. It can adapt to variable-length inputs with fixed batch sizes and is a general dynamic batch filling mechanism.

[0117] S150 includes S1501-S1504:

[0118] S1501. Based on the new inference engine, batch-divide the adapted input data;

[0119] In a cascaded system, the number of outputs N from upstream models, such as text detection, is dynamically changing. For example, an image might detect 5 text boxes or 15. However, to maximize GPU utilization, the inference engine typically requires a fixed batch size B, such as B=8. The system groups the N input samples according to the preset batch size B, generating N / B) batches.

[0120] S1501 includes steps A1-A2:

[0121] A1. Extract batch size from the first input dimension based on the new inference engine;

[0122] The batch size value B is extracted from the first input tensor scanned, typically with dimension shape[0]. If the dimension in the metadata is a fixed value (e.g., [8, 3, 224, 224]), then B is always equal to 8. If the dimension in the metadata is dynamic (e.g., [-1, 3, 224, 224]), the system will check the default batch value preset in the initialization configuration file (e.g., B=4 is configured). Because the new engine has not yet received real data, it cannot infer the batch size from an empty tensor, so it must rely on the configuration as a fallback.

[0123] It does not change the static graph structure inside the model; it simply reads out the data that is fixed in the model and hands it over to the scheduler for use.

[0124] A2. Divide the adapted input data into batches based on the batch size.

[0125] Once the determined B value is obtained, the actual segmentation of the N input samples to be inferred begins.

[0126] S1502. If the last batch is insufficient during partitioning, it will be filled with zero values.

[0127] As an example, if N=10 and B=8, the dataset is divided into batch 1 (8 samples) and batch 2 (2 samples). For the last batch with fewer than B samples, batch 2 has only 2 samples, which are automatically padded to B samples with zero-valued tensors. This padding does not change the model weights or affect the features of the real data; it is purely to meet the inference engine's strict requirements on tensor shape.

[0128] S1503. Perform deep reasoning on the multi-batch input data to obtain effective reasoning results, where the reasoning results of the last batch containing padding data are removed.

[0129] Batch inference is performed and false positives are removed. All padded batches are then sequentially fed into a new inference engine for deep inference. For the last batch, although the inference engine outputs B results, the system strictly retains only the first M valid results. As an example, M = the number of true samples in the last batch; invalid inference results generated by zero-padding are discarded.

[0130] If these invalid results are not discarded, subsequent post-processing will parse out a large number of garbled characters or empty strings, polluting the final output.

[0131] S1504. The valid reasoning results are concatenated and output in the original order to obtain reasoning results that conform to the output method indicated by the output node information.

[0132] All valid results extracted from each batch are concatenated into a complete result list, strictly following the original input order before grouping. This ensures the consistency of the data flow throughout the cascaded system.

[0133] For example, if the detection model outputs 10 bounding boxes, after batch processing by the recognition model, the 10 recognized texts will still be arranged in the order of boxes 1 to 10, and will not be disrupted due to batch segmentation, thus ensuring a strict correspondence with the coordinates of the upstream bounding boxes.

[0134] In some embodiments, S150 further includes S1505-S1508:

[0135] When downgrading and switching engines, although the hardware or backend is changed, the cached input node names and cached output dimension rules remain valid because the same model is being switched to a different backend. This means that during the downgrading / recovery process, no manual intervention or secondary scanning is required, achieving instantaneous and seamless switching.

[0136] Instead of relying on external maintenance personnel to manually restart or switch traffic, it achieves dual protection of "fault tolerance during loading" and "self-healing at runtime" through built-in monitoring, circuit breaking, detection, and back-switch logic.

[0137] S1505. Call the new inference engine to perform deep inference on the adapted input data.

[0138] Ideally, the system would call the highest priority inference engine to perform actual inference on the processed batch of data and return the results.

[0139] In some embodiments, an automatic engine switching step is set to handle abnormal states. When the main engine triggers a preset disable condition, it will not directly report an error and crash, but will instead perform a two-step action of identification and degradation.

[0140] S1506. If the inference engine being invoked meets the preset engine disabling conditions, then based on the preset priority sorting table for engines, it will be downgraded and switched to the next backup engine. The engine disabling conditions include at least one of engine failure, engine timeout, resource exhaustion, and cost overrun.

[0141] The preset disabling conditions include at least one of the following: engine failure, engine timeout, resource shortage, and cost overrun.

[0142] If an anomaly is detected by real-time monitoring, for example, if three consecutive inference timeouts exceed the threshold, or if NVML detects a video memory OOM (Out of Memory) error.

[0143] Based on the provider priority sorting table configured during system initialization, the system automatically switches the current request and subsequent new requests to the next available backup engine.

[0144] Downgrading involves changing the engine type, such as from TensorRT to ONNX Runtime, or changing the model precision, such as from FP16 to INT8. The entire process is completed in milliseconds, and upper-layer business applications, such as clients calling the API, are completely unaware of it, perhaps only noticing a slight slowdown in response speed, but the service remains uninterrupted.

[0145] S1507. Periodically detect the superior engine of the backup engine. If the superior engine meets the preset engine recovery conditions, switch back to the superior engine.

[0146] After the system is downgraded to the backup engine, it will not remain in a low-performance state indefinitely, but will start a background daemon coroutine / thread.

[0147] Using an exponential backoff strategy such as 10s, 20s, and 40s, lightweight probe requests are periodically sent to the disabled parent priority engine. These are typically a simplified model or a single matrix operation. When the probe successfully reaches a preset threshold consecutively, such as 5 consecutive successes (success rate > 95%), the main engine is considered to have recovered, and the process switches back to the parent priority engine.

[0148] S1508. Perform deep reasoning on the adapted input data again based on the superior priority engine, and output the reasoning result in the output mode indicated by the output node information.

[0149] Once the recovery of the parent engine is confirmed, the system performs a smooth rollback. To prevent the massive influx of requests during recovery from overwhelming the newly recovered main engine again, 1% to 5% of the traffic is initially switched back as a test. If no issues are found, all traffic is gradually switched back to the high-priority engine. At this point, the system re-performs deep inference on the subsequent adaptation data based on the recovered parent engine and outputs the results according to the cached output node information, returning to the optimal state.

[0150] This application provides a method, apparatus, computer device, and storage medium for model migration across inference engines. The method includes: in response to a model migration instruction, loading a target model onto a new local inference engine, the new inference engine being different from the original inference engine of the target model, the target model being used for information recognition and extraction in document understanding; performing a metadata scan on the newly loaded target model based on the new inference engine to obtain metadata of input and output nodes, the metadata including node name, node dimension, and data type; using preset probe pseudo-data as input data, performing an initial inference detection on the target model based on the new inference engine to obtain new output node information under engine migration; performing adaptation processing on preset input data based on the input metadata of the target model's input nodes, the adaptation processing including at least one of dimension layout transformation, normalization, channel order adjustment, and scaling; performing deep inference on the adapted input data based on the new inference engine, and outputting the inference result in the output mode indicated by the output node information. This application does not employ manual hard-coding for engine adaptation, nor does it attempt to modify the model during loading. Instead, as soon as the model enters a new engine, the system automatically and thoroughly scans the metadata of the interfaces under the new engine, passively receiving the input metadata returned by the new engine. All processing strategies are not written into the code logic but are calculated and derived in real time from the scanned metadata. Regardless of the specifications of the new engine, the system can adapt adaptively, demonstrating extremely strong compatibility. This solves the technical problem in existing technologies where manual hard-coding for deep learning model adaptation across inference engines yields poor adaptation results.

[0151] Figure 2 This is a schematic block diagram of a model migration device across inference engines provided in an embodiment of this application. Figure 2As shown, corresponding to the above-described cross-inference engine model migration method, this application also provides a cross-inference engine model migration apparatus 600. This cross-inference engine model migration apparatus 600 includes a unit for executing the above-described cross-inference engine model migration method, and can be configured in a terminal such as a desktop computer, tablet computer, or laptop computer. Specifically, please refer to... Figure 2 The cross-inference engine model transfer device 600 includes a model loading unit 601, a meta-information scanning unit 602, a first-inference detection unit 603, an input adaptation unit 604, and a deep inference unit 605, wherein:

[0152] The loading model unit 601 is used to load the target model onto a new local inference engine in response to a model migration instruction. The new inference engine is different from the original inference engine of the target model. The target model is used for information recognition and extraction in document understanding.

[0153] Meta-information scanning unit 602 is used to perform meta-information scanning on the newly loaded target model based on the new inference engine to obtain the meta-information of input and output nodes. The meta-information includes node name, node dimension and data type.

[0154] The first inference detection unit 603 is used to take the preset probe pseudo data as input data and perform the first inference detection on the target model based on the new inference engine to obtain the new output node information under the engine migration.

[0155] The input adaptation unit 604 is used to perform adaptation processing on preset input data based on the input metadata of the target model input node. The adaptation processing includes at least one of dimension layout transformation, normalization, channel order adjustment and scaling.

[0156] The deep inference unit 605 is used to perform deep inference on the adapted input data based on the new inference engine, and output the inference results in the output mode indicated by the output node information.

[0157] In some embodiments, the metadata scanning unit 602 performs metadata scanning on the newly loaded target model based on the new inference engine to obtain the metadata of the input and output nodes, specifically for:

[0158] Meta-information scanning is performed on the newly loaded target model based on the new inference engine;

[0159] If the scan determines that the target model is a multi-input model, the original input data obtained from the scan is processed to obtain multiple tensors, including relation matrix tensors, text encoding tensors, and identity matrix tensors.

[0160] Based on the relation matrix tensor, the text encoding tensor, and the identity matrix tensor, an ordered external input list is constructed, which represents the business processing logic.

[0161] The scanned input node names are mapped to tensors in the external input list to construct a mapping dictionary. The mapping dictionary includes meta-information of input and output nodes, including node name, node dimension, and data type.

[0162] In some embodiments, the initial inference detection unit 603, when performing initial inference detection on the target model based on a new inference engine using preset probe pseudo data as input data, obtains new output node information under engine migration, specifically used for:

[0163] Generate image pseudo-tensors that meet the input requirements of the target model;

[0164] The image pseudo tensor is input as probe pseudo data into the new inference engine, and a forced forward graph calculation is performed to obtain the output tensor.

[0165] The output tensor is parsed to obtain new output node information under engine migration. The new output node information includes the number of outputs, dimension axis semantics, and dynamic sequence length.

[0166] In some embodiments, the input adaptation unit 604 performs adaptation processing on preset input data based on the input metadata of the target model input nodes, specifically for:

[0167] Using the input metadata of the target model's input nodes as a reference, the layout dimensions of the preset input data are transformed;

[0168] Normalize the input data after layout transformation;

[0169] Scaling the normalized input data;

[0170] The channel order is automatically matched and adjusted to obtain input data that is compatible with the target model.

[0171] In some embodiments, the deep inference unit 605 performs deep inference on the adapted input data based on the new inference engine and outputs the inference result in the output manner indicated by the output node information, specifically for:

[0172] The new inference engine is used to batch-divide the adapted input data.

[0173] If the last batch is insufficient during partitioning, it will be filled with zero values;

[0174] The input data from the multiple batches is subjected to deep inference batch by batch to obtain effective inference results, of which the inference results of the last batch containing padding data are removed.

[0175] The valid reasoning results are concatenated and output in their original order to obtain reasoning results that conform to the output method indicated by the output node information.

[0176] In some embodiments, the deep inference unit 605, in performing deep inference on the adapted input data based on the new inference engine and outputting the inference result in the output manner indicated by the output node information, is further specifically used for:

[0177] The new inference engine is invoked to perform deep inference on the adapted input data.

[0178] If the inference engine being invoked meets the preset engine disabling conditions, it will be downgraded to the next backup engine based on the preset priority ranking table for engines. The engine disabling conditions include at least one of engine failure, engine timeout, resource exhaustion, and cost overrun.

[0179] Periodically detect the primary engine of the backup engine. If the primary engine meets the preset engine recovery conditions, switch back to the primary engine.

[0180] The input data after adaptation is re-inferred based on the superior priority engine, and the inference results are output in the output mode indicated by the output node information.

[0181] In some embodiments, batch partitioning of the adapted input data based on the new inference engine includes:

[0182] The batch size is extracted from the first input dimension based on the new inference engine;

[0183] The adapted input data is divided into batches based on the batch size.

[0184] In summary, the cross-inference engine model migration device 600 in this embodiment loads a target model onto a new local inference engine in response to a model migration command. The new inference engine is different from the original inference engine of the target model. The target model is used for information recognition and extraction in document understanding. Based on the new inference engine, the newly loaded target model is scanned for metadata to obtain the metadata of the input and output nodes. The metadata includes node name, node dimension, and data type. Preset probe pseudo-data is used as input data, and the target model is subjected to initial inference detection based on the new inference engine to obtain new output node information under engine migration. Based on the input metadata of the target model's input nodes, the preset input data is adapted. The adaptation process includes at least one of dimension layout transformation, normalization, channel order adjustment, and scaling. Based on the new inference engine, deep inference is performed on the adapted input data, and the inference result is output in the output mode indicated by the output node information. This application does not employ manual hard-coding for engine adaptation, nor does it attempt to modify the model during loading. Instead, as soon as the model enters a new engine, the system automatically and thoroughly scans the metadata of the interfaces under the new engine, passively receiving the input metadata returned by the new engine. All processing strategies are not written into the code logic but are calculated and derived in real time from the scanned metadata. Regardless of the specifications of the new engine, the system can adapt adaptively, demonstrating extremely strong compatibility. This solves the technical problem in existing technologies where manual hard-coding for deep learning model adaptation across inference engines yields poor adaptation results.

[0185] It should be noted that those skilled in the art can clearly understand that the specific implementation process of the above-mentioned cross-inference engine model migration device and each unit can be referred to the corresponding description in the foregoing method embodiments. For the sake of convenience and brevity, it will not be repeated here.

[0186] The aforementioned model transfer mechanism across inference engines can be implemented as a computer program, which can, for example... Figure 3 It runs on the computer device shown.

[0187] Please see Figure 3 , Figure 3 This is a schematic block diagram of a computer device 700 provided in an embodiment of this application. The computer device 700 can be a terminal or a server. The terminal can be an electronic device with communication functions, such as a smartphone, tablet, laptop, desktop computer, personal digital assistant, or wearable device. The server can be a standalone server or a server cluster composed of multiple servers.

[0188] See Figure 3The computer device 700 includes a processor 702, a memory, and a network interface 705 connected via a system bus 701. The memory may include a non-volatile storage medium 703 and internal memory 704.

[0189] The non-volatile storage medium 703 may store an operating system 7031 and a computer program 7032. The computer program 7032 includes program instructions that, when executed, cause the processor 702 to perform a model migration method across inference engines.

[0190] The processor 702 provides computing and control capabilities to support the operation of the entire computer device 700.

[0191] The internal memory 704 provides an environment for the execution of the computer program 7032 in the non-volatile storage medium 703. When the computer program 7032 is executed by the processor 702, the processor 702 can execute a model migration method across inference engines.

[0192] This network interface 705 is used for network communication with other devices. Those skilled in the art will understand that... Figure 3 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device 700 to which the present application is applied. The specific computer device 700 may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0193] The processor 702 is used to run the computer program 7032 stored in the memory to perform the following steps:

[0194] In response to the model migration instruction, the target model is loaded onto a new local inference engine. The new inference engine is different from the original inference engine of the target model. The target model is used for information recognition and extraction in document understanding.

[0195] Based on the new inference engine, the newly loaded target model is scanned for metadata to obtain the metadata of the input and output nodes. The metadata includes the node name, node dimension, and data type.

[0196] Using the preset probe pseudo data as input data, the target model is subjected to the first inference detection based on the new inference engine to obtain the new output node information under the engine migration.

[0197] Based on the input metadata of the target model input nodes, the preset input data is adapted. The adaptation process includes at least one of dimensional layout transformation, normalization, channel order adjustment, and scaling.

[0198] The new inference engine performs deep inference on the adapted input data and outputs the inference results in the manner indicated by the output node information.

[0199] It should be understood that in the embodiments of this application, the processor 702 may be a central processing unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0200] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program includes program instructions and can be stored in a storage medium, which is a computer-readable storage medium. The program instructions are executed by at least one processor in the computer system to implement the process steps of the embodiments of the above methods.

[0201] It should be noted that the specific embodiments are merely illustrative examples intended to aid in understanding the technical solutions of the present invention, and are not intended to limit the scope of protection of the present invention. The scope of protection of this application is determined by the contents of the claims. Specific embodiments, parameter ranges, or technical features described in the specification should not be construed as undue limitation or expansion of the scope of protection of the claims. The technical effects described in the specification are only used to illustrate the innovation of the present invention. Any technical solution that does not simultaneously possess all the technical features of the present invention, even if it claims to solve the same technical problem, does not fall within the scope of protection of the present invention.

[0202] Therefore, this application also provides a storage medium. This storage medium can be a computer-readable storage medium. The storage medium stores a computer program, wherein the computer program includes program instructions. When executed by a processor, the program instructions cause the processor to perform the following steps:

[0203] In response to the model migration instruction, the target model is loaded onto a new local inference engine. The new inference engine is different from the original inference engine of the target model. The target model is used for information recognition and extraction in document understanding.

[0204] Based on the new inference engine, the newly loaded target model is scanned for metadata to obtain the metadata of the input and output nodes. The metadata includes the node name, node dimension, and data type.

[0205] Using the preset probe pseudo data as input data, the target model is subjected to the first inference detection based on the new inference engine to obtain the new output node information under the engine migration.

[0206] Based on the input metadata of the target model input nodes, the preset input data is adapted. The adaptation process includes at least one of dimensional layout transformation, normalization, channel order adjustment, and scaling.

[0207] The new inference engine performs deep inference on the adapted input data and outputs the inference results in the manner indicated by the output node information.

[0208] The storage medium can be any computer-readable storage medium that can store program code, such as a USB flash drive, external hard drive, read-only memory (ROM), magnetic disk, or optical disk.

[0209] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this application.

[0210] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For example, the division of each unit is merely a logical functional division, and there may be other division methods in actual implementation. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed.

[0211] The steps in the methods of this application embodiment can be adjusted, merged, or deleted according to actual needs. The units in the apparatus of this application embodiment can be merged, divided, or deleted according to actual needs. Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0212] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, terminal, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application.

[0213] It should be noted that any software tools or components not belonging to this company appearing in the embodiments of this application are merely illustrative examples and do not represent actual use. All user personal information involved in the embodiments of this application is authorized (with the knowledge and consent) by the relevant parties or fully authorized by all parties, and the executing entity may obtain it through various legal and compliant means. The collection, storage, use, processing, transmission, provision, and disclosure of the information, data, and signals involved all comply with relevant laws and regulations and do not violate public order and good morals.

[0214] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A model transfer method across inference engines, characterized in that, The method includes: In response to the model migration instruction, the target model is loaded onto a new local inference engine. The new inference engine is different from the original inference engine of the target model. The target model is used for information recognition and extraction in document understanding. Based on the new inference engine, the newly loaded target model is scanned for metadata to obtain the metadata of the input and output nodes. The metadata includes the node name, node dimension, and data type. Using the preset probe pseudo data as input data, the target model is subjected to the first inference detection based on the new inference engine to obtain the new output node information under the engine migration. Based on the input metadata of the target model input nodes, the preset input data is adapted. The adaptation process includes at least one of dimensional layout transformation, normalization, channel order adjustment, and scaling. The new inference engine performs deep inference on the adapted input data and outputs the inference results in the manner indicated by the output node information.

2. The method according to claim 1, characterized in that, Based on the new inference engine, a metadata scan is performed on the newly loaded target model to obtain the metadata of the input and output nodes, including: Meta-information scanning is performed on the newly loaded target model based on the new inference engine; If the scan determines that the target model is a multi-input model, the original input data obtained from the scan is processed to obtain multiple tensors, including relation matrix tensors, text encoding tensors, and identity matrix tensors. Based on the relation matrix tensor, the text encoding tensor, and the identity matrix tensor, an ordered external input list is constructed, which represents the business processing logic. The scanned input node names are mapped to tensors in the external input list to construct a mapping dictionary. The mapping dictionary includes meta-information of input and output nodes, including node name, node dimension, and data type.

3. The method according to claim 1, characterized in that, Using pre-set probe pseudo-data as input data, the target model is subjected to initial inference detection based on the new inference engine, resulting in new output node information under engine migration, including: Generate image pseudo-tensors that meet the input requirements of the target model; The image pseudo tensor is input as probe pseudo data into the new inference engine, and a forced forward graph calculation is performed to obtain the output tensor. The output tensor is parsed to obtain new output node information under engine migration. The new output node information includes the number of outputs, dimension axis semantics, and dynamic sequence length.

4. The method according to claim 1, characterized in that, Based on the input metadata of the target model's input nodes, the preset input data is adapted, including: Using the input metadata of the target model's input nodes as a reference, the layout dimensions of the preset input data are transformed; Normalize the input data after layout transformation; Scaling the normalized input data; The channel order is automatically matched and adjusted to obtain input data that is compatible with the target model.

5. The method according to claim 1, characterized in that, Based on the new inference engine, deep inference is performed on the adapted input data, and the inference results are output in the output manner indicated by the output node information. The method also includes: The new inference engine is invoked to perform deep inference on the adapted input data. If the inference engine being invoked meets the preset engine disabling conditions, it will be downgraded to the next backup engine based on the preset priority ranking table for engines. The engine disabling conditions include at least one of engine failure, engine timeout, resource exhaustion, and cost overrun. Periodically detect the primary engine of the backup engine. If the primary engine meets the preset engine recovery conditions, switch back to the primary engine. The adapted input data is re-inferred based on the superior priority engine, and the inference results are output in the manner indicated by the output node information.

6. The method according to claim 1, characterized in that, Based on the new inference engine, deep inference is performed on the adapted input data, and the inference results are output in the output manner indicated by the output node information, including: The new inference engine is used to batch-divide the adapted input data. If the last batch is insufficient during partitioning, it will be filled with zero values; The input data from the multiple batches is subjected to deep inference batch by batch to obtain effective inference results, of which the inference results of the last batch containing padding data are removed. The valid reasoning results are concatenated and output in their original order to obtain reasoning results that conform to the output method indicated by the output node information.

7. The method according to claim 1, characterized in that, The new inference engine performs batch partitioning on the adapted input data, including: The batch size is extracted from the first input dimension based on the new inference engine; The adapted input data is divided into batches based on the batch size.

8. A model transfer device across inference engines, characterized in that, The device includes: The loading model unit is used to load the target model onto a new local inference engine in response to a model migration instruction. The new inference engine is different from the original inference engine of the target model. The target model is used for information recognition and extraction in document understanding. The meta-information scanning unit is used to scan the newly loaded target model based on the new inference engine to obtain the meta-information of the input and output nodes. The meta-information includes the node name, node dimension, and data type. The initial inference detection unit is used to take the preset probe pseudo data as input data and perform the initial inference detection on the target model based on the new inference engine to obtain the new output node information under the engine migration. An input adaptation unit is used to adapt preset input data based on the input metadata of the target model input nodes. The adaptation process includes at least one of dimensional layout transformation, normalization, channel order adjustment, and scaling. The deep inference unit is used to perform deep inference on the adapted input data based on the new inference engine, and output the inference results in the output manner indicated by the output node information.

9. A computer device for model transfer across inference engines, characterized in that, The method includes a memory, a processor, and a cross-inference engine model migration program stored in the memory and executable on the processor, wherein the processor executes the cross-inference engine model migration program to implement the steps of the cross-inference engine model migration method according to any one of claims 1 to 7.

10. A storage medium, characterized in that, The medium stores a program for implementing a model migration method across inference engines, which is executed by a processor to implement the steps of the model migration method across inference engines as described in any one of claims 1 to 7.

Images