Data processing method, device and equipment based on machine learning model inference

Abstract

The method comprises the following steps: obtaining platform device information according to a preset configuration file, performing initialization processing on the platform device information, and determining a cache space; preprocessing original input data to be processed to obtain preprocessed data stored in the cache space; performing model inference processing on the preprocessed data to obtain model inference data stored in the cache space; performing post-processing on the model inference data to obtain inference data stored in the cache space, and storing a target pointer corresponding to a third position of the inference data in the cache space in a target data dictionary. The data in the inference whole process is processed by means of the pointer and the data dictionary, which not only improves the data processing efficiency, but also improves the model inference efficiency.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, and in particular to a data processing method, apparatus, and device based on machine learning model reasoning. Background Technology

[0002] With the development of big data and artificial intelligence technologies, an increasing number of business scenarios, such as financial risk control, online advertising, product recommendation, and smart cities, are employing machine learning techniques to improve service quality and intelligent decision-making. For specific tasks, after training the model in a designated training environment, it needs to be packaged and deployed as an online inference service. Users can then utilize this inference service when using the same runtime environment as the training environment.

[0003] In related technologies, model inference is typically performed on dedicated inference devices. However, each dedicated inference device has different inference capabilities. Therefore, when multiple dedicated inference devices are executing the entire model inference process, data corresponding to multiple inference stages will inevitably be generated. This large amount of data leads to low model inference efficiency.

[0004] Therefore, improving the efficiency of model inference is a pressing technical problem that needs to be solved. Summary of the Invention

[0005] This disclosure provides a data processing method, apparatus, and device based on machine learning model inference. It processes data throughout the inference process using pointers and a data dictionary, which not only improves data processing efficiency but also enhances model inference efficiency.

[0006] In a first aspect, this disclosure provides a data processing method based on machine learning model inference, including:

[0007] The platform device information is obtained according to the preset configuration file, the platform device information is initialized, and the cache space corresponding to the platform device information is determined.

[0008] The original input data to be processed is preprocessed to obtain preprocessed data stored in the cache space, and the input pointer corresponding to the first position of the preprocessed data in the cache space is stored in the target data dictionary;

[0009] The preprocessed data obtained from the cache space according to the input pointer is subjected to model inference processing to obtain model inference data, and the output pointer corresponding to the second position of the model inference data in the cache space is stored in the target data dictionary;

[0010] The model inference data obtained from the cache space according to the output pointer is post-processed to obtain inference data stored in the cache space, and the target pointer corresponding to the third position of the inference data in the cache space is stored in the target data dictionary.

[0011] Preferably, according to the data processing method based on machine learning model inference provided in this disclosure, the configuration file includes at least: configuration platform parameters and model path parameters;

[0012] The step of obtaining platform device information according to a preset configuration file, initializing the platform device information, and determining the cache space corresponding to the platform device information includes:

[0013] The backend interfaces are selected based on the configured platform parameters;

[0014] Based on the model path parameters, the preset model file is processed to obtain the model information generated in the inference device corresponding to the backend interface;

[0015] Based on the model information, the cache space is created in the inference device corresponding to the platform device information.

[0016] Preferably, in the data processing method based on machine learning model inference provided in this disclosure, the step of filtering backend interfaces according to the configuration platform parameters includes:

[0017] Create a model class corresponding to the preset inference interface;

[0018] Based on the model class and the configuration platform parameters, determine the backend interface that has been registered in the configuration file.

[0019] Preferably, according to the data processing method based on machine learning model inference provided in this disclosure, the step of processing a preset model file based on the model path parameters to obtain model information generated in the inference device corresponding to the backend interface includes:

[0020] The model file stored in the host memory is obtained based on the model path parameters;

[0021] The model file in the host memory is transferred to the inference device corresponding to the backend interface, so that the inference device can parse the model file and obtain the model information.

[0022] Preferably, according to the data processing method based on machine learning model inference provided in this disclosure, the step before preprocessing the original input data to be processed to obtain preprocessed data stored in the cache space includes:

[0023] The target data dictionary is obtained by initializing the preset data dictionary.

[0024] The original input data is stored in the host memory, and the original data pointer corresponding to the storage location of the original input data in the host memory is stored in the target data dictionary, so as to retrieve the original input data from the host memory using the original data pointer.

[0025] Preferably, according to the data processing method based on machine learning model inference provided in this disclosure, the step of performing model inference processing on the preprocessed data obtained from the cache space according to the input pointer to obtain model inference data includes:

[0026] Obtain the execution function from the inference interface;

[0027] The inference function in the model class is loaded based on the running function, and the preprocessed data obtained by the input pointer is processed by model inference to obtain the model inference data.

[0028] Secondly, this disclosure also provides a data processing apparatus based on machine learning model inference, comprising:

[0029] An initialization module is used to obtain platform device information according to a preset configuration file, perform initialization processing on the platform device information, and determine the cache space corresponding to the platform device information.

[0030] The preprocessing module is used to preprocess the raw input data to be processed, obtain preprocessed data stored in the cache space, and store the input pointer corresponding to the first position of the preprocessed data in the cache space into the target data dictionary;

[0031] The model inference module is used to perform model inference processing on the preprocessed data obtained from the cache space according to the input pointer to obtain model inference data, and store the output pointer corresponding to the second position of the model inference data in the cache space into the target data dictionary;

[0032] The post-processing module is used to post-process the model inference data obtained from the cache space according to the output pointer, to obtain inference data stored in the cache space, and to store the target pointer corresponding to the third position of the inference data in the cache space into the target data dictionary.

[0033] Thirdly, this disclosure also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of any of the above-described data processing methods based on machine learning model inference.

[0034] Fourthly, this disclosure also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the data processing method based on machine learning model inference as described above.

[0035] Fifthly, this disclosure also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the data processing method based on machine learning model inference as described above.

[0036] This disclosure provides a data processing method, apparatus, and device based on machine learning model inference. The method involves: acquiring platform device information according to a preset configuration file; initializing the platform device information to determine a cache space corresponding to the platform device information; preprocessing the raw input data to be processed to obtain preprocessed data stored in the cache space; storing the input pointer corresponding to the first position of the preprocessed data in the cache space into a target data dictionary; performing model inference processing on the preprocessed data obtained from the cache space based on the input pointer to obtain model inference data; storing the output pointer corresponding to the model inference data at the second position in the cache space into the target data dictionary; and post-processing the model inference data obtained from the cache space based on the output pointer to obtain inference data stored in the cache space; storing the target pointer corresponding to the inference data at the third position in the cache space into the target data dictionary. By processing data throughout the inference process using pointers and a data dictionary, both data processing efficiency and model inference efficiency are improved. Attached Figure Description

[0037] To more clearly illustrate the technical solutions in this disclosure or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0038] Figure 1 This is one of the flowcharts illustrating a data processing method based on machine learning model inference provided in this disclosure embodiment;

[0039] Figure 2 This is a schematic diagram of the interface and model class of the standard inference interface in this embodiment of the disclosure;

[0040] Figure 3 This is an embodiment of the present disclosure. Figure 1 One of the flowcharts for step S100;

[0041] Figure 4 This is an embodiment of the present disclosure. Figure 3 One of the flowcharts for step S310;

[0042] Figure 5 This is an embodiment of the present disclosure. Figure 3 One of the flowcharts for step S320;

[0043] Figure 6 This is a schematic diagram of the structure of a data processing device based on machine learning model inference provided in an embodiment of this disclosure;

[0044] Figure 7 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this disclosure. Detailed Implementation

[0045] To make the objectives, technical solutions, and advantages of this disclosure clearer, the technical solutions of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, not all embodiments. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0046] The Standard Inference Interface (InferDK) encapsulates the underlying inference capabilities provided by the backend. It is used to load and initialize models, control data transmission between devices and hosts or between devices, including the model driver execution interface. It ensures thread safety for model inference and provides access to basic model parameters. By encapsulating the Standard Inference Interface, the upper-level processes are unaware of the underlying hardware when performing model inference operations. Furthermore, by extending the backend of the Standard Inference Interface, it can support more model inference capabilities. The upper-level processes are the logical layer above the Standard Inference Interface's logical layer, corresponding to the service and process framework.

[0047] Backend: Typically, the backend uses the native inference runtime API provided by the hardware device manufacturer for encapsulation.

[0048] Image Processing Interface (ProcessDK): This interface unifies and encapsulates the processing capabilities of different hardware devices, including image processing units. This shields the upper layers from the various interfaces provided by each hardware processing unit. For image data input inference, it provides atomic capabilities such as typical image matting and scaling.

[0049] Data dictionary: A data dictionary is a directory that users can access to record database and application metadata. In this embodiment of the disclosure, it is used to store multiple data pointer information to quickly retrieve data in the cache space.

[0050] Pointer: In computer science, a pointer is an object in a programming language that uses an address to directly point to a value stored in another location in computer memory. In this embodiment of the disclosure, it is a value pointing to the location information of data stored in the cache space.

[0051] Cache space: In the computer field, cache space actually refers to cache registers, which temporarily store data sent from peripherals so that the processor can retrieve it, or temporarily store data sent by the processor to peripherals (back-end inference devices). With cache space, the high-speed CPU and the slow-speed peripherals can coordinate and buffer each other, achieving synchronization of data transmission.

[0052] The following is combined with Figures 1-7 This disclosure describes a data processing method, apparatus, and device based on machine learning model inference, which can process data throughout the inference process using pointers and a data dictionary, thereby improving both data processing efficiency and model inference efficiency.

[0053] The specific implementation is illustrated in the following embodiments. First, a data processing method based on machine learning model inference in the embodiments of this disclosure is described.

[0054] like Figure 1 As shown, it is a schematic diagram of the implementation process of a data processing method based on machine learning model reasoning provided in an embodiment of this disclosure. A data processing method based on machine learning model reasoning may include, but is not limited to, steps S100 to S400.

[0055] S100: Obtain platform device information according to a preset configuration file, initialize the platform device information, and determine the cache space corresponding to the platform device information;

[0056] S200, preprocess the original input data to be processed to obtain preprocessed data stored in the cache space, and store the input pointer corresponding to the first position of the preprocessed data in the cache space into the target data dictionary;

[0057] S300, perform model inference processing on the preprocessed data obtained from the cache space according to the input pointer to obtain model inference data, and store the output pointer corresponding to the second position of the model inference data in the cache space into the target data dictionary;

[0058] S400, post-process the model inference data obtained from the cache space according to the output pointer to obtain inference data stored in the cache space, and store the target pointer corresponding to the third position of the inference data in the cache space into the target data dictionary.

[0059] In step S100 of some embodiments, platform device information is obtained according to a preset configuration file, the platform device information is initialized, and a cache space corresponding to the platform device information is determined. It is understood that the platform device information can be obtained through a user-defined configuration file, using either YAML or JSON configuration format.

[0060] Furthermore, YAML is a highly readable format used to express data serialization.

[0061] JSON (JavaScript Object Notation) is a lightweight data interchange format.

[0062] Further, the platform device information obtained from the configuration file is initialized to determine the cache space. Specifically, the steps include creating a model class corresponding to a preset inference interface, determining the backend interface registered in the configuration file based on the model class and the configured platform parameters, obtaining the model file stored in the host memory based on the model path parameters, transferring the model file from the host memory to the inference device corresponding to the backend interface, and using the inference device to parse the model file to obtain the model information. Finally, based on the model information, the cache space is created in the inference device corresponding to the platform device information.

[0063] It should be noted that the configuration file must include at least: platform parameters and model path parameters.

[0064] ModelPath parameter: The model repository where the model file is located, or the path where the model file is stored. It can be a model file stored in OSS cloud storage or local storage. The type is string.

[0065] Configure platform parameters: The inference backend for running the model file, such as tensorrt, cann, openvino, mnn, etc., and the type is string.

[0066] Of course, for reference Figure 2 As shown, the configuration file may also include, but is not limited to:

[0067] Model Name: This is a unique identifier for this model. It is unique within the same software development kit (SDK) and is of string type.

[0068] The memory reuse function (Use_device_buffer) determines whether to reuse the memory of the inference device. It is a boolean type.

[0069] Further, refer to Figure 2 The diagram shows the interface and model classes of the standard inference interface (InferDK).

[0070] The inference interface may include, but is not limited to, the following functions:

[0071] Run function and ModelLoader function.

[0072] The model class (ModelLoader) can contain at least, but is not limited to, the following functions:

[0073] Inference function, Transfer function, GetDims function, Register function.

[0074] A model class corresponds to multiple backend ports, and each backend port can contain at least, but is not limited to, the following functions:

[0075] Inference function, Transfer function, GetDims function, and DeviceId.

[0076] It is understandable that by calling the run function in the inference interface, the inference functions in the model class and multiple backend ports can be invoked to achieve model inference.

[0077] Transfer functions are used to transfer data between the host side and the inference device side.

[0078] Data metric functions are used to control the dimension and size of input or output data.

[0079] The registration function is used to identify whether a backend port has been registered, in order to filter out available backend ports.

[0080] Optionally, the backend object names corresponding to multiple backend ports can be TrtModelLoader, CannModelLoader, and CpuModelLoader.

[0081] In step S200 of some embodiments, the original input data to be processed is preprocessed to obtain preprocessed data stored in the cache space, and the input pointer corresponding to the first position of the preprocessed data in the cache space is stored in the target data dictionary. It can be understood that after executing step S100, which involves obtaining platform device information according to a preset configuration file, initializing the platform device information, and determining the cache space corresponding to the platform device information, the original input data to be processed is preprocessed to obtain preprocessed data. This preprocessed data is then stored in the cache space created in step S100, and the input pointer corresponding to the first position of the preprocessed data in the cache space is stored in the target data dictionary.

[0082] For example, the cache space may include, but is not limited to, a first position, a second position, a third position, and a fourth position. Preprocessed data is stored in the first position, and an input pointer pointing to the first position is stored in the target data dictionary, allowing for quick retrieval of the preprocessed data from the cache space via the input pointer in the target data dictionary.

[0083] In step S300 of some embodiments, the preprocessed data obtained from the cache space according to the input pointer is subjected to model inference processing to obtain model inference data, and the output pointer corresponding to the second position of the model inference data in the cache space is stored in the target data dictionary. It can be understood that after performing step S200 (preprocessing the original input data to be processed to obtain preprocessed data stored in the cache space, and storing the input pointer corresponding to the first position of the preprocessed data in the cache space in the target data dictionary), the preprocessed data is first quickly obtained from the cache space according to the input pointer in the target data dictionary, and then model inference processing is performed on the preprocessed data. Specifically, the preprocessed data can be asynchronously transmitted to the inference device corresponding to the standard inference interface (InferDK) to load the inference model for inference processing, thereby obtaining model inference data.

[0084] Furthermore, the model inference data is stored in the second location of the cache space, and the output pointer pointing to the second location is stored in the target data dictionary. This allows the model inference data to be quickly retrieved from the cache space through the output pointer in the target data dictionary, thereby improving the inference efficiency of the model inference.

[0085] In step S400 of some embodiments, the model inference data obtained from the cache space according to the output pointer is post-processed to obtain inference data stored in the cache space, and the target pointer corresponding to the third position of the inference data in the cache space is stored in the target data dictionary. It can be understood that after performing step S300—processing the pre-processed data obtained from the cache space according to the input pointer to obtain model inference data, and storing the output pointer corresponding to the second position of the model inference data in the cache space in the target data dictionary—the specific execution steps can be: quickly obtaining the model inference data from the cache space according to the output pointer in the target data dictionary, performing post-processing on the model inference data to obtain the inference data.

[0086] Optionally, the inference data is stored in the third location in the cache space, and the target pointer pointing to the third location is stored in the target data dictionary. This allows the inference data to be quickly retrieved from the cache space through the target pointer in the target data dictionary, thereby improving the efficiency of reusing inference data after model inference.

[0087] Optionally, the specific steps for post-processing the model inference data to obtain inference data can be to use a preset segmentation rule to segment each model inference data separately to obtain inference data, wherein the segmentation rule is to segment each model inference data into inference data of the same number as a preset quantity threshold.

[0088] In some embodiments, reference Figure 3 As shown, step S100 may also include, but is not limited to, steps S310 to S330.

[0089] S310, Filter out the backend interfaces according to the configuration platform parameters;

[0090] S320, Based on the model path parameters, the preset model file is processed to obtain the model information generated in the inference device corresponding to the backend interface;

[0091] S330, Based on the model information, create the cache space in the inference device corresponding to the platform device information.

[0092] In step S310 of some embodiments, backend interfaces are selected based on the configuration platform parameters. It can be understood that the specific steps for selecting backend interfaces based on configuration platform parameters can be: creating a model class corresponding to a preset inference interface; and determining the backend interfaces already registered in the configuration file based on the model class and the configuration platform parameters.

[0093] Optionally, the inference interface may include, but is not limited to, the standard inference interface (InferDK).

[0094] The corresponding model class is created based on the standard inference interface. Based on the configuration platform parameters in the model class, the backend interfaces that have been registered in the configuration file are determined. For example, if there are multiple backend interfaces connected to the model class, only the backend interfaces that have been registered using the registration function can be used.

[0095] In step S320 of some embodiments, a preset model file is processed based on the model path parameters to obtain model information generated in the inference device corresponding to the backend interface. It can be understood that after step S310, which filters out the backend interface based on the configuration platform parameters, the model file is processed based on the model path parameters to obtain model information.

[0096] Optionally, the model information is generated in the inference device corresponding to the backend interface. Each backend interface corresponds to one backend inference device, and each backend inference device has a pre-reserved backend interface at the factory, which can be used to invoke the inference performance of the backend inference device.

[0097] In step S330 of some embodiments, the cache space is created in the inference device corresponding to the platform device information based on the model information. It can be understood that after step S320, which processes the preset model file based on the model path parameters to obtain the model information generated in the inference device corresponding to the backend interface, the specific execution steps can be: creating a cache space in the inference device based on the model information parsed in step S320, for temporarily storing relevant data generated during the entire model inference process. This relevant data includes at least: preprocessed data, model inference data, and inference data.

[0098] Alternatively, data cached in the cache space can be transferred between host memory and inference device via a transfer function.

[0099] Furthermore, during data transmission, it is not necessary to transmit a large amount of data; only pointer information pointing to the data storage location needs to be transmitted. Since the pointer information is stored in the target data dictionary, only the target data dictionary needs to be transmitted. This reduces data transmission and enables quick retrieval of relevant data, thereby greatly improving the efficiency of model inference.

[0100] In some embodiments, reference Figure 4 As shown, step S310 may also include, but is not limited to, steps S410 to S420.

[0101] S410, Create a model class corresponding to the preset inference interface according to the preset inference interface;

[0102] S420, Based on the model class and the configuration platform parameters, determine the backend interface that has been registered in the configuration file.

[0103] In step S410 of some embodiments, a model class corresponding to the inference interface is created according to a preset inference interface. It is understood that creating a model class corresponding to the inference interface is used to determine the backend interfaces corresponding to multiple backends based on the model class.

[0104] In step S420 of some embodiments, the backend interfaces registered in the configuration file are determined based on the model class and the configuration platform parameters. It can be understood that after step S410, which creates a model class corresponding to the inference interface based on a preset inference interface, the specific execution steps may include creating a corresponding model class based on a standard inference interface, and determining the backend interfaces registered in the configuration file based on the configuration platform parameters in the model class. For example, if multiple backend interfaces are connected to the model class, only the backend interfaces that have been registered using the registration function can be used.

[0105] In some embodiments, reference Figure 5 As shown, step S320 may also include, but is not limited to, steps S510 to S520.

[0106] S510, obtain the model file stored in the host memory according to the model path parameters;

[0107] S520, the model file in the host memory is transferred to the inference device corresponding to the backend interface, so as to use the inference device to parse the model file and obtain the model information.

[0108] In step S510 of some embodiments, the model file stored in the host memory is obtained according to the model path parameters. It can be understood that after step S420 is completed, which determines the backend interface registered in the configuration file according to the model class and the configuration platform parameters, the specific execution steps can be: obtaining the model file from the cloud database or local database according to the model path adoption number, storing the model file in the host memory, and transmitting it to the inference device for parsing.

[0109] In step S520 of some embodiments, the model file in the host memory is transferred to the inference device corresponding to the backend interface, so that the inference device can parse the model file to obtain the model information. It can be understood that after executing step S510, the specific execution steps may include transferring the model file in the host memory to the inference device corresponding to the backend interface, and using the inference device to parse the model file to obtain the model information.

[0110] In some embodiments, the method prior to step S200, which involves preprocessing the raw input data to be processed to obtain preprocessed data stored in the cache space, includes:

[0111] The target data dictionary is obtained by initializing the preset data dictionary.

[0112] The original input data is stored in the host memory, and the original data pointer corresponding to the storage location of the original input data in the host memory is stored in the target data dictionary, so as to retrieve the original input data from the host memory using the original data pointer.

[0113] It is understandable that before preprocessing the raw input data to be processed to obtain the preprocessed data stored in the cache space, the preset data dictionary is first initialized to obtain the target data dictionary, which is used to store pointer information pointing to the relevant data stored in the cache space.

[0114] The raw input data is stored in the host memory, and the raw data pointer corresponding to the storage location of the raw input data in the host memory is stored in the target data dictionary. In this way, the raw input data can be quickly retrieved from the host memory using the raw data pointer, thereby speeding up the efficiency of model inference.

[0115] In some embodiments, the preprocessed data obtained from the cache space according to the input pointer is subjected to model inference processing to obtain model inference data, including:

[0116] Obtain the execution function from the inference interface;

[0117] The inference function in the model class is loaded based on the running function, and the preprocessed data obtained by the input pointer is processed by model inference to obtain the model inference data.

[0118] It is understandable that obtaining the run function in the inference interface can be done by obtaining the Run function, using the Run function to load the inference function in the model class, and then using the inference function in the model class and the inference function in the backend port to quickly obtain preprocessed data from the cache space using the input pointer. The preprocessed data is then transmitted asynchronously to the inference device corresponding to the standard inference interface (InferDK) to load the inference model for inference processing, thus obtaining the model inference data, which greatly improves the efficiency of model inference.

[0119] Preferably, in some embodiments, the standard inference interface also initializes the transfer function based on whether the device-side memory (Use_device_buffer) needs to be reused. The transfer function is also implemented according to the transfer capability interface provided by different back-end devices. If the original input data to be processed is memory data provided by a non-inference hardware device, then data transfer (H2D) is required; otherwise, memory reuse is performed directly using pointers (D2D).

[0120] Optionally, data transmission can be performed from host to device (H2D), from device to device (D2D), from device to host (D2H), or from host to host (H2H).

[0121] Preferably, for the multiple back-ends of the image processing interface (ProcessDK): since an inference device can consist of multiple dedicated chips, each dedicated chip performs a specific processing task, such as video decoding (decoding H264 video streams into images) is performed by a dedicated chip, which is more efficient than non-dedicated chips.

[0122] The image processing interface (ProcessDK) can transfer computing tasks suitable for dedicated chip processing from the host control chip to more efficient dedicated chips, achieving overall optimization of the data processing flow.

[0123] Preferably, the data processing flow includes at least: preprocessing, model inference, and post-processing. The complex inference process is controlled by the host CPU, encoding / decoding / rendering operations are handled by the encoding / decoding chip, and model inference is handled by a GPU / NPU suitable for matrix operations.

[0124] This disclosure provides a data processing method based on machine learning model inference. The method involves: obtaining platform device information according to a preset configuration file; initializing the platform device information to determine a cache space corresponding to the platform device information; preprocessing the raw input data to be processed to obtain preprocessed data stored in the cache space; storing the input pointer corresponding to the first position of the preprocessed data in the cache space into a target data dictionary; performing model inference processing on the preprocessed data obtained from the cache space based on the input pointer to obtain model inference data; storing the output pointer corresponding to the model inference data at the second position in the cache space into the target data dictionary; and post-processing the model inference data obtained from the cache space based on the output pointer to obtain inference data stored in the cache space; storing the target pointer corresponding to the inference data at the third position in the cache space into the target data dictionary. By processing data throughout the inference process using pointers and a data dictionary, both data processing efficiency and model inference efficiency are improved.

[0125] The data processing apparatus based on machine learning model inference provided in this disclosure is described below. The data processing apparatus based on machine learning model inference described below and the data processing method based on machine learning model inference described above can be referred to in correspondence.

[0126] Reference Figure 6 As shown, a data processing device based on machine learning model inference includes:

[0127] The initialization module 610 is used to obtain platform device information according to a preset configuration file, perform initialization processing on the platform device information, and determine the cache space corresponding to the platform device information.

[0128] The preprocessing module 620 is used to preprocess the raw input data to be processed, obtain preprocessed data stored in the cache space, and store the input pointer corresponding to the first position of the preprocessed data in the cache space into the target data dictionary;

[0129] The model inference module 630 is used to perform model inference processing on the preprocessed data obtained from the cache space according to the input pointer to obtain model inference data, and store the output pointer corresponding to the second position of the model inference data in the cache space into the target data dictionary;

[0130] The post-processing module 640 is used to post-process the model inference data obtained from the cache space according to the output pointer, to obtain inference data stored in the cache space, and to store the target pointer corresponding to the third position of the inference data in the cache space into the target data dictionary.

[0131] According to the data processing apparatus based on machine learning model inference provided in this disclosure, the configuration file includes at least: configuration platform parameters, model path parameters, and an initialization module 610, which is specifically used to filter out backend interfaces according to the configuration platform parameters.

[0132] Based on the model path parameters, the preset model file is processed to obtain the model information generated in the inference device corresponding to the backend interface;

[0133] Based on the model information, the cache space is created in the inference device corresponding to the platform device information.

[0134] According to the data processing device based on machine learning model inference provided in this disclosure, the initialization module 610 is specifically used to create a model class corresponding to the inference interface according to the preset inference interface;

[0135] Based on the model class and the configuration platform parameters, determine the backend interface that has been registered in the configuration file.

[0136] According to the data processing device based on machine learning model inference provided in this disclosure, the initialization module 610 is specifically used to obtain the model file stored in the host memory according to the model path parameters;

[0137] The model file in the host memory is transferred to the inference device corresponding to the backend interface, so that the inference device can parse the model file and obtain the model information.

[0138] A data processing apparatus based on machine learning model inference provided in this disclosure includes:

[0139] The storage module is used to initialize a preset data dictionary to obtain the target data dictionary before preprocessing the raw input data to be processed to obtain the preprocessed data stored in the cache space.

[0140] The original input data is stored in the host memory, and the original data pointer corresponding to the storage location of the original input data in the host memory is stored in the target data dictionary, so as to retrieve the original input data from the host memory using the original data pointer.

[0141] According to the data processing apparatus based on machine learning model inference provided in this disclosure, the model inference module 630 is specifically used to obtain the running function in the inference interface;

[0142] The inference function in the model class is loaded based on the running function, and the preprocessed data obtained by the input pointer is processed by model inference to obtain the model inference data.

[0143] This disclosure provides a data processing apparatus based on machine learning model inference. It obtains platform device information according to a preset configuration file, initializes the platform device information, and determines a cache space corresponding to the platform device information. It preprocesses the raw input data to be processed, obtaining preprocessed data stored in the cache space, and stores the input pointer corresponding to the first position of the preprocessed data in the cache space into a target data dictionary. It then performs model inference processing on the preprocessed data obtained from the cache space based on the input pointer, obtaining model inference data, and stores the output pointer corresponding to the second position of the model inference data in the cache space into the target data dictionary. Finally, it post-processes the model inference data obtained from the cache space based on the output pointer, obtaining inference data stored in the cache space, and stores the target pointer corresponding to the third position of the inference data in the cache space into the target data dictionary. By processing data throughout the inference process using pointers and a data dictionary, it not only improves data processing efficiency but also enhances model inference efficiency.

[0144] Figure 7 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 7As shown, the electronic device may include: a processor 710, a communication interface 720, a memory 730, and a communication bus 740, wherein the processor 710, the communication interface 720, and the memory 730 communicate with each other through the communication bus 740. The processor 710 can call logical instructions in the memory 730 to execute a data processing method based on machine learning model inference. This method includes: obtaining platform device information according to a preset configuration file; initializing the platform device information; determining a cache space corresponding to the platform device information; preprocessing the raw input data to be processed to obtain preprocessed data stored in the cache space; and storing the input pointer corresponding to the first position of the preprocessed data in the cache space into a target data dictionary; performing model inference processing on the preprocessed data obtained from the cache space according to the input pointer to obtain model inference data; and storing the output pointer corresponding to the second position of the model inference data in the cache space into the target data dictionary; and post-processing the model inference data obtained from the cache space according to the output pointer to obtain inference data stored in the cache space; and storing the target pointer corresponding to the third position of the inference data in the cache space into the target data dictionary.

[0145] Furthermore, the logical instructions in the aforementioned memory 730 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this disclosure, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this disclosure. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0146] On the other hand, this disclosure also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the data processing method based on machine learning model inference provided by the above methods. The method includes: obtaining platform device information according to a preset configuration file; initializing the platform device information; determining a cache space corresponding to the platform device information; preprocessing the raw input data to be processed to obtain preprocessed data stored in the cache space; and storing the input pointer corresponding to the first position of the preprocessed data in the cache space into a target data dictionary; performing model inference processing on the preprocessed data obtained from the cache space according to the input pointer to obtain model inference data; and storing the output pointer corresponding to the second position of the model inference data in the cache space into the target data dictionary; and postprocessing the model inference data obtained from the cache space according to the output pointer to obtain inference data stored in the cache space; and storing the target pointer corresponding to the third position of the inference data in the cache space into the target data dictionary.

[0147] In another aspect, this disclosure also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements the data processing method based on machine learning model inference provided by the above methods. The method includes: obtaining platform device information according to a preset configuration file; initializing the platform device information; determining a cache space corresponding to the platform device information; preprocessing the raw input data to be processed to obtain preprocessed data stored in the cache space; and storing the input pointer corresponding to a first position of the preprocessed data in the cache space into a target data dictionary; performing model inference processing on the preprocessed data obtained from the cache space according to the input pointer to obtain model inference data; and storing the output pointer corresponding to a second position of the model inference data in the cache space into the target data dictionary; and postprocessing the model inference data obtained from the cache space according to the output pointer to obtain inference data stored in the cache space; and storing the target pointer corresponding to a third position of the inference data in the cache space into the target data dictionary.

[0148] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0149] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0150] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this disclosure, and are not intended to limit them. Although this disclosure has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this disclosure.

Claims

1. A data processing method based on machine learning model inference, characterized in that, include: The platform device information is obtained according to the preset configuration file, the platform device information is initialized, and the cache space corresponding to the platform device information is determined. The configuration file includes at least: configuration platform parameters; filtering backend interfaces based on the configuration platform parameters; the filtering backend interfaces based on the configuration platform parameters includes: creating a model class corresponding to the inference interface based on a preset inference interface; determining the backend interfaces registered in the configuration file based on the model class and the configuration platform parameters; the backend interfaces are used to generate model information in the corresponding inference device; The original input data to be processed is preprocessed to obtain preprocessed data stored in the cache space, and the input pointer corresponding to the first position of the preprocessed data in the cache space is stored in the target data dictionary; Obtain the running function in the inference interface; load the inference function in the model class based on the running function, perform model inference processing on the preprocessed data obtained from the cache space according to the input pointer, obtain model inference data, and store the output pointer corresponding to the second position of the model inference data in the cache space into the target data dictionary; The model inference data obtained from the cache space according to the output pointer is post-processed to obtain inference data stored in the cache space, and the target pointer corresponding to the third position of the inference data in the cache space is stored in the target data dictionary.

2. The data processing method based on machine learning model inference according to claim 1, characterized in that, The configuration file shall include at least: model path parameters; The step of obtaining platform device information according to a preset configuration file, initializing the platform device information, and determining the cache space corresponding to the platform device information includes: Based on the model path parameters, the preset model file is processed to obtain the model information generated in the inference device corresponding to the backend interface; Based on the model information, the cache space is created in the inference device corresponding to the platform device information.

3. The data processing method based on machine learning model inference according to claim 2, characterized in that, The step of processing the preset model file based on the model path parameters to obtain model information generated in the inference device corresponding to the backend interface includes: The model file stored in the host memory is obtained based on the model path parameters; The model file in the host memory is transferred to the inference device corresponding to the backend interface, so that the inference device can parse the model file and obtain the model information.

4. The data processing method based on machine learning model inference according to claim 1, characterized in that, The method, preceding the preprocessing of the raw input data to be processed to obtain preprocessed data stored in the cache space, includes the following steps: The target data dictionary is obtained by initializing the preset data dictionary. The original input data is stored in the host memory, and the original data pointer corresponding to the storage location of the original input data in the host memory is stored in the target data dictionary, so as to retrieve the original input data from the host memory using the original data pointer.

5. A data processing device based on machine learning model inference, characterized in that, include: An initialization module is used to obtain platform device information according to a preset configuration file, perform initialization processing on the platform device information, and determine the cache space corresponding to the platform device information. The configuration file includes at least: configuration platform parameters; filtering backend interfaces based on the configuration platform parameters; the filtering backend interfaces based on the configuration platform parameters includes: creating a model class corresponding to the inference interface based on a preset inference interface; determining the backend interfaces registered in the configuration file based on the model class and the configuration platform parameters; the backend interfaces are used to generate model information in the corresponding inference device; The preprocessing module is used to preprocess the raw input data to be processed, obtain preprocessed data stored in the cache space, and store the input pointer corresponding to the first position of the preprocessed data in the cache space into the target data dictionary; The model inference module is used to obtain the running function in the inference interface; load the inference function in the model class based on the running function, perform model inference processing on the preprocessed data obtained from the cache space according to the input pointer, obtain model inference data, and store the output pointer corresponding to the second position of the model inference data in the cache space into the target data dictionary; The post-processing module is used to post-process the model inference data obtained from the cache space according to the output pointer, to obtain inference data stored in the cache space, and to store the target pointer corresponding to the third position of the inference data in the cache space into the target data dictionary.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the data processing method based on machine learning model reasoning as described in any one of claims 1 to 4.

7. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the data processing method based on machine learning model reasoning as described in any one of claims 1 to 4.

8. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the data processing method based on machine learning model reasoning as described in any one of claims 1 to 4.

Images