Cross-platform model deployment framework, electronic equipment and storage medium
By using the framework interface abstraction layer and runtime library of the cross-platform model deployment framework, the problem of diverse model deployment methods is solved, enabling rapid deployment and improved inference performance, while reducing development difficulty and resource waste.
Patent Information
- Application Number
- CN202511535790.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-10-24
- Publication Date
- 2026-01-20
AI Technical Summary
In existing technologies, the diverse deployment methods of artificial intelligence models lead to repetitive work, wasting human and material resources, and inconsistent evaluation standards across different platforms affect inference performance.
This paper provides a cross-platform model deployment framework, including a framework interface abstraction layer and a runtime library. Through a unified abstract interface and runtime support, it enables rapid deployment of models on different platforms and improves inference performance.
It accelerates model deployment, ensures consistency across different platforms, improves inference performance, and enables rapid deployment updates when the inference runtime environment is updated, saving manpower and resources.
Smart Images
Figure CN121364862A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of artificial intelligence, and more particularly, to a cross-platform model deployment framework, an electronic device and a storage medium. BACKGROUND
[0002] In recent years, with the rapid development of artificial intelligence technology, various artificial intelligence models have emerged like mushrooms after rain. At the same time, in order to adapt to the needs of different artificial intelligence models, various hardware accelerators (GPU, TPU, BPU, etc.) and software computing libraries (OpenCL, OpenGL, Vulkan, etc.) have also appeared. The diversity of hardware platforms and the diversity of software computing libraries also lead to the diversity of deployment methods and the diversity of timeliness differences. In this way, it further leads to a large amount of repetitive work in engineering deployment of artificial intelligence models, which is a great waste of manpower and material resources. SUMMARY
[0003] Therefore, the present application provides a cross-platform model deployment framework, an electronic device and a storage medium, which are used to enable developers to realize rapid deployment on different platforms according to the framework, thereby avoiding the waste of manpower and material resources.
[0004] In order to achieve the above-mentioned purpose, the present scheme is as follows:
[0005] A cross-platform model deployment framework applied to an electronic device, the cross-platform model deployment framework comprising a framework interface abstraction layer and a plurality of runtime libraries connected with the framework interface abstraction layer, wherein:
[0006] The framework interface abstraction layer is configured to connect an application framework layer and provide a unified abstract analysis interface for the application framework layer.
[0007] The runtime libraries are configured to provide running support for the framework interface abstraction layer.
[0008] Optionally, the framework interface abstraction layer is configured with a configuration management module, a memory management module, a sensor management module, a message sending module, a model inference module and a model scheduling module.
[0009] The abstract analysis interface comprises a memory management interface, a sensor unified management interface, a data exchange interface, a model loading and inference interface and a unified interface of underlying functions.
[0010] Optionally, the configuration management module is configured to provide the abstract analysis interface and realize management and analysis of configuration files through the abstract analysis interface.
[0011] Optionally, the memory management module is configured to provide the memory management interface, and implement memory allocation and release through the memory management interface, and uniformly manage memory allocation of the inference platform.
[0012] Optionally, the sensor management module is configured to provide the sensor uniform management interface, and complete initialization, start, stop or data acquisition of the sensor through the sensor uniform management interface.
[0013] Optionally, the message sending module is configured to provide the data exchange interface, and implement sending of post-processing content through the data exchange interface.
[0014] Optionally, the model inference module is configured to provide the model loading inference interface, and implement a model inference process through the model loading inference interface, the model inference process including a model loading process, a data preprocessing process, an inference process and a post-processing process of the model.
[0015] Optionally, the model scheduling module is configured to implement an abstract model scheduling method, and implement scheduling and inference of the model through the abstract interface.
[0016] An electronic device, comprising at least one processor and a memory connected to the processor, wherein:
[0017] The memory is configured to store computer programs or instructions;
[0018] The processor is configured to execute the computer programs or instructions, so that the electronic device implements the cross-platform model deployment framework as described above.
[0019] A storage medium applied to an electronic device, the storage medium carrying one or more computer programs, the one or more computer programs being executable by the electronic device, so that the electronic device implements the cross-platform model deployment framework as described above.
[0020] From the above technical solutions, it can be seen that the present application discloses a cross-platform model deployment framework, an electronic device and a storage medium, the framework being applied to an electronic device, and specifically comprising a framework interface abstraction layer and a plurality of runtime libraries connected to the framework interface abstraction layer. The framework interface abstraction layer is configured to connect an application framework layer, and is configured to provide a unified abstract interface for the application framework layer; the runtime library is configured to provide runtime support for the framework interface abstraction layer. Compared with a traditional model deployment scheme, the present application scheme can accelerate the deployment speed of the model, ensure the uniformity of the model in different platform evaluation standards, and improve the inference performance. If the inference running environment is updated, the deployment and update of the model on a new platform can be quickly completed according to the framework, which can greatly save manpower and material resources required for deployment. BRIEF DESCRIPTION OF DRAWINGS
[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or prior art description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative effort on the basis of these drawings.
[0022] Figure 1 A schematic diagram of a cross-platform model deployment framework according to an embodiment of the present application;
[0023] Figure 2 A schematic diagram of a framework interface abstraction layer according to an embodiment of the present application;
[0024] Figure 3 A block diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION
[0025] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative effort fall within the scope of the present application.
[0026] Figure 1 A schematic diagram of a cross-platform model deployment framework according to an embodiment of the present application.
[0027] As shown in Figure 1 , the cross-platform model deployment framework provided by the present application is applied to an electronic device, which can be understood as a computer, a server or a cloud platform with data calculation and information processing capabilities. The cross-platform model deployment framework specifically includes a framework interface abstraction layer 10 and a plurality of runtime libraries 20 connected with the framework interface abstraction layer.
[0028] The framework interface abstraction layer is used to connect an application framework layer and provide a unified abstract interface for the application framework layer. The runtime library is used to provide running support for the framework interface abstraction layer.
[0029] As shown in Figure 2 , the framework interface abstraction layer is configured with a configuration management module ConfigMgr, a memory management module ObjectPoolMgr, a sensor management module SensorMgr, a message sending module MsgIPCMgr, a model inference module ModeMgr and a model scheduling module PipeLineMgr.
[0030] The configuration management module is configured to provide an abstract parsing interface and manage and parse the configuration file through the abstract parsing interface. The abstract parsing interface includes a memory management interface, a sensor unified management interface, a data exchange interface, a model loading and reasoning interface, and a unified interface of underlying functions. Specifically, the configuration management module provides a parse() interface, which is responsible for parsing the configuration file, parsing the model and camera configuration parameters, and providing the parsed configuration parameters for other modules. Other modules can use the configuration information to start, initialize, and perform other operations on the module.
[0031] The memory management module is configured to provide a memory management interface and allocate and release memory through the memory management interface. The memory management module is configured to uniformly manage the memory allocation of the reasoning platform. Specifically, the memory management module provides memory allocation and release interfaces, is responsible for the unified management of system memory, acquiresBuffer() is responsible for applying for memory, releaseBuffer() is responsible for releasing memory, and the underlying is responsible for implementing the differentiation and customization of the memory of each platform.
[0032] The sensor management module is configured to provide a sensor unified management interface and complete the initialization, start, stop, or data acquisition of the sensor through the sensor unified management interface. Specifically, the sensor management module provides init(), run(), timeAlignment(), and other interfaces, which are used for the initialization, start, time alignment, and other operations of the sensor. The sensor information synchronization is implemented through the ros standardized message mechanism.
[0033] The message sending module is configured to provide a data exchange interface and send the post-processing content through the data exchange interface. Specifically, the message sending module provides convert() and publish() interfaces, which are responsible for converting and sending the model reasoning result.
[0034] The model reasoning module is configured to provide a model loading and reasoning interface and implement the model reasoning process through the model loading and reasoning interface. The model reasoning process includes the loading process of the model, the data preprocessing process, the reasoning process, and the post-processing process. Specifically, the model reasoning module provides init(), loadMode(), prepare(), infer(), postProc(), and other interfaces, which are responsible for the loading and initialization of the model, the model reasoning, and the analysis of the reasoning result. The three reasoning stages of the model reasoning can run in parallel to accelerate the reasoning speed of the model, reduce the time delay, and improve the reasoning performance. At the same time, the thread pool technology is naturally adaptive and can automatically adjust the number of threads to adapt to the reasoning task amount, achieving the optimization of resources and reasoning performance.
[0035] The model scheduling module is used to implement an abstract model scheduling method and schedule and infer the model through an abstract interface. The model scheduling module calls specific inference models by using a thread pool technology and is divided into three thread pools, i.e., a preprocessing thread pool, an inference thread pool and a post-processing thread pool, which respectively implement the calling of model preprocessing, inference and post-processing functions. Specifically, the module provides prepare(), infer() and postProc() interfaces, is responsible for scheduling all models of the inference system, and includes model preprocessing, inference, post-processing and other operations.
[0036] The model deployment framework simulates each operation step in the model deployment process through an abstract parsing interface, realizes the abstraction of the model deployment process, hides the implementation from the application framework layer, shields the platform environment of model deployment, including the hardware platform environment and the software platform environment, and thus realizes cross-platform model deployment. In a specific platform deployment process, the framework abstract interface is realized by calling the runtime library of the platform, and thus the rapid transplantation and deployment of the model are realized. The framework solves the inconsistency of inference performance caused by the model deployment method, the development repeatability caused by platform diversity, and isolates the underlying specific implementation environment.
[0037] For the hardware acceleration function provided by the corresponding platform itself, the scheme of the application abstracts a set of API interfaces, realizes the calling of the acceleration function in the link by using an OP registration mechanism, adopts a FastRPC interaction mechanism for communication, more optimally and more quickly utilizes the hardware module, and maximizes the utilization of platform resources. According to the API interface provided by the C++ runtime library of the specific platform, the abstract interface is realized, the interface layer development of the inference framework is completed, the main process pulls up each module according to the process dependency relationship, and the whole inference process is driven by data.
[0038] Through the unified interface provided by the inference deployment framework, the specific interface function is realized by calling the C++ runtime library of the specific running platform at the bottom, the cross-platform deployment of the model can be quickly completed, and the inference performance of the model is ensured. At the same time, when the specific interface is updated, the interface can be quickly updated and iterated by modifying the specific calling interface.
[0039] From the above technical solution can be seen, the embodiment provides a cross-platform model deployment framework, which is applied to an electronic device and specifically includes a framework interface abstraction layer and a plurality of runtime libraries connected with the framework interface abstraction layer. The framework interface abstraction layer is configured to connect an application framework layer and provide a unified abstract interface for the application framework layer, and the runtime library is configured to provide running support for the framework interface abstraction layer. Compared with a traditional model deployment scheme, the scheme provided in the application can accelerate the deployment speed of a model, ensure the uniformity of the model in different platform evaluation standards, and improve inference performance. If an inference running environment is updated, the framework can be used to quickly complete deployment and update of the model on a new platform, thereby greatly saving manpower and resources required for deployment.
[0040] The framework developer is mainly responsible for efficient scheduling and implementation of the framework model, and does not need to care about the underlying C++ implementation; the C++ runtime library developer is committed to developing different backend specifications and providing more convenient APIs, and the separation of tasks is very meaningful in practice, and the design reduces the difficulty of development.
[0041] The application framework layer and the platform layer of the application are decoupled, the coupling degree of the program is reduced, the expansion of the program is improved, and the OCP development principle is met. The application framework layer only needs to care about the form of the interface and does not need to care about the specific implementation, as long as the compatibility of the interface is maintained, modification or upgrade of the platform layer is not perceived by the user. After decoupling, it is also convenient for multiple people to develop, after designing the interface, each module only interacts through the interface, and each module can be completed.
[0042] Figure 3 A block diagram of an electronic device according to an embodiment of the application.
[0043] Reference will be made to the following description Figure 3 which shows a structural schematic diagram suitable for implementing an electronic device in the embodiments of the present disclosure. The terminal device in the embodiments of the present disclosure can include, but is not limited to, a mobile terminal such as a mobile phone, a notebook computer, a digital broadcast receiver, a PDA (Personal Digital Assistant), a PAD (Tablet Personal Computer), a PMP (Portable Multimedia Player), a vehicle terminal (for example, a vehicle navigation terminal), and the like, and a fixed terminal such as a digital TV, a desktop computer, and the like. The electronic device is only an example, and should not bring any limitation to the functions and use range of the embodiments of the present disclosure.
[0044] The electronic device can include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 301 that can perform various appropriate actions and processes according to programs stored in a read-only memory (ROM) 302 or loaded into a random access memory (RAM) 303 from an input device 306. Various programs and data required for the operation of the electronic device are also stored in the RAM. The processing device, the ROM, and the RAM are connected to each other through a bus 304. An input / output (I / O) interface 305 is also connected to the bus 304.
[0045] Generally, the following devices can be connected to the I / O interface: input devices including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; output devices 307 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 308 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 309. The communication devices 309 can allow the electronic device to communicate wirelessly or wiredly with other devices to exchange data. Although the electronic device is shown as having various devices, it should be understood that all of the shown devices are not required to be implemented or possessed. More or fewer devices can be alternatively implemented or possessed.
[0046] The present application also provides a computer-readable storage medium embodiment.
[0047] The above computer-readable storage medium is applied to an electronic device and carries one or more programs, which, when executed by the electronic device, enable the electronic device to implement a cross-platform model deployment framework. The framework is applied to an electronic device and specifically includes a framework interface abstraction layer and a plurality of runtime libraries connected to the framework interface abstraction layer. The framework interface abstraction layer is used to connect an application framework layer and provide a unified abstract interface for the application framework layer; and the runtime library is used to provide runtime support for the framework interface abstraction layer. Compared with traditional model deployment solutions, the present application can accelerate the deployment speed of the model, ensure the uniformity of the model in different platform evaluation standards, and improve the inference performance. If the inference running environment is updated, the framework can be used to quickly complete the deployment and update of the model on the new platform, which can greatly save the manpower and material resources required for deployment.
[0048] It should be noted that the computer-readable medium in the present disclosure can be a computer-readable signal medium or a computer-readable storage medium or any combination of the two. The computer-readable storage medium may, for example, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or apparatus, or any combination of the above. More specific examples of the computer-readable storage medium can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.
[0049] In the present disclosure, the computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in connection with an instruction execution system, apparatus or device. In the present disclosure, the computer-readable signal medium can include a data signal that carries computer-readable program code in a baseband or as part of a carrier wave. Such a propagated data signal can take on many forms, including but not limited to electro-magnetic, optical, or any suitable combination thereof. The computer-readable signal medium can also be any computer-readable medium that is not a storage medium and that can communicate, propagate or transport program code for use by or in connection with an instruction execution system, apparatus or device. Program code embodied on a computer-readable medium can be transmitted using any suitable medium, including but not limited to wire, cable, optical fiber, RF, etc., or any suitable combination of the above.
[0050] Each of the embodiments in the present specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be mutually referred to.
[0051] Although the preferred embodiments of the present application have been described, those skilled in the art can make further changes and modifications to the embodiments once they know the basic inventive concept. Therefore, the appended claims are intended to include the preferred embodiments and all changes and modifications falling within the scope of the embodiments of the present application.
[0052] Finally, it needs to be pointed out that in this document, relational terms such as first and second and the like can only be used to distinguish one entity or action from another entity or action, without necessarily requiring or implying any such actual relationship or order between such entities or actions. Moreover, the terms "comprises", "comprising", or any other variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without more limitations, an element defined by an "comprising" statement serves as a means plus function alternative.
[0053] The above detailed description of the technical solutions provided by the present application has been given, and the principles and implementation manners of the present application are described by applying specific examples in this document. The above description of the examples is only for helping to understand the method of the present application and its core idea; meanwhile, for those skilled in the art, according to the idea of the present application, the specific implementation manners and application ranges will have changes, and the above description of the content of the specification should not be understood as a limitation of the present application.
Claims
1. A cross-platform model deployment framework applied to an electronic device, characterized in that, The cross-platform model deployment framework comprises a framework interface abstraction layer and a plurality of runtime libraries connected with the framework interface abstraction layer, wherein: The framework interface abstraction layer is configured to connect an application framework layer and provide a unified abstract analysis interface for the application framework layer. The runtime libraries are configured to provide running support for the framework interface abstraction layer.
2. The cross-platform model deployment framework of claim 1, wherein, The framework interface abstraction layer is configured with a configuration management module, a memory management module, a sensor management module, a message sending module, a model inference module and a model scheduling module. The abstract analysis interface comprises a memory management interface, a sensor unified management interface, a data exchange interface, a model loading inference interface and a unified interface of underlying functions.
3. The cross-platform model deployment framework of claim 2, wherein, The configuration management module is configured to provide the abstract analysis interface and implement management and analysis of configuration files through the abstract analysis interface.
4. The cross-platform model deployment framework of claim 2, wherein, The memory management module is configured to provide the memory management interface and implement allocation and release of memory through the memory management interface, and uniformly manage memory allocation of an inference platform.
5. The cross-platform model deployment framework of claim 2, wherein, The sensor management module is configured to provide the sensor unified management interface and complete initialization, start, stop or data acquisition of sensors through the sensor unified management interface.
6. The cross-platform model deployment framework of claim 2, wherein, The message sending module is configured to provide the data exchange interface and implement sending of post-processing content through the data exchange interface.
7. The cross-platform model deployment framework of claim 2, wherein, The model inference module is configured to provide the model loading inference interface and implement a model inference process through the model loading inference interface, wherein the model inference process comprises a model loading process, a data preprocessing process, an inference process and a post-processing process of the model.
8. The cross-platform model deployment framework of claim 2, wherein, The model scheduling module is configured to implement an abstract model scheduling method and implement scheduling and inference of a model through the abstract interface.
9. An electronic device, comprising: The electronic device comprises at least one processor and a memory connected with the processor, wherein: The memory is configured to store computer programs or instructions. The processor is configured to execute the computer programs or instructions to enable the electronic device to implement the cross-platform model deployment framework as claimed in any one of claims 1-8.
10. A storage medium for use in an electronic device, the storage medium comprising: The storage medium carries one or more computer programs, which can be executed by the electronic device to enable the electronic device to implement the cross-platform model deployment framework as claimed in any one of claims 1-8.