A deep learning reasoning method, device, equipment and medium

By encapsulating and integrating the AI ​​inference framework in the Android system and communicating with the Linux system through socket, the complexity of AI development and acceleration capability call problems under the Android system are solved, and deep learning AI model inference under the Android+Linux fusion system is realized, reducing the problem of AI model inference in the Android+Linux fusion system. cost and complexity.

CN118709793BActive Publication Date: 2024-11-15CHENGDU AJIAXI INTELLIGENT TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411178924.0
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-08-27
Publication Date
2024-11-15
Estimated Expiration
2044-08-27

AI Technical Summary

Technical Problem

Under the Android system, containerization technology is used to achieve complex and fragmented operation of Linux system and AI development, and it is difficult to call the acceleration capability of the underlying Android system, especially the maximum acceleration capability of the GPU or NPU.

Method used

Encapsulate and integrate multiple AI inference frameworks in the Android system to form an Aidlite-SDK inference framework, and communicate with the Linux system through the socket communication channel, send remote process calls RPC requests to realize AI inference, and feedback the results, reducing the inference framework Complexity and cost simplifies the acceleration capability call of heterogeneous systems.

Benefits of technology

By integrating and encapsulating the AI ​​inference framework and establishing socket communication channels, the fragmentation and complexity of the inference framework are reduced, the investment cost is reduced, and the acceleration capability call from the upper Linux system to the underlying Android system is simplified, realizing the Android+Linux Deep learning AI model reasoning under a fusion system.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118709793B_ABST
    Figure CN118709793B_ABST
Patent Text Reader

Abstract

The present application discloses a deep learning reasoning method, device, equipment and medium, which relates to the field of artificial intelligence technology and is used to solve the problems of complexity, fragmentation and high cost of existing reasoning frameworks when developing AI. The method is applied to the Android+Linux fusion system, including: encapsulating and integrating multiple preset AI reasoning frameworks in the Android system to obtain the Aidlite‑SDK reasoning framework; building a socket communication channel between the Aidlite‑SDK reasoning framework and the Linux system; according to the socket communication channel, using the Linux system to send a remote procedure call RPC request to the Aidlite‑SDK reasoning framework; wherein the remote procedure call RPC request contains AI reasoning data; in response to the remote procedure call RPC request, using the Aidlite‑SDK reasoning framework to perform AI reasoning, and feeding back the corresponding AI reasoning results to the Linux system to reduce the fragmentation, complexity and investment cost of the reasoning framework.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology and provides a deep learning reasoning method, device, equipment and medium. Background Art

[0002] At present, under the premise of open source Android system, the use of containerization technology to implement the Linux system running on Android applications has become the mainstream of customization and personalization in the current Android market. However, it is not easy to use the underlying computing power of edge chips for AI development based on the Linux system containerized by Android applications. Moreover, if you want to run all some commonly used reasoning frameworks under the Linux system, it is not only complex and fragmented, but also costly. In addition, if you want to call the maximum acceleration capability of each reasoning framework (for example, GPU or NPU), you also need to solve the heterogeneous problem of acceleration capability from the upper Linux system to the underlying Android system. Summary of the invention

[0003] The present application provides a deep learning reasoning method, apparatus, device and medium for solving the problems of complexity, fragmentation and high cost of existing reasoning frameworks when developing AI.

[0004] On the one hand, a deep learning reasoning method is provided, which is applied to an Android+Linux fusion system, and the method includes:

[0005] Package and integrate multiple preset AI reasoning frameworks in the Android system to obtain the Aidlite-SDK reasoning framework;

[0006] Building a socket communication channel between the Aidlite-SDK inference framework and the Linux system;

[0007] According to the socket communication channel, the Linux system is used to send a remote procedure call RPC request to the Aidlite-SDK reasoning framework; wherein the remote procedure call RPC request includes AI reasoning data;

[0008] In response to the remote procedure call RPC request, the Aidlite-SDK reasoning framework is used to perform AI reasoning, and the corresponding AI reasoning result is fed back to the Linux system.

[0009] Optionally, the step of encapsulating and integrating multiple preset AI reasoning frameworks in the Android system to obtain the Aidlite-SDK reasoning framework includes:

[0010] Customize the Model class related to the model; wherein the Model class includes the full path of the model file, the path of the model weight parameter, the model input type, the model output type, the shape information of the model input, and the shape information of the model output;

[0011] Customize the Config class related to parameter configuration; wherein the Config class includes the acceleration type, framework type, output node name, and number of threads of the underlying model call;

[0012] Customize the Interpreter class related to the model reasoning core; wherein the Interpreter class includes model initialization configuration, model loading, setting model data, model reasoning, and obtaining the inferred data.

[0013] Optionally, before establishing a socket communication channel between the Aidlite-SDK inference framework and the Linux system, the method further includes:

[0014] A socket communication framework for communicating with the Android system is designed in the Linux system by using C++ language or Python language.

[0015] Optionally, the step of designing a socket communication framework for communicating with the Android system in the Linux system by using C++ language or Python language includes:

[0016] Read the model information sent by the Aidlite client; wherein the model information includes the model Modle-related properties, model configuration-related properties, model initialization, loading type, setting model data, inference model, obtaining model data and releasing model as agreed upon by the protocol;

[0017] Parsing the model information in the Linux system to obtain parsed model information;

[0018] According to the parsed model information, a socket communication framework for communicating with the Android system is designed in the Linux system by using C++ language or Python language.

[0019] Optionally, after designing a socket communication framework for communicating with the Android system in the Linux system by using the C++ language or the Python language, the method further includes:

[0020] Encapsulating the socket communication framework to obtain an encapsulated socket communication framework;

[0021] The step of sending a remote procedure call (RPC) request to the Aidlite-SDK reasoning framework using the Linux system according to the socket communication channel includes:

[0022] According to the encapsulated socket communication framework, the Linux system is used to send a remote procedure call RPC request to the Aidlite-SDK reasoning framework.

[0023] Optionally, the step of sending a remote procedure call (RPC) request to the Aidlite-SDK inference framework using the Linux system according to the socket communication channel includes:

[0024] According to the socket communication channel and the data transmission framework MMKV, the Linux system is used to send a remote procedure call RPC request to the Aidlite-SDK reasoning framework.

[0025] Optionally, the multiple preset AI reasoning frameworks include TFLite framework, Snpe framework, PaddlePaddle framework, ONNX framework, QNN framework, RKNN framework, TNN framework and NCNN framework.

[0026] On the one hand, a deep learning reasoning device is provided, which is applied to an Android+Linux fusion system, and the device includes:

[0027] The inference framework integration unit is used to package and integrate multiple preset AI inference frameworks in the Android system to obtain the Aidlite-SDK inference framework;

[0028] A communication channel building unit, used to build a socket communication channel between the Aidlite-SDK reasoning framework and the Linux system;

[0029] An RPC request sending unit, configured to send a remote procedure call RPC request to the Aidlite-SDK inference framework using the Linux system according to the socket communication channel; wherein the remote procedure call RPC request includes AI inference data;

[0030] The AI ​​reasoning unit is used to respond to the remote procedure call RPC request, use the Aidlite-SDK reasoning framework to perform AI reasoning, and feed back corresponding AI reasoning results to the Linux system.

[0031] On the one hand, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein any one of the above methods is implemented when the processor executes the computer program.

[0032] In one aspect, a computer storage medium is provided, on which computer program instructions are stored, and when the computer program instructions are executed by a processor, any of the above methods is implemented.

[0033] Compared with the prior art, the beneficial effects of this application are:

[0034] In this application, when performing AI reasoning, first, multiple preset AI reasoning frameworks can be encapsulated and integrated in the Android system to obtain the Aidlite-SDK reasoning framework; then, a socket communication channel can be built between the Aidlite-SDK reasoning framework and the Linux system; next, according to the socket communication channel, the Linux system can be used to send a remote procedure call RPC request to the Aidlite-SDK reasoning framework; wherein the remote procedure call RPC request contains AI reasoning data; finally, the remote procedure call RPC request can be responded to to use the Aidlite-SDK reasoning framework for AI reasoning, and the corresponding AI reasoning results can be fed back to the Linux system. Therefore, in this application, since various AI reasoning frameworks are integrated and encapsulated together, not only can the fragmentation and complexity of the reasoning framework be greatly reduced, but also the investment cost can be reduced. Moreover, since a socket communication channel is also built between the Aidlite-SDK reasoning framework and the Linux system, the difficulty of calling the heterogeneous unit acceleration capability from the upper Linux system to the underlying Android system is reduced, so that, under the Android+Linux fusion system, the deep learning AI model reasoning can be processed by the Android proxy. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related technologies, the drawings required for use in the embodiments or the related technical descriptions are briefly introduced below. Obviously, the drawings described below are only the embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0036] Figure 1 A schematic diagram of an application scenario provided for an embodiment of the present application;

[0037] Figure 2 A schematic diagram of a principle framework of deep learning reasoning provided in an embodiment of the present application;

[0038] Figure 3 A schematic diagram of a flow chart of a deep learning reasoning method provided in an embodiment of the present application;

[0039] Figure 4 A specific code schematic diagram of deep learning reasoning provided in an embodiment of the present application;

[0040] Figure 5 Another specific code schematic diagram of deep learning reasoning provided in an embodiment of the present application;

[0041] Figure 6 A schematic diagram of a deep learning reasoning device provided in an embodiment of the present application.

[0042] Markings in the figure: 10-deep learning reasoning device, 101-processor, 102-memory, 103-I / O interface, 104-database, 60-deep learning reasoning device, 601-reasoning framework integration unit, 602-communication channel building unit, 603-RPC request sending unit, 604-AI reasoning unit. DETAILED DESCRIPTION

[0043] In order to make the purpose, technical scheme and advantages of the present application clearer, the technical scheme in the embodiment of the present application will be clearly and completely described below in conjunction with the drawings in the embodiment of the present application. Obviously, the described embodiment is only a part of the embodiment of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application. In the absence of conflict, the embodiments in the present application and the features in the embodiments can be combined with each other arbitrarily. In addition, although the logical order is shown in the flow chart, in some cases, the steps shown or described can be performed in an order different from that here.

[0044] At present, under the premise of open source Android system, the use of containerization technology to implement the Linux system running on Android applications has become the mainstream of customization and personalization in the current Android market. However, it is not easy to use the underlying computing power of edge chips for AI development based on the Linux system containerized by Android applications. Moreover, if you want to run all some commonly used reasoning frameworks under the Linux system, it is not only complex and fragmented, but also costly. In addition, if you want to call the maximum acceleration capability of each reasoning framework (for example, GPU or NPU), you also need to solve the heterogeneous problem of acceleration capability from the upper Linux system to the underlying Android system.

[0045] Based on this, the embodiment of the present application provides a deep learning reasoning method, in which, first, multiple preset AI reasoning frameworks can be encapsulated and integrated in the Android system to obtain the Aidlite-SDK reasoning framework; then, a socket communication channel can be built between the Aidlite-SDK reasoning framework and the Linux system; next, according to the socket communication channel, the Linux system can be used to send a remote procedure call RPC request to the Aidlite-SDK reasoning framework; wherein the remote procedure call RPC request contains AI reasoning data; finally, the remote procedure call RPC request can be responded to to use the Aidlite-SDK reasoning framework for AI reasoning, and the corresponding AI reasoning results can be fed back to the Linux system. Therefore, in the embodiment of the present application, since various AI reasoning frameworks are integrated and packaged together, not only can the fragmentation and complexity of the reasoning framework be greatly reduced, but also the investment cost is reduced. Moreover, since a socket communication channel is also built between the Aidlite-SDK reasoning framework and the Linux system, the difficulty of calling the heterogeneous unit acceleration capability from the upper Linux system to the underlying Android system is reduced, so that, in the Android+Linux fusion system, the deep learning AI model reasoning can be processed by the Android proxy.

[0046] After introducing the design ideas of the embodiments of the present application, the following briefly introduces the application scenarios to which the technical solutions of the embodiments of the present application can be applied. It should be noted that the application scenarios introduced below are only used to illustrate the embodiments of the present application and are not limited. In the specific implementation process, the technical solutions provided by the embodiments of the present application can be flexibly applied according to actual needs.

[0047] like Figure 1 FIG. 1 is a schematic diagram of an application scenario provided by an embodiment of the present application. The application scenario may include a deep learning inference device 10.

[0048] Among them, the deep learning reasoning device 10 can be used to perform AI reasoning through deep learning, for example, it can be a personal computer (PC), a server and a laptop. The deep learning reasoning device 10 may include one or more processors 101, a memory 102, an I / O interface 103 and a database 104. Specifically, the processor 101 can be a central processing unit (CPU), or a digital processing unit, etc. The memory 102 can be a volatile memory (volatile memory), such as a random-access memory (RAM); the memory 102 can also be a non-volatile memory (non-volatile memory), such as a read-only memory, a flash memory (flash memory), a hard disk drive (HDD) or a solid-state drive (SSD); or the memory 102 is any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited to this. The memory 102 can be a combination of the above memories. The memory 102 may store some program instructions of the deep learning reasoning method provided in the embodiment of the present application, which can be used to implement the steps of the deep learning reasoning method provided in the embodiment of the present application when executed by the processor 101, so as to solve the problems of complexity, fragmentation and high cost of the existing reasoning framework when developing AI. The database 104 may be used to store data such as multiple preset AI reasoning frameworks, Aidlite-SDK reasoning frameworks, AI reasoning data, AI reasoning results, Model classes, Config classes, Interpreter classes and model information involved in the solution provided in the embodiment of the present application.

[0049] In the embodiment of the present application, the deep learning reasoning device 10 can obtain multiple preset AI reasoning frameworks through the I / O interface 103, and then the processor 101 of the deep learning reasoning device 10 will reduce the fragmentation, complexity and investment cost of the reasoning framework according to the program instructions of the deep learning reasoning method provided in the embodiment of the present application in the memory 102. In addition, multiple preset AI reasoning frameworks, Aidlite-SDK reasoning frameworks, AI reasoning data, AI reasoning results, Model class, Config class, Interpreter class and model information data can also be stored in the database 104.

[0050] like Figure 2As shown, it is a schematic diagram of a principle framework of deep learning reasoning provided in an embodiment of the present application. In the background, in the Android system, a Linux system (for example, Ubuntu, a Linux distribution operating system mainly for desktop applications, or Debian, an operating system based on the Linux kernel) is virtualized through containerization technology.

[0051] Among them, if you want to bring out the AI ​​capabilities of Linux in the Linux system, you need to let the Linux system perform AI development. Therefore, in the embodiment of the present application, you can use the Android AI application (Aidlite Sdk Engine) to encapsulate and integrate multiple preset AI reasoning frameworks in the Android system (for example, TFLite reasoning framework, Snpe reasoning framework, PaddlePaddle reasoning framework, ONNX reasoning framework, QNN reasoning framework, RKNN reasoning framework, TNN reasoning framework, NCNN reasoning framework, etc.), thereby obtaining the Aidlite-SDK reasoning framework.

[0052] Then, you can build a remote procedure call (RPC), that is, build a socket communication channel between the Aidlite-SDK inference framework and the Linux system, so that the Linux system can communicate with the AI ​​application of the Android system.

[0053] Next, the Aidlite client application programming interface is called uniformly in the Linux system to develop AI in the Linux system environment and give full play to the AI ​​capabilities of containerized Linux. That is, the Aidlite client application programming interface can be called uniformly to send remote procedure call (RPC) requests to the Aidlite-SDK inference framework; wherein the remote procedure call (RPC) request contains AI inference data.

[0054] Finally, by responding to the remote procedure call (RPC) request, the Aidlite-SDK reasoning framework is used to perform AI reasoning, and the corresponding AI reasoning results are fed back to the Linux system.

[0055] Of course, the method provided in the embodiment of the present application is not limited to Figure 1 The application scenarios shown can also be used in other possible application scenarios, and the embodiments of the present application are not limited thereto. Figure 1 The functions that can be realized by each device in the application scenario shown will be described in the subsequent method embodiments, and will not be described in detail here. Below, the method of the embodiment of the present application will be introduced in conjunction with the accompanying drawings.

[0056] like Figure 3As shown, it is a flowchart of a deep learning reasoning method provided by an embodiment of the present application. The method is applied to an Android+Linux fusion system and can be Figure 1 The deep learning inference device 10 is used to execute, and specifically, the process of the method is described as follows.

[0057] Step 301: Package and integrate multiple preset AI reasoning frameworks in the Android system to obtain the Aidlite-SDK reasoning framework.

[0058] In the embodiment of the present application, the multiple preset AI reasoning frameworks may include TFLite framework, Snpe framework, PaddlePaddle framework, ONNX framework, QNN framework, RKNN framework, TNN framework and NCNN framework. Moreover, the Aidlite-SDK reasoning framework is a software development kit (SDK).

[0059] However, in actual applications, since the designs of various AI reasoning frameworks are inconsistent, in order to be compatible with the inconsistencies in the designs of various Android AI reasoning frameworks, in the embodiments of the present application, some classes need to be customized to facilitate the calling of the encapsulated and integrated Aidlite-SDK reasoning framework.

[0060] Specifically, first, you can customize the Model class related to the model; the Model class includes public member variables, for example, the full path of the model file required by the framework to load the model (variable modelAbsolutePath), the path of the original model weight parameters of individual reasoning frameworks (for example, TNN, NCNN reasoning frameworks, etc.) (variable modelWeightPath), model input / output types (variable inputType / variable outputType, for example, the model has 32-bit floating-point data input and output, or int8 quantized input and output, or int64 long integer input and output, so the type can be an enumeration type), model input / output shape information (variable inShape / variable outShape, which represents the shape information of the model input and output, and its type is a two-dimensional long integer long[][], and because the model may have multiple inputs and multiple outputs, two dimensions are selected. In addition, because the model may have a large shape parameter, a long integer is selected).

[0061] Then, since different models of different AI reasoning frameworks have different configurations, in an embodiment of the present application, a class can be designed to meet the differentiated configuration of the framework, that is, the Config class related to the parameter configuration (the parameter configuration class required by the reasoning framework) can be customized; wherein, the member attributes of the Config class include the acceleration type of the underlying model call (variable accelerateType, for example, the AI ​​reasoning framework can be set to use CPU, GPU or NPU when running), the framework type (variable frameworkType, for example, TYPE_TFLITE, TYPE_SNPE, TYPE_QNN, TYPE_TNN, TYPE_PADDLE_PADDLE, TYPE_NCNN, TYPE_MNN, TYPE_ONNX, TYPE_MNN), the output node name (variable outNodes, for example, for a multi-output model, the output node of the model needs to be specified) and the number of threads (variable numberOfThreads, for example, when using the CPU for AI reasoning, several threads are required for reasoning).

[0062] Next, you can customize the Interpreter class related to the model reasoning core (mainly the core class of Aidlite SDK model reasoning); among them, the Interpreter class includes model initialization configuration, model loading, setting model data, model reasoning, and obtaining inferred data. Specifically, when loading the model, first, you can set the model config that has been initialized in the previous step to the interpreter for initialization configuration. Then, after the interpreter is initialized, you can load the model, that is, load the local model into the memory through the AI ​​reasoning framework (the correct loading of the model is a necessary condition before the model can be reasoned). In addition, after the model is loaded successfully, the model data is set. The model infers one or more frames of images, and it is necessary to set the preprocessed image data to the interpreter interpreter.

[0063] Furthermore, after setting the data for the inference framework, the inference framework can be used for reasoning.

[0064] Step 302: Build a socket communication channel between the Aidlite-SDK inference framework and the Linux system.

[0065] After the SDK design of the Aidlite-SDK inference framework is completed, in order to reduce the difficulty of calling the heterogeneous unit acceleration capabilities from the upper-level Linux system to the underlying Android system, in an embodiment of the present application, a communication framework for the Linux system in the Android+Linux fusion system to communicate with the Android system (for example, network communication, pipeline communication) can also be designed. This communication framework can simultaneously meet the needs of the Linux system designed based on C++ to communicate with the Android system background inference service, and the Linux system designed based on Python to communicate with the Android system background service.

[0066] Furthermore, in the embodiment of the present application, before building a socket communication channel between the Aidlite-SDK reasoning framework and the Linux system, a socket communication framework for communicating with the Android system can also be designed in the Linux system through C++ or Python. Among them, the Android system background reasoning service can be used as the socket server, and the Linux system designed based on C++\Python can be used as the socket client.

[0067] Based on this, we can directly design a socket communication framework for communicating with the Android system in the Linux system through C++ or Python. Specifically, first, we can read the model information sent by the Aidlite client; the model information includes the model Modle-related properties, model configuration-related properties, model initialization, loading type, setting model data, inference model, obtaining model data, and releasing model; then, we can parse the model information in the Linux system to obtain the parsed model information; finally, we can design a socket communication framework for communicating with the Android system in the Linux system through C++ or Python based on the parsed model information.

[0068] Among them, when communicating based on the socket communication framework (designed by C++ language / Python language), first, you can set the IP address and port of the Aidlite server in the Android system; then, based on the IP address and port of the Aidlite server, you can try to let the Linux system connect to the Aidlite server; next, you can let the Linux system send a message to the Aidlite server; then, the Linux system can receive the result generated by the Aidlite server response, and finally, you can close the socket communication channel.

[0069] In addition, in order to facilitate users to perform AI reasoning, after designing a socket communication framework for communicating with the Android system in the Linux system through C++ or Python, in the embodiment of the present application, the socket communication framework can also be encapsulated to obtain an encapsulated socket communication framework. Furthermore, when the Linux system is used to send a remote procedure call RPC request to the Aidlite-SDK reasoning framework according to the socket communication channel, the Linux system can be used to send a remote procedure call RPC request to the Aidlite-SDK reasoning framework directly according to the encapsulated socket communication framework.

[0070] Step 303: According to the socket communication channel, a remote procedure call (RPC) request is sent to the Aidlite-SDK inference framework using the Linux system.

[0071] In an embodiment of the present application, the remote procedure call (RPC) request includes AI reasoning data.

[0072] In actual applications, when there is communication transmission, data transmission also needs to be considered to send the data set by the user in the Aidlite client to the Android inference background service.

[0073] Specifically, a remote procedure call RPC request can be sent to the Aidlite-SDK reasoning framework using the Linux system according to the socket communication channel and the data transmission framework MMKV, wherein the data transmission framework MMKV supports a data framework for cross-platform transmission.

[0074] Step 304: Respond to the remote procedure call (RPC) request, use the Aidlite-SDK reasoning framework to perform AI reasoning, and feed back the corresponding AI reasoning results to the Linux system.

[0075] In the embodiment of the present application, after the Android system receives the remote procedure call RPC request, the Android system will respond to the remote procedure call RPC request to use a specific reasoning framework in the Aidlite-SDK reasoning framework (the reasoning framework corresponding to the loaded reasoning model) for AI reasoning; then, after the reasoning is completed, in order to directly obtain the result data predicted by the model, the corresponding AI reasoning result can be directly fed back to the Linux system. In addition, if the developer no longer uses the loaded reasoning model, the loaded reasoning model can be released.

[0076] In one possible implementation, when implementing the SDK for communication between the Linux system and the Android system, in order to make the communication transmission faster, in an embodiment of the present application, both the Linux system and the Android system can use a unified diagnostic service (UDS) to communicate, or use the cross-process communication binder mechanism on the Android platform (cross-compile the binder mechanism into the ARM-Linux kernel).

[0077] In addition, the medium for data transmission can also be optimized, that is, the file descriptor can be directly operated in the memory through shared memory to achieve the purpose of data transmission. Specific embodiment 1:

[0079] Take the deep learning inference process of Qualcomm Yolov5.dlc model as an example.

[0080] The example is a Linux C++ client that calls the Qualcomm SNPE framework, passes in the DLC model, and sets the data for the NPU acceleration unit to perform 10 inferences. First, after encapsulating and integrating the Aidlite-SDK inference framework and building the socket communication channel, you can directly set the relevant AI inference data in the Android system, and then you can directly perform inference based on the AI ​​inference data and obtain the time consumption of the AI ​​inference data, such as Figure 4 As shown, a specific code diagram of deep learning reasoning provided in an embodiment of the present application is provided.

[0081] Specifically, after encapsulating and integrating the Aidlite-SDK inference framework and building the socket communication channel, the parameters required for AI inference can be directly encapsulated in the C++ client of the Android system, such as the model path, model input, output tensor type, model input size information, output size information, and which computing unit (CPU, GPU, NPU) to use for inference, etc. Based on this, the C++ client can pass these parameters to the Android encapsulated AI server through the buildInterpreter method encapsulated by the socket, and then the AI ​​server can parse these parameters.

[0082] Then, the C++ client can start calling the socket-encapsulated interface, such as loading the model interface, setting the data interface, and the inference interface, and then obtain the inference data interface to obtain the parameters just parsed.

[0083] And before calling the data setting interface, because the interface method parameters are not suitable for passing image loading data, the data can be written to MMKV. Then, when the AI ​​server receives the signal from the C++ client calling the data setting interface, the image tensor data just saved by the C++ client can be retrieved from MMKV.

[0084] Similarly, when obtaining the inference data interface, the AI ​​server can write the inference result data to MMKV in advance when returning the return value of the inference data interface. After the C++ client calls the inference data acquisition function, it can retrieve the inference result data from MMKV and perform post-inference processing. Specific embodiment 2:

[0086] Take the deep learning inference process of Google's Yolov5.tflite model as an example.

[0087] The example is a Linux C++ client that calls the Google tflite framework, passes in the tflite model, and sets the data for the CPU acceleration unit to perform one inference. First, after encapsulating and integrating the Aidlite-SDK inference framework and building the socket communication channel, you can directly set the relevant AI inference data in the Android system, and then you can directly perform inference based on the AI ​​inference data and obtain the time consumption of the AI ​​inference data, such as Figure 5 As shown, another specific code diagram of deep learning reasoning provided in an embodiment of the present application is shown.

[0088] Specifically, similar to Specific Embodiment 1, after encapsulating and integrating the Aidlite-SDK reasoning framework and building the socket communication channel, the parameters required for AI reasoning can be directly encapsulated in the C++ client of the Android system, such as the model path, model input, output tensor type, model input size information, output size information, and which computing unit (CPU, GPU, NPU) is used for reasoning, etc. Based on this, the C++ client can pass these parameters to the Android encapsulated AI server through the buildInterpreter method of the socket encapsulation, and then the AI ​​server can parse these parameters.

[0089] Then, the C++ client can start calling the socket-encapsulated interface, such as loading the model interface, setting the data interface, and the inference interface, and then obtain the inference data interface to obtain the parameters just parsed.

[0090] And before calling the data setting interface, because the interface method parameters are not suitable for passing image loading data, the data can be written to MMKV. Then, when the AI ​​server receives the signal from the C++ client calling the data setting interface, the image tensor data just saved by the C++ client can be retrieved from MMKV.

[0091] Similarly, when obtaining the inference data interface, the AI ​​server can write the inference result data to MMKV in advance when returning the return value of the inference data interface. After the C++ client calls the inference data acquisition function, it can retrieve the inference result data from MMKV and perform post-inference processing.

[0092] In summary, since various AI reasoning frameworks are integrated and packaged together, not only can the fragmentation and complexity of the reasoning framework be greatly reduced, but also the investment cost can be reduced. In addition, since a socket communication channel is built between the Aidlite-SDK reasoning framework and the Linux system, the difficulty of calling the heterogeneous unit acceleration capability from the upper Linux system to the underlying Android system is reduced. Therefore, in the Android+Linux fusion system, deep learning AI model reasoning can be processed by the Android proxy.

[0093] Based on the same inventive concept, the present application embodiment provides a deep learning reasoning device 60, which is applied to an Android+Linux fusion system, such as Figure 6 As shown, the deep learning reasoning device 60 includes:

[0094] The inference framework integration unit 601 is used to package and integrate multiple preset AI inference frameworks in the Android system to obtain the Aidlite-SDK inference framework;

[0095] A communication channel building unit 602 is used to build a socket communication channel between the Aidlite-SDK reasoning framework and the Linux system;

[0096] The RPC request sending unit 603 is used to send a remote procedure call RPC request to the Aidlite-SDK reasoning framework using the Linux system according to the socket communication channel; wherein the remote procedure call RPC request contains AI reasoning data;

[0097] The AI ​​reasoning unit 604 is used to respond to remote procedure call (RPC) requests, perform AI reasoning using the Aidlite-SDK reasoning framework, and feed back corresponding AI reasoning results to the Linux system.

[0098] Optionally, the reasoning framework integration unit 601 is further used for:

[0099] Customize the Model class related to the model; the Model class includes the full path of the model file, the path of the model weight parameter, the model input type, the model output type, the shape information of the model input, and the shape information of the model output;

[0100] Customize the Config class related to parameter configuration; the Config class includes the underlying acceleration type, framework type, output node name, and number of threads of the model call;

[0101] Customize the Interpreter class related to the model reasoning core; the Interpreter class includes model initialization configuration, model loading, setting model data, model reasoning, and obtaining the inferred data.

[0102] Optionally, the communication channel building unit 602 is further used for:

[0103] Design a socket communication framework in the Linux system to communicate with the Android system using C++ or Python.

[0104] Optionally, the communication channel building unit 602 is further used for:

[0105] Read the model information sent by the Aidlite client; the model information includes the model Modle-related properties, model configuration-related properties, model initialization, loading type, setting model data, inferring model, obtaining model data and releasing model.

[0106] Parse the model information in the Linux system to obtain the parsed model information;

[0107] Based on the parsed model information, a socket communication framework for communicating with the Android system is designed in the Linux system using C++ or Python.

[0108] Optionally, the communication channel building unit 602 is further used for:

[0109] Encapsulate the socket communication frame to obtain the encapsulated socket communication frame;

[0110] According to the socket communication channel, the steps of sending a remote procedure call RPC request to the Aidlite-SDK inference framework using the Linux system include:

[0111] According to the encapsulated socket communication framework, the Linux system is used to send a remote procedure call RPC request to the Aidlite-SDK inference framework.

[0112] Optionally, the RPC request sending unit 603 is further configured to:

[0113] According to the socket communication channel and the data transmission framework MMKV, the Linux system is used to send a remote procedure call RPC request to the Aidlite-SDK inference framework.

[0114] The deep learning reasoning device 60 can be used to perform Figure 3 The method executed in the embodiment shown in FIG. 6 is therefore the function that can be realized by each functional module of the deep learning reasoning device 60. Figure 3 The description of the illustrated embodiment is not repeated in detail.

[0115] In some possible implementations, various aspects of the method provided in the present application may also be implemented in the form of a program part, which includes a program code. When the program part is run on a computer device, the program code is used to enable the computer device to execute the steps of the method according to various exemplary embodiments of the present application described above in this specification. For example, the computer device may execute the following steps: Figure 3 The method performed in the illustrated embodiment.

[0116] Those skilled in the art can understand that all or part of the steps of the above method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above method embodiment are executed; and the aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, read-only memory (ROM), random access memory (RAM), magnetic disks or optical disks. Alternatively, if the above integrated unit of the present invention is implemented in the form of a software function module and sold or used as an independent part, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present invention can be essentially or in other words, the part that contributes to the prior art can be embodied in the form of a software part, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in various embodiments of the present invention. And the aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROM, RAM, magnetic disks or optical disks.

[0117] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.

[0118] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A deep learning reasoning method, characterized in that: Applied to the Android+Linux fusion system, the method includes: Package and integrate multiple preset AI reasoning frameworks in the Android system to obtain the Aidlite-SDK reasoning framework; Building a socket communication channel between the Aidlite-SDK inference framework and the Linux system; According to the socket communication channel, the Linux system is used to send a remote procedure call RPC request to the Aidlite-SDK reasoning framework; wherein the remote procedure call RPC request includes AI reasoning data; In response to the remote procedure call (RPC) request, the Aidlite-SDK reasoning framework is used to perform AI reasoning, and the corresponding AI reasoning result is fed back to the Linux system; Before building a socket communication channel between the Aidlite-SDK inference framework and the Linux system, the method further includes: Read the model information sent by the Aidlite client; wherein the model information includes the model Modle-related properties, model configuration-related properties, model initialization, loading type, setting model data, inference model, obtaining model data and releasing model as agreed upon by the protocol; Parsing the model information in the Linux system to obtain parsed model information; According to the parsed model information, a socket communication framework for communicating with the Android system is designed in the Linux system by using C++ language or Python language; Encapsulating the socket communication framework to obtain an encapsulated socket communication framework; The step of sending a remote procedure call (RPC) request to the Aidlite-SDK reasoning framework using the Linux system according to the socket communication channel includes: According to the encapsulated socket communication framework, the Linux system is used to send a remote procedure call RPC request to the Aidlite-SDK reasoning framework.

2. The method according to claim 1, characterized in that The step of encapsulating and integrating multiple preset AI reasoning frameworks in the Android system to obtain the Aidlite-SDK reasoning framework includes: Customize the Model class related to the model; wherein the Model class includes the full path of the model file, the path of the model weight parameter, the model input type, the model output type, the shape information of the model input, and the shape information of the model output; Customize the Config class related to parameter configuration; wherein the Config class includes the acceleration type, framework type, output node name, and number of threads of the underlying model call; Customize the Interpreter class related to the model reasoning core; wherein the Interpreter class includes model initialization configuration, model loading, setting model data, model reasoning, and obtaining the inferred data.

3. The method according to claim 1, characterized in that The step of sending a remote procedure call (RPC) request to the Aidlite-SDK reasoning framework using the Linux system according to the socket communication channel includes: According to the socket communication channel and the data transmission framework MMKV, the Linux system is used to send a remote procedure call RPC request to the Aidlite-SDK reasoning framework.

4. The method according to claim 1, characterized in that The multiple preset AI reasoning frameworks include TFLite framework, Snpe framework, PaddlePaddle framework, ONNX framework, QNN framework, RKNN framework, TNN framework and NCNN framework.

5. A deep learning reasoning device, characterized in that: Applied to Android+Linux fusion system, the device includes: The inference framework integration unit is used to package and integrate multiple preset AI inference frameworks in the Android system to obtain the Aidlite-SDK inference framework; A communication channel building unit, used to build a socket communication channel between the Aidlite-SDK reasoning framework and the Linux system; An RPC request sending unit, configured to send a remote procedure call RPC request to the Aidlite-SDK inference framework using the Linux system according to the socket communication channel; wherein the remote procedure call RPC request includes AI inference data; An AI reasoning unit, for responding to the remote procedure call RPC request, performing AI reasoning using the Aidlite-SDK reasoning framework, and feeding back corresponding AI reasoning results to the Linux system; Before building a socket communication channel between the Aidlite-SDK inference framework and the Linux system, it also includes: Read the model information sent by the Aidlite client; wherein the model information includes the model Modle-related properties, model configuration-related properties, model initialization, loading type, setting model data, inference model, obtaining model data and releasing model as agreed upon by the protocol; Parsing the model information in the Linux system to obtain parsed model information; According to the parsed model information, a socket communication framework for communicating with the Android system is designed in the Linux system by using C++ language or Python language; Encapsulating the socket communication framework to obtain an encapsulated socket communication framework; The step of sending a remote procedure call (RPC) request to the Aidlite-SDK reasoning framework using the Linux system according to the socket communication channel includes: According to the encapsulated socket communication framework, the Linux system is used to send a remote procedure call RPC request to the Aidlite-SDK reasoning framework.

6. An electronic device, characterized in that: The device comprises: A memory for storing program instructions; A processor is used to call the program instructions stored in the memory, and execute any method according to claims 1-4 according to the obtained program instructions.

7. A storage medium, characterized in that: The storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute any one of the methods in claims 1-4.