A large language model fast reasoning method and system based on an Android system deployment

Abstract

The application discloses a large language model fast reasoning method and system based on an Android system deployment, and comprises the following steps: constructing an OpenGL environment in a PC terminal, cloning a large language model reasoning framework in the OpenGL environment, selecting GPU hardware acceleration in the OpenGL environment, and obtaining an so file after compiling and converting the large language model reasoning framework and the hardware acceleration; acquiring a large language model resource file, and converting and quantifying the large language model resource file into a GGUF format file; preparing an Android NDK environment by using an Android development component, converting the GGUF format file of the large language model into an LLM.so file in the Android NDK environment, and configuring dynamic library linkage in the Android NDK environment; creating an Android NDK project, configuring GPU calling instructions of the large language model reasoning in the Android NDK project, starting the large language model reasoning, and running in a CPU of the corresponding PC terminal.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of large language model deployment and reasoning, in particular to a large language model fast reasoning method and system based on Android system deployment. BACKGROUND

[0002] At present, the traditional large language model is deployed on the PC end and uses the CPU of the PC end for reasoning. The large language model deployed and reasoned on the PC end has the problems of inconvenience to carry and the PC end needs to be in a network connection environment to perform efficient reasoning, so that the network security of the PC end large language model reasoning is low. In addition, the traditional PC end large language model reasoning is in a network environment and only relies on the CPU of the PC end for reasoning, so that the efficiency of the large language model reasoning is low. Therefore, how to fully utilize the PC end GPU for efficient reasoning in a safe offline state is a technical problem to be solved at present. SUMMARY

[0003] One of the purposes of the present application is to provide a large language model fast reasoning method and system based on Android system deployment. The method and system construct an OpenGL environment on the PC end. The OpenGL is an Open Graphics Library, which is a cross-platform GPU graphics rendering development standard interface and can be used for graphics rendering and large language model reasoning calling, but not limited to. The present application accelerates the GPU of the PC end system by using the constructed OpenGL environment, so that the large language model reasoning fully utilizes the PC end GPU, thereby greatly improving the efficiency of the large language model reasoning on the PC end.

[0004] One of the purposes of the present application is to provide a large language model fast reasoning method and system based on Android system deployment. The method and system constructs an Android NDK environment on the PC end. The NDK is a Native Development Kit, which is an Android-based native development kit. It can compile and generate Android code running on the PC end. In the Android NDK environment, the large language model file downloaded and standardized format converted is cross-compiled into an so file meeting the Android NDK environment, and the so file is loaded by the dynamic link of the Android system, and the GPU reasoning calling of the PC end is performed according to the so file.

[0005] One of the purposes of the present application is to provide a large language model fast reasoning method and system based on Android system deployment, which pre-downloads Android integrated development tool Android Studio on the PC end, and creates an Android system on the PC end using the Android integrated development tool Android Studio. Since the Android system has OpenGL built-in, the present application can use Android XML to open OpenGL-related components, so that the so file containing the corresponding large language model reasoning file can be run through the OpenGL calling PC end GPU, thereby significantly improving the efficiency of large language model reasoning. Offline reasoning ensures the security of reasoning data.

[0006] To achieve at least one of the above-mentioned purposes, the present application further provides a large language model fast reasoning method based on Android system deployment, which comprises:

[0007] An OpenGL environment is constructed in the PC end, and a large language model reasoning framework is cloned in the OpenGL environment. The large language model reasoning framework and hardware acceleration are compiled and converted to obtain an so file.

[0008] A large language model resource file is obtained, and the large language model resource file is converted and quantized into a GGUF format file.

[0009] An Android NDK environment is prepared using an Android development component, and the GGUF format file of the large language model is converted into an LLM.so file in the Android NDK environment. The dynamic library link is configured in the above-mentioned Android NDK environment.

[0010] An Android NDK project is created, and GPU calling instructions for large language model reasoning are configured in the Android NDK project. The GPU calling instructions are used to start large language model reasoning, so that the large language model runs in the corresponding PC end GPU.

[0011] According to one of the preferred embodiments of the present application, the NDK environment preparation method comprises: downloading Android integrated development tool Android Studio, and selecting build tool CMake in SDK Manager. The NDK environment is generated according to the build tool CMake, and the application binary interface ABI of the PC end processor is configured in the NDK environment. The environment variable of the NDK environment is configured using the build tool CMake, wherein the environment variable includes the NDK storage path.

[0012] According to another preferred embodiment of the present application, the method for cloning the large language model inference framework comprises the following steps: installing a Git tool in the OpenGL environment, installing a compiler and a build tool CMake, obtaining a network address of a large language model inference framework, cloning the corresponding large language model inference framework into a specified directory through a cloning instruction by using the compiler and the build tool CMake, adding a hardware acceleration instruction to the PC end through the build tool CMake, and performing a compilation operation on the cloned large language model inference framework and the hardware acceleration instruction through the build tool CMake to generate an executable so file in a bin subdirectory.

[0013] According to another preferred embodiment of the present application, the method for converting the GGUF format file comprises the following steps: obtaining a large language model resource file, parsing the large language model to obtain key parameters of the large language model, wherein the key parameters comprise a hidden layer size, an intermediate layer size, a number of hidden layers, and a number of attention heads, and performing tensor mapping processing of the GGUF name on the key parameters and corresponding weights according to a TensorNameMap to obtain the GGUF format file after quantization processing.

[0014] According to another preferred embodiment of the present application, the method for cross-compiling comprises the following steps: obtaining a PC end processor source code, parsing the source code, and constructing an Android system interface layer Sysroot, the Android system interface layer Sysroot obtains an Android system header file and a special library file, and a key difference adaptation rule is configured through a binary interface ABI, wherein the key difference adaptation rule of the binary interface ABI comprises a byte sequence adaptation rule, a function call convention adaptation rule, and a memory alignment adaptation rule, which are used to adapt the corresponding PC end processor source code to the Android system to obtain an LLM.so file.

[0015] According to another preferred embodiment of the present application, the method for cross-compiling comprises the following steps: locating a tool chain in a configuration stage of compilation, and setting a pointer to a corresponding header file and an OpenGL library of the Android system through an Android system interface layer Sysroot, encoding the PC end processor source code into an architecture-independent intermediate code in an encoding stage, and converting and compiling the PC end processor source code to the Android system code through a binary interface ABI adaptation rule mark injection.

[0016] According to another preferred embodiment of the present application, the method for building an Android NDK project comprises the following steps: installing a tool chain through an Android integrated development tool Android Studio, the tool chain comprises an NDK version, a tool type, and an SDK, wherein the tool type is configured as a build tool CMake, and the SDK is further configured as a command line tool for including a corresponding large language model loading command.

[0017] According to another preferred embodiment of the present application, the CMakeLists.txt file is configured by using the build tool CMake, the library dynamic library is added by using the CMakeLists.txt, and the LLM.so file is loaded by dynamic library link loading, wherein LibOpenGL.so is configured by using Android xml, LLM.so instructions are generated after the LLM.so file is loaded, and the PC end GPU is called according to the LLM.so instructions to perform inference of the large language model.

[0018] In order to achieve at least one of the above-mentioned purposes, the present application further provides a large language model fast inference system deployed based on an Android system, wherein the system executes the above-mentioned large language model fast inference method deployed based on an Android system.

[0019] The present application further provides a computer readable storage medium, wherein the computer readable storage medium stores a computer program, and the computer program is executed by a processor to realize the above-mentioned large language model fast inference method deployed based on an Android system. BRIEF DESCRIPTION OF DRAWINGS

[0020] Fig. 1 A large language model fast inference method deployed based on an Android system is shown in the whole flowchart.

[0021] Fig. 2 An Android file generation flowchart in the present application is shown.

[0022] Fig. 3 A GPU pulling method based on an Android file in the present application is shown in the flowchart. DETAILED DESCRIPTION

[0023] The following description is provided to disclose the present application so that those skilled in the art can implement the present application. The preferred embodiments in the following description are only examples, and other obvious modifications can be made by those skilled in the art. The basic principles defined in the following description can be applied to other embodiments, modifications, improvements, equivalents and other technical solutions without departing from the spirit and scope of the present application.

[0024] It can be understood that the term "one" should be understood as "at least one" or "one or more", that is, in one embodiment, the number of one element can be one, and in another embodiment, the number of the element can be multiple, and the term "one" cannot be understood as a limitation on the number.

[0025] Please refer to Figs. 1-3The application discloses a large language model fast reasoning method and system based on an Android system deployment, and mainly comprises the following steps:

[0026] S01, an OpenGL environment is constructed in a PC terminal, a large language model reasoning framework is cloned in the OpenGL environment, GPU hardware acceleration is selected in the OpenGL environment, and after the large language model reasoning framework and hardware acceleration are compiled and converted, an so file is obtained;

[0027] S02, a large language model resource file is obtained, and the large language model resource file is converted and quantified into a GGUF format file;

[0028] S03, an Android NDK environment is prepared by using an Android development component, the GGUF format file of the large language model is converted into an LLM.so file in the Android NDK environment, and dynamic library linking is configured in the Android NDK environment;

[0029] S04, an Android NDK project is created, GPU calling instructions of large language model reasoning are configured in the Android NDK project, large language model reasoning is started according to the GPU calling instructions, and the large language model is enabled to run in a corresponding PC terminal GPU.

[0030] Specifically, in one of the preferred embodiments of the application, the construction method of the OpenGL environment comprises the following steps: first, a graphics card driver is downloaded in the PC terminal, wherein the graphics card driver includes but is not limited to NVIDI A, AMD and Intel, an OpenGL header file and a library are installed by using a package manager Vcpkg, a window management library GLFW is installed by using the package manager Vcpkg, an IDE (integrated development environment) is further configured, a project is created and a dependency is added through the IDE (integrated development environment), wherein the added dependency includes a project attribute and a corresponding library directory, for example, the added dependency can be opengl32.lib and glfw3.lib, wherein the opengl32.lib is a core library file, used for declaration and definition of OpenGL core functions, and actual function code is generated through a dynamic link library opengl32.dll. The glfw3.lib is a static link library, providing a window management function, used for management of input boxes, mice and keyboards. In this embodiment, the core of the added dependency is the opengl32.lib, used for generating execution code of corresponding OpenGL core functions.

[0031] Further, after the construction of the OpenGL environment is completed, the application further performs large language model framework cloning in the OpenGL environment and opens the hardware acceleration function, and the specific method comprises the following steps: downloading an Android integrated development tool Android Studio, selecting a build tool CMake in an SDK Manager, generating the NDK environment according to the build tool CMake, and configuring an application binary interface ABI of the PC end processor in the NDK environment; the environment variable of the NDK environment is configured by using the build tool CMake, wherein the environment variable comprises an NDK storage path. When the cloning command is executed, a llama.cpp folder is created as a cloning project, and an option including but not limited to -DLLAMA_CUBLAS=ON is selected in a general command as a corresponding GPU hardware acceleration instruction. Since different GPUs correspond to different hardware acceleration instructions, the application will not repeat the description. The NDK environment provides an editing function for the Android underlying code language, and then a so file can be generated for the Android system NDK project, so that the application can realize large language model reasoning based on the Android underlying code, and the efficiency of reasoning is improved.

[0032] The conversion method of the GGUF format file comprises the following steps: obtaining a large language model resource file, and analyzing the large language model to obtain key parameters of the large language model, wherein the key parameters comprise a hidden layer size, an intermediate layer size, a number of hidden layers and a number of attention heads, and the key parameters and corresponding weights are subjected to tensor mapping processing of a GGUF name according to a TensorNameMap, and the GGUF format file is obtained after quantization processing.

[0033] Further, the application further needs to cross-compile the GGUF format file after completing the GGUF format file conversion of the large language model, so that the code generated on the PC end can be converted and used in the environment of the installed system to meet the compatibility of the Android system. The cross-compilation method comprises the following steps: obtaining PC end processor source code, parsing the source code, and constructing an Android system interface layer Sysroot. The Android system interface layer Sysroot obtains an Android system header file and a special library file, and configures key difference adaptation rules through the binary interface ABI. The key difference adaptation rules of the binary interface ABI include byte sequence adaptation rules, function call convention adaptation rules, and memory alignment adaptation rules, which are used to adapt the PC end processor source code to the Android system to obtain an LLM.so file. The cross-compilation method further comprises the following steps: positioning the tool chain during the configuration stage of the compilation, and setting the corresponding header file and OpenGL library of the Android system through the Android system interface layer Sysroot. During the coding stage, the PC end processor source code is coded into architecture-independent intermediate code, and the adaptation rules of the binary interface ABI are marked and injected to convert and compile the Android system code.

[0034] The Android NDK project construction method comprises the following steps: installing a tool chain through an Android integrated development tool Android Studio, wherein the tool chain comprises an NDK version, a tool type, and an SDK. The tool type is configured as a construction tool CMake, and the SDK is further configured as a command line tool, which is used to include a corresponding large language model loading command.

[0035] In one of the preferred embodiments of the application, the CMakeLists.txt file is configured by using the construction tool CMake, the library dynamic library is added by using the CMakeLists.txt, and the LLM.so file is loaded through dynamic library linking. The LibOpenGL.so is configured by using the Android xml, and the LLM.so instruction is generated after the LLM.so file is loaded. The PC end GPU is called according to the LLM.so instruction to perform inference of the large language model. It should be noted that the LibOpenGL library file is opened by using the configuration file of the Android xml in the application, so that the related so file can run in the corresponding PC end GPU. The configuration file code is <uses-library android:name="libOpenCL.so" android:required="true" / > .

[0036] The processes described above with reference to the flowcharts can be implemented as computer software programs in accordance with embodiments of the present disclosure. Embodiments of the present disclosure include a computer program product comprising a computer program carried on a computer readable medium, the computer program comprising program code for performing the methods illustrated by the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication section, and / or installed from a detachable medium. When the computer program is executed by a central processing unit (CPU), the above-described functions are performed in the methods of the present application. It should be noted that the computer readable medium described above in the present application can be a computer readable signal medium or a computer readable storage medium or any combination of the two. The computer readable storage medium may, for example, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or apparatus, or any combination of the above. More specific examples of the computer readable storage medium can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, the computer readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, device or apparatus. In the present application, the computer readable signal medium can include a data signal carried in a baseband or as a part of a carrier wave, which carries computer readable program code. Such a propagated data signal can take on many forms, including but not limited to, an electromagnetic signal, an optical signal, or any suitable combination thereof. The computer readable signal medium can also be any computer readable medium that can send, propagate or transfer a program for use by or in connection with an instruction execution system, device or apparatus. The program code contained on the computer readable medium can be transmitted by any suitable medium, including but not limited to, wireless, wire, optical cable, RF, or the like, or any suitable combination thereof.

[0037] The computer program product of the present application can be a computer program product comprising a computer-readable medium bearing computer program code embodied therein for use with a computer. The computer program code can be code defining and / or implementing the present application. The computer program code can be written in any suitable computer readable programming language. The computer program code can be stored in a computer- readable storage medium, such as, but not limited to, any type of disk including an optical disk, a CD-ROM, a CD-R, a CD-RW, a DVD, a flash memory, a ROM, a RAM, a magnetic disk or hard drive, or any other suitable type of medium including a medium that holds the software for a particular or specialized computing purpose, or any suitable combination of media. The computer program product can be a computer program product distributed to end users, whether as a stand-alone program, as part of a physical system, or as a software download. The computer program product can be distributed on a physical medium, such as, but not limited to, a floppy disk, a CD-ROM, a CD-R, a CD-RW, a DVD, a flash memory, a ROM, a RAM, a magnetic disk or hard drive, or any other suitable type of medium, or any suitable combination of media. The computer program product can be distributed from a program distribution center, either as a tangible medium or via electronic delivery, such as from a Web site via the Internet, or from one computer to another via electronic transfer, such as by e-mail. The computer program product can be distributed in an encrypted manner, such as via encryption or via password protection.

[0038] Those skilled in the art will understand that the application described above and illustrated in the accompanying drawings is presented by way of example only and is not limiting as to the present application. The intent is to cover all modifications and alternatives of the present application falling within the scope of the application.

Claims

1. A large language model fast inference method based on an Android system deployment, characterized in that, The method comprises: An OpenGL environment is constructed in a PC terminal, a large language model inference framework is cloned in the OpenGL environment, GPU hardware acceleration is selected in the OpenGL environment, and after the large language model inference framework and hardware acceleration are compiled and converted, an so file is obtained for use by an Android system NDK project; A large language model resource file is obtained, and the large language model resource file is converted and quantized into a GGUF format file; An Android NDK environment is generated by using an Android development component, the GGUF format file of the large language model is cross-compiled in the Android NDK environment to convert it into an LLM.so file, and a dynamic library link is configured in the above-mentioned Android NDK environment; An Android NDK project is created, GPU calling instructions for large language model inference are configured in the Android NDK project, and according to the GPU calling instructions, large language model inference is started, so that the large language model runs in the corresponding PC terminal GPU.

2. The method according to claim 1, wherein, The generation of the NDK environment comprises: downloading an Android integrated development tool Android Studio, selecting a build tool CMake in an SDK Manager, generating the NDK environment according to the build tool CMake, and configuring an application binary interface ABI of the PC terminal processor in the NDK environment; and configuring an environment variable of the NDK environment by using the build tool CMake, wherein the environment variable comprises an NDK storage path.

3. The method of claim 1, wherein the method is based on an Android system deployment of a large language model rapid inference method. The cloning of the large language model inference framework comprises: installing a Git tool in the OpenGL environment, installing a compiler and a build tool CMake, obtaining a network address of a large language model inference framework, cloning a corresponding large language model inference framework into a specified directory by using the compiler and the build tool CMake through a cloning instruction, adding a hardware acceleration instruction in the PC terminal by using the build tool CMake, and performing a compilation operation on the cloned large language model inference framework and the hardware acceleration instruction by using the build tool CMake to generate an executable so file in a bin subdirectory.

4. The method of claim 1, wherein the method is based on an Android system. The conversion of the GGUF format file comprises: obtaining a large language model resource file, and analyzing the large language model to obtain key parameters of the large language model, wherein the key parameters comprise: a hidden layer size, an intermediate layer size, a number of hidden layers, and a number of attention heads; and performing tensor mapping processing of the GGUF name on the key parameters and corresponding weights according to a TensorNameMap, and obtaining the GGUF format file through quantization processing.

5. The method according to claim 2, wherein, The cross-compilation includes: obtaining PC processor source code, parsing the source code, and constructing an Android system interface layer Sysroot. The Android system interface layer Sysroot obtains Android system header files and special library files, and configures key difference adaptation rules through the binary interface ABI, wherein the key difference adaptation rules of the binary interface ABI include byte sequence adaptation rules, function call convention adaptation rules, and memory alignment adaptation rules, which are used to adapt the corresponding PC processor source code to the Android system to obtain an LLM.so file.

6. The method of claim 5, wherein the method is based on an Android system. The cross-compilation includes: locating the tool chain during the configuration phase of compilation, and setting a pointer to the corresponding header files and OpenGL library of the Android system through the Android system interface layer Sysroot. During the coding phase, the PC processor source code is coded into architecture-independent intermediate code, and is converted and compiled into the Android system code through the adaptation rules of the binary interface ABI.

7. The method of claim 1, wherein the method is based on an Android system deployment of a large language model rapid inference method. The creation of the Android NDK project includes: installing a tool chain through an Android integrated development tool Android Studio, the tool chain including an NDK version, a tool type, and an SDK, wherein the tool type is configured as a build tool CMake, and the SDK is further configured as a command line tool for including a corresponding large language model loading command.

8. The method of claim 5, wherein the method is based on an Android system. The build tool CMake is configured by using the CMakeLists.txt file, the CMakeLists.txt is used for adding a dynamic library, and the LLM.so file is loaded through dynamic library linking. The LibOpenGL.so is configured by using Android xml, and after the LLM.so file is loaded, an LLM.so instruction is generated, and the PC end GPU is called according to the LLM.so instruction to perform inference of the large language model.

9. A large language model fast inference system deployed based on an Android system, characterized in that, The system implements the large language model fast inference method based on the Android system deployment in any one of claims 1-8.

10. A computer readable storage medium, the computer readable storage medium storing a computer program, the computer program being executed by a processor to implement the large language model fast inference method based on the Android system deployment in any one of claims 1-8.

Images