Method and system for reasoning large language model based on Android mobile terminal
By configuring repetition penalties and stopping conditions for negative prompts on the Android side, combined with hardware device handles and thread pool management, we achieved reasonable content length answers and resource savings for large language models, and solved the problem of invalid loop output and performance improvement of large language models in Android.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- HANGZHOU TIANKUAN TECH
- Filing Date
- 2026-01-26
- Publication Date
- 2026-05-19
AI Technical Summary
Existing technologies have issues with the inability to stop answering when reasoning large language models on the Android side. The NPU backend adaptation is not perfect, there is a lot of room for improvement in reasoning performance, and it cannot reason normally with the latest large models such as qwen3-1.7b and qwen3-4b.
By configuring sampling parameters for duplicate penalties, setting stop conditions for reverse prompts and loop inference conditions, obtaining hardware device handles and registration information, creating a thread pool, implementing responses of reasonable content length, and providing a session token reuse mechanism to reduce redundant computation.
It solves the problem of invalid loop inference output in large language models, improves the adaptability of token processing on different backends, significantly improves inference performance and saves resources.
Smart Images
Figure CN122064485A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of large language model reasoning technology, and in particular to a method and system for reasoning large language models based on Android mobile devices. Background Technology
[0002] Currently, the following situations exist in the process of inferring large LLM language models on the Android side: (1) Large models can be run on Android devices using the open-source llama.cpp or MNN-LLM inference framework, and different inference backend runtime libraries can be generated through cross-compilation; (2) Qualcomm's QNN inference framework can be used for high-speed inference. The inference process includes converting the hugging face format model to ONNX format, converting the ONNX format model to QNN IR format using the QNN SDK tool, and converting it to DLC format using Hexagon, thereby generating model_dlc.so and model_dlc.dlc for NPU inference on the Android side. However, the existing technologies mentioned above have the following technical problems: 1. When using the open-source llama.cpp and MNN-LLM framework for inference, the latest large models (qwen3-1.7b, qwen3-4b, etc.) may encounter the problem of being unable to stop answering; 2. When using the open-source llama.cpp and MNN-LLM framework, the inference adaptation to the NPU backend is not very perfect, and sometimes the latest large models cannot be inferred normally; 3. There is a lot of room for improvement in inference performance when using open-source inference frameworks. Summary of the Invention
[0003] One of the objectives of this invention is to provide a method and system for reasoning large language models based on Android mobile devices. The method and system enable the large language model to achieve answers of reasonable content length after updates by configuring sampling parameters for repetition penalties in the sampler, setting stopping conditions for reverse prompts, and setting loop reasoning conditions through three truncation reasoning methods, thus avoiding the problem of the answer being unable to stop due to ineffective loop reasoning output.
[0004] One of the objectives of this invention is to provide a method and system for inference of a large language model based on an Android mobile device. The method and system obtain the handle and registration information of the NPU / GPU / CPU backend hardware device, and construct a batch processing thread pool according to the parameters of the NPU / GPU / CPU backend hardware device to process large-scale tokens adapted to different backends. Furthermore, a single-token thread pool is created, so that the large language model can simultaneously adapt to the processing of single tokens and large-scale tokens from different backend hardware, significantly improving the adaptability of the large language model to the processing of tokens from different backends.
[0005] One of the objectives of this invention is to provide a method and system for inferring large language models on an Android mobile device. The method and system create and maintain an additional thread pool, which is attached to the context. Therefore, this invention eliminates the need to temporarily create and destroy thread pools during the inference process to handle high concurrency in the context. As a result, it can achieve significant resource savings and improve the performance of large language model inference.
[0006] One of the objectives of this invention is to provide a method and system for reasoning large language models on Android mobile devices. The method and system provide a session token reuse mechanism. This mechanism pre-caches some calculation results from previous sessions and compares the word embedding token with the session token. If the two tokens are the same, the calculation results from previous sessions in the cache are used directly. If they are different, the session token is truncated and recalculated. Therefore, this invention can significantly reduce the calculation amount of prompts with the same prefix and significantly improve the processing speed of prompts with the same prefix.
[0007] To achieve at least one of the above-mentioned objectives, the present invention further provides a method for reasoning large language models based on Android mobile devices, the method comprising:
[0008] Initialize the large language model backend library, and create thread pools in batches for the device parameters of the multiple backends of the large language model backend library, and at the same time create a single token thread pool to adapt to different types of backend devices.
[0009] Initialize the large language model sampler, configure sampling parameters including repetition penalties, configure inference loop conditions based on the number of remaining tokens, and configure inference stop conditions with reverse hints.
[0010] An additional thread pool is pre-created and maintained, which is attached to the context, so that the large language model can directly call the additional thread pool when performing contextual reasoning.
[0011] The session calculation result is cached, and it is determined whether the input token and the cached session token are the same. If they are the same, the cached session calculation result is reused.
[0012] According to a preferred embodiment of the present invention, the initialization method of the large language model backend library includes: calling the tk_llm_backend_init instruction to initialize the large language model backend library, and initializing the CPU, GPU and NPU hardware drivers including the backend library; calling the tk_llm_numa_init instruction to initialize non-uniform memory access support; and further calling the common_init_from_params function to load and read the large language model file, allocate memory and create tk_llm_model and tk_llm_context objects; when the large language model is successfully loaded, obtaining the vocabulary of the large language model prompt words and the interactive dialogue template.
[0013] According to another preferred embodiment of the present invention, the thread pool creation method includes: obtaining the handle and structured registration information of the backend library hardware device by calling the tk_llm_core_ggml_backend_dev_by_type instruction, and obtaining the core parameters of each hardware device; creating a batch processing thread pool based on the hardware core parameters to process batch tokens, wherein the core parameters include the number of CPU cores, the number of GPU stream processors, and the number of NPU computing units; and further creating a single token thread pool to process a single token.
[0014] According to another preferred embodiment of the present invention, an additional thread pool is pre-constructed based on the core parameters of the hardware device. The additional thread pool is attached to the inference context by calling the tk_llm_attach_threadpool instruction, which is used to process context information during the inference process and reduce the resource overhead of generating and destroying temporary threads.
[0015] According to another preferred embodiment of the present invention, the large language model sampler initialization method includes: calling the tk_llm_common_sampler_init instruction to create a sampler object, and configuring the following sampler parameters: configuring the temperature parameter to control the randomness of the sampler output, configuring the kernel sampling parameter top_p to calculate the cumulative probability threshold of the token, configuring the top_k sampling parameter to calculate and select the k tokens with the highest probability; and configuring the repeat_penalty sampling parameter to reduce the output of duplicate tokens.
[0016] According to another preferred embodiment of the present invention, the configuration method of the inference loop condition includes: determining the number of remaining tokens in the current large language model based on the value of the n_remain parameter; if the value of the n_remain parameter is not equal to 0 or tokens need to be generated, then performing loop inference for the corresponding large language model; otherwise, jumping to the cleanup phase, wherein the cleanup phase includes executing the following commands in sequence: calling tk_llm_common_sampler_free to release the sampler, calling tk_llm__backend_free to release the backend, calling tk_llm_core_ggml_threadpool_free_fn to release the thread pool, and returning a status code indicating normal exit of the SDK call.
[0017] According to another preferred embodiment of the present invention, the method for configuring the inference stop condition of the reverse hint includes: obtaining the strings of the most recent n tokens, checking whether each reverse hint string appears at the end of the output, and checking whether the last token obtained for a single token reverse hint is an EOG token. If so, the current inference action is stopped, the interactive template is called to output and enter the interactive mode; otherwise, the last token of the strings of the most recent n tokens is checked to see if it is an EOG token. If not, the inference action is continued.
[0018] According to another preferred embodiment of the present invention, the method for reusing the session calculation result includes: pre-caching the m processed token session files into KV key-value pairs as session_Tokens; obtaining the token in the current word embedding and comparing it with the token in session_Tokens; if the token in the current embedding is the same as the token in session_Tokens, then skipping the token decoding step, directly calling the tk_llm_state_load_file instruction to load the previously saved session file, and using the KV key-value pairs to restore the previous dialogue content; otherwise, directly truncating session_Tokens and recalculating the session content.
[0019] To achieve at least one of the above-mentioned objectives, the present invention further provides a large language model system for inference on Android mobile devices, wherein the system executes the above-mentioned method for large language model inference on Android mobile devices.
[0020] The present invention further provides a computer-readable storage medium storing a computer program, which is executed by a processor to implement the above-described method for reasoning large language models based on Android mobile devices. Attached Figure Description
[0021] Figure 1 The diagram shown is a flowchart of a method for reasoning large language models based on Android mobile devices according to the present invention. Detailed Implementation
[0022] The following description is intended to disclose the present invention and enable those skilled in the art to implement it. The preferred embodiments described below are merely examples, and other obvious variations will occur to those skilled in the art. The basic principles of the invention defined in the following description can be applied to other embodiments, modifications, improvements, equivalents, and other technical solutions that do not depart from the spirit and scope of the invention.
[0023] It is understood that the term "a" should be understood as "at least one" or "one or more," that is, in one embodiment, the number of an element can be one, while in another embodiment, the number of the element can be multiple, and the term "a" should not be understood as a limitation on the number.
[0024] Please combine Figure 1 This invention provides a method and system for reasoning large language models on Android mobile devices. The method includes the following steps:
[0025] S01. Initialize the large language model backend library, and create thread pools in batches for the device parameters of the multiple backends of the large language model backend library, and at the same time create a single token thread pool to adapt to different types of backend devices.
[0026] S02. Initialize the large language model sampler, configure sampling parameters including duplicate penalties, configure inference loop conditions based on the number of remaining tokens, and configure inference stop conditions for reverse hints.
[0027] S03. Pre-create and maintain an additional thread pool, which is attached to the context so that the large language model can directly call the additional thread pool when performing contextual reasoning.
[0028] S04. Cache the session calculation result and determine whether the input Token and the cached session Token are the same. If they are the same, reuse the cached session calculation result.
[0029] Specifically, the method further includes the following technical solutions: First, the present invention can use, but is not limited to, SDK calls to call and parse the large language model. The SDK call method includes calling the `tk_llm_common_params_parse` command to parse command-line parameters. These command-line parameters include relevant configuration parameters of the large language model itself, specifically including: model path (model), prompt word (prompt), system prompt word (system_prompt), context size (ctx_size), number of generated tokens (n), sampling parameters (temperature, top-p, etc.), dialogue mode switch (cnv / -no-cnv), etc. After parsing, the general module is further initialized, and the processing logic for relevant inputs is set. The present invention further verifies the above-mentioned configuration parameters, including but not limited to checking whether the context size (ctx_size) is at least 8, checking the RoPE (RotaryPosition Embedding) frequency parameter, and outputting warning messages (e.g., whether the model path exists) if there are any abnormal values.
[0030] It is worth mentioning that this invention configures backend libraries adapted to different types of hardware during the initialization of the large language model. The configuration method includes: calling `tk_llm_backend_init` to initialize the tk-llm backend library, and using `tk_llm_backend_init` to set the global state and initialize the hardware drivers for various backend libraries, such as CPU, GPU, and NPU. In one preferred embodiment of this invention, it checks whether the backend library hardware device has an NPU or GPU; if so, inference is preferentially performed on the NPU or GPU. Further, the `tk_llm_numa_init` instruction is called to initialize NUMA (Non-Uniform Memory Access) support, optimizing memory access performance on multi-processor systems. Then, the model file is loaded through the `common_init_from_params` function interface. It is worth noting that the framework for loading model files in this invention is based on .gguf format model files with a ggml structure. The `common_init_from_params` function interface can perform the following actions: reading the model file, allocating memory, and creating `tk_llm_model` and `tk_llm_context` objects. If model loading fails, the program will output an error message and exit. After successful loading, you can directly obtain the model's vocabulary (tk_llm_model_get_vocab) and the initialized interactive templates (tk_llm_common_chat_templates_init).
[0031] It is worth mentioning that this invention will configure the thread pool as follows: The `tk_llm_core_ggml_backend_dev_by_type` command will be called to obtain the handle of the NPU / GPU / CPU backend hardware device; and the `tk_llm_core_ggml_backend_dev_by_type` command will be used to obtain the registration structure information of the backend hardware device. Furthermore, a batch processing thread pool will be created based on core hardware parameters including the number of CPU cores, GPU stream processors, and NPU computing units. This batch processing thread pool is used to process large numbers of tokens. Additionally, a single-token thread pool will be created to process a single token. When multiple tokens with the same name exist, only one thread pool is created for that token, allowing tokens with the same name to be reused by the same thread pool without additional computation, effectively saving computing resources. Based on the creation and configuration of the above thread pools, this invention can greatly utilize the adaptation processing of different hardware devices and the processing of large batches of tokens, as well as the processing of small numbers of tokens. This allows large language models to adapt to various unexpected token situations during iterative releases, avoiding the technical problem of large language models being unable to reason after iteration.
[0032] Furthermore, this invention also provides a pre-configured additional thread pool, wherein the additional thread pool is attached to the inference context by calling the `tk_llm_attach_threadpool` instruction to process context information during inference, reducing the resource overhead of creating and destroying temporary threads. Those skilled in the art will understand that creating and destroying threads in a thread pool consumes significant resources. Therefore, this invention, by maintaining an additional thread pool, can significantly reduce the number of temporary threads created and destroyed during the context processing of a large language model, thereby greatly reducing resource consumption during inference. This allows computational resources to be concentrated on more core inference parts, effectively improving the inference performance of large language models.
[0033] Furthermore, this invention provides a session management technique for large language models, particularly a session reuse mechanism. This mechanism includes the following components: First, pre-processed m token session files are cached in key-value pairs as previous session tokens. The token in the current word embedding is retrieved and compared with the token in session tokens. If the token in the current embedding is the same as the token in session tokens, the token decoding step is skipped, and the previously saved session file is loaded directly using the tk_llm_state_load_file instruction. The previous dialogue content is then restored using the key-value pairs. Otherwise, session tokens are truncated, and the session content is recalculated. When the tk_llm_state_load_file instruction is called, the key-value cache and context state are obtained, thus enabling the reuse of previous session content without computation, effectively saving computational resources.
[0034] The session management further provides an interaction mode and a raw mode. The technical details of the interaction mode include: if a system prompt is input, it is formatted as a chat message using `tk_llm_common_chat_format_single`; if a user prompt is input, it is also formatted as a chat message; further, `tk_llm_common_chat_templates_apply` is called to obtain the application chat interaction template and generate a complete prompt that conforms to the model's expected format; for example, for the ChatML format, the following interaction template prompt can be generated: `<|im_start|>system\n{system_prompt}<|im_end|>\n<|im_start|>user\n{user_prompt}<|im_end|>\n<|im_start|>assistant\n`
[0035] If the current mode is not interactive, the large language model directly uses the original prompts for reasoning.
[0036] This invention further provides a method for initializing a large language model sampler. The method includes: calling the `tk_llm_common_sampler_init` instruction to create a sampler object and configuring the following sampler parameters: configuring a `temperature` parameter to control the randomness of the sampler output; configuring a kernel sampling parameter `top_p` to calculate the cumulative probability threshold of tokens; configuring a `top_k` sampling parameter to calculate and select the k tokens with the highest probabilities; and configuring a `repeat_penalty` sampling parameter as a repetition penalty parameter to reduce the output of duplicate tokens. The `repeat_penalty` parameter can be set to 1-1.5, with a larger value resulting in less repeated content.
[0037] The present invention also provides an encoder model processing technique, wherein the encoder model processing technique method includes:
[0038] If the large language model has an encoder (such as T5, BART, etc. encoder-decoder models): first call tk_llm_model_encode to encode the input; clear the input token sequence; add the decoder start token; if it is a pure decoder model (such as GPT), skip the encoding step directly.
[0039] The present invention further provides the following method for loop reasoning and provides a stopping condition for loop reasoning: the number of remaining tokens in the current large language model is determined according to the value of the n_remain parameter. If the value of the n_remain parameter is not equal to 0 or tokens need to be generated, the loop reasoning of the corresponding large language model is executed; otherwise, the large language model stops reasoning and jumps to the cleanup phase.
[0040] It should be noted that the present invention requires the following context management method during the loop reasoning process: When there are tokens to be processed, context management is performed first. The management method includes: determining whether the context exceeds the maximum length; when the context length is full, a context shifting action is performed, which includes first calling tk_llm_memory_seq_rm to delete the middle part of the KV cache, and then calling tk_llm_memory_seq_add to move the position of the subsequent tokens forward; or an extended uplink length action is performed, which includes shifting the token position by tk_llm_memory_seq_add, then scaling the position encoding by tk_llm_memory_seq_div, and finally calling tk_llm_memory_seq_add again to adjust the position.
[0041] Following the aforementioned contextual reasoning actions, the obtained batch tokens need to undergo the following batch decoding process: tk_llm_model_decode performs the actual reasoning calculations. The specific operation flow is as follows:
[0042] First, the embedding layer is searched, then the Transformer layer is forward propagated, attention is calculated, the feedforward network is calculated, the KV cache is updated, and the logits (log probability) of the last token is calculated. For each batch of tokens processed, n_past is increased by the corresponding number, representing the total number of tokens processed. If session caching is enabled, the newly processed tokens are added to session_Tokens.
[0043] Further, check whether all input tokens have been processed:
[0044] If the tokens have been exhausted, a new token needs to be generated to initiate the sampling process:
[0045] (1) When sampling a new token, call tk_llm_common_sampler_sample
[0046] (2) Accept the sampling results: call tk_llm_common_sampler_accept
[0047] (3) Add the new token to the sampler’s history.
[0048] (4) Update the duplicate penalty context
[0049] (5) Finally, add the sampled token to the embedding, and decrement the remaining number of tokens by 1.
[0050] If the token is not used up, another token needs to be entered and processed:
[0051] Retrieve the next token from embedding_inp and call tk_llm_common_sampler_accept to add the input token to the sampler context (for duplicate penalty).
[0052] It should be noted that the configuration method for the inference stopping condition of the reverse hint described in this invention includes: obtaining the strings of the most recent n tokens, wherein the strings of the most recent 32 tokens are preferred in this invention; checking whether each reverse hint string appears at the end of the output; and simultaneously checking whether the last token obtained for a single token reverse hint is an EOG token. If so, the current inference action is stopped, the interactive template is invoked, and the output enters the interactive mode. Otherwise, the last token of the strings of the most recent n tokens is checked to see if it is an EOG token. If not, the inference action continues. By recognizing the EOG token of the reverse hint and its position, the output of a large language model can be effectively truncated, reducing disordered and unstoppable output results.
[0053] The present invention further provides the following processing operations in the cleaning phase:
[0054] When the exit conditions are met (including but not limited to completion of generation, user logout, encounter of EOG Token, etc.), the cleanup phase begins, which includes the following four parts:
[0055] 1. Save the session
[0056] If a session file is specified and the saving conditions are met:
[0057] Call tk_llm__state_save_file;
[0058] Save the complete session state, including: token sequence; key-value cache; context state.
[0059] II. Performance Statistics:
[0060] Call tk_llm__common_perf_print to print detailed performance statistics:
[0061] Total number of tokens processed; number of prompt tokens and number of tokens generated; processing speed (Tokens / second); total time elapsed.
[0062] III. Resource Release:
[0063] Release resources sequentially to prevent memory leaks:
[0064] (1) Call tk_llm_common_sampler_free to release the sampler.
[0065] (2) Call tk_llm__backend_free to release the backend.
[0066] (3) Call tk_llm_core_ggml_threadpool_free_fn to release the thread pool.
[0067] IV. End of SDK call
[0068] A status code of 0 indicates a normal exit.
[0069] The processes described in the flowcharts above, as disclosed in the embodiments of this invention, can be implemented as computer software programs. Embodiments of this invention include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication component, and / or installed from a removable medium. When the computer program is executed by a central processing unit (CPU), the methods of this application are not limited to the aforementioned functions. It should be noted that the computer-readable medium described above can be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. Computer-readable storage media can be, for example, but not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wire segments, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on a computer-readable medium may be transmitted using any suitable medium, including but not limited to: wireless segments, wire segments, optical fibers, RF, etc., or any suitable combination thereof.
[0070] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.
[0071] Those skilled in the art should understand that the embodiments of the present invention described above and shown in the accompanying drawings are merely examples and do not limit the present invention. The purpose of the present invention has been fully and effectively achieved. The functions and structural principles of the present invention have been shown and explained in the embodiments. Without departing from the stated principles, the implementation of the present invention may have any variations or modifications.
Claims
1. A method for reasoning large language models based on Android mobile devices, characterized in that, The method includes: Initialize the large language model backend library, and create thread pools in batches for the device parameters of the multiple backends of the large language model backend library, and at the same time create a single token thread pool to adapt to different types of backend devices. Initialize the large language model sampler, configure sampling parameters including repetition penalties, configure inference loop conditions based on the number of remaining tokens, and configure inference stop conditions with reverse hints. An additional thread pool is pre-created and maintained, which is attached to the context, so that the large language model can directly call the additional thread pool when performing contextual reasoning. The session calculation result is cached, and it is determined whether the input token and the cached session token are the same. If they are the same, the cached session calculation result is reused.
2. The method for reasoning large language models based on Android mobile devices according to requirement 1, characterized in that, The initialization method of the large language model backend library includes: calling the tk_llm_backend_init instruction to initialize the large language model backend library, and initializing the CPU, GPU and NPU hardware drivers including the backend library; calling the tk_llm_numa_init instruction to initialize non-uniform memory access support; and further calling the common_init_from_params function to load and read the large language model file, allocate memory and create tk_llm_model and tk_llm_context objects. When the large language model is successfully loaded, the vocabulary of the large language model prompt words and the interactive dialogue template are obtained.
3. A method for reasoning large language models based on Android mobile devices according to requirement 1, characterized in that, The thread pool creation method includes: obtaining the handle and structured registration information of the backend library hardware device by calling the tk_llm_core_ggml_backend_dev_by_type instruction, and obtaining the core parameters of each hardware device; creating a batch processing thread pool based on the hardware core parameters to process batch tokens, wherein the core parameters include the number of CPU cores, the number of GPU stream processors, and the number of NPU computing units; and further creating a single token thread pool to process a single token.
4. A method for reasoning large language models based on Android mobile devices according to requirement 1, characterized in that, An additional context pool is pre-built based on the core parameters of the hardware device. The additional thread pool is attached to the inference context by calling the tk_llm_attach_threadpool instruction. This is used to process context information during the inference process and reduce the resource overhead of generating and destroying temporary threads.
5. A method for reasoning large language models based on Android mobile devices according to requirement 1, characterized in that, The large language model sampler initialization method includes: calling the tk_llm_common_sampler_init instruction to create a sampler object and configuring the following sampler parameters: configuring the temperature parameter to control the randomness of the sampler output, configuring the kernel sampling parameter top_p to calculate the cumulative probability threshold of the token, configuring the top_k sampling parameter to calculate and select the k tokens with the highest probability; and configuring the repeat_penalty sampling parameter to reduce the output of duplicate tokens.
6. A method for reasoning large language models based on Android mobile devices according to requirement 1, characterized in that, The configuration method for the inference loop condition includes: determining the number of remaining tokens in the current large language model based on the value of the n_remain parameter; if the value of the n_remain parameter is not equal to 0 or tokens need to be generated, then the loop inference of the corresponding large language model is executed; otherwise, the process jumps to the cleanup phase. The cleanup phase includes executing the following commands in sequence: calling tk_llm_common_sampler_free to release the sampler, calling tk_llm__backend_free to release the backend, calling tk_llm_core_ggml_threadpool_free_fn to release the thread pool, and returning a status code indicating that the SDK call exited normally.
7. A method for reasoning large language models based on Android mobile devices according to requirement 1, characterized in that, The configuration method for the reasoning stop condition of the reverse hint includes: obtaining the strings of the most recent n tokens, checking whether each reverse hint string appears at the end of the output, and checking whether the last token obtained for a single token reverse hint is EOGToken. If it is, stop the current reasoning action, call the interactive template output, and enter the interactive mode. Otherwise, continue to check whether the last token of the strings of the most recent n tokens is EOG Token. If not, continue to execute the reasoning action.
8. A method for reasoning large language models based on Android mobile devices according to requirement 1, characterized in that, The method for reusing the session calculation results includes: pre-caching the m processed token session files into KV key-value pairs as session_Tokens; obtaining the token in the current word embedding and comparing it with the token in session_Tokens; if the token in the current embedding is the same as the token in session_Tokens, skipping the token decoding step and directly calling the tk_llm_state_load_file instruction to load the previously saved session file, and using the KV key-value pairs to restore the previous dialogue content; otherwise, directly truncating session_Tokens and recalculating the session content.
9. A large language model system for inference based on Android mobile devices, characterized in that, The system executes the Android mobile terminal reasoning large language model method according to any one of claims 1-8.
10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that is executed by a processor to implement the method for reasoning large language models based on an Android mobile terminal as described in any one of claims 1-8.