Question and answer method of end-side large language model, intelligent device and storage medium

Abstract

The application discloses a question and answer method of an end-side large language model, an intelligent device and a storage medium, relates to the technical field of artificial intelligence, and applies to an intelligent device locally deployed with at least two large language models, and comprises the following steps: when a request to be replied to of a user is received, performance state data of the intelligent device is acquired; the current task complexity is determined according to the request to be replied to; the target reasoning model is determined from the at least two large language models with different model parameter quantities according to the current task complexity and the performance state data; the request to be replied to is input into the target reasoning model, and a current answer is obtained. According to the task complexity and the device performance state, the application dynamically selects a large language model with adaptive parameters to perform reasoning, reduces power consumption and calculation overhead on the premise of ensuring the answer quality, thereby prolonging the device endurance, avoiding interruption of the question and answer service due to excessively high load of the device, and achieving balance between the device performance state and the task processing demand.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a question-answering method, intelligent device, and storage medium for a large language model on an edge. Background Technology

[0002] With the development of artificial intelligence technology, large language models have demonstrated strong processing capabilities in user interaction tasks. Extending these capabilities to mobile scenarios has led to the development of smart devices such as AI-powered recording power banks, power banks with screens, AI-powered recording earphone cases, meeting recording devices, and voice recorders. When using these devices, users can send them requests for a response, and the devices will respond to those requests.

[0003] However, these devices are battery-powered and limited in size, and their processor computing power is subject to limitations. In existing methods, devices typically process all pending request tasks using the same large language model. When the device is performing well, calling the large model for simple tasks leads to unnecessary resource consumption; when the device is in a performance-constrained state such as low battery or overheating, calling the same large language model for complex tasks may cause excessive device load, even forcing the device to interrupt the question-answering service. Therefore, existing single-model-based processing methods cannot balance device performance and task processing needs when device performance changes. Thus, how to enable smart devices to provide usable intelligent question-answering services to users under performance constraints has become an urgent technical problem to be solved. Summary of the Invention

[0004] The main purpose of this application is to provide a question-answering method, intelligent device and storage medium based on a large language model on the device side, which aims to solve the technical problem that the existing intelligent device question-answering response method based on a single model cannot balance the device performance status and task processing requirements when the device performance status changes.

[0005] To achieve the above objectives, this application proposes a question-answering method based on a large language model on an edge, applicable to a smart device with at least two large language models deployed locally, each with a different number of model parameters. The method includes: Upon receiving a user's pending response request, the performance status data of the smart device is obtained; Determine the complexity of the current task based on the pending request; The target inference model is determined from the at least two large language models based on the current task complexity and the performance status data; The request to be answered is input into the target reasoning model to obtain the current answer.

[0006] In one embodiment, the step of determining the complexity of the current task based on the request to be responded to includes: The intent to be responded to is identified, the intent identification result is obtained, and the task type of the request to be responded to is determined based on the intent identification result; Obtain the response requirement information of the request to be responded to, wherein the response requirement information is information used to characterize the response constraints of the request to be responded to; Based on the task type and the response requirement information, determine the current task complexity of the request to be responded to.

[0007] In one embodiment, the task complexity is simple or complex; The step of determining the target inference model from the at least two large language models based on the current task complexity and the performance status data includes: Determine the preset judgment conditions satisfied by the performance status data, and determine the current inference resource level of the smart device based on the satisfied preset judgment conditions. The current inference resource level is a high-performance level, a balanced level, an energy-saving level, or a protection level. When the task complexity is complex and the current inference resource level is high performance or balanced, a large language model with a model parameter count higher than a preset first parameter count threshold is identified as the target inference model. When the task complexity is complex and the current inference resource level is energy-saving or protection level, a large language model with no more than a preset first parameter threshold is determined as the target inference model. When the task complexity is simple, a large language model with no more than a preset first parameter threshold is determined as the target inference model.

[0008] In one embodiment, the step of inputting the request to be answered into the target inference model to obtain the current answer includes: When the task type of the request to be replied to meets the preset local task type, the request to be replied to is vectorized and encoded to obtain the current request vector; Based on the current request vector, a search is performed in the local vector database to obtain target associated data for replying to the request to be replied to. The local vector database includes associated data corresponding to different request vectors. The pending response request and the target-related data are input into the target reasoning model to obtain the current answer.

[0009] In one embodiment, the local vector database includes a first data sub-database and a second data sub-database. The first data sub-database is constructed based on the knowledge data pre-uploaded by the user, and the second data sub-database is constructed based on the user's historical dialogue data. The first data sub-database includes a plurality of first data fragments, and the second data sub-database includes a plurality of second data fragments. The step of retrieving target associated data for replying to the request based on the current request vector in the local vector database includes: A target data sub-database is determined based on the task type, and a target data fragment matching the current request vector is retrieved from the target data sub-database. The target data fragments are integrated to obtain target associated data for responding to the request to be responded to.

[0010] In one embodiment, before the step of obtaining the performance status data of the smart device upon receiving a user's pending response request, the method further includes: Record historical voice interactions or meeting speeches authorized by the user to obtain voice dialogue data; Obtain the user's historical question-and-answer records with the smart device; The voice dialogue data and the historical question-and-answer records are used as historical dialogue data, and a second data sub-database is constructed based on the historical dialogue data; The step of determining the target data sub-database according to the task type and retrieving the target data fragment matching the current request vector from the target data sub-database includes: If the target data sub-library is determined to be the second data sub-library based on the task type, a target data segment matching the current request vector is retrieved from the voice dialogue data or the historical question-and-answer records based on the task type.

[0011] In one embodiment, the step of inputting the request to be replied to and the target associated data into the target inference model to obtain the current answer includes: Obtain the standard inference parameters of the target inference model, and adjust the standard inference parameters according to the performance status data to obtain the running inference parameters of the target inference model; Configure the target inference model according to the running inference parameters, and input the request to be replied to and the target associated data into the configured target inference model to obtain the current answer.

[0012] In one embodiment, the step of obtaining the standard inference parameters of the target inference model and adjusting the standard inference parameters according to the performance status data to obtain the running inference parameters of the target inference model includes: Obtain the standard inference parameters of the target inference model; The priority order of each performance dimension in the performance status data is determined, and the dimension weight corresponding to each performance dimension is determined according to the priority order. The priority order is determined based on the degree of deviation between the corresponding value of different performance dimensions in the performance status data and the standard value of different performance dimensions. The greater the degree of deviation, the higher the priority of the corresponding performance dimension. The weights of each dimension are combined to obtain a comprehensive weight; The standard inference parameters are adjusted according to the comprehensive weights to obtain the running inference parameters of the target inference model.

[0013] Furthermore, to achieve the above objectives, this application also proposes an intelligent device, which includes: a memory, a processor, and a question-answering program for an edge-side large language model stored in the memory and executable on the processor, the question-answering program being configured to implement the steps of the question-answering method for the edge-side large language model as described above.

[0014] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a question-answering program for a terminal-side large language model is stored. When the question-answering program is executed by a processor, it implements the steps of the question-answering method for a terminal-side large language model as described above.

[0015] This application discloses a question-answering method, a smart device, and a storage medium based on a large language model on an edge. The method is applied to a smart device with at least two large language models deployed locally, each with a different number of model parameters. The method includes: upon receiving a user's pending response request, acquiring the performance status data of the smart device; determining the current task complexity based on the pending response request; determining a target inference model from at least two large language models based on the current task complexity and performance status data; and inputting the pending response request into the target inference model to obtain the current answer.

[0016] Because this application can dynamically select a target inference model suitable for the current scenario from at least two large language models with different parameter numbers based on the current task complexity of the request to be answered and the current performance status data of the smart device, it avoids the waste of computing resources and increased power consumption caused by calling a large parameter model for simple tasks, and also avoids the decrease in answer quality caused by calling a small parameter model for complex tasks. Thus, it effectively reduces the power consumption and computing overhead of the smart device while ensuring the quality of the answer, extends the device's battery life, and avoids the device from interrupting the question-and-answer service due to excessive load, achieving a balance between device performance status and task processing requirements. Attached Figure Description

[0017] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0018] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a flowchart illustrating the first embodiment of the question-answering method for the client-side large language model in this application; Figure 2 This is a schematic diagram of the model lightweighting process; Figure 3 This is a schematic diagram of the decision path of the objective reasoning model of this application; Figure 4 This is a flowchart illustrating the second embodiment of the question-answering method for the client-side large language model in this application; Figure 5 This is a schematic diagram of the reasoning process of the target reasoning model of this application; Figure 6 This is a flowchart illustrating the third embodiment of the question-answering method for the client-side large language model in this application; Figure 7 This is a schematic diagram illustrating the dynamic computing power allocation of the intelligent device in this application; Figure 8 This is a system architecture diagram of the question-answering method for the client-side large language model in this application; Figure 9 This is a schematic diagram of the entire process of the question-answering method of the client-side large language model in this application; Figure 10 This is a schematic diagram of the structure of the smart device of this application.

[0020] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0021] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0022] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0023] This application provides a question-answering method for a large language model on the client side, with reference to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the question-answering method for a large language model on the client side of this application. In this embodiment, the method includes steps S10 to S40: Step S10: Upon receiving a user's pending response request, obtain the performance status data of the smart device.

[0024] It should be noted that the executing entity of the method in this embodiment can be an intelligent device with functions such as data processing, voice acquisition, power management, and model inference, such as a portable intelligent device powered by a battery and limited in size, including: AI recording power bank, mobile power bank with screen, AI recording earphone case, meeting recording device, or voice recorder, etc. Here, the AI ​​recording power bank (referred to as "intelligent device") is used as an example to illustrate this embodiment and the following embodiments.

[0025] It should be understood that smart devices can pre-deploy at least two large language models with different parameter sizes locally. These large language models are used to perform reasoning tasks such as natural language understanding, generation, question answering, or intent recognition locally on the smart device, and the large language models are obtained by lightweighting the pre-trained large language models.

[0026] It should also be noted that pre-trained large language models can be language models pre-trained on large-scale text corpora, or they can be directly selected open-source network-based large language models. Their parameter count typically reaches billions or even hundreds of billions, resulting in massive model file sizes and enormous inference computations, making them unsuitable for direct operation on resource-constrained smart devices. However, since smart devices connect to cloud servers via a network, the cloud server can perform lightweight processing to obtain large language models that can be deployed on smart devices, and then deploy these large language models on smart devices via network connections.

[0027] The lightweighting process can include one or more combinations of the following methods: quantizing the model weights of a pre-trained large language model using INT8, INT4, or mixed precision quantization to reduce model size and memory usage by lowering numerical precision, while simultaneously accelerating inference using the processor's low-precision computation instructions; using knowledge distillation to guide the training of a student model with fewer parameters using a teacher model with a larger number of parameters, ensuring the inference quality of the lightweight model remains above a preset quality threshold; performing structured pruning to remove unimportant connections or neurons from the pre-trained large language model to reduce model size; introducing low-rank adaptation parameters, decomposing the original weight matrix into a product of low-rank matrices, and merging the adaptation parameters with the original weights after training to reduce storage overhead; and performing model sharding to divide the model parameters into multiple shards, supporting on-demand loading so that the device does not need to load all parameters at once during inference, thereby reducing peak runtime memory usage.

[0028] For intelligent devices with limited computing power, such as microcontroller units (MCUs) or coprocessors, the large language model in such devices can be a sub-model with the smallest number of parameters, which is only used for wake word recognition, intent classification, and fast response. However, for intelligent devices equipped with ARM Cortex-A, RISC-V AI extensions, neural network processing units (NPUs), digital signal processors (DSPs), or embedded graphics processors (GPUs), the large language model in such devices can be a deep inference model with a large number of parameters.

[0029] Furthermore, based on the lightweight deployment of the large language model on smart devices, the smart devices can also optimize the inference process for the local hardware platform: the embedding layer, normalization layer, matrix multiplication, activation function, attention calculation and output projection in the large language model are fused together, and multiple consecutive calculation operations are merged into one operation, thereby reducing the generation and reading / writing of intermediate tensors, and reducing memory access latency and bandwidth usage.

[0030] Furthermore, for smart devices using different hardware platforms, corresponding acceleration instruction sets and computing libraries can be configured: for ARM architecture processors, NEON or SME instructions are configured for vectorization acceleration; for smart devices with DSPs, DSP-specific instructions are called for matrix operation acceleration; for devices with NPUs, model operators are mapped to the NPU for computation through the NPU compiler; for RISC-V AI extension processors, corresponding AI extension instructions are configured; and for embedded GPUs, parallel acceleration is achieved through GPU computing frameworks (such as OpenCL or Vulkan). Through the above instruction-level adaptation for different hardware platforms, the inference process of large language models can be deeply integrated with the underlying hardware of smart devices, fully leveraging the computing power of various hardware platforms. Simultaneously, for autoregressive generation tasks, smart devices can also manage key-value pairs (KV Cache) to avoid redundant computation of prefixes of already generated content; and dynamically decide whether to retain the complete cache to accelerate subsequent generation, compress the cache to save memory, or use a sliding window cache to control memory usage based on context length, remaining memory, and device temperature status, thereby further reducing redundant computation overhead in continuous dialogue or multi-turn inference scenarios.

[0031] Furthermore, this can be referenced here. Figure 2 This paper provides an exemplary illustration of the process of obtaining a large language model that can be locally deployed on smart devices by lightweighting a pre-trained large language model. Figure 2 This is a schematic diagram of the model lightweighting process.

[0032] Depend on Figure 2 As can be seen, taking the original large-scale model LLaMA as an example of a pre-trained large-scale language model, this original large-scale model sequentially includes an embedding layer, multiple sequentially stacked Transformer modules (Transformer Block 1 to Transformer Block N), and an output layer (Lm Head). Each Transformer module contains sub-layers such as self-attention computation, feedforward networks, and layer normalization. The number of parameters in this type of pre-trained large-scale language model typically reaches billions or even hundreds of billions, making it difficult to directly deploy on resource-constrained intelligent devices.

[0033] Therefore, lightweight processing can be performed on this type of pre-trained large language model. Specifically, this includes: quantizing the model weights with INT8 or INT4 to reduce numerical precision; using knowledge distillation to guide the training of a small-parameter student model using a large-parameter teacher model; performing structured pruning to remove unimportant connections or neurons from the model; and introducing low-rank adaptation parameters (such as LoRA / Adapter) and merging the adaptation parameters with the original weights after training to reduce storage overhead. These various lightweighting methods can be combined to significantly reduce storage volume and runtime memory usage without compromising inference quality beyond a preset threshold.

[0034] Building upon the lightweight deployment, the model inference process can be further optimized. For example... Figure 2 As shown, continuous mergeable operators such as LayerNorm, MatMul, BiasAdd, and SiLU activation functions in large language models can be fused to reduce the generation and read / write operations of intermediate tensors, thereby reducing memory access latency and bandwidth usage.

[0035] Simultaneously, instruction-level adaptation can be performed for the local hardware platform of smart devices: NEON instructions are configured for vectorization acceleration for ARM architecture processors; DSP-equipped smart devices are accelerated by calling DSP-specific instructions for matrix operations; NPU-equipped devices are mapped to the NPU for computation through the NPU compiler; embedded GPUs are accelerated in parallel through the GPU computing framework; and corresponding AI extension instructions are configured for RISC-V AI extension processors.

[0036] It should be understood that the large language model obtained through the above-mentioned multi-layered lightweighting, operator fusion and hardware adaptation can run efficiently on resource-constrained smart devices.

[0037] Furthermore, when deploying large language models on smart devices, the model files of each large language model can be divided into multiple segments according to a preset segment size (e.g., 1MB per segment). Each segment is then encrypted using a symmetric encryption algorithm (e.g., AES-256). The encryption key for each segment can be derived from the unique hardware identifier of the smart device (e.g., chip serial number or internal key of the security chip), making the model files non-interchangeable between different smart devices. This effectively prevents the model from being illegally copied and running on other unauthorized smart devices, and ensures that ordinary user processes and external debugging interfaces have no right to access or modify the model files of the aforementioned large language models.

[0038] Furthermore, upon each startup or before each invocation of the target inference model, the smart device can perform integrity checks on each model file fragment, for example, by comparing digital signatures or verifying hash values ​​to detect whether the model file has been tampered with or corrupted. If the verification passes, the smart device decrypts the required fragments and loads them into memory for inference. If an integrity check fails, the smart device prevents the loading of the model file, marks the model as unavailable, and outputs a corresponding error message to the user.

[0039] It should also be noted that the request to be responded to can be a request entered by the user to the smart device via voice, text, key, or touch. For example, a user can issue a voice command to the smart device to "open the recording," or a user can enter a text request on the device screen such as "Please summarize the content of the meeting just now," thereby triggering an interactive question-and-answer process with the smart device.

[0040] It should be understood that performance status data is data used to characterize the current resource availability of a smart device. It can consist of device data from different performance dimensions, such as battery level, temperature, charging / discharging status, memory usage, and processor load. Therefore, performance status data can include at least one of the following: battery level, temperature, charging status, memory usage, and processor load.

[0041] Specifically, smart devices can obtain current power data and charging / discharging status data through power management chips, current temperature data through temperature sensors, and current memory usage and processor load data through the operating system interface.

[0042] For example, when a smart device receives a user's pending response request, it can simultaneously read one or more of the following device data: power data from the power management chip (e.g., current power 65%), temperature data from the temperature sensor (e.g., current temperature 38 degrees Celsius), charging status data (e.g., not charging), memory usage (e.g., 45%), and processor load (e.g., 30%), thereby obtaining the performance status data of the smart device at that moment.

[0043] Step S20: Determine the complexity of the current task based on the request to be replied to.

[0044] It should be noted that the current task complexity is used to characterize the difficulty of the reasoning task corresponding to the request to be replied to, and it can be determined comprehensively based on the task type of the request to be replied to and the answer requirement information.

[0045] Task complexity reflects the attributes of a user request across multiple dimensions, including: whether the request to be answered is a simple instruction or a complex instruction that requires deep reasoning, whether it requires access to the device's local vector database, whether it depends on the context of the dialogue history, whether the expected response is a brief result or a long summary, and whether the user is in an interactive state that requires a rapid response.

[0046] In practice, intelligent devices can classify user requests into different complexity levels by comprehensively evaluating the above dimensions, thereby providing a basis for decision-making in selecting an appropriate inference model.

[0047] Furthermore, to illustrate in detail how to determine the complexity of the current task based on the pending request, step S20 specifically includes: steps S201~S203: Step S201: Perform intent recognition on the request to be replied to, obtain intent recognition results, and determine the task type of the request to be replied to based on the intent recognition results.

[0048] It should be noted that intent recognition can be achieved through a lightweight intent classification model or rule engine deployed locally on the device. This lightweight intent classification model can be a classifier fine-tuned from a small pre-trained language model such as BERT-Tiny, and its number of parameters can be much smaller than the large language models deployed on smart devices.

[0049] It should also be noted that if the request to be replied to is input via voice, keypad, or touch, the smart device can first convert the request to be replied to into text form through voice-to-text conversion technology or a pre-set input mapping relationship.

[0050] In its implementation, a smart device can input a text-based request for a response into an intent classification model, which then outputs an intent recognition result. This result can include task types such as device control, simple question answering, meeting retrieval, summary generation, complex reasoning, and multi-turn dialogue. The smart device determines the task type corresponding to the request based on the intent recognition result, and this request can correspond to one or more task types.

[0051] For example, if a user inputs "open recording", the intent classification model can output "device control", and thus determine the task type as device control; if a user inputs "what did member A say at the meeting", the intent classification model outputs "meeting retrieval", and the smart device can then determine the task type as meeting retrieval.

[0052] Step S202: Obtain the response requirement information of the request to be responded to, wherein the response requirement information is information used to characterize the response constraints of the request to be responded to.

[0053] It should be understood that response request information describes the user's expected response and its constraints, and can be extracted from the request to be responded to. Response request information may include at least one of the following: the text length of the request to be responded to, whether local knowledge retrieval is required, whether context reliance is required, the expected response length, and the urgency of the interaction.

[0054] It should be understood that smart devices can extract answer request information from pending requests through rule parsing or lightweight information extraction models.

[0055] For example, a user enters "Briefly summarize the main points of today's meeting," where "briefly summarize" indicates a short expected response; a user enters "What were the three risk points we just discussed?" where "just discussed" indicates a need to rely on the context of the current session, and "three risk points" indicates an expected enumerated response; a user enters "Please look up the product solution from last time," where "look up" indicates a need to search the local knowledge base.

[0056] Step S203: Determine the current task complexity of the request to be replied to based on the task type and the response requirement information.

[0057] It should be noted that the complexity of the current task can be divided into two levels: simple or complex, or a more granular classification method can be used. This embodiment does not impose any restrictions on this. The following explanation uses a simple / complex classification as an example.

[0058] It should be understood that smart devices can input task type and response requirements into a preset complexity determination model, or perform a lookup judgment using a preset rule table. Specifically, if the task type is device control (such as "turn on recording" or "check battery level"), the smart device can directly determine the current task complexity as simple, regardless of the response requirements. If the task type is summary generation or complex reasoning, the smart device can directly determine the current task complexity as complex, regardless of the response requirements. If the task type is question-and-answer, the smart device can further determine the complexity based on a comprehensive assessment of the text length, knowledge retrieval requirements, contextual dependencies, response length requirements, and the urgency of the interaction within the response requirements.

[0059] For example, if a user's pending response request is "What time is it now?", and the task type is simple question and answer, and the response information shows a short text length, no knowledge retrieval required, no contextual dependency, and a short response length, then the current task complexity is determined to be simple.

[0060] If a user's pending response request is "Please summarize all the risk points and corresponding solutions mentioned in the meeting just now," and the task type is summary generation, and the response request information shows that it requires retrieving meeting records, depends on the current session context, and expects a lengthy response, then the complexity of the current task is determined to be complex.

[0061] Step S30: Determine the target inference model from the at least two large language models based on the current task complexity and the performance status data.

[0062] It should be noted that the at least two large language models deployed in a smart device can have different numbers of model parameters. The number of model parameters refers to the total number of trainable parameters contained in a large language model, which can be used to characterize the storage volume, inference computation load, and semantic understanding capability of the large language model. That is, the larger the number of model parameters of a large language model, the stronger its inference capability, but also the greater its device storage space, the computational resources required for inference, and the power consumption.

[0063] For example, a smart device can deploy a larger model with 1B parameters and a smaller model with 0.1B parameters, wherein the larger model has a stronger semantic understanding and generation capability but higher inference power consumption, and the smaller model has a weaker semantic understanding capability but lower inference power consumption.

[0064] It should be understood that the current task complexity measures the degree to which the user's pending request requires the model's inference capabilities, while performance status data characterizes the level of computing resources currently available to the intelligent device. Therefore, when selecting a target inference model, the intelligent device can comprehensively consider both dimensions and choose the one most suitable for the current scenario from the large language models deployed on the intelligent device, in order to achieve a dynamic balance between response quality and resource consumption.

[0065] Furthermore, to specifically illustrate the determination logic of the target reasoning model, step S30 also includes: steps S301~S304: Step S301: Determine the preset judgment conditions satisfied by the performance status data, and determine the current inference resource level of the smart device according to the preset judgment conditions satisfied. The current inference resource level is a high-performance level, a balanced level, an energy-saving level, or a protection level.

[0066] It should be noted that the inference resource level can be used to characterize the sufficiency of computing resources currently available for intelligent devices to perform inference tasks. Since device data in different performance dimensions of the performance status data can be converted into quantified values ​​for different performance dimensions, the preset judgment conditions can be pre-defined rules used to map the values ​​of the performance status data to discrete inference resource levels.

[0067] It should be understood that the preset judgment conditions may include preset thresholds corresponding to each performance dimension, and the preset thresholds may include upper and lower thresholds. The intelligent device can compare the current value of each performance dimension with the corresponding preset threshold, determine the currently satisfied preset judgment conditions based on the comparison results, and thus determine the corresponding inference resource level.

[0068] It should also be noted that if multiple preset judgment conditions corresponding to the high performance level, balance level, energy saving level, or protection level are met simultaneously, the level that ranks higher in the priority order of protection level > energy saving level > balance level > high performance level can be determined as the current inference resource level.

[0069] Specifically, the high-performance level (P0) can be used to characterize the current state of a smart device having sufficient computing resources. The current inference resource level can be determined to be high-performance when the performance status data meets the following conditions: the device is charging; the device temperature is below a preset first temperature threshold; and the device's current battery level is above a preset first battery level threshold.

[0070] For example, the preset first temperature threshold can be set to 40 degrees Celsius, and the preset first battery level threshold can be set to 50%. That is, when the smart device is charging, the temperature is below 40 degrees Celsius, and the battery level is above 50%, it is determined to be at the high performance level.

[0071] The Balanced Level (P1) characterizes the current state of a smart device with moderate computing resources. At this level, the smart device can support regular inference tasks but cannot support prolonged high-load operation. The current inference resource level is determined to be Balanced when the performance status data meets the following conditions: the smart device is not charging; the device temperature is below a preset first temperature threshold; and the device's current battery level is between a preset first battery threshold and a preset second battery threshold. The preset second battery threshold is lower than the preset first battery threshold.

[0072] For example, the preset first temperature threshold is set to 40 degrees Celsius, the preset first battery level threshold is set to 50%, and the preset second battery level threshold is set to 20%. That is, when the device is not charging, the temperature is below 40 degrees Celsius, and the battery level is between 20% and 50%, it is determined to be in a balanced state.

[0073] The Energy Saving Level (P2) characterizes a smart device's current state of strained computing resources. In this case, the smart device needs to limit inference load to extend battery life and control temperature rise. The current inference resource level can be determined to be Energy Saving Level when the performance status data meets the following conditions: the smart device is in a state of external discharge; the device temperature is higher than a preset second temperature threshold but lower than a preset third temperature threshold; and the device's current battery level is between a preset second battery level threshold and a preset third battery level threshold. Specifically, the preset third temperature threshold is higher than the preset second temperature threshold, and the preset third battery level threshold is lower than the preset second battery level threshold.

[0074] For example, the external discharge state means that the device is charging other external devices (such as mobile phones); the preset second temperature threshold can be set to 40 degrees Celsius, and the preset third temperature threshold can be set to 48 degrees Celsius; the preset second power threshold is set to 20%, and the preset third power threshold is set to 10%. That is, when the device is discharging externally, the temperature is between 40 degrees Celsius and 48 degrees Celsius, and the power is between 10% and 20%, it is determined to be at the energy-saving level.

[0075] Protection level (P3) can be used to characterize the state of a smart device where computing resources are severely limited. In this case, the smart device must prioritize basic operation and security, and limit or suspend inference tasks. The current inference resource level can be determined to be protection level when the performance status data meets the following conditions: the smart device's current battery level is lower than the preset third battery threshold; or the device temperature is higher than the preset third temperature threshold; or the device is in a high-power external discharge state and the recording and saving task is in progress.

[0076] For example, the preset third battery threshold can be set to 10%, and the preset third temperature threshold can be set to 50 degrees Celsius. That is, when the device battery is below 10% or the temperature is above 50 degrees Celsius, it is determined to be in a protection level. At this level, deep inference is paused, necessary device control and simple Q&A are retained, and complex tasks are queued or prompted to the user for later processing.

[0077] It should also be noted that the specific values ​​of the above preset thresholds (e.g., 40 degrees Celsius, 50 degrees Celsius, 10%, 20%, 50%, etc.) are all illustrative examples. In actual applications, the judgment thresholds corresponding to each performance dimension in each inference resource level can be customized according to the specific hardware specifications, battery capacity, heat dissipation design and product requirements of the smart device. This embodiment does not impose any restrictions on this.

[0078] Furthermore, the performance status data acquired by the smart device may also include at least one of discharge power data, recording task status data, and recording storage status data, which can be used as a factor in determining the current inference resource level. The recording task status data is used to indicate whether the smart device is in a recording state, the recording storage status data is used to indicate whether the smart device is performing a recording data write and storage operation, and the discharge power data is used to indicate the output power of the smart device when supplying power to external devices.

[0079] For example, when the recording task status data indicates that the smart device is currently in a recording state and the recording save status data is in an active state, the current inference resource level can be set to no higher than the balance level (P1), thereby avoiding the competition between the inference task and the recording task for processor and memory resources, which could lead to recording interruption or data loss.

[0080] When the discharge power data exceeds a preset power threshold, the current inference resource level can be set to no higher than the energy-saving level (P2) to avoid inference tasks competing with the power supply function for battery energy, which could lead to unstable output voltage or power interruption. For example, the preset power threshold can be set to 10W.

[0081] It should be noted that the above-mentioned judgment factors have higher priority than the preset judgment conditions corresponding to the aforementioned performance dimensions. That is, when the intelligent device determines the current inference resource level to be high-performance (P0) based on the aforementioned performance dimension data, and the recording task status data indicates that it is currently in recording state and the recording save state is active, the intelligent device can downgrade the current inference resource level to the balanced level (P1) or a lower level.

[0082] The specific downgrade strategy can refer to the example above. Alternatively, when it is determined that the output power of the smart device is higher than a preset power threshold when it is in a recording state, performing a recording data writing and storage operation, or supplying power to an external device, the inference resource level obtained based on the data of each performance dimension is downgraded by one level to serve as the current inference resource level of the smart device.

[0083] Step S302: When the task complexity is complex and the current inference resource level is high performance or balanced, the large language model with more than the preset first parameter threshold is determined as the target inference model.

[0084] It should be noted that the preset first parameter threshold can be used to distinguish between large language models with a large number of parameters and large language models with a small number of parameters among the multiple large language models deployed by the smart device. Its specific value can be preset according to the parameter distribution of the models deployed by the device.

[0085] Taking a smart device with two large language models, 1B and 0.1B, as an example, the preset threshold for the first parameter quantity can be set to 0.5B. In this case, the 1B model is a large parameter quantity model, and the 0.1B model is a small parameter quantity model.

[0086] It should be understood that when the task complexity is "complex," it indicates that the user's pending response requires the model to have strong semantic understanding and reasoning generation capabilities. If the inference resource level is "high-performance" or "balanced," it means that the smart device currently has sufficient power and heat dissipation to support the operation of a model with a large number of parameters. Therefore, the smart device can select a target inference model from a set of models with a large number of parameters that exceeds a preset first parameter threshold. For example, it can directly select the large language model with the highest number of parameters as the target inference model, thereby obtaining better response quality with higher resource consumption.

[0087] Furthermore, based on the above selection of a large number of parameters model, the target inference model can be further refined according to the specific differences between the current inference resource level and the high-performance level or the balanced level.

[0088] Understandably, at the high-performance level, the smart device is charging and has sufficient power and a low temperature, which can support the operation of a large number of parameters model at a high power consumption level for a long time. At the balanced level, the device is not charging and has a medium power level. Although it can support the operation of a large number of parameters model, power consumption needs to be controlled to avoid the battery being consumed too quickly.

[0089] In practical implementation, if a smart device deploys three or more large language models with different parameter counts, then there may be multiple models with parameter counts exceeding a preset first parameter count threshold. For example, a smart device may deploy three large language models: 1.5B, 0.7B, and 0.1B. If the preset first parameter count threshold is set to 0.5B, then both the 1.5B and 0.7B models are considered large parameter count models.

[0090] In this scenario, when the current inference resource level is high performance, the intelligent device can select the model with the largest number of parameters (such as the 1.5B model) as the target inference model from the large parameter model, thereby allowing for a higher number of threads, a longer context window, and higher generation quality parameters, and responding to user requests with the best answer quality.

[0091] When the current inference resource level is balanced, since the smart device is not charging and its power level is moderate, the smart device can select a model with a relatively small number of parameters (such as the 0.7B model) as the target inference model from the large parameter model, so as to reduce power consumption and extend the battery life of the smart device while ensuring the quality of the answer.

[0092] Step S303: When the task complexity is complex and the current inference resource level is energy-saving or protection level, a large language model with no more than a preset first parameter threshold is determined as the target inference model.

[0093] It should be noted that when the task complexity is "complex" and the inference resource level is "energy-saving" or "protection," it indicates that the intelligent device is in a state of external discharge, with high temperature and low battery, resulting in relatively strained computing resources. In this state, considering that forcibly calling a model with a large number of parameters for inference might cause the intelligent device's battery to deplete rapidly, the temperature to rise further, and ultimately force the device to interrupt the question-and-answer service, the intelligent device can select a target inference model from a set of models with fewer parameters than a preset first parameter threshold.

[0094] Furthermore, based on the above selection of a small parameter model, the target inference model can be further refined according to the specific differences between the energy-saving level and the protection level.

[0095] It should be noted that under the energy-saving level, intelligent devices can still support inference of models with a small number of parameters, but power consumption must be strictly controlled; while under the protection level, intelligent devices can hardly support inference of models with a small number of parameters, and the basic operation and safety of the equipment must be prioritized.

[0096] In practical implementation, if a smart device deploys three or more large language models with different parameter counts, then there may be multiple models with parameter counts below a preset first parameter count threshold. For example, a smart device may deploy four large language models: 1.5B, 0.7B, 0.3B, and 0.1B. If the preset first parameter count threshold is set to 0.5B, then the 0.3B and 0.1B models are considered small parameter count models.

[0097] In this scenario, when the current inference resource level is energy-saving, the smart device can select the 0.3B model as the target inference model, thereby reducing the number of threads and shortening the dialogue context length while ensuring that the smart device can still generate usable answers under low power conditions.

[0098] When the current inference resource level is at the protection level, the intelligent device selects the model with the fewest parameters (such as the 0.1B model) to complete the response with the lowest power consumption. At this time, the intelligent device can add requests with complex task complexity to the waiting queue and output a prompt to the user indicating that an answer is pending.

[0099] Step S304: When the task complexity is simple, a large language model with no more than a preset first parameter threshold is determined as the target inference model.

[0100] It should be noted that when the task complexity is set to "simple," it means that the user request only requires the model to have basic semantic understanding or simple instruction parsing capabilities, without the need for deep inference capabilities of a model with a large number of parameters. In this case, regardless of whether the current inference resource level of the intelligent device is high-performance, balanced, energy-saving, or protection-oriented, the intelligent device can select the target inference model from a set of small-parameter models with fewer than a preset first parameter threshold.

[0101] It should be understood that for simple tasks, the difference in response quality between a large language model with more than a preset first parameter threshold and a small parameter model is negligible. The small parameter model has a faster inference speed and lower power consumption. Therefore, under all inference resource levels, simple tasks should prioritize calling the small parameter model to complete the inference task with the lowest power consumption and computational overhead, thus avoiding the resource waste caused by calling the large parameter model for simple tasks.

[0102] Furthermore, based on the above selection of a model with a small number of parameters, the target inference model can be further refined according to the specific differences between different levels of inference resources.

[0103] Specifically, following the example above, if the small parameter model includes 0.3B and 0.1B models, then when the current inference resource level is high performance level or balanced level, the intelligent device can choose 0.3B as the target inference model; when the current inference resource level is energy saving level or protection level, the intelligent device can choose 0.1B as the target inference model.

[0104] Furthermore, you can also refer to this section. Figure 3 The process of determining the target inference model based on the current task complexity and performance status data described in this application will be explained. Figure 3 This is a schematic diagram of the decision path of the reasoning model for the objective of this application.

[0105] Depend on Figure 3 It can be seen that after receiving a request to be responded to, the intelligent device simultaneously obtains performance status data and determines the complexity of the current task. When the task complexity is simple, it indicates that the user request only requires basic semantic understanding or simple instruction parsing capabilities, without the need for deep inference with a large number of parameters. A small number of parameters model can be directly selected as the target inference model to complete the response with the lowest power consumption.

[0106] When the task complexity is complex, it indicates that the user request requires the model to have strong semantic understanding and reasoning generation capabilities. At this time, the intelligent device can further determine the current inference resource level (P0 high performance level, P1 balanced level, P2 energy-saving level, P3 protection level) based on performance status data, and select the target inference model according to the differences in different levels: At P0 level, the smart device is charging and has sufficient power and heat dissipation, so a large parameter model can be selected to respond to user requests with optimal quality. At P1 level, the smart device is not charging and has a moderate power level, so a large parameter model can be selected but its inference parameters are limited to achieve a balance between response quality and power consumption. At P2 level, the smart device is discharging and has low power and high temperature, so a small parameter model can be selected and the number of threads and the context length can be further reduced. At P3 level, the smart device has extremely low power or excessive temperature, so the model with the fewest parameters can be selected to execute a minimal response to generate a short summary or prompt the user to charge before processing, prioritizing device safety.

[0107] Finally, the target reasoning model performs reasoning, obtains the current answer and outputs it, or outputs an answer and waits for prompts at the P3 level.

[0108] Step S40: Input the request to be answered into the target reasoning model to obtain the current answer.

[0109] It should be noted that after the target reasoning model is determined, the smart device can process the request to be replied to according to the input format required by the target reasoning model, and then input it into the target reasoning model to perform reasoning and obtain the current answer.

[0110] It should be understood that the smart device formats the user's input request for a response: First, the text-based request is encoded to obtain a current request vector. Then, a task instruction prefix is ​​added to the current request vector. Based on this prefixed current request vector, a dialogue template is constructed, and so on, to obtain input data that conforms to the input specifications of the target inference model. Finally, the smart device loads the processed input data into the target inference model, which performs forward inference calculations to generate the response text, i.e., the current answer, word by word.

[0111] Specifically, smart devices can also combine secure chips or trusted execution environments (TEEs) to protect the model files and local data of the target inference model: when the target inference model performs inference, model parameters and sensitive data during the inference process (such as requests to be answered, retrieved local data, and generated answers) are loaded in a separate, isolated trusted execution environment. This prevents the inference process from being read or tampered with by malicious processes from other applications on the smart device, thereby ensuring the reliability of the answer results and user privacy.

[0112] Understandably, after the target inference model generates the answer text, the smart device can present the current answer to the user through a preset output method. This output method may include: displaying the answer text on the smart device screen, broadcasting it via a speaker or headphones, or synchronizing the answer to a mobile terminal APP paired with the device via a wireless communication module.

[0113] In addition, smart devices can also write the pending requests and answer texts from this interactive Q&A process to a local cache for quick responses to subsequent similar requests.

[0114] Furthermore, when a smart device is connected to a cloud server, if the smart device is in an offline environment and the user authorizes a response from the cloud, the smart device can first quickly generate the current answer based on the local target reasoning model, and then the pre-trained large language model in the cloud server can further supplement a more complete deep answer analysis. That is, the current answer and the deep answer analysis together constitute the complete response to the request for a reply.

[0115] However, if the smart device is in an offline environment or the user has not authorized the cloud to respond, the current answer generated by the smart device based on the local target reasoning model is the complete response to the request for a reply.

[0116] Among them, the situation where the user does not authorize the cloud response can be either the user manually choosing to disconnect the cloud connection on the smart device, or the situation where the cloud connection is automatically disconnected when the pending request involves sensitive privacy information.

[0117] Specifically, smart devices can identify whether the request to be responded to contains privacy-sensitive information through a rule engine or a lightweight classification model. The privacy-sensitive information includes one or more of the following: user name, contact information, meeting content, address book, to-do list, and other personally identifiable information.

[0118] When the pending request is detected as a privacy-sensitive request, the smart device marks the pending request as a local mandatory processing request, prohibiting the uploading of the pending request and the reasoning process data generated based on the target reasoning model to the cloud server or any external device. All retrieval, reasoning and answer generation steps are completed locally on the device.

[0119] Furthermore, in cases where the user has not authorized a cloud-based response, the smart device can immediately clear the temporary cached data from the question-and-answer interaction after outputting all responses, retaining only the necessary dialogue summary in the smart device's local encrypted storage area for subsequent contextual coherence. This ensures that privacy-sensitive data does not leave the smart device, further reducing the risk of privacy leaks.

[0120] This embodiment can dynamically select a target inference model suitable for the current scenario from at least two large language models with different parameter sizes based on the current task complexity of the request to be answered and the current performance status data of the smart device. For simple tasks, a model with fewer parameters is selected to avoid unnecessary waste of computing resources and increased power consumption; for complex tasks, a model with more parameters is selected when the device performance is good to ensure the quality of the answer, and a model with fewer parameters is selected when the device performance is limited to ensure that the question-answering service can still be provided. This effectively reduces the power consumption and computing overhead of the smart device while ensuring the quality of the answer, extends the device's battery life, and prevents the device from interrupting the question-answering service due to excessive load.

[0121] Based on the first embodiment of this application, in the second embodiment of this application, the content that is the same as or similar to the first embodiment described above can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 4 , Figure 4 This is a flowchart illustrating the second embodiment of the question-answering method for the client-side large language model in this application.

[0122] In this embodiment, considering that the request to be replied to may be a request related to local knowledge, when the request to be replied to involves local knowledge, the smart device can implement Retrieval-Augmented Generation (RAG) based on the local vector database to output the current answer, that is, the retrieved local data is used as target related data and input into the target inference model along with the request to be replied to. Therefore, step S40 also includes: steps S401~S403: Step S401: When the task type of the request to be replied to meets the preset local task type, the request to be replied to is vectorized and encoded to obtain the current request vector.

[0123] It should be noted that the preset local task types are pre-defined task categories that require intelligent local storage data to complete an answer. The task type for the request to be answered can include one or more of the following: device control, simple question and answer, meeting retrieval, summary generation, complex reasoning, and multi-turn dialogue.

[0124] It should be understood that meeting retrieval tasks involve searching for transcribed meeting recordings or meeting minutes stored locally on smart devices; summary generation tasks require summarizing and refining locally stored meeting content, dialogue records, and other data; complex reasoning tasks typically require multi-step reasoning by combining local knowledge bases or historical dialogue context; and multi-turn dialogue tasks rely on historical question-and-answer records saved by smart devices to maintain the continuity of the dialogue. Meanwhile, device control tasks (such as "open recording") only require executing the corresponding device operation command and do not need to access local knowledge data; simple question-and-answer tasks (such as "What time is it?") only require the model to answer based on its own parameter memory, and similarly do not require access to local knowledge data.

[0125] The intelligent device can then determine that meeting retrieval, summary generation, complex reasoning, and multi-turn dialogue belong to the preset local task types; device control and simple question and answer do not belong to the preset local task types.

[0126] It should be understood that vectorization encoding refers to the process of converting a text-based request for a response into a numerical vector representation.

[0127] In practical implementation, smart devices can call a locally deployed embedding model to encode the request to be replied to, thereby mapping the text-based request to be replied to into a fixed-dimensional semantic vector, namely the current request vector. This current request vector can be used to represent the semantic information of the request to be replied to. The aforementioned embedding model can be a lightweight encoder that is paired with the deployed large language model, or it can be an independent small semantic encoding model with a much smaller number of parameters than the various language models deployed in the smart device.

[0128] Step S402: Search the local vector database according to the current request vector to obtain target associated data for replying to the request to be replied to. The local vector database includes associated data corresponding to different request vectors.

[0129] It should be noted that the local vector database is a collection of vector indexes stored locally on the device, used to support efficient semantic retrieval. The local vector database stores the mapping relationship between different request vectors and associated data, that is, each record contains a pre-stored vector and its corresponding associated data (such as knowledge fragments, meeting minutes, question-answer pairs, etc.).

[0130] After receiving the current request vector, the smart device can perform a similarity search in the local vector database to find several pre-stored vectors that are most similar to the current request vector in the semantic space, and then obtain the data fragments corresponding to these pre-stored vectors and combine them into target associated data.

[0131] It should be understood that smart devices can use the Approximate Nearest Neighbor (ANN) retrieval algorithm to build an index on the local vector database, thereby performing retrieval in the local vector database. The index structure can adopt, for example, a Hierarchical Navigable Small World (HNSW), an Inverted File Index (IVF), or Product Quantization (PQ) to control the computing resources and storage overhead of the smart device while ensuring retrieval accuracy.

[0132] It should be noted that the intelligent device can manage the local vector database by eliminating data segments based on their access frequency, timestamps, and remaining storage capacity. When local storage space is insufficient or the total amount of data exceeds the preset capacity limit, the intelligent device can mark eligible pre-stored vectors as pending elimination according to the strategy of prioritizing the lowest access frequency or the earliest storage time, and perform deletion operations during the device's idle period to release storage space, thereby maintaining the efficient operation of the local vector database under the limited storage resources of the intelligent device. For frequently accessed data segments, the intelligent device also retains them in the cache to improve retrieval efficiency.

[0133] It should also be noted that the local vector database can also use an encrypted storage method, that is, all data fragments stored in the local vector database are encrypted with a symmetric encryption algorithm before being written, thereby ensuring the data security of the local vector database.

[0134] Furthermore, upon startup or before retrieving data from the local vector database, smart devices can perform integrity checks on data segments within the local vector database using hash value comparison or CRC verification to detect whether the data has been tampered with or corrupted. If an integrity check fails, the smart device can mark the corresponding data segment as unusable or delete it from the local vector database, thereby ensuring the data security and integrity of the local vector database.

[0135] In practical implementation, the smart device can calculate the similarity (such as cosine similarity or inner product similarity) between the current request vector and each pre-stored vector in the local vector database, and use the associated data corresponding to the pre-set number of pre-stored vectors with the highest similarity (such as Top-3 or Top-5) as the retrieval results to obtain the target associated data for replying to the request to be replied to.

[0136] Furthermore, to specifically illustrate the internal structure and retrieval logic of the local vector database, step S402 also includes: steps S4021~S4022: Step S4021: Determine the target data sub-database according to the task type, and retrieve the target data fragment that matches the current request vector from the target data sub-database.

[0137] It should be noted that the target data sub-database is a sub-database in the local vector database that matches the task type of the current request to be responded to. This local vector database may include a first data sub-database and a second data sub-database. The first data sub-database is constructed based on knowledge data pre-uploaded by the user (such as product manuals, personal documents, etc.) and includes several first data fragments; the second data sub-database is constructed based on the user's historical dialogue data and includes several second data fragments.

[0138] It should be understood that smart devices can determine the target data sub-database based on the task type. Task types include one or more of the following: device control, simple question answering, meeting retrieval, summary generation, complex reasoning, and multi-turn dialogue. Device control and simple question answering do not involve local knowledge data and can therefore be performed without triggering a retrieval. Meeting retrieval, summary generation, and multi-turn dialogue all rely on historical dialogue data, allowing smart devices to determine a second data sub-database as the target data sub-database.

[0139] Furthermore, for complex reasoning, intelligent devices can make judgments based on the data sources upon which the reasoning relies: if it relies on knowledge data pre-uploaded by the user, then the first data sub-database is determined as the target data sub-database; if it relies on historical dialogue data, then the second data sub-database is determined as the target data sub-database; if it relies on both types of data, then both the first and second data sub-databases are determined as target data sub-databases.

[0140] After determining the target data sub-database, the smart device can calculate the similarity between the current request vector and each pre-stored vector in the target data sub-database, and take the data segment corresponding to the preset number of pre-stored vectors with the highest similarity as the target data segment that best matches the current request vector.

[0141] Furthermore, to illustrate in detail how to construct the second data sub-database in the local vector database, steps S001-S003 may be included before step S10: Step S001: Record the historical voice interaction process or meeting speech process authorized by the user to obtain voice dialogue data.

[0142] It should be noted that users can engage in various language interactions while using smart devices, and can actively trigger recording of these interactions. Specifically, after user authorization, the smart device can record the user's speeches, voice commands, dictation, and other speech processes using a built-in microphone array or an external microphone, obtaining raw voice dialogue data.

[0143] After obtaining the voice dialogue data, the smart device can also call the locally deployed speech recognition module to transcribe the raw voice data, converting it into text-based dialogue data, which facilitates subsequent text segmentation and vectorization encoding.

[0144] In practice, when a user triggers a recording operation, the smart device can display the rules for collecting recording information, informing the user that the device can perform operations such as text conversion, data storage, and data analysis on the recorded voice dialogue data, and then prompt the user with an authorization request. After the user confirms the authorization, the smart device then records the user's historical voice interaction process or meeting speech, obtaining the raw voice dialogue data and converting it into text-based voice dialogue data.

[0145] Step S002: Obtain the historical question-and-answer records between the user and the smart device.

[0146] It should be noted that historical question-and-answer records refer to the question-and-answer data generated during the user's interactions with the smart device. Specifically, after each question-and-answer interaction, the smart device can store the user's pending response request and the corresponding current answer in key-value pairs in local storage. Then, when a second sub-database needs to be built or updated, the smart device can read all historical question-and-answer records from local storage.

[0147] In addition, smart devices can periodically clean up historical question and answer records according to preset elimination strategies, such as deleting records that exceed a preset time threshold or retaining a preset number of recent records, in order to control local storage usage.

[0148] Step S003: Use the voice dialogue data and the historical question-and-answer records as historical dialogue data, and construct a second data sub-database based on the historical dialogue data.

[0149] It should be noted that smart devices can combine the transcribed text of voice dialogue data with historical question-and-answer records as historical dialogue data to construct a second data sub-library. Specifically, smart devices can segment historical dialogue data according to semantic boundaries to obtain several second data segments; each second data segment can be vectorized and encoded to obtain a corresponding vector; and each second data segment and its corresponding vector can be stored in the second data sub-library. Thus, the second data segments can be single-turn question-and-answer pairs, slices of content from consecutive multi-turn dialogues, or segments of paragraphs from conference speeches, etc.

[0150] For example, for the transcribed text of the voice dialogue data, the smart device can segment it according to the meeting identifier (ID), timestamp, speaker, and topic tag, dividing the speech content of a single meeting into multiple semantically complete data segments. Each data segment is labeled with the corresponding meeting identifier, speaker, and timestamp information to support subsequent refined retrieval by meeting dimension. For historical question and answer records, the smart device can treat the user's question and the device's answer in each interaction as an independent question and answer pair, or segment multiple consecutive rounds of dialogue according to the dialogue window to obtain data segments composed of continuous question and answer sequences.

[0151] It should be understood that, based on this, the retrieval logic in step S4021 when the target data sub-database is the second data sub-database can be implemented through the following steps: Step S4021': If the target data sub-library is determined to be the second data sub-library according to the task type, a target data segment matching the current request vector is retrieved from the voice dialogue data or the historical question and answer records according to the task type.

[0152] It should be noted that when a smart device determines the target data sub-database as the second data sub-database based on the task type, the smart device can perform a search within the data range contained in the second data sub-database.

[0153] It should also be noted that, since the second data sub-database contains both second data segments from voice dialogue data and second data segments from historical question and answer records, the smart device can determine the retrieval source based on further subdivision of the task type during retrieval, and then retrieve the target data segment that matches the current request vector.

[0154] Step S4022: Integrate the target data fragments to obtain target associated data for replying to the request to be replied to.

[0155] In the specific implementation, after retrieving the target data fragments from the target data sub-database, the smart device can integrate and process the retrieved target data fragments: First, sort the multiple target data fragments from high to low according to their similarity scores with the current request vector, and select a preset number of target data fragments with the highest similarity; then, remove redundant information and duplicate content from each target data fragment; finally, concatenate the sorted target data fragments according to a preset format to generate a structured context text block, which is the target associated data used to reply to the request to be replied to.

[0156] Step S403: Input the request to be replied to and the target associated data into the target reasoning model to obtain the current answer.

[0157] It should be noted that after obtaining the target-related data, the smart device combines the request to be replied to with the target-related data to construct an enhanced model input. The smart device can use a preset prompt word template to concatenate the request to be replied to and the target-related data into a unified input sequence.

[0158] For example, the prompt template could be constructed as: "Please answer the user's question based on the following reference information. Reference information: {target related data}. User question: {request to be answered}."

[0159] In a practical implementation, a smart device can encode the constructed input sequence according to the format required by the target reasoning model and then input it into the target reasoning model. The target reasoning model then generates an answer that incorporates local knowledge data based on reference information and user questions, and outputs the current answer.

[0160] It should be understood that the target reasoning model generates answers by combining target-related data. Since the target-related data contains local knowledge fragments that semantically match the request to be answered, the target reasoning model can generate more accurate answers based on the aforementioned target-related data. This effectively improves the shortcomings of simply relying on model parameter memory, which results in vague answers or an inability to answer questions about private knowledge.

[0161] Furthermore, you can also refer to this section. Figure 5 This application describes the process by which the target reasoning model outputs the current answer. Figure 5 This is a schematic diagram of the reasoning process of the target reasoning model of this application.

[0162] Depend on Figure 5 It is understood that after receiving a user's pending response request, the smart device first determines whether the task type of the pending response request meets the preset local task type. The preset local task types include meeting retrieval, summary generation, complex reasoning, and multi-turn dialogue. All of the above task types require meeting records, historical dialogues, or knowledge data pre-uploaded by the user stored locally to complete the response.

[0163] If the preset local task type is met, the smart device can execute an offline RAG process based on a local vector database: calling the locally deployed embedding model to vectorize the request to be replied to, obtaining a current request vector that can represent the semantics of the request text. Simultaneously, the smart device can determine the target data sub-database based on the task type: when the task type is conference retrieval, summary generation, or multi-turn dialogue, the second data sub-database (built based on historical dialogue data) is determined as the target data sub-database; when the task type involves querying and referencing user-uploaded knowledge data, the first data sub-database (built based on user-uploaded knowledge data) is determined as the target data sub-database; when the task type is complex reasoning and simultaneously depends on both types of data, both the first and second data sub-databases are determined as target data sub-databases.

[0164] Next, the smart device calculates the similarity between the current request vector and each pre-stored vector in the target data sub-database, retrieves the target data fragments that match the current request vector, and then sorts the retrieved target data fragments from high to low similarity to obtain the Top-K target data fragments that match the current request vector.

[0165] After obtaining the target data fragment, the intelligent device sorts the multiple target data fragments that can be retrieved from high to low similarity, removes duplicates, and concatenates them to integrate them into target-related data, which serves as contextual information for the target reasoning model to refer to.

[0166] Finally, the smart device can input the request to be replied to and the target-related data together into the target reasoning model using a preset prompt word template. The target reasoning model then generates a current answer that incorporates local knowledge data based on the target-related data and outputs it through screen display, voice broadcast, or APP synchronization.

[0167] If the preset local task type is not met (e.g., device control or simple question and answer), it indicates that the request does not involve local knowledge data and does not require triggering a retrieval process. The smart device can skip the retrieval step and directly input the request to be answered into the target inference model. After performing inference, the target inference model outputs the current answer.

[0168] This embodiment triggers local vector retrieval when the task type of the request to be replied to meets the preset local task type. It can selectively retrieve target-related data semantically matching the current request from the local vector database, and input the retrieval results along with the request to be replied to into the target inference model. This allows the model to reference local meeting records, historical dialogues, and pre-uploaded knowledge data when generating answers, effectively improving the accuracy and information richness of private knowledge question answering in offline scenarios. Simultaneously, this embodiment can also select a matching target data sub-database from the first and second data sub-databases for retrieval based on the task type, avoiding the waste of computational resources caused by global retrieval and improving retrieval efficiency and hit rate. Since the entire retrieval and inference process is completed locally on the smart device, the user's voice dialogue data, historical question-and-answer records, and uploaded knowledge data do not need to be uploaded to the cloud, thus ensuring the security of user privacy data.

[0169] Based on the first and second embodiments of this application, in the third embodiment of this application, the content that is the same as or similar to that in embodiments one and two above can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 6 , Figure 6 This is a flowchart illustrating the third embodiment of the question-answering method for the client-side large language model in this application.

[0170] In this embodiment, to ensure that the resource consumption of the target inference model during the inference process is adapted to the current computing resource level of the intelligent device, the inference parameters of the target inference model can also be adjusted. Therefore, step S403 further includes: steps S4031~S4032: Step S4031: Obtain the standard inference parameters of the target inference model, and adjust the standard inference parameters according to the performance status data to obtain the running inference parameters of the target inference model.

[0171] It should be noted that standard inference parameters are the inference parameters of the target inference model under default configuration, which are used to control the computational resource consumption and generation behavior during model runtime. Standard inference parameters may include at least one of the following: model accuracy, upper limit of context length, upper limit of thread count, batch size, and sampling parameters.

[0172] Among them, model precision refers to the numerical precision of weights and activation values ​​during model inference, including INT8, INT4, or mixed precision. Higher precision results in better inference quality but also increases computational and memory usage. The upper limit of context length is the maximum length of the input sequence that the model can process in a single inference. The larger this value, the richer the context information that the model can refer to, but also increases computational cost. The upper limit of the number of threads is the maximum number of CPU or NPU parallel threads that the model can use during inference. The larger this value, the faster the inference speed but also the higher the instantaneous power consumption. The batch size is the number of samples processed simultaneously in a single forward propagation of the model. The larger this value, the higher the computational efficiency but also the higher the memory usage. Sampling parameters are parameters that control the randomness and diversity of the output during text generation, such as temperature coefficient and kernel sampling threshold (Top-p).

[0173] It should be understood that intelligent devices can adjust the standard inference parameters based on the aforementioned performance status data to obtain the operating inference parameters of the target inference model under the current resource conditions.

[0174] Furthermore, to specifically illustrate how to adjust the standard inference parameters based on device data for each performance dimension in the performance status data, step S4031 also includes: steps S40311~S40314: Step S40311: Obtain the standard inference parameters of the target inference model.

[0175] It should be noted that smart devices can read the standard inference parameters corresponding to the target inference model from local storage. Each large language model can correspond to a set of preset standard inference parameters. These standard inference parameters are pre-configured based on the size of the large language model's parameters and the hardware specifications of the smart device during model deployment and are stored in the smart device's local storage.

[0176] For example, models with a large number of parameters can be configured with higher standard inference parameters (e.g., a maximum of 4 threads and a maximum context length of 4096), while models with a small number of parameters can be configured with lower standard inference parameters (e.g., a maximum of 2 threads and a maximum context length of 2048). After determining the target inference model, the smart device can read the standard inference parameters corresponding to the target inference model as the baseline values ​​for subsequent adjustments.

[0177] Step S40312: Determine the priority order of each performance dimension in the performance status data, and determine the dimension weight corresponding to each performance dimension according to the priority order. The priority order is determined based on the degree of deviation between the corresponding values ​​of different performance dimensions in the performance status data and the standard values ​​of different performance dimensions. The greater the degree of deviation, the higher the priority of the corresponding performance dimension.

[0178] It should be noted that each performance dimension may include at least one of the following: battery level, temperature, charging status, discharging status, memory usage, and processor load. The standard values ​​for each performance dimension can be reference values ​​or expected values ​​under normal operating conditions.

[0179] For example, the standard value for temperature can be set to the middle or upper limit of the recommended temperature range for normal operation of the smart device, and the standard value for battery power can be set to the full battery value or high battery threshold of the smart device, etc.

[0180] It should be understood that the degree of deviation is the degree of difference between the current value of each performance dimension and its standard value. The degree of deviation can include the degree of downward deviation when the current value is lower than the preset lower threshold, or the degree of upward deviation when the current value is higher than the preset upper threshold.

[0181] For example, the standard value for temperature can be set to 37 degrees Celsius, and the preset upper threshold can be set to 40 degrees Celsius. When the current temperature is 45 degrees Celsius, the deviation is (45 degrees Celsius - 40 degrees Celsius) = 5 degrees Celsius upward deviation. The standard value for battery level can be set to 100%, and the preset lower threshold can be set to 20%. When the current battery level is 15%, the deviation is (20% - 15%) = 5%. The above values ​​are for illustrative purposes only. In actual applications, the standard values ​​and preset thresholds can be set according to the specific hardware specifications of the smart device.

[0182] It should be understood that smart devices can calculate the degree of deviation between the current value of each performance dimension and its standard value, and sort the performance dimensions from largest to smallest according to the degree of deviation to determine the priority order of each performance dimension. The performance dimension with the larger the deviation, the higher its priority. After determining the priority order, the smart device then determines the corresponding dimension weight for each performance dimension based on the priority order.

[0183] The weights for each performance dimension can be determined using a preset weight allocation rule. This rule assigns the highest priority performance dimension the largest weight, the lowest priority performance dimension the smallest weight, and the remaining performance dimensions are assigned decreasing weights in order of priority.

[0184] Step S40313: Merge the weights of each dimension to obtain the comprehensive weight.

[0185] It should be understood that smart devices can use a weighted product approach to fuse the weights of each dimension and obtain a comprehensive weight. A weighted product is a method of multiplying the weights corresponding to each performance dimension to obtain the comprehensive weight.

[0186] For example, if the dimension weight corresponding to the power dimension is 0.8, the dimension weight corresponding to the temperature dimension is 0.6, and the dimension weight corresponding to the load dimension is 0.9, then the overall weight = 0.8 * 0.6 * 0.9 = 0.432.

[0187] In addition, in some other embodiments, the above fusion operation can also be performed using a weighted summation method, that is, a weighted average or a linear weighted summation of the weights of each dimension. This embodiment does not limit this.

[0188] Step S40314: Adjust the standard inference parameters according to the comprehensive weight to obtain the running inference parameters of the target inference model.

[0189] It should be understood that the adjustment method adopted by the intelligent device can be: multiplying the parameter values ​​in the standard inference parameters with the comprehensive weight, and using the product as the parameter value of the corresponding item in the running inference parameters.

[0190] For example, if the standard value for the upper limit of the number of threads is 4 and the overall weight is 0.75, then the adjusted upper limit of the number of threads is round(4*0.75)=3; if the standard value for the upper limit of the context length is 4096 and the overall weight is 0.75, then the adjusted upper limit of the context length is max(512, round(4096*0.75))=3072. Furthermore, lower limits can be set for some parameters, such as an upper limit of context length not less than 512 and an upper limit of the number of threads not less than 1, to ensure that the target inference model can still provide basic question-answering services when resources are severely limited.

[0191] It should also be noted that the overall weight is positively correlated with the inference performance indicated by the running inference parameters. That is, the larger the overall weight, the higher the inference performance corresponding to the adjusted running inference parameters (such as higher thread count, longer context, and higher model accuracy), and the smaller the overall weight, the lower the inference performance corresponding to the adjusted running inference parameters. This allows the target inference model to run at high performance when the device performance is good. As a result, intelligent devices can automatically reduce resource consumption when device performance is limited, achieving dynamic adaptation of the inference process to the device performance state.

[0192] Furthermore, in some embodiments, the smart device can also directly execute an inference parameter adjustment strategy based on the recording task state: the smart device can acquire recording task state data to characterize whether the smart device is in a recording state; when the recording task state data indicates that the smart device is in a recording state, the upper limit of the number of threads can be directly and forcibly reduced (e.g., from 4 threads to 2 threads), and the upper limit of the context length can be shortened (e.g., from 4096 to 2048) to determine the running inference parameters adapted to the recording state. When the recording task ends, i.e., when the smart device returns to a non-recording state, the smart device can revert to using the running inference parameters obtained by adjusting the standard inference parameters with comprehensive weights.

[0193] By adopting the aforementioned inference parameter adjustment strategy based on the recording task status, intelligent devices can avoid competition between inference tasks and recording tasks for processor and memory resources, thereby ensuring the integrity of the recording process and the reliability of the recording data.

[0194] Step S4032: Configure the target inference model according to the running inference parameters, and input the request to be replied to and the target associated data into the configured target inference model to obtain the current answer.

[0195] In a specific implementation, the intelligent device can configure the target inference model according to the aforementioned running inference parameters, including setting the model's precision mode, upper limit of context length, upper limit of the number of threads, batch size, and sampling parameters.

[0196] Once configured, the smart device can encode the request to be replied to and the target related data according to the input format required by the target inference model, and input them into the configured target inference model to perform inference, so that the model generates and outputs the current answer under the currently configured inference parameters.

[0197] Furthermore, this can be referenced here. Figure 7 This application describes the process of adjusting the inference parameters of the target inference model, i.e., the dynamic computing power allocation process. Figure 7 This is a schematic diagram of the dynamic computing power allocation of the intelligent device in this application.

[0198] Depend on Figure 7 It is known that intelligent devices monitor performance status data in real time. The performance status data includes current battery level (e.g., 35%), current temperature (e.g., 42℃), charging status (e.g., charging), memory usage (e.g., 68%), processor load (e.g., 75%), recording task status (e.g., recording), and external discharge status (e.g., high power output).

[0199] After acquiring the aforementioned performance status data, the smart device can determine the weight of each performance dimension based on its current value and the corresponding preset threshold. For example, when the smart device is in a high-power discharge state and a recording task is in progress, the weights of the discharge state dimension and the recording task dimension are increased to prioritize power supply stability and the integrity of the recording task; when the smart device is in a charging state and the temperature is high, the weight of the temperature dimension is increased to prioritize controlling the device's temperature rise.

[0200] Next, the intelligent device integrates the weights of each dimension to obtain a comprehensive weight, and determines the current inference resource level based on the comprehensive weight, for example... Figure 7 The P1 balance level is shown.

[0201] Subsequently, the intelligent device determines the corresponding inference parameters based on the current inference resource level, including model accuracy (e.g., INT8), number of concurrent threads (e.g., 4 threads), maximum context length (e.g., 2048), batch size (e.g., 1), and inference frequency (e.g., medium). Each inference parameter is positively correlated with the current inference resource level: the higher the current inference resource level, the higher the inference performance corresponding to the parameters (e.g., higher number of threads, longer context length, higher model accuracy); the lower the current inference resource level, the lower the inference performance corresponding to the parameters (e.g., lower number of threads, shorter context length, lower model accuracy).

[0202] Finally, the intelligent device can configure the target inference model according to the above-mentioned running inference parameters, so that the target inference model can adapt to the current performance state of the intelligent device during operation, thereby realizing dynamic computing power allocation in the inference process.

[0203] This embodiment can obtain the standard inference parameters of the target inference model, determine the dimension weights based on the deviation of each performance dimension in the performance status data, and dynamically adjust the standard inference parameters after fusing them to obtain the comprehensive weight. This makes the parameters such as the number of threads, context length, and model accuracy of the target inference model match the current performance status of the device, such as battery power, temperature, and load. In this way, while ensuring the quality of the usable answers output by the smart device, the power consumption and computing resource consumption of the inference process can be effectively controlled, avoiding situations such as a sudden drop in battery life or temperature runaway caused by excessive inference load.

[0204] Furthermore, you can also refer to this section. Figure 8 and Figure 9 This paper describes the entire process of the question-answering method of the client-side large language model in this application. Figure 8 This is a system architecture diagram of the question-answering method based on the client-side large language model in this application. Figure 9This is a schematic diagram of the entire process of the question-answering method of the client-side large language model in this application.

[0205] Depend on Figure 8 As can be seen, the overall system architecture of the question-answering method of the edge-side large language model in this application uses portable smart devices (such as AI recording power banks, mobile power banks with screens, AI recording earphone cases, voice recorders, etc.) as the execution body. Among them, the smart device deploys an edge-side large model inference engine, which may include a model management module, a device performance awareness module, a model selection and scheduling module, a runtime inference parameter configuration module, an inference execution module, and a local vector database.

[0206] The model management module manages at least two locally deployed large language models with different parameter counts (such as small parameter models, medium parameter models, and large parameter models); the device performance awareness module monitors the performance status data of the smart device in real time, including battery status, temperature status, charging status, memory usage, processor load, and network status; the local vector database stores the local data required for offline RAG, including a first data sub-database built based on user-uploaded knowledge data and a second data sub-database built based on user historical dialogue data.

[0207] The model selection and scheduling module is connected to the device performance perception module and the model management module, respectively. It is used to determine the target inference model from at least two large language models managed by the model management module based on the performance status data monitored by the device performance perception module and the current task complexity corresponding to the request to be responded to. The runtime inference parameter configuration module is used to determine the runtime inference parameters of the target inference model based on performance status data; the inference execution module is used to configure the target inference model according to the runtime inference parameters, and input the request to be replied to and the target association data retrieved from the local vector database into the configured target inference model to perform inference and obtain the current answer.

[0208] Among them, the data flow is between the device performance perception module and the model selection and scheduling module; the control flow is between the model selection and scheduling module and the model management module, and between the running inference parameter configuration module and the inference execution module; and the interaction / communication is between the inference execution module and the user.

[0209] Through the above architecture, a closed-loop question-and-answer service with dynamic computing power allocation and offline local knowledge retrieval enhancement can be implemented on resource-constrained smart devices.

[0210] Based on the above system architecture Figure 9 This further illustrates the entire process of the question-answering method of the client-side large language model in this application.

[0211] Depend on Figure 9It can be seen that after receiving a user's request for a response, the smart device can simultaneously execute two paths: Path 1 is to obtain performance status data, including at least one of the following: battery data, temperature data, charging status data, memory usage data, and processor load data; Path 2 is to perform intent recognition on the request for a response and obtain the information on the need for the response, and comprehensively determine whether the complexity of the current task is simple or complex.

[0212] Next, the intelligent device can determine the target inference model from at least two large language models with different parameter sizes based on task complexity and performance status data: When the task complexity is simple, regardless of the current performance status of the smart device, a model with a small number of parameters can be selected as the target inference model to complete the response with the lowest power consumption.

[0213] When the task complexity is complex, the intelligent device can further determine the current inference resource level (high performance level, balanced level, energy saving level, or protection level) based on performance status data: if it is a high performance level or balanced level, the intelligent device can choose a model with a large number of parameters as the target inference model to ensure the quality of the answer; if it is an energy saving level or protection level, the intelligent device can choose a model with a small number of parameters as the target inference model to ensure that it can still provide available services under resource-constrained conditions.

[0214] After determining the target inference model, the intelligent device can determine whether the request to be responded to needs to be retrieved using a local vector database. If the task type is a preset local task type such as conference retrieval, summary generation, complex inference, or multi-turn dialogue, the local RAG process is triggered, sequentially executing the steps of request vectorization, determining the target data sub-database (first data sub-database and / or second data sub-database) according to the task type, retrieving and obtaining target data fragments, and integrating them into target-related data.

[0215] If the task type is device control or simple question and answer, local knowledge data is not involved, and the device skips the retrieval step.

[0216] In addition, smart devices can obtain standard inference parameters of the target inference model, determine the weight of each dimension based on the degree of deviation of each performance dimension in the performance status data and integrate them into a comprehensive weight, and adjust the standard inference parameters according to the comprehensive weight to obtain running inference parameters that match the current performance status.

[0217] Finally, the smart device can configure the target inference model according to the running inference parameters, input the request to be replied to and the target related data (if performing RAG retrieval) into the target inference model to perform inference, obtain the current answer and output it.

[0218] The question-answering method based on the large language model in this application achieves a balance between answer quality and device performance by dynamically selecting an appropriate target inference model based on task complexity and performance status. Through the construction and retrieval of a local vector database, the smart device can still perform enhanced generation based on user history dialogues and pre-uploaded knowledge data in offline scenarios, effectively improving the accuracy and information richness of offline knowledge-based question answering. By dynamically adjusting inference parameters according to performance status data, parameters such as model accuracy, context length, and number of threads adapt to changes in device power consumption, temperature, and other conditions. This effectively reduces the power consumption and computational overhead of the smart device while ensuring answer quality, extending the device's battery life, and avoiding interruptions to the question-answering service due to excessive load, thus achieving a balance between device performance and task processing requirements.

[0219] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the question-answering method of the large language model on the client side of this application. Any simple transformations based on this technical concept are within the protection scope of this application.

[0220] In addition, this application also provides a smart device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to execute the question-and-answer method of the edge-side large language model in the above embodiment 1.

[0221] The following is for reference. Figure 10 , Figure 10 This is a schematic diagram of the structure of the smart device of this application. The smart device in the embodiments of this application may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, personal digital assistants (PDAs), tablet computers (PADs), portable media players (PMPs), etc., as well as fixed terminals such as digital TVs, desktop computers, etc. Figure 10 The smart device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0222] like Figure 10As shown, the intelligent device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 1002 or a program loaded from storage device 1003 into random access memory (RAM) 1004. The RAM 1004 also stores various programs and data required for the operation of the intelligent device. The processing unit 1001, ROM 1002, and RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to the I / O interface 1006: input devices 1007 including, for example, touchscreens, touchpads, keyboards, mice, image sensors, microphones, accelerometers, gyroscopes, etc.; output devices 1008 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 1003 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1009. Communication device 1009 allows smart devices to communicate wirelessly or wiredly with other devices to exchange data. While the figures show smart devices with various systems, it should be understood that implementing or having all of the systems shown is not required. More or fewer systems may be implemented alternatively.

[0223] The intelligent device provided in this application employs the question-answering method of the edge-side large language model in the above embodiments, which can solve the technical problem that intelligent devices struggle to balance answer quality and power consumption under performance constraints. Compared with the prior art, the beneficial effects of the intelligent device provided in this application are the same as those of the question-answering method of the edge-side large language model provided in the above embodiments, and other technical features of this intelligent device are the same as those disclosed in the method of the first embodiment above, and will not be repeated here.

[0224] This application also provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to execute the question-and-answer method of the end-side large language model in the above embodiments.

[0225] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0226] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the question-answering method of the aforementioned endpoint large language model, thereby solving the technical problems of the question-answering method of the endpoint large language model. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as the beneficial effects of the question-answering method of the endpoint large language model provided in the above embodiments, and will not be repeated here.

[0227] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other elements in the process, method, article, or system that includes that element.

[0228] The above embodiment numbers are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments. They are only some embodiments of this application and do not limit the scope of this application. All equivalent structural transformations made under the technical concept of this application and using the content of this application specification and drawings, or direct / indirect applications in other related technical fields, are included within the protection scope of this application.

Claims

1. A question-answering method for a large edge-side language model, characterized in that, The method, applicable to intelligent devices with at least two large language models deployed locally, each with a different number of model parameters, includes: Upon receiving a user's pending response request, the performance status data of the smart device is obtained; Determine the complexity of the current task based on the pending request; The target inference model is determined from the at least two large language models based on the current task complexity and the performance status data; The request to be answered is input into the target reasoning model to obtain the current answer.

2. The method as described in claim 1, characterized in that, The step of determining the complexity of the current task based on the request to be responded to includes: The intent to be responded to is identified, the intent identification result is obtained, and the task type of the request to be responded to is determined based on the intent identification result; Obtain the response requirement information of the request to be responded to, wherein the response requirement information is information used to characterize the response constraints of the request to be responded to; Based on the task type and the response requirement information, determine the current task complexity of the request to be responded to.

3. The method as described in claim 1, characterized in that, The task complexity is described as simple or complex. The step of determining the target inference model from the at least two large language models based on the current task complexity and the performance status data includes: Determine the preset judgment conditions satisfied by the performance status data, and determine the current inference resource level of the smart device based on the satisfied preset judgment conditions. The current inference resource level is a high-performance level, a balanced level, an energy-saving level, or a protection level. When the task complexity is complex and the current inference resource level is high performance or balanced, a large language model with a model parameter count higher than a preset first parameter count threshold is identified as the target inference model. When the task complexity is complex and the current inference resource level is energy-saving or protection level, a large language model with no more than a preset first parameter threshold is determined as the target inference model. When the task complexity is simple, a large language model with no more than a preset first parameter threshold is determined as the target inference model.

4. The method as described in claim 1, characterized in that, The step of inputting the request to be answered into the target inference model to obtain the current answer includes: When the task type of the request to be replied to meets the preset local task type, the request to be replied to is vectorized and encoded to obtain the current request vector; Based on the current request vector, a search is performed in the local vector database to obtain target associated data for replying to the request to be replied to. The local vector database includes associated data corresponding to different request vectors. The pending response request and the target-related data are input into the target reasoning model to obtain the current answer.

5. The method as described in claim 4, characterized in that, The local vector database includes a first data sub-database and a second data sub-database. The first data sub-database is constructed based on the knowledge data pre-uploaded by the user, and the second data sub-database is constructed based on the user's historical dialogue data. The first data sub-database includes several first data segments, and the second data sub-database includes several second data segments. The step of retrieving target associated data for replying to the request based on the current request vector in the local vector database includes: A target data sub-database is determined based on the task type, and a target data fragment matching the current request vector is retrieved from the target data sub-database. The target data fragments are integrated to obtain target associated data for responding to the request to be responded to.

6. The method as described in claim 5, characterized in that, Before the step of obtaining the performance status data of the smart device upon receiving a user's pending response request, the method further includes: Record historical voice interactions or meeting speeches authorized by the user to obtain voice dialogue data; Obtain the user's historical question-and-answer records with the smart device; The voice dialogue data and the historical question-and-answer records are used as historical dialogue data, and a second data sub-database is constructed based on the historical dialogue data; The step of determining the target data sub-database according to the task type and retrieving the target data fragment matching the current request vector from the target data sub-database includes: If the target data sub-library is determined to be the second data sub-library based on the task type, a target data segment matching the current request vector is retrieved from the voice dialogue data or the historical question-and-answer records based on the task type.

7. The method as described in claim 4, characterized in that, The step of inputting the request to be answered and the target associated data into the target inference model to obtain the current answer includes: Obtain the standard inference parameters of the target inference model, and adjust the standard inference parameters according to the performance status data to obtain the running inference parameters of the target inference model; Configure the target inference model according to the running inference parameters, and input the request to be replied to and the target associated data into the configured target inference model to obtain the current answer.

8. The method as described in claim 7, characterized in that, The step of obtaining the standard inference parameters of the target inference model and adjusting the standard inference parameters according to the performance status data to obtain the running inference parameters of the target inference model includes: Obtain the standard inference parameters of the target inference model; The priority order of each performance dimension in the performance status data is determined, and the dimension weight corresponding to each performance dimension is determined according to the priority order. The priority order is determined based on the degree of deviation between the corresponding value of different performance dimensions in the performance status data and the standard value of different performance dimensions. The greater the degree of deviation, the higher the priority of the corresponding performance dimension. The weights of each dimension are combined to obtain a comprehensive weight; The standard inference parameters are adjusted according to the comprehensive weights to obtain the running inference parameters of the target inference model.

9. A smart device, characterized in that, The intelligent device includes: a memory, a processor, and a question-answering program stored in the memory and executable on the processor, the question-answering program being configured to implement the question-answering method of the edge-side large language model as described in any one of claims 1 to 8.

10. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a question-answering program is stored on the storage medium. When the question-answering program is executed by the processor, it implements the question-answering method of the end-side large language model as described in any one of claims 1 to 8.

Images