End-side large model inference acceleration method, device, equipment, storage medium and product

Abstract

The application discloses an end-side large model reasoning acceleration method, device, equipment, storage medium and product, the method comprises the following steps: receiving the prompt word information input by a user, starting a parallel execution reasoning thread and a cooperative thread; sending the prompt word information to a first large model of an edge cloud end through the cooperative thread for reasoning to obtain reasoning guidance information; and sending the reasoning guidance information to the coding stage of a second large model of a terminal equipment through the reasoning thread to perform reasoning on the prompt word information to obtain reply information of the prompt word information; by adding the reasoning guidance information of the second large model of the edge cloud end to the coding stage of the second large model of the terminal equipment, the reasoning of the second large model is guided, and the reasoning performance of the end-side large model can be improved without affecting the reasoning response speed of the end-side large model.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method, apparatus, device, storage medium, and product for accelerating large-scale model inference on the edge. Background Technology

[0002] Large Language Models (LLMs) on the device side refer to large language models (LLMs) that run efficiently on resource-constrained terminal devices (such as computers, tablets, and mobile phones). While maintaining high performance, LLMs offer faster data processing speeds and better privacy protection than server-side LLMs, meeting users' needs for real-time performance, privacy protection, and offline capabilities.

[0003] Larger models have more parameters and greater capabilities, but also require more computing power. To reduce computational and storage demands on the device side and improve inference, current methods primarily involve reducing model size through large model compression or using dedicated hardware acceleration chips to speed up the inference process. However, compression of large models reduces their capabilities, impacting inference performance. Furthermore, using dedicated hardware acceleration chips on terminal devices is limited by their resource constraints; these chips have significantly less computing power than those on servers and do not support horizontal scaling for performance improvement, further affecting inference performance. Summary of the Invention

[0004] To address the problems existing in the prior art, embodiments of the present invention provide a method, apparatus, device, storage medium, and product for accelerating edge-side large model inference, which improves the inference performance of edge-side large models without affecting the inference response speed of edge-side large models.

[0005] In a first aspect, embodiments of the present invention provide a method for accelerating large-model inference on an edge device, applied to a terminal device, including: Receive prompt information from the user and start inference threads and cooperating threads to execute in parallel; The prompt word information is sent to the first large model in the edge cloud for inference through the collaborative thread to obtain inference guidance information; The reasoning thread inserts the reasoning guidance information into the encoding stage of the second major model of the terminal device to reason about the prompt word information, thereby obtaining the response information of the prompt word information.

[0006] As an improvement to the above scheme, the inference guidance information is inserted into the encoding stage of the second major model of the terminal device through the inference thread to infer the prompt word information, thereby obtaining the response information of the prompt word information, including: The inference thread inputs the prompt word information into the second large model for inference; In the pre-filling stage of the second major model, the prompt word information is pre-filled to obtain the first word key value vector; During the decoding phase of the second major model, it is determined whether to receive the inference guidance information output by the first major model; If the reasoning guidance information is received, the reasoning guidance information is inserted into the first word key value vector for reasoning encoding to obtain the response information of the prompt word information.

[0007] As an improvement to the above scheme, the reasoning guidance information is inserted into the first lexical key-value vector for reasoning encoding to obtain the response information of the prompt word information, including: The reasoning guidance information is pre-filled to obtain the second word key value vector of the reasoning guidance information; The first word key vector is appended to the end of the second word key vector to generate the third word key vector. The response information is obtained by inferring and encoding the third word key value vector.

[0008] As an improvement to the above scheme, the reasoning guidance information includes: first guidance word information or at least one second guidance word information; Wherein, the first guiding meta-information includes the keyword meta-information output by the first large model after reasoning about the prompt word information; The second guiding meta-information includes the keyword meta-information output by the first large model after completing partial reasoning on the prompt word information.

[0009] As an improvement to the above scheme, the second guiding meta-information includes the keyword meta-information generated by the first large model during the encoding stage according to a preset timeout response strategy; the timeout response strategy is used to indicate that the keyword meta-information generated during the current inference time is output when the current inference time reaches a preset inference time threshold.

[0010] As an improvement to the above scheme, the reasoning guidance information is pre-filled to obtain a second lexical key-value vector of the reasoning guidance information, including: The reasoning guidance information is segmented into words to obtain a segmented sequence; The word segmentation sequence is subjected to word embedding processing to obtain the first word embedding vector; Add positional encoding to the word embedding vector to obtain the second word embedding vector; Attention calculation and feedforward network calculation are performed on the second word embedding vector to generate the second word key value vector.

[0011] Secondly, embodiments of the present invention provide an edge-side large model inference acceleration device, applied to a terminal device, comprising: The prompt word receiving module is used to receive prompt word information input by the user and start inference threads and cooperating threads that are executed in parallel; The prompt word sending module is used to send the prompt word information to the first large model in the edge cloud for inference through the collaborative thread, so as to obtain inference guidance information; The reasoning module is used to infer the prompt word information by inserting the reasoning guidance information into the encoding stage of the second major model of the terminal device through the reasoning thread, so as to obtain the response information of the prompt word information.

[0012] Thirdly, embodiments of the present invention provide an edge-side large model inference acceleration device, comprising: a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor executes the computer program to implement the edge-side large model inference acceleration method as described in any one of the first aspects.

[0013] Fourthly, embodiments of the present invention provide a computer-readable storage medium storing a computer program, wherein, when the computer program is executed, it controls the device where the computer-readable storage medium is located to perform the edge-side large model inference acceleration method as described in any one of the first aspects.

[0014] Fifthly, embodiments of the present invention provide a computer program product, including a computer program or instructions, which, when executed by a processor, implement the edge-side large model inference acceleration method as described in any one of the first aspects.

[0015] Compared to existing technologies, this invention provides a method, apparatus, device, storage medium, and product for accelerating large-scale inference on the edge. When a terminal device receives prompt information input by a user, it initiates parallel execution of an inference thread and a collaborative thread. The collaborative thread sends the prompt information to a first large-scale model in the edge cloud for inference, obtaining inference guidance information. The inference thread then inserts the inference guidance information into the encoding stage of a second large-scale model on the terminal device to infer the prompt information, obtaining a response to the prompt information. This invention improves the inference performance of the edge-side large-scale model without affecting its inference response speed by inserting inference guidance information from the edge cloud into the encoding stage of the second large-scale model on the terminal device. Attached Figure Description

[0016] To more clearly illustrate the technical solution of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a flowchart of an edge-side large model inference acceleration method provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of the interaction process between the edge cloud and the terminal device provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of another interaction process between the edge cloud and the terminal device provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of the reasoning guidance information insertion process in the encoding stage of the second major model of the terminal device provided in this embodiment of the invention; Figure 5 This is a structural block diagram of an end-side large model inference acceleration device provided in an embodiment of the present invention; Figure 6 This is a structural block diagram of an end-side large model inference acceleration device provided in an embodiment of the present invention. Detailed Implementation

[0018] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] It is understood that the various numerical designations used in the embodiments of this invention are merely for descriptive convenience and are not intended to limit the scope of this application. The order of the process numbers does not imply the order of execution; the execution order of each process should be determined by its function and internal logic.

[0020] In embodiments of the invention, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, without necessarily requiring or implying any such actual relationship or order between these entities or operations. The terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising" does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes said element. The term "a plurality or several" refers to two or more.

[0021] See Figure 1 , Figure 1 This is a flowchart of a method for accelerating large-model inference on the edge, provided by an embodiment of the present invention. The method for accelerating large-model inference on the edge, applied to a terminal device, includes: S11: Receive prompt information from the user and start parallel execution of inference and cooperating threads; S12: The prompt word information is sent to the first large model in the edge cloud for inference through the collaborative thread to obtain inference guidance information; S13: The reasoning guidance information is inserted into the encoding stage of the second major model of the terminal device through the reasoning thread to reason about the prompt word information and obtain the response information of the prompt word information.

[0022] Understandably, the inference process of a large model includes a prefill phase and an encoding phase. The prefill phase is executed only once and generates the first token, while the encoding phase is executed autoregressively to continuously generate the next token.

[0023] This invention, taking into account that edge cloud computing is close to user-side terminal devices and has the characteristics of low latency and fast response, deploys a powerful and well-resourced primary model in the edge cloud, which performs collaborative reasoning with the primary model deployed on the terminal devices, thereby supplementing and improving the reasoning capabilities of the primary models on each terminal device. For example... Figure 2 As shown, the specific process of collaborative reasoning between the first major model in the edge cloud and the second major model in the terminal device is as follows: When the terminal device receives the prompt word information input by the user, it triggers an inference task. At this time, parallel execution of the inference thread and the coordination thread is started. The inference thread executes the inference task corresponding to the user-input prompt word information, that is, inference is performed by the second large model of the terminal device. Simultaneously, the coordination thread is awakened to synchronously send the prompt word information to the first large model in the edge cloud, so as to instruct the first large model in the edge cloud to process the inference task corresponding to the prompt word information in parallel, that is, to perform inference synchronously by the first large model in the edge cloud. When the coordination thread receives the inference guidance information returned by the first large model, it inserts (i.e., supplements) the inference guidance information into the decoding stage of the second large model through the receiver, such as before a certain decoding step. This allows the second large model of the terminal device to have more input information in the subsequent decoding stage, so that it can match better tokens during sampling and obtain the response information of the prompt word information. The second large model of the terminal device outputs the response information to the user in a streaming manner until the inference is completed. This invention improves the inference performance of the large model on the edge cloud by inserting inference guidance information of the second large model in the encoding stage of the second large model on the terminal device.

[0024] The reasoning guidance information includes: first guidance word information or at least one second guidance word information; Wherein, the first guiding meta-information includes the keyword meta-information output by the first large model after reasoning about the prompt word information; The second guiding meta-information includes the keyword meta-information output by the first large model after completing partial reasoning on the prompt word information.

[0025] Specifically, the second guiding meta-information includes keyword meta-information generated by the first large model during the encoding phase according to a preset timeout response strategy; the timeout response strategy is used to indicate that when the current inference time reaches a preset inference time threshold, the keyword meta-information generated during the current inference time will be output.

[0026] For example, during the encoding phase, whenever the inference time reaches a preset inference time threshold, the first large model outputs keyword meta-information generated for the corresponding inference time (including multiple word elements / key information output by the first large model inference), as a second guiding word element information.

[0027] In this embodiment of the invention, the response mechanism of the first major model in the edge cloud includes a normal response strategy and a timeout response strategy, such as... Figure 2As shown. Under a normal response strategy, such as in a normal network environment, the first large model outputs complete inference guidance information (i.e., the aforementioned first guiding lexical information, all lexical / key information output by the first large model after inference) after completing all inference of the prompt word information. Under a timeout response strategy, such as in a weak network environment, to ensure that the second large model on the terminal device can receive the inference guidance information returned from the edge cloud, a timeout interruption judgment is performed. Whenever the inference duration reaches a preset inference time threshold (e.g., 2 seconds), the stage-wise inference guidance information generated by the first large model within that inference duration (i.e., the aforementioned second guiding lexical information, i.e., the lexical / key information generated by the first large model within 2 seconds) is immediately returned. That is, the inference is interrupted and returned halfway, so that the second large model receives as much as possible. For example, every 2 seconds, the keyword lexical information output by the first large model during that 2-second inference time is returned as inference guidance information and sent to the second large model on the terminal device.

[0028] If the second largest model does not receive inference guidance information from the first largest model after inference is completed, or receives inference guidance information from the first largest model only after inference is completed, then the inference guidance information is discarded and not processed.

[0029] For example, such as Figure 3 As shown, the collaborative threads on the terminal device send the prompt word information in the form of requests to the first large model in the edge cloud for inference in parallel; Furthermore, the first major model in the edge cloud has strong computing power. In order to provide prompts to the second major model on the terminal device as soon as possible, the prompt word information is optimized to accelerate the generation of some keyword meta-information (including several word elements / key information), which is then input into the second major model for reasoning. This can accelerate the reasoning process of the second model and improve the reasoning accuracy.

[0030] For example, in edge cloud environments, it's necessary to optimize prompt word information and generate key information for the primary model. The primary model can then optimize the prompt word information based on the prompt word template. An example of prompt word information optimization is shown below: Suppose the prompt message issued by the second major model of the terminal device is: xxxxxxxx; List the key information for the answer to the question based on the prompt word template. The format of the list is: Key information: xxx, xxx, xxx.

[0031] Requirements: List at least 10 points, with each point containing no more than 10 key information points, and the total number of words not exceeding 100.

[0032] In this process, the prompt words need to be replaced with specific prompt words from the terminal device. Based on the prompt word template mentioned above, the first large model can optimize the prompt word information and generate key information related to the prompt word information very quickly through reasoning, as guidance information for reasoning. For example, the key information optimized by the first large model for "What is a large model?" is "large parameter scale, strong computing resources, multi-domain knowledge, deep learning ability, natural language processing, general model, self-evolution, reasoning ability, high resource consumption, and interpretability challenge".

[0033] In the decoding phase, the first major model at the edge cloud determines whether to interrupt the return of partial inference guidance information to the terminal device based on a pre-set timeout response strategy. For example, a timeout response strategy is set to ensure that the inference time does not exceed a threshold (e.g., 2 seconds). If the inference time exceeds 2 seconds, the generated keyword metadata (tokens) are returned to the terminal device, that is, the newly generated keyword metadata (tokens) are returned every 2 seconds to prevent the terminal device's second major model from completing its inference task before the message is returned. Under normal response conditions, after the first major model in the edge cloud completes inference quickly, it returns all the key keyword meta-information (tokens) that need to be focused on in this prompt word information as inference guidance information to the terminal device, so that the key keyword meta-information (tokens) can perform better local inference under this guidance; This invention, by setting two response mechanisms—a normal response strategy and a timeout response strategy—for the first major model in the edge cloud, ensures that even if the network is unstable, the second major model of the terminal device can still obtain some phased guidance information, avoiding zero feedback in the edge cloud under weak network conditions and ensuring that the core guidance intent is at least partially transmitted.

[0034] In one optional embodiment, the inference guidance information is inserted into the encoding stage of the second major model of the terminal device through the inference thread to infer the prompt word information, thereby obtaining the response information of the prompt word information, including: The inference thread inputs the prompt word information into the second large model for inference; In the pre-filling stage of the second major model, the prompt word information is pre-filled to obtain the first word key value vector; During the decoding phase of the second major model, it is determined whether to receive the inference guidance information output by the first major model; If the reasoning guidance information is received, the reasoning guidance information is inserted into the first word key value vector for reasoning encoding to obtain the response information of the prompt word information.

[0035] For example, such as Figure 3As shown, the second major model of the terminal device is also simultaneously undergoing pre-filling and autoregressive inference processes. In the pre-filling stage of the second major model, pre-filling calculations are performed on each lexical unit in the prompt word information. This includes attention calculations and feedforward network calculations, generating key-value pairs for each lexical unit, and then integrating all lexical unit key-value pairs to generate the first lexical unit key-value vector. For example: the prompt word information is segmented to obtain a prompt word segmentation sequence; the segmented prompt word sequence is then embedded to obtain a prompt word embedding vector; positional encoding is added to the prompt word embedding vector; attention calculations and feedforward network calculations are performed on the positionally encoded prompt word embedding vector to generate the first lexical unit key-value vector. The specific pre-filling calculation process is described below.

[0036] During the decoding stage of the second major model, a receiving flag is added to determine whether to receive inference guidance information from the first major model. If the receiver of the cooperative thread receives inference guidance information from the first major model (received in a FIFO queue), the receiving flag is enabled. Before the next decoding, once the receiving identifier is detected as enabled, the currently received inference guidance information is pre-filled to obtain the second token key-value vector, which is then appended to the currently calculated first token key-value vector. In subsequent attention calculations during the decoding phase, with more input information, better tokens will be matched during sampling.

[0037] Specifically, the reasoning guidance information is inserted into the first word key-value vector for reasoning encoding to obtain the response information of the prompt word information, including: The reasoning guidance information is pre-filled to obtain the second word key value vector of the reasoning guidance information; The first word key vector is appended to the end of the second word key vector to generate the third word key vector. The response information is obtained by inferring and encoding the third word key value vector.

[0038] Specifically, the reasoning guidance information is pre-filled to obtain a second lexical key-value vector, including: The reasoning guidance information is segmented into words to obtain a segmented sequence; The word segmentation sequence is subjected to word embedding processing to obtain the first word embedding vector; Add positional encoding to the word embedding vector to obtain the second word embedding vector; Attention calculation and feedforward network calculation are performed on the second word embedding vector to generate the second word key value vector.

[0039] At the terminal device, after receiving inference guidance information from the first major model in the edge cloud, this inference guidance information is "inserted" into the encoding stage of the second major model on the terminal device, such as... Figure 4 As shown, the specific procedure for cutting in line is as follows: When the second model receives the inference guidance information, it needs to undergo calculations such as tokenization, embedding, positional encoding, attention, and feedforward networks in the Transformer architecture, which is equivalent to a pre-filling process. To obtain the second token key-value vector (K'V' Cache) of the inference guidance information through the attention mechanism, sample sampling is not required. Only the K'V' Cache of each token in all inference guidance information tokens needs to be calculated in parallel. The inference guidance information tokens undergo a linear transformation of their own sequence to obtain three matrices: Q', K', and V'. Q' is multiplied by the transpose of K' and then softmax normalized to obtain a temporary A' value. A' is then multiplied by V' to finally obtain the K'V' Cache of that inference guidance information token. That is, the K'V' Cache comes from the attention algorithm. The calculation formula for the K'V' Cache of a single attention head is: (1); Where Q', K', and V' represent the query, key, and value vectors, respectively; T represents the transpose symbol.

[0040] Assuming the sequence length seq_length of the inference guidance information tokens is n, the K'V'Cache of the i-th token is calculated as follows: (2); in, It is the query vector of the i-th token in the reasoning guidance information tokens, i n, , ,... The key vector representing all lexical units in the reasoning guidance information tokens; , ,... This represents the value vector of all lexical units in the reasoning guidance information tokens.

[0041] The K'V' Cache for all reasoning guidance information tokens is calculated as follows: (3); in, , ,... This represents the query vector containing all terms in the inference guidance information tokens.

[0042] Furthermore, each attention contains h heads, and the cache matrices need to be concatenated in h dimensions. That is, the specific representation of the K'V' cache concatenation of multi-head attention is as follows: (4); Where, j h. Since large models are typically multi-layered networks, assuming an N-layer Transformer Block, it's necessary to concatenate the caches of each multi-head in N dimensions. Therefore, a single pre-filling calculation produces the final K'V' cache value: (5); After the above calculations, the K'V' Cache of inference guidance information is obtained. To prevent the encoding of the first word key-value vector (KV Cache) of the prompt word information calculated in the current inference from being disordered, the second word key-value vector of the inference guidance information (i.e., K'V' Cache, i.e., the above Cache (K', V')) is appended to the beginning of the current inference KV Cache. That is, the KV Cache is concatenated after the guidance information K'V' Cache. (6); The concatenated KV Cache (i.e., the FinalCache(K,V) above) has a sequence length increased by n inference guidance information compared to the original. In the next decoding calculation, seq_length needs to be increased by n, i.e., seq_length = seq_length + n. In subsequent token inference decoding, the input of inference guidance information is increased through a concatenation algorithm, resulting in a more comprehensive output response.

[0043] It should be noted that the process of pre-filling the prompt information to generate the first lexical key-value vector KV Cache is the same as the process of pre-filling the inference guidance information to generate the second lexical key-value vector K'V' Cache (i.e., the above formulas 1-5), and will not be repeated here.

[0044] At this point, the terminal device has completed the insertion of information from the edge cloud. This embodiment of the invention supplements the reference information for inference sampling by inserting inference guidance information from the edge cloud into the inference decoding process of the terminal device. For example, the second large model of the terminal device itself can only calculate 5 relevant information, but due to the auxiliary input of the first large model in the edge cloud, the word relationships are greatly supplemented, more sampling results are obtained, and finally 10 relevant information are output, which greatly improves the inference performance of the large model on the edge.

[0045] Compared with the prior art, the beneficial effects of the embodiments of the present invention are as follows: The terminal device starts asynchronous parallel inference threads and collaborative threads, which hardly affects or slows down the inference response speed of the large model on the edge. At the same time, the large model on the edge cloud provides auxiliary guidance to the large model on the edge. On the one hand, it does not need to offload the inference tasks that originally belonged to the terminal device to the edge cloud, avoiding the risk of data loss or inference task interruption. On the other hand, it can significantly improve the performance of the large model on the edge, especially for the processing of long texts, and can extract more accurate outline information. The large model on the device side only needs to interact with the large model on the edge and cloud only once, and it has a timeout policy, so it can be executed even in a weak network environment.

[0046] See Figure 5 , Figure 5 This is a structural block diagram of an edge-side large model inference acceleration device provided in an embodiment of the present invention. The edge-side large model inference acceleration device is applied to a terminal device and includes: The prompt word receiving module 11 is used to receive prompt word information input by the user and start the inference thread and the cooperative thread to be executed in parallel; The prompt word sending module 12 is used to send the prompt word information to the first large model in the edge cloud for inference through the collaborative thread, so as to obtain inference guidance information; The reasoning module 13 is used to reason about the prompt word information by inserting the reasoning guidance information into the encoding stage of the second major model of the terminal device through the reasoning thread, so as to obtain the response information of the prompt word information.

[0047] In an optional embodiment, the inference module 13 includes: The first reasoning unit is used to input the prompt word information into the second large model for reasoning through the reasoning thread; In the pre-filling stage of the second major model, the prompt word information is pre-filled to obtain the first word key value vector; During the decoding stage of the second major model, it is determined whether to receive the inference guidance information output by the first major model; and if it is determined that the inference guidance information is received, the inference guidance information is inserted into the first word key value vector for inference encoding to obtain the response information of the prompt word information.

[0048] In one optional embodiment, the inference guidance information is inserted into the first lexical key-value vector for inference encoding to obtain the response information of the prompt word information, including: The reasoning guidance information is pre-filled to obtain the second word key value vector of the reasoning guidance information; The first word key vector is appended to the end of the second word key vector to generate the third word key vector. The response information is obtained by inferring and encoding the third word key value vector.

[0049] In one optional embodiment, the reasoning guidance information includes: first guidance lexical information or at least one second guidance lexical information; Wherein, the first guiding meta-information includes the keyword meta-information output by the first large model after reasoning about the prompt word information; The second guiding meta-information includes the keyword meta-information output by the first large model after completing partial reasoning on the prompt word information.

[0050] In one optional embodiment, the second guiding meta-information includes keyword meta-information generated by the first large model during the encoding phase according to a preset timeout response strategy; the timeout response strategy is used to indicate that the keyword meta-information generated during the current inference time is output when the current inference time reaches a preset inference time threshold.

[0051] In one optional embodiment, the inference guidance information is pre-filled to obtain a second lexical key-value vector of the inference guidance information, including: The reasoning guidance information is segmented into words to obtain a segmented sequence; The word segmentation sequence is subjected to word embedding processing to obtain the first word embedding vector; Add positional encoding to the word embedding vector to obtain the second word embedding vector; Attention calculation and feedforward network calculation are performed on the second word embedding vector to generate the second word key value vector.

[0052] It should be noted that the working process of each module in the edge-side large model inference acceleration device described in the embodiments of the present invention can refer to the working process of the edge-side large model inference acceleration method described in the above embodiments, and the technical effect achieved is the same as that of the edge-side large model inference acceleration method described in the above embodiments, which will not be repeated here.

[0053] See Figure 6 , Figure 6 This is a structural block diagram of the edge-side large model inference acceleration device provided in an embodiment of the present invention. The edge-side large model inference acceleration device includes a processor 21, a memory 22, and a computer program stored in the memory 22 and executable on the processor 21. When the processor 21 executes the computer program, it implements the steps in the above embodiments of the edge-side large model inference acceleration methods, such as steps S11 to S13.

[0054] For example, the computer program may be divided into one or more modules or units, which are stored in the memory 22 and executed by the processor 21 to complete the present invention. The one or more modules or units may be a series of computer program instruction segments capable of performing specific functions, which describe the execution process of the computer program in the edge-side large model inference acceleration device.

[0055] The edge-side large model inference acceleration device may include, but is not limited to, processor 21 and memory 22. Those skilled in the art will understand that the schematic diagram is merely an example of an edge-side large model inference acceleration device and does not constitute a limitation on the device. It may include more or fewer components than illustrated, or combine certain components, or use different components. For example, the edge-side large model inference acceleration device may also include input / output devices, network access devices, buses, etc.

[0056] The processor 21 can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor. The processor 21 is the control center of the edge-side large model inference acceleration device, connecting various parts of the device via various interfaces and lines.

[0057] The memory 22 can be used to store the computer programs and / or modules. The processor 21 implements various functions of the edge-side large model inference acceleration device by running or executing the computer programs and / or modules stored in the memory 22 and calling the data stored in the memory 22. The memory 22 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the mobile phone (such as audio data, phonebook, etc.). In addition, the memory 22 may include high-speed random access memory, and may also include non-volatile memory, such as hard disk, memory, plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0058] Wherein, if the modules or units integrated in the edge-side large model inference acceleration device are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the above-described embodiments of the present invention can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by the processor 21, it can implement the steps of the above-described method embodiments. Wherein, the computer program includes computer program code, which can be in the form of source code, object code, executable file, or some intermediate form, etc. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal, and software distribution medium, etc.

[0059] It should be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Furthermore, in the accompanying drawings of the device embodiments provided by this invention, the connection relationships between modules indicate that they have communication connections, which can be specifically implemented as one or more communication buses or signal lines. Those skilled in the art can understand and implement this without any creative effort.

[0060] The above description represents the preferred embodiments of the present invention. It should be noted that, for those skilled in the art, various improvements and modifications can be made without departing from the principles of the present invention, and these improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A method for accelerating large-scale inference on the edge, characterized in that, Applied to terminal devices, including: Receive prompt information from the user and start inference threads and cooperating threads to execute in parallel; The prompt word information is sent to the first large model in the edge cloud for inference via the collaborative thread to obtain inference guidance information; the inference guidance information includes: first guidance word meta-information or at least one second guidance word meta-information; wherein, the first guidance word meta-information includes keyword meta-information output by the first large model after inference of the prompt word information; the second guidance word meta-information includes keyword meta-information output by the first large model after partial inference of the prompt word information; The inference thread inserts the inference guidance information into the encoding stage of the second major model of the terminal device to infer the prompt word information, thereby obtaining the response information of the prompt word information. The inference thread inserts the inference guidance information into the encoding stage of the second major model of the terminal device to infer the prompt word information, thereby obtaining the response information of the prompt word information, including: The inference thread inputs the prompt word information into the second large model for inference; In the pre-filling stage of the second major model, the prompt word information is pre-filled to obtain the first word key value vector; During the decoding phase of the second major model, it is determined whether to receive the inference guidance information output by the first major model; If the reasoning guidance information is received, the reasoning guidance information is inserted into the first word key value vector for reasoning encoding to obtain the response information of the prompt word information.

2. The edge-side large model inference acceleration method as described in claim 1, characterized in that, The reasoning guidance information is inserted into the first lexical key-value vector for reasoning encoding to obtain the response information of the prompt word information, including: The reasoning guidance information is pre-filled to obtain the second word key value vector of the reasoning guidance information; The first word key vector is appended to the end of the second word key vector to generate the third word key vector. The response information is obtained by inferring and encoding the third word key value vector.

3. The edge-side large model inference acceleration method as described in claim 1, characterized in that, The second guiding meta-information includes the keyword meta-information generated by the first large model during the encoding stage according to a preset timeout response strategy; the timeout response strategy is used to indicate that the keyword meta-information generated during the current inference time is output when the current inference time reaches a preset inference time threshold.

4. The edge-side large model inference acceleration method as described in claim 2, characterized in that, The reasoning guidance information is pre-filled to obtain a second lexical key-value vector, including: The reasoning guidance information is segmented into words to obtain a segmented sequence; The word segmentation sequence is subjected to word embedding processing to obtain the first word embedding vector; Add positional encoding to the word embedding vector to obtain the second word embedding vector; Attention calculation and feedforward network calculation are performed on the second word embedding vector to generate the second word key value vector.

5. An end-side large-model inference acceleration device, characterized in that, Applied to terminal devices, including: The prompt word receiving module is used to receive prompt word information input by the user and start inference threads and cooperating threads that are executed in parallel; The prompt word sending module is used to send the prompt word information to the first large model in the edge cloud for inference via the collaborative thread to obtain inference guidance information; the inference guidance information includes: first guidance word meta-information or at least one second guidance word meta-information; wherein, the first guidance word meta-information includes keyword meta-information output by the first large model after inference of the prompt word information; the second guidance word meta-information includes keyword meta-information output by the first large model after partial inference of the prompt word information; The reasoning module is used to infer the prompt word information by inserting the reasoning guidance information into the encoding stage of the second major model of the terminal device through the reasoning thread, so as to obtain the response information of the prompt word information. The reasoning module includes: The first reasoning unit is used to input the prompt word information into the second large model for reasoning through the reasoning thread; In the pre-filling stage of the second major model, the prompt word information is pre-filled to obtain the first word key value vector; During the decoding stage of the second major model, it is determined whether to receive the inference guidance information output by the first major model; and if it is determined that the inference guidance information is received, the inference guidance information is inserted into the first word key value vector for inference encoding to obtain the response information of the prompt word information.

6. An edge-side large model inference acceleration device, characterized in that, include: A processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor, when executing the computer program, implements the edge-side large model inference acceleration method as described in any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein, when the computer program is executed, it controls the device where the computer-readable storage medium is located to perform the edge-side large model inference acceleration method as described in any one of claims 1 to 4.

8. A computer program product, comprising a computer program or instructions, characterized in that, When the computer program or instructions are executed by the processor, they implement the edge-side large model inference acceleration method according to any one of claims 1 to 4.

Images