Large model decoding system, method, related device and computer program product

Abstract

The present application discloses a large model decoding system, method, related device, and computer program product. The system includes a terminal device and a cloud server. A small-scale approximate model is arranged on the terminal device for decoding, and then a target model with a larger scale is used on the cloud server to confirm or correct the current decoding result of the approximate model. In the case of limited terminal hardware resources, the inference effect of the local model of the terminal device can be improved. At the same time, there is no need for the cloud server to decode each token, which can reduce the service load of the cloud server.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology. More specifically, it relates to a large model decoding system, method, related devices, and computer program products. Background Art

[0002] With the development of artificial intelligence technology, large language models (LLMs, simply referred to as "large models") have achieved remarkable success in various natural language processing tasks in the general field. A large model refers to a Transformer language model with a large number of parameters, which are trained on large-scale text data. Large models have powerful capabilities in understanding natural language and solving complex tasks (through text generation).

[0003] Existing large model deployment methods generally include two implementation schemes: cloud large models and local large models. The cloud large model scheme deploys the large model on a cloud server. Cloud large models generally have a large parameter scale and can handle complex tasks and a large amount of data, but they require a large amount of computing and memory resources, and the usage cost will be very high. The local large model scheme deploys the large model locally on a terminal device. Limited by the hardware resources of the terminal device, the parameter scale of the local large model is generally small, so the inference effect is generally average. Summary of the Invention

[0004] In view of the above problems, this application is proposed to provide a large model decoding system, method, related devices, and computer program products to improve the inference effect of the local model on the terminal through the coordinated cooperation between the terminal and the cloud. The specific solutions are as follows:

[0005] In a first aspect, this application provides a large model decoding system, including: a terminal device and a cloud server. A target model is configured on the cloud server, and an approximate model is configured on the terminal device. The target model is a large model, and the approximate model is a model with a smaller parameter scale than the target model;

[0006] The terminal device is used to perform autoregressive serial decoding based on the prefix using the configured approximate model, and when the set upload condition is reached, send the current decoding result to the cloud server;

[0007] The cloud server is used to infer and determine the text unit token (token) accepted in the current decoding result using the configured target model and sample the next token information. A new decoding result is composed of the accepted token and the next token information, and the new decoding result is returned to the terminal device;

[0008] The terminal device is further configured to update the local decoding result based on the new decoding result, determine whether an end condition is met, and if so, use the updated local decoding result as the final decoding result; if not, continue to execute the step of performing autoregressive serial decoding based on the prefix using the configured approximate model.

[0009] In a possible design, in another implementation of the first aspect of the embodiments of the present application, the current decoding result includes a consecutive first number r of tokens and the first probability value of each token.

[0010] After sending the current decoding result to the cloud server, the terminal device suspends the decoding process until it receives the new decoding result returned by the cloud server, and continues the decoding process after determining that the end condition is not met.

[0011] The process by which the cloud server uses the configured target model to infer and determine the text unit tokens accepted in the current decoding result and sample the next token information includes:

[0012] The cloud server uses the configured target model to determine the tokens accepted in the current decoding result, samples each candidate result of the next token and its second probability value, and selects the candidate result with the highest second probability value as the sampled next token information.

[0013] In a possible design, in another implementation of the first aspect of the embodiments of the present application, the current decoding result obtained by the terminal device through autoregressive serial decoding based on the prefix includes each candidate result of each token and its first probability value.

[0014] After sending the current decoding result to the cloud server, the terminal device continues the decoding process until it determines that the end condition is met.

[0015] The process by which the cloud server uses the configured target model to infer and determine the text unit tokens accepted in the current decoding result and sample the next token information includes:

[0016] The cloud server uses the configured target model to determine the tokens accepted in the current decoding result, samples each candidate result of the next token and its second probability value, and uses the candidate results and their second probability values as the sampled next token information.

[0017] The process by which the terminal device updates the local decoding result based on the new decoding result includes:

[0018] The terminal device updates the token at the same sequence position in the local decoding result by using the accepted token in the new decoding result, and determines whether there is an overlap between each candidate result in the next token information in the new decoding result and each candidate result of the token at the same sequence position in the local decoding result. If there is an overlap, the first probability value of the overlapping candidate result locally is updated to the second probability value, and the sorting of each candidate result is refreshed.

[0019] In a possible design, in another implementation manner of the first aspect of the embodiments of the present application, the process in which the terminal device updates the local decoding result based on the new decoding result further includes:

[0020] If it is determined that there is no overlap between each candidate result in the next token information in the new decoding result and each candidate result of the token at the same sequence position in the local decoding result, the approximate model is controlled to roll back to the position of the next token in the new decoding result and continue serial decoding.

[0021] In a possible design, in another implementation manner of the first aspect of the embodiments of the present application, the current decoding result includes r tokens and the first probability value of each token;

[0022] The process in which the cloud server uses the configured target model to infer and determine the accepted token in the current decoding result and sample the next token information includes:

[0023] The cloud server concatenates the r tokens in the current decoding result with the local prefix and inputs them into the target model for forward propagation to obtain the second probability value of each token. Based on the first probability value and the second probability value of each token, the accepted token is selected from the current decoding result, and the next token information is sampled.

[0024] In a possible design, in another implementation manner of the first aspect of the embodiments of the present application, the current decoding result includes the candidate results of each of the r tokens and the first probability value of each candidate result;

[0025] The process in which the cloud server uses the configured target model to infer and determine the accepted token in the current decoding result and sample the next token information includes:

[0026] The cloud server invokes the target model, and based on the prefix of the local and the first probability values of each candidate result of each token in the current decoding result, determines the final result that can be accepted as a token from each candidate result of each token. If none of the candidate results of a token can be accepted, the information of that token is resampled.

[0027] In a possible design, in another implementation manner of the first aspect of the embodiments of the present application, the process by which the terminal device determines whether the end condition is satisfied includes:

[0028] The terminal device determines whether the length of the token in the new decoding result reaches a threshold or whether it contains a set termination symbol. If the judgment result of any one of the conditions is yes, it is determined that the end condition is satisfied; otherwise, it is determined that the end condition is not satisfied.

[0029] In a possible design, in another implementation manner of the first aspect of the embodiments of the present application, the cloud server is further configured to:

[0030] After obtaining the new decoding result, determine whether the end condition is satisfied. If not, continuously detect whether the decoding result sent by the terminal device is received. If so, end the current decoding process.

[0031] In a second aspect, a large model decoding method is provided, which is applied to a terminal device in a decoding system. The decoding system further includes a cloud server, and a target model is configured on the cloud server, and an approximate model is configured on the terminal device. The target model is a large model, and the approximate model is a small model with a smaller parameter scale than the target model. The method includes:

[0032] Using the configured approximate model, perform autoregressive serial decoding based on the prefix, and when the set upload condition is reached, send the current decoding result to the cloud server, so that the cloud server uses the target model to infer and determine the text unit token accepted in the current decoding result and sample the information of the next token, and a new decoding result is composed of the accepted token and the next token information;

[0033] Obtain the new decoding result returned by the cloud server, and update the local decoding result based on the new decoding result;

[0034] Determine whether the end condition is satisfied. If so, use the updated local decoding result as the final decoding result. If not, continue to execute the step of performing autoregressive serial decoding based on the prefix using the configured approximate model.

[0035] In a possible design, in another implementation manner of the second aspect of the embodiments of the present application, the current decoding result includes r consecutive tokens of the first quantity and the first probability value of each token;

[0036] After the terminal device sends the current decoding result to the cloud server, it pauses the decoding process until it receives the new decoding result returned by the cloud server, and continues to execute the decoding process after determining that the end condition is not met.

[0037] In a possible design, in another implementation manner of the second aspect of the embodiments of the present application, the current decoding result includes each candidate result of each token and its first probability value; the new decoding result includes the tokens accepted in the current decoding result determined by the cloud server, and each candidate result of the sampled next token and its second probability value;

[0038] After the terminal device sends the current decoding result to the cloud server, it continuously performs the decoding process until it determines that the end condition is met;

[0039] The process of updating the local decoding result based on the new decoding result includes:

[0040] Using the tokens accepted in the new decoding result to update the tokens at the same sequence position in the local decoding result, and determining whether there is an overlap between each candidate result of the next token in the new decoding result and each candidate result of the token at the same sequence position in the local decoding result;

[0041] If there is an overlap, update the first probability value of the local overlapping candidate result to the second probability value, and refresh the sorting of each candidate result;

[0042] If there is no overlap, control the approximate model to roll back to the position of the next token in the new decoding result and continue serial decoding.

[0043] In a third aspect, a large model decoding method is provided, which is applied to a cloud server in a decoding system. The decoding system further includes a terminal device. A target model is configured on the cloud server, and an approximate model is configured on the terminal device. The target model is a large model, and the approximate model is a small model with a smaller parameter scale than the target model; the method includes:

[0044] Obtain the current decoding result sent by the terminal device, where the current decoding result is obtained by the terminal device performing autoregressive serial decoding using the approximate model;

[0045] Using the target model, infer and determine the text unit token accepted in the current decoding result and sample the next token information, and form a new decoding result from the accepted token and the next token information;

[0046] Return the new decoding result to the terminal device so that the terminal device updates the local decoding result based on the new decoding result.

[0047] In a possible design, in another implementation manner of the third aspect of the embodiments of the present application, the process of using the configured target model to infer and determine the text unit token accepted in the current decoding result and sample the next token information includes:

[0048] Using the configured target model, determine the token accepted in the current decoding result, sample each candidate result of the next token and its second probability value, and select the candidate result with the highest second probability value as the sampled next token information.

[0049] In a possible design, in another implementation manner of the third aspect of the embodiments of the present application, the process of using the configured target model to infer and determine the text unit token accepted in the current decoding result and sample the next token information includes:

[0050] Using the configured target model, determine the token accepted in the current decoding result, sample each candidate result of the next token and its second probability value, and use the candidate results and their second probability values as the sampled next token information.

[0051] In a possible design, in another implementation manner of the third aspect of the embodiments of the present application, the current decoding result includes r tokens and the first probability value of each token;

[0052] Using the configured target model, the process of inferring and determining the token accepted in the current decoding result and sampling the next token information includes:

[0053] Concatenate the r tokens in the current decoding result with the local prefix and input them into the target model for forward propagation to obtain the second probability value of each token. Based on the first probability value and the second probability value of each token, select the accepted token from the current decoding result and sample the next token information.

[0054] In a possible design, in another implementation manner of the third aspect of the embodiments of the present application, the current decoding result includes candidate results of r tokens respectively and a first probability value of each candidate result;

[0055] The process of using the configured target model to infer and determine the tokens accepted in the current decoding result and sample the information of the next token includes:

[0056] Call the target model, and based on the local prefix and the first probability values of the candidate results of each token in the current decoding result, determine the final results that can be accepted as tokens from the candidate results of each token. If none of the candidate results of a token can be accepted, resample the information of this token.

[0057] Fourth aspect, there is provided an electronic device, including: a memory and a processor;

[0058] The memory is used to store a program;

[0059] The processor is used to execute the program to implement each step of the large model decoding method described in any one of the foregoing second aspect or third aspect of the present application.

[0060] Fifth aspect, there is provided a readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, each step of the large model decoding method described in any one of the foregoing second aspect or third aspect of the present application is implemented.

[0061] Sixth aspect, there is provided a computer program product, including a computer program. When the computer program is executed by a processor, each step of the large model decoding method described in any one of the foregoing second aspect or third aspect of the present application is implemented.

[0062] By means of the above technical solutions, the present application provides a decoding system in which a terminal device and a cloud server cooperate. The terminal device uses a small-scale approximate model for decoding, and then the cloud server uses a larger-scale target model to confirm or correct the current decoding result of the approximate model. In the case where the terminal hardware resources are limited, the inference effect of the local model of the terminal device can be improved. At the same time, there is no need for the cloud server to decode each token one by one, which can reduce the service load of the cloud server.

[0063] Specifically, deploy the target model on the cloud server and deploy an approximate model with a smaller parameter scale locally on the terminal device. The terminal device uses the approximate model for autoregressive serial decoding and sends the current decoding result to the cloud server. Since the approximate model has a relatively smaller parameter scale, the hardware resource requirements for the terminal device are lower, and moreover, the memory access requirements can be reduced. Through the approximate model, consecutive tokens can be quickly decoded and generated. The cloud server adopts the target model, which can infer and determine the tokens accepted in the current decoding result and sample the information of the next token to form a new decoding result. Since the target model does not need to decode each token one by one, but only evaluates the current decoding result generated by the approximate model, and resamples the unacceptable tokens once again, the inference overhead of the target model is greatly reduced, and the service load of the cloud server is reduced.

[0064] The cloud server returns the new decoding result to the terminal device for the terminal device to update the local decoding result and continue autoregressive serial decoding when it is determined that the end condition is not met. By using the target model to evaluate the decoding result of the approximate model, the obtained new decoding result is more accurate. Then, by using this new decoding result to update the local decoding result, the inference effect of the local model of the terminal device can be improved. Description of the Drawings

[0065] By reading the following detailed description of the preferred embodiments, various other advantages and benefits will become clear to those of ordinary skill in the art. The drawings are only for the purpose of illustrating the preferred embodiments and are not considered to be a limitation of the present application. Moreover, throughout the drawings, the same reference numerals are used to represent the same components. In the drawings:

[0066] Figure 1 It is a schematic diagram of the architecture of the large model decoding system provided by the embodiment of the present application;

[0067] Figure 2 It is a schematic diagram of the structure of a terminal device provided by the embodiment of the present application;

[0068] Figure 3 It is a schematic diagram of the structure of a cloud server provided by the embodiment of the present application;

[0069] Figure 4 It is a signaling interaction diagram of the large model decoding method for end-cloud collaboration provided by the embodiment of the present application;

[0070] Figure 5 It is a schematic diagram of the process flow of the large model decoding method on the terminal device side provided by the embodiment of the present application;

[0071] Figure 6This application provides a schematic flow diagram of a large model decoding method on the cloud server side;

[0072] Figure 7 This is a schematic structural diagram of an electronic device provided by an embodiment of this application. Detailed implementation manners

[0073] Next, the technical solutions in the embodiments of this application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of this application. Obviously, the described embodiments are only a part of the embodiments of this application, rather than all the embodiments. Based on the embodiments in this application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of this application.

[0074] Before introducing the solution of this application, first, relevant concepts involved in this article are explained:

[0075] prompt: An instruction. When interacting with an AI (such as an artificial intelligence model), the instruction that needs to be sent to the AI can be a text description, such as "Please recommend a popular music for me" entered when you interact with the AI, or a parameter description in a certain format. For example, when asking the AI to draw a picture in a certain format, relevant drawing parameters need to be described.

[0076] Large language model: (Large language model, LLM) can also be simply referred to as a large model. Generally, it refers to a language model with a large number of parameters and capabilities. It learns the statistical laws and semantic relationships of language by pre-training on a large-scale text data. These models usually use unsupervised learning methods to predict the next word or fill in the missing word to capture the context and semantic information of the language. Large language models can generate coherent sentences, answer questions, complete translation tasks, etc. The characteristic of LLM is its huge scale, including billions or even more parameters, which helps them learn complex patterns in language data. The emerging capabilities of large language models include in-context learning, instruction following, and step-by-step reasoning capabilities, etc. With the release of ChatGPT, research and applications related to LLM have gradually exploded, such as Google's PaLM model, Meta's LLaMA model, etc.

[0077] Furthermore, this application provides a large model decoding method for end-cloud collaboration, which can be applied to a Figure 1 system architecture as shown in the figure. This system can include a terminal device 100 and a cloud server 200. The server 200 can include one or more servers ( Figure 1 taking including one server as an example for illustration).

[0078] The terminal device 100 and the cloud server 200 can cooperate to execute the large model decoding method provided in the embodiments of the present application.

[0079] Next, the product form of the terminal device 100 will be described. Figure 1 in the terminal device 100;

[0080] The terminal device 100 in the embodiments of the present application can be a mobile phone, a tablet computer, a teaching large screen, a wearable device, a vehicle-mounted device, a conference terminal, an augmented reality (AR) / virtual reality (VR) device, a notebook computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), etc. The embodiments of the present application do not make any restrictions on this.

[0081] Figure 2 FIG. shows an optional schematic diagram of the hardware structure of the terminal device 100.

[0082] Referring to Figure 2 as shown, the terminal device 100 may include a radio frequency unit 110, a memory 120, an input unit 130, a display unit 140, a camera 150 (optional), an audio circuit 160, a speaker 161, a microphone 162, a headphone jack 163 (optional), a processor 170, an external interface 180, a power supply 190, and other components. Those skilled in the art can understand that Figure 2 merely examples of a terminal or a multifunctional device, and do not constitute a limitation on the terminal or the multifunctional device, and may include more or fewer components than shown in the figure, or combine certain components, or different components.

[0083] The input unit 130 can be used to receive input digital or character information and generate key signal inputs related to the user settings and function controls of the portable multifunctional device. Specifically, the input unit 130 can include a touch screen 131 and / or other input devices 132. The touch screen 131 can collect touch operations of the user thereon or nearby (such as operations of the user using any suitable object such as a finger, joint, stylus, etc. on or near the touch screen), and drive corresponding connection devices according to a preset program. The touch screen can detect the touch actions of the user on the touch screen, convert the touch actions into touch signals and send them to the processor 170, and can receive and execute the commands sent by the processor 170; the touch signals at least include contact coordinate information. The touch screen 131 can provide an input interface and an output interface between the terminal 100 and the user. In addition, various types such as resistive, capacitive, infrared, and surface acoustic wave can be used to implement the touch screen. In addition to the touch screen 131, the input unit 130 can also include other input devices. Specifically, the other input devices 132 can include, but are not limited to, one or more of a physical keyboard, function keys (such as volume control keys, power on / off keys, etc.), trackballs, mice, joysticks, etc.

[0084] Among them, the input device 132 can receive input data, etc.

[0085] The display unit 140 can be used to display information input by the user or information provided to the user, various menus of the terminal 100, interactive interfaces, file displays, and / or the playback of any multimedia file. In the embodiments of the present application, the display unit 140 can be used to display various interactive interfaces, processing results, etc. in the large model decoding method.

[0086] The memory 120 can be used to store instructions and data. The memory 120 mainly includes a storage instruction area and a storage data area. The storage data area can store various data, such as multimedia files, texts, etc.; the storage instruction area can store software units such as an operating system, applications, instructions required for at least one function, etc., or their subsets and extended sets. It can also include non-volatile random access memory; it provides the processor 170 with management of the hardware, software, and data resources in the computing processing device, supports control software and applications. It is also used for the storage of multimedia files and the storage of running programs and applications.

[0087] The processor 170 is the control center of the terminal 100, connecting various parts of the entire terminal 100 through various interfaces and lines. By running or executing instructions stored in the memory 120 and invoking data stored in the memory 120, it performs various functions of the terminal 100 and processes data, thereby exercising overall control over the terminal device. Optionally, the processor 170 may include one or more processing units; preferably, the processor 170 may integrate an application processor and a modem processor. Among them, the application processor mainly processes the operating system, user interface, application programs, etc., and the modem processor mainly processes wireless communication. It can be understood that the above-mentioned modem processor may not be integrated into the processor 170. In some embodiments, the processor and the memory can be implemented on a single chip. In some embodiments, they can also be separately implemented on independent chips. The processor 170 can also be used to generate corresponding operation control signals, send them to corresponding components of the computing and processing device, read and process data in the software, especially read and process the data and programs in the memory 120, so that each functional module therein executes corresponding functions, thereby controlling the corresponding components to act according to the requirements of the instructions.

[0088] Among them, the memory 120 can be used to store software codes related to the large model decoding method. The processor 170 can execute the steps of the large model decoding method or schedule other units (such as the above-mentioned input unit 130 and display unit 140) to implement corresponding functions.

[0089] The radio frequency unit 110 (optional) can be used for receiving and transmitting information or signals during a call. For example, after receiving the downlink information from the base station, it is sent to the processor 170 for processing. Additionally, the data designed for uplink is sent to the base station. Generally, the RF circuit includes but is not limited to an antenna, at least one amplifier, a transceiver, a coupler, a low noise amplifier (LNA), a duplexer, etc. In addition, the radio frequency unit 110 can also communicate with network devices and other devices through wireless communication. This wireless communication can use any communication standard or protocol, including but not limited to Global System of Mobile communication (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Long Term Evolution (LTE), email, Short Messaging Service (SMS), etc.

[0090] Among them, in the embodiment of this application, the radio frequency unit 110 can send data to the cloud server 200 and receive the processing result sent by the cloud server 200. Exemplarily, the radio frequency unit 110 sends the decoding result of the approximate model to the cloud server 200, and the cloud server 200 calls the target model to evaluate the decoding result and generates a new decoding result to be returned to the terminal device 100.

[0091] It should be understood that the radio frequency unit 110 is optional and can be replaced by other communication interfaces, such as a network interface.

[0092] The terminal device 100 also includes a power supply 19,0 (such as a battery) for powering each component. Preferably, the power supply can be logically connected to the processor 170 through a power management system, so as to realize functions such as management of charging, discharging, and power consumption management through the power management system.

[0093] The terminal device 100 also includes an external interface 180. This external interface can be a standard Micro USB interface or a multi-pin connector, and can be used to connect the terminal 100 to other devices for communication, or can be used to connect a charger to charge the terminal 100.

[0094] It should be noted that in the translation of "The terminal device 100 also includes a power supply 19,0 (such as a battery) for powering each component. Preferably, the power supply can be logically connected to the processor 170 through a power management system, so as to realize functions such as management of charging, discharging, and power consumption management through the power management system. ", there may be an error in the original text where "power supply 19,0" should probably be "power supply 190". The translation is based on the provided text.Although not shown, the terminal device 100 may further include a flash, a wireless fidelity (WiFi) module, a Bluetooth module, sensors with different functions, etc., which will not be elaborated herein. Some or all of the methods described below can be applied to the terminal device 100 as shown in Figure 2 shown.

[0095] Next, the product form of the cloud server 200 will be described. Figure 1 in the cloud server 200;

[0096] Figure 3 A schematic structural diagram of a cloud server 200 is provided. As shown in Figure 3 shown, the cloud server 200 includes a bus 201, a processor 202, a communication interface 203, and a memory 204. The processor 202, the memory 204, and the communication interface 203 communicate with each other through the bus 201.

[0097] The bus 201 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For the sake of simplicity of representation, Figure 3 only a thick line is used to represent it in the figure, but it does not mean that there is only one bus or one type of bus.

[0098] The processor 202 may be any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP), etc.

[0099] The memory 204 may include a volatile memory, such as a random access memory (RAM). The memory 204 may also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD), or a solid state drive (SSD).

[0100] Among them, the memory 204 can be used to store software codes related to the large model decoding method, and the processor 202 can execute the steps of the large model decoding method or schedule other units to implement corresponding functions.

[0101] It should be understood that the above terminal device 100 and cloud server 200 can be centralized or distributed devices, and the processors in the above terminal device 100 and cloud server 200 (such as processor 170 and processor 202) can be hardware circuits (such as application specific integrated circuit (ASIC), field-programmable gate array (FPGA), general processor, digital signal processor (DSP), microprocessor or microcontroller, etc.), or a combination of these hardware circuits. For example, the processor can be a hardware system with the function of executing instructions, such as CPU, DSP, etc., or a hardware system without the function of executing instructions, such as ASIC, FPGA, etc., or a combination of the above hardware system without the function of executing instructions and the hardware system with the function of executing instructions.

[0102] This application provides a cloud-edge collaborative large model decoding solution, which can be applied to various task scenarios that rely on large models for decoding. The following will introduce several applicable scenarios of the cloud-edge collaborative large model decoding solution of this application respectively.

[0103] Human-computer dialogue scenario: The user chats with the AI robot software deployed on the terminal device. For the current conversation content input by the user, it can be decoded through the cloud-edge collaborative large model decoding solution of this application to obtain the decoded reply content, which is output in multiple modalities such as text, voice, and video.

[0104] Intelligent education scenario: The user can study on a learning machine, and the learning machine can answer questions raised by the user based on large model technology, such as querying knowledge points, modifying wrong questions, and writing outlines for compositions. At this time, the cloud-edge collaborative large model decoding solution of this application can be used to decode the user's question to obtain the decoded reply content, which is output through the learning machine.

[0105] In addition, the cloud-edge collaborative large model decoding solution of this application can also be applied to many other scenarios, which will not be elaborated one by one in this case.

[0106] The inference of large models usually adopts the autoregressive decoding method. It is necessary to perform serial decoding token by token. When generating each token, all parameters need to be transferred from the storage unit to the computing unit, and their inference process is quite slow, and the process is often restricted by the memory access speed. Therefore, the memory access bandwidth becomes the bottleneck of large model inference.

[0107] The following is an example of a processing flow of autoregressive decoding:

[0108] 1. The large model uses the prefix as the input. After processing and normalizing the output result into a probability distribution, it generates the distribution of the next token.

[0109] 2. Concatenate the generated token with the prefix to form a new prefix, and repeat step 1 until EOS (set end symbol) is generated or the maximum number of tokens is reached.

[0110] In autoregressive decoding of large models, the generated tokens come out one by one. Because only the part with a sequence length of 1 is effectively calculated each time, but all parameters corresponding to the entire prefix need to be accessed, the computational memory access ratio is very low.

[0111] Speculative Sampling, also known as Speculative Decoding, is a method for accelerating large model inference. It can improve the inference speed of large models without sacrificing the generation effect.

[0112] Speculative Sampling is a decoding method that can significantly optimize the computational memory access ratio. It generally uses two models: one is the original target model, also called the large model; the other is a much smaller approximate model, also called the small model. The small model is used for autoregressive serial decoding, while the large model is used to evaluate the decoding results of the small model. During the decoding process, the decoding of some tokens is relatively easy, while the decoding of some tokens is very difficult. Therefore, simple token generation can be handed over to the small model, while difficult tokens are handed over to the large model. The small model here can adopt the same structure as the original model but with fewer parameters. The small model not only has less computational volume, but more importantly, it reduces the memory access requirements.

[0113] The following is an example of an optional processing flow of speculative sampling:

[0114] 1. Use the small model Mq to perform autoregressive decoding to continuously generate r tokens.

[0115] 2. Concatenate the generated r tokens with the prefix and send them into the large model Mp to perform a forward pass.

[0116] 3. Compare the logits results of the large and small models to evaluate whether to accept each token as the final result output. If it is found that a certain token cannot be accepted, re-decode this token. Repeat step 1.

[0117] 4. If the generation results of the small model are all acceptable, sample the next token using the large model. Repeat step 1.

[0118] In the above step 2, the r tokens and the prefix are combined as the input of the large model. Compared with autoregressive decoding, although the computational amount is the same, the r tokens can participate in the calculation simultaneously, and the computational memory ratio is significantly improved.

[0119] In the above step 3, the process of evaluating whether a token is acceptable can adopt the following judgment method:

[0120] If q(x) > p(x) (p and q represent the sampling probabilities of the large and small models respectively, that is, the probability distribution after logits normalization), then reject the generation of this token with a probability of 1 - p(x) / q(x), and resample a token from a new probability distribution p’(x) = norm(max(0, p(x) - q(x))).

[0121] Traditional large model deployment solutions generally adopt cloud deployment or terminal local deployment.

[0122] For the cloud large model solution, whether using autoregressive decoding or speculative decoding, the involved models are deployed on the cloud server, and the decoding process requires a large amount of computing and memory resources, resulting in a high service load on the cloud server.

[0123] For the terminal local deployment solution, limited by the terminal hardware resources, the parameter scale of the deployed model is small, so the inference effect is generally average.

[0124] Obviously, both the existing pure cloud large model solution and the pure terminal local large model solution have defects. In view of this, the present application provides a cloud-edge collaborative large model decoding solution, which can deploy the target model on the cloud server and deploy a smaller approximate model locally on the terminal. The approximate model uses autoregressive decoding to generate the decoding result and transmits it to the cloud server, and the target model is used to confirm or correct the decoding result, which can improve the inference effect of the local model of the terminal device while reducing the service load of the cloud server.

[0125] Next, in combination with Figure , the large model decoding system provided by the embodiments of the present application will be introduced: The system may include: a terminal device

[0126] The target model is configured on the cloud server 200, and the approximate model is configured on the terminal device 100. The target model is a large model, and the approximate model is a model with a smaller parameter scale than the target model. The approximate model can be a model with the same structure as the target model but fewer parameters, or it can be other types of models with different structures from the target model.

[0127] The process of decoding using the large model decoding system of this embodiment can be implemented according to the following process:

[0128] Step S10: The terminal device 100 uses the configured approximate model to perform autoregressive serial decoding based on the prefix to obtain the current decoding result.

[0129] Among them, the decoding result can include the generated continuous first number r of tokens and the first probability value of each token. Among them, r can be an integer greater than 1. Exemplarily, r takes the value of 5.

[0130] It should be noted here that when decoding with the approximate model, for each token, 1 or more candidate results can be obtained, as well as the first probability value corresponding to each candidate result. In this step, for each token, the candidate result with the highest first probability value can be selected from its candidate results as the token, and the decoding result is composed of each token and its first probability value.

[0131] In addition, in this step, the candidate results of each token and the corresponding first probability value can also be composed into the decoding result, that is, the decoding result includes the candidate results of each of the r tokens and the first probability value of each candidate result.

[0132] The trigger condition for the terminal device 100 to perform decoding using the approximate model can be based on the prompt instruction currently input by the user. For example, during the human-computer interaction process, when receiving the problem information currently input by the user, the current round of decoding process is triggered. Of course, in addition, it can also be based on the instructions of other functional modules to trigger the current round of decoding process.

[0133] As used in this embodiment, the "current round" refers to starting from the moment when the terminal device 100 triggers the decoding using the approximate model, through the collaborative decoding of the terminal device 100 and the cloud server 200 until the end condition is reached, to obtain the complete decoding result of the current round.

[0134] The prefix at the initial decoding moment (that is, the moment when the terminal device 100 starts decoding using the approximate model when the trigger condition is met) can be the input prompt instruction. When step S10 is iteratively executed subsequently, the local decoding result can be updated based on the new decoding result returned by the cloud server, and the updated local decoding result is used as the new prefix.

[0135] Step S11: The terminal device 100 sends the current decoding result to the cloud server 200.

[0136] Specifically, the terminal device 100 may send the current decoding result to the cloud server 200 when the set upload condition is met. Among them, there can be various set upload conditions. For example, the number of newly decoded tokens reaches the set quantity threshold r, or the set upload period is reached, etc. The current decoding result can be the newly decoded result between the last upload time and the current time.

[0137] The terminal device 100 can transmit the decoding result to the cloud server 200 in a wired or wireless manner.

[0138] It can be understood that since the approximate model deployed on the terminal device 100 is a model with a relatively small parameter scale, compared with the target model deployed in the cloud, its performance is relatively low. It can correctly decode relatively simple tokens, but may decode complex tokens incorrectly. That is, the tokens in the decoding result obtained through the approximate model may be completely correct or may contain incorrect tokens. To ensure the decoding effect of the terminal local model, the terminal device 100 can send the decoding result to the cloud server 200 for the cloud server 200 to confirm or correct the decoding result through the target model.

[0139] Step S12: The cloud server 200 uses the target model to evaluate and process the current decoding result to obtain a new decoding result.

[0140] Specifically, the cloud server 200 can use the target model to infer and determine the accepted tokens in the current decoding result and sample the next token information. The accepted tokens and the next token information form a new decoding result.

[0141] Specifically, when the cloud server 200 infers and determines that all tokens in the current decoding result are completely accepted, it can sample the next token information; when it cannot accept a certain token, it resamples the token to obtain the resampled next token information), and the accepted tokens and the next token information form a new decoding result.

[0142] Among them, the next token information may include each candidate result of the sampled next token and its second probability value. Or, the next token information may also only include the candidate result with the highest probability value among each candidate result of the sampled next token and its second probability value.

[0143] When the cloud server 200 determines whether it can receive each token in the current decoding result, it can be implemented according to the traditional speculative sampling algorithm, or other improved algorithms can be used for implementation, which will be elaborated later.

[0144] It should be explained that in this embodiment, the cloud server 200 uses the target model to evaluate the uploaded current decoding result, resamples unacceptable tokens once, or samples the next token when all tokens are fully accepted, without the need to decode each token one by one as in the existing large model decoding scheme. In this embodiment, the evaluation calculation object of the target model is multiple consecutive tokens, and the parameters corresponding to all prefix positions can be called at one time to complete the evaluation of multiple tokens, with a lower calculation memory access ratio and reduced inference overhead of the cloud server. At the same time, the new decoding result obtained through the cloud server 200 has higher accuracy compared to the decoding result of the local approximate model of the terminal device 100, that is, the accuracy of the decoding result is ensured.

[0145] Step S13: The cloud server 200 sends the new decoding result to the terminal device 100.

[0146] Step S14: The terminal device 100 updates the local decoding result based on the new decoding result.

[0147] Step S15: The terminal device 100 determines whether the end condition is met. For example, it is determined whether the length of the token in the new decoding result reaches a set threshold, or whether a set termination symbol is included. If the judgment result of any condition is yes, it can be determined that the end condition is met; otherwise, it is determined that the end condition is not met.

[0148] When it is determined in step S15 that the end condition is not met, it means that the current round of decoding has not ended and tokens need to be continuously decoded and generated. At this time, the locally updated decoding result can be used as the new prefix, and then return to execute step S10.

[0149] When it is determined in step S15 that the end condition is met, step S16 can be executed.

[0150] Step S16: The terminal device 100 uses the updated local decoding result as the final decoding result.

[0151] Specifically, when it is determined that the end condition is met, it indicates that the current round of decoding has ended, and the updated local decoding result can be used as the final decoding result. For the final decoding result, it can be output and displayed directly or after being processed. For example, during a human-machine conversation, the final decoding result can be output and displayed in various modal forms such as text and voice. In addition, in some other application scenarios, the final decoding result can also be passed to other functional modules for downstream task processing, which will not be elaborated here.

[0152] In the decoding system with end-cloud collaboration provided by the embodiments of the present application, a small-scale approximate model is used for decoding on the terminal device, and then a larger-scale target model is adopted on the cloud server to confirm or correct the current decoding result of the approximate model. In the case where the terminal hardware resources are limited, the inference effect of the local model of the terminal device can be improved. At the same time, there is no need for the cloud server to decode each token one by one, which can reduce the service load of the cloud server.

[0153] Specifically, the target model is deployed on the cloud server, and an approximate model with a smaller parameter scale is deployed locally on the terminal device. The terminal device uses the approximate model for autoregressive serial decoding and sends the current decoding result to the cloud server. Since the parameter scale of the approximate model is relatively smaller, the requirements for the hardware resources of the terminal device are lower, and moreover, the need for memory access can be reduced. Through the approximate model, consecutive tokens can be quickly decoded and generated. The cloud server adopts the target model to be able to infer and determine the tokens accepted in the current decoding result and sample the information of the next token to form a new decoding result. Since the target model does not need to decode each token one by one, but only evaluates the current decoding result generated by the approximate model, and only resamples the unacceptable tokens once, the inference overhead of the target model is greatly reduced, and the service load of the cloud server is reduced.

[0154] The cloud server returns the new decoding result to the terminal device for the terminal device to update the local decoding result and continue autoregressive serial decoding when it is determined that the end condition is not met. By using the target model to evaluate the decoding result of the approximate model, the obtained new decoding result is more accurate. Furthermore, using this new decoding result to update the local decoding result can improve the inference effect of the local model of the terminal device.

[0155] Further optionally, after obtaining the new decoding result, the cloud server 200 can also add a step of determining whether the end condition is met. This determination process is similar to the process of the terminal device 100 determining whether the end condition is met, which will not be elaborated here.

[0156] When the cloud server 200 determines that the end condition is not met, it can continuously detect whether it has received the decoding result sent by the terminal device. When it determines that the end condition is met, it can end the current decoding process.

[0157] In some embodiments of the present application, a process of a terminal device 100 and a cloud server 200 cooperating to perform collaborative decoding using the speculative sampling algorithm is introduced.

[0158] Based on the above embodiments, for the process of the terminal device 10 in step S10 using an approximate model to perform autoregressive serial decoding based on the prefix, it can be to pause the decoding process when the set upload condition is reached during decoding. For example, when the number of tokens obtained by decoding reaches the set threshold r, the decoding process is paused, and the consecutive r tokens obtained by decoding and the first probability value of each token are used as the current decoding result and uploaded to the cloud server 200. After the terminal device 10 pauses decoding, it waits for the cloud server 200 to return a new decoding result until it receives the new decoding result, and only continues to execute the decoding process after determining that the end condition is not met.

[0159] Further, for the process of the cloud server 200 using the configured target model to infer and determine the tokens accepted in the current decoding result and sample the next token information, it can include:

[0160] Using the configured target model, determine the tokens accepted in the current decoding result according to the speculative sampling algorithm, sample the candidate results of the next token and their second probability values, and select the candidate result with the highest second probability value as the sampled next token information.

[0161] Specifically, the cloud server 200 concatenates the r tokens in the current decoding result with the local prefix and inputs them into the target model for forward propagation to obtain the second probability value of each token. Based on the first probability value and the second probability value of each token, select the accepted tokens from the current decoding result and sample the next token to form a new decoding result, and update the new decoding result as the local prefix (when the cloud server first receives the current decoding result sent by the terminal device 100 in this round, the local prefix can be a null value, that is, directly input the r tokens in the current decoding result into the target model for forward propagation).

[0162] After receiving the new decoding result returned by the cloud server 200, the terminal device 100 can use the new decoding result to update the local decoding result and determine whether the termination conditions are met, such as whether the length of the new decoding result reaches a set length threshold or whether the new decoding result contains a set end character. If any of these conditions are met, the current decoding round is considered to be completed and the latest local decoding result can be output as the final decoding result. Otherwise, the current decoding round is considered to be not yet completed, and the steps of performing autoregressive serial decoding based on the prefix can be continued. It should be noted that when the decoding steps are continued, the prefix can use the locally updated decoding result, that is, the autoregressive serial decoding process continues based on the latest local decoding result.

[0163] The solution provided in this embodiment applies the speculative sampling algorithm to the collaborative decoding system of the terminal device and the cloud server, which can not only improve the inference effect of the local model of the terminal device, but also avoid the problems of high cloud service load and low computational memory access ratio that exist in simply using a large cloud model for autoregressive serial decoding.

[0164] In other embodiments of the present application, considering that there must be communication between the terminal device 100 and the cloud server 200 during the collaborative decoding process, the communication process may introduce network delays. If the aforementioned speculative sampling algorithm is directly adopted, since the terminal device 100 suspends decoding after uploading the current decoding result and waits for the cloud server 200 to return a new decoding result, this process will inevitably cause output delays of the terminal device 100, especially in some scenarios with strong immediacy, such as human-computer interactive dialogue scenarios. The terminal device cannot output and display the decoding result while waiting for the cloud server to return the new decoding result, which brings a user experience with excessive delay.

[0165] In order to solve this problem, an improved solution is provided in this embodiment:

[0166] In this embodiment, terminal device 100 uses the configured approximate model to perform prefix-based autoregressive serial decoding. The resulting decoding result includes candidate results for each token and their first probability values. When a preset upload condition is met (e.g., the number of tokens included in the current decoding result reaches a preset threshold r, or other optional conditions), the current decoding result is uploaded to cloud server 200.

[0167] It should be particularly noted that after the terminal device 100 sends the current decoding result to the cloud server 200, it continues the decoding process until it determines that the end condition is met. That is, it will not pause and wait after uploading the current decoding result. At this time, the terminal device 100 can output in a timely manner based on the continuously decoded results locally, reducing the output delay in the human-computer interaction process and improving the user experience. Of course, the decoded result output at this time is not necessarily the final decoded result. Subsequently, it is necessary to wait for the evaluation result returned by the cloud server 200 and update the local decoded result based on the evaluation result, so as to refresh the output display.

[0168] Furthermore, the cloud server 200 uses the configured target model to infer and determine the tokens accepted in the current decoding result and sample the next token information. This process may include:

[0169] The cloud server 200 uses the configured target model to determine the tokens accepted in the current decoding result, samples the candidate results of the next token and their second probability values, and uses the candidate results and their second probability values as the sampled next token information. The new decoding result composed of the accepted tokens and the sampled next token information is sent to the terminal device 100.

[0170] Specifically, the current decoding result includes the candidate results of r tokens and the first probability value of each candidate result. On this basis, the process by which the cloud server 200 uses the target model to infer and determine the tokens accepted in the current decoding result and sample the next token information may include:

[0171] The cloud server 200 calls the target model, and based on the local prefix and the first probability values of the candidate results of each token in the current decoding result, determines the final results that can be accepted as tokens from the candidate results of each token. If the accepted candidate results can be selected for each token, the candidate results of the next token and their second probability values can be sampled. If none of the candidate results of a certain token can be accepted, the information of this token is resampled.

[0172] For example:

[0173] The current decoding result includes the candidate results of two tokens, t1 and t2. The candidate results of t1 include s11, s12, and s13, and the candidate results of t2 include s21, s22, and s23.

[0174] After reasoning by the target model, if it is determined that s11 among the candidate results of t1 can be accepted as the final result of t1, then s11 can be taken as t1; if it is determined that s23 among the candidate results of t2 can be accepted as the final result of t2, then s23 can be taken as t2. At this time, the information of the next token (t3) can be sampled, that is, the candidate results of t3 and their second probability values are sampled.

[0175] In another case, after reasoning by the target model, if it is determined that s11 among the candidate results of t1 can be accepted as the final result of t1, then s11 can be taken as t1; if none of the candidate results of t2 can be accepted, then the candidate results of t2 and their second probability values can be resampled.

[0176] The terminal device 100 updates the local decoding result based on the new decoding result, and determines whether the end condition is satisfied. If so, the updated local decoding result is taken as the final decoding result. If not, the step of performing autoregressive serial decoding based on the prefix using the configured approximate model is continued.

[0177] Among them, the process by which the terminal device 100 updates the local decoding result based on the new decoding result may include:

[0178] The terminal device 100 uses the accepted tokens in the new decoding result to update the tokens at the same sequence position in the local decoding result, and determines whether there is an overlap between the candidate results in the next token information in the new decoding result and the candidate results of the tokens at the same sequence position in the local decoding result. If there is an overlap, the first probability value of the overlapping candidate results in the local is updated to the second probability value, and the sorting of the candidate results is refreshed.

[0179] Exemplarily, the new decoding result includes tx1, tx2, tx3, and Tx4, where tx1 - tx3 are the accepted tokens, and Tx4 is the sampled next token information, which includes three candidate results, sx1, sx2, and sx3, and their respective second probability values are px1, px2, and px3.

[0180] The local decoding result includes Ty1, Ty2, Ty3, and Ty4. Taking Ty4 as an example, it includes four candidate results, sy1, sy2, sy3, and sy4, and their respective first probability values are py1, py2, py3, and py4.

[0181] After receiving the new decoding result, the terminal device can use tx1, tx2, and tx3 to update the local Ty1, Ty2, and Ty3.

[0182] For the 4th token:

[0183] Compare whether there are overlapping candidate results in Tx4 and Ty4. Assume that sx1 and sy1 overlap. Then, the second probability value px1 of sx1 can be used to replace the first probability value py1 of the local sy1. After replacement, the probability values of the candidate results of the 4th local token are as follows:

[0184] sy1 - px1;

[0185] sy2 - py2;

[0186] sy3 - py3;

[0187] sy4 - py4.

[0188] Among them, the part before "-" is the candidate result, and the part after "-" is the corresponding probability value.

[0189] The candidate results can be re - sorted according to the latest probability values of the candidate results of the 4th token above.

[0190] Adopting the solution provided in this embodiment, the terminal device 100 can continuously perform decoding. At the same time, by combining the next - token information fed back by the cloud server 200 with the information of the token at the corresponding sequence position decoded locally, the judgment of overlapping candidate results is carried out. If an overlapping candidate result can be matched, in order to improve the local model inference effect, the second probability value of the overlapping candidate result can be used to replace the first probability value locally, that is, taking the second probability value of the overlapping candidate result obtained by the target model as the standard, and re - sorting the candidate results according to the replaced probability value can improve the decoding result accuracy. Obviously, adopting the solution of this embodiment can improve the decoding rate during end - cloud collaborative decoding, reduce the network latency that may occur during the end - cloud collaboration process, and improve the user experience.

[0191] Furthermore, if the terminal device 100 determines that there is no overlap between the candidate results in the next - token information in the new decoding result and the candidate results of the token at the same sequence position in the local decoding result, it means that all the candidate results of the token at the same sequence position in the local decoding result are not accepted, and the approximate model can be made to roll back to the position of the next - token in the new decoding result and continue serial decoding.

[0192] Still taking the previous example for illustration:

[0193] Assume that there are no overlapping candidate results in Tx4 and Ty4, then the approximate model can be made to fallback to the 4th token for re-decoding. It can be understood that at this time, since the terminal device 100 has updated the local decoding result, the 4th token can be re-decoded based on the locally updated decoding result, which can improve the accuracy of the re-decoding result.

[0194] In some embodiments of the present application, from the perspective of the terminal device 100 in the foregoing decoding system, the large model decoding method of the present application will be described.

[0195] Combined with ​ As shown, the large model decoding method may include:

[0196] Step S100: Use an approximate model to perform autoregressive serial decoding based on the prefix to obtain the current decoding result.

[0197] Among them, the prefix at the initial decoding moment is the input prompt instruction.

[0198] Step S110: When the set upload condition is reached, send the current decoding result to the cloud server.

[0199] Specifically, by sending the current decoding result to the cloud server, so that the cloud server uses the target model to infer and determine the text unit token and sampling next token information accepted in the current decoding result, and a new decoding result is composed of the accepted token and the next token information.

[0200] Step S120: Obtain the new decoding result returned by the cloud server, and update the local decoding result based on the new decoding result.

[0201] Step S130: Determine whether the end condition is satisfied. If not, continue to execute step S100. If so, execute step S140.

[0202] Step S140: Use the updated local decoding result as the final decoding result.

[0203] The large model decoding method provided in this embodiment decodes using a small-scale approximate model on the terminal device, and then uses a larger-scale target model on the cloud server to confirm or correct the decoding result of the approximate model, which can ensure the inference effect of the local model of the terminal device. At the same time, there is no need for the cloud server to decode each token one by one, which can reduce the service load of the cloud server.

[0204] The current decoding result obtained in the above step S100 can have two forms:

[0205] In one case, the current decoding result includes r consecutive tokens of the first quantity and the first probability value of each token. On this basis, after the terminal device sends the current decoding result to the cloud server, the decoding process is paused until a new decoding result returned by the cloud server is received, and after determining that the end condition is not met, the decoding process continues.

[0206] In another case, the current decoding result includes each candidate result of each token and its first probability value. The current decoding result is sent to the cloud server, and the cloud server uses the target model to reason and determine which candidate results of each token can be accepted as the final result of the token. If there is a token whose candidate results cannot be accepted, the information of this token is resampled to obtain a new decoding result, which includes the tokens accepted in the current decoding result, and the candidate results of the next token sampled and their second probability values.

[0207] On this basis, after the terminal device sends the current decoding result to the cloud server, the decoding process continues until it is determined that the end condition is met. The process of the terminal device updating the local decoding result based on the new decoding result includes:

[0208] Using the tokens accepted in the new decoding result to update the tokens at the same sequence position in the local decoding result, and determining whether there is an overlap between the candidate results of the next token in the new decoding result and the candidate results of the tokens at the same sequence position in the local decoding result;

[0209] If there is an overlap, update the first probability value of the local overlapping candidate result to the second probability value and refresh the sorting of each candidate result;

[0210] If there is no overlap, control the approximate model to roll back to the position of the next token in the new decoding result and continue serial decoding.

[0211] For other implementation details of the terminal device, reference can be made to the description of the embodiments on the side of the large model decoding system in the foregoing text.

[0212] In some embodiments of the present application, from the perspective of the cloud server 200 in the foregoing decoding system, the large model decoding method of the present application is described.

[0213] Combined with ​ As shown, the large model decoding method may include:

[0214] Step S200, obtain the current decoding result sent by the terminal device.

[0215] As described above, the current decoding result may include the candidate result with the highest probability value for each of the r tokens and its first probability value; alternatively, the current decoding result may also include several candidate results for each of the r tokens and their first probability values.

[0216] Among them, the current decoding result is obtained by the terminal device using an approximate model for autoregressive serial decoding. The specific decoding process can refer to the relevant description above and will not be elaborated here.

[0217] Step S210: Use the target model to infer and determine the tokens accepted in the current decoding result and sample the next token information, and form a new decoding result from the accepted tokens and the next token information.

[0218] Step S220: Return the new decoding result to the terminal device.

[0219] Specifically, the cloud server 200 returns the new decoding result to the terminal device so that the terminal device can update the local decoding result based on the new decoding result.

[0220] In some possible implementations, after obtaining the new decoding result, the cloud server 200 may further add the following steps:

[0221] Step S230: Determine whether the end condition is satisfied.

[0222] This determination method can refer to the introduction of the end condition determination above and will not be elaborated here.

[0223] When it is determined that the end condition is satisfied, this round of decoding process can be ended. When it is determined that the end condition is not satisfied, step S200, the step of obtaining the decoding result sent by the terminal device, can be returned, that is, it can continuously detect whether the decoding result sent by the terminal device is received.

[0224] In the large model decoding method provided in this embodiment, the target model (large model) is deployed on the cloud server. This target model does not decode token by token, but only confirms or corrects the results decoded by the approximate model uploaded by the terminal device, effectively reducing the service load of the cloud server. At the same time, the cloud server uses the target model to confirm or correct the decoding result of the approximate model, which can improve the accuracy of the decoding result of the local model of the terminal device and enhance the final inference effect of the local model of the terminal device.

[0225] For the above step S210, the process of using the target model to infer and determine the tokens accepted in the current decoding result and sample the next token information

[0226] It may be to use the target model to determine the tokens accepted in the current decoding result, sample the candidate results of the next token and their second probability values, and select the candidate result with the highest second probability value as the information of the sampled next token.

[0227] In addition, it may also be the information of the sampled next token composed of the candidate results of the next token and their second probability values.

[0228] That is, in this embodiment, the cloud server 200 may use the candidate result with the highest probability value of the sampled next token as the information of the next token, add it to the new decoding result and send it to the terminal device for the terminal device to update the local decoding result and guide the subsequent decoding process. In addition, the cloud server 200 may also use the candidate results of the sampled next token and their second probability values as the information of the next token, add it to the new decoding result and send it to the terminal device for the terminal device to update the local decoding result and guide the subsequent decoding process. The process of the terminal device performing relevant processing based on the new decoding result may refer to the relevant introduction above and will not be elaborated here.

[0229] In some possible implementations, the current decoding result uploaded by the terminal device includes r tokens and the first probability value of each token. On this basis, the above step S210, the process of using the target model to reason and determine the tokens accepted in the current decoding result and sample the information of the next token includes:

[0230] Concatenate the r tokens in the current decoding result with the local prefix, input them into the target model for forward propagation to obtain the second probability value of each token, and based on the first probability value and the second probability value of each token, select the accepted tokens from the current decoding result and sample the information of the next token.

[0231] In other possible implementations, the current decoding result uploaded by the terminal device includes the candidate results of each of the r tokens and the first probability value of each candidate result. On this basis, the above step S210, the process of using the target model to reason and determine the tokens accepted in the current decoding result and sample the information of the next token includes:

[0232] Call the target model, and based on the local prefix and the first probability value of each candidate result of each token in the current decoding result, determine the final result that can be accepted as a token from each candidate result of each token. If there is a token whose candidate results cannot be accepted, resample the information of this token.

[0233] For other implementation details of the cloud server, reference may be made to the description of the embodiments on the side of the large model decoding system in the foregoing text.

[0234] An embodiment of the present application also provides an electronic device. Refer to ​ As shown, it shows a schematic structural diagram of an electronic device suitable for implementing the electronic device in the embodiments of the present application. The electronic device in the embodiments of the present application may include, but is not limited to, fixed terminals such as mobile phones, tablet computers, teaching large screens, wearable devices, cloud servers, and the like. ​ The electronic device shown is merely an example and should not impose any limitations on the functions and usage scope of the embodiments of the present application.

[0235] As ​ shown, the electronic device may include a processing device (such as a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 602 or the program loaded from the storage device 608 into the random access memory (RAM) 603, so as to implement the large model decoding method on the side of the terminal device in the foregoing embodiments of the present application, or implement the large model decoding method on the side of the cloud server in the foregoing embodiments of the present application. When the electronic device is powered on, various programs and data required for the operation of the electronic device are also stored in the RAM 603. The processing device 601, the ROM 602, and the RAM 603 are connected to each other through a bus 604. The input / output (I / O) interface 605 is also connected to the bus 604.

[0236] Generally, the following devices may be connected to the I / O interface 605: an input device 606 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 608 including, for example, a memory card, a hard disk, etc.; and a communication device 609. The communication device 609 can allow the electronic device to communicate with other devices wirelessly or wiredly to exchange data. Although ​ the electronic device with various devices is shown, it should be understood that it is not required to implement or have all the shown devices. More or fewer devices may be alternatively implemented or had.

[0237] An embodiment of the present application also provides a computer program product including computer-readable instructions, which, when running on an electronic device, enable the electronic device to implement any of the large model decoding methods on the side of the terminal device provided in the embodiments of the present application, or implement any of the large model decoding methods on the side of the cloud server provided in the embodiments of the present application.

[0238] In an embodiment of the present application, a computer-readable storage medium is further provided. The storage medium carries one or more computer programs. When the one or more computer programs are executed by an electronic device, any large model decoding method on the terminal device side provided by the embodiments of the present application can be implemented, or any large model decoding method on the cloud server side provided by the embodiments of the present application can be implemented.

[0239] In addition, it should be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. In addition, in the accompanying drawings of the device embodiments provided in the present application, the connection relationships between the modules indicate that there is a communication connection between them, which can be specifically implemented as one or more communication buses or signal lines.

[0240] Through the description of the above embodiments, those skilled in the art can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course, it can also be implemented by dedicated hardware including application-specific integrated circuits, dedicated CPUs, dedicated memories, dedicated components, etc. Generally, functions completed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structures for implementing the same function can also be various, such as analog circuits, digital circuits or dedicated circuits. However, for the present application, in more cases, software program implementation is a better implementation method. Based on such an understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. The computer software product is stored in a readable storage medium, such as a floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk or optical disc of a computer, etc., and includes several instructions for causing a computer device (which can be a personal computer, training device, or network device, etc.) to execute the methods described in the various embodiments of the present application.

[0241] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product.

[0242] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions described in the embodiments of the present application are generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from one website, computer, training device, or data center to another website, computer, training device, or data center by wire (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wirelessly (such as infrared, wireless, microwave, etc.). The computer-readable storage medium may be any available medium that can be stored by a computer or a data storage device such as a training device or a data center that includes one or more integrated available media. The available medium may be a magnetic medium (such as a floppy disk, hard disk, magnetic tape), an optical medium (such as a DVD), or a semiconductor medium (such as a solid state disk (SSD)), etc.

[0243] The various embodiments in this specification are described in a progressive manner. Each embodiment focuses on the differences from other embodiments. The various embodiments can be combined as needed, and the same or similar parts can be referred to each other.

Claims

1. A large model decoding system, characterized in that, Including: A terminal device and a cloud server, where a target model is configured on the cloud server, and an approximate model is configured on the terminal device. The target model is a large model, and the approximate model is a model with a smaller parameter scale than the target model; The terminal device is used to perform autoregressive serial decoding based on a prefix using the configured approximate model, and when a set upload condition is met, send the current decoding result to the cloud server, and continue the decoding process after sending the current decoding result to the cloud server until it is determined that the end condition is satisfied; the current decoding result includes each candidate result of each token and its first probability value; The cloud server is used to determine the tokens accepted in the current decoding result using the configured target model, sample to obtain each candidate result of the next token and its second probability value, use the candidate results and the second probability value as the information of the sampled next token, form a new decoding result with the accepted token and the next token information, and return the new decoding result to the terminal device; The terminal device is further used to update the token at the same sequence position in the local decoding result using the token accepted in the new decoding result, and determine whether there is an overlap between each candidate result in the next token information in the new decoding result and each candidate result of the token at the same sequence position in the local decoding result. If there is an overlap, update the first probability value of the local overlapping candidate result to the second probability value and refresh the sorting of each candidate result. If it is determined that there is no overlap between each candidate result in the next token information in the new decoding result and each candidate result of the token at the same sequence position in the local decoding result, control the approximate model to roll back to the position of the next token in the new decoding result and continue serial decoding; determine whether the end condition is satisfied. If so, use the updated local decoding result as the final decoding result. If not, continue to execute the step of performing autoregressive serial decoding based on the prefix using the configured approximate model.

2. The system according to claim 1, wherein The current decoding result includes a consecutive first number r of tokens and the first probability value of each token; The terminal device pauses the decoding process after sending the current decoding result to the cloud server until it receives the new decoding result returned by the cloud server, and continues the decoding process after determining that the end condition is not satisfied; Select the candidate result with the highest second probability value as the information of the sampled next token.

3. The system according to claim 1, wherein The current decoding result includes r tokens and the first probability value of each token; The cloud server concatenates r tokens in the current decoding result with the local prefix and inputs them into the target model for forward propagation to obtain the second probability value of each token. Based on the first probability value and the second probability value of each token, the accepted tokens are selected from the current decoding result, and the information of the next token is sampled.

4. The system according to claim 1, wherein The current decoding result includes the candidate results of each of the r tokens and the first probability value of each candidate result; The cloud server invokes the target model and determines, based on the local prefix and the first probability values of the candidate results of each token in the current decoding result, the final result that can be accepted as a token from the candidate results of each token. If none of the candidate results of a token can be accepted, the information of this token is resampled.

5. The system according to any one of claims 1-4, characterized in that, The process by which the terminal device determines whether the end condition is met includes: The terminal device determines whether the token length in the new decoding result reaches a threshold or whether it contains a set termination symbol. If the judgment result of any of the conditions is yes, it is determined that the end condition is met; otherwise, it is determined that the end condition is not met.

6. A large model decoding method, characterized in that, A terminal device applied to a decoding system, the decoding system further includes a cloud server, a target model is configured on the cloud server, and an approximate model is configured on the terminal device. The target model is a large model, and the approximate model is a small model with a smaller parameter scale than the target model; the method includes: Using the configured approximate model, perform autoregressive serial decoding based on the prefix, and when the set upload condition is reached, send the current decoding result to the cloud server, and continue the decoding process after sending the current decoding result to the cloud server until it is determined that the end condition is met, so that the cloud server uses the target model to determine the accepted tokens in the current decoding result, and samples the candidate results and their second probability values of the next token. The candidate results and their second probability values are used as the information of the sampled next token, and the new decoding result is composed of the accepted tokens and the information of the next token; the current decoding result includes the candidate results of each token and their first probability values; Obtain the new decoding result returned by the cloud server, update the token at the same sequence position in the local decoding result with the token accepted in the new decoding result, and determine whether there is an overlap between each candidate result in the next token information in the new decoding result and each candidate result of the token at the same sequence position in the local decoding result. If there is an overlap, update the first probability value of the overlapping candidate result in the local to the second probability value, and refresh the sorting of each candidate result. If it is determined that there is no overlap between each candidate result in the next token information in the new decoding result and each candidate result of the token at the same sequence position in the local decoding result, control the approximate model to roll back to the position of the next token in the new decoding result and continue serial decoding; Determine whether the end condition is satisfied. If so, use the updated local decoding result as the final decoding result. If not, continue to execute the step of performing autoregressive serial decoding based on the prefix using the configured approximate model.

7. The method according to claim 6, characterized in that, The current decoding result includes a continuous first number r of tokens and the first probability value of each token; After the terminal device sends the current decoding result to the cloud server, it pauses the decoding process until it receives the new decoding result returned by the cloud server, and continues the decoding process after determining that the end condition is not satisfied.

8. A large model decoding method, characterized in that, Applied to the cloud server in a decoding system, the decoding system further includes a terminal device. A target model is configured on the cloud server, and an approximate model is configured on the terminal device. The target model is a large model, and the approximate model is a small model with a smaller parameter scale than the target model; The method includes: Obtain the current decoding result sent by the terminal device, where the current decoding result is obtained by the terminal device performing autoregressive serial decoding using the approximate model; The current decoding result includes each candidate result of each token and its first probability value; Use the target model to determine the token accepted in the current decoding result, sample to obtain each candidate result of the next token and its second probability value. Use each candidate result and its second probability value as the information of the sampled next token. The accepted token and the next token information form a new decoding result; Return the new decoding result to the terminal device, so that the terminal device uses the tokens accepted in the new decoding result to update the tokens at the same sequence positions in the local decoding result, and determine whether there is an overlap between the candidate results in the next token information in the new decoding result and the candidate results of the tokens at the same sequence positions in the local decoding result. If there is an overlap, update the first probability value of the local overlapping candidate result to the second probability value, and refresh the sorting of each candidate result. If it is determined that there is no overlap between the candidate results in the next token information in the new decoding result and the candidate results of the tokens at the same sequence positions in the local decoding result of the terminal device, the terminal device controls the approximate model to go back to the position of the next token in the new decoding result and continue serial decoding.

9. The method according to claim 8, wherein Select the candidate result with the highest second probability value as the sampled next token information.

10. The method according to claim 8, characterized in that The current decoding result includes r tokens and the first probability value of each token; Concatenate the r tokens in the current decoding result with the local prefix, and input them into the target model for forward propagation to obtain the second probability value of each token. Based on the first probability value and the second probability value of each token, select the accepted tokens from the current decoding result, and sample the next token information.

11. The method according to claim 8, wherein The current decoding result includes the candidate results of r tokens respectively and the first probability value of each candidate result; Call the target model, and based on the local prefix and the first probability value of each candidate result of each token in the current decoding result, determine the final result that can be accepted as a token from each candidate result of each token. If none of the candidate results of a token can be accepted, resample the information of this token.

12. An electronic device, characterized in that, Comprising: A memory and a processor; The memory is used for storing programs; The processor is used for executing the program to implement each step of the large model decoding method as described in any one of claims 6 - 7 or 8 - 11.

13. A readable storage medium, on which a computer program is stored, characterized in that, When the computer program is executed by the processor, it implements each step of the large model decoding method as described in any one of claims 6 - 7 or 8 - 11.

14. A computer program product, comprising a computer program, characterized in that, When this computer program is executed by the processor, it implements each step of the large model decoding method as described in any one of claims 6 - 7 or 8 - 11.

Images