Inference method, related device and inference system

Abstract

The invention discloses an inference method, a related device and an inference system, which are applied to the technical field of inference acceleration, and the inference method comprises the following steps: a cloud server performs model distillation on a first model to obtain a second model, copies the second model to obtain a shadow model, and sends the second model to a terminal; deploying a second model locally by the terminal, and reasoning the user request by using the local second model; the terminal sends the reasoning related data to a cloud server; the cloud server performs parameter optimization on the shadow model by utilizing the reasoning related data and the first model, and sends the optimized parameters of the shadow model to the terminal; and the terminal performs parameter optimization on a local second model according to the optimized parameters of the shadow model. The reasoning process is completed by the local small model at the end side, so that the reasoning method disclosed by the invention has relatively high reasoning speed, and in addition, the reasoning method provided by the invention can realize optimization of the end side model through the shadow model at the cloud side.

Description

Technical Field

[0001] The present application relates to the field of reasoning acceleration technology, and in particular to a reasoning method, related devices and a reasoning system. Background Art

[0002] Most of the current inference schemes are as follows: the terminal receives the user request and sends the user request to the cloud server. After the cloud server receives the user request from the terminal, it uses the deployed large model to infer the user request. After obtaining the inference result of the user request, it sends the inference result of the user request to the terminal.

[0003] Since the large models of cloud servers have more parameters and more complex structures, the inference speed is slower, which affects the user experience. Summary of the invention

[0004] In view of this, the present application provides a reasoning method, a related device and a reasoning system to solve the problem that the existing reasoning scheme has a slow reasoning speed, thereby affecting the user experience. The technical solution is as follows:

[0005] A first aspect of the present application provides an inference method, which is applied to a cloud server, wherein a first model is deployed on the cloud server, and the inference method includes:

[0006] Obtain a second model by performing model distillation on the first model, and obtain a shadow model by copying the second model;

[0007] Sending the second model to the terminal so that the terminal locally deploys the second model and uses the local second model to reason about the user request;

[0008] receiving the inference-related data sent by the terminal, and optimizing parameters of the shadow model using the inference-related data and the first model;

[0009] The optimized parameters of the shadow model are sent to the terminal, so that the terminal optimizes the parameters of the local second model according to the optimized parameters of the shadow model.

[0010] In a possible implementation, the reasoning-related data includes: the user request;

[0011] The using the inference-related data, aided by the first model, to optimize parameters of the shadow model includes:

[0012] determining a difference in reasoning between the second model or the shadow model and the first model for the user request;

[0013] Parameters of the shadow model are optimized according to the inference difference.

[0014] In a possible implementation, the inference-related data further includes: a first inference result obtained by the terminal using the local second model to infer the user request, and first probability information of each word in the first inference result;

[0015] Determining a difference in reasoning between the second model and the first model for the user request includes:

[0016] Using the first model, according to the user request and the first inference result, obtaining second probability information of each word in the first inference result;

[0017] The prediction loss of the second model is determined according to the first probability information and the second probability information of each word in the first inference result, wherein the prediction loss of the second model can characterize the difference in inference between the second model and the first model for the user request.

[0018] In a possible implementation manner, determining the prediction loss of the second model according to the first probability information and the second probability information of each word in the first inference result includes:

[0019] For each word in the first inference result, determine the difference between the first probability information of the word and the second probability information of the word, and obtain a probability information difference representation value as the prediction loss of the second model on the word;

[0020] The prediction losses of the second model on each word of the first inference result are fused to obtain the final prediction loss of the second model.

[0021] In a possible implementation, determining a difference between the shadow model and the first model in reasoning about the user request includes:

[0022] Inferring the user request using the shadow model to obtain a second inference result of the user request and first probability information of each word in the second inference result;

[0023] Using the first model, according to the user request and the second inference result, obtaining second probability information of each word in the second inference result;

[0024] The prediction loss of the shadow model is determined according to the first probability information and the second probability information of each word in the second inference result, wherein the prediction loss of the shadow model can characterize the difference in inference between the shadow model and the first model for the user request.

[0025] In a possible implementation, the first probability information of any word in an inference result is the first probability or the first candidate word probability distribution corresponding to the word in the inference result, the first candidate word probability distribution corresponding to any word in an inference result is the candidate word probability distribution obtained in the process of predicting the word, and the first probability corresponding to any word in an inference result is obtained from the first candidate word probability distribution corresponding to the word;

[0026] Using the first model, according to the user request and an inference result of the user request, obtaining second probability information of each word in the inference result, including:

[0027] At the tth prediction step:

[0028] Using the first model, based on the user request and the first t words in the inference result, predict the candidate word probability distribution of the t+1th word, and use the candidate word probability distribution of the t+1th word as the second candidate word probability distribution corresponding to the t+1th word in the inference result;

[0029] Alternatively, using the first model, based on the user request and the first t words in the inference result, predicting the probability distribution of candidate words for the t+1th word, and obtaining a second probability corresponding to the t+1th word in the inference result from the probability distribution of candidate words for the t+1th word;

[0030] The second probability or the second candidate word probability distribution corresponding to the t+1th word in the inference result is used as the second probability information of the t+1th word in the inference result; wherein t ranges from 0 to N-1, and N is the length of the inference result.

[0031] In a possible implementation, determining a difference between the shadow model and the first model in reasoning about the user request includes:

[0032] Using the first model, reasoning the user request to obtain a third reasoning result of the user request and first probability information of each word in the third reasoning result;

[0033] Using the shadow model, according to the user request and the third reasoning result, obtaining second probability information of each word in the third reasoning result;

[0034] The prediction loss of the shadow model is determined according to the first probability information and the second probability information of each word in the third reasoning result, wherein the prediction loss of the shadow model can characterize the reasoning difference between the shadow model and the first model for the user request.

[0035] In a possible implementation, the first probability information of any word in the third reasoning result is the first probability or the first candidate word probability distribution corresponding to the word in the third reasoning result, the first candidate word probability distribution corresponding to any word in the third reasoning result is the candidate word probability distribution obtained in the process of predicting the word, and the first probability corresponding to any word in the third reasoning result is obtained from the first candidate word probability distribution corresponding to the word;

[0036] The using the shadow model, according to the user request and the third reasoning result, to obtain second probability information of each word in the third reasoning result, includes:

[0037] At the tth prediction step:

[0038] Using the shadow model, based on the user request and the first t words in the third reasoning result, predict the candidate word probability distribution of the t+1th word, and use the candidate word probability distribution of the t+1th word as the second candidate word probability distribution corresponding to the t+1th word in the third reasoning result;

[0039] Alternatively, using the shadow model, based on the user request and the first t words in the third reasoning result, predicting the probability distribution of candidate words for the t+1th word, obtaining the probability corresponding to the t+1th word in the third reasoning result from the probability distribution of candidate words for the t+1th word, and using the probability corresponding to the t+1th word as the second probability corresponding to the t+1th word in the third reasoning result;

[0040] The second probability or the second candidate word probability distribution corresponding to the t+1th word in the third inference result is used as the second probability information of the t+1th word in the third inference result; wherein t ranges from 0 to M-1, and M is the length of the third inference result.

[0041] In a possible implementation manner, before sending the optimized parameters of the shadow model to the terminal, the method further includes:

[0042] Determine whether a preset parameter return condition is met, wherein the parameter return condition is that the time interval between the last parameter sending to the terminal reaches a preset time interval, or the number of parameter optimizations of the shadow model after the last parameter sending to the terminal reaches a preset number of parameter optimizations;

[0043] If the parameter return condition is met, the optimized parameters of the shadow model are sent to the terminal.

[0044] A second aspect of the present application provides an inference method, which is applied to a terminal. The inference method includes:

[0045] Receiving a second model sent by the cloud server, and deploying the second model locally, wherein the second model is a model obtained by performing model distillation on the first model of the cloud server;

[0046] When receiving a user request, using the local second model to infer the user request to obtain an inference result of the user request;

[0047] Sending the inference-related data to a cloud server so that the cloud server uses the inference-related data and the first model to optimize parameters of a shadow model, wherein the shadow model is obtained by the cloud server by copying the second model;

[0048] Receive the optimized parameters of the shadow model sent by the cloud server, and optimize the parameters of the local second model according to the optimized parameters of the shadow model.

[0049] In a possible implementation manner, the reasoning-related data includes the user request; or, the reasoning-related data includes the user request, the reasoning result, and probability information of each word in the reasoning result.

[0050] A third aspect of the present application provides an inference device, which is applied to a cloud server, wherein a first model is deployed on the cloud server, and the inference device comprises: a model distillation module, a shadow model acquisition module, a model sending module, an inference-related data receiving module, a model optimization module, and a parameter sending module;

[0051] The model distillation module is used to obtain a second model by performing model distillation on the first model;

[0052] The shadow model acquisition module is used to obtain the shadow model by copying the second model;

[0053] The model sending module is used to send the second model to the terminal, so that the terminal deploys the second model locally and uses the local second model to infer the user request;

[0054] The inference-related data receiving module is used to receive the inference-related data sent by the terminal;

[0055] The model optimization module is used to optimize the parameters of the shadow model by using the reasoning related data and the first model;

[0056] The parameter sending module is used to send the optimized parameters of the shadow model to the terminal, so that the terminal optimizes the parameters of the local second model according to the optimized parameters of the shadow model.

[0057] A fourth aspect of the present application provides an inference device, which is applied to a terminal, and the inference device includes: a model receiving module, a model deployment module, an inference module, an inference-related data sending module, a parameter receiving module, and a model optimization module;

[0058] The model receiving module is used to receive the second model sent by the cloud server;

[0059] The model deployment module is used to deploy the second model locally, wherein the second model is a model obtained by performing model distillation on the first model of the cloud server;

[0060] The reasoning module is used to, when receiving a user request, use the local second model to reason about the user request to obtain a reasoning result of the user request;

[0061] The inference-related data sending module is used to send the inference-related data to the cloud server, so that the cloud server uses the inference-related data and the first model to optimize the parameters of the shadow model, wherein the shadow model is obtained by the cloud server by copying the second model;

[0062] The parameter receiving module is used to receive the optimized parameters of the shadow model sent by the cloud server;

[0063] The model optimization module is used to optimize the parameters of the local second model according to the optimized parameters of the shadow model.

[0064] A fifth aspect of the present application provides a cloud server, comprising at least one processor and a memory connected to the processor, wherein:

[0065] The memory is used to store computer programs;

[0066] The processor is used to execute the computer program so that the cloud server can implement the steps of any one of the above-mentioned reasoning methods.

[0067] A sixth aspect of the present application provides a terminal, comprising at least one processor and a memory connected to the processor, wherein:

[0068] The memory is used to store computer programs;

[0069] The processor is used to execute the computer program so that the terminal can implement the steps of any one of the above-mentioned reasoning methods.

[0070] In a seventh aspect, the present application provides a computer storage medium, which carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement the steps of any one of the above-mentioned reasoning methods.

[0071] An eighth aspect of the present application provides a computer program product, comprising computer-readable instructions, which, when executed on an electronic device, enables the electronic device to implement the steps of any one of the above-mentioned reasoning methods.

[0072] A ninth aspect of the present application provides an inference system, comprising: a terminal and a cloud server, wherein a first model is deployed on the cloud server;

[0073] The cloud server is configured to obtain a second model by performing model distillation on the first model, obtain a shadow model by copying the second model, and send the second model to the terminal;

[0074] The terminal is used to receive the second model, deploy the second model locally, and, when receiving a user request, use the local second model to infer the user request and send the inference-related data to the cloud server;

[0075] The cloud server is further configured to receive the inference-related data, and optimize the parameters of the shadow model using the inference-related data in combination with the first model, and send the optimized parameters of the shadow model to the terminal;

[0076] The terminal is further used to receive the optimized parameters of the shadow model, and optimize the parameters of the local second model according to the optimized parameters of the shadow model.

[0077] By means of the above technical scheme, in the reasoning method provided by the present application, the cloud server obtains the second model by performing model distillation on the first model, and obtains the shadow model by copying the second model, and then sends the second model to the terminal. After the terminal receives the second model, it deploys the second model locally, and then uses the local second model to reason about the user request. After the terminal reasoning is completed, the reasoning-related data is sent to the cloud server. When the cloud server receives the reasoning-related data, it uses the reasoning-related data, supplemented by the first model, to optimize the parameters of the shadow model, and then sends the optimized parameters of the shadow model to the terminal. After the terminal receives the optimized parameters of the shadow model, it optimizes the parameters of the local second model according to the received parameters. In the reasoning method provided by the present application, the cloud server obtains the second model by performing model distillation on the first model and provides it to the terminal, so that the terminal can deploy a small-scale second model with better performance locally, and then use the local second model to reason about the user request, thereby significantly improving the reasoning speed. In addition, the reasoning method provided by the present application can also optimize the parameters of the second model on the terminal side by optimizing the parameters of the cloud-side shadow model. By optimizing the second model on the terminal side, the reasoning effect on the terminal side can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0078] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0079] Figure 1 A schematic diagram of a system architecture involved in this application;

[0080] Figure 2 A schematic diagram of a hardware structure of a terminal provided in an embodiment of the present application;

[0081] Figure 3 A schematic diagram of a hardware structure of a server provided in an embodiment of the present application;

[0082] Figure 4 A flowchart of an inference method applied to a cloud server provided in an embodiment of the present application;

[0083] Figure 5 A flowchart of a reasoning method applied to a terminal provided in an embodiment of the present application;

[0084] Figure 6 A flowchart of a method for implementing reasoning and model optimization by interaction between a cloud server and a terminal provided in an embodiment of the present application;

[0085] Figure 7 A schematic diagram of the interaction between the cloud server and the terminal provided in an embodiment of the present application;

[0086] Figure 8 A flowchart of another method for implementing reasoning and model optimization by interaction between a cloud server and a terminal provided in an embodiment of the present application;

[0087] Fig. 9 A flowchart of another method for implementing reasoning and model optimization by interaction between a cloud server and a terminal provided in an embodiment of the present application;

[0088] Fig.10 A schematic diagram of the structure of an inference device applied to a cloud server provided in an embodiment of the present application;

[0089] Fig.11 A schematic diagram of the structure of an inference device applied to a terminal provided in an embodiment of the present application. DETAILED DESCRIPTION

[0090] The following describes the embodiments of the present application in conjunction with the drawings in the embodiments of the present application. The terms used in the implementation method section of the present application are only used to explain the specific embodiments of the present application, and are not intended to limit the present application.

[0091] The embodiments of the present application are described below in conjunction with the accompanying drawings. Those skilled in the art will appreciate that, with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0092] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and need not be used to describe a specific order or sequential order. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, which is only to describe the distinction mode adopted by the objects of the same attributes when describing in the embodiments of the present application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, so that the process, method, system, product or equipment comprising a series of units need not be limited to those units, but may include other units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0093] The system architecture involved in the present application may include a terminal 101 and a cloud server 102. The terminal 101 may interact with the cloud server 102 via a network (wired network or wireless network). The cloud server 102 may include one or more servers ( Figure 1 A server is included as an example for explanation).

[0094] Next, the product form of the above-mentioned terminal is described.

[0095] The above-mentioned terminal can be a mobile phone, a tablet computer, a wearable device, a vehicle-mounted device, a robot, an augmented reality (AR) / virtual reality (VR) device, a laptop computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), etc., and the embodiments of the present application do not impose any restrictions on this.

[0096] Figure 2 An optional hardware structure diagram of a terminal is shown.

[0097] refer to Figure 2 As shown, the terminal may include a radio frequency unit 210, a memory 220, an input unit 230, a display unit 240, a camera 250 (optional), an audio circuit 260 (optional), a speaker 261 (optional), a microphone 262 (optional), an earphone jack 263 (optional), a processor 270, an external interface 280, a power supply 290 and other components. Those skilled in the art will appreciate that Figure 2 This is merely an example of a terminal and does not constitute a limitation on the terminal. The terminal may include more or fewer components than shown in the figure, or a combination of certain components, or different components.

[0098] The input unit 230 can be used to receive input digital or character information, and generate key signal input related to the user settings and function control of the terminal. Specifically, the input unit 230 may include a touch screen 231 (optional) and / or other input devices 232. The touch screen 231 can collect the user's touch operations on or near it (such as the user's operation on or near the touch screen using any suitable object such as fingers, joints, stylus, etc.), and drive the corresponding connection device according to a pre-set program. The touch screen can detect the user's touch action on the touch screen, convert the touch action into a touch signal and send it to the processor 270, and can receive and execute the command sent by the processor 270; the touch signal at least includes the contact point coordinate information. The touch screen 231 can provide an input interface and an output interface between the terminal and the user. In addition, the touch screen can be implemented using multiple types such as resistive, capacitive, infrared, and surface acoustic waves. In addition to the touch screen 231, the input unit 230 can also include other input devices. Specifically, other input devices 232 may include, but are not limited to, one or more of a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, a joystick, and the like.

[0099] The display unit 240 may be used to display information input by a user or information provided to a user, various menus of the terminal, an interactive interface, file display, and / or playback of any multimedia file.

[0100] The memory 220 can be used to store instructions and data. The memory 220 can mainly include an instruction storage area and a data storage area. The data storage area can store various data, such as multimedia files, texts, etc. The instruction storage area can store software units such as operating systems, applications, and instructions required for at least one function, or their subsets and extensions. It can also include a non-volatile random access memory; it provides the processor 270 with hardware, software, and data resources including management of computing and processing equipment, and supports control software and applications. It is also used for the storage of multimedia files, and the storage of running programs and applications.

[0101] The processor 270 is the control center of the terminal. It uses various interfaces and lines to connect various parts of the entire terminal. By running or executing instructions stored in the memory 220 and calling data stored in the memory 220, it executes various functions of the terminal and processes data, thereby controlling the terminal as a whole. Optionally, the processor 270 may include one or more processing units; preferably, the processor 270 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface and application program, and the modem processor mainly processes wireless communication. It is understandable that the above-mentioned modem processor may not be integrated into the processor 270. In some embodiments, the processor and the memory may be implemented on a single chip, and in some embodiments, they may also be implemented separately on separate chips. The processor 270 may also be used to generate corresponding operation control signals, send them to corresponding components of the computing and processing device, read and process data in the software, especially read and process data and programs in the memory 220, so that each functional module therein performs corresponding functions, thereby controlling the corresponding components to act according to the requirements of the instructions.

[0102] Among them, the memory 220 can be used to store software codes related to the reasoning method, the processor 270 can execute the software codes in the memory 220, and can also schedule other units (such as the above-mentioned input unit 230 and display unit 240) to achieve corresponding functions.

[0103] The radio frequency unit 210 (optional) can be used for receiving and sending information or receiving and sending signals during a call, for example, after receiving the downlink information of the base station, it is sent to the processor 270 for processing; in addition, the designed uplink data is sent to the base station. Generally, the radio frequency unit 210 includes but is not limited to an antenna, at least one amplifier, a transceiver, a coupler, a low noise amplifier (LowNoiseAmplifier, LNA), a duplexer, etc. In addition, the radio frequency unit 210 can also communicate with network devices and other devices through wireless communication. The wireless communication can use any communication standard or protocol, including but not limited to Global System of Mobile communication (Global System of Mobile communication, GSM), General Packet Radio Service (General PacketRadio Service, GPRS), Code Division Multiple Access (Code Division Multiple Access, CDMA), Wideband Code Division Multiple Access (Wideband Code Division Multiple Access, WCDMA), Long Term Evolution (Long Term Evolution, LTE), email, Short Messaging Service (SMS), etc.

[0104] In the embodiment of the present application, the radio frequency unit 210 can send data to other devices, and can also receive data sent by other devices. It should be understood that the radio frequency unit 210 is optional, and can be replaced by other communication interfaces, such as a network port.

[0105] The terminal also includes a power supply 290 (such as a battery) for supplying power to various components. Preferably, the power supply can be logically connected to the processor 270 through a power management system, so that the power management system can manage charging, discharging, power consumption and other functions.

[0106] The terminal also includes an external interface 280, which may be a standard Micro USB interface or a multi-pin connector, and may be used to connect the terminal to communicate with other devices, or to connect a charger to charge the terminal.

[0107] Although not shown, the terminal may further include a flashlight, a wireless fidelity (WiFi) module, a Bluetooth module, sensors with different functions, etc., which will not be described in detail here.

[0108] Next, the product form of the above-mentioned cloud server is described.

[0109] Figure 3 A structural diagram of the above-mentioned cloud server is provided, such as Figure 3 As shown, the cloud server may include a bus 301, a processor 302, a communication interface 303 and a memory 304. The processor 302, the memory 304 and the communication interface 303 communicate with each other via the bus 301.

[0110] The bus 301 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 3 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0111] The processor 302 may be any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0112] The memory 304 may include a volatile memory, such as a random access memory (RAM). The memory 304 may also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD), or a solid state drive (SSD).

[0113] The memory 304 may be used to store software codes related to the reasoning method. The processor 302 may call the software codes stored in the memory 304 or schedule other units to implement corresponding functions.

[0114] The processors in the above-mentioned terminals and cloud servers (such as processor 270 and processor 302) can be hardware circuits (such as application specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), general-purpose processors, digital signal processors (DSP), microprocessors or microcontrollers, etc.), or a combination of these hardware circuits. For example, the processor can be a hardware system with an instruction execution function, such as a CPU, DSP, etc., or a hardware system without an instruction execution function, such as an ASIC, FPGA, etc., or a combination of the above-mentioned hardware systems without an instruction execution function and hardware systems with an instruction execution function.

[0115] Most of the current inference solutions use large models on the cloud side to infer user requests. However, the large models on the cloud side have more parameters and more complex structures, so the inference speed is slow.

[0116] We conducted research on the problem of slow inference speed of the current inference scheme. The initial idea was that when receiving a user request, we first determined whether the user request existed in the end-side cache. If the user request existed in the end-side cache, the inference result corresponding to the user request in the end-side cache was used as the inference result obtained by inferring the user request. If the user request did not exist in the end-side cache, the large model on the cloud side was used to infer the user request, and the user request and its corresponding inference result were stored in the end-side cache.

[0117] The above idea can quickly give inference results for user requests that exist in the end-side cache. However, for user requests that do not exist in the end-side cache, it is still necessary to use the large model on the cloud side to infer the user requests. Moreover, in actual business scenarios, the content of user requests varies, and the actual cache hit probability is low, which means that the above idea does not effectively improve the inference speed.

[0118] In view of this, the inventor of this case continued to conduct research, and through continuous research, finally proposed a reasoning method with better effect, which can significantly improve the reasoning speed and thus improve the user experience. Next, the reasoning method provided in this application will be introduced through the following embodiments.

[0119] If you look at the technical solution of this application from the cloud server side, please refer to Figure 4 , is a flowchart of a reasoning method provided in an embodiment of the present application, and the method may include:

[0120] Step S401: obtaining a second model by performing model distillation on the first model, and obtaining a shadow model by copying the second model.

[0121] Among them, the first model is a large-scale model, the scale of the second model is smaller than the first model, and the storage and computing resources required are much smaller than the first model. The effect of the second model is close to that of the first model, that is, the second model is a small-scale model whose effect is close to that of the large-scale first model.

[0122] In a possible implementation, the process of obtaining the second model by performing model distillation on the first model may include: constructing an initial second model; and training the initial second model using the training data in the training data set and the first model. The architecture, input and output of the initial second model are consistent with those of the second model, the scale of the initial second model is smaller than that of the first model, and the training data set includes multiple training data, each of which includes a training user request and a real inference result corresponding to the training user request.

[0123] The process of training the initial second model using the training data in the training data set and the first model may include: obtaining training data from the training data set; inputting the training user request in the training data into the first model and the second model for inference, respectively, to obtain the inference result output by the first model and the inference result output by the second model; determining the first loss (i.e., distillation loss) that characterizes the inference difference between the second model and the first model based on the inference result output by the first model and the inference result output by the second model, and determining the second loss based on the inference result output by the second model and the real inference result corresponding to the training user request; and updating the parameters of the second model based on the first loss and the second loss. The second model is iteratively trained for multiple times according to the above process until the training end condition is met (such as reaching the set number of training iterations, model convergence, etc.), and the trained second model is the second model obtained by model distillation of the first model.

[0124] Step S402: Send the second model to the terminal so that the terminal deploys the second model locally and uses the local second model to infer the user request.

[0125] After obtaining a second model that is smaller in scale than the first model and has an effect close to the first model, the second model is sent to the terminal. After receiving the second model, the terminal deploys the second model locally, and when a user request is obtained, the local second model is used to infer the user request.

[0126] Step S403: receiving the inference-related data sent by the terminal, and optimizing the parameters of the shadow model using the inference-related data and the first model.

[0127] After the terminal completes the inference for the user request, the inference-related data can be sent to the cloud server. After receiving the inference-related data fed back by the terminal, the cloud server uses the inference-related data and the first model to optimize the parameters of the shadow model. The inference-related data at least includes the user request.

[0128] In one possible implementation, the process of optimizing the parameters of the shadow model using the inference-related data and the first model may include: determining the inference differences between the second model or the shadow model and the first model for user requests; and optimizing the parameters of the shadow model based on the inference differences.

[0129] Step S404: sending the optimized parameters of the shadow model to the terminal, so that the terminal optimizes the parameters of the local second model according to the optimized parameters of the shadow model.

[0130] After optimizing the shadow model, the cloud server sends the optimized parameters of the shadow model to the terminal, and the terminal optimizes the parameters of the local second model according to the optimized parameters of the shadow model.

[0131] The reasoning method provided in the embodiment of the present application first obtains the second model by performing model distillation on the first model, obtains the shadow model by copying the second model, and then sends the second model to the terminal so that the terminal can deploy the second model locally and use the local second model to reason about the user request. Then, when receiving the reasoning related data sent by the terminal, the shadow model is optimized by using the reasoning related data and the first model, and finally the optimized parameters of the shadow model are sent to the terminal so that the terminal can optimize the parameters of the local second model according to the optimized parameters of the shadow model. The reasoning method provided in the embodiment of the present application, by performing model distillation on the large-scale first model to obtain a small-scale second model and providing it to the terminal, enables the terminal to deploy the small-scale second model with better effect locally, and then use the local second model to reason about the user request, thereby significantly improving the reasoning speed. In addition, the reasoning method provided in the embodiment of the present application can also optimize the parameters of the second model on the terminal by optimizing the parameters of the shadow model. By optimizing the second model on the terminal, the reasoning effect on the terminal side can be improved.

[0132] Correspondingly, if you look at the technical solution of this application from the terminal side, please refer to Figure 5 , is a flowchart of a reasoning method provided in an embodiment of the present application, and the method may include:

[0133] Step S501: Receive the second model sent by the cloud server, and deploy the second model locally.

[0134] Among them, the second model is a model obtained by distilling the large-scale first model of the cloud server, and the second model is a small-scale model whose effect is close to the large-scale first model.

[0135] Step S502: When a user request is received, the user request is inferred using the local second model to obtain an inference result of the user request.

[0136] When a user request is received, the user request is input into the local second model for inference to obtain an inference result output by the second model.

[0137] Step S503: Send the inference-related data to the cloud server so that the cloud server can use the inference-related data and the first model to optimize the parameters of the shadow model.

[0138] The shadow model is obtained by copying the second model from the cloud server.

[0139] After the terminal completes the reasoning for the user's request, it can send the reasoning-related data to the cloud server. After the cloud server receives the reasoning-related data fed back by the terminal, it uses the reasoning-related data and the first model to optimize the parameters of the shadow model, and sends the optimized parameters of the shadow model to the terminal.

[0140] Step S504: receiving the optimized parameters of the shadow model sent by the cloud server, and optimizing the parameters of the local second model according to the optimized parameters of the shadow model.

[0141] After receiving the optimized parameters of the shadow model, the terminal optimizes the parameters of the local second model according to the optimized parameters of the shadow model. Through steps S503 to S504, the terminal can realize self-learning of the local second model, thereby improving the reasoning effect of the local second model.

[0142] The inference method provided in the embodiment of the present application can obtain a small-scale second model that is close to the effect of the large-scale first model of the cloud server, and then the second model can be deployed locally. When a user request is received, the local second model can be directly used to infer the user request. Since the terminal uses the local small-scale second model to infer the user request, the inference speed is faster. In addition, the inference method provided in the embodiment of the present application can also feed back the inference-related data to the cloud server, so that the cloud server can use the inference-related data, supplemented by the first model, to optimize the parameters of the shadow model. Then, after obtaining the optimized parameters of the shadow model sent by the cloud server, the parameters of the local second model can be optimized according to the optimized parameters of the shadow model, thereby improving the performance of the local second model.

[0143] Based on the above embodiments, the technical solution of the present application is described in detail below through the interaction between the cloud server and the terminal. Figure 6 A flow chart showing a method of implementing inference and model optimization by interaction between a cloud server and a terminal may include:

[0144] Step S601: The cloud server obtains a second model by performing model distillation on the first model, and obtains a shadow model by copying the second model.

[0145] The cloud server distills the large-scale first model to obtain a small-scale second model with an effect close to that of the large-scale first model. After obtaining the second model, the second model is copied and the copied model is used as a shadow model.

[0146] The implementation process of the cloud server obtaining the second model by performing model distillation on the first model can be found in the relevant parts of the above embodiment, which will not be described in detail in this embodiment.

[0147] Step S602: The cloud server sends the second model to the terminal.

[0148] like Figure 7 As shown, after obtaining a small-scale second model whose effect is close to that of the large-scale first model, the cloud server copies the second model, retains the copied model as a shadow model of the second model on this side, and pushes the second model to the terminal.

[0149] Step S603: The terminal receives the second model and deploys the received second model locally.

[0150] After receiving the second model from the cloud server, the terminal deploys the second model locally for processing user requests.

[0151] Step S604: When the terminal receives the user request, the terminal uses the local second model to infer the user request to obtain an inference result of the user request.

[0152] like Figure 7 As shown, when the application on the terminal receives a user request, the user request is input into the local second model for inference, and the inference result output by the second model is fed back to the application for output or processing.

[0153] Since the terminal locally deploys a small-scale second model that is close to the effect of the large-scale first model on the cloud side, the terminal can directly use the locally deployed small-scale second model to infer the received user requests.

[0154] The terminal uses the locally deployed small-scale second model to infer the user request and can provide the inference result quickly.

[0155] Step S605: The terminal sends the inference-related data to the cloud server.

[0156] The reasoning-related data in this embodiment may include: a user request, a first reasoning result, and first probability information of each word in the first reasoning result. The first reasoning result is the reasoning result obtained by the terminal using the local second model to reason about the user request, and the first probability information of each word in the first reasoning result is obtained in the process of obtaining the first reasoning result.

[0157] It should be noted that the first probability information of any word in the first inference result may be the first probability corresponding to the word. <eos> ”， " <eos>" indicates the end of the sequence. The first probabilities corresponding to each word in the first inference result are 0.91, 0.93, 0.98, 0.99, 0.99 respectively. That is, the first probability corresponding to "Ke" is 0.91, the first probability corresponding to "D" is 0.93, the first probability corresponding to "Xing" is 0.98, the first probability corresponding to "H" is 0.99, " <eos>The first probability corresponding to "..." is 0.99.

[0158] In addition to being the first probability corresponding to the word, the first probability information of any word in the first inference result can also be the probability distribution of the first candidate word corresponding to the word. The probability distribution of the first candidate word corresponding to any word in the first inference result is the candidate word probability distribution obtained by the second model in the process of predicting the word. The first probability corresponding to any word in the first inference result is obtained from the probability distribution of the first candidate word corresponding to the word.

[0159] Step S606: The cloud server uses the first model to obtain the second probability information of each word in the first inference result according to the user request and the first inference result.

[0160] The second probability information of any word in the first inference result can be the second probability corresponding to the word or the probability distribution of the second candidate word corresponding to the word.

[0161] It should be noted that if the first probability information of each word in the first inference result is the first probability corresponding to the word, the second probability information of each word in the first inference result is the second probability corresponding to the word. Similarly, if the first probability information of each word in the first inference result is the probability distribution of the first candidate word corresponding to the word, the second probability information of each word in the first inference result is the probability distribution of the second candidate word corresponding to the word.

[0162] In a possible implementation, the process of using the first model to obtain the probability distribution of the second candidate word corresponding to each word in the first inference result according to the user request and the first inference result includes: at the t-th prediction step, using the first model, based on the user request and the first t words in the first inference result, predicting the candidate word probability distribution of the (t + 1)-th word (at the t-th prediction step, the probabilities corresponding to each candidate word, that is, the probabilities that the (t + 1)-th word is each candidate word), and the predicted candidate word probability distribution of the (t + 1)-th word is used as the probability distribution of the second candidate word corresponding to the (t + 1)-th word in the first inference result. Wherein, t ranges from 0 to N - 1, and N is the length of the first inference result.

[0163] Exemplarily, the user request is "The artificial intelligence company that... is", and the first inference result is "Ke D Xing H" <eos>": At the 0th prediction step, using the first model and based on the user request, predict the probability distribution of candidate words for the 1st word (input "… artificial intelligence company is" into the first model to predict the next word). The predicted probability distribution of candidate words for the 1st word is used as the second candidate word probability distribution corresponding to the 1st word "Ke" in the first inference result. At the 1st prediction step, using the first model and based on the user request and the 1st word "Ke" in the first inference result, predict the probability distribution of candidate words for the 2nd word (input "… artificial intelligence company is Ke" into the first model to predict the next word). The predicted probability distribution of candidate words for the 2nd word is used as the second candidate word probability distribution corresponding to the 2nd word "D" in the first inference result. At the 2nd prediction step, using the first model and based on the user request and the 1st word "Ke" and the 2nd word "D" in the first inference result, predict the probability distribution of candidate words for the 3rd word (input "… artificial intelligence company is Ke D" into the first model to predict the next word). The predicted probability distribution of candidate words for the 3rd word is used as the second candidate word probability distribution corresponding to the 3rd word "Xing" in the first inference result. And so on for subsequent prediction steps.

[0164] In a possible implementation, the process of using the first model to obtain the second probability corresponding to each word in the first inference result based on the user request and the first inference result includes: at the tth prediction step, using the first model and based on the user request and the first t words in the first inference result, predict the probability distribution of candidate words for the (t + 1)th word, and obtain the probability corresponding to the (t + 1)th word in the first inference result from the probability distribution of candidate words for the (t + 1)th word as the second probability corresponding to the (t + 1)th word in the first inference result. Where t ranges from 0 to N - 1, and N is the length of the first inference result.

[0165] Exemplarily, the user request is "… artificial intelligence company is", and the first inference result is "Ke D Xing H <eos>": At the 0th prediction step, using the first model and based on the user request, predict the probability distribution of candidate words for the 1st word (input "… artificial intelligence company is" into the first model to predict the next word), obtain the probability corresponding to "Ke" in the first inference result from the probability distribution of candidate words for the 1st word, and use it as the second probability corresponding to the 1st word "Ke" in the first inference result; at the 1st prediction step, using the first model and based on the user request and the 1st word "Ke" in the first inference result, predict the probability distribution of candidate words for the 2nd word (input "… artificial intelligence company is Ke" into the first model to predict the next word), obtain the probability corresponding to the 2nd word "D" in the first inference result from the probability distribution of candidate words for the 2nd word, and use it as the second probability corresponding to the 2nd word "D" in the first inference result; at the 2nd prediction step, using the first model and based on the user request and the 1st word "Ke" and the 2nd word "D" in the first inference result, predict the probability distribution of candidate words for the 3rd word (input "… artificial intelligence company is Ke D" into the first model to predict the next word), obtain the probability corresponding to the 3rd word "Xing" in the first inference result from the probability distribution of candidate words for the 3rd word (assuming the probability distribution of candidate words for the 3rd word is {"Xun": 0.99, "Xing": 0.61,...}, then obtain the probability corresponding to "Xing" from this candidate word probability distribution), and use it as the second probability corresponding to the 3rd word "Xing" in the first inference result. The subsequent prediction steps are carried out in the same way.

[0166] Step S607: The cloud server determines the prediction loss of the second model according to the first probability information and the second probability information of each word in the first inference result.

[0167] Among them, the prediction loss of the second model can represent the inference difference between the second model and the first model for the user request.

[0168] Specifically, the process of determining the prediction loss of the second model according to the first probability information and the second probability information of each word in the first inference result may include:

[0169] Step S6071: For each word C in the first inference result i , determine the difference between the first probability information of word C i and the second probability information of word C i , obtain a probability information difference characterization value, and use it as the prediction loss of the second model on word C i .

[0170] For word C i , the first probability information is the first probability corresponding to word C i , and the second probability information of word C i is the second probability corresponding to word C i Corresponding to the second probability case, the second model is obtained in word C i The process of predicting the loss on word C may include: calculating word C i The first probability corresponding to word C i The probability difference representation value of the corresponding second probability is used as the second model in word C i The prediction loss on .

[0171] If the word C i The corresponding first probability is denoted as p i1 , the word C i The corresponding second probability is denoted as p i2 In one possible implementation, C can be calculated i The corresponding first probability p i1 With word C i The corresponding second probability p i2 The square of the difference is used as the second model in word C i The prediction loss L on Ci ,Right now:

[0172] L Ci =(p i1 -p i2 ) 2 (1)

[0173] It should be noted that this embodiment is not limited to using the above formula (1) to calculate the second model in the word C i The prediction loss L on Ci , for example, we can also calculate C i The corresponding first probability p i1 With word C i The corresponding second probability p i2 The absolute value of the difference is used as the second model in word C i The prediction loss L on Ci .

[0174] In word C i The first probability information is word C i The corresponding probability distribution of the first candidate word, and word C i The second probability information is word C i Under the corresponding probability distribution of the second candidate word, the second model is obtained in word C i There are many ways to implement the predicted loss, and this embodiment provides the following two implementation methods.

[0175] Get the second model in word C i The first implementation of the prediction loss on: For each candidate word, from word C i The first probability corresponding to the candidate word is obtained from the probability distribution of the first candidate word corresponding to the word, and the second probability corresponding to the candidate word is obtained from the probability distribution of the second candidate word corresponding to the word, and the difference between the first probability corresponding to the candidate word and the second probability corresponding to the candidate word is determined (for example, the square of the difference between the two probabilities is calculated), and the probability difference representation value corresponding to the candidate word is obtained; the probability difference representation values ​​corresponding to each candidate word are merged (for example, the probability difference representation values ​​corresponding to each candidate word are averaged), and the fusion result is used as the word C i The corresponding prediction loss.

[0176] Get the second model in word C i The second implementation of the prediction loss on: according to the word C i The corresponding first candidate word probability distribution, according to the preset screening strategy, select some candidate words from each candidate word, and select the candidate words according to the word C i The corresponding second candidate word probability is obtained by filtering some candidate words from each candidate word according to the preset filtering strategy, merging and deduplicating all the filtered candidate words, and forming a candidate word set S by the candidate words obtained after merging and deduplicating. i ; For the candidate word set S i Each candidate word in C i The first probability corresponding to the candidate word is obtained from the probability distribution of the corresponding first candidate word, and the first probability corresponding to the candidate word is obtained from the word C i The second probability corresponding to the candidate word is obtained from the corresponding second candidate word probability distribution, and the difference between the first probability corresponding to the candidate word and the second probability corresponding to the candidate word is determined (for example, the square of the difference between the two probabilities is calculated), and the probability difference representation value corresponding to the candidate word is obtained; the candidate word set S i The probability difference representation values ​​corresponding to each candidate word in are fused (for example, the probability difference representation values ​​corresponding to each candidate word are averaged), and the fusion result is used as the word C i The corresponding prediction loss.

[0177] In a possible implementation, according to word C i The corresponding first candidate word probability distribution, according to the preset screening strategy, the process of screening some candidate words from each candidate word may include: i The probabilities corresponding to the candidate words in the corresponding first candidate word probability distribution are sorted in descending order, and the candidate words corresponding to the first K probabilities are used as the selected candidate words (of course, the word C can also be i The probabilities corresponding to the candidate words in the corresponding first candidate word probability distribution are sorted in ascending order, and the candidate words corresponding to the last K probabilities are used as the selected candidate words), where the specific value of K can be determined according to the actual application scenario. i The corresponding second candidate word probability distribution, according to the preset screening strategy, the process of screening some candidate words from each candidate word is similar, and this embodiment will not be described in detail here.

[0178] Exemplarily, the candidate words are word A, word B, word C, and word D. The probability corresponding to word A is 0.2, the probability corresponding to word B is 0.6, the probability corresponding to word C is 0.15, and the probability corresponding to word D is 0.05. The four probabilities can be sorted in order from large to small, and after sorting, they are 0.6, 0.2, 0.15, and 0.05. Assuming K is 2, the words corresponding to 0.6 and 0.2 respectively (i.e., word A and word B) are selected as candidate words.

[0179] In another possible implementation, according to word C i The corresponding first candidate word probability distribution, according to the preset screening strategy, the process of screening some candidate words from each candidate word may include: i The probabilities corresponding to the candidate words in the corresponding probability distribution of the first candidate word are sorted in descending order, and are accumulated one by one from the first probability until the accumulated probability is greater than or equal to the preset probability. The candidate words corresponding to the first probability to the last probability involved in the accumulation are taken as the selected candidate words (of course, the word C can also be i The probabilities corresponding to the candidate words in the corresponding probability distribution of the first candidate word are sorted in ascending order, and are accumulated one by one from the last probability forward in order until the accumulated probability is greater than or equal to the preset probability, and the candidate words corresponding to the last probability ~ the last probability involved in the accumulation are taken as the selected candidate words). According to word C i The corresponding second candidate word probability distribution, according to the preset screening strategy, the process of screening some candidate words from each candidate word is similar, and this embodiment will not be described in detail here.

[0180] Exemplarily, the candidate words are word A, word B, word C, and word D. The probability corresponding to word A is 0.2, the probability corresponding to word B is 0.6, the probability corresponding to word C is 0.15, and the probability corresponding to word D is 0.05. The four probabilities can be sorted in order from large to small, and after sorting, they are 0.6, 0.2, 0.15, and 0.05. Assuming the preset probability is 0.9, accumulation starts from 0.6, 0.6 plus 0.2 is 0.8, which is less than the preset probability of 0.9, and accumulation continues. 0.8 plus 0.15 is 0.95, which is greater than the preset probability of 0.9, and accumulation ends. The words corresponding to the probabilities of 0.6, 0.2, and 0.15 respectively (i.e., word A, word B, and word C) participating in the accumulation are selected as candidate words.

[0181] Step S6072: The prediction loss of the second model on each word of the first inference result is integrated to obtain the final prediction loss of the second model.

[0182] In one possible implementation, the prediction losses of the second model on each word of the first inference result may be integrated by averaging, that is, the prediction losses of the second model on each word of the first inference result are averaged, and the obtained mean is used as the final prediction loss of the second model.

[0183] It should be noted that this embodiment is not limited to the method of averaging to fuse the prediction losses of the second model on each word of the first reasoning result. For example, the prediction losses of the second model on each word of the first reasoning result can also be fused by directly summing.

[0184] Step S608: The cloud server optimizes parameters of the shadow model according to the predicted loss of the second model.

[0185] The cloud server can update the parameters of the shadow model using the gradient descent method according to the prediction loss of the second model.

[0186] Step S609: The cloud server sends the optimized parameters of the shadow model to the terminal.

[0187] After optimizing the parameters of the shadow model, the cloud server sends the optimized parameters of the shadow model to the terminal.

[0188] In a possible implementation, the cloud server may send the optimized parameters of the shadow model to the terminal each time after optimizing the parameters of the shadow model.

[0189] Taking into account the limited resources of the terminal, in order to minimize the impact on the terminal, in another possible implementation method, the cloud server can determine whether the preset parameter return conditions are met after each parameter optimization of the shadow model. If the parameter return conditions are met, the optimized parameters of the shadow model will be sent to the terminal.

[0190] Among them, the parameter return condition can be that the time interval between the last parameter sending to the terminal reaches a preset time interval, and the parameter return condition can also be that the number of parameter optimizations of the shadow model after the last parameter sending to the terminal reaches a preset number of parameter optimizations.

[0191] It should be noted that when the parameter return conditions are met, the cloud server will send the last optimized parameters to the terminal.

[0192] Step S610: the terminal receives the optimized parameters of the shadow model, and optimizes the parameters of the local second model according to the optimized parameters of the shadow model.

[0193] The terminal optimizes the parameters of the local second model according to the optimized parameters of the shadow model, thereby improving the reasoning effect of the local second model.

[0194] The inference method provided in the embodiment of the present application is that the cloud server obtains a second model by performing model distillation on the first model and provides it to the terminal, so that the terminal can deploy a small-scale second model with better effect locally, and then use the local small-scale second model to infer user requests, thereby significantly improving the inference speed. In addition, the inference method provided in the embodiment of the present application can also optimize the parameters of the shadow model on the cloud side according to the reasoning difference between the second model on the terminal side and the first model on the cloud side for user requests, thereby realizing parameter optimization of the second model on the terminal side. By optimizing the second model on the terminal side, the inference effect on the terminal side can be improved.

[0195] See also Figure 8 , Figure 8 A flow chart showing another method of implementing inference and model optimization by interaction between a cloud server and a terminal may include:

[0196] Step S801: The cloud server obtains a second model by performing model distillation on the first model, and obtains a shadow model by copying the second model.

[0197] Step S802: The cloud server sends the second model to the terminal.

[0198] Step S803: The terminal receives the second model and deploys the received second model locally.

[0199] Step S804: When the terminal receives a user request, the terminal uses the local second model to infer the user request.

[0200] For the related introduction of steps S801 to S804 , reference may be made to the related introduction of steps S601 to S604 in the above embodiment, which will not be described in detail in this embodiment.

[0201] Step S805: The terminal sends the inference-related data to the cloud server.

[0202] The inference-related data in this embodiment includes user requests.

[0203] Step S806: The cloud server uses the shadow model to infer the user request to obtain a second inference result of the user request and first probability information of each word in the second inference result.

[0204] The cloud server inputs the user request into the shadow model for reasoning to obtain a second reasoning result of the user request and first probability information of each word in the second reasoning result.

[0205] The first probability information of any word in the second reasoning result can be the first probability corresponding to the word in the second reasoning result, or it can be the first candidate word probability distribution corresponding to the word in the second reasoning result. The first candidate word probability distribution corresponding to any word in the second reasoning result is the candidate word probability distribution obtained by the shadow model in the process of predicting the word. The first probability corresponding to any word in the second reasoning result is obtained from the first candidate word probability distribution corresponding to the word.

[0206] Step S807: The cloud server uses the first model to obtain second probability information of each word in the second reasoning result according to the user request and the second reasoning result.

[0207] In this embodiment, the implementation process of "the cloud server uses the first model to obtain the second probability information of each word in the second reasoning result according to the user request and the second reasoning result" is similar to the implementation process of "step S606: the cloud server uses the first model to obtain the second probability information of each word in the first reasoning result according to the user request and the first reasoning result" in the above embodiment (replacing the "first reasoning result" in the implementation process of step S606 with "second reasoning result" is the implementation process of step S807), which is not repeated in this embodiment.

[0208] Step S808: The cloud server determines the prediction loss of the shadow model according to the first probability information and the second probability information of each word in the second inference result.

[0209] Among them, the prediction loss of the shadow model can represent the difference in reasoning between the shadow model and the first model for user requests.

[0210] In this embodiment, the implementation process of "the cloud server determines the prediction loss of the shadow model according to the first probability information and the second probability information of each word in the second reasoning result" is similar to the implementation process of "step S607: the cloud server determines the prediction loss of the second model according to the first probability information and the second probability information of each word in the first reasoning result" in the above embodiment (replacing the "first reasoning result" with the "second reasoning result" and replacing the "second model" with the "shadow model" in the implementation process of step S607 is the implementation process of step S808), which is not elaborated in this embodiment.

[0211] Step S809: The cloud server optimizes the parameters of the shadow model according to the predicted loss of the shadow model.

[0212] The cloud server can update the parameters of the shadow model using the gradient descent method according to the prediction loss of the shadow model.

[0213] Step S810: The cloud server sends the optimized parameters of the shadow model to the terminal.

[0214] Taking into account the limited resources of the terminal, in order to minimize the impact on the terminal, preferably, the cloud server can determine whether the preset parameter return conditions are met after each parameter optimization of the shadow model (for example, the time interval between the last time the parameters were sent to the terminal reaches the preset time interval, or the number of shadow model parameter optimizations since the last time the parameters were sent to the terminal reaches the preset number of parameter optimizations). If the parameter return conditions are met, the optimized parameters of the shadow model are sent to the terminal.

[0215] Step S811: the terminal receives the optimized parameters of the shadow model, and optimizes the parameters of the locally deployed second model according to the optimized parameters of the shadow model.

[0216] The terminal optimizes the parameters of the local second model according to the optimized parameters of the shadow model, thereby improving the reasoning effect of the local second model.

[0217] The inference method provided in the embodiment of the present application is that the cloud server obtains a second model by performing model distillation on the first model and provides it to the terminal, so that the terminal can deploy a small-scale second model with better inference effect locally, and then use the local second model to infer user requests, thereby significantly improving the inference speed. In addition, the inference method provided in the embodiment of the present application can also optimize the parameters of the shadow model on the cloud side according to the inference difference between the shadow model on the cloud side and the first model on the cloud side for user requests, thereby realizing parameter optimization of the second model on the terminal side. By optimizing the second model on the terminal side, the inference effect on the terminal side can be improved.

[0218] See also Fig. 9 , Fig. 9 A flow chart showing another method of implementing inference and model optimization by interaction between a cloud server and a terminal may include:

[0219] Step S901: The cloud server obtains a second model by performing model distillation on the first model, and obtains a shadow model by copying the second model.

[0220] Step S902: The cloud server sends the second model to the terminal.

[0221] Step S903: The terminal receives the second model and deploys the received second model locally.

[0222] Step S904: When the terminal receives a user request, the terminal uses the local second model to infer the user request.

[0223] For the related introduction of step S901 to step S904, reference may be made to the related introduction of step S601 to step S604 in the above embodiment, which will not be described in detail in this embodiment.

[0224] Step S905: The terminal sends the inference-related data to the cloud server.

[0225] The inference-related data in this embodiment includes user requests.

[0226] Step S906: The cloud server uses the first model to infer the user request to obtain a third inference result of the user request and first probability information of each word in the third inference result.

[0227] The cloud server inputs the user request into the first model for inference to obtain a third inference result requested by the user and first probability information of each word in the third inference result.

[0228] The first probability information of any word in the third reasoning result can be the first probability corresponding to the word in the third reasoning result, or it can be the first candidate word probability distribution corresponding to the word in the third reasoning result. The first candidate word probability distribution corresponding to any word in the third reasoning result is the candidate word probability distribution obtained by the first model in the process of predicting the word. The first probability corresponding to any word in the third reasoning result is obtained from the first candidate word probability distribution corresponding to the word.

[0229] Step S907: The cloud server uses the shadow model to obtain second probability information of each word in the third reasoning result according to the user request and the third reasoning result.

[0230] The second probability information of any word in the third reasoning result may be the second probability corresponding to the word in the third reasoning result, or may be the second candidate word probability distribution corresponding to the word in the third reasoning result.

[0231] It should be noted that if the first probability information of each word in the third reasoning result is the first probability corresponding to the word, then the second probability information of each word in the third reasoning result is the second probability corresponding to the word. Similarly, if the first probability information of each word in the third reasoning result is the first candidate word probability distribution corresponding to the word, then the second probability information of each word in the third reasoning result is the second candidate word probability distribution corresponding to the word.

[0232] In a possible implementation, the process of obtaining the probability distribution of the second candidate word corresponding to each word in the third reasoning result by using the shadow model according to the user request and the third reasoning result includes: in the t-th prediction step, using the shadow model, based on the user request and the first t words in the third reasoning result, predicting the probability distribution of the candidate word of the t+1-th word, and using the predicted probability distribution of the candidate word of the t+1-th word as the probability distribution of the second candidate word corresponding to the t+1-th word in the third reasoning result. Wherein, t ranges from 0 to M-1, and M is the length of the third reasoning result.

[0233] In a possible implementation, the process of obtaining the second probability corresponding to each word in the third reasoning result by using the shadow model according to the user request and the third reasoning result includes: in the t-th prediction step, using the first model, based on the user request and the first t words in the third reasoning result, predicting the candidate word probability distribution of the t+1-th word, obtaining the probability corresponding to the t+1-th word in the third reasoning result from the candidate word probability distribution of the t+1-th word as the second probability corresponding to the t+1-th word in the third reasoning result. Wherein, t ranges from 0 to M-1, and M is the length of the third reasoning result.

[0234] Step S908: The cloud server determines the prediction loss of the shadow model according to the first probability information and the second probability information of each word in the third inference result.

[0235] Among them, the prediction loss of the shadow model can represent the difference in reasoning between the shadow model and the first model for user requests.

[0236] In this embodiment, the implementation process of "the cloud server determines the prediction loss of the shadow model according to the first probability information and the second probability information of each word in the third reasoning result" is similar to the implementation process of "step S607: the cloud server determines the prediction loss of the second model according to the first probability information and the second probability information of each word in the first reasoning result" in the above embodiment (replacing the "first reasoning result" with the "third reasoning result" and replacing the "second model" with the "shadow model" in the implementation process of step S607 is the implementation process of step S908), which is not elaborated in this embodiment.

[0237] Step S909: The cloud server optimizes the parameters of the shadow model according to the predicted loss of the shadow model.

[0238] The cloud server can update the parameters of the shadow model using the gradient descent method according to the prediction loss of the shadow model.

[0239] Step S910: The cloud server sends the optimized parameters of the shadow model to the terminal.

[0240] Taking into account the limited resources of the terminal, in order to minimize the impact on the terminal, preferably, the cloud server can determine whether the preset parameter return conditions are met after each parameter optimization of the shadow model (for example, the time interval between the last time the parameters were sent to the terminal reaches the preset time interval, or the number of shadow model parameter optimizations since the last time the parameters were sent to the terminal reaches the preset number of parameter optimizations). If the parameter return conditions are met, the optimized parameters of the shadow model are sent to the terminal.

[0241] Step S911: the terminal receives the optimized parameters of the shadow model, and optimizes the parameters of the local second model according to the optimized parameters of the shadow model.

[0242] The terminal optimizes the parameters of the local second model according to the optimized parameters of the shadow model, thereby improving the reasoning effect of the local second model.

[0243] The inference method provided in the embodiment of the present application is that the cloud server obtains a second model by performing model distillation on the first model and provides it to the terminal, so that the terminal can deploy a small-scale second model with better inference effect locally, and then use the local second model to infer user requests, thereby significantly improving the inference speed. In addition, the inference method provided in the embodiment of the present application can also optimize the parameters of the shadow model on the cloud side according to the inference difference between the shadow model on the cloud side and the first model on the cloud side for user requests, thereby realizing parameter optimization of the second model on the terminal side. By optimizing the second model on the terminal side, the inference effect on the terminal side can be improved.

[0244] The present application also provides a device corresponding to the reasoning method provided in the above embodiment. Fig.10 , Fig.10 A structural schematic diagram of an inference device provided in an embodiment of the present application, the inference device is applied to a cloud server, and the inference device may include: a model distillation module 1001, a shadow model acquisition module 1002, a model sending module 1003, an inference-related data receiving module 1004, a model optimization module 1005 and a parameter sending module 1006.

[0245] The model distillation module 1001 is used to obtain a second model by performing model distillation on the first model.

[0246] The shadow model acquisition module 1002 is used to obtain a shadow model by copying the second model.

[0247] The model sending module 1003 is used to send the second model to the terminal so that the terminal deploys the second model locally and uses the local second model to infer the user request.

[0248] The reasoning-related data receiving module 1004 is used to receive the reasoning-related data sent by the terminal.

[0249] The model optimization module 1005 is used to optimize the parameters of the shadow model by using the inference-related data and the first model.

[0250] The parameter sending module 1006 is used to send the optimized parameters of the shadow model to the terminal, so that the terminal optimizes the parameters of the local second model according to the optimized parameters of the shadow model.

[0251] In one possible implementation, the inference-related data includes a user request.

[0252] When the model optimization module 1005 optimizes the parameters of the shadow model by using the inference-related data and the first model, it is specifically used to:

[0253] determining a difference in reasoning between the second model or the shadow model and the first model for the user request;

[0254] According to the inference difference, the parameters of the shadow model are optimized.

[0255] In a possible implementation, the reasoning-related data also includes: a first reasoning result obtained by the terminal using a local second model to reason about the user request, and first probability information of each word in the first reasoning result.

[0256] When determining the reasoning difference between the second model and the first model for the user request, the model optimization module 1005 is specifically configured to include:

[0257] Using the first model, according to the user request and the first inference result, obtain second probability information of each word in the first inference result;

[0258] The prediction loss of the second model is determined according to the first probability information and the second probability information of each word in the first reasoning result, wherein the prediction loss of the second model can characterize the reasoning difference between the second model and the first model for the user request.

[0259] In a possible implementation, when the model optimization module 1005 determines the prediction loss of the second model according to the first probability information and the second probability information of each word in the first reasoning result, it is specifically used to:

[0260] For each word in the first inference result, determine the difference between the first probability information of the word and the second probability information of the word, and obtain a probability information difference representation value as the prediction loss of the second model on the word;

[0261] The prediction loss of the second model on each word of the first inference result is fused to obtain the final prediction loss of the second model.

[0262] In a possible implementation, when determining the reasoning difference between the shadow model and the first model for the user request, the model optimization module 1005 is specifically configured to:

[0263] Reasoning the user request using the shadow model to obtain a second reasoning result of the user request and first probability information of each word in the second reasoning result;

[0264] Using the first model, according to the user request and the second inference result, obtaining second probability information of each word in the second inference result;

[0265] The prediction loss of the shadow model is determined according to the first probability information and the second probability information of each word in the second reasoning result, wherein the prediction loss of the shadow model can characterize the reasoning difference between the shadow model and the first model for the user request.

[0266] In one possible implementation, the first probability information of any word in an inference result is the first probability or the first candidate word probability distribution corresponding to the word in the inference result, the first candidate word probability distribution corresponding to any word in an inference result is the candidate word probability distribution obtained in the process of predicting the word, and the first probability corresponding to any word in an inference result is obtained from the first candidate word probability distribution corresponding to the word.

[0267] When the model optimization module 1005 uses the first model to obtain the second probability information of each word in the inference result according to the user request and an inference result of the user request, it is specifically used to:

[0268] At the tth prediction step:

[0269] Using the first model, based on the user request and the first t words in the inference result, predict the probability distribution of candidate words for the t+1th word, and use the probability distribution of candidate words for the t+1th word as the probability distribution of the second candidate words corresponding to the t+1th word in the inference result;

[0270] Alternatively, using the first model, based on the user request and the first t words in the inference result, predicting the probability distribution of candidate words for the t+1th word, and obtaining the second probability corresponding to the t+1th word in the inference result from the probability distribution of candidate words for the t+1th word;

[0271] The second probability or the second candidate word probability distribution corresponding to the t+1th word in the inference result is used as the second probability information of the t+1th word in the inference result; wherein t ranges from 0 to N-1, and N is the length of the inference result.

[0272] In a possible implementation, when determining the reasoning difference between the shadow model and the first model for the user request, the model optimization module 1005 is specifically configured to:

[0273] Using the first model, reasoning the user request to obtain a third reasoning result of the user request and first probability information of each word in the third reasoning result;

[0274] Using the shadow model, according to the user request and the third inference result, obtain second probability information of each word in the third inference result;

[0275] The prediction loss of the shadow model is determined according to the first probability information and the second probability information of each word in the third reasoning result, wherein the prediction loss of the shadow model can characterize the reasoning difference between the shadow model and the first model for the user request.

[0276] In one possible implementation, the first probability information of any word in the third reasoning result is the first probability or the first candidate word probability distribution corresponding to the word in the third reasoning result, the first candidate word probability distribution corresponding to any word in the third reasoning result is the candidate word probability distribution obtained in the process of predicting the word, and the first probability corresponding to any word in the third reasoning result is obtained from the first candidate word probability distribution corresponding to the word.

[0277] When the model optimization module 1005 uses the shadow model to obtain the second probability information of each word in the third reasoning result according to the user request and the third reasoning result, it is specifically used to:

[0278] At the tth prediction step:

[0279] Using the shadow model, based on the user request and the first t words in the third reasoning result, predict the probability distribution of candidate words for the t+1th word, and use the probability distribution of candidate words for the t+1th word as the probability distribution of the second candidate word corresponding to the t+1th word in the third reasoning result;

[0280] Alternatively, using the shadow model, based on the user request and the first t words in the third reasoning result, predicting the probability distribution of candidate words for the t+1th word, obtaining the probability corresponding to the t+1th word in the third reasoning result from the probability distribution of candidate words for the t+1th word, and using the probability corresponding to the t+1th word as the second probability corresponding to the t+1th word in the third reasoning result;

[0281] The second probability or the second candidate word probability distribution corresponding to the t+1th word in the third reasoning result is used as the second probability information of the t+1th word in the third reasoning result; wherein t ranges from 0 to M-1, and M is the length of the third reasoning result.

[0282] In a possible implementation, the reasoning device further includes: a parameter feedback determination module.

[0283] The parameter return determination module is used to determine whether the preset parameter return conditions are met, wherein the parameter return conditions are that the time interval between the last time the parameters were sent to the terminal reaches the preset time interval, or the number of parameter optimizations of the shadow model after the last time the parameters were sent to the terminal reaches the preset number of parameter optimizations.

[0284] The parameter sending module 1006 is specifically configured to send the optimized parameters of the shadow model to the terminal when the parameter return condition is met.

[0285] The inference device provided in the embodiment of the present application performs model distillation on the large-scale first model to obtain a small-scale second model and provides it to the terminal, so that the terminal can deploy the small-scale second model with better effect locally, and then use the local second model to infer user requests, thereby significantly improving the inference speed. In addition, the inference device provided in the embodiment of the present application can also optimize the parameters of the second model on the terminal by optimizing the parameters of the shadow model. By optimizing the second model on the terminal, the inference effect on the terminal side can be improved.

[0286] See also Fig.11 , Fig.11 A structural schematic diagram of another inference device provided in an embodiment of the present application, wherein the inference device is applied to a terminal, and the inference device may include: a model receiving module 1101, a model deployment module 1102, an inference module 1103, an inference-related data sending module 1104, a parameter receiving module 1105 and a model optimization module 1106.

[0287] The model receiving module 1101 is used to receive the second model sent by the cloud server.

[0288] The model deployment module 1102 is used to deploy the second model locally, wherein the second model is obtained by performing model distillation on the first model of the cloud server.

[0289] The reasoning module 1103 is used to, when receiving a user request, use the local second model to reason about the user request to obtain the reasoning result of the user request.

[0290] The inference-related data sending module 1104 is used to send the inference-related data to the cloud server so that the cloud server can use the inference-related data and the first model to optimize the parameters of the shadow model. The shadow model is obtained by the cloud server by copying the second model.

[0291] The parameter receiving module 1105 is used to receive the optimized parameters of the shadow model sent by the cloud server.

[0292] The model optimization module 1106 is used to optimize the parameters of the local second model according to the optimized parameters of the shadow model.

[0293] In one possible implementation, the inference-related data includes a user request.

[0294] In another possible implementation, the reasoning-related data includes a user request, a reasoning result, and probability information of each word in the reasoning result.

[0295] The inference device provided in the embodiment of the present application can obtain a small-scale second model that is close to the effect of the large-scale first model of the cloud server, and then the second model can be deployed locally. When a user request is received, the local second model can be directly used to infer the user request. Since the terminal uses the local small-scale second model to infer the user request, the inference speed is faster. In addition, the inference device provided in the embodiment of the present application can also feed back the inference-related data to the cloud server, so that the cloud server can use the inference-related data, supplemented by the first model, to optimize the parameters of the shadow model. Then, after obtaining the optimized parameters of the shadow model sent by the cloud server, the parameters of the local second model can be optimized according to the optimized parameters of the shadow model, thereby improving the performance of the local second model.

[0296] The embodiment of the present application further provides a cloud server, the cloud server comprising at least one processor and a memory connected to the processor, wherein:

[0297] The memory is used to store computer programs;

[0298] The processor is used to execute the computer program so that the cloud server can implement the steps of the reasoning method applied to the cloud server provided in the above embodiment.

[0299] The embodiment of the present application further provides a terminal, which includes at least one processor and a memory connected to the processor, wherein:

[0300] The memory is used to store computer programs;

[0301] The processor is used to execute the computer program so that the terminal can implement the steps of the reasoning method applied to the terminal provided in the above embodiment.

[0302] An embodiment of the present application also provides a computer storage medium, which carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement the steps of the reasoning method provided in the above embodiment.

[0303] An embodiment of the present application also provides a computer program product, including computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements the steps of the reasoning method provided in the above embodiment.

[0304] An embodiment of the present application also provides an inference system, which includes a terminal and a cloud server, and a first model is deployed on the cloud server.

[0305] The cloud server is used to obtain a second model by performing model distillation on the first model, obtain a shadow model by copying the second model, and send the second model to the terminal.

[0306] The terminal is used to receive the second model and deploy the second model locally, and when receiving a user request, use the local second model to infer the user request and send the inference-related data to the cloud server.

[0307] The cloud server is also used to receive the inference-related data, and use the inference-related data, supplemented by the first model, to optimize the parameters of the shadow model, and send the optimized parameters of the shadow model to the terminal.

[0308] The terminal is also used to receive the optimized parameters of the shadow model and optimize the parameters of the local second model according to the optimized parameters of the shadow model.

[0309] The more specific implementation process of the cloud server and the terminal interacting to realize reasoning and model optimization can be found in the above-mentioned reasoning method embodiment, which will not be described in detail in this embodiment.

[0310] It should also be noted that the device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed over multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the drawings of the device embodiments provided by the present application, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines.

[0311] Through the description of the above implementation mode, the technicians in the field can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course, it can also be implemented by special hardware including special integrated circuits, special CPUs, special memories, special components, etc. In general, all functions completed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be various, such as analog circuits, digital circuits or special circuits. However, for the present application, software program implementation is a better implementation mode in more cases. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer floppy disk, a U disk, a mobile hard disk, a ROM, a RAM, a disk or an optical disk, etc., including a number of instructions to enable a computer device (which can be a personal computer, a training device, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0312] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0313] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, a computer, a training device, or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, training device, or data center. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device, a data center, etc. that includes one or more available media integrations. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)), etc.< / eos> < / eos> < / eos> < / eos> < / eos>

Claims

1. A reasoning method, characterized in that: Applied to a cloud server, a first model is deployed on the cloud server, and the reasoning method includes: Obtain a second model by performing model distillation on the first model, and obtain a shadow model by copying the second model; Sending the second model to the terminal so that the terminal locally deploys the second model and uses the local second model to reason about the user request; receiving the inference-related data sent by the terminal, and optimizing parameters of the shadow model using the inference-related data and the first model; The optimized parameters of the shadow model are sent to the terminal, so that the terminal optimizes the parameters of the local second model according to the optimized parameters of the shadow model.

2. The inference method according to claim 1, characterized in that: The inference-related data includes: the user request; The using the inference-related data, aided by the first model, to optimize parameters of the shadow model includes: determining a difference in reasoning between the second model or the shadow model and the first model for the user request; Parameters of the shadow model are optimized according to the inference difference.

3. The inference method according to claim 2, characterized in that: The inference-related data also includes: a first inference result obtained by the terminal using the local second model to infer the user request, and first probability information of each word in the first inference result; Determining a difference in reasoning between the second model and the first model for the user request includes: Using the first model, according to the user request and the first inference result, obtaining second probability information of each word in the first inference result; The prediction loss of the second model is determined according to the first probability information and the second probability information of each word in the first inference result, wherein the prediction loss of the second model can characterize the difference in inference between the second model and the first model for the user request.

4. The inference method according to claim 3, characterized in that: The determining the prediction loss of the second model according to the first probability information and the second probability information of each word in the first inference result includes: For each word in the first inference result, determine the difference between the first probability information of the word and the second probability information of the word, and obtain a probability information difference representation value as the prediction loss of the second model on the word; The prediction losses of the second model on each word of the first inference result are fused to obtain the final prediction loss of the second model.

5. The inference method according to claim 2, characterized in that: Determining a difference in reasoning between the shadow model and the first model for the user request includes: Inferring the user request using the shadow model to obtain a second inference result of the user request and first probability information of each word in the second inference result; Using the first model, according to the user request and the second inference result, obtaining second probability information of each word in the second inference result; The prediction loss of the shadow model is determined according to the first probability information and the second probability information of each word in the second inference result, wherein the prediction loss of the shadow model can characterize the difference in inference between the shadow model and the first model for the user request.

6. The inference method according to claim 3 or 5, characterized in that: The first probability information of any word in an inference result is the first probability or the first candidate word probability distribution corresponding to the word in the inference result. The first candidate word probability distribution corresponding to any word in an inference result is the candidate word probability distribution obtained in the process of predicting the word. The first probability corresponding to any word in an inference result is obtained from the first candidate word probability distribution corresponding to the word. Using the first model, according to the user request and an inference result of the user request, obtaining second probability information of each word in the inference result, including: At the tth prediction step: Using the first model, based on the user request and the first t words in the inference result, predict the candidate word probability distribution of the t+1th word, and use the candidate word probability distribution of the t+1th word as the second candidate word probability distribution corresponding to the t+1th word in the inference result; Alternatively, using the first model, based on the user request and the first t words in the inference result, predicting the probability distribution of candidate words for the t+1th word, and obtaining a second probability corresponding to the t+1th word in the inference result from the probability distribution of candidate words for the t+1th word; The second probability or the second candidate word probability distribution corresponding to the t+1th word in the inference result is used as the second probability information of the t+1th word in the inference result; wherein t ranges from 0 to N-1, and N is the length of the inference result.

7. The inference method according to claim 2, characterized in that: Determining a difference in reasoning between the shadow model and the first model for the user request includes: Using the first model, reasoning the user request to obtain a third reasoning result of the user request and first probability information of each word in the third reasoning result; Using the shadow model, according to the user request and the third reasoning result, obtaining second probability information of each word in the third reasoning result; The prediction loss of the shadow model is determined according to the first probability information and the second probability information of each word in the third reasoning result, wherein the prediction loss of the shadow model can characterize the reasoning difference between the shadow model and the first model for the user request.

8. The inference method according to claim 7, characterized in that: The first probability information of any word in the third reasoning result is the first probability or the first candidate word probability distribution corresponding to the word in the third reasoning result. The first candidate word probability distribution corresponding to any word in the third reasoning result is the candidate word probability distribution obtained in the process of predicting the word. The first probability corresponding to any word in the third reasoning result is obtained from the first candidate word probability distribution corresponding to the word. The using the shadow model, according to the user request and the third reasoning result, to obtain second probability information of each word in the third reasoning result, includes: At the tth prediction step: Using the shadow model, based on the user request and the first t words in the third reasoning result, predict the candidate word probability distribution of the t+1th word, and use the candidate word probability distribution of the t+1th word as the second candidate word probability distribution corresponding to the t+1th word in the third reasoning result; Alternatively, using the shadow model, based on the user request and the first t words in the third reasoning result, predicting the probability distribution of candidate words for the t+1th word, obtaining the probability corresponding to the t+1th word in the third reasoning result from the probability distribution of candidate words for the t+1th word, and using the probability corresponding to the t+1th word as the second probability corresponding to the t+1th word in the third reasoning result; The second probability or the second candidate word probability distribution corresponding to the t+1th word in the third inference result is used as the second probability information of the t+1th word in the third inference result; wherein t ranges from 0 to M-1, and M is the length of the third inference result.

9. The inference method according to claim 1, characterized in that: Before sending the optimized parameters of the shadow model to the terminal, the method further includes: Determine whether a preset parameter return condition is met, wherein the parameter return condition is that the time interval between the last parameter sending to the terminal reaches a preset time interval, or the number of parameter optimizations of the shadow model after the last parameter sending to the terminal reaches a preset number of parameter optimizations; If the parameter return condition is met, the optimized parameters of the shadow model are sent to the terminal.

10. A reasoning method, characterized in that Applied to a terminal, the inference method includes: Receiving a second model sent by the cloud server, and deploying the second model locally, wherein the second model is a model obtained by performing model distillation on the first model of the cloud server; When receiving a user request, using the local second model to infer the user request to obtain an inference result of the user request; Sending the inference-related data to a cloud server so that the cloud server uses the inference-related data and the first model to optimize parameters of a shadow model, wherein the shadow model is obtained by the cloud server by copying the second model; Receive the optimized parameters of the shadow model sent by the cloud server, and optimize the parameters of the local second model according to the optimized parameters of the shadow model.

11. The inference method according to claim 10, characterized in that: The inference-related data includes the user request; Alternatively, the inference-related data includes the user request, the inference result, and probability information of each word in the inference result.

12. An inference device, characterized in that: Applied to a cloud server, a first model is deployed on the cloud server, and the reasoning device includes: a model distillation module, a shadow model acquisition module, a model sending module, a reasoning related data receiving module, a model optimization module and a parameter sending module; The model distillation module is used to obtain a second model by performing model distillation on the first model; The shadow model acquisition module is used to obtain the shadow model by copying the second model; The model sending module is used to send the second model to the terminal, so that the terminal deploys the second model locally and uses the local second model to infer the user request; The inference-related data receiving module is used to receive the inference-related data sent by the terminal; The model optimization module is used to optimize the parameters of the shadow model by using the reasoning related data and the first model; The parameter sending module is used to send the optimized parameters of the shadow model to the terminal, so that the terminal optimizes the parameters of the local second model according to the optimized parameters of the shadow model.

13. An inference device, characterized in that: Applied to a terminal, the inference device comprises: a model receiving module, a model deployment module, an inference module, an inference-related data sending module, a parameter receiving module and a model optimization module; The model receiving module is used to receive the second model sent by the cloud server; The model deployment module is used to deploy the second model locally, wherein the second model is a model obtained by performing model distillation on the first model of the cloud server; The reasoning module is used to, when receiving a user request, use the local second model to reason about the user request to obtain a reasoning result of the user request; The inference-related data sending module is used to send the inference-related data to the cloud server, so that the cloud server uses the inference-related data and the first model to optimize the parameters of the shadow model, wherein the shadow model is obtained by the cloud server by copying the second model; The parameter receiving module is used to receive the optimized parameters of the shadow model sent by the cloud server; The model optimization module is used to optimize the parameters of the local second model according to the optimized parameters of the shadow model.

14. A cloud server, characterized in that: The method comprises at least one processor and a memory connected to the processor, wherein: The memory is used to store computer programs; The processor is used to execute the computer program so that the cloud server can implement the steps of the reasoning method as described in any one of claims 1 to 9.

15. A terminal, characterized in that: The method comprises at least one processor and a memory connected to the processor, wherein: The memory is used to store computer programs; The processor is used to execute the computer program so that the terminal can implement the steps of the reasoning method as described in any one of claims 10 to 11.

16. A computer storage medium, characterized in that: The storage medium carries one or more computer programs, and when the one or more computer programs are executed by an electronic device, the electronic device can implement the steps of the reasoning method as described in any one of claims 1 to 9, or implement the steps of the reasoning method as described in any one of claims 10 to 11.

17. A computer program product, characterized in that The method comprises computer-readable instructions, which, when executed on an electronic device, enable the electronic device to implement the steps of the reasoning method as described in any one of claims 1 to 9, or implement the steps of the reasoning method as described in any one of claims 10 to 11.

18. An inference system, characterized in that include: A terminal and a cloud server, wherein a first model is deployed on the cloud server; The cloud server is configured to obtain a second model by performing model distillation on the first model, obtain a shadow model by copying the second model, and send the second model to the terminal; The terminal is used to receive the second model, deploy the second model locally, and, when receiving a user request, use the local second model to infer the user request and send the inference-related data to the cloud server; The cloud server is further configured to receive the inference-related data, and optimize the parameters of the shadow model using the inference-related data in combination with the first model, and send the optimized parameters of the shadow model to the terminal; The terminal is further used to receive the optimized parameters of the shadow model, and optimize the parameters of the local second model according to the optimized parameters of the shadow model.

Images