A reasoning method, related device and reasoning system

Abstract

The application discloses a reasoning method, a related device and a reasoning system, and applies to the technical field of reasoning acceleration. The reasoning method comprises the following steps: a cloud server obtains a second model by performing model distillation on a first model, obtains a shadow model by copying the second model, and sends the second model to a terminal; the terminal locally deploys the second model, and uses the local second model to perform reasoning on a user request; the terminal sends reasoning-related data to the cloud server; the cloud server uses the reasoning-related data to assist the first model in performing parameter optimization on the shadow model, and sends the optimized parameters of the shadow model to the terminal; and the terminal performs parameter optimization on the local second model according to the optimized parameters of the shadow model. Since the reasoning process is completed by the local small model on the terminal side, the reasoning method disclosed by the application has high reasoning speed. In addition, the reasoning method provided by the application can optimize the model on the terminal side through the shadow model on the cloud side.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of inference acceleration, in particular to an inference method, related device and inference system. BACKGROUND

[0002] At present, most of the inference schemes are as follows: a terminal receives a user request, sends the user request to a cloud server, the cloud server receives the user request from the terminal, uses a deployed large model to perform inference on the user request, and after obtaining the inference result of the user request, sends the inference result of the user request to the terminal.

[0003] Since the large model of the cloud server has more parameters and a more complex structure, the inference speed is slow, and slow inference speed will affect user experience. SUMMARY

[0004] Therefore, the present application provides an inference method, related device and inference system to solve the problem of slow inference speed of the existing inference scheme, thereby affecting user experience. The technical solutions are as follows:

[0005] The first aspect of the present application provides an inference method applied to a cloud server, wherein a first model is deployed on the cloud server, and the inference method comprises:

[0006] obtaining a second model by model distillation on the first model, and obtaining a shadow model by copying the second model;

[0007] sending the second model to a terminal, so that the terminal deploys the second model locally, and uses the local second model to perform inference on a user request;

[0008] receiving inference-related data sent by the terminal, using the inference-related data, and assisting the first model to optimize parameters of the shadow model;

[0009] sending the optimized parameters of the shadow model to the terminal, so that the terminal optimizes parameters of the local second model according to the optimized parameters of the shadow model.

[0010] In a possible implementation manner, the inference-related data comprises the user request.

[0011] The use of the inference-related data, and the assistance of the first model to optimize the parameters of the shadow model comprises:

[0012] determining the inference difference of the second model or the shadow model and the first model for the user request;

[0013] According to the inference difference, the shadow model is optimized in parameters.

[0014] In a possible implementation, the inference-related data further includes: first inference results obtained by the terminal using the local second model to infer the user request, and first probability information of each word in the first inference results.

[0015] The inference difference of the second model and the first model for the user request is determined, including:

[0016] According to the user request and the first inference results, second probability information of each word in the first inference results is obtained by using the first model.

[0017] According to the first probability information and the second probability information of each word in the first inference results, a prediction loss of the second model is determined, wherein the prediction loss of the second model can represent the inference difference of the second model and the first model for the user request.

[0018] In a possible implementation, the prediction loss of the second model is determined according to the first probability information and the second probability information of each word in the first inference results, including:

[0019] For each word in the first inference results, a difference between the first probability information of the word and the second probability information of the word is determined to obtain a probability information difference representation value as the prediction loss of the second model on the word.

[0020] The prediction losses of the second model on each word in the first inference results are fused to obtain a final prediction loss of the second model.

[0021] In a possible implementation, the inference difference of the shadow model and the first model for the user request is determined, including:

[0022] The user request is inferred by using the shadow model to obtain second inference results of the user request and first probability information of each word in the second inference results.

[0023] According to the user request and the second inference results, second probability information of each word in the second inference results is obtained by using the first model.

[0024] According to the first probability information and the second probability information of each word in the second inference results, a prediction loss of the shadow model is determined, wherein the prediction loss of the shadow model can represent the inference difference of the shadow model and the first model for the user request.

[0025] In a possible implementation, the first probability information of any word in a reasoning result is a first probability of the word in the reasoning result or a first candidate word probability distribution of the word, the first candidate word probability distribution of any word in a reasoning result is a candidate word probability distribution obtained in the process of predicting the word, and the first probability of any word in a reasoning result is obtained from the first candidate word probability distribution of the word.

[0026] The first model is used to obtain second probability information of each word in the reasoning result according to the user request and the reasoning result of the user request, including:

[0027] In the t th prediction step:

[0028] The first model is used to predict a candidate word probability distribution of the (t+1) th word according to the user request and the first t words in the reasoning result, and the candidate word probability distribution of the (t+1) th word is used as the second candidate word probability distribution of the (t+1) th word in the reasoning result.

[0029] Alternatively, the first model is used to predict a candidate word probability distribution of the (t+1) th word according to the user request and the first t words in the reasoning result, and the second probability of the (t+1) th word in the reasoning result is obtained from the candidate word probability distribution of the (t+1) th word.

[0030] The second probability or the second candidate word probability distribution of the (t+1) th word in the reasoning result is used as the second probability information of the (t+1) th word in the reasoning result, where t is from 0 to N-1, and N is the length of the reasoning result.

[0031] In a possible implementation, determining the reasoning difference between the shadow model and the first model for the user request includes:

[0032] The first model is used to perform reasoning on the user request to obtain a third reasoning result of the user request and first probability information of each word in the third reasoning result.

[0033] The second probability information of each word in the third reasoning result is obtained according to the user request and the third reasoning result by using the shadow model.

[0034] The prediction loss of the shadow model is determined according to the first probability information and the second probability information of each word in the third reasoning result, where the prediction loss of the shadow model can represent the reasoning difference between the shadow model and the first model for the user request.

[0035] In a possible implementation, the first probability information of any word in the third reasoning result is a first probability corresponding to the word in the third reasoning result or a first candidate word probability distribution corresponding to the word in the third reasoning result, the first candidate word probability distribution corresponding to the word in the third reasoning result is a candidate word probability distribution obtained in a process of predicting the word, and the first probability corresponding to the word in the third reasoning result is obtained from the first candidate word probability distribution corresponding to the word.

[0036] The third reasoning result is obtained by using the shadow model and according to the user request and the second reasoning result.

[0037] In the tth prediction step:

[0038] The candidate word probability distribution of the t+1th word is predicted by using the shadow model and according to the user request and the first t words in the third reasoning result, and the candidate word probability distribution of the t+1th word is used as the second candidate word probability distribution corresponding to the t+1th word in the third reasoning result.

[0039] Alternatively, the candidate word probability distribution of the t+1th word is predicted by using the shadow model and according to the user request and the first t words in the third reasoning result, and the probability corresponding to the t+1th word is obtained from the candidate word probability distribution of the t+1th word, and the probability corresponding to the t+1th word is used as the second probability corresponding to the t+1th word in the third reasoning result.

[0040] The second probability or the second candidate word probability distribution corresponding to the t+1th word in the third reasoning result is used as the second probability information of the t+1th word in the third reasoning result, where t is from 0 to M-1, and M is the length of the third reasoning result.

[0041] In a possible implementation, before the optimized parameters of the shadow model are sent to the terminal, the method further includes:

[0042] It is determined whether a preset parameter return condition is met, where the parameter return condition is that a time interval from a time when the parameters are last sent to the terminal to a current time reaches a preset time interval, or a number of times of parameter optimization of the shadow model after the parameters are last sent to the terminal reaches a preset number of times of parameter optimization.

[0043] If the parameter return condition is met, the optimized parameters of the shadow model are sent to the terminal.

[0044] The second aspect of the application provides a reasoning method applied to a terminal, and the reasoning method includes:

[0045] receive a second model sent by the cloud server and deploy the second model locally, wherein the second model is a model obtained by model distillation on the first model of the cloud server;

[0046] when receiving a user request, infer the user request by using the second model locally to obtain an inference result of the user request;

[0047] send inference-related data to the cloud server, so that the cloud server performs parameter optimization on a shadow model by using the inference-related data and with the aid of the first model, wherein the shadow model is obtained by the cloud server by copying the second model;

[0048] receive the optimized parameters of the shadow model sent by the cloud server and perform parameter optimization on the second model locally according to the optimized parameters of the shadow model.

[0049] In a possible implementation, the inference-related data includes the user request; or the inference-related data includes the user request, the inference result, and probability information of each word in the inference result.

[0050] The third aspect of the present application provides an inference device applied to a cloud server, wherein the cloud server has a first model deployed thereon, and the inference device comprises a model distillation module, a shadow model acquisition module, a model sending module, an inference-related data receiving module, a model optimization module, and a parameter sending module.

[0051] The model distillation module is configured to obtain a second model by model distillation on the first model.

[0052] The shadow model acquisition module is configured to obtain a shadow model by copying the second model.

[0053] The model sending module is configured to send the second model to a terminal, so that the terminal deploys the second model locally and infers a user request by using the second model locally.

[0054] The inference-related data receiving module is configured to receive inference-related data sent by the terminal.

[0055] The model optimization module is configured to perform parameter optimization on the shadow model by using the inference-related data and with the aid of the first model.

[0056] The parameter sending module is configured to send the optimized parameters of the shadow model to the terminal, so that the terminal performs parameter optimization on the second model locally according to the optimized parameters of the shadow model.

[0057] The fourth aspect of the present application provides a reasoning device applied to a terminal, the reasoning device comprising a model receiving module, a model deploying module, a reasoning module, a reasoning related data sending module, a parameter receiving module and a model optimizing module;

[0058] The model receiving module is configured to receive a second model sent by a cloud server.

[0059] The model deploying module is configured to deploy the second model locally, wherein the second model is a model obtained by model distillation on a first model of the cloud server.

[0060] The reasoning module is configured to, when receiving a user request, perform reasoning on the user request by using the second model locally to obtain a reasoning result of the user request.

[0061] The reasoning related data sending module is configured to send reasoning related data to the cloud server, so that the cloud server performs parameter optimization on a shadow model by using the reasoning related data and the first model, wherein the shadow model is obtained by the cloud server by copying the second model.

[0062] The parameter receiving module is configured to receive the optimized parameters of the shadow model sent by the cloud server.

[0063] The model optimizing module is configured to perform parameter optimization on the second model locally according to the optimized parameters of the shadow model.

[0064] The fifth aspect of the present application provides a cloud server comprising at least one processor and a memory connected with the processor, wherein:

[0065] The memory is configured to store a computer program.

[0066] The processor is configured to execute the computer program, so that the cloud server can implement the steps of any one of the reasoning methods.

[0067] The sixth aspect of the present application provides a terminal comprising at least one processor and a memory connected with the processor, wherein:

[0068] The memory is configured to store a computer program.

[0069] The processor is configured to execute the computer program, so that the terminal can implement the steps of any one of the reasoning methods.

[0070] The seventh aspect of the present application provides a computer storage medium, the storage medium carrying one or more computer programs, when the one or more computer programs are executed by an electronic device, the electronic device can realize the steps of any one of the inference methods.

[0071] The eighth aspect of the present application provides a computer program product, comprising computer readable instructions, when the computer readable instructions run on an electronic device, the electronic device realizes the steps of any one of the inference methods.

[0072] The ninth aspect of the present application provides an inference system, comprising: a terminal and a cloud server, the cloud server is deployed with a first model;

[0073] The cloud server is configured to obtain a second model by model distillation on the first model, obtain a shadow model by copying the second model, and send the second model to the terminal;

[0074] The terminal is configured to receive the second model, deploy the second model locally, and when receiving a user request, use the local second model to infer the user request, and send inference related data to the cloud server;

[0075] The cloud server is further configured to receive the inference related data, use the inference related data, and assist the first model to optimize parameters of the shadow model, and send the optimized parameters of the shadow model to the terminal;

[0076] The terminal is further configured to receive the optimized parameters of the shadow model, and optimize the local second model according to the optimized parameters of the shadow model.

[0077] By the technical scheme, the inference method provided in the application comprises the following steps: a cloud server obtains a second model by model distillation on a first model, and obtains a shadow model by copying the second model; the cloud server sends the second model to a terminal; after receiving the second model, the terminal deploys the second model locally, and then uses the local second model to infer a user request; after the terminal completes the inference, the terminal sends inference-related data to the cloud server; after receiving the inference-related data, the cloud server uses the inference-related data to optimize the parameters of the shadow model in conjunction with the first model, and then sends the optimized parameters of the shadow model to the terminal; and after receiving the optimized parameters of the shadow model, the terminal optimizes the parameters of the local second model according to the received parameters. The inference method provided in the application enables the terminal to deploy a small-scale second model with better performance locally, so that the terminal can use the local second model to infer a user request, thereby significantly improving the inference speed. In addition, the inference method provided in the application can optimize the parameters of the second model on the terminal side by optimizing the parameters of the shadow model on the cloud side, thereby improving the inference effect on the terminal side. BRIEF DESCRIPTION OF DRAWINGS

[0078] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or the prior art description. Obviously, the drawings in the following description are only embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of the provided drawings.

[0079] Figure 1 A schematic diagram of a system architecture related to the present application;

[0080] Figure 2 A schematic diagram of a hardware structure of a terminal provided in an embodiment of the present application;

[0081] Figure 3 A schematic diagram of a hardware structure of a server provided in an embodiment of the present application;

[0082] Figure 4 A flowchart of an inference method applied to a cloud server provided in an embodiment of the present application;

[0083] Figure 5 A flowchart of an inference method applied to a terminal provided in an embodiment of the present application;

[0084] Figure 6 A flowchart of a method of interaction between a cloud server and a terminal to implement inference and model optimization provided in an embodiment of the present application;

[0085] Figure 7 A schematic diagram of cloud server interaction with terminal provided by an embodiment of the present application is shown in FIG. 1.

[0086] Figure 8 A schematic diagram of another way of implementing reasoning and model optimization by cloud server interaction with terminal provided by an embodiment of the present application is shown in FIG. 2.

[0087] Figure 9 A schematic diagram of still another way of implementing reasoning and model optimization by cloud server interaction with terminal provided by an embodiment of the present application is shown in FIG. 3.

[0088] Figure 10 A structural schematic diagram of a reasoning device applied to a cloud server provided by an embodiment of the present application is shown in FIG. 4.

[0089] Figure 11 A structural schematic diagram of a reasoning device applied to a terminal provided by an embodiment of the present application is shown in FIG. 5. DETAILED DESCRIPTION

[0090] The embodiments of the present application are described below in conjunction with the accompanying drawings. The terms used in the embodiment part of the present application are only used to explain the specific embodiments of the present application, and are not intended to limit the present application.

[0091] The embodiments of the present application are described below in conjunction with the accompanying drawings. Those skilled in the art can know that the technical solutions provided by the embodiments of the present application are also applicable to similar technical problems as technology develops and new scenarios appear.

[0092] The terms "first", "second", and the like in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily describe a specific order or sequence. It should be understood that the terms used in this way can be interchanged under appropriate circumstances, and this is only a way of distinguishing the objects with the same attributes in the description of the embodiments of the present application. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, so that the processes, methods, systems, products or devices containing a series of units do not have to be limited to those units, but can include other units not clearly listed or inherent to these processes, methods, products or devices.

[0093] The system architecture involved in the present application can include a terminal 101 and a cloud server 102, and the terminal 101 can interact with the cloud server 102 through a network (wired network or wireless network). The cloud server 102 can include one or more servers (one server is taken as an example for description in the embodiment). Figure 1

[0094] The product form of the terminal is described next.​

[0095] The terminal described above can be a mobile phone, a tablet computer, a wearable device, a vehicle-mounted device, a robot, an augmented reality (AR) / virtual reality (VR) device, a notebook computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), and the like, and embodiments of the present application do not limit the same.

[0096] Figure 2 An optional hardware structure diagram of the terminal is shown.

[0097] Reference Figure 2 As shown, the terminal can include a radio frequency unit 210, a memory 220, an input unit 230, a display unit 240, a camera 250 (optional), an audio circuit 260 (optional), a speaker 261 (optional), a microphone 262 (optional), a headphone jack 263 (optional), a processor 270, an external interface 280, a power supply 290, and the like. Those skilled in the art can understand that the terminal is not limited to the components shown in the figure, and can include more or fewer components, or combine certain components, or different components. Figure 2 The above is merely an example of the terminal and does not constitute a limitation on the terminal, and the terminal can include more or fewer components than those shown in the figure, or combine certain components, or different components.

[0098] The input unit 230 can be used to receive input digital or character information, and generate key signal input related to user settings and function control of the terminal. Specifically, the input unit 230 can include a touch screen 231 (optional) and / or other input devices 232. The touch screen 231 can collect user touch operations (such as user operations on or near the touch screen using a finger, a joint, a stylus, or any suitable object) and drive the corresponding connection device according to the pre-set program. The touch screen can detect the user's touch action on the touch screen, convert the touch action into a touch signal and send it to the processor 270, and can receive commands from the processor 270 and execute them; the touch signal at least includes touch point coordinate information. The touch screen 231 can provide an input interface and an output interface between the terminal and the user. In addition, the touch screen can be implemented in various types such as resistive, capacitive, infrared, and surface acoustic wave. In addition to the touch screen 231, the input unit 230 can also include other input devices. Specifically, the other input devices 232 can include one or more of a physical keyboard, function keys (such as volume control buttons, on-off buttons, etc.), trackballs, mice, joysticks, and the like.

[0099] The display unit 240 can be used to display information input by the user or information provided to the user, various menus of the terminal, interactive interfaces, file display and / or playback of any multimedia file.

[0100] The memory 220 can be used to store instructions and data. The memory 220 may primarily include an instruction storage area and a data storage area. The data storage area can store various types of data, such as multimedia files and text. The instruction storage area can store software units such as operating systems, applications, and instructions required for at least one function, or subsets or extended sets thereof. It may also include non-volatile random access memory. It provides the processor 270 with hardware, software, and data resources for managing the computing device, supporting control software and applications. It is also used for storing multimedia files, as well as storing running programs and applications.

[0101] The processor 270 is the control center of the terminal, connecting various parts of the terminal through various interfaces and lines. It executes instructions stored in the memory 220 and calls data stored in the memory 220 to perform various functions and process data, thereby controlling the terminal as a whole. Optionally, the processor 270 may include one or more processing units; preferably, the processor 270 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the modem processor may not be integrated into the processor 270. In some embodiments, the processor and memory can be implemented on a single chip; in some embodiments, they can also be implemented on separate chips. The processor 270 can also be used to generate corresponding operation control signals, send them to the corresponding components of the computing device, read and process data in the software, especially read and process data and programs in the memory 220, so that the various functional modules therein perform corresponding functions, thereby controlling the corresponding components to act according to the instructions.

[0102] The memory 220 can be used to store software code related to the reasoning method. The processor 270 can execute the software code in the memory 220, and can also schedule other units (such as the above-mentioned input unit 230 and display unit 240) to achieve the corresponding functions.

[0103] The RF unit 210 (optional) can be used to receive and send signals in the process of information or communication, for example, receiving the downlink information of the base station, and processing by the processor 270. In addition, the uplink data is sent to the base station. Generally, the RF unit 210 includes but is not limited to an antenna, at least one amplifier, a transceiver, a coupler, a low noise amplifier (LNA), a duplexer, etc. In addition, the RF unit 210 can also communicate with network devices and other devices through wireless communication. The wireless communication can use any communication standard or protocol, including but not limited to global system for mobile communication (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), long term evolution (LTE), email, short messaging service (SMS), etc.

[0104] In the embodiments of the present application, the RF unit 210 can send data to other devices, and can also receive data sent by other devices. It should be understood that the RF unit 210 is optional, which can be replaced by other communication interfaces, for example, a network interface.

[0105] The terminal also includes a power supply 290 (such as a battery) for supplying power to each component. Preferably, the power supply can be logically connected to the processor 270 through a power management system, so as to realize the functions of power management, discharge management, and power consumption management through the power management system.

[0106] The terminal also includes an external interface 280, which can be a standard Micro USB interface, or a multi-pin connector, and can be used to connect the terminal to other devices for communication, or to connect a charger to charge the terminal.

[0107] Although not shown, the terminal can also include a flash, a wireless fidelity (WiFi) module, a Bluetooth module, different function sensors, etc., which will not be described here.

[0108] Next, the product form of the cloud server described above is described.

[0109] Figure 3 A structural diagram of the cloud server described above is provided, as shown inFigure 3 As shown, the cloud server can include a bus 301, a processor 302, a communication interface 303, and a memory 304. The processor 302, the memory 304, and the communication interface 303 communicate through the bus 301.

[0110] The bus 301 can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 3 In the figure, only one thick line is used to represent, but it does not mean that there is only one bus or one type of bus.

[0111] The processor 302 can be any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP), etc.

[0112] The memory 304 can include a volatile memory, such as a random access memory (RAM). The memory 304 can also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a mechanical hard disk drive (HDD), or a solid state drive (SSD).

[0113] The memory 304 can be used to store software code related to the inference method, and the processor 302 can call the software code stored in the memory 304, or can schedule other units to implement the corresponding functions.

[0114] The processor in the terminal and the cloud server (for example, the processor 270 and the processor 302) can be a hardware circuit (such as an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a general-purpose processor, a digital signal processor (DSP), a microprocessor, or a microcontroller, etc.), or a combination of these hardware circuits. For example, the processor can be a hardware system with an instruction execution function, such as a CPU, a DSP, etc., or a hardware system without an instruction execution function, such as an ASIC, an FPGA, etc., or a combination of the hardware system without an instruction execution function and the hardware system with an instruction execution function.

[0115] Current reasoning schemes are mostly inferred by a large model on the cloud side for a user request, and the large model on the cloud side has more parameters and a more complex structure, so the reasoning speed is slow.

[0116] In view of the problem of slow reasoning speed in the current reasoning scheme, research is conducted. Initially, the idea is that when a user request is received, it is first determined whether the user request exists in the end-side cache. If the user request exists in the end-side cache, the reasoning result corresponding to the user request in the end-side cache is used as the reasoning result obtained by reasoning the user request. If the user request does not exist in the end-side cache, the user request is reasoned by using a large model on the cloud side, and the user request and the reasoning result corresponding thereto are stored in the end-side cache.

[0117] The above idea can quickly give a reasoning result for a user request existing in the end-side cache, but for a user request not existing in the end-side cache, the user request still needs to be reasoned by using a large model on the cloud side. Moreover, in actual business scenarios, the content of the user request is variable, and the actual cache hit probability is low, which means that the above idea does not effectively improve the reasoning speed.

[0118] Therefore, the present inventors continue to research, and finally propose a better reasoning method, which can significantly improve the reasoning speed and thus improve the user experience. Next, the reasoning method provided by the present application will be introduced through the following embodiments.

[0119] If the technical scheme of the present application is viewed from the side of the cloud server, please refer to Figure 4 A flowchart of a reasoning method provided by an embodiment of the present application is shown in the figure, which can include the following steps.

[0120] Step S401: obtaining the second model by model distillation on the first model, and obtaining the shadow model by copying the second model.

[0121] The first model is a large-scale model, the second model has a smaller scale than the first model, and the second model requires much less storage and computing resources than the first model. The effect of the second model is close to that of the first model, that is, the second model is a small-scale model with an effect close to that of the large-scale first model.

[0122] In a possible implementation, the process of obtaining the second model by model distillation on the first model can include: constructing an initial second model; training the initial second model with the training data in the training data set and with the assistance of the first model. The architecture, input and output of the initial second model are consistent with those of the second model. The initial second model has a smaller scale than the first model. The training data set includes a plurality of training data. Each piece of training data includes a training user request and a real inference result corresponding to the training user request.

[0123] The process of training the initial second model with the training data in the training data set and with the assistance of the first model can include: obtaining training data from the training data set; inputting the training user request in the training data into the first model and the second model respectively for inference to obtain the inference result output by the first model and the inference result output by the second model; determining a first loss (i.e., a distillation loss) representing the inference difference between the second model and the first model according to the inference result output by the first model and the inference result output by the second model, and determining a second loss according to the inference result output by the second model and the real inference result corresponding to the training user request; and updating the parameters of the second model according to the first loss and the second loss. The second model is iteratively trained multiple times according to the above process until a training end condition (such as reaching a set training iteration number, model convergence, etc.) is met. The trained second model is the second model obtained by model distillation on the first model.

[0124] Step S402: sending the second model to the terminal, so that the terminal deploys the second model locally and uses the local second model to infer the user request.

[0125] After obtaining the second model with a smaller scale than the first model and an effect close to that of the first model, the second model is sent to the terminal. After the terminal receives the second model, the terminal deploys the second model locally. When a user request is obtained, the terminal uses the local second model to infer the user request.

[0126] Step S403: receiving inference-related data sent by the terminal, and using the inference-related data to assist the first model in optimizing the parameters of the shadow model.

[0127] After the terminal completes the inference for the user request, the terminal can send inference-related data to the cloud server. After receiving the inference-related data fed back by the terminal, the cloud server performs parameter optimization on the shadow model by using the inference-related data and with the aid of the first model. The inference-related data at least includes the user request.

[0128] In a possible implementation, the process of performing parameter optimization on the shadow model by using the inference-related data and with the aid of the first model can include: determining the inference difference between the second model or the shadow model and the first model for the user request; and performing parameter optimization on the shadow model according to the inference difference.

[0129] Step S404: Send the optimized parameters of the shadow model to the terminal, so that the terminal performs parameter optimization on the local second model according to the optimized parameters of the shadow model.

[0130] After the cloud server optimizes the shadow model, the cloud server sends the optimized parameters of the shadow model to the terminal, and the terminal performs parameter optimization on the local second model according to the optimized parameters of the shadow model.

[0131] The inference method provided by the embodiment of the present application first obtains the second model by model distillation on the first model, and obtains the shadow model by copying the second model, then sends the second model to the terminal, so that the terminal deploys the second model locally and performs inference on the user request by using the local second model, then performs parameter optimization on the shadow model by using the inference-related data and with the aid of the first model when receiving the inference-related data sent by the terminal, and finally sends the optimized parameters of the shadow model to the terminal, so that the terminal performs parameter optimization on the local second model according to the optimized parameters of the shadow model. The inference method provided by the embodiment of the present application provides the small-scale second model obtained by model distillation on the large-scale first model to the terminal, so that the terminal can deploy the small-scale second model with better effect locally, and then perform inference on the user request by using the local second model, thereby significantly improving the inference speed. In addition, the inference method provided by the embodiment of the present application can also perform parameter optimization on the second model on the terminal by performing parameter optimization on the shadow model, and by optimizing the second model on the terminal, the inference effect on the terminal side can be improved.

[0132] Correspondingly, if the technical solution of the present application is viewed from the terminal side, please refer to Figure 5 A flowchart of an inference method provided by the embodiment of the present application is shown in FIG. 5. The method can include the following steps.

[0133] Step S501: Receive the second model sent by the cloud server, and deploy the second model locally.

[0134] The second model is a model obtained by model distillation on the large-scale first model of the cloud server, and the second model is a small-scale model with an effect close to that of the large-scale first model.

[0135] Step S502: When receiving the user request, the second local model is used to infer the user request to obtain an inference result of the user request.

[0136] When receiving the user request, the user request is input into the second local model for inference to obtain an inference result output by the second local model.

[0137] Step S503: The inference-related data is sent to the cloud server, so that the cloud server uses the inference-related data to assist the first model to optimize the parameters of the shadow model.

[0138] The shadow model is obtained by copying the second model by the cloud server.

[0139] After the terminal completes the inference on the user request, the inference-related data can be sent to the cloud server. After the cloud server receives the inference-related data fed back by the terminal, the inference-related data is used to assist the first model to optimize the parameters of the shadow model, and the optimized parameters of the shadow model are sent to the terminal.

[0140] Step S504: The optimized parameters of the shadow model sent by the cloud server are received, and the second local model is optimized according to the optimized parameters of the shadow model.

[0141] After the terminal receives the optimized parameters of the shadow model, the second local model is optimized according to the optimized parameters of the shadow model. Through steps S503-S504, the terminal can realize self-learning of the second local model, thereby improving the inference effect of the second local model.

[0142] The inference method provided by the embodiment of the application can obtain a small-scale second model with an effect close to that of the large-scale first model of the cloud server, and then the second model can be deployed locally. When receiving a user request, the second local model can be directly used to infer the user request. Since the terminal uses the small-scale second model locally to infer the user request, the inference speed is fast. In addition, the inference method provided by the embodiment of the application can feed back inference-related data to the cloud server, so that the cloud server uses the inference-related data to assist the first model to optimize the parameters of the shadow model. Then, after obtaining the optimized parameters of the shadow model sent by the cloud server, the second local model can be optimized according to the optimized parameters of the shadow model, thereby improving the performance of the second local model.

[0143] On the basis of the above embodiments, the technical solutions of the present application are described in detail below through the interaction between the cloud server and the terminal, Figure 6 A flowchart of one way of implementing reasoning and model optimization through the interaction between the cloud server and the terminal is shown, which can include:

[0144] Step S601: The cloud server obtains a second model by model distillation on the first model, and obtains a shadow model by copying the second model.

[0145] The cloud server distills the large-scale first model to obtain a small-scale second model close in effect to the large-scale first model. After obtaining the second model, the second model is copied, and the copied model is used as a shadow model.

[0146] The implementation process of the cloud server obtaining a second model by model distillation on the first model can be referred to the related part in the above embodiments, which will not be repeated here.

[0147] Step S602: The cloud server sends the second model to the terminal.

[0148] As shown in the figure, Figure 7 After obtaining a small-scale second model close in effect to the large-scale first model, the cloud server copies the second model, and the copied model is used as a shadow model of the second model and retained on the cloud side, and the second model is pushed to the terminal.

[0149] Step S603: The terminal receives the second model and deploys the received second model locally.

[0150] After receiving the second model from the cloud server, the terminal deploys the second model locally for processing user requests.

[0151] Step S604: When the terminal receives a user request, the local second model is used to reason the user request to obtain a reasoning result of the user request.

[0152] As shown in the figure, Figure 7 When the application on the terminal receives a user request, the user request is input into the local second model for reasoning, and the reasoning result output by the second model is fed back to the application for output or processing.

[0153] Since the terminal locally deploys a small-scale second model close in effect to the large-scale first model on the cloud side, the terminal can directly use the locally deployed small-scale second model to reason the received user request.

[0154] The terminal uses the locally deployed small-scale second model to reason the user request, which can quickly give a reasoning result.

[0155] Step S605: The terminal sends the inference-related data to the cloud server.

[0156] The inference-related data in this embodiment may include: a user request, a first inference result, and first probability information for each word in the first inference result. Among them, the first inference result is the inference result obtained by the terminal using the second model locally for the user request, and the first probability information for each word in the first inference result is obtained during the process of obtaining the first inference result.

[0157] It should be noted that the first probability information for any word in the first inference result may be the first probability corresponding to the word. Exemplarily, the first inference result is "Science D Star H" <eos> ”， " <eos>” indicates the end of the sequence. The first probabilities corresponding to each word in the first inference result are 0.91, 0.93, 0.98, 0.99, 0.99 respectively. That is, the first probability corresponding to "科" is 0.91, the first probability corresponding to "D" is 0.93, the first probability corresponding to "星" is 0.98, the first probability corresponding to "H" is 0.99, " <eos>The corresponding first probability is 0.99.

[0158] The first probability information of any word in the first inference result can be the first probability corresponding to the word, or a first candidate word probability distribution corresponding to the word. The first candidate word probability distribution corresponding to any word in the first inference result is a candidate word probability distribution obtained by the second model in the process of predicting the word. The first probability corresponding to any word in the first inference result is obtained from the first candidate word probability distribution corresponding to the word.

[0159] Step S606: The cloud server obtains the second probability information of each word in the first inference result according to the user request and the first inference result by using the first model.

[0160] The second probability information of any word in the first inference result can be the second probability corresponding to the word, or a second candidate word probability distribution corresponding to the word.

[0161] It should be noted that if the first probability information of each word in the first inference result is the first probability corresponding to the word, the second probability information of each word in the first inference result is the second probability corresponding to the word. Similarly, if the first probability information of each word in the first inference result is the first candidate word probability distribution corresponding to the word, the second probability information of each word in the first inference result is the second candidate word probability distribution corresponding to the word.

[0162] In a possible implementation, the process of obtaining the second candidate word probability distribution corresponding to each word in the first inference result according to the user request and the first inference result by using the first model includes: in the t th prediction step, using the first model to predict the candidate word probability distribution of the t+1 th word based on the user request and the first t words in the first inference result (in the t th prediction step, each candidate word corresponds to a probability, i.e., the probability of the t+1 th word for each candidate word), and the predicted candidate word probability distribution of the t+1 th word is taken as the second candidate word probability distribution corresponding to the t+1 th word in the first inference result. Wherein, t is from 0 to N-1, and N is the length of the first inference result.

[0163] For example, the user request is "… artificial intelligence company is", and the first inference result is "Ko D star H <eos>" in the first prediction step, the first model is used to predict the candidate word probability distribution of the first word based on the user request (the input of "… artificial intelligence company is" into the first model for the prediction of the next word), and the predicted candidate word probability distribution of the first word is taken as the second candidate word probability distribution corresponding to the first word "Ke" in the first reasoning result; in the first prediction step, the first model is used to predict the candidate word probability distribution of the second word based on the user request and the first word "Ke" in the first reasoning result (the input of "… artificial intelligence company is Ke" into the first model for the prediction of the next word), and the predicted candidate word probability distribution of the second word is taken as the second candidate word probability distribution corresponding to the second word "D" in the first reasoning result; in the second prediction step, the first model is used to predict the candidate word probability distribution of the third word based on the user request and the first word "Ke" and the second word "D" in the first reasoning result (the input of "… artificial intelligence company is Ke D" into the first model for the prediction of the next word), and the predicted candidate word probability distribution of the third word is taken as the second candidate word probability distribution corresponding to the third word "Xing" in the first reasoning result, and the subsequent prediction steps are similar.

[0164] In a possible implementation, the process of obtaining, by using the first model, the second probability corresponding to each word in the first reasoning result based on the user request and the first reasoning result includes: in the t-th prediction step, the first model is used to predict the candidate word probability distribution of the t+1-th word based on the user request and the first t words in the first reasoning result, and the probability corresponding to the t+1-th word in the first reasoning result is obtained from the candidate word probability distribution of the t+1-th word as the second probability corresponding to the t+1-th word in the first reasoning result. Wherein, t is from 0 to N-1, and N is the length of the first reasoning result.

[0165] For example, the user request is "… artificial intelligence company is", and the first reasoning result is "Ke D Xing H <eos>": in the 0th prediction step, the first model is used to predict the candidate word probability distribution of the 1st word based on the user request (input "… artificial intelligence company is" into the first model to predict the next word), and the probability corresponding to "ke" in the first reasoning result is obtained from the candidate word probability distribution of the 1st word as the second probability corresponding to the 1st word "ke" in the first reasoning result; in the 1st prediction step, the first model is used to predict the candidate word probability distribution of the 2nd word based on the user request and the 1st word "ke" in the first reasoning result (input "… artificial intelligence company is ke" into the first model to predict the next word), and the probability corresponding to "D" in the first reasoning result is obtained from the candidate word probability distribution of the 2nd word as the second probability corresponding to the 2nd word "D" in the first reasoning result; in the 2nd prediction step, the first model is used to predict the candidate word probability distribution of the 3rd word based on the user request and the 1st word "ke" and the 2nd word "D" in the first reasoning result (input "… artificial intelligence company is ke D" into the first model to predict the next word), and the probability corresponding to "xing" in the first reasoning result is obtained from the candidate word probability distribution of the 3rd word as the second probability corresponding to the 3rd word "xing" in the first reasoning result (assuming that the candidate word probability distribution of the 3rd word is {"xun": 0.99, "xing": 0.61,...}, then the probability corresponding to "xing" is obtained from the candidate word probability distribution), and subsequent prediction steps are similar.

[0166] Step S607: The cloud server determines the prediction loss of the second model according to the first probability information and the second probability information of each word in the first reasoning result.

[0167] The prediction loss of the second model can represent the reasoning difference between the second model and the first model for the user request.

[0168] Specifically, the process of determining the prediction loss of the second model according to the first probability information and the second probability information of each word in the first reasoning result can include:

[0169] Step S6071, for each word C i in the first reasoning result, determine the difference between the first probability information of the word C i and the second probability information of the word C i , obtain a probability information difference representation value as the prediction loss of the second model on the word C i .

[0170] The first probability information of the word C i is the first probability corresponding to the word C i , and the second probability information of the word C i is the second probability corresponding to the word C i In the case of the corresponding second probability, a process of obtaining a prediction loss of the second model on the word C i may include: calculating a prediction loss of the second model on the word C i . i The probability difference value of the corresponding first probability and the corresponding second probability is taken as the prediction loss of the second model on the word C i .

[0171] If the corresponding first probability of the word C i is denoted as p i1 , and the corresponding second probability of the word C i is denoted as p i2 , in a possible implementation, the square of the difference between the corresponding first probability p i of the word C i1 and the corresponding second probability p i of the word C i2 may be calculated as the prediction loss L i of the second model on the word C Ci , that is:

[0172] L Ci =(p i1 -p i2 ) 2 (1)

[0173] It should be noted that the embodiment does not limit to calculating the prediction loss L i of the second model on the word C Ci by using the above formula (1), for example, the absolute value of the difference between the corresponding first probability p i of the word C i1 and the corresponding second probability p i of the word C i2 may also be calculated as the prediction loss L i of the second model on the word C Ci .

[0174] In the case that the first probability information of the word C i is the first candidate word probability distribution corresponding to the word C i , and the second probability information of the word C i is the second candidate word probability distribution corresponding to the word C i , there are multiple implementation manners of obtaining the prediction loss of the second model on the word C i , and the following two implementation manners are provided in the embodiment.

[0175] The first implementation manner of obtaining the prediction loss of the second model on the word C i : for each candidate word, the prediction loss of the second model on the word C i obtaining the first probability corresponding to the candidate word from the first candidate word probability distribution corresponding to the word C and obtaining the second probability corresponding to the candidate word from the second candidate word probability distribution corresponding to the word C, determining the difference between the first probability corresponding to the candidate word and the second probability corresponding to the candidate word (for example, calculating the square of the difference between the two probabilities), obtaining the probability difference representation value corresponding to the candidate word; fusing the probability difference representation values corresponding to the respective candidate words (for example, averaging the probability difference representation values corresponding to the respective candidate words), and taking the fusion result as the predicted loss of the word C i corresponding to the word C.

[0176] obtaining the second probability corresponding to the candidate word from the second candidate word probability distribution corresponding to the word C i corresponding to the word C. i obtaining the first probability corresponding to the candidate word from the first candidate word probability distribution corresponding to the word C and obtaining the second probability corresponding to the candidate word from the second candidate word probability distribution corresponding to the word C, determining the difference between the first probability corresponding to the candidate word and the second probability corresponding to the candidate word (for example, calculating the square of the difference between the two probabilities), obtaining the probability difference representation value corresponding to the candidate word; fusing the probability difference representation values corresponding to the respective candidate words (for example, averaging the probability difference representation values corresponding to the respective candidate words), and taking the fusion result as the predicted loss of the word C i corresponding to the word C. i corresponding to the word C. i obtaining the first probability corresponding to the candidate word from the first candidate word probability distribution corresponding to the word C and obtaining the second probability corresponding to the candidate word from the second candidate word probability distribution corresponding to the word C, determining the difference between the first probability corresponding to the candidate word and the second probability corresponding to the candidate word (for example, calculating the square of the difference between the two probabilities), obtaining the probability difference representation value corresponding to the candidate word; fusing the probability difference representation values corresponding to the respective candidate words (for example, averaging the probability difference representation values corresponding to the respective candidate words), and taking the fusion result as the predicted loss of the word C i corresponding to the word C. i corresponding to the word C. i corresponding to the word C. i corresponding to the word C.

[0177] In a possible implementation manner, the process of screening part of the candidate words from the candidate words according to the first candidate word probability distribution corresponding to the word C can include: sorting the probabilities corresponding to the respective candidate words in the first candidate word probability distribution corresponding to the word C in descending order, and taking the candidate words corresponding to the first K probabilities as the screened candidate words (of course, the probabilities corresponding to the respective candidate words in the first candidate word probability distribution corresponding to the word C can also be sorted in ascending order, and the candidate words corresponding to the last K probabilities can be taken as the screened candidate words), where the specific value of K can be determined according to an actual application scenario. i corresponding to the word C. i corresponding to the word C. i corresponding to the word C. i The process of screening part of the candidate words from each candidate word according to the corresponding second candidate word probability distribution and a preset screening strategy is similar, and details are not described herein.

[0178] For example, the candidate words are word A, word B, word C and word D, the probability corresponding to word A is 0.2, the probability corresponding to word B is 0.6, the probability corresponding to word C is 0.15, and the probability corresponding to word D is 0.05. The four probabilities can be sorted in descending order, and the sorted order is 0.6, 0.2, 0.15 and 0.05. Assuming that K is 2, the words (i.e., word A and word B) corresponding to 0.6 and 0.2 are selected as the screened candidate words.

[0179] In another possible implementation, according to the word C i The process of screening part of the candidate words from each candidate word according to the corresponding first candidate word probability distribution and a preset screening strategy can include: the probability corresponding to word C i The probabilities corresponding to each candidate word in the corresponding first candidate word probability distribution are sorted in descending order, and the probabilities are sequentially accumulated from the first probability. The process is stopped until the accumulated probability is greater than or equal to the preset probability. The candidate words corresponding to the first probability and the last probability participating in the accumulation are selected as the screened candidate words (of course, the candidate words corresponding to the word C i The probabilities corresponding to each candidate word in the corresponding first candidate word probability distribution are sorted in descending order, and the probabilities are sequentially accumulated from the first probability. The process is stopped until the accumulated probability is greater than or equal to the preset probability. The candidate words corresponding to the first probability and the last probability participating in the accumulation are selected as the screened candidate words (of course, the candidate words corresponding to the word C i The process of screening part of the candidate words from each candidate word according to the corresponding second candidate word probability distribution and a preset screening strategy is similar, and details are not described herein.

[0180] For example, the candidate words are word A, word B, word C and word D, the probability corresponding to word A is 0.2, the probability corresponding to word B is 0.6, the probability corresponding to word C is 0.15, and the probability corresponding to word D is 0.05. The four probabilities can be sorted in descending order, and the sorted order is 0.6, 0.2, 0.15 and 0.05. Assuming that the preset probability is 0.9, the probabilities are sequentially accumulated from 0.6. 0.6 plus 0.2 is 0.8, which is less than the preset probability 0.9. The accumulation continues, 0.8 plus 0.15 is 0.95, which is greater than the preset probability 0.9. The accumulation is stopped. The words (i.e., word A, word B and word C) corresponding to the probabilities 0.6, 0.2 and 0.15 participating in the accumulation are selected as the screened candidate words.

[0181] In step S6072, the prediction loss of the second model on each word of the first inference result is fused to obtain the final prediction loss of the second model.

[0182] In a possible implementation, the prediction loss of the second model on each word of the first inference result can be fused by averaging, that is, the prediction loss of the second model on each word of the first inference result is averaged, and the average value obtained is taken as the final prediction loss of the second model.

[0183] It should be noted that the embodiment does not limit the fusion of the prediction loss of the second model on each word of the first inference result by averaging, for example, the prediction loss of the second model on each word of the first inference result can also be fused by directly summing.

[0184] Step S608: The cloud server optimizes the parameters of the shadow model according to the prediction loss of the second model.

[0185] The cloud server can update the parameters of the shadow model by using the gradient descent method according to the prediction loss of the second model.

[0186] Step S609: The cloud server sends the optimized parameters of the shadow model to the terminal.

[0187] After optimizing the parameters of the shadow model, the cloud server sends the optimized parameters of the shadow model to the terminal.

[0188] In a possible implementation, the cloud server can send the optimized parameters of the shadow model to the terminal after each optimization of the parameters of the shadow model.

[0189] In order to reduce the impact on the terminal as much as possible, considering the limited resources of the terminal, in another possible implementation, the cloud server can determine whether a preset parameter return condition is met after each optimization of the parameters of the shadow model, and if the parameter return condition is met, the cloud server sends the optimized parameters of the shadow model to the terminal.

[0190] The parameter return condition can be that the time interval from the last time the parameters are sent to the terminal reaches a preset time interval, or the parameter return condition can be that the number of parameter optimizations of the shadow model after the last time the parameters are sent to the terminal reaches a preset number of parameter optimizations.

[0191] It should be noted that when the parameter return condition is met, the cloud server sends the last optimized parameters to the terminal.

[0192] Step S610: The terminal receives the optimized parameters of the shadow model, and optimizes the parameters of the local second model according to the optimized parameters of the shadow model.

[0193] The terminal optimizes the parameters of the local second model according to the optimized parameters of the shadow model, which can improve the inference effect of the local second model.

[0194] The inference method provided in the embodiments of the present application enables the terminal to deploy the small-scale second model with better effect locally, and then the local small-scale second model can be used to infer the user request, thereby significantly improving the inference speed. In addition, the inference method provided in the embodiments of the present application can also optimize the parameters of the shadow model on the cloud side according to the inference difference of the second model on the terminal side and the first model on the cloud side for the user request, thereby realizing the parameter optimization of the second model on the terminal side. By optimizing the second model on the terminal side, the inference effect on the terminal side can be improved.

[0195] Please refer to Figure 8 , Figure 8 Another way of interaction between the cloud server and the terminal to realize inference and model optimization is shown in the flowchart, which can include the following steps.

[0196] Step S801: The cloud server obtains a second model by model distillation on a first model, and obtains a shadow model by copying the second model.

[0197] Step S802: The cloud server sends the second model to the terminal.

[0198] Step S803: The terminal receives the second model and deploys the received second model locally.

[0199] Step S804: When the terminal receives a user request, the second model locally is used to infer the user request.

[0200] The related introduction of steps S801-S804 can be referred to the related introduction of steps S601-S604 in the above embodiments, which will not be repeated here.

[0201] Step S805: The terminal sends inference-related data to the cloud server.

[0202] The inference-related data in the embodiments includes a user request.

[0203] Step S806: The cloud server infers the user request using the shadow model to obtain a second inference result of the user request and first probability information of each word in the second inference result.

[0204] The cloud server inputs the user request into the shadow model to infer, to obtain a second inference result of the user request and first probability information of each word in the second inference result.

[0205] The first probability information of any word in the second reasoning result can be the first probability corresponding to the word in the second reasoning result, or can be the first candidate word probability distribution corresponding to the word in the second reasoning result, the first candidate word probability distribution corresponding to any word in the second reasoning result being a candidate word probability distribution obtained by the shadow model in the process of predicting the word, and the first probability corresponding to any word in the second reasoning result being obtained from the first candidate word probability distribution corresponding to the word.

[0206] Step S807: The cloud server obtains, by using the first model, second probability information of each word in the second reasoning result according to the user request and the second reasoning result.

[0207] In this embodiment, the implementation process of "the cloud server obtains, by using the first model, second probability information of each word in the second reasoning result according to the user request and the second reasoning result" is similar to that of "step S606: the cloud server obtains, by using the first model, second probability information of each word in the first reasoning result according to the user request and the first reasoning result" in the above embodiment (the implementation process of step S807 is obtained by replacing "the first reasoning result" in the implementation process of step S606 with "the second reasoning result"), and thus will not be described here in detail.

[0208] Step S808: The cloud server determines the prediction loss of the shadow model according to the first probability information and the second probability information of each word in the second reasoning result.

[0209] The prediction loss of the shadow model can represent the difference between the shadow model and the first model in reasoning the user request.

[0210] In this embodiment, the implementation process of "the cloud server determines the prediction loss of the shadow model according to the first probability information and the second probability information of each word in the second reasoning result" is similar to that of "step S607: the cloud server determines the prediction loss of the second model according to the first probability information and the second probability information of each word in the first reasoning result" in the above embodiment (the implementation process of step S808 is obtained by replacing "the first reasoning result" in the implementation process of step S607 with "the second reasoning result" and replacing "the second model" with "the shadow model"), and thus will not be described here in detail.

[0211] Step S809: The cloud server performs parameter optimization on the shadow model according to the prediction loss of the shadow model.

[0212] The cloud server can update the parameters of the shadow model by using the gradient descent method according to the prediction loss of the shadow model.

[0213] Step S810: The cloud server sends the optimized parameters of the shadow model to the terminal.

[0214] In consideration of the limited resources of the terminal, in order to reduce the impact on the terminal as much as possible, preferably, the cloud server can determine whether a preset parameter return condition is met after each parameter optimization of the shadow model, and if the parameter return condition is met, the optimized parameters of the shadow model are sent to the terminal.

[0215] Step S811: The terminal receives the optimized parameters of the shadow model, and performs parameter optimization on the locally deployed second model according to the optimized parameters of the shadow model.

[0216] The terminal performs parameter optimization on the local second model according to the optimized parameters of the shadow model, which can improve the inference effect of the local second model.

[0217] The inference method provided by the embodiment of the application can provide the second model obtained by model distillation of the first model to the terminal, so that the terminal can deploy the small-scale second model with better inference effect locally, and then can use the local second model to infer the user request, thereby significantly improving the inference speed. In addition, the inference method provided by the embodiment of the application can also perform parameter optimization on the cloud-side shadow model according to the inference difference of the cloud-side shadow model and the cloud-side first model for the user request, thereby realizing parameter optimization of the terminal-side second model. By optimizing the terminal-side second model, the inference effect of the terminal side can be improved.

[0218] Please refer to Figure 9 , Figure 9 Another way of interaction between the cloud server and the terminal to realize inference and model optimization is shown in the flowchart, which can include:

[0219] Step S901: The cloud server obtains a second model by model distillation of a first model, and obtains a shadow model by copying the second model.

[0220] Step S902: The cloud server sends the second model to the terminal.

[0221] Step S903: The terminal receives the second model, and deploys the received second model locally.

[0222] Step S904: When the terminal receives a user request, the local second model is used to infer the user request.

[0223] The related introduction of steps S901 to S904 can be referred to the related introduction of steps S601 to S604 in the above embodiment, which will not be repeated here.

[0224] Step S905: The terminal sends the inference-related data to the cloud server.

[0225] The inference-related data in this embodiment includes the user request.

[0226] Step S906: The cloud server uses the first model to infer the user request to obtain a third inference result of the user request and first probability information of each word in the third inference result.

[0227] The cloud server inputs the user request into the first model for inference to obtain a third inference result of the user request and first probability information of each word in the third inference result.

[0228] The first probability information of any word in the third inference result can be a first probability corresponding to the word in the third inference result, or a first candidate word probability distribution corresponding to the word in the third inference result. The first candidate word probability distribution corresponding to any word in the third inference result is a candidate word probability distribution obtained by the first model in the process of predicting the word. The first probability corresponding to any word in the third inference result is obtained from the first candidate word probability distribution corresponding to the word.

[0229] Step S907: The cloud server uses the shadow model to obtain second probability information of each word in the third inference result according to the user request and the third inference result.

[0230] The second probability information of any word in the third inference result can be a second probability corresponding to the word in the third inference result, or a second candidate word probability distribution corresponding to the word in the third inference result.

[0231] It should be noted that if the first probability information of each word in the third inference result is the first probability corresponding to the word, the second probability information of each word in the third inference result is the second probability corresponding to the word. Similarly, if the first probability information of each word in the third inference result is the first candidate word probability distribution corresponding to the word, the second probability information of each word in the third inference result is the second candidate word probability distribution corresponding to the word.

[0232] In one possible implementation, the process of obtaining the second candidate word probability distribution corresponding to each word in the third inference result according to the user request and the third inference result using the shadow model includes: in the t th prediction step, using the shadow model to predict the candidate word probability distribution of the t+1 th word according to the user request and the first t words in the third inference result, and taking the predicted candidate word probability distribution of the t+1 th word as the second candidate word probability distribution corresponding to the t+1 th word in the third inference result. Wherein, t is from 0 to M-1, and M is the length of the third inference result.

[0233] In a possible implementation, the process of obtaining, according to the user request and the third reasoning result, the second probability corresponding to each word in the third reasoning result by using the shadow model includes: at the tth prediction step, using the first model, taking the user request and the first t words in the third reasoning result as the basis, predicting a candidate word probability distribution of the t+1th word, and obtaining, from the candidate word probability distribution of the t+1th word, a probability corresponding to the t+1th word in the third reasoning result as the second probability corresponding to the t+1th word in the third reasoning result. Wherein, t is from 0 to M-1, and M is the length of the third reasoning result.

[0234] Step S908: The cloud server determines the prediction loss of the shadow model according to the first probability information and the second probability information of each word in the third reasoning result.

[0235] The prediction loss of the shadow model can represent the reasoning difference between the shadow model and the first model for the user request.

[0236] In this embodiment, the implementation process of "the cloud server determines the prediction loss of the shadow model according to the first probability information and the second probability information of each word in the third reasoning result" is similar to the implementation process of "the cloud server determines the prediction loss of the second model according to the first probability information and the second probability information of each word in the first reasoning result" in the above embodiment (replacing "the first reasoning result" in the implementation process of step S607 with "the third reasoning result" and replacing "the second model" with "the shadow model" is the implementation process of step S908), and this embodiment will not be repeated here.

[0237] Step S909: The cloud server performs parameter optimization on the shadow model according to the prediction loss of the shadow model.

[0238] The cloud server can use the gradient descent method to update the parameters of the shadow model according to the prediction loss of the shadow model.

[0239] Step S910: The cloud server sends the optimized parameters of the shadow model to the terminal.

[0240] Considering the limited resources of the terminal, in order to reduce the impact on the terminal as much as possible, preferably, the cloud server can judge whether a preset parameter return condition is met after performing parameter optimization on the shadow model each time, and if the parameter return condition is met, the cloud server sends the optimized parameters of the shadow model to the terminal.

[0241] Step S911: The terminal receives the optimized parameters of the shadow model, and performs parameter optimization on the local second model according to the optimized parameters of the shadow model.

[0242] The terminal performs parameter optimization on the local second model according to the optimized parameters of the shadow model, which can improve the inference effect of the local second model.

[0243] The inference method provided by the embodiments of the present application can provide the terminal with the second model obtained by model distillation on the first model, so that the terminal can deploy the small-scale second model with better inference effect locally, and then infer the user request by using the local second model, thereby significantly improving the inference speed. In addition, the inference method provided by the embodiments of the present application can also perform parameter optimization on the cloud-side shadow model according to the inference difference of the cloud-side shadow model and the cloud-side first model for the user request, thereby realizing parameter optimization on the terminal-side second model. By optimizing the terminal-side second model, the inference effect of the terminal side can be improved.

[0244] The embodiments of the present application also provide a device corresponding to the inference method provided by the above embodiments. Please refer to Figure 10 , Figure 10 A structural schematic diagram of an inference device provided by the embodiments of the present application is provided. The inference device is applied to a cloud server, and can include a model distillation module 1001, a shadow model acquisition module 1002, a model sending module 1003, an inference related data receiving module 1004, a model optimization module 1005, and a parameter sending module 1006.

[0245] The model distillation module 1001 is configured to obtain a second model by performing model distillation on a first model.

[0246] The shadow model acquisition module 1002 is configured to obtain a shadow model by copying the second model.

[0247] The model sending module 1003 is configured to send the second model to a terminal, so that the terminal deploys the second model locally and infers a user request by using the local second model.

[0248] The inference related data receiving module 1004 is configured to receive inference related data sent by the terminal.

[0249] The model optimization module 1005 is configured to perform parameter optimization on the shadow model by using the inference related data and the first model.

[0250] The parameter sending module 1006 is configured to send the optimized parameters of the shadow model to the terminal, so that the terminal performs parameter optimization on the local second model according to the optimized parameters of the shadow model.

[0251] In a possible implementation, the inference-related data comprises the user request.

[0252] The model optimization module 1005 is specifically configured to:

[0253] determine the inference difference between the second model or the shadow model and the first model for the user request.

[0254] perform parameter optimization on the shadow model according to the inference difference.

[0255] In a possible implementation, the inference-related data further comprises: a first inference result obtained by the terminal using the local second model to perform inference on the user request, and first probability information of each word in the first inference result.

[0256] The model optimization module 1005 is specifically configured to:

[0257] obtain, using the first model, second probability information of each word in the first inference result according to the user request and the first inference result.

[0258] determine a prediction loss of the second model according to the first probability information and the second probability information of each word in the first inference result, wherein the prediction loss of the second model can represent the inference difference between the second model and the first model for the user request.

[0259] In a possible implementation, the model optimization module 1005 is specifically configured to:

[0260] for each word in the first inference result, determine a difference between the first probability information of the word and the second probability information of the word to obtain a probability information difference representation value as the prediction loss of the second model on the word.

[0261] fuse the prediction losses of the second model on each word in the first inference result to obtain the final prediction loss of the second model.

[0262] In a possible implementation, the model optimization module 1005 is specifically configured to:

[0263] perform inference on the user request using the shadow model to obtain a second inference result of the user request and first probability information of each word in the second inference result.

[0264] The model optimization module 1005 is specifically configured to: obtain, by using the first model, second probability information of each word in a reasoning result of the user request according to the user request and the reasoning result.

[0265] The model optimization module 1005 is specifically configured to: determine a prediction loss of the shadow model according to the first probability information and the second probability information of each word in the second reasoning result, wherein the prediction loss of the shadow model can represent a reasoning difference between the shadow model and the first model for the user request.

[0266] In a possible implementation, the first probability information of any word in a reasoning result is a first probability corresponding to the word in the reasoning result or a first candidate word probability distribution, the first candidate word probability distribution corresponding to any word in a reasoning result is a candidate word probability distribution obtained in the process of predicting the word, and the first probability corresponding to any word in a reasoning result is obtained from the first candidate word probability distribution corresponding to the word.

[0267] The model optimization module 1005 is specifically configured to:

[0268] In the t th prediction step:

[0269] The model optimization module 1005 is specifically configured to: predict a candidate word probability distribution of the t+1 th word by using the first model according to the user request and the first t words in the reasoning result, and the candidate word probability distribution of the t+1 th word is used as the second candidate word probability distribution corresponding to the t+1 th word in the reasoning result.

[0270] Alternatively, the model optimization module 1005 is specifically configured to: predict a candidate word probability distribution of the t+1 th word by using the first model according to the user request and the first t words in the reasoning result, and obtain the second probability corresponding to the t+1 th word in the reasoning result from the candidate word probability distribution of the t+1 th word.

[0271] The second probability or the second candidate word probability distribution corresponding to the t+1 th word in the reasoning result is used as the second probability information of the t+1 th word in the reasoning result, wherein t is from 0 to N-1, and N is the length of the reasoning result.

[0272] In a possible implementation, the model optimization module 1005 is specifically configured to:

[0273] The model optimization module 1005 is specifically configured to: perform reasoning on the user request by using the first model to obtain a third reasoning result of the user request and first probability information of each word in the third reasoning result.

[0274] The model optimization module 1005 is specifically configured to: obtain, by using the shadow model, second probability information of each word in the third reasoning result according to the user request and the third reasoning result.

[0275] According to the first probability information and the second probability information of each word in the third reasoning result, a prediction loss of the shadow model is determined, where the prediction loss of the shadow model can represent a reasoning difference between the shadow model and the first model for the user request.

[0276] In a possible implementation, the first probability information of any word in the third reasoning result is a first probability corresponding to the word in the third reasoning result or a first candidate word probability distribution, the first candidate word probability distribution corresponding to any word in the third reasoning result is a candidate word probability distribution obtained in the process of predicting the word, and the first probability corresponding to any word in the third reasoning result is obtained from the first candidate word probability distribution corresponding to the word.

[0277] When the model optimization module 1005 obtains the second probability information of each word in the third reasoning result according to the user request and the third reasoning result by using the shadow model, the model optimization module 1005 is specifically configured to:

[0278] In the t th prediction step:

[0279] The shadow model is used to predict a candidate word probability distribution of a (t+1) th word according to the user request and the first t words in the third reasoning result, the candidate word probability distribution of the (t+1) th word serving as a second candidate word probability distribution corresponding to the (t+1) th word in the third reasoning result;

[0280] Alternatively, the shadow model is used to predict a candidate word probability distribution of a (t+1) th word according to the user request and the first t words in the third reasoning result, and a probability corresponding to the (t+1) th word in the third reasoning result is obtained from the candidate word probability distribution of the (t+1) th word, the probability corresponding to the (t+1) th word serving as a second probability corresponding to the (t+1) th word in the third reasoning result;

[0281] The second probability or the second candidate word probability distribution corresponding to the (t+1) th word in the third reasoning result serves as the second probability information of the (t+1) th word in the third reasoning result; where t is from 0 to M-1, and M is the length of the third reasoning result.

[0282] In a possible implementation, the reasoning device further includes a parameter feedback determination module.

[0283] The parameter feedback determination module is configured to determine whether a preset parameter feedback condition is met, where the parameter feedback condition is that a time interval from the last time when the parameter is sent to the terminal reaches a preset time interval, or a number of times of parameter optimization of the shadow model after the last time when the parameter is sent to the terminal reaches a preset number of times of parameter optimization.

[0284] The parameter sending module 1006 is specifically configured to send the optimized parameter of the shadow model to the terminal when the parameter feedback condition is met.

[0285] The inference device provided by the embodiments of the present application can provide a small-scale second model obtained by model distillation on a large-scale first model to a terminal, so that the terminal can deploy the small-scale second model with better effect locally, and then use the local second model to infer a user request, thereby significantly improving the inference speed. In addition, the inference device provided by the embodiments of the present application can also optimize the parameters of the second model on the terminal by optimizing the parameters of the shadow model, thereby improving the inference effect on the terminal side.

[0286] Please refer to Figure 11 , Figure 11 The structure diagram of another inference device provided by the embodiments of the present application is shown in FIG. 11. The inference device is applied to a terminal, and can include a model receiving module 1101, a model deployment module 1102, an inference module 1103, an inference-related data sending module 1104, a parameter receiving module 1105, and a model optimization module 1106.

[0287] The model receiving module 1101 is configured to receive a second model sent by a cloud server.

[0288] The model deployment module 1102 is configured to deploy the second model locally. The second model is a model obtained by model distillation on a first model of the cloud server.

[0289] The inference module 1103 is configured to, when receiving a user request, use the local second model to infer the user request, and obtain an inference result of the user request.

[0290] The inference-related data sending module 1104 is configured to send inference-related data to the cloud server, so that the cloud server uses the inference-related data to assist the first model to optimize the parameters of a shadow model. The shadow model is obtained by the cloud server copying the second model.

[0291] The parameter receiving module 1105 is configured to receive the optimized parameters of the shadow model sent by the cloud server.

[0292] The model optimization module 1106 is configured to optimize the parameters of the local second model according to the optimized parameters of the shadow model.

[0293] In a possible implementation manner, the inference-related data includes the user request.

[0294] In another possible implementation manner, the inference-related data includes the user request, the inference result, and probability information of each word in the inference result.

[0295] The reasoning device provided in the embodiments of the present application can obtain a small-scale second model close to the large-scale first model effect of the cloud server, and then deploy the second model locally. When a user request is received, the second model in the local can be directly used to reason the user request. Since the terminal uses the small-scale second model in the local to reason the user request, the reasoning speed is faster. In addition, the reasoning device provided in the embodiments of the present application can also feed back reasoning-related data to the cloud server, so that the cloud server uses the reasoning-related data to assist the first model to optimize the parameters of the shadow model. Then, after obtaining the optimized parameters of the shadow model sent by the cloud server, the second model in the local can be optimized according to the optimized parameters of the shadow model, so that the performance of the local second model is improved.

[0296] The embodiments of the present application also provide a cloud server, which comprises at least one processor and a memory connected with the processor, wherein:

[0297] The memory is used to store a computer program;

[0298] The processor is used to execute the computer program, so that the cloud server can implement the steps of the reasoning method applied to the cloud server provided in the above embodiments.

[0299] The embodiments of the present application also provide a terminal, which comprises at least one processor and a memory connected with the processor, wherein:

[0300] The memory is used to store a computer program;

[0301] The processor is used to execute the computer program, so that the terminal can implement the steps of the reasoning method applied to the terminal provided in the above embodiments.

[0302] The embodiments of the present application also provide a computer storage medium, which carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement the steps of the reasoning method provided in the above embodiments.

[0303] The embodiments of the present application also provide a computer program product, which comprises computer readable instructions. When the computer readable instructions run on an electronic device, the electronic device can implement the steps of the reasoning method provided in the above embodiments.

[0304] The embodiments of the present application also provide a reasoning system, which comprises a terminal and a cloud server. The first model is deployed on the cloud server.

[0305] The cloud server is used to obtain a second model by model distillation on the first model, obtain a shadow model by copying the second model, and send the second model to the terminal.

[0306] The terminal is configured to receive the second model, locally deploy the second model, and, when receiving a user request, perform inference on the user request using the locally deployed second model and send inference-related data to the cloud server.

[0307] The cloud server is further configured to receive the inference-related data, and, using the inference-related data, perform parameter optimization on the shadow model in conjunction with the first model, and send the optimized parameters of the shadow model to the terminal.

[0308] The terminal is further configured to receive the optimized parameters of the shadow model, and perform parameter optimization on the locally deployed second model according to the optimized parameters of the shadow model.

[0309] The more specific implementation process of the inference and model optimization achieved by the interaction between the cloud server and the terminal can be referred to the inference method embodiments described above, and will not be described here.

[0310] In addition, it should be noted that the apparatus embodiments described above are only schematic, and the units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, i.e., they can be located in one place or distributed on multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the embodiments. In addition, in the apparatus embodiments provided by the present application, the connection relationship between the modules indicates that there is a communication connection between them, which can be implemented as one or more communication buses or signal lines.

[0311] Through the above description of the embodiments, those skilled in the art can clearly understand that the present application can be implemented by means of software and the necessary general hardware, and of course, it can also be implemented by special hardware including special integrated circuits, special CPUs, special memories, special components, etc. Generally, functions completed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structure for implementing the same function can also be various, such as analog circuits, digital circuits or special circuits. However, for the present application, software program implementation is a better embodiment. Based on this understanding, the technical solutions of the present application can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer's floppy disk, U disk, mobile hard disk, ROM, RAM, magnetic disk or optical disk, etc., including a plurality of instructions to make a computer device (which can be a personal computer, training device, or network device, etc.) execute the methods described in various embodiments of the present application.

[0312] In the above embodiments, the implementation can be wholly or partially by software, hardware, firmware, or any combination thereof. When implemented by software, the implementation can be wholly or partially in the form of a computer program product.

[0313] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions described in the embodiments of the present application are wholly or partially generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another, for example, the computer instructions can be transmitted from one website, computer, training device or data center to another website, computer, training device or data center through wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can store or a data storage device such as a training device, a data center, etc. integrated with one or more available media. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a magnetic tape), an optical medium (for example, a DVD), or a semiconductor medium (for example, a solid state disk (SSD)), etc.< / eos> < / eos> < / eos> < / eos> < / eos>

Claims

1. An inference method, characterized by, The application is applied to a cloud server, wherein a first model is deployed on the cloud server, and the inference method comprises the following steps: A second model is obtained by model distillation on the first model, and a shadow model is obtained by copying the second model; The second model is sent to a terminal, so that the terminal locally deploys the second model, and infers a user request by using the local second model; Inference-related data sent by the terminal is received, and the shadow model is parameter-optimized by using the inference-related data and the first model; The optimized parameters of the shadow model are sent to the terminal, so that the terminal parameter-optimizes the local second model according to the optimized parameters of the shadow model; The inference-related data comprises the following steps: The inference result obtained by inferring the user request by using one of the two models and the first probability information of each word in the inference result are obtained; the second probability information of each word in the inference result is obtained by using the other model of the two models according to the user request and the inference result; one of the two models is the first model, and the other model is the shadow model or the second model; According to the first probability information and the second probability information of each word in the inference result, a prediction loss representing the inference difference of the two models is determined; and the shadow model is parameter-optimized according to the prediction loss; The prediction loss representing the inference difference of the two models is determined according to the first probability information and the second probability information of each word in the inference result, comprising the following steps: In the case that the probability information of a word is a candidate word probability distribution corresponding to the word, for each word in the inference result: according to the first candidate word probability distribution corresponding to the word and the second candidate word probability distribution corresponding to the word respectively, a part of candidate words is selected from each candidate word according to a preset screening strategy, the selected candidate words are combined and de-duplicated, and a candidate word set is formed by the combined and de-duplicated candidate words; the first probability corresponding to each candidate word in the candidate word set is obtained from the first candidate word probability distribution corresponding to the word, and the second probability corresponding to each candidate word in the candidate word set is obtained from the second candidate word probability distribution corresponding to the word, and the probability difference representation value corresponding to each candidate word in the candidate word set is determined according to the first probability and the second probability corresponding to each candidate word in the candidate word set; the probability difference representation values corresponding to each candidate word in the candidate word set are fused to obtain the prediction loss corresponding to the word; The prediction loss corresponding to each word in the inference result is fused to obtain the prediction loss representing the inference difference of the two models.

2. The inference method of claim 1, wherein, The inference result obtained by inferring the user request by using one of the two models and the first probability information of each word in the inference result are obtained, comprising the following steps: The first inference result obtained by inferring the user request by using the second model and the first probability information of each word in the first inference result are obtained. The second probability information of each word in the inference result is obtained by using another model of the two models according to the user request and the inference result, and the second probability information of each word in the inference result includes: The second probability information of each word in the first inference result is obtained by using the first model according to the user request and the first inference result; The prediction loss representing the inference difference of the two models is determined according to the first probability information and the second probability information of each word in the inference result, and the prediction loss representing the inference difference of the two models includes: The prediction loss of the second model is determined according to the first probability information and the second probability information of each word in the first inference result, and the prediction loss of the second model can represent the inference difference of the second model and the first model for the user request.

3. The inference method of claim 1, wherein, The inference result obtained by using one of the two models to infer the user request and the first probability information of each word in the inference result include: The second inference result obtained by using the shadow model to infer the user request and the first probability information of each word in the second inference result are obtained; The second probability information of each word in the inference result is obtained by using another model of the two models according to the user request and the inference result, and the second probability information of each word in the inference result includes: The second probability information of each word in the second inference result is obtained by using the first model according to the user request and the second inference result; The prediction loss representing the inference difference of the two models is determined according to the first probability information and the second probability information of each word in the inference result, and the prediction loss representing the inference difference of the two models includes: The prediction loss of the shadow model is determined according to the first probability information and the second probability information of each word in the second inference result, and the prediction loss of the shadow model can represent the inference difference of the shadow model and the first model for the user request.

4. The inference method of claim 1, wherein, The first probability information of any word in the inference result is the first probability or the first candidate word probability distribution corresponding to the word in the inference result, the first candidate word probability distribution corresponding to any word in the inference result is the candidate word probability distribution obtained in the process of predicting the word, and the first probability corresponding to any word in the inference result is obtained from the first candidate word probability distribution corresponding to the word; The second probability information of each word in the inference result is obtained by using the first model according to the user request and the inference result, and the second probability information of each word in the inference result includes: In the tth prediction step: The candidate word probability distribution of the t+1th word is predicted by using the first model according to the user request and the first t words in the inference result, and the candidate word probability distribution of the t+1th word is used as the second candidate word probability distribution corresponding to the t+1th word in the inference result; Or, the candidate word probability distribution of the t+1th word is predicted by using the first model according to the user request and the first t words in the inference result, and the second probability corresponding to the t+1th word in the inference result is obtained from the candidate word probability distribution of the t+1th word. The second probability or the second candidate word probability distribution corresponding to the t+1th word in the inference result is taken as the second probability information of the t+1th word in the inference result, where t is from 0 to N-1, and N is the length of the inference result.

5. The inference method of claim 1, wherein, The inference result obtained by performing inference on the user request by using one of the two models and the first probability information of each word in the inference result are obtained. The third inference result obtained by performing inference on the user request by using the first model and the first probability information of each word in the third inference result are obtained. The second probability information of each word in the inference result is obtained according to the user request and the inference result by using the other of the two models, and the second probability information of each word in the inference result is obtained according to the user request and the third inference result by using the shadow model. The second probability information of each word in the third inference result is obtained according to the user request and the third inference result by using the shadow model. The prediction loss representing the inference difference between the two models is determined according to the first probability information and the second probability information of each word in the inference result. The prediction loss of the shadow model is determined according to the first probability information and the second probability information of each word in the third inference result, where the prediction loss of the shadow model can represent the inference difference between the shadow model and the first model for the user request.

6. The inference method of claim 5, wherein, The first probability information of any word in the third inference result is the first probability or the first candidate word probability distribution corresponding to the word in the third inference result, the first candidate word probability distribution corresponding to any word in the third inference result is the candidate word probability distribution obtained in the process of predicting the word, and the first probability corresponding to the word is obtained from the first candidate word probability distribution corresponding to the word. The second probability information of each word in the inference result is obtained according to the user request and the inference result by using the other of the two models, and the second probability information of each word in the third inference result is obtained according to the user request and the third inference result by using the shadow model. In the tth prediction step: The candidate word probability distribution of the t+1th word is predicted by using the shadow model according to the user request and the first t words in the third inference result, and the candidate word probability distribution of the t+1th word is taken as the second candidate word probability distribution corresponding to the t+1th word in the third inference result. Alternatively, the candidate word probability distribution of the t+1th word is predicted by using the shadow model according to the user request and the first t words in the third inference result, the probability corresponding to the t+1th word is obtained from the candidate word probability distribution of the t+1th word, and the probability corresponding to the t+1th word is taken as the second probability corresponding to the t+1th word in the third inference result. The second probability or the second candidate word probability distribution corresponding to the t+1th word in the third inference result is taken as the second probability information of the t+1th word in the third inference result, where t is from 0 to M-1, and M is the length of the third inference result.

7. The inference method of claim 1, wherein, Before the optimized parameters of the shadow model are sent to the terminal, the following further includes: determining whether a preset parameter return condition is met, wherein the parameter return condition is that a time interval from a last time when the parameters are sent to the terminal reaches a preset time interval, or a number of times of parameter optimization of the shadow model after the last time when the parameters are sent to the terminal reaches a preset number of times of parameter optimization; if the parameter return condition is met, performing sending of the optimized parameters of the shadow model to the terminal.

8. An inference method, comprising: The inference method is applied to a terminal and includes the following steps: receiving a second model sent by a cloud server and deploying the second model locally, wherein the second model is a model obtained by model distillation on a first model of the cloud server; when a user request is received, inferring the user request by using the second model locally to obtain an inference result of the user request; sending inference-related data to the cloud server, so that the cloud server performs parameter optimization on a shadow model by using the inference-related data and the first model, wherein the shadow model is obtained by copying the second model by the cloud server; the cloud server performing parameter optimization on the shadow model by using the inference-related data and the first model includes: obtaining an inference result obtained by inferring the user request by using one of the two models and first probability information of each word in the inference result; obtaining second probability information of each word in the inference result by using the other of the two models according to the user request and the inference result; one of the two models is the first model, and the other is the shadow model or the second model; determining a prediction loss representing a difference in inference of the two models according to the first probability information and the second probability information of each word in the inference result; and performing parameter optimization on the shadow model according to the prediction loss; the cloud server determines the prediction loss representing the difference in inference of the two models according to the first probability information and the second probability information of each word in the inference result, including: in the case that the probability information of a word is a candidate word probability distribution corresponding to the word, for each word in the inference result: according to the first candidate word probability distribution corresponding to the word and the second candidate word probability distribution corresponding to the word, respectively, a preset screening strategy is used to screen part of the candidate words from each candidate word, and the screened candidate words are combined and de-duplicated to form a candidate word set; the first probability corresponding to each candidate word in the candidate word set is obtained from the first candidate word probability distribution corresponding to the word, and the second probability corresponding to each candidate word in the candidate word set is obtained from the second candidate word probability distribution corresponding to the word, and a probability difference representation value corresponding to each candidate word in the candidate word set is determined according to the first probability and the second probability corresponding to each candidate word in the candidate word set; and the probability difference representation values corresponding to the candidate words in the candidate word set are fused to obtain a prediction loss corresponding to the word; and the prediction losses corresponding to the words in the inference result are fused to obtain the prediction loss representing the difference in inference of the two models. receive the optimized parameters of the shadow model sent by the cloud server, and perform parameter optimization on the local second model according to the optimized parameters of the shadow model.

9. An inference apparatus characterized by comprising: The cloud server is applied to, and the cloud server is deployed with a first model, and the inference device comprises a model distillation module, a shadow model acquisition module, a model sending module, an inference related data receiving module, a model optimization module and a parameter sending module. The model distillation module is configured to obtain a second model by performing model distillation on the first model. The shadow model acquisition module is configured to obtain a shadow model by copying the second model. The model sending module is configured to send the second model to a terminal, so that the terminal locally deploys the second model, and uses the local second model to infer a user request. The inference related data receiving module is configured to receive inference related data sent by the terminal. The model optimization module is configured to use the inference related data, with the aid of the first model, to perform parameter optimization on the shadow model. The parameter sending module is configured to send the optimized parameters of the shadow model to the terminal, so that the terminal performs parameter optimization on the local second model according to the optimized parameters of the shadow model. When the model optimization module uses the inference related data, with the aid of the first model, to perform parameter optimization on the shadow model, it is specifically configured to: obtain an inference result obtained by one of the two models inferring the user request and first probability information of each word in the inference result; and obtain second probability information of each word in the inference result according to the user request and the inference result by using the other of the two models; one of the two models is the first model, and the other is the shadow model or the second model. According to the first probability information and the second probability information of each word in the inference result, a prediction loss representing the inference difference between the two models is determined; and the shadow model is optimized according to the prediction loss. When the model optimization module determines the prediction loss representing the inference difference between the two models according to the first probability information and the second probability information of each word in the inference result, it is specifically configured to: In a case where the probability information of a word is a candidate word probability distribution corresponding to the word, for each word in the inference result: according to the first candidate word probability distribution corresponding to the word and the second candidate word probability distribution corresponding to the word respectively, a preset screening strategy is used to screen part of the candidate words from each candidate word, the screened candidate words are merged and deduplicated, and a candidate word set is formed by the candidate words after the merging and deduplication; a first probability corresponding to each candidate word in the candidate word set is obtained from the first candidate word probability distribution corresponding to the word, and a second probability corresponding to each candidate word in the candidate word set is obtained from the second candidate word probability distribution corresponding to the word, a probability difference representation value corresponding to each candidate word in the candidate word set is determined according to the first probability and the second probability corresponding to each candidate word in the candidate word set; and the probability difference representation values corresponding to the candidate words in the candidate word set are fused to obtain a prediction loss corresponding to the word. The prediction losses corresponding to the words in the inference result are fused to obtain a prediction loss representing the inference difference between the two models.

10. An inference apparatus characterized by comprising: The inference device is applied to a terminal and includes a model receiving module, a model deployment module, an inference module, an inference related data sending module, a parameter receiving module, and a model optimization module. The model receiving module is configured to receive a second model sent by a cloud server. The model deployment module is configured to deploy the second model locally, where the second model is a model obtained by model distillation on a first model of the cloud server. The inference module is configured to, when receiving a user request, perform inference on the user request by using the second model locally to obtain an inference result of the user request. The parameter receiving module is configured to receive a parameter sent by the cloud server. The model optimization module is configured to optimize the second model by using the parameter. The inference-related data sending module is configured to send inference-related data to a cloud server, so that the cloud server performs parameter optimization on a shadow model by using the inference-related data and the first model, wherein the shadow model is obtained by copying the second model by the cloud server; the cloud server performing parameter optimization on the shadow model by using the inference-related data and the first model comprises: obtaining an inference result obtained by one of the two models performing inference on the user request and first probability information of each word in the inference result; obtaining second probability information of each word in the inference result by using the other of the two models according to the user request and the inference result; one of the two models is the first model, and the other is the shadow model or the second model; determining a prediction loss representing inference difference of the two models according to the first probability information and the second probability information of each word in the inference result; and performing parameter optimization on the shadow model according to the prediction loss; the cloud server determines the prediction loss representing inference difference of the two models according to the first probability information and the second probability information of each word in the inference result, comprising: in the case that the probability information of a word is a candidate word probability distribution corresponding to the word, for each word in the inference result: according to the first candidate word probability distribution corresponding to the word and the second candidate word probability distribution corresponding to the word respectively, a part of candidate words are screened from each candidate word according to a preset screening strategy, the screened candidate words are combined and de-duplicated, and a candidate word set is formed by the combined and de-duplicated candidate words; a first probability corresponding to each candidate word in the candidate word set is obtained from the first candidate word probability distribution corresponding to the word, and a second probability corresponding to each candidate word in the candidate word set is obtained from the second candidate word probability distribution corresponding to the word, and a probability difference representation value corresponding to each candidate word in the candidate word set is determined according to the first probability and the second probability corresponding to each candidate word in the candidate word set; the probability difference representation values corresponding to the candidate words in the candidate word set are fused respectively to obtain a prediction loss corresponding to the word; and the prediction losses corresponding to the words in the inference result are fused respectively to obtain the prediction loss representing inference difference of the two models; The parameter receiving module is configured to receive the optimized parameters of the shadow model sent by the cloud server. The model optimization module is configured to perform parameter optimization on the second model locally according to the optimized parameters of the shadow model.

11. A cloud server, characterized by The memory is configured to store a computer program. The processor is configured to execute the computer program, so that the cloud server can implement the steps of the inference method in any one of claims 1-7. The memory is configured to store a computer program.

12. A terminal, characterized by comprising: The processor is configured to execute the computer program. ​ The processor is configured to execute the computer program to enable the terminal to implement the steps of the inference method according to claim 8.

13. A computer storage medium, characterized in that The storage medium carries one or more computer programs, which, when executed by an electronic device, enable the electronic device to implement the steps of the inference method according to any one of claims 1-7, or implement the steps of the inference method according to claim 8.

14. A computer program product, characterised in that, The computer readable instructions, when executed on an electronic device, cause the electronic device to implement the steps of the inference method according to any one of claims 1-7, or implement the steps of the inference method according to claim 8.

15. An inference system, comprising: Comprise: a terminal and a cloud server, wherein the cloud server is deployed with a first model; the cloud server is configured to obtain a second model by model distillation on the first model, obtain a shadow model by copying the second model, and send the second model to the terminal; the terminal is configured to receive the second model, deploy the second model locally, and when a user request is received, perform inference on the user request by using the local second model, and send inference-related data to the cloud server; the cloud server is further configured to receive the inference-related data, and perform parameter optimization on the shadow model by using the inference-related data and the first model, and send the optimized parameters of the shadow model to the terminal; the terminal is further configured to receive the optimized parameters of the shadow model, and perform parameter optimization on the local second model according to the optimized parameters of the shadow model; when the cloud server performs parameter optimization on the shadow model by using the inference-related data and the first model, and sends the optimized parameters of the shadow model to the terminal, the cloud server is specifically configured to: obtain an inference result of one of the two models on the user request and first probability information of each word in the inference result; and obtain second probability information of each word in the inference result by using the other model of the two models according to the user request and the inference result; one of the two models is the first model, and the other model is the shadow model or the second model; determine a prediction loss representing inference difference between the two models according to the first probability information and the second probability information of each word in the inference result; and perform parameter optimization on the shadow model according to the prediction loss; when the cloud server determines the prediction loss representing inference difference between the two models according to the first probability information and the second probability information of each word in the inference result, the cloud server is specifically configured to: In a case where the probability information of a word is a candidate word probability distribution corresponding to the word, for each word in the inference result: according to the first candidate word probability distribution corresponding to the word and the second candidate word probability distribution corresponding to the word respectively, a preset screening strategy is used to screen part of the candidate words from each candidate word, the screened candidate words are merged and deduplicated, and a candidate word set is formed by the candidate words after the merging and deduplication; a first probability corresponding to each candidate word in the candidate word set is obtained from the first candidate word probability distribution corresponding to the word, a second probability corresponding to each candidate word in the candidate word set is obtained from the second candidate word probability distribution corresponding to the word, and a probability difference representation value corresponding to each candidate word in the candidate word set is determined according to the first probability and the second probability corresponding to each candidate word in the candidate word set; the probability difference representation values corresponding to the candidate words in the candidate word set are fused respectively to obtain a prediction loss corresponding to the word; the prediction losses corresponding to the words in the inference result are fused respectively to obtain a prediction loss representing the inference difference between the two models.

Images