Data processing method, computing device and system

Abstract

The invention provides a data processing method, a computing device and a system. The data processing method comprises the following steps: reasoning input data by utilizing a first model deployed in first computing equipment to obtain a first reasoning result; the first computing device sends the first reasoning result to the second computing device; the first computing device determines new input data of the first model by using the verification result; the verification result is obtained by verifying the first reasoning result by using a second reasoning result of the second model; wherein the second model is deployed on a second computing device. According to the data processing method, the reasoning task is partially unloaded to the first model of the first computing device, on one hand, the computing power cost of the second computing device can be reduced by using the computing power of the first computing device; and on the other hand, the reasoning result of the first model is verified by using the second reasoning result of the second model, so that the reasoning precision of the whole reasoning task can be ensured.

Description

Technical Field

[0001] This invention relates to the field of computers, and more specifically, to a data processing method, computing device, and system. Background Technology

[0002] With the development of AI technology, large language models (LLMs) have gradually entered people's lives, bringing many positive impacts and conveniences. LLMs typically have billions or even trillions of parameters, enabling them to handle complex natural language understanding and generation tasks. Due to the enormous number of parameters in an LLM, it is usually deployed in the cloud, with cloud-side devices providing computing power. However, deploying large language models solely in the cloud fails to utilize the computing resources of edge devices, resulting in wasted edge computing resources and failing to achieve the goal of reducing cloud computing costs through edge computing power. Summary of the Invention

[0003] This invention provides a data processing method, computing device, and system that can reduce the computational cost of model inference.

[0004] In a first aspect, the present invention provides a data processing method, the method comprising: a first computing device inferring from input data using a first model to obtain a first inference result, the first model existing in the first computing device; the first computing device sending the first inference result to a second computing device; the first computing device determining new input data for the first model based on a verification result, the verification result being obtained by verifying the first inference result using a second inference result, the second inference result being obtained by the second computing device inferring from the first inference result using a second model, the second model existing in the second computing device.

[0005] In this solution, by deploying the first model on the first computing device and the second model on the second computing device, the computing power of the first computing device is utilized. On the other hand, the inference results of the second model are used to verify and correct the inference results of the first model, which can improve the accuracy of the inference results of the first model. At the same time, transferring some of the inference tasks of the second model to the first model can reduce the computing power cost of the second computing device.

[0006] In conjunction with the first aspect, in one possible implementation of the first aspect, the method further includes: the first computing device receiving a verification result returned by the second computing device, the verification result being obtained by the second computing device verifying the first inference result using the second inference result of the second model.

[0007] In this scheme, the verification action is performed by the second computing device, which then sends the verification result to the first computing device. This avoids the second computing device sending the second calculation result to the first computing device, reducing latency caused by data transmission and improving inference efficiency. It is understandable that transmitting the second inference result takes longer than transmitting the verification result; therefore, the time spent by the second computing device sending the second inference result to the first computing device is longer than the time spent sending the verification result back to the first computing device.

[0008] In conjunction with the first aspect, in one possible implementation of the first aspect, the second model is homologous to the first model, and the number of parameters in the first model is less than the number of parameters in the second model.

[0009] In this scheme, the first model and the second model share a common origin, resulting in a higher acceptance rate when using them for collaborative reasoning. The first model has a smaller number of parameters, allowing it to be deployed on computing devices with lower computational power, thus making full use of these devices.

[0010] In conjunction with the first aspect, in one possible implementation of the first aspect, the computing power of the first computing device is less than that of the second computing device. It is understood that the computing resources required by the first model are matched with the computing resources available to the first computing device.

[0011] In this scheme, the computing power of the first computing device is relatively small. Therefore, some devices with smaller computing power can be selected to deploy the first model, so as to make use of the devices with smaller computing power and reduce the computing power cost of the second computing device.

[0012] In conjunction with the first aspect, in one possible implementation of the first aspect, the second reasoning result is obtained by the second model reasoning from the first reasoning result.

[0013] In this scheme, the first inference result is re-inferred by the second model, thereby correcting the inference result of the first model and improving the accuracy of the inference result of the first model.

[0014] In conjunction with the first aspect, in one possible implementation of the first aspect, the first inference result includes: a first text sequence, the first text sequence including a plurality of target text units arranged in sequence; a plurality of first probability distributions, the plurality of first probability distributions corresponding one-to-one with the plurality of target text units, the first probability distributions being used to indicate the probability of each target text unit corresponding to each text unit in the set of text units.

[0015] In conjunction with the first aspect, in one possible implementation of the first aspect, sending the first inference result to the second computing device includes: for each first probability distribution generated, sending a first network data packet to the second computing device, the first network data packet including data of the currently generated first probability distribution.

[0016] In this scheme, the first computing device sends each generated first probability distribution to the second computing device before the inference of the first model is completed, allowing the transmission of the first probability distributions to proceed in parallel. Compared to sending all first probability distributions after the inference of the first model is finished, this scheme can improve the transmission efficiency of the first probability distributions and reduce the time delay caused by sending the first probability distributions.

[0017] In conjunction with the first aspect, in one possible implementation of the first aspect, sending the first inference result to the second computing device includes: after the first text sequence is generated, sending a second network data packet to the second computing device, wherein the second network data packet includes data of the first text sequence.

[0018] In conjunction with the first aspect, in one possible implementation of the first aspect, the first network data packet includes a first identifier, which is used to indicate the target text unit corresponding to the first probability distribution data contained in the first network data packet.

[0019] In this scheme, by setting a first identifier, the second computing device can determine the target text unit corresponding to the currently received first probability distribution through the first identifier, which facilitates the second computing device to reconstruct multiple received first probability distributions.

[0020] In conjunction with the first aspect, in one possible implementation of the first aspect, the first network data packet includes a second identifier, which is used to indicate the inference round corresponding to the first probability distribution data contained in the first network data packet.

[0021] In this scheme, by setting a second identifier, the second computing device can determine which inference round the currently received first probability distribution was generated by the second identifier.

[0022] In conjunction with the first aspect, in one possible implementation of the first aspect, the first network data packet includes a third identifier, which is used to indicate an inference request corresponding to the first probability distribution data contained in the first network data packet.

[0023] In this scheme, by setting a third identifier, the second computing device can determine which inference request the currently received first probability distribution originated from through the third identifier.

[0024] In conjunction with the first aspect, in one possible implementation of the first aspect, the third identifier includes a device identifier, which is used to uniquely identify the first computing device.

[0025] In this scheme, by setting a device identifier, the second computing device can determine which computing device sent the first probability distribution it currently receives. Compared with directly sending an inference request to the second computing device and then having the second computing device return a request ID to the first computing device that sent the inference request, the first network data packet in this scheme carries a device identifier that can indicate the source of the request. This eliminates the need for the second computing device to return a request ID, saving the time required to return the request ID and improving data transmission efficiency.

[0026] In conjunction with the first aspect, in one possible implementation of the first aspect, the third identifier includes an inference task identifier, which is used to indicate the inference task corresponding to the first probability distribution data contained in the first network data packet when the first computing device executes multiple inference tasks.

[0027] In this scheme, by setting an inference task identifier, the second computing device can determine which inference task the first computing device is currently executing, the currently received first probability distribution belongs to.

[0028] In a second aspect, the present invention provides another data processing method, the method comprising: a second computing device receiving a first inference result sent by a first computing device; the first inference result being obtained by the first computing device using a first model to infer input data; the first model existing in the first computing device; the second computing device using a second model to infer the first inference result to obtain a second inference result; the second model existing in the second computing device; the second computing device using the second inference result to verify the first inference result to obtain a verification result; and the second computing device sending the verification result to the first computing device.

[0029] In this solution, by deploying the first model on the first computing device and the second model on the second computing device, the computing power of the first computing device is utilized. On the other hand, the inference results of the second model are used to verify and correct the inference results of the first model, which can improve the accuracy of the inference results of the first model. At the same time, transferring some of the inference tasks of the second model to the first model can reduce the computing power cost of the second computing device.

[0030] In conjunction with the second aspect, in one possible implementation of the second aspect, the first inference result includes: a first text sequence, the first text sequence including a plurality of target text units arranged in sequence; a plurality of first probability distributions, the plurality of first probability distributions corresponding one-to-one with the plurality of target text units, the first probability distributions being used to indicate the probability that each text unit in the set of text units is the corresponding target text unit.

[0031] In conjunction with the second aspect, in one possible implementation of the second aspect, the method further includes: upon receiving data of a first probability distribution, converting the received first probability distribution data into data that the second model can recognize.

[0032] In this scheme, each time the second computing device receives data from a first probability distribution, it converts it into data that the second model can recognize. This means that the conversion of data from each first probability distribution is performed in parallel, and the conversion of data from the first probability distribution is performed in parallel with the inference of the first model and the inference of the second model. This reduces the time delay caused by the conversion of data from the first probability distribution.

[0033] In conjunction with the second aspect, in one possible implementation of the second aspect, the method further includes: after receiving the data of the first text sequence, converting the data of the first text sequence into data that the second model can recognize.

[0034] In this scheme, the data transformation of the first probability distribution and the data transformation of the first text sequence are performed in parallel, which can reduce the time delay caused by data transformation.

[0035] In conjunction with the second aspect, in one possible implementation of the second aspect, the second model is homologous to the first model, and the number of parameters in the first model is less than the number of parameters in the second model.

[0036] Thirdly, the present invention provides a computing device comprising: a first inference module for inferring received input data using a first model to obtain a first inference result; the first model residing in the computing device; a first transceiver module for sending the first inference result to a second computing device; and a first data processing module for determining new input data for the first model based on a verification result, wherein the verification result is obtained by verifying the first inference result using a second inference result, the second inference result being obtained by the second computing device by inferring the first inference result using a second model, and the second model residing in the second computing device.

[0037] In conjunction with the third aspect, in one possible implementation of the third aspect, the verification result is obtained by the second computing device verifying the first inference result using the second inference result of the second model; the first transceiver module is also used to receive the verification result returned by the second computing device.

[0038] In this scheme, the verification action is performed by the second computing device, which then sends the verification result to the first computing device. This avoids the second computing device sending the second calculation result to the first computing device, reducing latency caused by data transmission and improving inference efficiency. It is understandable that transmitting the second inference result takes longer than transmitting the verification result; therefore, the time spent by the second computing device sending the second inference result to the first computing device is longer than the time spent sending the verification result back to the first computing device.

[0039] In conjunction with the third aspect, in one possible implementation of the third aspect, the verification result includes indication information and text units. The indication information is used to indicate the position of the text units or to indicate whether the target text units are accepted. The first data processing module is specifically used to: determine new input data for the first model based on the indication information and text units.

[0040] In conjunction with the third aspect, in one possible implementation of the third aspect, the verification result includes a text sequence, and the first data processing module is specifically used to use the text sequence as new input data for the first model.

[0041] In conjunction with the third aspect, in one possible implementation, the computing device is a terminal device, and the second computing device is an edge computing device or a cloud device. The terminal device has relatively low computing power; therefore, it can run a first model with a small number of parameters. Deploying the first model on the terminal device allows for the utilization of its computing power, reducing the inference cost of the cloud device or edge computing device.

[0042] In conjunction with the third aspect, one possible implementation involves an edge computing device as the computing device and a cloud device as the second computing device. Edge computing devices have relatively low computing power, therefore, they can run a first model with a small number of parameters. Deploying the first model on an edge computing device allows for the utilization of its computing power, reducing the inference cost on the cloud device.

[0043] In conjunction with the third aspect, in one possible implementation of the third aspect, the first inference result includes: a first text sequence, the first text sequence including a plurality of target text units arranged in sequence; a plurality of first probability distributions, the plurality of first probability distributions corresponding one-to-one with the plurality of target text units, the first probability distributions being used to indicate the probability that each text unit in the set of text units is the corresponding target text unit.

[0044] In conjunction with the third aspect, in one possible implementation of the third aspect, the first transceiver module is specifically used to: send a first network data packet to the second computing device for each first probability distribution generated, the first network data packet including the data of the currently generated first probability distribution.

[0045] In conjunction with the third aspect, in one possible implementation of the third aspect, the first transceiver module is specifically used to: after the first text sequence is generated, send a second network data packet to the second computing device, the second network data packet including the data of the first text sequence.

[0046] In conjunction with the third aspect, in one possible implementation of the third aspect, the first network data packet includes a first identifier, which is used to indicate the target text unit corresponding to the first probability distribution data contained in the first network data packet.

[0047] In conjunction with the third aspect, in one possible implementation of the third aspect, the first network data packet includes a second identifier, which is used to indicate the inference round corresponding to the first probability distribution data contained in the first network data packet.

[0048] In conjunction with the third aspect, in one possible implementation of the third aspect, the first network data packet includes a third identifier, which is used to indicate the inference request corresponding to the first probability distribution data contained in the first network data packet.

[0049] In conjunction with the third aspect, in one possible implementation of the third aspect, the third identifier includes a device identifier, which is used to uniquely identify a computing device.

[0050] In conjunction with the third aspect, in one possible implementation of the third aspect, the third identifier includes an inference task identifier, which is used to indicate the inference task corresponding to the first probability distribution data contained in the first network data packet when the computing device performs multiple inference tasks.

[0051] In conjunction with the third aspect, in one possible implementation of the third aspect, the second reasoning result is obtained by the second model reasoning from the first reasoning result.

[0052] In conjunction with the third aspect, in one possible implementation of the third aspect, the second model is homologous to the first model, and the number of parameters in the first model is less than the number of parameters in the second model.

[0053] Fourthly, the present invention provides another computing device, comprising: a second transceiver module for receiving a first inference result sent by a first computing device; the first inference result is obtained by the first computing device using a first model to infer input data; the first model exists in the first computing device; a second processing module for inferring the first inference result using a second model to obtain a second inference result; the second model exists in the computing device; a verification module for verifying the first inference result using the second inference result to obtain a verification result; the second transceiver module is further configured to send the verification result to the first computing device.

[0054] In conjunction with the fourth aspect, in one possible implementation of the fourth aspect, the first inference result includes: a first text sequence, the first text sequence including a plurality of target text units arranged in sequence; a plurality of first probability distributions, the plurality of first probability distributions corresponding one-to-one with the plurality of target text units, the first probability distributions being used to indicate the probability that each text unit in the set of text units is the corresponding target text unit.

[0055] In conjunction with the fourth aspect, in one possible implementation of the fourth aspect, the device further includes: a second data processing module, configured to convert the received first probability distribution data into data recognizable by the second model each time a first probability distribution data is received; or

[0056] In conjunction with the fourth aspect, in one possible implementation of the fourth aspect, the device further includes: a second data processing module, used to convert the data of the first text sequence into data that the second model can recognize after receiving the data of the first text sequence.

[0057] In conjunction with the fourth aspect, in one possible implementation of the fourth aspect, the second model is homologous to the first model, and the number of parameters in the first model is less than the number of parameters in the second model.

[0058] Fifthly, the present invention provides a computing system, which includes a first computing device and a second computing device; wherein the first computing device is the computing device provided in the third aspect, and the second computing device is the computing device provided in the fourth aspect.

[0059] A sixth aspect provides a computer device including a memory and a processor, the memory for storing instructions, the processor for executing the instructions stored in the memory, and the execution of the instructions stored in the memory causes the processor to perform the method of the first aspect or any possible implementation thereof.

[0060] A seventh aspect provides a computer device including a memory and a processor, the memory for storing instructions, the processor for executing the instructions stored in the memory, and the execution of the instructions stored in the memory causes the processor to perform the method of the second aspect or any possible implementation thereof.

[0061] Eighthly, the present invention provides a computer storage medium storing a computer program that, when executed by a computer, causes the computer to implement the method of the first aspect or any possible implementation thereof.

[0062] Ninthly, the present invention provides a computer storage medium storing a computer program that, when executed by a computer, causes the computer to implement the method of the second aspect or any possible implementation thereof.

[0063] In a tenth aspect, the present invention provides a computer program product storing instructions that, when executed by a computer, cause the computer to implement the method of the first aspect or any possible implementation thereof.

[0064] Eleventhly, the present invention provides a computer program product storing instructions that, when executed by a computer, cause the computer to implement the method of the second aspect or any possible implementation thereof.

[0065] The second through eleventh aspects all possess various possible designs similar to the first aspect and can produce corresponding technical effects, which will not be elaborated here. Attached Figure Description

[0066] Figure 1 This is a schematic diagram of the structure of an edge-cloud collaborative system provided in an embodiment of the present invention;

[0067] Figure 2 A flowchart of a data processing method provided in an embodiment of the present invention;

[0068] Figure 3 This is a schematic diagram of the first reasoning result obtained by reasoning using the first model in the data processing method provided in this embodiment of the invention;

[0069] Figure 4 This is a schematic diagram of the first reasoning result obtained by reasoning using the first model in the data processing method provided in this embodiment of the invention;

[0070] Figure 5 This is a schematic diagram of the first reasoning result obtained by reasoning using the second model in the data processing method provided in this embodiment of the invention;

[0071] Figure 6 This is a schematic diagram of a user-inputted question when performing reasoning using the data processing method provided in this embodiment of the invention;

[0072] Figure 7 A flowchart illustrating reasoning using the data processing method provided in this embodiment of the invention;

[0073] Figure 8 This is a schematic diagram of the final reasoning result obtained after reasoning using the data processing method provided in the embodiments of the present invention;

[0074] Figure 9This is a schematic diagram of the structure of a terminal device provided in an embodiment of the present invention;

[0075] Figure 10 This is a schematic diagram of the structure of a terminal device provided in an embodiment of the present invention;

[0076] Figure 11 A schematic diagram of the structure of a cloud device provided in an embodiment of the present invention;

[0077] Figure 12 A schematic diagram of the structure of a cloud device provided in an embodiment of the present invention;

[0078] Figure 13 This is a schematic diagram of the structure of an edge-cloud collaborative system provided in an embodiment of the present invention. Detailed Implementation

[0079] The technical solutions in the embodiments of the present invention will now be described with reference to the accompanying drawings.

[0080] This invention can be applied to scenarios where some inference tasks of a large language model are offloaded to devices with limited computing power. Specifically, it can be applied to scenarios where it is necessary to reduce the inference cost of a large language model on the cloud side. In these scenarios, inference tasks based on user requests are partially executed by devices with limited computing power and partially executed by devices with high computing power.

[0081] Devices with limited computing power can be terminal devices, such as mobile terminals like smartphones, tablets, learning machines, and PCs; or edge computing devices, such as edge intelligent stations and edge computing servers. These terminal and edge computing devices possess some AI computing power, but it is limited. Small-scale large language models can be deployed on them.

[0082] Devices with strong computing capabilities can be cloud-based devices, which can be servers or server clusters. These cloud-based devices can also be called computing nodes or cloud-side computing clusters. Devices with strong computing capabilities can also be edge computing devices, such as edge inference servers. These cloud and edge computing devices possess strong AI computing power, enabling the deployment of large-scale language models.

[0083] Understandably, model size refers to the number of model parameters. The larger the model size, the greater the computational and storage resources required. Generally speaking, the smaller the model size, the faster its inference speed, but the lower its inference accuracy.

[0084] Regarding the first model and the second model in this embodiment of the invention, the first model has a smaller number of parameters, while the second model has a larger number of parameters. Thus, the first model can be used to quickly infer from the input data and generate several possible results. Then, the second model is used to verify the inference results of the first model, improving inference accuracy and accelerating the inference speed of the second model.

[0085] Specifically, the embodiments of the present invention can be applied to the following scenarios.

[0086] I. End-to-Cloud Collaboration Scenarios.

[0087] In edge-cloud collaboration scenarios, there are terminal devices and cloud devices. In this scenario, terminal devices possess a certain level of AI computing power, allowing for the deployment of small-scale large language models (Model 1). Cloud devices have stronger AI computing power, enabling the deployment of large-scale large language models (Model 2).

[0088] II. End-to-edge collaboration scenario.

[0089] In edge-device collaboration scenarios, there are terminal devices and edge computing devices. In this scenario, terminal devices possess a certain level of AI computing power, allowing for the deployment of small-scale large language models (Model 1). Edge computing devices have stronger AI computing power, enabling the deployment of large-scale large language models (Model 2).

[0090] III. End-edge-cloud collaboration scenarios.

[0091] The edge-cloud collaboration scenario includes terminal devices, edge computing devices, and cloud devices. In this scenario, terminal devices lack AI computing power and are mainly used for user interaction; edge computing devices possess a certain level of AI computing power, allowing the deployment of small-scale large language models (Model 1); and cloud devices have strong AI computing power, enabling the deployment of large-scale large language models (Model 2).

[0092] It's important to note that having AI computing power can be understood as the device possessing an AI chip. Accordingly, in the three scenarios mentioned above, "the terminal device / edge computing device possesses some AI computing power" means that the terminal device / edge computing device has an AI chip, but its AI chip's computing power is lower than that of the AI ​​chip in the cloud device. In the third scenario, "the terminal device lacks AI computing power" means that the terminal device does not have an AI chip.

[0093] The technical solution of this invention will be described below using an edge-cloud collaboration scenario as an example. First, let's consider... Figure 1 This invention describes the end-to-cloud collaborative system 100 provided in an embodiment of the invention.

[0094] like Figure 1As shown, the edge-cloud collaborative system 100 includes a terminal device 110 and a cloud device 120. The computing power of the terminal device 110 is less than that of the cloud device. The terminal device 110 includes a first model 111 and an application program 112. Figure 1 The diagram illustrates applications 1, 2, ..., n. The first model 111 performs a preliminary inference task to obtain a first inference result. Application 112 is implemented based on the first model 111 running on terminal device 110. Application 120 provides inference services to users, who input inference requests through the application. Cloud device 120 includes a second model 121, which performs a later inference task to obtain a second inference result. Cloud device 120 verifies the first inference result using the first and second inference results to obtain a verification result. Cloud device 120 sends the verification result to terminal device 110. Terminal device 110 updates its input data based on the verification result and uses the updated input data as input to the first model. After receiving new input data, the first model 111 enters the next round of inference. This cycle continues until the inference task termination condition is met.

[0095] In some embodiments, the first model 111 and the second model 121 are large language models. The size of the first model 111 is smaller than the size of the second model 121. That is, the first model 111 is a small-scale large language model, and the second model 121 is a large-scale large language model. Here, "size" can be understood as the number of parameters. Therefore, the computing power required by the first model 111 is less than that required by the second model 121. Compared with the technical solution of deploying both the first model 111 and the second model 121 on cloud devices, the technical solution of this embodiment of the invention, by deploying the first model 111 on the terminal device 110, can utilize the inference capabilities of the terminal device 110 and reduce cloud-side inference costs.

[0096] In some embodiments, the first model 111 and the second model 121 are homologous. For large oracle models, homologousness can be understood as having the same model structure, the same tokenizer, and the same training method. The first model 111 can be obtained by performing structured pruning, fine-tuning, or quantization on the second model 121.

[0097] It is understandable that the first and second models can also be other models, such as computer vision models, audio models, multimodal large models, etc. The following text will use the example of both the first and second models being large language models for illustration.

[0098] Figure 2 A flowchart illustrating a data processing method 200 according to an embodiment of the present invention is shown. Figure 2 As shown, method 200 is executed by a terminal device and a cloud device, wherein the terminal device is, for example, a... Figure 1The terminal device 110 shown, the cloud device is, for example, Figure 1 The cloud device 120 shown. The method 200 includes steps 210 to 250.

[0099] Step 210: The terminal device uses the first model to reason about the input data and obtains the first reasoning result.

[0100] Specifically, the terminal device receives an inference request sent by the user. This inference request requests the processing of an inference task, and executing the inference task yields the text output desired by the user. After receiving the user's inference request, when the first model performs its first round of inference, the input data to the first model is prompts provided by the user, such as prompts obtained based on the user's inference request (e.g., a user question).

[0101] In some embodiments, after the first round of inference, the first model performs multiple rounds of inference, and the input data for the first inference of each subsequent round of inference is obtained by performing step 250.

[0102] In some embodiments, the first inference result includes a first text sequence comprising a plurality of target text units arranged in sequence. Figure 3 An example of the first text sequence is shown. Figure 3 Each word and each punctuation mark in the text is a target text unit (target token). Figure 3 The first text sequence in the dataset consists of 7 target text units.

[0103] In some embodiments, the first model is an autoregressive model. Autoregression means that the model outputs word by word, that is, each output of the autoregressive model contains only a new text unit (token). When outputting each new token, the autoregressive model needs to use all previously output tokens as input for inference.

[0104] In this embodiment of the invention, the first model obtains γ new tokens in each round of inference. Here, γ is the inference configuration parameter. In one round of inference, the first model performs γ inferences, obtaining a new token each time. The following uses... Figure 3 The reasoning process of the first model will be explained using the first text sequence shown as an example.

[0105] like Figure 3As shown, assume γ is 7. The first model obtains the input data for the first inference based on the user's inference request. After the first inference, it outputs token1 (the first token), which is the word "the". Then, using "the" as input, it performs the second inference, outputting token1 and token2 (the second token), which is "the dog". Next, using "the dog" as input, it performs the third inference, outputting token1, token2, and token3 (the third token), which is "the dog is". This process repeats until the current round of 7 inferences is completed, obtaining 7 tokens (token1 to token7).

[0106] In some embodiments, the first inference result further includes multiple first probability distributions. Each of these first probability distributions corresponds one-to-one with a plurality of target text units in the first text sequence; that is, each target text unit in the first text sequence corresponds to its own unique first probability distribution. This first probability distribution indicates the probability that each text unit in the text unit set is a corresponding target text unit. In this embodiment, the text unit set is a vocabulary containing different words (tokens / text units). The first probability distribution represents the probability that each word in the vocabulary is a target text unit in this inference. Assuming there are N text units in the text unit set, then the first probability distribution contains N probability values, each indicating the probability that a text unit is a target text unit corresponding to the first probability distribution. It is understood that the target text unit obtained in this inference is selected from the vocabulary based on the first probability distribution obtained in this inference. For example, the selection method may be to select the word with the highest probability value as the target text unit. This embodiment does not limit the method of selecting target words.

[0107] Figure 4 This is an example of the first reasoning result. For example... Figure 4 As shown, the first inference result contains 7 target text units (token1 to token7) and 7 first probability distributions (p1 to p7). Specifically, p1 corresponds to token1, p2 to token2, p3 to token3, p4 to token4, p5 to token5, p6 to token6, and p7 to token7. As mentioned above, each first probability distribution includes N probability values ​​(N is the number of text units in the text unit set).

[0108] Step 220: The terminal device sends the first inference result to the cloud device.

[0109] Specifically, the terminal device converts the first inference result output by the first model into data that can be transmitted over the network, and then sends it to the cloud device through the network.

[0110] In some embodiments, the input and output data of the first model are both tensor data. This type of data cannot be directly transmitted over the network. Therefore, it is necessary to first convert the tensor data into data that can be transmitted over the network (network transmission data).

[0111] Step 230: After receiving the first inference result sent by the terminal device, the cloud device uses the second model to infer the first inference result and obtain the second inference result.

[0112] Specifically, the cloud device receives the first inference result sent by the terminal device as data that can be transmitted over the network. The cloud device first converts the data that can be transmitted over the network into data that the second model can recognize, and then uses it as input data for the second model to perform inference.

[0113] Specifically, in some embodiments, the input and output values ​​of the second model are both tensor data, and the data that can be transmitted over the network cannot be recognized by the second model. Therefore, after the cloud device receives the first inference result data sent by the terminal device, it first converts the first inference result data into tensor data.

[0114] In some embodiments, the second model infers the first text sequence in the first inference result, that is, the input data of the second model includes the first text sequence in the first inference result.

[0115] In some embodiments, similar to the first inference result, the second inference result includes a second text sequence consisting of multiple text units (tokens) and multiple second probability distributions. Each of the multiple second probability distributions corresponds one-to-one with a multiple token. The second probability distributions are used to indicate the probability that each text unit in the set of text units corresponds to a specific token.

[0116] Specifically, in some embodiments, the second model is an autoregressive model, which, after one inference, yields a new token and a new probability distribution (the second probability distribution). Therefore, after using the second model to infer the first text sequence in the first inference result, the resulting second inference result will include γ+1 tokens and γ+1 second probability distributions. Each of the γ+1 second probability distributions corresponds one-to-one with one of the γ+1 tokens. These γ+1 tokens constitute the second text sequence. The second text sequence differs from the first text sequence in that it includes a new token at the end. As mentioned above, γ is the inference configuration parameter of the first model.

[0117] Figure 5 This is an example of the second reasoning result. For example... Figure 5 As shown, the second inference result contains 8 tokens (token1 to token8) and 8 first probability distributions (q1 to q8). Specifically, q1 corresponds to token1, q2 to token2, q3 to token3, q4 to token4, q5 to token5, q6 to token6, q7 to token7, and q8 to token8. Tokens 1 to 7 are identical to those in the first text sequence, while token8 is a new token inferred by the second model. q1 to q8 represent the new probability distributions inferred by the second model.

[0118] Step 240: The cloud device uses the second inference result and the first inference result to verify the first inference result and obtain the verification result.

[0119] Since the cloud device uses the second model to further infer the first inference result output by the first model, and the inference accuracy of the second model is higher than that of the first model, the accuracy of the first inference result can be judged by combining the first and second inference results.

[0120] In one example, the validation result includes indication information and a replacement token (or a new token). The indication information indicates whether the target text unit output by the first model is accepted. The indication information may be, for example, "accept" or "reject". If the indication information indicates that a target text unit is not accepted, the validation result includes a replacement token for that unaccepted target text unit; if all target text units are accepted, the validation result includes a new token obtained after inference by the second model. The aforementioned indication information and replacement token (or new token) are collectively referred to as the validation result information.

[0121] After receiving the verification result, the terminal device can update the first text sequence in the first inference result according to the prompts in the instruction information, and use the updated first text sequence as the new input data of the first model for the first model to perform the first inference in the next round of inference.

[0122] For example, the instruction information could be a sequence: [accept, accept, accept, reject, accept, reject, accept], where the verification result includes the replacement token "in". After the cloud device sends the sequence of instruction information and the replacement token to the terminal device, the terminal device determines that the first rejected target text unit is token4, replaces the original "on" with "in", and discards tokens 5 through 7. The text sequence consisting of the original tokens 1 through 3 and the text unit "in" is used as the new input data for the first model.

[0123] For example, with Figure 4 For example, the indication information can be a sequence: [accept, accept, accept, accept, accept, accept, accept], and the verification result includes the new token "and". After the cloud device sends the sequence of indication information and the new token to the terminal device, the terminal device determines that all target text units have been accepted, adds "and" to the end of the original first text sequence to form a new text sequence, and uses this new text sequence as the new input data for the first model.

[0124] For example, with Figure 4 For example, the instruction message could be: "Accept All". "Accept All" means that the entire target text sequence is accepted, and the verification result contains the new token. Subsequent actions are similar to the aforementioned embodiments and will not be repeated here.

[0125] In another example, the verification result is a text sequence containing multiple tokens. If every target text unit in the first text sequence is accepted, the text sequence comprises γ+1 text units, where the first γ text units are identical to the first text sequence, and the last text unit is a new token inferred by the second model. If one or more target text units in the first text sequence are rejected, the text sequence comprises h+1 text units, where the first h text units are identical to the first h text units in the first text sequence, and the last text unit is a replacement token obtained by the cloud device after resampling the first rejected target text unit. This replacement token is used to replace the first rejected target text unit. h represents the number of target text units preceding the first rejected target text unit in the first text sequence.

[0126] In some embodiments, if all target text units are accepted, the cloud device sends the second text sequence from the second inference result to the terminal device; or the cloud device sends only the last token in the second text sequence and indication information to the terminal device, the indication information indicating that all target text units are accepted. If a target text unit is rejected, the cloud device sends the terminal device indication information and a replacement token obtained by resampling the first rejected target text unit, wherein the indication information indicates the target text unit that the replacement token is used to replace or the position of the replaced target text unit in the first text sequence; or the cloud device replaces the first rejected target text unit with the replacement token to form a new text sequence, sends the new text sequence to the terminal device, and the terminal device directly uses the received text sequence as new input data for inference; wherein the text sequence includes all target text units before the first rejected target text unit in the first text sequence and the replacement token.

[0127] In some embodiments, if the verification result sent from the cloud device to the terminal device includes indication information and a text unit (replacement token or new token), the indication information is used to indicate the position of the text unit. Through this indication information, the terminal device can determine the position of the text unit in the first text sequence, thereby determining how to update the first text sequence and using the updated first text sequence as new input data. For example, assuming the text unit is "in", the indication information indicates the position of the text unit as follows: Figure 4 If token4 is shown, the terminal device will replace "on" with "in" in token4 and discard tokens 5 through 7. The text sequence consisting of the original tokens 1 through 3 and the text unit "in" will be used as the new input data for the first model.

[0128] Whether a target text unit is accepted can be determined by comparing a first probability distribution and a second probability distribution. Specifically, in some embodiments, if the probability value of a target text unit in the second probability distribution is less than its probability value in the first probability distribution, then the target text unit is rejected.

[0129] Specifically, to reduce false positives, in some embodiments, for a given target text unit, if the ratio of its probability value in the second probability distribution to its probability value in the first probability distribution is less than r, then the target text unit is rejected. r is a random number conforming to a normal distribution, and r < 1.

[0130] In some embodiments, a comparison-by-comparison approach can be used to determine whether to accept the target text unit. The following, in conjunction with... Figure 4 and Figure 5 Describe the process of comparing each item one by one. It should be noted that only judgment is required. Figure 4 Whether tokens 1 through 7 are accepted.

[0131] In some embodiments, tokens can be evaluated sequentially from token1 to token7. First, it is determined whether token1 is accepted. This involves comparing p1 and q1. Assuming the probability value for "the" in p1 is p11, and the probability value for "the" in q1 is q11, then if q11 is less than p11, the target text unit token1 is rejected. Furthermore, to reduce false positives, if p11 and q11 satisfy q11 / p11 < r, then the target text unit token1 is rejected. Conversely, if p11 < q11, then the target text unit token1 is accepted.

[0132] Similarly, when determining whether token2 should be accepted, assuming the probability of "dog" in p2 is p22 and the probability of "dog" in q2 is q22, if q22 < p22, or q22 / p22 < r, then the target text unit token2 is rejected. Conversely, if q22 > p22, then the target text unit token2 is accepted.

[0133] The verification of token3, token4, token5, token6, and token7 is similar to that of token1 and token2, and will not be repeated here.

[0134] As mentioned above, Figure 4 After all seven target text units in the first text sequence shown have been verified, if all verification results are "acceptable," then in the next round of inference, the input to the first model includes the following: Figure 5 The tokens shown are 1 through 8. Assuming that one or more target text units are rejected after evaluation, the cloud device will resample the first rejected target text unit to obtain a replacement token. In the next round of inference, the input data of the first model includes the target text units preceding the first rejected target text unit and the replacement token. For example, if token 7 is rejected, the cloud device will resample the 7th target text unit to obtain a replacement token. In the next round of inference, the input data of the first model includes a text sequence consisting of token 1, token 2, token 3, token 4, token 5, token 6, and the replacement token.

[0135] It is worth noting that when multiple target text units are rejected, all target text units following the first rejected target text unit in the first text sequence may have inaccurate reasoning and therefore will all be rejected.

[0136] In some embodiments, when verifying tokens in the order of token1 to token7, verification of subsequent tokens stops after the first target text unit is rejected. This verification rule can save verification time.

[0137] Step 250: Determine the input data for the next round of inference based on the verification results sent by the cloud device.

[0138] As described above, the first model is an autoregressive model. During inference, it uses the result obtained from the previous inference as input for new inference. In this embodiment of the invention, after the first model performs one round of inference, the second model is used to correct the inference result of the first model, making the inference result of the first model more accurate. The updated inference result is used as input for the next round of inference, which can make the next round of inference result more accurate.

[0139] As mentioned earlier, if the verification result contains indication information indicating whether the target text unit is accepted, a replacement token for the first unaccepted target text unit (or a new token provided when all target text units are accepted), then the terminal device, based on the indication information and the replacement token (or new token), forms a new text sequence as new input data for the next round of reasoning.

[0140] If the verification result is a text sequence, then the text sequence can be directly used as input data for the next round of reasoning.

[0141] The data processing method of this invention deploys a first model on a terminal device and a second model on a cloud device. On the one hand, it utilizes idle computing power on the terminal side; on the other hand, it transfers some inference tasks of the second model to the first model, and uses the second model with higher inference accuracy to verify and correct the inference results of the first model, which can improve the accuracy of the inference results of the first model; at the same time, the second model can obtain an inference result containing multiple tokens in one inference, which improves the inference efficiency of the second model and reduces the inference cost on the cloud side.

[0142] In the above embodiments, the verification action is performed by a cloud device. It is understood that the verification action can also be performed by a terminal device. If the verification action is performed by a terminal device, then after the cloud device obtains the second inference result using the second model, it sends the second inference result to the terminal device. The terminal device then uses the second inference result to verify the first inference result, obtaining a verification result. After obtaining the verification result, the terminal device determines the new input data for the first model based on the verification result. Specifically, the cloud device sends the second probability distribution from the second inference result to the terminal device. The terminal device uses the second probability distribution and the first probability distribution to verify the first text sequence, and determines the input data for the next inference based on the verification result.

[0143] In some embodiments, when the terminal device sends the first inference result to the cloud device, it sends the generated first probability distribution to the cloud device as soon as it is generated. As described above, in one round of inference of the first model, γ inferences are performed, and each inference yields a new token and a first probability distribution. After completing one round of inference, the first model obtains γ tokens and γ first probability distributions. Since the first probability distribution contains a large amount of data, this embodiment of the invention sends each generated first probability distribution to the cloud device. This parallel transmission method avoids sending all first probability distributions simultaneously and improves the transmission efficiency of the first probability distribution.

[0144] As mentioned earlier, the first probability distribution obtained after inference by the first model is a data format (tensor data) that the large language model can recognize, and cannot be directly transmitted over the network. Therefore, in some embodiments, before sending the first probability distribution, the terminal device converts it into network-transmittable data, forming a first network data packet, which contains the data of the first probability distribution (network-transmittable data). After receiving the first network data packet, the cloud device then converts it into tensor data that the second model can recognize.

[0145] In this embodiment of the invention, the transmission of the first probability distribution is performed in parallel. Therefore, the cloud device can also perform the conversion of the network-transmittable data of the first probability distribution in parallel. Since converting the network-transmittable data of the first probability distribution into tensor data takes a certain amount of time, using a parallel conversion method can improve the efficiency of data conversion.

[0146] In some embodiments, when the terminal device sends the first inference result to the cloud device, it sends the generated first text sequence to the cloud device after all target text units have been generated and formed a first text sequence.

[0147] As mentioned earlier, the first text sequence obtained after inference by the first model is in a data format (tensor data) that the large language model can recognize, and cannot be directly transmitted over the network. Therefore, before sending the first text sequence, the terminal device converts it into network-transmittable data, forming a second network data packet, which contains the data of the first text sequence (network-transmittable data). After receiving the second network data packet, the cloud device then converts it into tensor data that the second model can recognize.

[0148] In this embodiment of the invention, the first text sequence mainly contains token data, and the amount of token data is very small. Therefore, the network transmission time is short, and the time spent on data conversion of token data is also small. Thus, all token data can be transmitted and converted at once.

[0149] Since the inference of the second model relies solely on the first text sequence obtained from the inference of the first model, the cloud device, upon receiving the second network data packet, can quickly convert it into tensor data that the second model can recognize, and then use the second model for inference. While the second model is performing inference, the cloud device may not have yet completed the data transformation of the last received first probability distribution. Therefore, in this embodiment of the invention, the inference of the second model and the data transformation of the last first probability distribution can be performed in parallel, thereby reducing the time delay caused by data transformation.

[0150] In some embodiments, in order to enable the data of the first probability distribution received by the cloud device to correspond to the target text unit, each first probability distribution is provided with an identifier that can indicate the corresponding target text unit.

[0151] Specifically, in some embodiments, the first network data packet also includes a first identifier for indicating the corresponding target text unit, and the cloud device can determine which target text unit the data of the first probability distribution contained in the currently received first network data packet corresponds to based on the first identifier.

[0152] As mentioned earlier, the transmission and reception of the first probability distribution are performed in parallel; the terminal device sends one first probability distribution to the cloud device as soon as it generates it. This transmission method is to reduce the latency caused by transmitting the first probability distribution. Since the first probability distribution corresponds to a target text unit, in order for the cloud device to know which target text unit it corresponds to when it receives the data of the first probability distribution, this embodiment of the invention sets a first identifier for the first probability distribution. Through this first identifier, the cloud device can determine which target text unit in the first text sequence the currently received data of the first probability distribution belongs to. Specifically, in some embodiments, the first identifier can be as follows: Figure 3 The tokens shown are token1 to token7.

[0153] Specifically, in some embodiments, the first network data packet also includes a second identifier for indicating the corresponding inference round, and the cloud device can determine in which round of inference the data of the first probability distribution contained in the currently received first network data packet was generated based on the second identifier.

[0154] As mentioned earlier, the first model performs γ inferences in one round of inference. Completing an inference task requires one or more rounds of inference. Therefore, to enable the cloud device to know in which round of inference the data of the first probability distribution was generated when it receives it, this embodiment of the invention sets a second identifier for the first probability distribution. Through this second identifier, the cloud device can determine in which round of inference the currently received data of the first probability distribution was generated by the first model. Specifically, in some embodiments, the second identifier can be turn1 to turn K, where K is the number of inference rounds required to perform a certain inference task. K is a positive integer greater than or equal to 1.

[0155] Specifically, in some embodiments, the first network data packet also includes a third identifier for indicating the corresponding inference task, and the cloud device can determine from the third identifier which inference task the data of the first probability distribution contained in the currently received first network data packet was obtained.

[0156] Terminal devices may execute multiple inference tasks simultaneously. Therefore, by setting a third identifier, the cloud device can determine which inference task the currently received first probability distribution belongs to.

[0157] In some embodiments, the third identifier includes a device identifier. This device identifier uniquely identifies the terminal device performing the inference task. The cloud device can use this device identifier to determine which terminal device sent the data of the first probability distribution contained in the currently received first network data packet. Compared to directly sending an inference request to the cloud device and having the terminal device sending the inference request return a request ID, the first network data packet in this embodiment of the invention carries a device identifier that indicates the source of the request, eliminating the need for the cloud device to return a request ID, saving the time required to return the request ID, and improving data transmission efficiency.

[0158] In some embodiments, the third identifier includes an inference task identifier. As mentioned above, the same terminal device may execute multiple inference tasks; therefore, in order to distinguish multiple inference tasks, a unique inference task identifier can be attached to each inference task.

[0159] In some embodiments, the inference task identifier can be combined with the aforementioned device identifier to form a third identifier that can uniquely identify the inference task. For example, the MAC address of the terminal device can be used as the device identifier, and request1 to requestM can be used as the inference task identifier. In this way, an inference task can be uniquely identified, and the cloud device can know which terminal device the inference task belongs to. In other embodiments, a device identifier that can uniquely identify the terminal device can also be generated by machine, and the machine-generated device identifier can be combined with the inference task identifier to form a third identifier that can uniquely identify the inference task.

[0160] In this embodiment of the invention, in one round of inference, the terminal device's γ-fold inference, data conversion and transmission of the first probability distribution, and conversion and transmission of the first text sequence are performed in parallel; the cloud device's data conversion of the first probability distribution, data conversion of the first text sequence, and inference of the second model are also performed in parallel; this approach can reduce the time delay caused by network transmission and data conversion, thereby improving the efficiency of end-to-cloud collaborative inference.

[0161] To better understand the data processing method of this invention, the following will use... Figure 6 Taking the reasoning scenario shown as an example, combined with Figure 7 This invention describes a data processing method according to an embodiment of the invention. It is assumed that the reasoning task is to answer questions posed by a user, such as... Figure 6 The problem is shown. After... Figure 7 After the process shown, the user can obtain the following: Figure 8 The answer shown.

[0162] like Figure 7 As shown, the specific processing flow is as follows:

[0163] Step 710: The terminal device obtains the inference task requested by the user.

[0164] Specifically, users locate the application icon through the user interface of their terminal device, tap the icon, and enter their question. The user's question might be something like... Figure 6 As shown.

[0165] Step 720: The terminal device uses the first model to perform one inference on the input data to obtain a target text unit and a first probability distribution.

[0166] Specifically, the input data for the initial inference is obtained through the user's question. The input data for subsequent inferences is obtained from the results of the previous inference. It's important to note that the input data for the first inference in the same round is obtained from the inference results of the second model. The input data for subsequent inferences in the same round is obtained from the previous inference results of the first model.

[0167] Step 730: The terminal device processes the obtained first probability distribution to obtain a first network data packet. The first network data packet is data that can be transmitted over the network.

[0168] Specifically, the first network data packet also includes a first identifier that can identify the corresponding target text unit, a second identifier that can identify the current inference round, and a third identifier that can identify the current inference request / inference task.

[0169] Step 740: The terminal device sends the first network data packet it has obtained to the cloud device.

[0170] Step 750: The cloud device determines, based on the second identifier in the first network data packet it has received, whether the data of the first probability distribution it has received is the last one in this round of inference. If so, proceed to step 760.

[0171] Specifically, after receiving the first network data packet, the cloud device obtains the second identifier from the first network data packet. Using the second identifier, it can determine whether the currently received first probability distribution is the last probability distribution in this round of inference.

[0172] Step 760: The cloud device processes all the first network data packets to obtain tensor data of all the first probability distributions.

[0173] Specifically, the cloud device can obtain the first identifier, the second identifier, and the third identifier from the first network data packet. The corresponding inference task / inference request, inference round, and target text unit can be determined through the three identifiers. Therefore, all the data of the first probability distribution can be recombined into tensor data through the first identifier, the second identifier, and the third identifier.

[0174] Step 770: The terminal device determines whether the γth inference of this round of inference has been completed. If yes, proceed to step 780; otherwise, proceed to step 720.

[0175] Step 780: The terminal device processes the first text sequence composed of target text units to obtain a second network data packet. The second network data packet is data that can be transmitted over a network.

[0176] Step 790: The terminal device sends a second network data packet to the cloud device.

[0177] Step 711: The cloud device processes the second network data packet to obtain the tensor data of the first text sequence.

[0178] Step 712: The cloud device uses the second model to reason about the first text sequence and obtains the second reasoning result.

[0179] Specifically, the second inference result contains multiple second probability distributions and multiple text units; the multiple text units correspond one-to-one with the multiple second probability distributions.

[0180] Step 713: The cloud device verifies the first inference result using the first probability distribution in the first inference result and the second probability distribution in the second inference result, and obtains the verification result.

[0181] Specifically, the verification result can be a text sequence or data including indication information and a replacement token (or a new token). The indication information is used to indicate whether the corresponding target text unit is accepted.

[0182] Step 714: The cloud device sends the verification result to the terminal device.

[0183] Specifically, before sending the verification result to the terminal device, the cloud device converts the verification result into data that can be transmitted over the network.

[0184] Step 715: The terminal device determines whether the current inference result is the final inference result. If yes, proceed to step 717; otherwise, proceed to step 716.

[0185] Step 716: The terminal device determines the input data for the next round of inference based on the verification result. Then it proceeds to step 720.

[0186] Step 717: The terminal device displays to the user, as follows Figure 8 The final reasoning result is shown.

[0187] In this embodiment of the invention, speculative reasoning is performed using a first model on a terminal device, and then the reasoning results are verified using a second model on a cloud device. On the one hand, some reasoning tasks are offloaded to the terminal device, thus making use of idle computing power on the terminal side. On the other hand, the second model on the cloud side can obtain multiple tokens after one reasoning, which improves the reasoning efficiency of the cloud device, thereby improving the reasoning efficiency of the entire reasoning task and reducing the reasoning cost on the cloud side.

[0188] In this embodiment of the invention, by transmitting the first probability distribution data and the first text sequence data in parallel, the time delay caused by transmitting the first probability distribution is reduced. Data processing for multiple first probability distributions is performed in parallel, reducing the time delay caused by data conversion. The inference of the second model and the data processing of the last first probability distribution are also performed in parallel, further reducing the time delay caused by processing the first probability distribution.

[0189] The data processing method of the present invention has been described above. The following text, in conjunction with... Figures 9-13This describes embodiments of terminal devices, cloud devices, and end-to-cloud collaborative systems used to perform the corresponding methods described above.

[0190] Figure 9 This diagram illustrates a schematic block diagram of a terminal device 900 according to an embodiment of the present invention. The terminal device 900 includes:

[0191] The reasoning module 910 is used to reason about the input data using the first model to obtain the first reasoning result.

[0192] The transceiver module 930 is used to send the first inference result to the cloud device and to receive the verification result sent by the cloud device.

[0193] The data processing module 920 is used to determine the input data for the next round of inference based on the verification results.

[0194] In this embodiment of the invention, by deploying the first model on a terminal device and the second model on a cloud device, on the one hand, idle computing power on the terminal side is utilized; on the other hand, by transferring some inference tasks of the second model to the first model, and using the second model to verify and correct the inference results of the first model, the accuracy of the inference results of the first model can be improved; at the same time, the second model can obtain an inference result containing multiple tokens in one inference, which can improve the inference efficiency of the second model and reduce the computing power cost on the cloud side.

[0195] In some embodiments, the first inference result includes a first text sequence and multiple first probability distributions. The first text sequence includes multiple sequentially arranged target text units. Each target text unit corresponds one-to-one with one of the multiple first probability distributions. The first probability distributions indicate the probability that each text unit in the text unit set is a corresponding target text unit. In this embodiment, the text unit set can be understood as a vocabulary (or dictionary) containing many words. The first probability distribution includes N probability values, each corresponding to a word in the vocabulary, used to identify the probability that the word is a target text unit. Here, N represents the number of words collected in the vocabulary.

[0196] In some embodiments, the first model is an autoregressive model. The first model performs one inference operation each time it obtains a new target text unit. Therefore, assuming the number of target text units obtained by the first model in one round of inference is γ, then the number of inference operations required for the first model to perform one round of inference is γ. Each inference operation yields a new target text unit and a new first probability distribution.

[0197] The reasoning module 910 is specifically used to perform γ inferences using the first model in one round of reasoning to obtain the first reasoning result.

[0198] Specifically, in one round of reasoning, the reasoning module 910 uses the target text units obtained from previous reasoning as input to the first model for this reasoning. If it is the first round of reasoning, it extracts prompt words from the user's question and uses the extracted prompt words as input to the first model for this reasoning.

[0199] In some embodiments, the transceiver module 930 is specifically configured to send the currently generated first probability distribution to the cloud device each time the first model generates a first probability distribution. Since the second model only needs the first text sequence output by the first model as input data during inference, and is unrelated to the first probability distribution; and the first probability distribution has a large data volume, the transmission of the first probability distribution can be performed in parallel with the inference of the first model, the transmission of the first text sequence, and the inference of the second model. This reduces the time delay caused by transmitting the first probability distribution.

[0200] In some embodiments, the data processing module 920 is further configured to perform data processing on the currently generated first probability distribution each time the first model generates a first probability distribution, to obtain a first network data packet. The first network data packet is data that can be transmitted over a network.

[0201] The first probability distribution output from the first model is tensor data, which can be recognized by the first and second models, but cannot be directly transmitted over the network. Therefore, before sending the first probability distribution to the cloud device, the data processing module 920 converts the first probability distribution into data that can be transmitted over the network.

[0202] In some embodiments, the first network data packet sent to the cloud device includes a first identifier, so that the cloud device can determine the target text unit corresponding to the currently received first probability distribution based on the first identifier.

[0203] In some embodiments, the first network data packet sent to the cloud device includes a second identifier, so that the cloud device can determine the inference round corresponding to the currently received first probability distribution based on the second identifier.

[0204] In some embodiments, the first network data packet sent to the cloud device includes a third identifier, so that the cloud device can determine the inference request corresponding to the currently received first probability distribution based on the third identifier.

[0205] In some embodiments, the third identifier includes a device identifier, which is used to uniquely identify the terminal device performing the reasoning task.

[0206] In some embodiments, the third identifier includes a reasoning task identifier, which is used to identify the reasoning task when multiple reasoning tasks are performed.

[0207] In some embodiments, the transceiver module 930 is specifically used to send the first text sequence to the cloud device after the first model outputs the first text sequence.

[0208] Since the first text sequence has a small data volume, the time required to transmit the first text sequence over the network is less, so it is not necessary to send each target text unit separately, thus reducing the burden on cloud devices to process / transform data.

[0209] In some embodiments, the data processing module 920 is further configured to perform data processing on the first text sequence after the first model outputs the first text sequence to obtain a second network data packet. The second network data packet is data that can be transmitted over a network. The second network data packet includes the data of the first text sequence.

[0210] In some embodiments, the data processing module 920 is specifically used to: directly use the verification result as input data for the next round of inference.

[0211] In some embodiments, the data processing module 920 is specifically used to: determine the input data for the next round of inference based on the received verification result.

[0212] Figure 10 As shown in the figure, this embodiment of the invention also provides a terminal device 1000, which includes a processor 1010, a memory 1020, and a transceiver 1030. The processor 1010, memory 1020, and transceiver 1030 communicate with each other through an internal connection path. The memory 1020 is used to store instructions, and the processor 1010 is used to execute the instructions stored in the memory 1020 to control the transceiver 1030 to receive or transmit signals. Specifically, the processor 1010 is used to process data to be processed using a first model to obtain a first inference result. The transceiver 1030 is used to send the first inference result to a cloud device; the transceiver 1030 is also used to receive a verification result sent by the cloud device; the verification result is obtained by the cloud device verifying the first inference result using a second inference result and the first inference result, and the second inference result is obtained by the cloud device inferring the first inference result using a second model. The processor 1010 is also used to determine the input data for the next round of inference based on the verification result.

[0213] In this embodiment of the invention, by deploying the first model on a terminal device and the second model on a cloud device, on the one hand, the idle computing power of the terminal device is utilized; on the other hand, by transferring some of the inference tasks of the second model to the first model, and using the second model to verify and correct the inference results of the first model, the accuracy of the inference results of the first model can be improved; at the same time, the second model can obtain an inference result containing multiple tokens in one inference, which can improve the inference efficiency of the second model and reduce the computing power cost on the cloud side.

[0214] In some embodiments, the first inference result includes a first text sequence and multiple first probability distributions. The first text sequence includes multiple sequentially arranged target text units. Each target text unit corresponds one-to-one with one of the multiple first probability distributions. The first probability distributions indicate the probability that each text unit in the text unit set is a corresponding target text unit. In this embodiment, the text unit set can be understood as a vocabulary (or dictionary) containing many words. The first probability distribution includes N probability values, each corresponding to a word in the vocabulary, used to identify the probability that the word is a target text unit. Here, N represents the number of words collected in the vocabulary.

[0215] In some embodiments, the first model is an autoregressive model. The first model performs one inference operation each time it obtains a new target text unit. Therefore, assuming the number of target text units obtained by the first model in one round of inference is γ, then the number of inference operations required for the first model to perform one round of inference is γ. Each inference operation yields a new target text unit and a new first probability distribution.

[0216] The processor 1010 is specifically used to perform γ inferences using the first model in one round of inference to obtain the first inference result.

[0217] Specifically, in one round of inference, the processor 1010 uses the target text units obtained from previous inference as input to the first model for this inference. If it is the first round of inference, it extracts prompt words from the user's question and uses the extracted prompt words as input to the first model for this inference.

[0218] In some embodiments, transceiver 1030 is specifically configured to send the currently generated first probability distribution to the cloud device each time the first model generates a first probability distribution. Since the second model only needs the first text sequence output by the first model as input data during inference, and is unrelated to the first probability distribution; and the first probability distribution has a large data volume, the transmission of the first probability distribution can be performed in parallel with the inference of the first model, the transmission of the first text sequence, and the inference of the second model. This reduces the time delay caused by transmitting the first probability distribution.

[0219] In some embodiments, the processor 1010 is further configured to perform data processing on the currently generated first probability distribution each time the first model generates a first probability distribution, to obtain a first network data packet. The first network data packet is data that can be transmitted over a network.

[0220] The first probability distribution output from the first model is tensor data, which can be recognized by the first and second models, but cannot be directly transmitted over the network. Therefore, before sending the first probability distribution to the cloud device, the processor 1010 converts the first probability distribution into data that can be transmitted over the network.

[0221] In some embodiments, the first network data packet sent to the cloud device includes a first identifier, so that the cloud device can determine the target text unit corresponding to the currently received first probability distribution based on the first identifier.

[0222] In some embodiments, the first network data packet sent to the cloud device includes a second identifier, so that the cloud device can determine the inference round corresponding to the currently received first probability distribution based on the second identifier.

[0223] In some embodiments, the first network data packet sent to the cloud device includes a third identifier, so that the cloud device can determine the inference request corresponding to the currently received first probability distribution based on the third identifier.

[0224] In some embodiments, the third identifier includes a device identifier, which is used to uniquely identify the terminal device performing the reasoning task.

[0225] In some embodiments, the third identifier includes a reasoning task identifier, which is used to identify the reasoning task when multiple reasoning tasks are performed.

[0226] In some embodiments, transceiver 1030 is specifically used to send the first text sequence to a cloud device after the first model outputs the first text sequence.

[0227] Since the first text sequence has a small data volume, the time required to transmit the first text sequence over the network is less, so it is not necessary to send each target text unit separately, thus reducing the burden on cloud devices to process / transform data.

[0228] In some embodiments, the processor 1010 is further configured to perform data processing on the first text sequence after the first model outputs the first text sequence to obtain a second network data packet. The second network data packet is data that can be transmitted over a network. The second network data packet includes the data of the first text sequence.

[0229] In some embodiments, the processor 1010 is specifically configured to: directly use the verification result as input data for the next round of inference.

[0230] In some embodiments, the processor 1010 is specifically configured to: determine the input data for the next round of inference based on the received verification result.

[0231] It should be understood that Figure 9 The terminal device 900 shown Figure 10 The terminal device 1000 shown can be used to execute operations or processes related to the terminal device in the above method embodiments, and the operations and / or functions of each module in the terminal device 900 or the terminal device 1000 are respectively to implement the corresponding processes in the above method embodiments. For the sake of brevity, they will not be described in detail here.

[0232] Figure 11 A schematic block diagram of a cloud device 1100 according to an embodiment of the present invention is shown. The cloud device 1100 includes:

[0233] The transceiver module 1110 is used to receive the first inference result sent by the terminal device; the first inference result is obtained by the terminal device using the first model to infer the input data.

[0234] The inference module 1130 is used to infer the first inference result received by the cloud device using the second model to obtain the second inference result.

[0235] The verification module 1140 is used to verify the first inference result using the second inference result and the received first inference result to obtain a verification result.

[0236] The transceiver module 1110 is also used to send the verification result to the terminal device.

[0237] In this embodiment of the invention, by offloading part of the inference task to the first model on the terminal device, the idle computing power of the terminal device is utilized. Using the second model on the cloud device to verify and correct the inference results of the first model can improve the accuracy of the inference results of the first model; simultaneously, the second model on the cloud device can obtain an inference result containing multiple tokens in a single inference operation, which can improve the inference efficiency of the second model and reduce the computing power cost on the cloud side.

[0238] In some embodiments, the first inference result includes: a first text sequence and a plurality of first probability distributions. The first text sequence includes a plurality of target text units arranged in sequence; the plurality of target text units correspond one-to-one with the plurality of first probability distributions. The first probability distributions are used to indicate the probability that each text unit in the set of text units is a corresponding target text unit.

[0239] In some embodiments, the cloud device further includes a data processing module 1120. The data processing module 1120 is configured to convert the received first probability distribution data into data that can be recognized by the second model each time a first probability distribution data is received.

[0240] In some embodiments, the data processing module 1120 is further configured to convert the data of the first text sequence into data that the second model can recognize after receiving the data of the first text sequence.

[0241] In some embodiments, the reasoning module 1130 is specifically used to: use a second model to reason about the first text sequence in the first reasoning result to obtain a second reasoning result.

[0242] In some embodiments, the second inference result includes a second text sequence and a plurality of second probability distributions. The second text sequence includes a plurality of text units arranged in sequence. Each text unit corresponds one-to-one with one of the plurality of second probability distributions. The second probability distributions are used to indicate the probability that each text unit in the set of text units is a corresponding target text unit.

[0243] The second text sequence contains one more text unit than the first text sequence contains one more target text unit. Specifically, the second text sequence includes all the target text units from the first text sequence and a new text unit at the end of the second text sequence. This new text unit is obtained by the second model through one inference operation.

[0244] In some embodiments, the verification module 1140 is specifically used to verify multiple target text units using multiple second probability distributions and multiple first probability distributions to obtain verification results.

[0245] In some embodiments, the verification result includes indication information indicating whether the target text unit output by the first model is accepted. If the indication information indicates that a target text unit is not accepted, the verification result also includes a replacement token for the unaccepted target text unit. If all target text units are accepted, the verification result also includes a new token obtained after inference by the second model.

[0246] In some embodiments, the verification result is a text sequence containing multiple tokens. If each target text unit in the first text sequence is accepted, the text sequence comprises γ+1 text units, where the first γ text units are identical to the first text sequence, and the last text unit is a new token inferred by the second model. If one or more target text units in the first text sequence are rejected, the text sequence comprises h+1 text units, where the first h text units are identical to the first h text units in the first text sequence, and the last text unit is a replacement token obtained by the cloud device after resampling the first rejected target text unit. h represents the number of target text units preceding the first rejected target text unit in the first text sequence.

[0247] In some embodiments, the cloud device further includes a scheduling module 1150. The scheduling module 1150 is used to schedule the inference module 1130 to perform cloud-side inference using the second model after the cloud device receives the first text sequence.

[0248] Specifically, in some examples, after the data processing module 1120 converts the received second network data packet into tensor data that the second model can recognize, it notifies the scheduling module 1150 that the token data (first text sequence) is ready. After receiving this notification, the scheduling module 1150 schedules the inference module 1130 to perform cloud-side inference on the token data (first text sequence) using the second model to obtain the second inference result.

[0249] In some embodiments, the scheduling module 1150 is further configured to, after receiving the last data of the first probability distribution, use the scheduling verification module 1140 to verify the first inference result using the first inference result and the second inference result.

[0250] Specifically, in some examples, after the data processing module 1120 converts all received first network data packets into tensor data that the second model can recognize, it notifies the scheduling module 1150 that the first probability distribution data is ready. After receiving this notification, the scheduling module 1150 schedules the verification module 1140 to verify the target text units in the first text sequence using the first and second probability distributions, and obtains the verification result data.

[0251] like Figure 12As shown in the figure, this embodiment of the invention also provides a cloud device 1200, which includes a processor 1210, a memory 1220, and a transceiver 1230. The processor 1210, memory 1220, and transceiver 1230 communicate with each other through an internal connection path. The memory 1220 stores instructions, and the processor 1210 executes the instructions stored in the memory 1220 to control the transceiver 1230 to receive or transmit signals. The transceiver 1230 receives a first inference result sent by a terminal device, which is obtained by the terminal device using a first model to infer input data. The processor 1210 uses a second model to infer the received first inference result to obtain a second inference result. The processor 1210 also verifies the first inference result using the first and second inference results to obtain a verification result. The transceiver 1230 further sends the verification result to the terminal device.

[0252] In this embodiment of the invention, by deploying the first model on a terminal device and the second model on a cloud device, on the one hand, the idle computing power of the terminal device is utilized; on the other hand, by transferring some of the inference tasks of the second model to the first model, and using the second model to verify and correct the inference results of the first model, the accuracy of the inference results of the first model can be improved; at the same time, the second model can obtain an inference result containing multiple tokens in one inference, which can improve the inference efficiency of the second model and reduce the computing power cost on the cloud side.

[0253] In some embodiments, the first inference result includes: a first text sequence and a plurality of first probability distributions. The first text sequence includes a plurality of target text units arranged in sequence; the plurality of target text units correspond one-to-one with the plurality of first probability distributions. The first probability distributions are used to indicate the probability that each text unit in the set of text units is a corresponding target text unit.

[0254] In some embodiments, the processor 1210 is further configured to convert the received first probability distribution data into data that the second model can recognize each time a first probability distribution data is received.

[0255] In some embodiments, the processor 1210 is further configured to, after receiving data of the first text sequence, convert the data of the first text sequence into data that the second model can recognize.

[0256] In some embodiments, the processor 1210 is specifically configured to: use a second model to reason about a first text sequence in a first reasoning result to obtain a second reasoning result.

[0257] In some embodiments, the second inference result includes a second text sequence and a plurality of second probability distributions. The second text sequence includes a plurality of text units arranged in sequence. Each text unit corresponds one-to-one with one of the plurality of second probability distributions. The second probability distributions are used to indicate the probability that each text unit in the set of text units is a corresponding target text unit.

[0258] The second text sequence contains one more text unit than the first text sequence contains one more target text unit. Specifically, the second text sequence includes all the target text units from the first text sequence and a new text unit at the end of the second text sequence. This new text unit is obtained by the second model through one inference operation.

[0259] In some embodiments, the processor 1210 is specifically configured to: verify multiple target text units using multiple second probability distributions and multiple first probability distributions to obtain verification results.

[0260] In some examples, the validation result includes indication information indicating whether the target text unit output by the first model is accepted. If the indication information indicates that a target text unit is not accepted, the validation result also includes a replacement token for the unaccepted target text unit. If all target text units are accepted, the validation result also includes a new token obtained after inference by the second model.

[0261] In other examples, the validation result is a text sequence containing multiple tokens. If every target text unit in the first text sequence is accepted, the text sequence comprises γ+1 text units, where the first γ text units are identical to those in the first text sequence, and the last text unit is a new token inferred by the second model. If one or more target text units in the first text sequence are rejected, the text sequence comprises h+1 text units, where the first h text units are identical to the first h text units in the first text sequence, and the last text unit is a replacement token obtained by the cloud device after resampling the first rejected target text unit. h represents the number of target text units preceding the first rejected target text unit in the first text sequence.

[0262] It should be understood that Figure 11 The cloud device 1100 shown is or Figure 12 The cloud device 1200 shown can be used to perform operations or processes related to the cloud device in the above method embodiments, and the operations and / or functions of each module in the cloud device 1100 or cloud device 1200 are respectively to implement the corresponding processes in the above method embodiments. For the sake of brevity, they will not be described in detail here.

[0263] Figure 13 The diagram shows a schematic block diagram of an end-to-cloud collaborative system 1300 according to an embodiment of the present invention. The end-to-cloud collaborative system 1300 includes a terminal device 1310 and a cloud device 1320. The terminal device 1310 corresponds to the terminal device 900 or terminal device 1000 in the above embodiments, and the cloud device 1320 corresponds to the cloud device 1100 or cloud device 1200 in the above embodiments.

[0264] It should be understood that the above description includes examples of data processing methods, terminal devices, cloud devices, and edge-cloud collaborative systems in edge-cloud collaboration scenarios. In edge-edge collaboration scenarios, in the above method embodiments, operations or processes related to cloud devices can be executed by edge computing devices. Edge computing devices can have the same structure and function as cloud devices in the above structural embodiments.

[0265] In edge-cloud collaborative scenarios, in the above method embodiments, operations or processes related to the terminal device can be executed by the edge computing device. The edge computing device can have the same structure and function as the terminal device in the above structural embodiments. The difference is that the actions of receiving the user's inference request and displaying the final inference result to the user are still performed by the terminal device. After receiving the inference request from the user, the terminal device sends the inference request to the edge computing device. After receiving the final inference result from the edge computing device, the terminal device displays the final inference result to the user.

[0266] It should be understood that the aforementioned AI chip can be a central processing unit (CPU), a neural network processing unit (NPU), a graphics processing unit (GPU), a deep learning processing unit (DPU), a tensor processing unit (TPU), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), etc.

[0267] It should be understood that the processors (1010, 1210) in the embodiments of the present invention can be CPUs, or other general-purpose processors, DSPs, ASICs, FPGAs, or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor. The aforementioned processors 1010 and 1210 can also be a collection of multiple processors; for example, processors 1010 and 1210 may include a CPU and a GPU (or a DPU, NPU, etc.). The GPU, DPU, or NPU runs the first model or the second model, performing inference tasks. The CPU performs other tasks besides inference tasks, such as data processing tasks, verification tasks, data transmission and reception tasks, etc.

[0268] It should also be understood that the memory (1020, 1220) in the embodiments of the present invention can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (SLDRAM), and direct rambus RAM (DR RAM).

[0269] It should be noted that when the processor is a general-purpose processor, DSP, ASIC, FPGA, or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component, the memory (storage module) is integrated into the processor.

[0270] It should be noted that the memory of the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0271] It should be understood that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0272] It should be understood that in the above embodiments, the sequence number of each process does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0273] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0274] In the embodiments provided in this application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0275] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0276] In addition, the functional units in the above embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0277] If the aforementioned function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention embodiment, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in this invention embodiment. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0278] The above description is merely a specific implementation of the embodiments of the present invention, but the protection scope of this application is not limited thereto. Any variations or substitutions conceived within the technical scope disclosed in this application should be included within the protection scope of this application. Therefore, the protection scope of this application should be determined by the protection scope of the claims.

Claims

1. A data processing method, characterized in that, The method includes: The first computing device uses the first model to reason about the input data and obtains the first reasoning result, wherein the first model exists in the first computing device; The first computing device sends the first inference result to the second computing device; The first computing device determines new input data for the first model based on the verification result. The verification result is obtained by verifying the first inference result using the second inference result. The second inference result is obtained by the second computing device using the second model to infer the first inference result. The second model exists in the second computing device.

2. The data processing method according to claim 1, characterized in that, The first reasoning result includes: A first text sequence, comprising a plurality of target text units arranged in sequence; Multiple first probability distributions, each corresponding one-to-one with a plurality of target text units, wherein the first probability distributions are used to indicate the probability of each target text unit corresponding to each text unit in the text unit set.

3. The data processing method according to claim 2, characterized in that, Sending the first inference result to the second computing device includes: For each first probability distribution generated, a first network data packet is sent to the second computing device, the first network data packet including the data of the currently generated first probability distribution; or Sending the first inference result to the second computing device includes: After the first text sequence is generated, a second network data packet is sent to the second computing device, the second network data packet including the data of the first text sequence.

4. The data processing method according to claim 3, characterized in that: The first network data packet includes a first identifier, which is used to indicate the target text unit corresponding to the first probability distribution data contained in the first network data packet; or The first network data packet includes a second identifier, which indicates the inference round corresponding to the first probability distribution data contained in the first network data packet; or The first network data packet includes a third identifier, which is used to indicate the inference request corresponding to the first probability distribution data contained in the first network data packet.

5. The data processing method according to any one of claims 1 to 4, characterized in that: The second model is homologous to the first model, and the number of parameters in the first model is less than the number of parameters in the second model.

6. A data processing method, characterized in that, The method includes: The second computing device receives a first inference result sent by the first computing device; the first inference result is obtained by the first computing device using a first model to infer the input data; the first model exists in the first computing device; The second computing device uses the second model to reason about the first reasoning result to obtain a second reasoning result; the second model exists in the second computing device; The second computing device uses the second inference result to verify the first inference result, and obtains a verification result; The second computing device sends the verification result to the first computing device.

7. The data processing method according to claim 6, characterized in that, The first reasoning result includes: A first text sequence, comprising a plurality of target text units arranged in sequence; Multiple first probability distributions are provided, each corresponding one-to-one with a plurality of target text units. The first probability distributions are used to indicate the probability that each text unit in the text unit set is the corresponding target text unit.

8. The data processing method according to claim 7, characterized in that, The method further includes: For each piece of data received from the first probability distribution, the received data from the first probability distribution is converted into data that the second model can recognize; or After receiving the data from the first text sequence, the data from the first text sequence is converted into data that the second model can recognize.

9. A computing device, characterized in that, The device includes: A first inference module is used to infer the received input data using a first model to obtain a first inference result; the first model exists in the computing device. The first transceiver module is used to send the first inference result to the second computing device; The first data processing module is used to determine new input data for the first model based on the verification result. The verification result is obtained by verifying the first inference result using the second inference result. The second inference result is obtained by the second computing device using the second model to infer the first inference result. The second model exists in the second computing device.

10. The computing device according to claim 9, characterized in that: The verification result is obtained by the second computing device using the second inference result of the second model to verify the first inference result; The first transceiver module is further configured to receive the verification result returned by the second computing device.

11. The computing device according to claim 9, characterized in that, The verification result includes indication information and text units. The indication information is used to indicate the position of the text unit or to indicate whether the target text unit is accepted. The first data processing module is specifically used to: determine new input data for the first model based on the indication information and text units; or The verification result includes a text sequence, and the first data processing module is specifically used to use the text sequence as new input data for the first model.

12. The computing device according to claim 9, characterized in that, The computing device is a terminal device, and the second computing device is an edge computing device or a cloud device; or The computing device is an edge computing device, and the second computing device is a cloud device.

13. The computing device according to claim 9, characterized in that, The first reasoning result includes: A first text sequence, comprising a plurality of target text units arranged in sequence; Multiple first probability distributions are provided, each corresponding one-to-one with a plurality of target text units. The first probability distributions are used to indicate the probability that each text unit in the text unit set is the corresponding target text unit.

14. The computing device according to claim 13, characterized in that, The first transceiver module is specifically used for: For each first probability distribution generated, a first network data packet is sent to the second computing device, the first network data packet including the data of the currently generated first probability distribution; or After the first text sequence is generated, a second network data packet is sent to the second computing device, the second network data packet including the data of the first text sequence.

15. The computing device according to claim 14, characterized in that: The first network data packet includes a first identifier, which is used to indicate the target text unit corresponding to the first probability distribution data contained in the first network data packet; or The first network data packet includes a second identifier, which indicates the inference round corresponding to the first probability distribution data contained in the first network data packet; or The first network data packet includes a third identifier, which is used to indicate the inference request corresponding to the first probability distribution data contained in the first network data packet.

16. The computing device according to any one of claims 9 to 15, characterized in that: The second model is homologous to the first model, and the number of parameters in the first model is less than the number of parameters in the second model.

17. A computing device, characterized in that, The device includes: The second transceiver module is used to receive a first inference result sent by the first computing device; the first inference result is obtained by the first computing device using a first model to infer the input data; the first model exists in the first computing device; The second processing module is used to reason about the first reasoning result using the second model to obtain a second reasoning result; the second model exists in the computing device; The verification module is used to verify the first inference result using the second inference result to obtain a verification result; The second transceiver module is also used to send the verification result to the first computing device.

18. A computing system, characterized in that: It includes a first computing device and a second computing device; wherein the first computing device is the computing device according to any one of claims 9 to 16, and the second computing device is the computing device according to claim 17.

Images