Method for obtaining reasoning result, terminal equipment and computer readable storage medium

Abstract

The embodiment of the invention provides a reasoning result obtaining method, terminal equipment and a computer readable storage medium, in the reasoning result obtaining method, after the terminal equipment sends a network request to a server, the terminal equipment receives a server reasoning result sent by the server, the reasoning result of the server comprises a part of result obtained by reasoning of the server, then the terminal equipment presents the reasoning result of the server to a user word by word, reasoning is carried out according to the reasoning result and first content requested by the user, the reasoning result of the terminal equipment is obtained, and after the reasoning result of the server is presented, the terminal equipment sends the reasoning result to the user. The terminal equipment presents subsequent reasoning results of the terminal equipment to a user word by word, so that collaborative reasoning of the terminal equipment and the server side can be realized, the problem that the time consumed for reasoning the first character by the terminal equipment side is too long is solved by using the reasoning results of the server, and the terminal equipment performs reasoning according to the reasoning results of the server, so that the reasoning efficiency is improved. The reasoning precision of the terminal equipment side can be improved.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the technical field of intelligent terminal, in particular to a method for obtaining inference result, a terminal device and a computer readable storage medium. BACKGROUND

[0002] With the rapid development of big data, cloud computing and / or artificial intelligence technologies, large models, as an important achievement in the field of artificial intelligence, are gradually changing our production and life. As a node for interaction with users, a large number of requests are generated from the end side, and the end side also has certain computing power as a computing terminal. The existing end-side large model landing methods generally have the following two kinds: 1) cloud-side large model access method based on network request (request); 2) large model inference access method based on end-side computing power.

[0003] However, the cloud-side large model access method based on network request increases the throughput pressure of the cloud-side machine, and the cloud-side large model inference cost is high; and the large model inference access method based on end-side computing power has a long first inference time and poor accuracy. SUMMARY

[0004] Embodiments of the present application provide a method for obtaining inference result, a terminal device and a computer readable storage medium, to realize collaborative inference of the terminal device and the cloud side, use the inference result of the cloud side to improve the problem of long time consumption of the terminal device side for inferring the first character, and improve the inference accuracy of the terminal device side according to the inference result of the cloud side.

[0005] In a first aspect, embodiments of the present application provide a method for obtaining inference result, comprising: a terminal device sends a network request to a server; wherein the network request carries first content of a user request; receiving the inference result of the server sent by the server; wherein the inference result of the server includes part of the result obtained by the server; presenting the inference result of the server to the user character by character, and inferring according to the inference result of the server and the first content to obtain the inference result of the terminal device; after the inference result of the server is presented, presenting the inference result of the terminal device to the user character by character after the inference result of the server.

[0006] In the above method for obtaining the inference result, after the terminal device sends a network request to the server, it receives the server's inference result, which includes a portion of the result obtained through server inference. Then, the terminal device presents the server's inference result to the user character by character, and performs inference based on the server's inference result and the first content of the user's request to obtain the terminal device's inference result. After the server's inference result is presented, the terminal device presents the inference result following the server's inference result to the user character by character. This enables collaborative inference between the terminal device and the server, improving the problem of excessive time spent inferring the first character on the terminal device side using the server's inference result, and improving the inference accuracy on the terminal device side by performing inference based on the server's inference result.

[0007] In one possible implementation, the step of reasoning based on the server's reasoning result and the first content to obtain the terminal device's reasoning result includes: if the server's reasoning result contains a result corresponding to the terminal device's (N-1)th round of reasoning, then the result corresponding to the terminal device's (N-1)th round of reasoning is used as the input to the terminal device's (N)th round of reasoning to obtain the terminal device's (N)th round of reasoning result; where N is an integer, N≥2; if the server's reasoning result does not contain a result corresponding to the terminal device's (N-1)th round of reasoning, then the (N-1)th round of reasoning result is used as the input to the terminal device's (N)th round of reasoning to obtain the terminal device's (N)th round of reasoning result.

[0008] In one possible implementation, the terminal device can perform a first round of reasoning based on the first content to obtain the first round of reasoning result of the terminal device.

[0009] In one possible implementation, presenting the server's inference results to the user word by word includes: adjusting the time interval for word-by-word presentation; and presenting the server's inference results to the user word by word according to the adjusted time interval.

[0010] In one possible implementation, during the process of presenting the server's inference results to the user word by word, if the number of characters obtained by the terminal device through inference is equal to the number of characters already presented to the user, then the terminal device cancels the adjustment of the time interval for word-by-word presentation and directly presents the server's inference results to the user word by word.

[0011] In one possible implementation, receiving the server's inference result sent by the server includes: receiving the server's inference result sent by the server through a long connection with the terminal device; wherein the long connection is established between the server and the terminal device after the server receives the network request.

[0012] In one possible implementation, after receiving the inference result sent by the server, the method further includes: storing the inference result of the server in a first buffer.

[0013] In one possible implementation, after obtaining the inference result of the terminal device, the method further includes: storing the inference result of the terminal device in a second buffer.

[0014] Secondly, embodiments of this application provide a terminal device, including: one or more processors; a memory; multiple applications; and one or more computer programs, wherein the one or more computer programs are stored in the memory, and the one or more computer programs include instructions that, when executed by the terminal device, cause the terminal device to perform the method provided in the first aspect.

[0015] It should be understood that the second aspect of the embodiments of this application is consistent with the technical solution of the first aspect of the embodiments of this application, and the beneficial effects achieved by each aspect and the corresponding feasible implementation are similar, and will not be described again.

[0016] Thirdly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when run on a computer, causes the computer to perform the method provided in the first aspect.

[0017] Fourthly, embodiments of this application provide a computer program that, when executed by a computer, performs the method provided in the first aspect.

[0018] In one possible design, the program in the fourth aspect can be stored wholly or partially on a storage medium packaged with the processor, or it can be stored wholly or partially on a memory not packaged with the processor. Attached Figure Description

[0019] Figures 1(a)-1(b) This is a schematic diagram illustrating how a large-scale model can be deployed on the edge.

[0020] Figure 2 This is a schematic diagram of an edge-cloud collaborative acceleration scheme for deep neural networks.

[0021] Figure 3This is a schematic diagram of an edge-cloud collaborative inference scheme for neural network operator fusion;

[0022] Figure 4 This is a schematic diagram of the structure of a terminal device provided in one embodiment of this application;

[0023] Figure 5 This is a schematic diagram illustrating an application scenario provided in one embodiment of this application;

[0024] Figure 6 A schematic diagram of the system architecture provided for one embodiment of this application;

[0025] Figure 7 A flowchart illustrating a method for obtaining reasoning results according to one embodiment of this application;

[0026] Figure 8 A schematic diagram illustrating the receipt of a user request according to an embodiment of this application;

[0027] Figure 9 A schematic diagram of a long connection provided in one embodiment of this application;

[0028] Figure 10 A schematic diagram of the inference module provided in one embodiment of this application;

[0029] Figure 11 A schematic diagram illustrating the optimization of accuracy and latency provided for one embodiment of this application;

[0030] Figure 12 A schematic diagram illustrating the rate-controlled return of inference results provided in one embodiment of this application;

[0031] Figure 13 This is a schematic diagram illustrating how the number of characters obtained by the terminal device 01 inference in one embodiment of this application catches up with the number of characters already presented to the user;

[0032] Figure 14 This is a schematic diagram illustrating the effect of one embodiment of this application;

[0033] Figure 15 This is a schematic diagram illustrating how the reasoning results are presented to the user, as provided in one embodiment of this application.

[0034] Figure 16 A flowchart illustrating a method for obtaining reasoning results according to one embodiment of this application;

[0035] Figure 17 This is a schematic diagram of the structure of a terminal device provided in another embodiment of this application. Detailed Implementation

[0036] The terminology used in the implementation section of this application is for the purpose of explaining specific embodiments of this application only, and is not intended to limit this application.

[0037] In existing related technologies, there are generally two ways to implement large models on the terminal side: 1) Cloud-side large model access method based on network request. After the user makes a request on the terminal device, the terminal device accesses the server to drive the cloud to perform inference, as shown in Figure 1(a).

[0038] 2) Access method for large model inference based on edge computing power: The large model is deployed on the edge. When the user makes a request on the terminal device, the edge computing power is used for inference, as shown in Figure 1(b). Figures 1(a)-1(b) This is a schematic diagram illustrating the deployment method of a large-scale model on the edge.

[0039] Of the two existing edge-side large model deployment solutions mentioned above, if the cloud-side large model access method based on network requests is used, the throughput pressure on the cloud-side machine will further increase with the increase in the number of users on the edge, and the inference cost of the cloud-side large model is relatively high. On the other hand, if the large model inference access method based on edge computing power is used, the inference time for the first character is relatively long due to the limitation of edge computing power. Furthermore, since the available memory of terminal devices is small, only small models with fewer parameters and lightweight design can be used. Compared with large models, the accuracy of such models is poor.

[0040] In the area of ​​edge-cloud collaborative acceleration, current edge-cloud collaborative acceleration solutions for deep neural networks generally involve splitting the model structure and decomposing the computational tasks for joint execution on both the edge and cloud sides, such as... Figure 2 As shown, Figure 2 This is a schematic diagram of a cloud-edge collaborative acceleration solution for deep neural networks. However, this solution cannot be applied to existing large models because, compared to general neural networks, large models have excessively large intermediate parameters, making them inconvenient to transfer between devices. Taking the LLaMa7B model as an example, the inter-layer data consists of a key-value cache (KVCache) (200MB) + intermediate results (80KB), totaling approximately 200MB. This puts excessive pressure on network bandwidth, and since the segmentation execution is serial, the edge also needs to load weights, resulting in excessive memory pressure on the edge.

[0041] Existing related technologies also provide an edge-cloud collaborative inference scheme for neural network operator fusion, such as Figure 3 As shown, Figure 3This diagram illustrates an edge-cloud collaborative inference scheme for neural network operator fusion. The scheme includes converting the neural network into a directed acyclic graph (DAG); dividing the DAG into several chain structures; performing a fusion operation on the network layers in each chain structure and replacing the fused network layers with the resulting fused block; predicting the inference time and output data size of each fused block and each unfused network layer based on the data to be inferred, and calculating the intermediate data transmission time based on the output data size and the network bandwidth between the edge and cloud; segmenting the neural network based on the inference time and the intermediate data transmission time, and performing edge-cloud collaborative inference based on the segmentation results. This scheme solves the minimum latency problem of network models with fusionable operators.

[0042] However, this solution targets traditional deep neural networks. Traditional deep neural networks have fewer nodes within a layer, resulting in less data between layers and lower network bandwidth requirements for intermediate result transmission. In contrast, large models have more nodes within a layer, and a single inference iteration consists of multiple incremental processes. Each incremental process requires executing the entire neural network, placing a significant strain on network bandwidth. Furthermore, the self-attention mechanism in large models requires acceleration using KVCache. The KVCache for each request essentially represents the intermediate results of the self-attention operation involving all tokens input to the request. For example, in the LLaMa7B model, the KVCache for a single request is 200MB, which puts immense pressure on network bandwidth.

[0043] To address the above issues, this application provides a method for obtaining inference results. A small-parameter model is deployed on the terminal device side, and a large-parameter model is deployed on the cloud side. These two models can perform collaborative inference. When a user initiates a request, inference requests are simultaneously sent to both the cloud side and the terminal device side, and the inference results returned by the cloud side are continuously received. The inference results returned by the cloud side are then returned to the user state at a controlled rate. After the first character of the inference is successfully inferred on the terminal device side, the cloud side terminates the inference process, and the terminal device corrects its own inference results based on the inference results returned by the cloud side.

[0044] The method for obtaining inference results provided in this application embodiment can be applied to terminal devices, wherein the terminal devices can be smartphones, tablets, wearable devices, in-vehicle devices, augmented reality (AR) / virtual reality (VR) devices, laptops, ultra-mobile personal computers (UMPCs), netbooks, or personal digital assistants (PDAs), etc.; this application embodiment does not impose any restrictions on the specific type of terminal device.

[0045] For example, Figure 4 This is a schematic diagram of the structure of a terminal device provided in one embodiment of this application, as shown below. Figure 4 As shown, the terminal device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, a sensor module 180, buttons 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an accelerometer sensor 180E, a distance sensor 180F, a proximity sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0046] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the terminal device 100. In other embodiments of this application, the terminal device 100 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0047] Processor 110 may include one or more processing units, such as application processors (APs), modem processors, graphics processing units (GPUs), image signal processors (ISPs), controllers, video codecs, digital signal processors (DSPs), baseband processors, and / or neural network processing units (NPUs). These different processing units may be independent devices or integrated into one or more processors.

[0048] The controller can generate operation control signals based on the instruction opcode and timing signals to complete the control of instruction fetching and execution.

[0049] The processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory can store instructions or data that the processor 110 has just used or that are used repeatedly. If the processor 110 needs to use the instruction or data again, it can retrieve it directly from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0050] In some embodiments, the processor 110 may include one or more interfaces. Interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0051] USB port 130 is a USB standard compliant interface, specifically a Mini USB port, Micro USB port, or USB Type-C port. USB port 130 can be used to connect a charger to charge terminal device 100, and can also be used for data transfer between terminal device 100 and peripheral devices. It can also be used to connect headphones for audio playback. This interface can also be used to connect other electronic devices, such as AR devices.

[0052] It is understood that the interface connection relationships between the modules illustrated in the embodiments of this application are merely illustrative and do not constitute a structural limitation on the terminal device 100. In other embodiments of this application, the terminal device 100 may also adopt different interface connection methods or a combination of multiple interface connection methods as described in the above embodiments.

[0053] The charging management module 140 receives charging input from a charger. The charger can be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 receives charging input from the wired charger via the USB interface 130. In some wireless charging embodiments, the charging management module 140 receives wireless charging input via the wireless charging coil of the terminal device 100. While charging the battery 142, the charging management module 140 can also supply power to the terminal device 100 via the power management module 141.

[0054] The power management module 141 connects the battery 142, the charging management module 140, and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140, providing power to the processor 110, internal memory 121, display screen 194, camera 193, and wireless communication module 160, etc. The power management module 141 can also monitor parameters such as battery capacity, battery cycle count, and battery health status (leakage current, impedance). In some other embodiments, the power management module 141 may also be located within the processor 110. In other embodiments, the power management module 141 and the charging management module 140 may be located in the same device.

[0055] The wireless communication function of the terminal device 100 can be implemented through antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, modem processor, and baseband processor.

[0056] Antennas 1 and 2 are used to transmit and receive electromagnetic wave signals. Each antenna in terminal device 100 can be used to cover one or more communication frequency bands. Different antennas can also be multiplexed to improve antenna utilization. For example, antenna 1 can be multiplexed as a diversity antenna for a wireless local area network. In some other embodiments, the antennas can be used in conjunction with a tuning switch.

[0057] The mobile communication module 150 can provide solutions for wireless communication, including 2G / 3G / 4G / 5G, applied to the terminal device 100. The mobile communication module 150 may include at least one filter, switch, power amplifier, low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves via antenna 1, and perform filtering, amplification, and other processing on the received electromagnetic waves before transmitting them to a modem processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modem processor and convert it into electromagnetic waves for radiation via antenna 1. In some embodiments, at least some functional modules of the mobile communication module 150 may be housed in the processor 110. In some embodiments, at least some functional modules of the mobile communication module 150 and at least some modules of the processor 110 may be housed in the same device.

[0058] The modem processor may include a modulator and a demodulator. The modulator modulates the low-frequency baseband signal to be transmitted into a mid-to-high frequency signal. The demodulator demodulates the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After processing by the baseband processor, the low-frequency baseband signal is transmitted to the application processor. The application processor outputs sound signals through an audio device (not limited to speaker 170A, receiver 170B, etc.) or displays images or videos through the display screen 194. In some embodiments, the modem processor may be a separate device. In other embodiments, the modem processor may be independent of the processor 110 and may be housed in the same device as the mobile communication module 150 or other functional modules.

[0059] The wireless communication module 160 can provide solutions for wireless communication applications on the terminal device 100, including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), and infrared (IR) technologies. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via antenna 2, performs frequency modulation and filtering of the electromagnetic wave signals, and sends the processed signal to processor 110. The wireless communication module 160 can also receive signals to be transmitted from processor 110, perform frequency modulation and amplification, and convert them into electromagnetic waves for radiation via antenna 2.

[0060] In some embodiments, antenna 1 of terminal device 100 is coupled to mobile communication module 150, and antenna 2 is coupled to wireless communication module 160, enabling terminal device 100 to communicate with networks and other devices via wireless communication technology. The wireless communication technology may include Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Time Division Code Division Multiple Access (TD-CDMA), Long Term Evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technologies, etc. The GNSS may include the Global Positioning System (GPS), the Global Navigation Satellite System (GLONASS), the BeiDou Navigation Satellite System (BDS), the Quasi-Zenith Satellite System (QZSS), and / or satellite-based augmentation systems (SBAS).

[0061] Terminal device 100 implements display functions through a GPU, display screen 194, and application processor. The GPU is a microprocessor for image processing, connected to the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations and for graphics rendering. Processor 110 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0062] Display screen 194 is used to display images, videos, etc. Display screen 194 includes a display panel. The display panel may be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a Miniled LED, a MicroLED, a Micro-OLED, a quantum dot light-emitting diode (QLED), etc. In some embodiments, terminal device 100 may include one or N displays 194, where N is a positive integer greater than 1.

[0063] Terminal device 100 can perform shooting functions through ISP, camera 193, video codec, GPU, display 194 and application processor.

[0064] The ISP (Image Signal Processor) is used to process data fed back from the camera 193. For example, when taking a picture, the shutter is opened, and light is transmitted through the lens to the camera's photosensitive element. The light signal is converted into an electrical signal, and the camera's photosensitive element transmits the electrical signal to the ISP for processing, transforming it into an image visible to the naked eye. The ISP can also perform algorithmic optimization of image noise, brightness, and skin tone. The ISP can also optimize parameters such as exposure and color temperature of the shooting scene. In some embodiments, the ISP can be set in the camera 193.

[0065] Camera 193 is used to capture still images or videos. An object is projected onto a photosensitive element by generating an optical image through the lens. The photosensitive element can be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, which is then passed to an ISP for conversion into a digital image signal. The ISP outputs the digital image signal to a DSP for processing. The DSP converts the digital image signal into image signals in standard RGB, YUV, or other formats. In some embodiments, the terminal device 100 may include one or N cameras 193, where N is a positive integer greater than 1.

[0066] A digital signal processor (DSP) is used to process digital signals. Besides digital image signals, it can also process other digital signals. For example, when terminal device 100 selects a frequency, the DSP can perform Fourier transforms on the frequency energy.

[0067] Video codecs are used to compress or decompress digital video. Terminal device 100 may support one or more video codecs. Thus, terminal device 100 can play or record videos in various encoding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG 2, MPEG 3, MPEG 4, etc.

[0068] NPU stands for Neural Network (NN) Computing Processor. By borrowing the structure of biological neural networks, such as the transmission patterns between neurons in the human brain, it can rapidly process input information and continuously learn on its own. NPUs enable intelligent cognitive applications in terminal devices, such as image recognition, facial recognition, speech recognition, and text understanding.

[0069] The external storage interface 120 can be used to connect an external storage card, such as a Micro SD card, to expand the storage capacity of the terminal device 100. The external storage card communicates with the processor 110 through the external storage interface 120 to perform data storage functions. For example, music, video, and other files can be saved on the external storage card.

[0070] Internal memory 121 can be used to store computer executable program code, which includes instructions. Internal memory 121 may include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback, image playback, etc.), etc. The data storage area may store data created during the use of terminal device 100 (such as audio data, phonebook, etc.). Furthermore, internal memory 121 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc. Processor 110 executes various functional applications and data processing of terminal device 100 by running instructions stored in internal memory 121 and / or instructions stored in memory located in the processor.

[0071] Terminal device 100 can implement audio functions, such as music playback and recording, through audio module 170, speaker 170A, receiver 170B, microphone 170C, headphone jack 170D, and application processor.

[0072] The audio module 170 is used to convert digital audio information into analog audio signals for output, and also to convert analog audio input into digital audio signals. The audio module 170 can also be used for encoding and decoding audio signals. In some embodiments, the audio module 170 may be located in the processor 110, or some functional modules of the audio module 170 may be located in the processor 110.

[0073] The speaker 170A, also known as a "loudspeaker," is used to convert audio electrical signals into sound signals. The terminal device 100 can listen to music or make hands-free calls through the speaker 170A.

[0074] The receiver 170B, also known as the "earpiece," is used to convert audio electrical signals into sound signals. When the terminal device 100 answers a phone call or voice message, the receiver 170B can be brought close to the listener's ear to hear the voice.

[0075] Microphone 170C, also known as a "microphone" or "voice transducer," is used to convert sound signals into electrical signals. When making a phone call or sending a voice message, the user can speak by bringing their mouth close to microphone 170C, inputting the sound signal into microphone 170C. Terminal device 100 may be equipped with at least one microphone 170C. In some embodiments, terminal device 100 may be equipped with two microphones 170C, which, in addition to collecting sound signals, can also perform noise reduction. In other embodiments, terminal device 100 may be equipped with three, four, or more microphones 170C, which can collect sound signals, reduce noise, identify the sound source, and perform directional recording, etc.

[0076] The 170D headphone jack is used to connect wired headphones. The 170D headphone jack can be a USB 130 interface or a 3.5mm Open Mobile Terminal Platform (OMTP) standard interface, a CTIA (Cellular Telecommunications Industry Association of the USA) standard interface.

[0077] Buttons 190 include a power button, volume buttons, etc. Buttons 190 can be mechanical buttons or touch-sensitive buttons. Terminal device 100 can receive button input and generate key signal inputs related to user settings and function control of terminal device 100.

[0078] Motor 191 can generate vibration alerts. Motor 191 can be used for incoming call vibration alerts or for touch vibration feedback. For example, different vibration feedback effects can correspond to touch operations performed on different applications (such as taking photos, playing audio, etc.). Motor 191 can also correspond to different vibration feedback effects for touch operations performed on different areas of the display screen 194. Different application scenarios (such as time reminders, receiving messages, alarm clocks, games, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also be customized.

[0079] Indicator 192 can be an indicator light, used to indicate charging status, power changes, or to indicate messages, missed calls, notifications, etc.

[0080] The SIM card interface 195 is used to connect a SIM card. The SIM card can be inserted into or removed from the SIM card interface 195 to make contact with and separate from the terminal device 100. The terminal device 100 can support one or N SIM card interfaces, where N is a positive integer greater than 1. The SIM card interface 195 can support Nano SIM cards, Micro SIM cards, SIM cards, etc. Multiple cards can be inserted into the same SIM card interface 195 simultaneously. The multiple cards can be of the same or different types. The SIM card interface 195 is also compatible with different types of SIM cards. The SIM card interface 195 is also compatible with external memory cards. The terminal device 100 interacts with the network through the SIM card to realize functions such as calls and data communication. In some embodiments, the terminal device 100 uses an eSIM, i.e., an embedded SIM card. The eSIM card can be embedded in the terminal device 100 and cannot be separated from the terminal device 100.

[0081] For ease of understanding, the following embodiments of this application will be described using the following methods: Figure 4 Taking the terminal device with the structure shown as an example, and in conjunction with the accompanying drawings and application scenarios, the method for obtaining the reasoning results provided in this application embodiment will be specifically described.

[0082] The method for obtaining reasoning results provided in this application embodiment can be applied to voice assistant applications (or mini-programs) on terminal devices 100 such as mobile phones, tablets, or in-vehicle devices, for example... Figure 5 As shown, Figure 5 This is a schematic diagram illustrating an application scenario provided in one embodiment of this application. Figure 5 In the application scenario shown, in response to the user's operation, after the terminal device 100 used by the user starts the voice assistant application (or mini-program), the terminal device 100 can obtain the user's request, and then the terminal device 100 can use the method provided in the embodiment of this application to obtain the reasoning result corresponding to the above request, and present the above reasoning result to the user.

[0083] Figure 6 A schematic diagram of the system architecture provided for one embodiment of this application, as shown below. Figure 6 As shown, the system architecture of this application embodiment includes one or more cloud servers 00 and terminal devices 01. The cloud server 00 may include a large model inference module 06 and a response module 05. The terminal device 01 may include a dynamic adjustment module 02, an end-side inference module 04 and a collaboration module 03.

[0084] 00: Cloud server, which carries large model question answering service. Cloud server 00 may include response module 05 and large model inference module 06, and provides services to terminal device 01 through network request service interface.

[0085] 01: Terminal device (such as a mobile phone, tablet computer, or in-vehicle device), serving as the human-computer interaction interface for users to use intelligent question-and-answer. Users can interact with terminal device 01 through clicking, touching, pressing buttons, voice, and / or gestures. It should be noted that, in this embodiment, terminal device 01 can be... Figure 4 The structure shown is implemented.

[0086] 02: The dynamic adjustment module interacts with the user, receives the user's reasoning requests, and presents the reasoning results to the user in an interactive and user-friendly manner.

[0087] 03: Collaboration module, which connects to the Internet via wired or wireless communication, constructs network requests based on the user's inference requests, sends the network requests to cloud server 00, and receives the inference results returned by cloud server 00.

[0088] 04: The terminal-side inference module is responsible for deploying the model on the terminal device's 01 side, performing local inference based on the inference request given by the user, and correcting the terminal-side inference results based on the cloud results.

[0089] 05: The response module, after receiving the network request sent by the collaboration module 03, performs load balancing, selects a large model inference container, forwards the network request to the selected inference container, and streams the inference results obtained by the selected inference container after executing the request back to the collaboration module 03.

[0090] 06: Large model inference module, performs large model inference based on the inference request forwarded by response module 05.

[0091] The following is combined Figure 6 The system architecture shown describes the method for obtaining inference results provided in the embodiments of this application.

[0092] Figure 7This is a flowchart illustrating a method for obtaining inference results according to an embodiment of this application. The method involves a dynamic adjustment module 02, an end-to-end inference module 04, and a collaboration module 03 on the terminal device 01 side, and a response module 05 and a large-scale model inference module 06 on the cloud server 00 side. More specifically, from a process perspective, the dynamic adjustment module 02 receives user requests, provides a friendly interactive response, and releases speed control; the collaboration module 03 constructs network requests, stores cloud-side results, and stores end-to-end results; the end-to-end inference module 04 performs end-to-end inference and corrects inference results; the response module 05 receives network requests and forwards inference results; and the large-scale model inference module 06 performs cloud-side inference.

[0093] like Figure 7 As shown, the methods for obtaining the above reasoning results may include:

[0094] Step 701: The user triggers the dynamic adjustment module to receive the user request through voice, key press and / or gesture operations.

[0095] For example, see some examples. Figure 8 The user can press and hold the power button 81 of the terminal device 01 (for example, press for 350ms to 450ms). In response to the user's operation of pressing and holding the power button 81, the terminal device 01 runs the voice assistant and displays the interface shown in 82. Then, the user can use voice to say their request, such as "How is the weather today?" The dynamic adjustment module in the terminal device 01 then receives the above user request. Figure 8 This is a schematic diagram illustrating the receipt of a user request according to one embodiment of this application.

[0096] Step 702: The collaboration module constructs a network request based on the user request, wherein the network request carries the server's inference time. Additionally, the network request also carries the first content of the user request; in this embodiment, the first content can be "What's the weather like today?". Then, the collaboration module distributes the network request to the cloud server 00 side via the network, and simultaneously transmits the first content to the edge-side inference module via a method call.

[0097] Specifically, the inference duration of the aforementioned server is used to limit the duration of inference performed by the cloud server 00. That is, within this inference duration, the cloud server 00 performs inference, and when the end time of this inference duration arrives, the cloud server 00 stops inference. In practical implementation, the inference duration of the aforementioned server can be set based on the inference duration of the first character of the terminal device 00. For example, assuming, based on empirical values, the inference duration of the first character of the terminal device 00 is 590ms, then the inference duration of the aforementioned server can be set to 600ms. Of course, this is only one way to set the inference duration of the aforementioned server, and this embodiment does not limit the method of setting the inference duration of the aforementioned server.

[0098] In step 703, the response module on the cloud server 00 receives the network request sent by the collaboration module and establishes a long connection with the terminal device 01 that issued the network request. The long connection allows the terminal device 01 to continuously receive the inference results sent by the cloud server 00.

[0099] In this embodiment, the long connection diagram between the cloud server 00 and the terminal device 01 can be as follows: Figure 9 As shown, Figure 9 This is a schematic diagram of a long connection provided in one embodiment of this application. Figure 9 In the process, after the response module in cloud server 00 receives a network request from the collaborative module of terminal device 01, the response module establishes a long connection with the collaborative module of terminal device 01. The response module then transmits data to the collaborative module of terminal device 01 through this long connection. Furthermore, the response module and the collaborative module exchange link detection packets. If both the response module and the collaborative module receive the link detection packets sent by the other, the response module can determine that the long connection is normal and can continue to use the long connection to transmit data to the collaborative module. However, if the response module in cloud server 00 disconnects the long connection with the collaborative module of terminal device 01, the link detection packets sent by the collaborative module to the response module will not receive a response, and subsequently, the collaborative module of terminal device 01 will also disconnect the long connection with the response module of cloud server 00.

[0100] Step 704: The end-side inference module in terminal device 01 receives the first content sent by the collaboration module and performs inference based on the first content.

[0101] Step 705: The large model inference module in cloud server 00 performs large model inference based on the network request received by the response module until the end time of the inference duration of the aforementioned server is reached.

[0102] Specifically, the large model inference module in cloud server 00 performs large model inference based on the first content carried in the aforementioned network request until the end of the inference time of the aforementioned server is reached.

[0103] By limiting the inference time of cloud server 00, it can be seen that in this embodiment, the large model inference module in cloud server 00 does not need to execute the complete inference process, but only needs to perform inference within the inference time. Therefore, the inference result of cloud server 00 is only a part of the result obtained by cloud server 00 inference.

[0104] In step 706, the large model inference module in cloud server 00 sends the large model inference result to the response module. The response module then streams the large model inference result back to the collaboration module in terminal device 01 through the long connection established in step 703.

[0105] In this embodiment, each time the large model inference module in cloud server 00 infers a character, it sends the inferred character to the response module. The response module then sends the inference result from cloud server 00 to the collaboration module in terminal device 01 via the aforementioned long connection. The collaboration module receives the inference result from cloud server 00 sent by the response module via the aforementioned long connection.

[0106] It is important to note that in this step, cloud server 00 transmits a token and the corresponding character to terminal device 01. If the vocabulary used by cloud server 00 is inconsistent with the vocabulary used by terminal device 01, then cloud server 00 needs to convert the output characters on the cloud side into the corresponding token according to the vocabulary used by terminal device 01.

[0107] Step 707: For the inference results returned by cloud server 00, the collaborative module maintains a first buffer shared with other modules, which stores the inference results returned by cloud server 00.

[0108] In step 708, the end-side inference module performs inference based on the inference results stored in the first buffer and the first content received by the end-side inference module in step 704 to obtain the inference result of the terminal device 01.

[0109] Specifically, in step 704, the edge-side inference module performs a first round of inference based on the first content sent by the collaboration module, and obtains the first round of inference result from the edge-side inference module. Since the inference time for the first character of terminal device 01 is relatively long, i.e., the first round of inference takes a considerable amount of time (generally around 600ms), the first buffer will store the inference result returned by cloud server 00 after the edge-side inference module performs the first round of inference. Thus, the edge-side inference module will subsequently perform inference based on the inference result stored in the first buffer to obtain the inference result from terminal device 01.

[0110] Specifically, if the inference results stored in the first buffer contain a result corresponding to the (N-1)th round of inference of the edge inference module, the edge inference module uses the result corresponding to the (N-1)th round of inference of the edge inference module stored in the first buffer as the input of the Nth round of inference of the edge inference module to obtain the Nth round of inference result of the edge inference module; where N is an integer, N≥2; if the inference results stored in the first buffer do not contain a result corresponding to the (N-1)th round of inference of the edge inference module, the edge inference module uses the (N-1)th round of inference result as the input of the Nth round of inference of the edge inference module to obtain the Nth round of inference result of the edge inference module.

[0111] The following example illustrates the reasoning process of the edge-side reasoning module. Figure 10 This is a schematic diagram of the inference module provided in one embodiment of this application, as shown below. Figure 10 As shown, after receiving the network request "How is the weather today?", the edge inference module performs the first round of inference based on the network request, and obtains the first round of inference result "Today". Then, when performing the second round of inference, the inference results stored in the first buffer contain the result "Today" corresponding to the first round of inference of the edge inference module. Therefore, the edge inference module uses "Today" as the input of the edge inference module for the second round of inference, and obtains the second round of inference result "Day". The process of the edge inference module performing the third to fifth rounds of inference is similar to the process of the second round of inference, and will not be described in detail here. The inference results of the third to fifth rounds of inference of the edge inference module are "Day", "Weather", and "Very" respectively.

[0112] During the sixth round of inference, the first buffer contains the result "Yes" corresponding to the fifth round of inference. Therefore, the edge inference module uses "Yes" as the input for the sixth round of inference and obtains the result "Yin" for the sixth round of inference. Since the end time of the inference duration has arrived, the cloud server 00 has not performed the sixth round of inference. Therefore, during the seventh round of inference, the first buffer does not contain the result corresponding to the sixth round of inference. Thus, the edge inference module uses the result "Yin" from the sixth round of inference as the input for the seventh round of inference and obtains the result "Tian" for the seventh round of inference.

[0113] from Figure 10As can be seen, the reasoning result of the fifth round of reasoning by the edge-side inference module is "very", while the reasoning result of the fifth round of reasoning by the cloud server 00 is "yes". It is obvious that the reasoning result of the fifth round of reasoning by the edge-side inference module is different from that of the cloud server 00. Since the reasoning accuracy of the cloud server 00 is high, in this embodiment, the edge-side inference module uses the reasoning result of the cloud server 00 as input for reasoning, thereby improving the accuracy of edge-side reasoning. In addition, in this embodiment, the cloud server 00 only performs reasoning within a limited reasoning time, thereby reducing the reasoning load of the cloud server 00.

[0114] See Figure 11 The inference result acquisition method provided in this embodiment, compared with pure cloud-side inference and pure terminal-side inference, can both maintain the high accuracy of cloud-side inference and reduce the load of cloud-side inference by using terminal-side inference. Figure 11 This is a schematic diagram illustrating the optimization of accuracy and latency provided for one embodiment of this application.

[0115] from Figure 11 It can be seen that the pure cloud-side inference scheme has the advantages of high accuracy and high speed, but it puts a lot of pressure on the cloud side and the inference cost of large cloud-side models is high; the pure edge-side inference scheme has low accuracy, long first-character inference time, and poor user experience. The embodiments of this application adopt an edge-cloud collaborative inference scheme, which can both maintain the high accuracy of cloud-side inference and reduce the load of cloud-side inference by using edge-side inference.

[0116] The edge-cloud collaborative reasoning solution provided in this application optimizes the edge latency from 590ms for the first character reasoning plus 35ms for each subsequent character reasoning to approximately 350ms for each character reasoning. In terms of accuracy, the edge reasoning module uses the reasoning results from the cloud server as input for reasoning, thereby improving the accuracy of edge reasoning.

[0117] Step 709: The dynamic adjustment module presents the inference results stored in the first buffer to the user word by word.

[0118] Specifically, when presenting the inference results stored in the first buffer, the dynamic adjustment module can interactively and user-friendly return the inference results stored in the first buffer to the user at a controlled speed. This interactive and user-friendly return refers to controlling the return speed of the inference results within the user's acceptable range. To achieve a sequential, word-by-word return effect for the inference results, this embodiment controls the return speed by controlling the interval between word-by-word returns, thereby improving the problem of long initial-word inference time on the client side using cloud-side inference results. Specifically, the dynamic adjustment module presents the inference results stored in the first buffer to the user word-by-word as follows: the dynamic adjustment module adjusts the time interval for word-by-word presentation, and then presents the inference results stored in the first buffer to the user word-by-word according to the adjusted time interval. A schematic diagram of the dynamic adjustment module controlling the speed of returning the inference results stored in the first buffer can be shown below. Figure 12 As shown, Figure 12 This is a schematic diagram illustrating the inference result rate control return provided in one embodiment of this application.

[0119] Figure 12 In the diagram, from bottom to top, the inference time for the first character of the terminal inference module is 3-4 seconds; the time from when the terminal device 01 sends a network request to the cloud server 00 to when the terminal device 01 receives the inference result from the cloud server 00 is the round trip time (RTT), which is 0.2 seconds; the inference time for the first character of the cloud server 00 is 100 ms, and the inference time for each subsequent character is 35 ms; when the dynamic adjustment module presents the inference result of the cloud server 00 stored in the first buffer to the user, it adjusts the time interval for each character presentation, setting the time interval for the first character presentation to 350 ms, the time interval for the second character (i.e., cloud increment 1) to 340 ms, the time interval for the third character (i.e., cloud increment 2) to 330 ms, and so on. The time interval for the first character to be presented refers to the time interval from the moment when the terminal device 00 sends the above network request to the moment when the first character is presented; the time interval for the presentation of the i-th character is the time interval between the presentation time of the (i-1)-th character and the presentation time of the i-th character, where i is an integer and i≥2.

[0120] In addition, it is understood that when presenting characters to the user, text, voice and / or images can be used, and this embodiment does not limit the way characters are presented.

[0121] Step 710: Since cloud-side inference ends after a limited inference time, cloud server 00 cannot complete the complete inference result. Subsequent inference needs to be completed by the terminal-side inference module. That is, in this embodiment, the terminal-side inference module executes the complete inference process. However, since the first character inference on the terminal side takes a long time, in order to improve this problem, the terminal device 01 takes advantage of the fast inference speed of cloud server 00. After receiving the inference result of cloud server 00, the terminal device 01 first presents the inference result of cloud server 00 to the user. After the inference result of cloud server 00 is presented, the dynamic adjustment module in terminal device 01 then presents the inference result after the inference result of cloud server 00 to the user character by character.

[0122] In this embodiment, the collaboration module also has a second buffer for storing the inference results on the terminal side, which stores the inference results of the terminal device 01.

[0123] Step 711: The dynamic adjustment module determines when to release the speed control return based on the length of the content in the second buffer. Specifically, if the number of characters inferred by terminal device 01 is equal to the number of characters presented to the user, it means that the inference speed of terminal device 01 has matched the progress of presentation to the user, and there is no need to perform a speed control return. All characters in the first and second buffers can be directly returned. Figure 13 As shown, Figure 13 This is a schematic diagram illustrating how the number of characters obtained by the terminal device 01 inference in one embodiment of this application catches up with the number of characters already presented to the user.

[0124] Figure 13 In the process, the inference time for the first character of the terminal-side inference module is 3-4 seconds. The time from when the terminal device 01 sends a network request to the cloud server 00 to when the terminal device 01 receives the inference result from the cloud server 00 is the round-trip time (RTT), which is 0.2 seconds. The inference time for the first character of the cloud server 00 is 100 ms, and the inference time for each subsequent character is 35 ms. When the dynamic adjustment module presents the inference result of the cloud server 00 stored in the first buffer to the user, it adjusts the time interval for each character to be presented. The time interval for presenting the first character is adjusted to 350 ms, the time interval for presenting the second character (i.e., cloud increment 1) is adjusted to 340 ms, the time interval for presenting the third character (i.e., cloud increment 2) is adjusted to 330 ms, and so on. The time interval for the first character to be presented refers to the time interval from the moment when the terminal device 00 sends the above network request to the moment when the first character is presented; the time interval for the presentation of the i-th character is the time interval between the presentation time of the (i-1)-th character and the presentation time of the i-th character, where i is an integer and i≥2.

[0125] After cloud server 00 obtains the 76th character (i.e., cloud increment 75), the inference time of cloud server 00 ends, and cloud server 00 stops inference. Subsequent inference results are obtained by the end-side inference module in terminal device 00. When the end-side inference module infers the 11th character (i.e., end increment 10), the character presented to the user is the 11th character (i.e., cloud increment 10) in the inference result of cloud server 00. This indicates that the number of characters obtained by terminal device 01 is equal to the number of characters presented to the user. At this point, the dynamic adjustment module no longer needs to perform speed control return; it can directly return all characters in the first and second buffers. Figure 13 It can also be seen that when the terminal device 01 infers the character "end-increment 10", the inference speed of the terminal device 01 has matched the progress of the presentation to the user. Subsequently, the speed control is released, and the characters in the first buffer are presented to the user at the terminal speed. After the cloud server 00 infers and obtains the 76th character (i.e., cloud-increment 75), the characters in the second buffer (end-increment 76, ...) are presented to the user at the terminal speed.

[0126] The method for obtaining reasoning results provided in this application embodiment can start from the intelligent question-and-answer process of the dynamic adjustment module triggered by the user through voice, key and / or gesture, obtain reasoning results through large model reasoning in edge-cloud collaboration, and then present the reasoning results word by word on the terminal device 01.

[0127] by Figure 8 For example, after a user verbally requests "What's the weather like today?", the dynamic adjustment module in terminal device 01 receives the user's request and then... Figure 14 As shown, terminal device 01 displays the aforementioned user request 1402 on interface 1401, and after obtaining the reasoning result, presents the reasoning result to the user word by word on interface 1401, as shown in 1403. The “…” in 1403 indicates a streaming effect, continuously receiving and presenting. Figure 14 This is a schematic diagram illustrating the effect of one embodiment of this application.

[0128] In this embodiment, after a user submits a smart question-answering request, the terminal device 01 and the cloud server 00 perform on-demand end-to-end cloud collaborative acceleration inference. Through end-to-end cloud collaboration, the terminal device 01 simultaneously sends the user's request to both the cloud-side large model and the end-to-end large model for inference. When presenting the inference result from the cloud server 00 to the user, the return speed is controlled by adjusting the interval between character-by-character returns. This allows the cloud-side inference result to improve the problem of long initial character inference time on the end-to-end. Furthermore, in this embodiment, the terminal device 01 uses the inference result from the cloud server 00 as input for inference, thereby improving the accuracy of end-to-end inference.

[0129] Additionally, in some examples, for this application Figure 7 Regarding step 709 of the illustrated embodiment, if the large model inference module in cloud server 00 completes the cloud-side large model inference within the inference time carried by the network request, the end-side inference module in terminal device 01 can terminate the end-side large model inference early. Thus, when presenting the inference results, terminal device 01 uses only the inference results returned from the cloud side and directly presents all characters in the first buffer to the user without speed control. It is understood that the process terminates after step 709 in this embodiment. Figure 7 In the illustrated embodiment, the steps following step 709 do not need to be executed again.

[0130] Figure 15 This is a schematic diagram illustrating the presentation of reasoning results to a user, as provided in one embodiment of this application. Figure 15 As shown, the inference time for the first character in the edge-side inference module is 3-4 seconds; the time from when the terminal device 01 sends a network request to the cloud server 00 to when the terminal device 01 receives the inference result from the cloud server 00 is the round-trip time (RTT). In this embodiment, the RTT is 0.2 seconds, the inference time for the first character in the cloud server 00 is 100ms, and the inference time for each subsequent character is 35ms. Figure 12 In comparison, the difference lies in, Figure 15 In the process, when the dynamic adjustment module presents the inference results of cloud server 00 stored in the first buffer to the user, it no longer adjusts the time interval for character-by-character presentation, but instead directly presents all characters in the first buffer to the user. For example... Figure 15 As shown, the time interval for the first character is 350ms, and each subsequent character is presented immediately after being received by the terminal device 01.

[0131] Furthermore, in this embodiment, the cloud server 00 completes the inference of the large cloud model within the inference time carried by the network request. Thus, after the large cloud model inference is completed, the edge inference module also terminates the edge large model inference and directly presents the cloud-side inference result to the user. This also achieves the desired result. Figure 14 The effect shown is as follows. In addition, the terminal device 01 can also save the power consumption of the terminal device 01 by interrupting the inference of the large model on the terminal side in advance.

[0132] As described above, the embodiments of this application provide a method for obtaining inference results, which can enable collaborative inference between the terminal device and the cloud side. The inference results from the cloud side can be used to improve the problem of excessive time consumption for inferring the first character on the terminal device side. Furthermore, the inference accuracy on the terminal device side can be improved by inferring based on the inference results from the cloud side.

[0133] Figure 16A flowchart of a method for obtaining inference results provided in one embodiment of this application is shown below. Figure 16 As shown, the methods for obtaining the above reasoning results may include:

[0134] Step 1601: The terminal device sends a network request to the server, wherein the network request carries the first content of the user request.

[0135] In this embodiment, the terminal device can adopt Figure 4 The structure shown is implemented as follows. Specifically, users can trigger the terminal device to receive user requests through voice, key presses, and / or gestures. An example of the terminal device receiving user requests is as follows: Figure 8 As shown, it will not be elaborated further here.

[0136] After receiving the user request, the terminal device constructs a network request based on the user request and includes the first content of the user request in the network request.

[0137] Additionally, the aforementioned network request may also carry the server's inference time. In this embodiment, the aforementioned server can be... Figure 6 Cloud server 00 in the above-mentioned server, the inference time can be found in [link to cloud server 00]. Figure 7 The description in step 702 of the illustrated embodiment will not be repeated here.

[0138] Step 1602: The terminal device receives the inference result from the server sent by the aforementioned server.

[0139] The inference result sent by the server can be a partial result obtained through inference. Because the inference time is limited in the network request, the server only performs inference within that time. Therefore, the server may not be able to complete the entire inference process, and the inference result sent by the server is only a partial result obtained through inference.

[0140] Specifically, the terminal device receiving the server's inference result sent by the server can be as follows: the terminal device receives the server's inference result sent by the server through a long connection with the terminal device; wherein, the long connection is established between the server and the terminal device after the server receives the network request. In this embodiment, a schematic diagram of establishing a long connection between the cloud server and the terminal device can be shown as follows. Figure 9 As shown, it will not be elaborated further here.

[0141] In some examples, after receiving the inference result sent by the server, the terminal device can also store the inference result in a first buffer. In this way, when the terminal device presents the server's inference result to the user word by word, it can present the inference result stored in the first buffer word by word. When the terminal device performs inference based on the server's inference result to obtain the terminal device's inference result, it can perform inference based on the inference result stored in the first buffer to obtain the terminal device's inference result.

[0142] Step 1603: The terminal device presents the inference result of the server to the user word by word, and performs inference based on the inference result of the server and the first content to obtain the inference result of the terminal device.

[0143] Specifically, the terminal device presents the server's inference results to the user word by word as follows: the terminal device adjusts the time interval for word-by-word presentation, and then presents the server's inference results to the user word by word according to the adjusted time interval. An example of the terminal device presenting the inference results is as follows: Figure 12 As shown, it will not be elaborated further here.

[0144] Furthermore, during the process of presenting the server's inference results to the user character by character, if the number of characters obtained by the terminal device through inference is equal to the number of characters already presented to the user, the terminal device can cancel the adjustment of the time interval for character-by-character presentation and directly present the server's inference results to the user character by character. A specific example is shown below. Figure 13 As shown, it will not be elaborated further here.

[0145] In this embodiment, the terminal device performs reasoning based on the inference results from the server and the first content to obtain the inference result of the terminal device. This can be achieved as follows: if the server's inference results contain a result corresponding to the (N-1)th round of reasoning by the terminal device, then the result corresponding to the (N-1)th round of reasoning by the server is used as the input for the Nth round of reasoning by the terminal device to obtain the Nth round of reasoning result; where N is an integer, N≥2; if the server's inference results do not contain a result corresponding to the (N-1)th round of reasoning by the terminal device, then the (N-1)th round of reasoning result of the terminal device is used as the input for the Nth round of reasoning by the terminal device to obtain the Nth round of reasoning result. Furthermore, when performing the first round of reasoning, the terminal device performs the first round of reasoning based on the first content to obtain the first round of reasoning result.

[0146] An example of inference performed by a terminal device can be as follows: Figure 10 As shown, it will not be elaborated further here.

[0147] Step 1604: After the inference results of the above server are presented, the terminal device presents the inference results of the terminal device that follow the inference results of the above server to the user word by word.

[0148] Since the server inference ends after a limited inference time, the server may not be able to obtain a complete inference result. Subsequent inference needs to be completed by the terminal device. In other words, in this embodiment, the terminal device performs the complete inference process. However, since the terminal device's first-word inference takes a long time, in order to improve this problem, the server's inference speed is utilized. After the terminal device receives the server's inference result, it can first present the server's inference result to the user. After the server's inference result is presented, the terminal device will then present the inference result following the server's inference result to the user word by word.

[0149] Furthermore, in step 1603 of this embodiment, after obtaining the inference result of the terminal device, the terminal device can also store the inference result in a second buffer. In this way, when presenting the inference result to the user, the terminal device can present the inference result stored in the second buffer that follows the server's inference result.

[0150] In some examples, if the server completes its inference within the specified inference time, the terminal device can terminate the inference process on the terminal side after the server inference is completed, and when presenting the inference results, the terminal device will use the server's inference results entirely.

[0151] In the above method for obtaining the inference result, after the terminal device sends a network request to the server, it receives the server's inference result, which includes a portion of the result obtained through server inference. Then, the terminal device presents the server's inference result to the user character by character, and performs inference based on the server's inference result and the first content of the user's request to obtain the terminal device's inference result. After the server's inference result is presented, the terminal device presents the inference result following the server's inference result to the user character by character. This enables collaborative inference between the terminal device and the server, improving the problem of excessive time spent inferring the first character on the terminal device side using the server's inference result, and improving the inference accuracy on the terminal device side by performing inference based on the server's inference result.

[0152] It is understood that some or all of the steps or operations in the above embodiments are merely examples, and other operations or variations thereof can be performed in the embodiments of this application. Furthermore, the steps may be performed in different orders as presented in the above embodiments, and it is not necessary to perform all the operations in the above embodiments.

[0153] It is understood that, in order to achieve the above-mentioned functions, the terminal device includes hardware and / or software modules corresponding to the execution of each function. Based on the algorithm steps of the various examples described in conjunction with the embodiments disclosed in this application, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed by hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application in conjunction with the embodiments, but such implementation should not be considered beyond the scope of this application.

[0154] This embodiment can divide the terminal device into functional modules according to the above method embodiment. For example, each function can be divided into its own functional module, or two or more functions can be integrated into one module. The integrated module can be implemented in hardware. It should be noted that the module division in this embodiment is illustrative and only represents one logical functional division. In actual implementation, there may be other division methods.

[0155] Figure 17 This is a schematic diagram of the structure of a terminal device provided in another embodiment of this application. In the case where functional modules are divided according to their respective functions, Figure 17 A schematic diagram of a possible composition of the terminal device 1700 involved in the above embodiments is shown, such as... Figure 17 As shown, the terminal device 1700 may include: a receiving unit 1701, a processing unit 1702, and a sending unit 1703;

[0156] The receiving unit 1701 can be used to support the terminal device 1700 in executing steps 1601 and 1604, and / or for other processes of the technical solutions described in the embodiments of this application.

[0157] Processing unit 1702 can be used to support terminal device 1700 in executing steps 1602, 1605 and 1606, etc., and / or other processes used in the technical solutions described in the embodiments of this application;

[0158] The sending unit 1703 can be used to support the terminal device 1700 in executing step 1603, etc., and / or in other processes of the technical solutions described in the embodiments of this application.

[0159] It should be noted that all relevant content of each step involved in the above method embodiments can be referenced from the functional description of the corresponding functional module, and will not be repeated here.

[0160] The terminal device 1700 provided in this embodiment is used to execute the above-described method for obtaining the reasoning result, and thus can achieve the same effect as the above-described method.

[0161] It should be understood that terminal device 1700 can correspond to Figure 4 The terminal device 100 shown. The functions of the receiving unit 1701 and the transmitting unit 1703 can be determined by... Figure 4 The processor 110, antenna 1, and mobile communication module 150 in the terminal device 100 shown are, and / or, implemented by the processor 110, antenna 2, and wireless communication module 160; the function of the processing unit 1702 can be implemented by the processor 110, touch sensor 180K, and display screen 194 in the terminal device 100 shown in FIG1.

[0162] When using integrated units, the terminal device 1700 may include a processing module, a storage module, and a communication module.

[0163] The processing module can be used to control and manage the actions of the terminal device 1700. For example, it can support the terminal device 1700 in executing the steps performed by the receiving unit 1701, processing unit 1702, and sending unit 1703. The storage module can support the terminal device 1700 in storing program code and data. The communication module can support communication between the terminal device 1700 and other devices.

[0164] The processing module can be a processor or controller, which can implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. The processor can also be a combination of functions that implement computing capabilities, such as a combination of one or more microprocessors, a combination of digital signal processing (DSP) and a microprocessor, etc. The storage module can be a memory. The communication module can specifically be a device that interacts with other electronic devices, such as radio frequency circuitry, a Bluetooth chip, and / or a Wi-Fi chip.

[0165] In one embodiment, when the processing module is a processor and the storage module is a memory, the terminal device 1700 involved in this embodiment can be a device having... Figure 4 The device with the structure shown.

[0166] This application also provides a computer-readable storage medium storing a computer program that, when run on a computer, causes the computer to execute this application. Figures 7-16The method provided in the illustrated embodiment.

[0167] This application also provides a computer program product, which includes a computer program that, when run on a computer, causes the computer to execute this application. Figures 7-16 The method provided in the illustrated embodiment.

[0168] In this application embodiment, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent the existence of A alone, A and B simultaneously, or B alone. A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" and similar expressions refer to any combination of these items, including any combination of singular or plural items. For example, at least one of a, b, and c can represent: a, b, c, a and b, a and c, b and c, or a and b and c, where a, b, and c can be single or multiple.

[0169] Those skilled in the art will recognize that the units and algorithm steps described in the embodiments disclosed herein can be implemented using electronic hardware, computer software, or a combination of electronic hardware and software. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0170] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0171] In the several embodiments provided in this application, any function, if implemented as a software functional unit and sold or used as an independent product, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, essentially, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0172] The above description is merely a specific embodiment of this application. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the protection scope of this application. The protection scope of this application should be determined by the protection scope of the claims.

Claims

1. A method for obtaining a reasoning result, characterized in that, include: The terminal device sends a network request to the server; wherein the network request carries the first content of the user's request; Receive the inference result sent by the server; wherein the inference result of the server includes a portion of the result obtained by the server inference; The inference results of the server are presented to the user word by word, and the inference results of the terminal device are obtained by inferring based on the inference results of the server and the first content. After the server has finished presenting its inference results, the terminal device will present the inference results following the server's inference results to the user word by word.

2. The method according to claim 1, characterized in that, The step of inferring the terminal device based on the inference result of the server and the first content includes: If the inference result of the server contains a result corresponding to the inference of the terminal device in round N-1, then the result corresponding to the inference of the terminal device in round N-1 is used as the input of the inference of the terminal device in round N to obtain the inference result of the terminal device in round N; where N is an integer and N≥2; If the inference results of the server do not contain a result corresponding to the (N-1)th round of inference of the terminal device, then the (N-1)th round of inference result of the terminal device is used as the input of the Nth round of inference of the terminal device to obtain the Nth round of inference result of the terminal device.

3. The method according to claim 2, characterized in that, Also includes: Based on the first content, a first round of reasoning is performed to obtain the first round of reasoning results of the terminal device.

4. The method according to claim 1, characterized in that, Presenting the server's reasoning results to the user word by word includes: Adjust the time interval for word-by-word presentation; The inference results from the server are presented to the user word by word at the adjusted time intervals.

5. The method according to claim 4, characterized in that, Also includes: In the process of presenting the server's inference results to the user word by word, if the number of characters obtained by the terminal device inference is equal to the number of characters already presented to the user, then the adjustment of the word-by-word presentation time interval is canceled, and the server's inference results are directly presented to the user word by word.

6. The method according to claim 1, characterized in that, The receipt of the inference result sent by the server includes: The server receives the inference result sent by the server through a long connection with the terminal device; wherein the long connection is established by the server with the terminal device after receiving the network request.

7. The method according to claim 1, characterized in that, After receiving the inference result sent by the server, the method further includes: The inference results of the server are stored in the first buffer.

8. The method according to claim 1, characterized in that, After obtaining the inference result of the terminal device, the method further includes: The inference results of the terminal device are stored in the second buffer.

9. A terminal device, characterized in that, include: One or more processors; Memory; Multiple applications; And one or more computer programs, wherein the one or more computer programs are stored in the memory, the one or more computer programs including instructions that, when executed by the terminal device, cause the terminal device to perform the method as described in any one of claims 1-8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when run on a computer, causes the computer to perform the method as described in any one of claims 1-8.

Images