Inference method and device of large language model, electronic equipment and readable storage medium

Abstract

The present disclosure provides a large language model inference method, device, electronic equipment and readable storage medium, relating to artificial intelligence technical fields such as natural language processing, large language model, speech recognition, cloud service. The large language model inference method comprises: obtaining a plurality of text segments obtained by performing streaming speech recognition on voice data input by a user; inputting the plurality of text segments into a large language model, performing pre-padding processing on each text segment by the large language model to obtain a pre-padding result of each text segment; in response to determining that the user stops inputting voice data, obtaining a target padding result according to the pre-padding results of all text segments; inputting the target pre-padding result into the large language model, performing forward propagation processing and generating output processing on the target pre-padding result by the large language model to obtain a predicted text, and taking the predicted text as an inference result of the large language model.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of Internet, in particular to the technical field of artificial intelligence such as natural language processing, large language model, speech recognition, cloud service, etc. A large language model inference method and device, an electronic device and a readable storage medium are provided. BACKGROUND

[0002] With the rapid development of large language model technology, more and more users begin to use large language models to meet the needs of work and life or problems encountered. In actual use, users have higher and higher requirements for the delay of large language models, especially in the voice dialogue scene. Users hope that after issuing a voice question, the large language model can reply within a short time. Therefore, how to improve the inference speed of the large language model is a technical problem to be solved. SUMMARY

[0003] According to a first aspect of the present disclosure, a large language model inference method is provided, comprising: obtaining a plurality of text segments obtained by performing streaming speech recognition on voice data input by a user; inputting the plurality of text segments into a large language model, performing pre-padding processing on each text segment by the large language model to obtain a pre-padding result of each text segment; in response to determining that the user stops inputting voice data, obtaining a target padding result according to the pre-padding results of all text segments; inputting the target pre-padding result into the large language model, performing forward propagation processing and output generation processing on the target pre-padding result by the large language model to obtain a predicted text, and taking the predicted text as an inference result of the large language model.

[0004] According to a second method of the present disclosure, a large language model inference device is provided, comprising: an obtaining unit configured to obtain a plurality of text segments obtained by performing streaming speech recognition on voice data input by a user; a first inference unit configured to input the plurality of text segments into a large language model, perform pre-padding processing on each text segment by the large language model to obtain a pre-padding result of each text segment; a processing unit configured to, in response to determining that the user stops inputting voice data, obtain a target padding result according to the pre-padding results of all text segments; and a second inference unit configured to input the target pre-padding result into the large language model, perform forward propagation processing and output generation processing on the target pre-padding result by the large language model to obtain a predicted text, and take the predicted text as an inference result of the large language model.

[0005] According to a third aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected with the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method as described above.

[0006] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to perform the method as described above.

[0007] According to a fifth aspect of the present disclosure, a computer program product is provided, comprising a computer program which, when executed by a processor, implements the method as described above.

[0008] It should be understood that the content described in this part is not intended to identify key or important features of the embodiments of the present disclosure, nor to limit the scope of the present disclosure. Other features of the present disclosure will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0009] The accompanying drawings are used to better understand the present scheme and do not limit the present disclosure. Among them:

[0010] Figure 1 is a schematic diagram according to the first embodiment of the present disclosure;

[0011] Figure 2 is a schematic diagram according to the second embodiment of the present disclosure;

[0012] Figure 3 is a schematic diagram according to the third embodiment of the present disclosure;

[0013] Figure 4 is a schematic diagram according to the fourth embodiment of the present disclosure;

[0014] Figure 5 is a block diagram of an electronic device for implementing the inference method of a large language model according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0015] Exemplary embodiments of the present disclosure are described below with reference to the accompanying drawings, which include various details of the embodiments of the present disclosure to help understanding, and should be considered as merely exemplary. Therefore, those of ordinary skill in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Also, in order to be clear and concise, descriptions of well-known functions and mechanisms are omitted in the following description.

[0016] Figure 1 is a schematic diagram according to the first embodiment of the present disclosure. AsFigure 1 As shown, the inference method of the large language model of the embodiment specifically includes the following steps:

[0017] S101, obtaining a plurality of text segments obtained by performing streaming speech recognition on the user input speech data;

[0018] S102, inputting the plurality of text segments into the large language model, performing pre-filling processing on each text segment by the large language model, and obtaining a pre-filling result of each text segment;

[0019] S103, in response to determining that the user stops inputting the speech data, obtaining a target filling result according to the pre-filling results of all text segments;

[0020] S104, inputting the target pre-filling result into the large language model, performing forward propagation processing and output generation processing on the target pre-filling result by the large language model, obtaining a predicted text, and taking the predicted text as the inference result of the large language model.

[0021] The inference method of the large language model of the embodiment is suitable for a voice dialogue scene, and a plurality of text segments obtained according to speech data are input into the large language model, and the large language model will immediately perform pre-filling processing on each text segment in the inference process after receiving each text segment. In the case of determining that the user stops inputting the speech data, the large language model performs forward propagation processing and output generation processing on the pre-filling results of all text segments in the remaining stage of the inference process, thereby obtaining the final predicted text.

[0022] Since the large language model in the embodiment starts pre-filling processing on the text segments in advance according to the time of user input speech data, it is not necessary to perform pre-filling processing on the complete text sequence after the user stops inputting the speech data, but only the target pre-filling result obtained needs to be processed in the remaining stage of the inference process. Therefore, the embodiment can greatly reduce the inference time of the large language model, reduce the output time of the predicted text, and improve the inference response speed of the large language model.

[0023] In the embodiment, the inference process of the large language model (Large Language Model, LLM) mainly includes three stages, namely, the pre-filling stage, the forward propagation stage and the output generation stage, and each stage is performed sequentially.

[0024] The pre-padding stage is used for the large language model to obtain a token vector sequence corresponding to the input text (segment or sequence). The pre-padding stage includes tokenization processing, masking processing, position encoding processing, embedding vector generation processing, key value caching (KVCaching) processing, and the like. The forward propagation stage is used for the large language model to obtain an output vector sequence by processing the token vector sequence obtained in the pre-padding stage through a plurality of Transformer modules. The output generation stage is used for obtaining an output text (i.e., predicted text) according to the output vector sequence obtained in the forward propagation stage.

[0025] The inference method of the large language model of the embodiment can be applied to a scenario in which a user has a conversation or chat with a virtual digital person or an artificial intelligence robot, that is, the large language model performs inference according to text data converted from voice data input by the user to the virtual digital person or the artificial intelligence robot, to obtain predicted text corresponding to the input voice data.

[0026] The plurality of text segments obtained in S101 are obtained by performing streaming voice recognition on the voice data input by the user. The streaming voice recognition process can include: collecting the voice data input by the user in real time, performing noise reduction, voice enhancement, and the like on the collected voice data, and performing streaming voice recognition on the voice data after processing by using a streaming voice recognition algorithm, to obtain the plurality of text segments.

[0027] In the embodiment, streaming voice recognition refers to cutting the voice data obtained in real time into a plurality of time slices (i.e., frames, each frame containing several to tens of milliseconds of voice data) while the user is speaking, and recognizing each time slice to obtain a plurality of text segments, instead of performing voice recognition on the complete voice data after waiting for the user to finish speaking.

[0028] After obtaining the plurality of text segments in S101, the embodiment inputs the plurality of text segments to the large language model in S102, and performs pre-padding processing on each text segment by the large language model to obtain a pre-padding result of each text segment. The pre-padding result of each text segment obtained in the embodiment is a token vector sequence.

[0029] In the implementation of inputting the plurality of text segments to the large language model in S102, the implementation manner can be as follows: generating a text data stream according to the plurality of text segments and time information of each text segment (for example, the collection time of the voice data of the text segment); and inputting different text segments in the generated text data stream to the large language model in sequence according to the time sequence.

[0030] In the process of inputting the voice data by the user, the large language model in this embodiment continuously receives the text segments in the text data stream, and after receiving each text segment, the large language model immediately performs the pre-padding processing on the currently received text segment, thereby obtaining the pre-padding result corresponding to the current text segment.

[0031] The pre-padding result of the text segment obtained in S102 of this embodiment is a token vector sequence, and the token vector sequence includes a vector corresponding to each token in the text segment. The token is the smallest unit that cannot be divided in the text, which can be a word, a token, a symbol, a phrase or any text part with independent semantic or syntactic function.

[0032] In the implementation of performing the pre-padding processing on each text segment by the large language model in S102, the large language model can perform the tokenization processing, the mask processing, the position encoding processing, the embedding vector generation processing and the key-value cache processing on the text segment in sequence, thereby obtaining the token vector sequence corresponding to the text segment and completing the pre-padding processing.

[0033] After obtaining the pre-padding result of each text segment in S102, the target pre-padding result is obtained according to the pre-padding results of all text segments in S103 in response to determining that the user stops inputting the voice data.

[0034] In S103, the user can be determined to stop inputting the voice data in the case that the large language model receives the voice end signal sent by the terminal device; or the user can be determined to stop inputting the voice data in the case that the large language model determines that the time difference between the current time and the input time of the previous text segment is greater than the preset time threshold.

[0035] That is, the large language model in this embodiment can determine whether the user stops inputting the voice data according to itself or the terminal device, thereby improving the accuracy of the determination and the accuracy of the target pre-padding result obtained.

[0036] The implementation manner that can be adopted by the embodiment when performing S103 to obtain the target pre-filling result according to the pre-filling results of all text segments can be: obtaining time information of the plurality of pre-filling results, and obtaining time information of the text segment corresponding to the pre-filling result as the time information of the corresponding pre-filling result; and in chronological order, sequentially splicing the plurality of pre-filling results, and taking the splicing result as the target pre-filling result, which is the token vector sequence corresponding to the complete voice data input by the user.

[0037] After performing S103 to obtain the target pre-filling result, the embodiment performs S104 to input the target pre-filling result to the large language model, and performs forward propagation processing and output generation processing on the target pre-filling result by the large language model to obtain a predicted text, which is taken as the inference result of the large language model.

[0038] That is, the embodiment divides the inference process of the large language model into two parts, the first part is to pre-fill each input text segment in advance, and the second part is to perform forward propagation processing and output generation processing on the obtained target pre-filling result.

[0039] Therefore, the embodiment achieves the purpose of completing the pre-filling stage in advance by using the time of the user input voice data, and after determining that the user ends the input of the voice data, the embodiment only needs to perform forward propagation processing and output generation processing on the obtained target pre-filling result, thereby greatly reducing the inference time of the large language model, reducing the output time of the predicted text, and improving the inference response speed of the large language model.

[0040] After performing S104 to obtain the predicted text, the embodiment can directly display the obtained predicted text to the user, so that the user can obtain a reply in the form of text.

[0041] In addition, after performing S104 to obtain the predicted text, the embodiment can further include the following content: generating reply voice data according to the predicted text; and playing the generated reply voice data to the user.

[0042] The embodiment can also play the generated target voice data to the user in a streaming output manner, so that the user can obtain a reply in the form of voice as soon as possible, thereby reducing the delay of the first packet reply.

[0043] That is, the embodiment can also reply in the form of target voice data converted from the predicted text, thereby improving the user experience when performing voice interaction.

[0044] Figure 2 is a schematic diagram according to the second embodiment of the present disclosure. Figure 2An architecture diagram of the inference method of the large language model of the embodiment is shown in FIG. 1. The architecture diagram includes a terminal device (i.e., a front end), a server end (i.e., a back end, such as a speech processing server end), and a large language model server end (for executing the inference method of the embodiment); the terminal device collects speech data input by a user in real time, and sends the collected speech data to the server end; the server end performs streaming speech recognition on the speech data, and sends a plurality of text segments obtained to the large language model server end; the large language model server end inputs the plurality of text segments to the large language model, and performs pre-padding processing on each text segment by the large language model to obtain a pre-padding result of each text segment; the terminal device generates a speech end signal when detecting that the user stops speech input, and sends the speech end signal to the large language model server end through the server end; the large language model server end inputs the target pre-padding result to the large language model after receiving the speech end signal, and performs forward propagation processing and generation output processing on the target pre-padding result by the large language model to obtain a predicted text; the large language model server end sends the predicted text to the server end, and the server end converts the predicted text into target speech data and sends the target speech data to the terminal device; and the terminal device plays the received target speech data.

[0045] Figure 3 is a schematic diagram according to the third embodiment of the present disclosure. Figure 3 The difference between the inference method of the large model of the embodiment and the inference method of a conventional large model is shown in FIG. 1. After the inference method of the embodiment obtains different text segments at different times, the large language model performs pre-padding processing, and when an end marker is received, the large language model performs forward propagation processing and generation output processing on the obtained pre-padding result, so that the predicted text obtained can be output at T3 time on the left. In contrast, the inference method of the conventional large model obtains different text segments at different times, first splices the plurality of text segments, and does not perform other processing. When an end marker is received, the inference method of the conventional large model performs pre-padding processing, forward propagation processing, and generation output processing on the spliced text segment sequence in sequence, so that the predicted text obtained can be output at T3 time on the right. There is a significant time delay between the left T3 time and the right T3 time, so the embodiment can greatly reduce the inference time of the large language model, reduce the output time of the predicted text, and improve the inference response speed of the large language model.

[0046] Figure 4 is a schematic diagram according to the fourth embodiment of the present disclosure. As shown in Figure 4 The inference device 400 of the large language model of the embodiment includes:

[0047] The acquisition unit 401 acquires a plurality of text segments obtained by performing streaming speech recognition on voice data input by a user.

[0048] The first inference unit 402 inputs the plurality of text segments into a large language model, performs pre-filling processing on each text segment by the large language model, and obtains a pre-filling result of each text segment.

[0049] The processing unit 403 obtains a target filling result according to the pre-filling results of all text segments in response to determining that the user stops inputting voice data.

[0050] The second inference unit 404 inputs the target pre-filling result into the large language model, performs forward propagation processing and output generation processing on the target pre-filling result by the large language model, obtains a predicted text, and takes the predicted text as an inference result of the large language model.

[0051] In this embodiment, the inference process of the large language model (LLM) mainly includes three stages, namely, a pre-filling stage, a forward propagation stage, and an output generation stage, which are performed sequentially.

[0052] The plurality of text segments acquired by the acquisition unit 401 are acquired by performing streaming speech recognition on voice data input by a user. The streaming speech recognition process can include: collecting voice data input by a user in real time, performing noise reduction, speech enhancement, etc. on the collected voice data, and then using a streaming speech recognition algorithm to perform streaming speech recognition on the processed voice data, thereby obtaining a plurality of text segments.

[0053] After the acquisition unit 401 acquires a plurality of text segments, the first inference unit 402 inputs the plurality of text segments into a large language model, and performs pre-filling processing on each text segment by the large language model, thereby obtaining a pre-filling result of each text segment. The pre-filling result of each text segment obtained by the first inference unit 402 is a token vector sequence.

[0054] When the first inference unit 402 inputs the plurality of text segments into the large language model, an implementation manner can be used as follows: generating a text data stream according to the plurality of text segments and time information (for example, the collection time of voice data of the text segment) of each text segment; and inputting different text segments in the generated text data stream into the large language model in chronological order.

[0055] In the process of receiving the speech data input by the user, the large language model in this embodiment continuously receives the text segments in the text data stream, and immediately performs pre-padding processing on the currently received text segment after receiving each text segment, thereby obtaining a pre-padding result corresponding to the current text segment.

[0056] The pre-padding result of the text segment obtained by the first inference unit 402 is specifically a token vector sequence, which includes a vector corresponding to each token in the text segment. A token is the smallest unit of text that cannot be divided, which can be a word, a token, a symbol, a phrase, or any text part with independent semantic or syntactic function.

[0057] When the large language model performs pre-padding processing on each text segment, the first inference unit 402 can perform tokenization processing, mask processing, position encoding processing, embedding vector generation processing, and key-value cache processing on the text segment in sequence by the large language model, thereby obtaining a token vector sequence corresponding to the text segment and completing the pre-padding processing.

[0058] After the first inference unit 402 obtains the pre-padding result of each text segment, the processing unit 403 obtains a target pre-padding result according to the pre-padding results of all text segments in response to determining that the user stops inputting speech data.

[0059] The processing unit 403 can determine that the user stops inputting speech data in the case that the large language model receives a speech end signal sent by the terminal device. The processing unit 403 can also determine that the user stops inputting speech data in the case that the large language model determines that the time difference between the current time and the input time of the previous text segment is greater than a preset time threshold.

[0060] That is, the processing unit 403 can determine whether the user stops inputting speech data according to the large language model itself or the terminal device, thereby improving the accuracy of the determination and the accuracy of the target pre-padding result obtained.

[0061] When obtaining the target pre-padding result according to the pre-padding results of all text segments, the processing unit 403 can adopt the following implementation manner: obtaining time information of the plurality of pre-padding results; sequentially splicing the plurality of pre-padding results in the order of time, and taking the splicing result as the target pre-padding result, which is a token vector sequence corresponding to the complete speech data input by the user.

[0062] After the target pre-filling result is obtained by the processing unit 403, the second inference unit 404 inputs the target pre-filling result into the large language model, and the large language model performs forward propagation processing and output generation processing on the target pre-filling result to obtain the predicted text, which is taken as the inference result of the large language model.

[0063] That is, the embodiment divides the inference process of the large language model into two parts, the first part is to pre-fill each input text segment in advance, and the second part is to perform forward propagation processing and output generation processing on the obtained target pre-filling result.

[0064] Therefore, the embodiment realizes the purpose of completing the pre-filling stage in advance by using the time of user input voice data, and after determining that the user ends the input of voice data, the embodiment only needs to perform forward propagation processing and output generation processing on the obtained target pre-filling result, thereby greatly reducing the inference time of the large language model, reducing the output time of the predicted text, and improving the inference response speed of the large language model.

[0065] After obtaining the predicted text, the second inference unit 404 can directly display the obtained predicted text to the user, so that the user can obtain the reply in the form of text.

[0066] In addition, after obtaining the predicted text, the second inference unit 404 can further include the following content: generating reply voice data according to the predicted text; and playing the generated reply voice data to the user.

[0067] That is, the embodiment can also convert the predicted text into the form of target voice data for reply, thereby improving the user experience when performing voice interaction.

[0068] In the technical solution of the present disclosure, the acquisition, storage and application of user personal information comply with relevant laws and regulations and do not violate public order and good customs.

[0069] According to the embodiments of the present disclosure, the present disclosure further provides an electronic device, a readable storage medium and a computer program product.

[0070] As Figure 5FIG. 1 illustrates a block diagram of an electronic device for a reasoning method of a large language model according to embodiments of the present disclosure. The electronic device is intended to represent various forms of digital computers, such as laptops, desktops, tablets, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular telephones, smartphones, wearable devices, and other similar computing devices. The components shown in FIG. 1, their connections and relationships, and their functions, are meant to be examples only, and are not meant to limit implementations of the present disclosure described and / or claimed in this document.

[0071] As shown in FIG. 1, the device 500 includes a computing unit 501 that can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 502 or a computer program loaded into a random access memory (RAM) 503 from a storage unit 508. In the RAM 503, various programs and data required for the operation of the device 500 can also be stored. The computing unit 501, the ROM 502, and the RAM 503 are connected to each other through a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504. Figure 5

[0072] Various components in the device 500 are connected to the I / O interface 505, including an input unit 506, such as a keyboard, a mouse, and the like; an output unit 507, such as various types of displays, speakers, and the like; a storage unit 508, such as a magnetic disk, an optical disk, and the like; and a communication unit 509, such as a network card, a modem, a wireless communication transceiver, and the like. The communication unit 509 allows the device 500 to exchange information / data with other devices through a computer network, such as the Internet, and / or various telecommunication networks.

[0073] The computing unit 501 can be various general and / or special purpose processing components with processing and computing capabilities. Some examples of the computing unit 501 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, and the like. The computing unit 501 performs various methods and processes described above, such as the reasoning method of a large language model. For example, in some embodiments, the reasoning method of a large language model can be implemented as a computer software program that is tangibly embodied in a machine-readable medium, such as the storage unit 508.

[0074] ​In some embodiments, portions or all of the computer program can be loaded onto the apparatus 500 via the ROM 502 and / or the communication unit 509. When the computer program is loaded onto the RAM 503 and executed by the computation unit 501, one or more steps of the inference method of the large language model described above can be performed. Alternatively, in other embodiments, the computation unit 501 can be configured to perform the inference method of the large language model by any other suitable means, for example, by means of firmware.

[0075] Various implementations of the systems and techniques described here can be realized in digital electronic circuitry, integrated circuitry, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on a chip (SOC), a complex programmable logic device (CPLD), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0076] Program code for carrying out methods of the present disclosure can be written in any combination of one or more programming languages. The program code can be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable processing apparatus to produce a machine, such that the program code, when executed by the processor or controller, implements the functions / acts specified in the flowcharts and / or block diagrams. The program code can be executed entirely on a machine, partially on a machine, partially on a machine and partially on a remote machine or entirely on a remote machine or server.

[0077] In the context of this disclosure, a machine-readable medium can be a tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include but is not limited to an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium will include one or more lines of electrical connections, portable computer disks, hard disk drives, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or Flash memory), optical fibers, portable compact disc read-only memories (CD-ROMs), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0078] To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0079] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0080] The computer system can include clients and servers. This relationship can be. remote, where each server is stored on a remote computer from a client. The clients and the servers can be connected through a communication network. The relationship can be a client-server relationship over a network. A server can be a cloud server, also known as a cloud computing server or a cloud host, which is a host product in the cloud computing service system, and solves the defects of large management difficulty and weak business scalability in traditional physical hosts and VPS services (Virtual Private Server, or VPS for short). The server can also be a server of a distributed system, or a server combined with a blockchain.

[0081] It should be understood that the steps shown above can be reordered, added, or deleted. For example, the steps described in the present disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in the present disclosure can be achieved, and the present disclosure is not limited herein.

[0082] The above detailed description does not constitute a limitation on the protection scope of the present disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modifications, equivalent replacements and improvements made within the spirit and principles of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A reasoning method of a large language model, comprising: obtaining a plurality of text segments obtained by performing streaming speech recognition on speech data input by a user; inputting the plurality of text segments into a large language model, performing pre-padding processing on each text segment by the large language model, and obtaining a pre-padding result of each text segment; in response to determining that the user stops inputting speech data, obtaining a target pre-padding result according to the pre-padding results of all text segments; inputting the target pre-padding result into the large language model, performing forward propagation processing and output generation processing on the target pre-padding result by the large language model, obtaining predicted text, and taking the predicted text as a reasoning result of the large language model; wherein the inputting the plurality of text segments into a large language model comprises: generating a text data stream according to the plurality of text segments and time information of each text segment; inputting different text segments in the text data stream into the large language model in chronological order; the obtaining a target pre-padding result according to the pre-padding results of all text segments comprises: obtaining time information of the plurality of pre-padding results; sequentially splicing the plurality of pre-padding results in chronological order, and taking a splicing result as the target pre-padding result.

2. The method of claim 1, wherein, The determining that the user stops inputting speech data comprises: in a case where the large language model receives a speech end signal sent by a terminal device, determining that the user stops inputting speech data.

3. The method of claim 1, wherein, The determining that the user stops inputting speech data comprises: in a case where the large language model determines that a time difference between a current time and an input time of a previous text segment is greater than a preset time threshold, determining that the user stops inputting speech data.

4. The method of claim 1, further comprising: after obtaining the predicted text, generating reply speech data according to the predicted text; playing the reply speech data to the user.

5. A reasoning apparatus of a large language model, comprising: an obtaining unit configured to obtain a plurality of text segments obtained by performing streaming speech recognition on speech data input by a user; a first reasoning unit configured to input the plurality of text segments into a large language model, perform pre-padding processing on each text segment by the large language model, and obtain a pre-padding result of each text segment; a processing unit configured to, in response to determining that the user stops inputting speech data, obtain a target pre-padding result according to the pre-padding results of all text segments; a second reasoning unit configured to input the target pre-padding result into the large language model, perform forward propagation processing and output generation processing on the target pre-padding result by the large language model, obtain predicted text, and take the predicted text as a reasoning result of the large language model; wherein the first reasoning unit, when inputting the plurality of text segments into a large language model, specifically performs: generating a text data stream according to the plurality of text segments and time information of each text segment; inputting different text segments in the text data stream into the large language model in chronological order; The processing unit specifically performs the following when obtaining the target prefill result according to the prefilled results of all text segments: obtain time information of the plurality of prefilled results; sequentially splice the plurality of prefilled results in chronological order, and take the spliced result as the target prefill result.

6. The apparatus of claim 5, wherein, The processing unit specifically performs the following when determining that the user stops inputting voice data: In a case where the large language model receives a voice end signal sent by the terminal device, it is determined that the user stops inputting voice data.

7. The apparatus of claim 5, wherein, The processing unit specifically performs the following when determining that the user stops inputting voice data: In a case where the large language model determines that a time difference between the current time and the input time of the previous text segment is greater than a preset time threshold, it is determined that the user stops inputting voice data.

8. The apparatus of claim 5, wherein the second inference unit is further configured to perform: after obtaining the predicted text, generating reply voice data according to the predicted text; playing the reply voice data to the user.

9. An electronic device, comprising: at least one processor; and a memory connected in communication with the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-4.

10. A non-transitory computer readable storage medium having stored thereon computer instructions, wherein, The computer instructions are used to enable the computer to perform the method of any one of claims 1-4.

11. A computer program product comprising a computer program which, when executed by a processor, implements the method of any one of claims 1-4.

Images