Voice data processing method and device and electronic equipment

Abstract

The invention provides a voice data processing method and device and electronic equipment. The method comprises the following steps: acquiring question text data; inputting the questionnaire text data into a large language model to generate an answer text for replying the questionnaire text data by using the large language model; in the process that the large language model outputs an answer text for the question text data, calling a voice generation service to convert a clause text into clause voice data when receiving one clause text in the answer text; and in the process of converting the phrase text into phrase voice data by the voice generation service, transmitting sub-voice data to a client when the sub-voice data is successfully converted for a preset duration, so that the client plays a corresponding answer voice based on the sub-voice data. Therefore, the waiting time before the text is converted into the voice and before the answer voice is transmitted can be reduced, and the processing efficiency is improved.

Description

Technical Field

[0001] The present application relates to the field of intelligent robot technology, and in particular to a voice data processing method, device and electronic equipment. Background Art

[0002] With the development of the Internet and the explosive growth of digital information, how to improve the efficiency and accuracy of intelligent question answering has become a research hotspot. Intelligent question answering (QA) aims to automatically provide answers to natural language questions raised by users and is one of the most representative tasks in the field of natural language processing (NLP). Large language model (LLM), also known as large language model, is an artificial intelligence model that aims to understand and generate human language and is often used in intelligent question answering scenarios (such as AI virtual people making phone calls, also known as real-time voice chat scenarios).

[0003] At present, in the scenario where the AI ​​virtual person makes a phone call, the client performs text recognition on the user's voice to obtain the question text, and then sends the question text to the server. The server generates a reply text through LLM, and then calls the Text To Speech (TTS) service to convert the complete reply text into reply voice data, and returns the reply voice data to the client. After receiving it, the client plays the voice as the voice answer of the AI ​​virtual person.

[0004] However, in the above-mentioned voice question and answer implementation, it is necessary to wait for LLM to output the complete reply text before executing the text-to-speech operation, and it is necessary to wait for TTS to convert the complete reply voice data before executing the transmission reply voice data operation. Therefore, the waiting time is relatively long and the processing efficiency is low. Summary of the invention

[0005] The purpose of the embodiments of the present application is to provide a method, device and electronic device for processing voice data to solve the problem of long waiting time and low efficiency in the current voice question and answer implementation method. The specific technical solution is as follows:

[0006] In a first aspect, the present application provides a voice data processing method, which is applied to a server, comprising:

[0007] Get the question text data;

[0008] Inputting the question text data into a large language model to generate a reply text for replying to the question text data using the large language model;

[0009] In the process of the large language model outputting the answer text for the question text data, each time a sentence text in the answer text is received, the speech generation service is called to convert the sentence text into sentence speech data;

[0010] In the process of the speech generation service converting the sentence text into sentence speech data, each time sub-speech data of a preset duration is successfully converted, the sub-speech data is transmitted to the client, so that the client plays the corresponding answer speech based on the sub-speech data.

[0011] In a second aspect, the present application provides a voice data processing method, which is applied to a client, comprising:

[0012] Whenever a sub-voice data is received from the server, a frame interpolation operation is performed on the sub-voice data to obtain data to be decoded, wherein the sub-voice data is a segment of voice data generated for a portion of the answer text;

[0013] The data to be decoded is decoded to obtain voice data to be played, and the voice data to be played is played.

[0014] In a possible implementation manner, performing a frame supplement operation on the sub-voice data to obtain the data to be decoded includes:

[0015] Determining the number of data frames included in the sub-voice data;

[0016] When the number is less than a preset threshold, splicing the next sub-voice data sent by the server to the sub-voice data to obtain spliced ​​sub-voice data;

[0017] Determine the number of data frames included in the spliced ​​sub-voice data;

[0018] When the number is greater than or equal to the preset threshold, determining the spliced ​​sub-speech data as the data to be decoded;

[0019] When the number is less than the preset threshold, the step of splicing the next sub-voice data sent by the server to the sub-voice data is re-executed until the number of data frames included in the spliced ​​sub-voice data is greater than or equal to the preset threshold.

[0020] In a possible implementation manner, performing a frame supplement operation on the sub-voice data to obtain the data to be decoded includes:

[0021] Obtaining previous sub-speech data corresponding to the sub-speech data;

[0022] Determine a preset number of data frames at the end of the previous sub-voice data;

[0023] A preset number of the data frames are concatenated before the sub-voice data.

[0024] In a possible implementation manner, the decoding the data to be decoded to obtain the voice data to be played includes:

[0025] Decoding the data to be decoded by a decoding thread to obtain decoded data;

[0026] Selecting a preset number of frames to be discarded starting from the first data frame of the decoded data;

[0027] A preset number of the to-be-discarded frames are discarded to obtain the to-be-played voice data.

[0028] In a possible implementation manner, before performing a frame filling operation on the sub-voice data to obtain the data to be decoded, the method further includes:

[0029] Determining whether the sub-speech data is the first segment of speech data generated for the answer text;

[0030] In the case where the sub-voice data is the first segment of voice data generated for the answer text, directly decoding the sub-voice data to obtain the voice data to be played;

[0031] In the case where the sub-voice data is not the first segment of voice data generated for the answer text, the step of performing a frame-filling operation on the sub-voice data to obtain data to be decoded is performed.

[0032] In a possible implementation manner, before performing a frame filling operation on the sub-voice data to obtain the data to be decoded, the method further includes:

[0033] In a case where the sub-voice data is not the first segment of voice data generated for the answer text, searching for file header data in the sub-voice data;

[0034] Deleting the file header data from the sub-voice data;

[0035] A frame-adding operation is performed on the sub-voice data from which the file header data is deleted.

[0036] In a third aspect, the present application provides a voice data processing device, applied to a server, comprising:

[0037] A data acquisition module, used to acquire question text data;

[0038] A text generation module, used for inputting the question text data into a large language model, so as to generate a reply text for replying to the question text data by using the large language model;

[0039] A speech conversion module is used for, in the process of the large language model outputting an answer text for the question text data, calling a speech generation service to convert the sentence text into sentence speech data each time a sentence text in the answer text is received;

[0040] The data transmission module is used to transmit the sub-voice data of a preset duration to the client when the voice generation service converts the sentence text into sentence voice data, so that the client can play the corresponding answer voice based on the sub-voice data.

[0041] In a fourth aspect, the present application provides a voice data processing device, applied to a client, comprising:

[0042] A data frame filling module, used for performing a frame filling operation on a sub-voice data sent by a server to obtain data to be decoded each time the sub-voice data is received, wherein the sub-voice data is a segment of voice data generated for a part of the answer text;

[0043] The data decoding module is used to decode the data to be decoded to obtain the voice data to be played, and play the voice data to be played.

[0044] In a possible implementation manner, the data frame filling module is specifically used to:

[0045] Determining the number of data frames included in the sub-voice data;

[0046] When the number is less than a preset threshold, splicing the next sub-voice data sent by the server to the sub-voice data to obtain spliced ​​sub-voice data;

[0047] Determine the number of data frames included in the spliced ​​sub-voice data;

[0048] When the number is greater than or equal to the preset threshold, determining the spliced ​​sub-speech data as the data to be decoded;

[0049] When the number is less than the preset threshold, the step of splicing the next sub-voice data sent by the server to the sub-voice data is re-executed until the number of data frames included in the spliced ​​sub-voice data is greater than or equal to the preset threshold.

[0050] In a possible implementation manner, the data frame filling module is further used to:

[0051] Obtaining previous sub-speech data corresponding to the sub-speech data;

[0052] Determine a preset number of data frames at the end of the previous sub-voice data;

[0053] A preset number of the data frames are concatenated before the sub-voice data.

[0054] In a possible implementation manner, the data decoding module is specifically used to:

[0055] Decoding the data to be decoded by a decoding thread to obtain decoded data;

[0056] Selecting a preset number of frames to be discarded starting from the first data frame of the decoded data;

[0057] A preset number of the to-be-discarded frames are discarded to obtain the to-be-played voice data.

[0058] In a possible implementation manner, the device further includes an execution module, configured to:

[0059] Determining whether the sub-speech data is the first segment of speech data generated for the answer text;

[0060] In the case where the sub-voice data is the first segment of voice data generated for the answer text, directly decoding the sub-voice data to obtain the voice data to be played;

[0061] In the case where the sub-voice data is not the first segment of voice data generated for the answer text, the step of performing a frame-filling operation on the sub-voice data to obtain data to be decoded is performed.

[0062] In a possible implementation manner, the method further includes a deletion module, which is used to:

[0063] In a case where the sub-voice data is not the first segment of voice data generated for the answer text, searching for file header data in the sub-voice data;

[0064] Deleting the file header data from the sub-voice data;

[0065] A frame-adding operation is performed on the sub-voice data from which the file header data is deleted.

[0066] In a third aspect, an electronic device is provided, comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;

[0067] Memory, used to store computer programs;

[0068] The processor is used to implement any method step described in the first aspect and the second aspect when executing the program stored in the memory.

[0069] In a fourth aspect, a computer-readable storage medium is provided, characterized in that a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, the method steps described in any one of the first and second aspects are implemented.

[0070] In a fifth aspect, a computer program product comprising instructions is provided, which, when executed on a computer, enables the computer to execute any of the above-mentioned speech data processing methods.

[0071] Beneficial effects of the embodiments of the present application:

[0072] The embodiment of the present application provides a method, device and electronic device for processing voice data. The present application first obtains question text data, then inputs the question text data into a large language model, so as to use the large language model to generate a reply text for the question text data. In the process of the large language model outputting the reply text for the question text data, each time a sentence text in the reply text is received, a voice generation service is called to convert the sentence text into sentence voice data, and in the process of the voice generation service converting the sentence text into sentence voice data, each time a sub-voice data of a preset time length is successfully converted, the sub-voice data is transmitted to a client, so that the client plays the corresponding reply voice based on the sub-voice data. In this way, the waiting time before converting text to voice and before transmitting the reply voice can be reduced, thereby improving processing efficiency.

[0073] Of course, implementing any product or method of the present application does not necessarily require achieving all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0074] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0075] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0076] One or more embodiments are exemplarily described by pictures in the corresponding drawings, and these exemplified descriptions do not constitute limitations on the embodiments. Elements with the same reference numerals in the drawings represent similar elements, and unless otherwise stated, the figures in the drawings do not constitute proportional limitations.

[0077] Figure 1A flowchart of a method for processing voice data provided in an embodiment of the present application;

[0078] Figure 2 A flowchart of another voice data processing method provided in an embodiment of the present application;

[0079] Figure 3 An example of generating mp3 voice data of three corresponding sentence texts for a complete answer text provided in an embodiment of the present application;

[0080] Figure 4 An overall process of processing voice data by a client provided in an embodiment of the present application;

[0081] Figure 5 A schematic diagram of the structure of a voice data processing device provided in an embodiment of the present application;

[0082] Figure 6 A schematic diagram of the structure of another voice data processing device provided in an embodiment of the present application;

[0083] Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0084] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0085] The disclosure below provides many different embodiments or examples to implement different structures of the present invention. In order to simplify the disclosure of the present invention, the parts and settings of specific examples are described below. Of course, they are only examples, and the purpose is not to limit the present invention. In addition, the present invention can repeat reference numbers and / or letters in different examples. This repetition is for the purpose of simplification and clarity, and does not itself indicate the relationship between the various embodiments and / or settings discussed.

[0086] Figure 1A flow chart of a voice data processing method provided for an embodiment of the present application. The method can be applied to one or more electronic devices such as smart phones, laptops, desktop computers, portable computers, servers, etc. In addition, the execution subject of the method can be hardware or software. When the above-mentioned execution subject is hardware, the execution subject can be one or more of the above-mentioned electronic devices. For example, a single electronic device can execute the method, or multiple electronic devices can cooperate with each other to execute the method. When the above-mentioned execution subject is software, the method can be implemented as multiple software or software modules, or as a single software or software module. No specific limitation is made here.

[0087] like Figure 1 As shown, the method specifically includes:

[0088] Step 101: Obtain question text data.

[0089] A voice data processing method provided in an embodiment of the present application is applied to an intelligent question and answer scenario, and the execution subject is a server. In a specific example, the intelligent question and answer scenario can be an AI (Artificial Intelligence) virtual person making a phone call, also known as real-time voice chat.

[0090] Question text data refers to question text that requires AI virtual people to give voice responses.

[0091] In one embodiment, the client records the question voice issued by the user and sends it to the server in real time. The server recognizes the question voice through the ASR (Automatic Speech Recognition) service to obtain the complete question text (ie, question text data).

[0092] In another embodiment, the user may directly input the question text data through the client, and the client may send the question text data to the server in real time.

[0093] Step 102: input the question text data into a large language model, so as to utilize the large language model to generate a reply text for replying to the question text data.

[0094] LLM (Large Language Model) refers to a deep learning model trained with a large amount of text data, which can generate natural language text or understand the meaning of language text. In practical applications, LLM can handle a variety of natural language tasks, such as text classification, question answering, and dialogue.

[0095] In the embodiment of the present application, after obtaining the question text data, the server can call the large language model, and generate a reply text to reply to the question text data by inputting the question text data into the large language model.

[0096] Step 103: During the process of the large language model outputting the answer text for the question text data, each time a sentence text in the answer text is received, the speech generation service is called to convert the sentence text into sentence speech data.

[0097] The speech generation service refers to a service that converts text into speech. In a specific example, the speech generation service may be a TTS service (Text-to-Speech Service), also known as speech synthesis or text reading.

[0098] In the embodiment of the present application, during the process of the large language model outputting the answer text word by word, the server can recognize the output text one by one. If a specific punctuation mark (e.g., a comma or a period) is recognized, a sentence text can be determined according to the specific punctuation mark, and each time a sentence text is determined, the operation of converting text to speech is performed, that is, the speech generation service is called to convert the sentence text into sentence speech data. Thus, the time for waiting for the large language model to generate text can be reduced, and the processing efficiency can be improved.

[0099] It should be noted that when a specific punctuation mark is recognized for the first time, all the text received before the specific punctuation mark is determined as a sentence text. When the specific punctuation mark is not recognized for the first time, all the text between the specific punctuation mark and the previous specific punctuation mark is determined as a sentence text.

[0100] Step 104: During the process of the speech generation service converting the sentence text into sentence speech data, each time sub-speech data of a preset duration is successfully converted, the sub-speech data is transmitted to the client, so that the client plays the corresponding answer speech based on the sub-speech data.

[0101] In an embodiment of the present application, in order to further improve processing efficiency, during the process of the speech generation service converting sentence text into sentence voice data, the server can monitor the conversion progress of the speech generation service, and when the speech generation service successfully converts sub-voice data of a preset duration (e.g., two seconds), the server can transmit the sub-voice data to the client for playback by the client.

[0102] In this way, the client can play the sub-voice data that has been received currently while receiving new sub-voice data, and when the current sub-voice data is played, it will then play the newly received sub-voice data, and so on, until all the sub-voice data corresponding to the answer text are played. Thus, the waiting time is further reduced and the processing efficiency is improved.

[0103] In addition, in another embodiment of the present application, before transmitting the sub-voice data to the client, the following steps may be included: first, the sub-voice data is encoded by mp3 (Moving Picture Experts Group Audio Layer III), then the encoded data is converted into binary text, then the binary text is converted into hex (hexadecimal) text, and finally, the sub-voice data of the hex text is transmitted to the client. This can speed up data transmission and reduce delays.

[0104] In the application, the sampling rate can be reduced as much as possible during encoding, the bit rate can be reduced as much as possible during transmission, and a single channel can be used to further increase the transmission speed. The specific sampling rate and bit rate can be set by the user according to actual needs to ensure the sound quality level required by the product, thereby achieving the goal of increasing the transmission speed while ensuring the sound quality.

[0105] In the solution provided by the embodiment of the present application, first, the question text data is obtained, and then the question text data is input into the large language model, so as to use the large language model to generate a reply text to the question text data, and in the process of the large language model outputting the reply text for the question text data, each time a sentence text in the reply text is received, the speech generation service is called to convert the sentence text into sentence voice data, and in the process of the speech generation service converting the sentence text into sentence voice data, each time a sub-voice data of a preset duration is successfully converted, the sub-voice data is transmitted to the client, so that the client plays the corresponding reply voice based on the sub-voice data. In this way, the waiting time before converting text to speech and before transmitting the reply voice can be reduced, thereby improving processing efficiency.

[0106] See also Figure 2 , is a flow chart of another embodiment of a voice data processing method provided in an embodiment of the present application, and the method is applied to a client. Figure 2 As shown, the process may include the following steps:

[0107] Step 201: when receiving a sub-voice data sent by a server, perform a frame interpolation operation on the sub-voice data to obtain data to be decoded, wherein the sub-voice data is a segment of voice data generated for a part of the answer text;

[0108] Step 202: Decode the data to be decoded to obtain voice data to be played, and play the voice data to be played.

[0109] For ease of understanding, step 201 and step 202 are described in a unified manner as follows:

[0110] In the application, the sub-voice data sent by the server is partial voice data converted for a sentence text. For example, a complete sentence text can be converted into 6 seconds of voice data. The server can transmit the 2 seconds of voice data as a sub-voice data to the client every time it converts 2 seconds of voice data.

[0111] In a specific example, Figure 3 As shown, for a complete answer text, 3 corresponding sentence texts of mp3 voice data are generated, that is, Figure 3 The mp3 voice data at index 0 is the complete mp3 voice data corresponding to a sentence text, the mp3 voice data at index 1 is the complete mp3 voice data corresponding to a sentence text, and the mp3 voice data at index 2 is the complete mp3 voice data corresponding to a sentence text.

[0112] For each complete mp3 voice data corresponding to a sentence text, it will be split into multiple sub-mp3 voice data (i.e., sub-voice data). For example, the sentence text "Okay" has 2 sub-mp3 voice data corresponding to index = 0, and the sentence text "Then I will tell you how to solve the Chang'an Black Tea case" has 7 sub-mp3 voice data corresponding to index = 1.

[0113] Since the server transmits short-duration sub-voice data to the client, it is difficult to ensure the decoding effect if the client directly decodes and plays, resulting in poor playback effect. Therefore, in the embodiment of the present application, each time a sub-voice data sent by the server is received, the sub-voice data is supplemented with a frame to obtain the data to be decoded, thereby ensuring the decoding effect of the data to be decoded during the decoding process, and then the data to be decoded is decoded to obtain the voice data to be played, and the voice data to be played is played. In this way, the playback effect of the voice data to be played can be improved, and the user experience can be improved.

[0114] In an application, if the sub-voice data contains too few data frames, the subsequent decoder may fail to decode. To this end, in one embodiment, the specific implementation of performing a frame supplement operation on the sub-voice data to obtain the data to be decoded may include the following steps:

[0115] Step A1, determining the number of data frames contained in the sub-voice data;

[0116] Step A2: when the number is less than a preset threshold, concatenate the next sub-voice data sent by the server to the sub-voice data to obtain concatenated sub-voice data;

[0117] Step A3, determining the number of data frames contained in the spliced ​​sub-voice data;

[0118] Step A4: when the number is greater than or equal to the preset threshold, determining the spliced ​​sub-speech data as the data to be decoded;

[0119] Step A5: When the number is less than the preset threshold, re-execute the step of splicing the next sub-voice data sent by the server to the sub-voice data until the number of data frames contained in the spliced ​​sub-voice data is greater than or equal to the preset threshold.

[0120] For ease of understanding, steps A1 to A5 are described in a unified manner as follows:

[0121] In this implementation, first, the number of data frames included in the sub-voice data is determined according to the frame header, such as Figure 3 As shown, usually the second streaming mp3 begins with a frame header of a data frame, which is 8 bits in total, such as fff348c4. In the application, a frame header and a complete string after the frame header correspond to a valid data frame. Thus, the number of data frames contained in the sub-voice data is determined.

[0122] If the number of data frames is less than a preset threshold (such as 10), first cache the sub-voice data, wait for the next sub-voice data to be transmitted, and then splice the two sub-voice data together. If the total number of data frames contained after splicing is still less than the preset threshold, wait for the next sub-voice data to be transmitted, and then splice them together until the total number of data frames contained in the spliced ​​data is greater than or equal to the preset threshold. End the splicing operation and perform subsequent decoding operations.

[0123] Through the above scheme, when the number of data frames contained in the sub-voice data is too small, the sub-voice data can be spliced ​​with the subsequent sub-voice data to obtain the data to be decoded for subsequent decoding. In this way, the success rate of subsequent decoding can be guaranteed and decoding failure can be avoided.

[0124] In addition, in the application, for sub-voice data containing only data frames, the decoding of the first few frames may fail. Therefore, in one embodiment, the specific implementation of performing frame-filling operation on the sub-voice data to obtain the data to be decoded may also include the following steps:

[0125] Step B1, obtaining the previous sub-speech data corresponding to the sub-speech data;

[0126] Step B2, determining a preset number of data frames at the end of the previous sub-voice data;

[0127] Step B3: splicing a preset number of the data frames before the sub-voice data.

[0128] For ease of understanding, steps B1 to B3 are described in a unified manner as follows:

[0129] In this implementation, for each current sub-voice data, the last few frames of data of the previous sub-voice data (for example, the last 4 frames of data) can be spliced ​​to the front of the current sub-voice data, thereby avoiding the problem of missing content in the decoded data due to possible failure of the first few frames during decoding.

[0130] Correspondingly, the specific implementation of decoding the data to be decoded to obtain the voice data to be played may include the following steps: decoding the data to be decoded through a decoding thread to obtain decoded data, selecting a preset number of frames to be discarded starting from the first data frame of the decoded data, discarding the preset number of frames to be discarded, and obtaining the voice data to be played.

[0131] In this implementation, in the case where a preset number of data frames are spliced ​​before the sub-voice data, a corresponding number of data frames are discarded after decoding, thereby avoiding subsequent repeated playback of the same content.

[0132] In addition, in another embodiment of the present application, before the sub-voice data is subjected to the frame-adding operation to obtain the data to be decoded, the following steps may also be included: determining whether the sub-voice data is the first segment of voice data generated for the answer text; if the sub-voice data is the first segment of voice data generated for the answer text, directly decoding the sub-voice data to obtain the voice data to be played; if the sub-voice data is not the first segment of voice data generated for the answer text, executing the step of performing the frame-adding operation on the sub-voice data to obtain the data to be decoded.

[0133] In the application, when the sub-voice data is the first voice data generated for the answer text (hereinafter referred to as the first voice data for ease of understanding), for example, Figure 3The sub-voice data with index 0 and starting with 49443304 generally has a complete file header and a sufficient number of data frames, so even if it is directly decoded, it will not affect the decoding effect, so no frame filling processing is performed. When the sub-voice data is not the first voice data, the number of data frames cannot be guaranteed, and a frame filling operation is required. At this time, the step of performing a frame filling operation on the sub-voice data to obtain the data to be decoded is performed. In this way, the sub-voice data with a sufficient number of data frames can be exempted from frame filling processing, thereby saving computing resources.

[0134] In addition, in another embodiment of the present application, before the sub-voice data is subjected to a frame-adding operation to obtain the data to be decoded, the following steps may also be included: when the sub-voice data is not the first segment of voice data generated for the answer text, searching for file header data in the sub-voice data, deleting the file header data from the sub-voice data, and performing a frame-adding operation on the sub-voice data from which the file header data has been deleted.

[0135] In the application, since the first voice data already contains a file header, the file headers in the subsequent sub-voice data are repeated and redundant, and repeatedly decoding them will waste resources.

[0136] Therefore, in this embodiment, it is possible to first check whether the non-first voice data contains file header data, for example, Figure 3 The sub-voice data with index 1 or 2 and starting with 49443304, where 49443304 means it has file header information and data frames, and fff348c4 means it only has data frames. For the sub-voice data containing file header data, the file header data is removed and only the data frame is retained. In this way, the file header data does not need to be decoded repeatedly during subsequent decoding, thereby saving decoding resources.

[0137] Optionally, the embodiment of the present application also provides an overall process of the client processing voice data, such as Figure 4 As shown, the specific steps are as follows.

[0138] (1) The client receives the streaming sub-mp3 (i.e., the sub-voice data, which is still a hexadecimal text at this time and needs to be saved as a file before decoding). It first determines whether the sub-mp3 is the first received data. If it is, it can be directly converted into a binary file and saved locally, and sent to the queue to wait for decoding. If not, the frame header of the data frame can be found to count how many valid data frames the sub-mp3 has.

[0139] (2) Determine whether the sub-mp3 has a file header. If so, remove the file header and keep only the data frame to facilitate subsequent unified processing.

[0140] (3) Determine how many data frames the sub-mp3 has based on the frame header. If it does not exceed a threshold (taking 10 as an example), it can be cached first and wait for the next sub-mp3 to be transmitted, and then spliced ​​together until the number of spliced ​​data frames is greater than or equal to 10.

[0141] (4) According to the MP3 frame header format, find the last 4 frames of the previous sub-MP3 and merge them with the current sub-MP3.

[0142] (5) Convert it into a binary file, save it locally, and add it to the queue.

[0143] (6) The decoding thread will extract files from the queue in order and decode the PCM data. Each frame of PCM data will be written into the PCMqueue.

[0144] (7) The playback thread reads data from the PCM queue and plays it.

[0145] (8) When the playback ends or the user interrupts, the next round of Q / A (question and answer) begins, that is, the user asks questions and waits for the big model to answer.

[0146] In the embodiment of the present application, each time a sub-voice data is received from the server, the sub-voice data is frame-filled to obtain the data to be decoded, thereby ensuring the decoding effect of the data to be decoded in the decoding process, and then the data to be decoded is decoded to obtain the voice data to be played, and the voice data to be played is played. Thus, the playback effect of the voice data to be played can be improved, and the user experience can be improved.

[0147] Based on the same technical concept, the embodiment of the present application also provides a voice data processing device, such as Figure 5 As shown, the device comprises:

[0148] A data acquisition module 51 is used to acquire question text data;

[0149] A text generation module 52, used for inputting the question text data into a large language model, so as to generate a reply text for replying to the question text data by using the large language model;

[0150] The speech conversion module 53 is used for, in the process of the large language model outputting the answer text for the question text data, calling the speech generation service to convert the sentence text into sentence speech data each time a sentence text in the answer text is received;

[0151] The data transmission module 54 is used to transmit the sub-voice data of a preset duration to the client when the voice generation service converts the sentence text into sentence voice data, so that the client can play the corresponding answer voice based on the sub-voice data.

[0152] Based on the same technical concept, the present application embodiment also provides another voice data processing device, such as Figure 6 As shown, the device comprises:

[0153] A data frame filling module 61 is used for performing a frame filling operation on a sub-voice data sent by the server to obtain data to be decoded, wherein the sub-voice data is a segment of voice data generated for a part of the answer text;

[0154] The data decoding module 62 is used to decode the data to be decoded to obtain the voice data to be played, and play the voice data to be played.

[0155] In a possible implementation manner, the data frame filling module is specifically used to:

[0156] Determining the number of data frames included in the sub-voice data;

[0157] When the number is less than a preset threshold, splicing the next sub-voice data sent by the server to the sub-voice data to obtain spliced ​​sub-voice data;

[0158] Determine the number of data frames included in the spliced ​​sub-voice data;

[0159] When the number is greater than or equal to the preset threshold, determining the spliced ​​sub-speech data as the data to be decoded;

[0160] When the number is less than the preset threshold, the step of splicing the next sub-voice data sent by the server to the sub-voice data is re-executed until the number of data frames included in the spliced ​​sub-voice data is greater than or equal to the preset threshold.

[0161] In a possible implementation manner, the data frame filling module is further used to:

[0162] Obtaining previous sub-speech data corresponding to the sub-speech data;

[0163] Determine a preset number of data frames at the end of the previous sub-voice data;

[0164] A preset number of the data frames are concatenated before the sub-voice data.

[0165] In a possible implementation manner, the data decoding module is specifically used to:

[0166] Decoding the data to be decoded by a decoding thread to obtain decoded data;

[0167] Selecting a preset number of frames to be discarded starting from the first data frame of the decoded data;

[0168] A preset number of the to-be-discarded frames are discarded to obtain the to-be-played voice data.

[0169] In a possible implementation manner, the device further includes an execution module, configured to:

[0170] Determining whether the sub-speech data is the first segment of speech data generated for the answer text;

[0171] In the case where the sub-voice data is the first segment of voice data generated for the answer text, directly decoding the sub-voice data to obtain the voice data to be played;

[0172] In the case where the sub-voice data is not the first segment of voice data generated for the answer text, the step of performing a frame-filling operation on the sub-voice data to obtain data to be decoded is performed.

[0173] In a possible implementation manner, the method further includes a deletion module, which is used to:

[0174] In a case where the sub-voice data is not the first segment of voice data generated for the answer text, searching for file header data in the sub-voice data;

[0175] Deleting the file header data from the sub-voice data;

[0176] A frame-adding operation is performed on the sub-voice data from which the file header data is deleted.

[0177] Based on the same technical concept, the embodiment of the present application also provides an electronic device, such as Figure 7 As shown, it includes a processor 111, a communication interface 112, a memory 113 and a communication bus 114, wherein the processor 111, the communication interface 112, and the memory 113 communicate with each other through the communication bus 114.

[0178] Memory 113, used for storing computer programs;

[0179] The processor 111 is used to execute the program stored in the memory 113 to implement the following steps:

[0180] Get the question text data;

[0181] Inputting the question text data into a large language model to generate a reply text for replying to the question text data using the large language model;

[0182] In the process of the large language model outputting the answer text for the question text data, each time a sentence text in the answer text is received, the speech generation service is called to convert the sentence text into sentence speech data;

[0183] In the process of converting the sentence text into sentence voice data by the voice generation service, each time sub-voice data of a preset duration is successfully converted, the sub-voice data is transmitted to the client, so that the client plays the corresponding answer voice based on the sub-voice data;

[0184] And, each time a sub-voice data sent by the server is received, a frame interpolation operation is performed on the sub-voice data to obtain data to be decoded, wherein the sub-voice data is a segment of voice data generated for a portion of the answer text;

[0185] The data to be decoded is decoded to obtain voice data to be played, and the voice data to be played is played.

[0186] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0187] The communication interface is used for communication between the above electronic device and other devices.

[0188] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0189] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0190] In another embodiment provided in the present application, a computer-readable storage medium is provided, in which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned voice data processing methods are implemented.

[0191] In another embodiment provided by the present application, a computer program product including instructions is also provided, which, when executed on a computer, enables the computer to execute any of the speech data processing methods in the above embodiments.

[0192] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0193] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a general hardware platform, and of course, by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0194] It should be understood that the terms used herein are only for the purpose of describing specific example embodiments and are not intended to be limiting. Unless the context clearly indicates otherwise, the singular forms "one", "an" and "said" as used herein may also be meant to include plural forms. The terms "include", "comprise", "contain", and "have" are inclusive, and therefore specify the existence of stated features, steps, operations, elements and / or parts, but do not exclude the existence or addition of one or more other features, steps, operations, elements, parts, and / or combinations thereof. The method steps, processes, and operations described herein are not interpreted as necessarily requiring them to be performed in the specific order described or illustrated, unless the execution order is clearly indicated. It should also be understood that additional or alternative steps may be used.

[0195] The foregoing is merely a specific embodiment of the present invention, which enables those skilled in the art to understand or implement the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A method for processing speech data, characterized in that: Applied to a server, the method comprises: Get the question text data; Inputting the question text data into a large language model to generate a reply text for replying to the question text data using the large language model; In the process of the large language model outputting the answer text for the question text data, each time a sentence text in the answer text is received, the speech generation service is called to convert the sentence text into sentence speech data; In the process of the speech generation service converting the sentence text into sentence speech data, each time sub-speech data of a preset duration is successfully converted, the sub-speech data is transmitted to the client, so that the client plays the corresponding answer speech based on the sub-speech data.

2. A method for processing speech data, characterized in that: Applied to a client, the method comprises: Whenever a sub-voice data is received from the server, a frame interpolation operation is performed on the sub-voice data to obtain data to be decoded, wherein the sub-voice data is a segment of voice data generated for a portion of the answer text; The data to be decoded is decoded to obtain voice data to be played, and the voice data to be played is played.

3. The method according to claim 2, characterized in that The performing a frame supplement operation on the sub-voice data to obtain data to be decoded includes: Determining the number of data frames included in the sub-voice data; When the number is less than a preset threshold, splicing the next sub-voice data sent by the server to the sub-voice data to obtain spliced ​​sub-voice data; Determine the number of data frames included in the spliced ​​sub-voice data; When the number is greater than or equal to the preset threshold, determining the spliced ​​sub-speech data as the data to be decoded; When the number is less than the preset threshold, the step of splicing the next sub-voice data sent by the server to the sub-voice data is re-executed until the number of data frames included in the spliced ​​sub-voice data is greater than or equal to the preset threshold.

4. The method according to claim 2, characterized in that: The performing a frame supplement operation on the sub-voice data to obtain data to be decoded includes: Obtaining previous sub-speech data corresponding to the sub-speech data; Determine a preset number of data frames at the end of the previous sub-voice data; A preset number of the data frames are concatenated before the sub-voice data.

5. The method according to claim 4, characterized in that The decoding process of the data to be decoded to obtain the voice data to be played includes: Decoding the data to be decoded by a decoding thread to obtain decoded data; Selecting a preset number of frames to be discarded starting from the first data frame of the decoded data; A preset number of the to-be-discarded frames are discarded to obtain the to-be-played voice data.

6. The method according to claim 2, characterized in that Before performing a frame supplement operation on the sub-voice data to obtain the data to be decoded, the method further includes: Determining whether the sub-speech data is the first segment of speech data generated for the answer text; In the case where the sub-voice data is the first segment of voice data generated for the answer text, directly decoding the sub-voice data to obtain the voice data to be played; In the case where the sub-voice data is not the first segment of voice data generated for the answer text, the step of performing a frame-filling operation on the sub-voice data to obtain data to be decoded is performed.

7. The method according to claim 6, characterized in that Before performing a frame supplement operation on the sub-voice data to obtain the data to be decoded, the method further includes: In a case where the sub-voice data is not the first segment of voice data generated for the answer text, searching for file header data in the sub-voice data; Deleting the file header data from the sub-voice data; A frame-adding operation is performed on the sub-voice data from which the file header data is deleted.

8. A voice data processing device, characterized in that: Applied to a server, the device comprises: A data acquisition module, used to acquire question text data; A text generation module, used for inputting the question text data into a large language model, so as to generate a reply text for replying to the question text data by using the large language model; A speech conversion module is used for, in the process of the large language model outputting an answer text for the question text data, calling a speech generation service to convert the sentence text into sentence speech data each time a sentence text in the answer text is received; The data transmission module is used to transmit the sub-voice data of a preset duration to the client when the voice generation service converts the sentence text into sentence voice data, so that the client can play the corresponding answer voice based on the sub-voice data.

9. A voice data processing device, characterized in that: Applied to a client, the device comprises: A data frame filling module, used for performing a frame filling operation on a sub-voice data sent by the server to obtain data to be decoded each time the sub-voice data is received, wherein the sub-voice data is a segment of voice data generated for a part of the answer text; The data decoding module is used to decode the data to be decoded to obtain the voice data to be played, and play the voice data to be played.

10. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; Memory, used to store computer programs; The processor is used to implement the voice data processing method described in any one of claims 1 to 7 when executing the program stored in the memory.

Images