Voice processing method and device and electronic equipment

Abstract

The invention discloses a voice processing method and device and electronic device.The voice processing method comprises the steps that in response to second input voice input by a user in the period of outputting first generated content, the output second generated content in the first generated content is determined; the first generation content represents a model reasoning result which is generated by utilizing a target model and aims at the first input voice; generating first prompt information according to the second input voice and the second generation content; and generating third generation content corresponding to the second input voice by using the target model according to the first prompt information.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to, but is not limited to, the technical field of computer, and particularly relates to a voice processing method and device and electronic equipment. BACKGROUND

[0002] A voice assistant or voice agent is an intelligent system based on artificial intelligence technology, which communicates with users through voice interaction, provides services or performs user tasks for users. The voice agent is widely used in smart home, information query and schedule management, entertainment and content consumption, navigation, shopping, medical health and other fields, and plays a great role in improving user work and life efficiency and convenience. SUMMARY

[0003] Therefore, the present disclosure provides at least a voice processing method, device and electronic equipment.

[0004] The technical solution of the present disclosure is implemented as follows:

[0005] In one aspect, the present disclosure provides a voice processing method, comprising:

[0006] In response to a second input voice obtained during output of the first generated content, determining second generated content that has been output in the first generated content; the first generated content represents a model inference result for the first input voice generated by using a target model;

[0007] Generating first prompt information according to the second input voice and the second generated content;

[0008] According to the first prompt information, generating third generated content corresponding to the second input voice by using the target model.

[0009] In some embodiments, in response to a second input voice obtained during output of the first generated content, determining second generated content that has been output in the first generated content, comprises:

[0010] In response to the second input voice obtained during output of the first buffered voice, determining an output voice in the first buffered voice; the first buffered voice represents a voice corresponding to the first generated content; the output voice represents a part of the voice buffered in advance in the first buffered voice;

[0011] According to the output voice, determining second generated content that has been output in the first generated content.

[0012] In some embodiments, the first buffered voice has a plurality of voice identifiers; the first generated content has a plurality of content identifiers; the plurality of voice identifiers and the plurality of content identifiers have a first correspondence relationship;

[0013] According to the output voice, determine the second generated content that has been output in the first generated content, comprising:

[0014] According to the at least one voice identifier corresponding to the output voice and the first correspondence relationship, determine the second generated content that has been output in the first generated content.

[0015] In some embodiments, according to the second input voice and the second generated content, generate the first prompt information, comprising:

[0016] Determine the first text corresponding to the second input voice and the second text corresponding to the second generated content respectively; wherein the second generated content represents the word token sequence or the key-value pair generated by the target model;

[0017] According to the first text and the second text, generate the first prompt information.

[0018] In some embodiments, according to the first text and the second text, generate the first prompt information, comprising:

[0019] Update the second prompt information with the first text and the second text to obtain the first prompt information; wherein the second prompt information represents the prompt information used to generate the first generated content.

[0020] In some embodiments, the method further comprises:

[0021] Obtain the current content generation speed of the target model and the content output speed of the first generated content;

[0022] According to the content output speed, update the content generation speed of the target model; wherein the updated content generation speed is lower than the content generation speed before updating.

[0023] In some embodiments, the content generation speed represents the word token generation speed or the key-value pair generation speed; and the content output speed represents the voice output speed.

[0024] In some embodiments, according to the content output speed, update the content generation speed of the target model, comprising:

[0025] According to the content generation speed and the content output speed, determine the adjustment factor;

[0026] Take the adjustment factor as the interval duration between the adjacent two word tokens or key-value pairs generated by the target model to update the content generation speed of the target model.

[0027] On the other hand, the present disclosure also provides a voice processing device, comprising:

[0028] The output content determination module determines second generated content that has been output in the first generated content in response to second input speech obtained during output of the first generated content; the first generated content represents a model processing result generated by using a target model for the first input speech;

[0029] The prompt information generation module generates first prompt information according to the second input speech and the second generated content;

[0030] The content generation module generates third generated content corresponding to the second input speech by using the target model according to the first prompt information.

[0031] In another aspect, the present disclosure further provides an electronic device, comprising at least one processor; the at least one processor is configured to run a voice agent; wherein,

[0032] The voice agent is configured to determine second generated content that has been output in the first generated content in response to second input speech obtained during output of the first generated content; the first generated content represents a model processing result generated by using a target model for the first input speech; generate first prompt information according to the second input speech and the second generated content; and generate third generated content corresponding to the second input speech by using the target model according to the first prompt information.

[0033] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, but not limiting the technical solutions of the present disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0034] The accompanying drawings incorporated in and forming a part of the specification, illustrate embodiments consistent with the present disclosure and serve to explain the technical solutions of the present disclosure together with the specification.

[0035] Figure 1 It is a voice interaction flowchart of a voice agent;

[0036] Figure 2 It is a voice interaction flowchart of a voice agent;

[0037] Figure 3 It is an implementation flowchart of a voice processing method provided by the present disclosure;

[0038] Figure 4 It is a key-value pair storage diagram in an embodiment of the present disclosure;

[0039] Figure 5 It is a voice processing flowchart in an embodiment of the present disclosure;

[0040] Figure 6A schematic diagram of a component structure of a voice processing device provided by the present disclosure is shown in FIG. 1.

[0041] Figure 7 A schematic diagram of a hardware entity of an electronic device provided by the present disclosure is shown in FIG. 2. DETAILED DESCRIPTION

[0042] In order to make the purposes, technical solutions and advantages of the present disclosure clearer, the technical solutions of the present disclosure are further described in detail below in combination with the accompanying drawings and embodiments, and the described embodiments should not be regarded as limitations of the present disclosure. All other embodiments obtained by those of ordinary skill in the art without making creative labor fall within the scope of protection of the present disclosure.

[0043] In the following description, "some embodiments" are related to a subset of all possible embodiments, but it can be understood that "some embodiments" can be the same subset or different subsets of all possible embodiments, and can be combined with each other without conflict.

[0044] The terms "first / second / third" involved are only to distinguish similar objects, and do not represent a specific order of the objects. It can be understood that "first / second / third" can interchange the specific order or sequence as allowed, so that the embodiments of the present disclosure described herein can be implemented in an order other than that illustrated or described herein.

[0045] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present disclosure belongs. The terms used herein are only for the purpose of describing the present disclosure and are not intended to limit the present disclosure.

[0046] In related technologies, the implementation schemes of voice agents mainly have the following two kinds:

[0047] The first one is the traditional but more mainstream pipeline scheme. For example, Figure 1As shown, in the pipeline scheme, the voice agent mainly includes an automatic speech recognition (ASR) module 110, a natural language understanding (NLU) module 120, and a text to speech (TTS) module 130. In the voice interaction process, first, the user voice 101 is input into the ASR module 110, and then the ASR module 110 converts the user voice 101 into text data and sends the text data to the NLU module 120; then, the NLU module 120 parses the received text data, generates a text reply to the user voice 101, and sends the text reply to the TTS module 130; then, the TTS module 130 composes the received text reply into voice data and stores the voice data in the voice cache module 140; finally, the system player 150 plays the voice in the voice cache module 140. Here, the NLU module 120 is usually implemented as a large language model (LLM) to enhance the understanding ability of the voice agent.

[0048] The second is a speech2speech scheme. As shown, Figure 2 As shown, in the speech2speech scheme, the voice agent mainly includes a speech encoder 210, a multimodal large language model (MLLM) 220, and a speech decoder 230. In the voice interaction process, first, the user voice 201 is input into the speech encoder 210 to encode the user voice 201 into a speech embedding vector using the speech encoder 210, and the speech embedding is input into the multimodal large language model 220; then, the multimodal large language model 220 understands the received speech embedding and generates a token sequence and a speech embedding, and the speech embedding is input into the speech decoder 230; finally, the speech decoder 230 decodes the received speech embedding to obtain a reply voice, and caches the reply voice to the voice cache module 240; finally, the system player 250 plays the voice in the voice cache module 240.

[0049] The implementation schemes of the above two voice agents both have the problem of asynchronization between "thinking" and "speaking" of the agent, that is, when playing the voice cached in the voice cache module, since the speed of receiving the voice by the user is limited and usually slower than the speed of generating the text or speech embedding by the model, the speed of outputting the voice by the voice agent will be significantly lower than the speed of generating the text or speech embedding by the model, thereby causing the problem of asynchronization between "thinking" and "speaking" of the voice agent.

[0050] In this way, during the speech playing process, if the user inputs new speech through the microphone before the text or speech embedding content generated by the model is played, the speech agent will stop the current playing progress and clear the speech cache module. However, the model as the brain of the speech agent has already "thought" and does not know where the user interrupts the speech output, that is, does not know which reply content has been played when the user inputs new speech information. In addition, in the natural interaction process, "barge in" usually contains important information, such as the user has already been impatient, the user is not interested in some content, the pronoun said by the user occurs in which context fragment, and the like. Therefore, in the speech interaction, how to make the speech agent naturally process the user barge in becomes a problem to be solved urgently.

[0051] Based on this, the present disclosure provides a speech processing method, which can be executed by an electronic device. The electronic device can be a notebook computer, a tablet computer, a desktop computer, a set-top box, a mobile device (for example, a mobile phone, a portable music player, a personal digital assistant, a dedicated message device, a portable game device), and various types of terminals. The electronic device can also be implemented as a server. The server can be a standalone physical server, a server cluster composed of multiple physical servers, or a distributed system. The server can also be a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and basic cloud computing services such as big data and artificial intelligence platforms.

[0052] In the following, the technical solutions in the embodiments of the present disclosure will be described clearly and completely in combination with the drawings in the embodiments of the present disclosure.

[0053] Figure 3 An implementation flowchart of a speech processing method provided by the present disclosure is shown in FIG. 1. The method includes the following steps S31 to S33. Figure 3

[0054] In step S31, in response to obtaining a second input speech input by a user during output of first generated content, second generated content that has been output in the first generated content is determined; the first generated content represents a model inference result generated by a target model for a first input speech.

[0055] ​Here, the first generated content refers to a model inference result generated by a target model according to a first input voice input by a user, that is, the first generated content is a reply content corresponding to the first input voice. The target model can be any type of model applied in the voice agent, such as a large language model, a multi-modal large language model, etc.

[0056] In some embodiments, the first generated content can be all generated content corresponding to the first input voice, that is, complete reply content corresponding to the first input voice.

[0057] In some embodiments, the first generated content can be part of the generated content corresponding to the first input voice, that is, in the model generation process, the first generated content is part of the reply content generated by the target model for the first input voice.

[0058] In some embodiments, the first generated content can be any data type.

[0059] In some embodiments, in the case of implementing the voice agent in a pipeline scheme, the first generated content can be text content, token sequence, etc. generated by a large language model.

[0060] In some embodiments, in the case of implementing the voice agent in a speech2speech scheme, the first generated content can be text content, token sequence, key-value pair, etc. generated by a multi-modal large language model; the key-value pair refers to a key (K) and a value (V) calculated based on an attention mechanism by the multi-modal large language model, and stored in a KV-cache in a data structure of a key-value pair tensor. As shown in Figure 4 , the K and V calculated based on the attention mechanism are stored as n key-value pairs in the KV-cache; each key-value pair has corresponding ID information (as shown in the first row 410 in Figure 4 ), V value (as shown in the second row 420 in Figure 4 ), and K value (as shown in the third row 430 in Figure 4 ); n is an integer greater than 3.

[0061] Here, outputting the first generated content refers to converting the first generated content into voice and outputting it to the user. For example, in the case of the first generated content being a token sequence, the token sequence is converted into voice and cached in a voice cache, so that the system player reads and plays the voice from the voice cache; for example, in the case of the first generated content being a plurality of key-value pairs, a corresponding token sequence is generated based on the plurality of key-value pairs, the token sequence is converted into voice and cached in a voice cache, so that the system player reads and plays the voice from the voice cache; and the like.

[0062] In some embodiments, the second input speech obtained during output of the first generated content can refer to the second input speech obtained when part of the first generated content is output, or can refer to the second input speech obtained when all of the first generated content is output.

[0063] Here, the second input speech can refer to a speech having content relevance with the first input speech. In some embodiments, the second input speech is supplementary information of the content expressed in the first input speech. For example, after the user inputs “recommend some Beijing scenic spots” (i.e., the first input speech), the user again inputs “I don’t like climbing” (i.e., the second input speech). In some embodiments, the second input speech is feedback information of the output content in the first generated content. For example, after the user inputs “recommend some Beijing scenic spots” (i.e., the first input speech), the target model generates the first generated content, and when the system player plays “you can go to the Summer Palace” in the first generated content, the user again inputs “does this place need to climb” (i.e., the second input speech).

[0064] The second generated content refers to the output content determined from the first generated content, i.e., the generated content corresponding to the output speech in the first generated content. In this way, as described above, if the second input speech is obtained when part of the first generated content is output, the second generated content refers to the part of the first generated content; if the second input speech is obtained when all of the first generated content is output, the second generated content is all of the first generated content.

[0065] In some embodiments, after obtaining the second input speech input by the user, the second generated content can be determined in any manner. In some embodiments, the second generated content can be determined by detecting the playing progress of the speech corresponding to the first generated content. In some embodiments, the output second generated content can be calculated according to the playing time length of the speech corresponding to the first generated content when the second input speech is obtained, and the speech playing speed in the speech buffer. For example, in the case where the playing time length of the speech corresponding to the first generated content is 3 seconds and the speech playing speed in the speech buffer is 3 word units per second, the second generated content can be determined as the first 9 word units in the first generated content.

[0066] In step S32, first prompt information is generated according to the second input speech and the second generated content.

[0067] Here, after the second generated content is determined, the first prompt information is generated according to the second input speech and the second generated content, so that the target model performs model inference based on the first prompt information.

[0068] In some embodiments, when the second generated content is a token sequence and a key-value pair, the second generated content can be converted into text data, so as to generate the first prompt information according to the text data and the second input speech.

[0069] In some embodiments, when the second generated content is a key-value pair, the second generated content can be converted into a token sequence, so as to generate the first prompt information according to the token sequence and the second input speech.

[0070] In some embodiments, the second input speech can be converted into text data, and the first prompt information can be generated based on the text data and the second generated content.

[0071] In some embodiments, the second input speech can be converted into a token sequence, and the first prompt information can be generated based on the token sequence and the second generated content.

[0072] In this way, taking the second generated content as part of the first prompt information, the brain of the voice intelligent agent, i.e., the target model, can explicitly determine the context information of the second input speech. For example, in the above example, when the first input speech is “Recommend some Beijing scenic spots”, and the second input speech is “Does this place need to climb a mountain?”, according to the second generated content “You can go to the Summer Palace” in the first prompt information, the target model determines that the “this place” expressed by the user in the second input speech refers to the Summer Palace, rather than other Beijing scenic spots (e.g., Jingshan Park, the Forbidden City, and Prince Gong’s Mansion) that the target model has generated.

[0073] In step S33, according to the first prompt information, the target model generates third generated content corresponding to the second input speech.

[0074] Here, after determining the first prompt information, the target model continues model generation according to the second input information and the second generated content in the first prompt information, to obtain third generated content corresponding to the second input speech.

[0075] In the voice processing method provided in the application, first, in response to the second input voice obtained during output of the first generated content, the second generated content that has been output in the first generated content is determined, wherein the first generated content represents a model inference result for the first input voice generated by using a target model; then, first prompt information is generated according to the second input voice and the second generated content; finally, third generated content corresponding to the second input voice is generated by using the target model according to the first prompt information. In this way, by feeding back the second generated content that has been output to the target model, the context information of the second input voice can be determined by the target model, that is, the target model can determine which content has been heard by the user when the second input voice is input, so as to align the model generated content (i.e., the content that the model “thinks”) and the content that the user has received (i.e., the content that the model or voice agent “says”), and then in the user interruption scenario, the target model can more accurately and naturally reply to the newly input voice of the user, thereby improving the fluency and efficiency of voice interaction.

[0076] In some embodiments, the step S31 of determining the second generated content that has been output in the first generated content in response to the second input voice obtained during output of the first generated content can be implemented as the following steps S311 and S312.

[0077] In the step S311, in response to the second input voice obtained during output of the first cached voice, the output voice in the first cached voice is determined; the first cached voice represents the voice corresponding to the first generated content; and the output voice represents the part of the voice cached in advance in the first cached voice.

[0078] Here, the first cached voice refers to voice information converted from the first generated content.

[0079] In some embodiments, the first cached voice can be the cached voice obtained after performing voice conversion on all the content of the first generated content. For example, in the case where the first generated content is full-content generated content for the first input voice, i.e., full reply content, the first cached voice is the cached voice corresponding to the full reply content of the first input voice; in the case where the first generated content is part of the reply content generated by the target model for the first input voice, the first cached voice is the cached voice corresponding to the part of the reply content.

[0080] In some embodiments, the first buffered speech can be a buffered speech obtained after performing speech conversion on part of the first generated content. For example, in a case where the first generated content is a complete reply content for the first input speech, the first buffered speech is a buffered speech corresponding to part of the complete reply content that is generated first; in a case where the first generated content is part of the reply content that is generated by the target model for the first input speech, the first buffered speech is a buffered speech corresponding to part of the part of the reply content that is generated first.

[0081] The output speech refers to speech in the first buffered speech that has been output to the user. Here, the output speech is part of the first buffered speech that is buffered first, i.e., when the user inputs the second input speech, the first buffered speech is not completely output but is interrupted by the user, so that the amount of content generated by the target model is different from the amount of content output to the user.

[0082] In some embodiments, the output speech in the first buffered speech can be determined in any manner.

[0083] In some embodiments, determining the output speech in the first buffered speech can refer to determining speech identification information that has been output in the first buffered speech; wherein the speech identification information is used to uniquely identify a speech segment in the first buffered speech.

[0084] In some embodiments, determining the output speech in the first buffered speech can refer to calculating the length of the output speech according to a preset speech playing speed and a played time length of the first generated content.

[0085] Step S312, determining second generated content that has been output from the first generated content according to the output speech.

[0086] Here, after determining the output speech, the second generated content corresponding to the output speech is determined from the first generated content. For example, in a case where the first generated content is a token sequence, the token sequence corresponding to the output speech is determined from the token sequence according to the output speech; for another example, in a case where the first generated content is a plurality of key-value pairs, at least one key-value pair corresponding to the output speech is determined from the plurality of key-value pairs; and the like.

[0087] In the above embodiments provided by the present application, the second generated content is located by the output speech in the first buffered speech, which can improve the accuracy of the determined second generated content, thereby improving the accuracy of the context information of the second input speech input to the target model, so that in the case where the user interrupts, the target model can also give natural, fluent and accurate reply information.

[0088] In some embodiments, the first cache voice has a plurality of voice identifiers; the first generated content has a plurality of content identifiers; and the plurality of voice identifiers and the plurality of content identifiers have a first correspondence.

[0089] A voice identifier refers to information that can uniquely identify a voice segment. In some embodiments, a voice identifier can be any type of identifier information that can identify a voice segment. In some embodiments, a voice identifier can be a timestamp. For example, a first cache voice is divided into a plurality of voice segments, and each voice segment has a corresponding start timestamp or end timestamp. In some embodiments, a voice identifier can be a frame number. For example, in the case of frame management of the first cache voice, each frame of the cache voice has a corresponding frame number.

[0090] A content identifier refers to identifier information of a text, a word unit, or a key-value pair in the first generated content. In some embodiments, a content identifier can be a storage location identifier of a text or a word unit in the first generated content, and can also be a word unit identifier (ID), a key-value pair ID, etc.

[0091] Since the first cache voice is a voice converted from the first generated content, the plurality of voice identifiers of the first cache voice and the plurality of content identifiers of the first generated content have a first correspondence.

[0092] In this way, the step S312 of determining the second generated content that has been output in the first generated content according to the output voice can be implemented as the following step S3121:

[0093] In step S3121, the second generated content that has been output in the first generated content is determined according to at least one voice identifier corresponding to the output voice and the first correspondence.

[0094] Here, since the plurality of voice identifiers and the plurality of content identifiers have a first correspondence, after determining the output voice, at least one content identifier can be determined according to at least one voice identifier of the output voice, and the second generated content that has been output can be determined according to the at least one content identifier. For example, according to the frame number of the output voice, the content identifier that has been output can be determined, and the second generated content that has been output can be determined.

[0095] In some embodiments, the first buffered voice includes a plurality of sampling points. In this way, the position of the text, the position of the word unit, or the ID of the key-value pair, etc. that has been output can be determined according to the number of sampling points that have been output and the total number of sampling points. For example, the first buffered voice is buffered in the sound card buffer of the electronic device in binary form, and the first buffered voice includes 48,000 sampling points. When the voice is played, the playing system of the electronic device can determine the number of sampling points output from the sound card buffer. In this way, according to the proportion of the number of sampling points that have been played (for example, 16,000) and the total number of sampling points, and the corresponding relationship between the first generated content and the first buffered voice, the position of the text, the position of the word unit, or the ID of the key-value pair (for example, the key-value pair IDs 0, 1, 2, and 3 in the above example are determined as the key-value pair IDs that have been played) that has been played, etc. can be determined. Figure 4

[0096] In the above embodiments provided by the present application, by the corresponding relationship between the voice identifier and the content identifier, or the proportional relationship between the number of sampling points that have been output and the total number of sampling points, the output content in the first generated content can be positioned in units of text, word units, or key-value pairs, which improves the positioning accuracy of the output content, so that the target model can obtain more accurate context information.

[0097] In some embodiments, after determining the second generated content that has been output, the generated content in the first generated content that has not been output, i.e., the content other than the second generated content, can be cleared. For example, in the scenario of buffering key-value pairs by using the KV-buffer, the key-value pairs corresponding to the second generated content (for example, the key-value pairs corresponding to the key-value pair IDs 0, 1, 2, and 3 described above) can be retained, and other key-value pairs in the KV-buffer can be deleted.

[0098] In some embodiments, the step S32 of generating the first prompt information according to the second input voice and the second generated content can be implemented as the following steps S321 to S322:

[0099] In step S321, the first text corresponding to the second input voice and the second text corresponding to the second generated content are respectively determined; wherein the second generated content represents a word unit sequence or a key-value pair generated by the target model.

[0100] ​Here, the second input speech is converted into text type data to obtain the first text. In some embodiments, the second input speech can be converted into text type data in any manner. For example, the conversion from speech to text can be implemented by using an ASR model, such as a Hidden Markov Model (HMM), a Seq2Seq with Attention model, a model based on a Transformer architecture, and the like.

[0101] In the case where the second generated content is a token sequence or a key-value pair, the second generated content is converted into text type data to obtain the second text. In some embodiments, the token sequence or the key-value pair can be converted into text data in any manner.

[0102] For example, in the case where the second generated content is a token sequence, in some embodiments, if the token and the text have a clear one-to-one correspondence, the token sequence can be directly mapped to text data; in some embodiments, if the token and the text do not have a one-to-one correspondence, the token sequence can be decoded by using a probabilistic generative model, or the token sequence can be decoded in combination with an external knowledge base or user preferences.

[0103] For another example, in the case where the second generated content is a key-value pair, since the key-value pair is stored in the KV buffer corresponding to the model decoder, the key-value pair can be directly decoded by using the model decoder to obtain the corresponding token sequence, and then the token sequence is converted into text data.

[0104] At step S322, the first prompt information is generated according to the first text and the second text.

[0105] Here, the first text and the second text are used as at least a part of the first prompt information. In this way, when the target model performs model inference based on the first prompt information, the second input speech newly input by the user and the generated content (i.e., the context information of the second input speech) that has been heard by the user can be obtained from the first prompt information.

[0106] In some embodiments, in the case where the second generated content is a token sequence, the first prompt information can be generated according to the token sequence and the first text without converting the token sequence into text type data.

[0107] In some embodiments, the generation of the first prompt information according to the first text and the second text, i.e., the above step S322, can be implemented as the following step S321:

[0108] In step S321, the first prompt information is obtained by updating the second prompt information with the first text and the second text. The second prompt information represents the prompt information for generating the first generated content.

[0109] Here, the second prompt information refers to the prompt information input into the target model of the voice intelligent agent, so that the target model generates the first generated content. It can be seen that the second prompt information at least includes the first input voice input by the user. For example, the above-mentioned "recommend some Beijing scenic spots".

[0110] In some embodiments, in addition to the first input voice, the second prompt information can also include other historical interaction information between the user and the voice intelligent agent. For example, the second prompt information includes historical interaction information between the user and the voice intelligent agent in the last week or day.

[0111] In some embodiments, the length of the historical interaction information included in the second prompt information is not greater than the maximum context length that the target model can take into account, for example, 2048 word units.

[0112] In some embodiments, the information output to the user in the historical interaction information refers to the information output to the user in the form of voice.

[0113] In some embodiments, updating the second prompt information with the first text and the second text refers to adding the first text and the second text to the second prompt information to obtain the first prompt information.

[0114] In the above embodiments provided by the present application, the first prompt information is obtained by updating the second prompt information, which can make the target model obtain more context information related to the second input voice when processing the second input voice, and further improve the accuracy of model reasoning.

[0115] The target model performs loop reasoning during the generation process, and each loop reasoning consumes a large amount of computing resources, which poses a great challenge to the end-side device with limited computing power and power consumption. At the same time, unlike the text feedback mode, when giving voice feedback, if the voice playback speed is too fast, it will affect the intelligibility of the playback content. For example, for Chinese, the intelligibility is best when 3-5 Chinese characters (corresponding to an average of 2-5 word units) are played per second. It can be seen that in voice interaction, the target model does not need to perform too much advance reasoning, and too much advance reasoning will occupy a large amount of computing resources and device power consumption of the electronic device.

[0116] Therefore, in some embodiments, the method further includes the following steps S34 to S35:

[0117] Step S34, obtaining a content generation speed of the target model at present and a content output speed of the first generated content.

[0118] Here, the content generation speed of the target model at present refers to a speed of generating content when the target model performs model inference at present. In some embodiments, the content generation speed represents a speed of generating tokens when the target model generates a token sequence, or a speed of generating key-value pairs when the target model generates a key-value pair.

[0119] In some embodiments, the content generation speed of the target model is related to a model size and a device configuration running the model. For example, the higher the complexity and the larger the size of the target model, the slower the content generation speed of the target model; the higher the hardware configuration of the device running the model, the faster the content generation speed of the target model.

[0120] The content output speed of the first generated content refers to an output speed of the first generated content when the first generated content is output to a user in a voice form.

[0121] In some embodiments, the content output speed represents a voice output speed. In some embodiments, the voice output speed can be a sampling speed of a voice sampling point in the first buffered voice, and can also be an output speed determined according to a voice identifier in the first buffered voice.

[0122] In some embodiments, the content output speed can also represent an output speed of a token or a key-value pair in the first generated content.

[0123] In some embodiments, the content output speed is related to a user setting and a timbre, etc. For example, the user can manually adjust a voice playing speed, etc.

[0124] Step S35, updating the content generation speed of the target model according to the content output speed, wherein the updated content generation speed is lower than the content generation speed before the updating.

[0125] Here, the content generation speed of the target model is adjusted according to the content output speed of the first generated content, so as to reduce the content generation speed of the target model.

[0126] In the above embodiments provided by the present application, the content generation speed of the target model is updated according to the content output speed of the first generated content, so as to reduce the generation speed of the target model (i.e., to reduce the speed of the target model in the loop inference), thereby reducing the device power consumption when the model is in the loop inference, and avoiding wasting of the computing resources due to overthinking of the target model.

[0127] In some implementations, updating the content generation speed of the target model based on the content output speed, i.e., step S35 above, can be implemented as steps S351 to S352:

[0128] Step S351: Determine the adjustment factor based on the content generation speed and the content output speed;

[0129] Step S352: Use the adjustment factor as the interval between two adjacent lexical units or key-value pairs generated by the target model to update the content generation speed of the target model.

[0130] Here, the adjustment factor refers to the parameter used to adjust the content generation speed of the target model.

[0131] When adjusting the content generation speed using a modifier, the modifier is used as the interval between two adjacent tokens or key-value pairs generated by the target model. For example, with a modifier of 1ms, the target model generates one token or key-value pair, and then generates the next token or key-value pair after a 1ms interval.

[0132] In some implementations, the adjustment factor can be determined in any way based on the content generation speed and the content output speed.

[0133] For example, when the electronic device running the voice agent is a terminal device with limited computing power and power consumption, the adjustment factor can be determined with the content generation speed being the same as the content output speed. Correspondingly, when the terminal device has high computing power and power consumption, the adjustment factor can be determined with the content generation speed being higher than the content output speed by a specified value.

[0134] For example, in scenarios where high fluency of voice interaction is required, the adjustment factor can be determined with the goal of quickly generating voice response content and ensuring smooth output.

[0135] In the embodiments provided in this application, by adjusting the content generation speed of the target model by adjusting the adjustment factor, the effect of reducing the loop inference speed of the target model can be achieved, and the implementation method is simple and easy to implement.

[0136] Below, in conjunction with Figure 5 The process of performing speech processing using the speech processing system provided by the speech processing method in this disclosure is explained.

[0137] like Figure 5As shown, the voice processing system 500 includes a voice agent 510, a voice buffer 520, a system player 530, a speed feedback module 540, and a content feedback module 550; wherein the voice agent 510 includes a history information control module 511 and a target model 512, and the target model 512 can be implemented as a multi-modal large language model, etc. In this way, the process of processing user input voice by using the voice processing system 500 is as follows:

[0138] First, the user inputs a first input voice 501 to the voice agent 510;

[0139] Here, the first input voice is “recommend some Beijing characteristic dishes”.

[0140] Then, the voice agent 510 generates reply content for the first input voice by using the target model 512, and converts the generated reply content into voice and stores it in the voice buffer 520;

[0141] For example, in response to the user input “recommend some Beijing characteristic dishes”, the target model 512 generates a token sequence, and the text content corresponding to the token sequence is “you can try Qianlong cabbage, Beijing sweet and sour shredded pork, liver tip, and Beijing roast duck”.

[0142] After that, the voice cached in the voice buffer 520 is output to the system player 530 one by one to play the reply content;

[0143] Here, the system player 530 can be implemented as a loudspeaker of an electronic device.

[0144] At the same time, when playing the cached voice, the speed feedback module 540 determines the content output speed of the reply content according to the playing speed of the cached voice in the voice buffer 520; at the same time, the speed feedback module 540 obtains the content generation speed of the model from the target model 512; then, the speed feedback module 540 calculates the adjustment factor k according to the content output speed and the content generation speed; finally, the speed feedback module 540 feeds back the adjustment factor k to the target model 512, so that the target model 512 updates its content generation speed according to the adjustment factor k;

[0145] At the same time, when playing the cached voice, the content feedback module 550 feeds back the voice playing progress in the voice buffer 520 to the history information control module 511;

[0146] In some embodiments, the content feedback module 550 feeds back the text position, token position, or key-value pair ID corresponding to the played cached voice to the history information control module 511.

[0147] After that, in the voice playing process, the user inputs a second input voice 502 to the voice agent 510;

[0148] Here, the second input speech is "How spicy is this dish?"

[0149] After receiving the second input speech 502, the historical information control module 511 concatenates the played reply content into the new prompt information according to the text position, the token position or the key-value pair ID corresponding to the played speech;

[0150] Here, the played reply content is, for example, "Try Qianlong cabbage", and the historical information control module 511 concatenates the text type data corresponding to the content into the new prompt information. In some embodiments, the new prompt information is:

[0151] "System description: You are a helpful assistant;

[0152] User prompt 1: Recommend some Beijing dishes.

[0153] Assistant reply: Try Qianlong cabbage

[0154] User prompt 2: How spicy is this dish?"

[0155] After that, the historical information control module 511 inputs the new prompt information into the target model 512, so that the target model 512 performs model inference based on the new prompt information.

[0156] As can be seen, in the above embodiments, the content feedback module 550 feeds back the played content to the historical information control module 511, so that when the user interrupts the speech playing, the historical information control module 511 can update the prompt information of the target model 512 based on the played content, so that the target model 512 can determine the position of the user interruption (i.e., the reply content the user has heard) according to the played content, so as to better understand the meaning of the second input speech, for example, the specific content referred to by the pronoun "this" in the second input speech is "Qianlong cabbage", thereby improving the inference accuracy for the second input speech; on the other hand, the speed feedback module 540 generates an adjustment factor, and the content generation speed of the target model 512 is updated using the adjustment factor, so that the speed of the target model 512 can be appropriately reduced according to the content playing speed, thereby reducing the requirements for the computing power and power consumption of the electronic device.

[0157] Based on the foregoing embodiments, the present disclosure provides a speech processing apparatus, which comprises various units and modules included in the units, and can be implemented by a processor in a computer device. Of course, it can also be implemented by a specific logic circuit. In the implementation process, the processor can be a central processing unit (CPU), a microprocessor unit (MPU), a digital signal processor (DSP), or a field programmable gate array (FPGA).

[0158] Figure 6 The present disclosure provides a speech processing apparatus, which comprises various units and modules included in the units, and can be implemented by a processor in a computer device. Of course, it can also be implemented by a specific logic circuit. In the implementation process, the processor can be a central processing unit (CPU), a microprocessor unit (MPU), a digital signal processor (DSP), or a field programmable gate array (FPGA). Figure 6 As shown in the figure, the speech processing apparatus 600 comprises an output content determination module 610, a prompt information generation module 620, and a content generation module 630, wherein:

[0159] The output content determination module 610 determines the second generated content that has been output in the first generated content in response to the second input speech obtained during output of the first generated content; the first generated content represents a model processing result for the first input speech generated by using a target model;

[0160] The prompt information generation module 620 generates first prompt information according to the second input speech and the second generated content;

[0161] The content generation module 630 generates third generated content corresponding to the second input speech by using the target model according to the first prompt information.

[0162] In some embodiments, the output content determination module 610 comprises:

[0163] A first determination module is configured to determine the output speech in the first buffered speech in response to the second input speech obtained during output of the first buffered speech; the first buffered speech represents the speech corresponding to the first generated content; the output speech represents the previously buffered part of the speech in the first buffered speech;

[0164] A second determination module is configured to determine the second generated content that has been output in the first generated content according to the output speech.

[0165] In some embodiments, the first buffered speech has a plurality of speech identifiers; the first generated content has a plurality of content identifiers; the plurality of speech identifiers have a first correspondence relationship with the plurality of content identifiers;

[0166] The second determining module is configured to determine, according to the at least one speech identifier corresponding to the output speech and the first correspondence relationship, second generated content that has been output in the first generated content.

[0167] In some embodiments, the prompt information generation module 620 is configured to:

[0168] determine a first text corresponding to the second input speech and a second text corresponding to the second generated content, respectively; and

[0169] generate the first prompt information according to the first text and the second text.

[0170] In some embodiments, the prompt information generation module 620 is configured to update second prompt information by using the first text and the second text to obtain the first prompt information; and the second prompt information represents prompt information used to generate the first generated content.

[0171] In some embodiments, the device 600 further includes a speed feedback module; the speed feedback module is configured to:

[0172] obtain a content generation speed of the target model at present and a content output speed of the first generated content;

[0173] update the content generation speed of the target model according to the content output speed; and the updated content generation speed is lower than the content generation speed before the update.

[0174] In some embodiments, the content generation speed represents a word element generation speed or a key-value pair generation speed; and the content output speed represents a speech output speed.

[0175] In some embodiments, the speed feedback module is configured to:

[0176] determine an adjustment factor according to the content generation speed and the content output speed;

[0177] use the adjustment factor as an interval duration between two adjacent word elements or key-value pairs generated by the target model to update the content generation speed of the target model.

[0178] Based on the above embodiments, the present application further provides an electronic device. As shown in FIG. 7, Figure 7 The electronic device 700 includes at least one processor 710; the at least one processor 710 is configured to run a voice intelligent agent; wherein

[0179] The voice agent is configured to determine second generated content that has been output in the first generated content in response to a second input voice obtained during output of the first generated content, the first generated content representing a model processing result for the first input voice generated by using a target model, generate first prompt information according to the second input voice and the second generated content, and generate third generated content corresponding to the second input voice by using the target model according to the first prompt information.

[0180] The descriptions of the device embodiments and the equipment embodiments above are similar to the descriptions of the method embodiments, and have similar beneficial effects to the method embodiments. In some embodiments, the device and the equipment provided by the embodiments of the present disclosure have functions and include modules or units for performing the methods described in the method embodiments. For technical details of the device and the equipment that are not disclosed in the embodiments of the present disclosure, please refer to the descriptions of the method embodiments of the present disclosure.

[0181] It should be noted that, in the embodiments of the present disclosure, if the voice processing method described above is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on such understanding, the technical solutions of the embodiments of the present disclosure can be embodied in the form of a software product, which is stored in a storage medium and includes a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in the embodiments of the present disclosure. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a magnetic disk or an optical disk, and various other media that can store program codes. Thus, the embodiments of the present disclosure are not limited to any particular hardware, software or firmware, or any combination of hardware, software and firmware.

[0182] The embodiments of the present disclosure provide a computer device, including a memory and a processor, the memory stores a computer program capable of running on the processor, and the processor implements part or all of the steps in the above method when executing the program.

[0183] The embodiments of the present disclosure provide a computer-readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement part or all of the steps in the above method. The computer-readable storage medium can be transitory or non-transitory.

[0184] The embodiment of the present disclosure provides a computer program, including computer readable code, wherein when the computer readable code runs in a computer device, a processor in the computer device executes part or all steps of the above method.

[0185] The embodiment of the present disclosure provides a computer program product, including a non-transitory computer readable storage medium storing a computer program, wherein when the computer program is read and executed by a computer, part or all steps of the above method are implemented. The computer program product can be implemented by hardware, software or a combination thereof. In some embodiments, the computer program product is embodied as a computer storage medium, and in other embodiments, the computer program product is embodied as a software product, such as a software development kit (SDK) or the like.

[0186] It should be noted that the above description of the various embodiments tends to emphasize the differences between the various embodiments, and the same or similar parts can be referred to each other. The above description of the device, storage medium, computer program and computer program product embodiments is similar to the description of the method embodiments, and has similar advantages. For technical details not disclosed in the device, storage medium, computer program and computer program product embodiments of the present disclosure, please refer to the description of the method embodiments of the present disclosure.

[0187] It should be understood that the "one embodiment" or "an embodiment" mentioned throughout the specification means that the specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present disclosure. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It should be understood that in various embodiments of the present disclosure, the size of the sequence number of each step / process does not mean the execution order, and the execution order of each step / process should be determined by its function and inherent logic, and should not constitute any limitation on the implementation process of the embodiments of the present disclosure. The above sequence number of the embodiments of the present disclosure is only for description, not representing the advantages or disadvantages of the embodiments.

[0188] It should be noted that, in the present document, the terms "comprising", "containing" or any other similar term are intended to encompass non-exclusive inclusion, such that processes, methods, articles, or apparatuses that comprise a list of elements are not limited to those elements, but can also include other elements not expressly listed, or also include elements inherent in such processes, methods, articles, or apparatuses. Without further limitation, an element defined by the phrase "comprising a" does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes the element.

[0189] In several embodiments provided by the present disclosure, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely illustrative, for example, the division of the units is only a logical function division, and actual implementation can have another division manner, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the various components shown or discussed can be through some interface, indirect coupling or communication connection between devices or units, which can be electrical, mechanical or other forms.

[0190] The units described above as separate components can or can not be physically separate, and the components shown as units can or can not be physical units; they can be located in one place or distributed on multiple network units; and part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.

[0191] In addition, each functional unit in each embodiment of the present disclosure can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be realized in the form of hardware or in the form of hardware plus software functional unit.

[0192] Those of ordinary skill in the art can understand that all or part of the steps of the above method embodiments can be completed by program instruction related hardware, and the aforementioned program can be stored in a computer readable storage medium, and the program executes the steps of the above method embodiments when executed; and the aforementioned storage medium includes: mobile storage device, read only memory (Read Only Memory, ROM), magnetic disc or optical disc, and various storage program codes.

[0193] Alternatively, the above-mentioned integrated units of the present disclosure, if implemented in the form of software function modules and sold or used as independent products, can also be stored in a computer-readable storage medium. Based on such understanding, the technical solutions of the present disclosure can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the methods described in the various embodiments of the present disclosure. The aforementioned storage medium includes: mobile storage devices, ROM, magnetic disks or optical disks, and various media that can store program codes.

[0194] The above is only an embodiment of the present disclosure, but the protection scope of the present disclosure is not limited thereto, and any person skilled in the art can easily think of changes or replacements within the technical range disclosed by the present disclosure, which should be covered within the protection scope of the present disclosure.

Claims

1. A speech processing method, comprising: In response to obtaining a second input voice from the user during the output of the first generated content, determine the second generated content that has been output in the first generated content; The first generated content representation utilizes the model inference results generated by the target model for the first input speech; Based on the second input voice and the second generated content, a first prompt message is generated; Based on the first prompt information, the third generated content corresponding to the second input speech is generated using the target model.

2. The method according to claim 1, wherein the step of obtaining the second input speech input by the user during the output of the first generated content and determining the second generated content that has been output in the first generated content includes: In response to obtaining the second input speech during the output of the first buffered speech, determine the already output speech in the first buffered speech; The first cached speech represents the speech corresponding to the first generated content; The output speech represents a portion of the previously cached speech in the first cached speech; Based on the output speech, determine the second generated content that has been output in the first generated content.

3. The method according to claim 2, wherein the first cached speech has multiple speech identifiers; the first generated content has multiple content identifiers; and the multiple speech identifiers and the multiple content identifiers have a first correspondence relationship; The step of determining the second generated content that has been output in the first generated content based on the output speech includes: Based on at least one voice identifier corresponding to the output voice and the first correspondence, determine the second generated content that has been output in the first generated content.

4. The method according to any one of claims 1 to 3, wherein generating the first prompt information based on the second input voice and the second generated content includes: The first text corresponding to the second input speech and the second text corresponding to the second generated content are determined respectively; wherein, the second generated content represents the word sequence or key-value pair generated by the target model; The first prompt message is generated based on the first text and the second text.

5. The method according to claim 4, wherein generating the first prompt information based on the first text and the second text comprises: The first prompt information is obtained by updating the second prompt information using the first text and the second text; wherein the second prompt information represents the prompt information used to generate the first generated content.

6. The method according to claim 2, further comprising: Obtain the current content generation speed of the target model and the content output speed of the first generated content; The content generation speed of the target model is updated based on the content output speed; wherein the updated content generation speed is lower than the original content generation speed.

7. The method according to claim 6, wherein the content generation speed characterizes the word generation speed or key-value pair generation speed; and the content output speed characterizes the speech output speed.

8. The method according to claim 7, wherein updating the content generation speed of the target model based on the content output speed comprises: The adjustment factor is determined based on the content generation speed and the content output speed; The adjustment factor is used as the interval between two adjacent lexical or key-value pairs generated by the target model to update the content generation speed of the target model.

9. A voice processing device, comprising: The output content determination module, in response to obtaining the second input voice from the user during the output of the first generated content, determines the second generated content that has been output in the first generated content; The first generated content representation utilizes the model processing result generated by the target model for the first input speech; The prompt message generation module generates a first prompt message based on the second input voice and the second generated content; The content generation module generates third content corresponding to the second input speech based on the first prompt information and using the target model.

10. An electronic device comprising at least one processor; said at least one processor being configured to run a voice agent; wherein, The voice agent is used to determine the second generated content that has been output in the first generated content in response to receiving a second input voice from the user during the output of the first generated content. The first generated content representation utilizes the model processing result generated by the target model for the first input speech; Based on the second input voice and the second generated content, a first prompt message is generated; Based on the first prompt information, the third generated content corresponding to the second input speech is generated using the target model.

Images