Audio interaction method and device of generative language model based on dialogue control module, medium, program product and terminal

Abstract

The invention provides an audio interaction method and device of a generative language model based on a dialogue control module, a medium, a program product and a terminal, natural and smooth communication between a user and Llama-pot is realized by adopting technical means of efficient clause affiliation operation and dynamic task request processing, information of a plurality of clauses is quickly integrated by establishing a request ID, and the user experience is improved. According to the method, the time stamps and the clause identifiers are combined to ensure accurate affiliation of the information, so that the problems of information delay and context incoherence in traditional pseudo-streaming interaction are effectively solved, the real-time performance and coherence of interaction are improved, and the satisfaction degree of users and the application effect of Llama-pot are remarkably improved.

Description

Technical Field

[0001] The present application relates to the field of speech recognition, and in particular to an audio interaction method, device, medium, program product and terminal based on a generative language model of a dialogue control module. Background Art

[0002] Llama-bot is a generative language model based on artificial intelligence, which uses a pseudo-streaming scheme to interact with the dialogue management module (DM). However, the existing technology has exposed multiple technical defects in practical applications, which seriously affects the user experience. Specifically, the traditional pseudo-streaming scheme often leads to long response delays when processing segmented information, which increases the connection time between various links. This delay makes users feel obvious incoherence when interacting with the system, reducing the fluency of the interaction. In addition, the existing information splitting problem often leads to context incoherence. A single piece of information input by the user may be unreasonably split into multiple rounds of display, making it difficult to understand the interactive content. In actual use, users will feel that the interaction process seems fragmented due to the delay and splitting of information, and cannot achieve a natural and smooth communication experience, which directly affects the application effect of Llama-bot and user satisfaction. Therefore, in view of the problems of reduced user experience and unreasonable information splitting, a new interaction method is urgently needed to improve response efficiency and context coherence, thereby improving the overall interaction experience of users when using Llama-bot. Summary of the invention

[0003] In view of the shortcomings of the prior art described above, the purpose of the present application is to provide an audio interaction method, device, medium, program product and terminal based on a generative language model of a dialogue control module, which is used to solve the problem that the existing pseudo-streaming audio interaction scheme of the generative language model causes response delay and call record splitting abnormalities when processing segmented information, seriously affecting the user experience and the fluency of the interaction.

[0004] To achieve the above-mentioned purpose and other related purposes, the first aspect of the present application provides an audio interaction method based on a generative language model of a dialogue control module. The method is applied to the dialogue control module, including: receiving task request information input by a user, establishing a request ID corresponding to the task request information, and sending the task request information to the generative language model, so that the generative language model generates corresponding multiple sentence text information based on the task request information; receiving multiple sentence text information sent back by the generative language model; and performing the following operations on the sentence text information one by one in the order of receipt: performing a sentence attribution operation on the current sentence text information based on the request ID to generate a reply text corresponding to the current sentence text information, and performing TTS audio broadcast on the reply text corresponding to the current sentence text information; in the TTS audio broadcast During this period, a sentence attribution operation and TTS audio broadcast are performed on the next sentence text information based on the request ID; until the received sentence text information contains a termination mark; the next round of task request information is received again, a next request ID corresponding to the next round of task request information is established, and the next round of task request information is sent to the generative language model; multiple sentence text information of the next round sent back by the generative language model is received; the following operations are performed on the sentence text information one by one in the order of reception: a sentence attribution operation is performed on the current sentence text information based on the next request ID to generate a reply text corresponding to the current sentence text information, and a TTS audio broadcast is performed on the reply text corresponding to the current sentence text information; during the TTS audio broadcast, a sentence attribution operation and TTS audio broadcast are performed on the next sentence text information based on the next request ID.

[0005] In some embodiments of the first aspect of the present application, the sentence text information includes: a sentence timestamp, a sentence ID; the process of performing a sentence attribution operation on the sentence text information based on the request ID includes: extracting all sentence text information of the same round containing the same request ID, and performing a sentence attribution operation based on the sentence timestamp and sentence ID of all sentence text information of the same round to generate a reply text corresponding to the current sentence text information.

[0006] In some embodiments of the first aspect of the present application, the method further includes: in response to an interruption instruction input by a user, stopping the TTS audio broadcast of the current round, and continuing to receive the remaining sentence text information of the current round sent back by the generative language model, and performing the sentence attribution operation on the remaining sentence text information of the current round to generate a reply text corresponding to the current sentence text information, and adding the reply text of the current sentence text information to the interaction log; receiving the next round of task request information input by the user again, sending the next round of task request information to the generative language model, and establishing a unique request ID corresponding to the next round of task request information; receiving multiple sentence text information corresponding to the next round of task request information sent back by the generative language model and performing a sentence attribution operation to generate the next round of reply text, performing TTS audio broadcast on the next round of reply text, and adding the next round of information to the interaction log.

[0007] In some embodiments of the first aspect of the present application, the method further includes: in response to a rollback instruction input by a user, stopping the TTS audio broadcast of the current round, and continuing to receive the remaining sentence text information of the current round sent back by the generative language model, and extracting the valid information sentence before the rollback instruction is issued according to the sentence timestamp of the remaining sentence text information of the current round; receiving the next round of task request information input by the user, performing a merging operation on the task request information of the current round, the next round of task request information and the valid information sentence to generate rollback request information, sending the rollback request information to the generative language model, and establishing a unique request ID corresponding to the rollback request information; receiving multiple sentence text information corresponding to the rollback request information sent back by the generative language model; performing a sentence attribution operation to generate the next round of reply text, performing TTS audio broadcast on the next round of reply text, and adding the next round of information to the interaction log.

[0008] In some embodiments of the first aspect of the present application, the method also includes: if it is detected that the multiple sentence text information sent back by the generative language model contains a preset end condition, ending the task request of the current round, and adding the task request information in the current round and the reply text to the interaction log.

[0009] In some embodiments of the first aspect of the present application, the process of receiving multiple sentence text information sent back by the generative language model also includes: if the generative language model has not returned the next frame of sentence text information within a preset time after returning the first frame of sentence text information, it is determined that the generative language model is abnormal, the preset fallback words are broadcast and the phone is hung up; and / or, if the generative language model does not send back the first frame of sentence text information within a preset time, it is determined that the generative language model is abnormal, the preset fallback words are broadcast, and the reply text is no longer broadcast; and / or, if any sentence text information returned by the generative language model contains a speech synthesis markup language error, it is determined that the model is abnormal, the preset fallback words are broadcast, and the reply text is no longer broadcast.

[0010] To achieve the above-mentioned purpose and other related purposes, the second aspect of the present application provides an audio interaction device based on a generative language model of a dialogue control module, the device comprising: a task request module: used to receive task request information input by a user, establish a unique request ID corresponding to the task request information, and send the task request information to the generative language model, so that the generative language model generates corresponding multiple sentence text information based on the task request information; an interaction management module: used to receive multiple sentence text information sent back by the generative language model; and perform the following operations on the sentence text information one by one in the order of receipt: perform a sentence attribution operation on the current sentence text information based on the request ID to generate a reply text corresponding to the current sentence text information, and perform TTS audio broadcast on the reply text corresponding to the current sentence text information; in the TTS During the S audio broadcast, a sentence attribution operation and TTS audio broadcast are performed on the next sentence text information based on the request ID; until the received sentence text information contains a termination mark; the next round of task request information is received again, a next request ID corresponding to the next round of task request information is established, and the next round of task request information is sent to the generative language model; multiple sentence text information of the next round sent back by the generative language model is received; the following operations are performed on the sentence text information one by one in the order of reception: a sentence attribution operation is performed on the current sentence text information based on the next request ID to generate a reply text corresponding to the current sentence text information, and a TTS audio broadcast is performed on the reply text corresponding to the current sentence text information; during the TTS audio broadcast, a sentence attribution operation and TTS audio broadcast are performed on the next sentence text information based on the next request ID.

[0011] To achieve the above-mentioned purpose and other related purposes, the third aspect of the present application provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the audio interaction method of the generative language model based on the dialogue control module is implemented.

[0012] To achieve the above-mentioned purpose and other related purposes, the fourth aspect of the present application provides a computer program product, which includes a computer program code. When the computer program code runs on a computer, the computer implements the audio interaction method of the generative language model based on the dialogue control module.

[0013] To achieve the above-mentioned purpose and other related purposes, the fifth aspect of the present application provides an electronic terminal, including a memory, a processor and a computer program stored in the memory; the processor executes the computer program to implement the audio interaction method of the generative language model based on the dialogue control module.

[0014] As described above, the audio interaction method, device, medium, program product and terminal of the generative language model based on the dialogue control module of the present application have the following beneficial effects: This solution effectively improves the interaction experience between the user and Llama-bot through an innovative audio interaction method based on the generative language model of the dialogue control module. This method not only significantly reduces the response delay, ensuring that the user can obtain the required information in the shortest time, but also ensures the contextual coherence of each round of dialogue content through sentence attribution operations. In addition, the system can flexibly respond to the user's interruption and fallback instructions, making the interaction more natural and smooth. These improvements have jointly contributed to the improvement of the efficiency of information processing and the user experience, enabling Llama-bot to better meet the real-time needs of users and enhance its application effect and user satisfaction. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 A flow chart of an embodiment of an audio interaction method based on a generative language model of a dialogue control module of the present application is shown.

[0016] Figure 2 A structural diagram of an embodiment of an audio interaction device based on a generative language model of a dialogue control module of the present application is shown.

[0017] Figure 3 A structural diagram of an embodiment of an audio interaction terminal based on a generative language model of a dialogue control module of the present application is shown. DETAILED DESCRIPTION

[0018] The following describes the embodiments of the present application through specific examples, and those skilled in the art can easily understand other advantages and effects of the present application from the contents disclosed in this specification. The present application can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present application. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.

[0019] Before further describing the present invention in detail, the nouns and terms involved in the embodiments of the present invention are explained. The nouns and terms involved in the embodiments of the present invention are applicable to the following interpretations:

[0020] <1> DM: Dialog Management Module is a key component used in chatbots or dialogue systems. It is responsible for monitoring and managing the user interaction process, processing the dialogue state, tracking context information, and generating appropriate responses based on user input to ensure the coherence and logic of the dialogue.

[0021] <2> LLAMA-BOT: A chatbot built on a large language model (LLAMA, Large Language Model Meta AI) that uses natural language processing technology for dialogue generation, can understand user input, and generate context-related responses. It is widely used in customer service, virtual assistants, and online interactive platforms.

[0022] <3> Sentence text information: refers to a structured data form in which a text is split and organized into sentences. Each sentence is extracted as an independent unit to facilitate subsequent analysis and processing. This information is of great significance in natural language processing (NLP) and text analysis, because this split can help the system understand semantic structure, perform sentiment analysis, and support tasks such as information retrieval.

[0023] <4> Sentence attribution operation: In natural language processing (NLP) tasks, it refers to the process of analyzing and classifying sentences in a sentence or text, aiming to identify the relationship between sentences to achieve deeper language understanding and data processing, and facilitate semantic analysis and information extraction.

[0024] <5> Timestamp: A type of information used to record the specific time when an event occurred, usually including the representation of date and time. It has a wide range of applications, such as event sequences in data logs, emails, blockchains, and other systems, helping to trace the event process and ensure time consistency.

[0025] <6> Dialogue turn: In a dialogue system or AI chatbot, it refers to the completion of a complete interaction process, including user input and system response. Dialogue turn is an important indicator for evaluating dialogue quality, user satisfaction and system understanding ability, and is crucial for optimizing dialogue models.

[0026] <7> SSML: Speech Synthesis Markup Language, is a markup language used to control text-to-speech (TTS) systems. SSML allows developers to adjust speech synthesis characteristics such as pitch, speed, pauses, and stress through specific tags, thereby improving the naturalness and comprehensibility of speech output. Through detailed annotations, SSML can significantly improve the accuracy of speech synthesis and help detect faults.

[0027] <8> TTS announcement: Text-to-Speech technology converts written text into audible speech. It is widely used in voice assistants, navigation systems, educational software, etc. It supports multiple languages ​​and voice styles and aims to provide barrier-free access and improve user experience.

[0028] To facilitate understanding of the embodiments of the present application, first Figure 1 Detailed description. Figure 1 A flow chart of an audio interaction method based on a generative language model of a dialogue control module in an embodiment of the present invention is shown. The audio interaction method based on a generative language model of a dialogue control module in this embodiment mainly includes the following steps:

[0029] Step S11: receiving task request information input by a user, establishing a request ID corresponding to the task request information, and sending the task request information to a generative language model, so that the generative language model generates a plurality of corresponding sentence text information based on the task request information.

[0030] In one embodiment of the present invention, the dialogue control module DM (Dialogue Management) refers to a component that controls the dialogue process and manages the dialogue state. DM is responsible for managing the dialogue state and deciding the next dialogue action. Specifically, DM is responsible for receiving user input, analyzing the current dialogue state, and deciding how to respond to user requests or questions. In terms of generative language models, it includes but is not limited to: Llama-bot. Llama-bot is a dialogue robot based on a large language model that uses a generative language model to generate human-like dialogue responses. Llama-bot can learn language patterns and semantic similarities by learning a large amount of text data, thereby generating natural and fluent dialogue responses.

[0031] In one embodiment of the present invention, when DM sends a task request content to Llama-bot, a unique request identifier (request_id) will be established to track the status and history of the current request. The request_id will serve as a unique identifier for the current request to ensure that the processing of the current request is performed in the correct context. When DM sends a task request content, the request content will include a series of parameters, such as user input, task type, etc. These parameters will be accepted and processed by Llama-bot to generate an accurate response. In this embodiment, request_id, as a unique identifier for the current request, will be used to track the status and history of the current request.

[0032] It is worth noting that the present invention ensures that each request can be accurately tracked and managed by receiving the task request information input by the user and establishing a unique request ID. In this way, once the user issues a request, the DM system can process it quickly and provide real-time feedback, thereby significantly reducing response delays. Through priority management and efficient task scheduling, this technology improves the smoothness of interaction with users and solves the problem of long waiting time for users during the interaction process.

[0033] In one embodiment of the present invention, the sentence details of the previous round of robot broadcasts will be added to the content of the current request, and the sentence details include: TTS broadcast time, sentence id, request_id to which the sentence belongs, and other information. This information will be passed to Llama-bot as a historical record of the current request, so that Llama-bot can understand the context of the current request and generate a more accurate response. In this embodiment, the sentence details of the previous round of robot broadcasts are input so that Llama-bot can better understand the context of the current request, generate more accurate and relevant responses, and avoid unnecessary repetition or disconnection. By utilizing historical records, Llama-bot can not only better understand user needs, but also learn and improve its own generation capabilities, thereby providing a better user experience.

[0034] In one embodiment of the present invention, the process of receiving multiple sentence text information sent back by the generative language model also includes: if the generative language model does not return the next frame of sentence text information within a preset time after returning the first frame of sentence text information, it is determined that the generative language model is abnormal, the preset fallback words are broadcast and the phone is hung up; and / or, if the generative language model does not send back the first frame of sentence text information within a preset time, it is determined that the generative language model is abnormal, the preset fallback words are broadcast, and the reply text is no longer broadcast; and / or, if any sentence text information returned by the generative language model contains a speech synthesis markup language error, it is determined that the model is abnormal, the preset fallback words are broadcast, and the reply text is no longer broadcast.

[0035] Furthermore, when the generative language model returns the sentence text information for the first time, the DM needs to wait for a certain period of time (for example, 10 seconds) to receive the next frame of sentence text information. If the next frame of sentence text information is still not received after exceeding the preset time length, the DM will determine that an abnormality has occurred in the generative language model, broadcast the preset fallback words and hang up. More preferably, the DM can also perform multiple operations such as retrying the connection, switching models, and feeding back error messages. Specifically, the DM can try to reconnect to the generative language model to obtain the next frame of sentence text information; the DM can also switch to a backup generative language model to continue providing services; the DM can send an error message to the user, notify the user of the current service abnormality, and provide further solutions.

[0036] In one embodiment of the present invention, in order to avoid awkward silence when the DM is not replied immediately, the DM instructs the TTS to play preset response words or phrases to maintain the fluency of the conversation. The response words include, but are not limited to, phrases that affirm the user's question or express interest. Exemplarily, the response words can be set to "This is an interesting question", "I also think this topic is attractive", "Yes, you are right", "It is indeed worth discussing", "This is an important point of view", etc. By playing these response words in a loop within a preset duration, the DM can enhance the user experience, increase interactivity, and reduce the embarrassment caused by response delays.

[0037] Furthermore, the process of containing a speech synthesis markup language error in any sentence text information returned by the generative language model includes: when the sentence text information returned by the generative language model contains a speech synthesis markup language (SSML) error, the DM will determine that the model is abnormal, broadcast a preset fallback speech, and no longer broadcast the reply text. More preferably, the DM can also perform one or more of the following operations: check SSML syntax, switch to a backup model, and feedback error information. The SSML syntax can be checked, and the cause of the error can be determined by checking the SSML syntax; the DM can switch to a backup generative language model to avoid SSML syntax errors; the DM can provide error information to users to inform them of service abnormalities and provide further solutions.

[0038] It should be noted that SSML (Speech Synthesis Markup Language) is a markup language used for text-to-speech synthesis, which allows developers to control the speed, pitch, volume and other properties of synthesized speech to improve the naturalness and comprehensibility of speech synthesis. When the sentence text information returned by the generative language model contains SSML syntax errors, it may cause speech playback failure or playback content errors, thus affecting the user experience. Therefore, when DM detects SSML syntax errors, it will determine that the model is abnormal and start the preset fallback speech to ensure that users receive error prompts to improve DM's reliability and user satisfaction.

[0039] In one embodiment of the present invention, the fallback phrases refer to some preset standardized responses when DM or Llama-bot encounters an abnormality or fails to provide normal services, which are used to promptly communicate the problem to the user and maintain a good user experience. Common fallback phrases include "Sorry, there is a problem with DM, please try again later", "We are working hard to solve this problem, please be patient", and "Sorry, we cannot provide the information you requested, please check your input or try again later". For example, when DM detects an SSML syntax error, it may broadcast "Sorry, we cannot process your request, please make sure the input content is in the correct format" so that the user knows the problem and guides them to try again.

[0040] In one embodiment of the present invention, the method further includes: if it is detected that the multiple sentence text information sent back by the generative language model contains a preset end condition, ending the task request of the current round, and adding the task request information in the current round and the reply text to the interaction log.

[0041] In one embodiment of the present invention, when it is detected that a preset end condition is included in the multiple sentence text information sent back by the generative language model, the specific process includes checking whether the sentence of the current reply contains an end mark. In this process, the streaming end condition (stream_end) refers to a specific signal or sign indicating the end of the task in the generated reply text by the generative language model. When stream_end is detected as true, it means that the current streaming request has ended, that is, the task request of this round is deemed to be completed. At this time, DM records the information of the task request and the corresponding reply text in the interaction log for subsequent query and analysis. Through this process, the accurate recording and tracking of task requests are ensured, thereby improving the interactive efficiency and manageability of DM.

[0042] Furthermore, the streaming end condition is a specific identifier automatically generated and added by the generative language model Llama-bot at the end of each sentence to identify the end status of the reply content. This identifier is intended to clearly indicate whether the current task request has been completed. In the process of processing user input and generating reply text, Llama-bot will determine when to appropriately end the current sentence based on the set logic DM. When specific conditions are met, such as reaching the upper limit of the content length, the appearance of specific keywords, or the user's intention has been fully expressed, Llama-bot will automatically insert the streaming end condition at the end of the generated sentence. This mechanism not only ensures the coherence and integrity between sentences, but also helps DM accurately identify the termination time of the task request, thereby significantly improving the interaction efficiency and user experience. Through this automated processing method, users can enjoy more fluent and natural dialogue feedback, while DM can also efficiently record and manage the information of each interaction.

[0043] Step S12: receiving multiple sentence text information sent back by the generative language model; and performing the following operations on the sentence text information one by one in the order of receiving: performing a sentence attribution operation on the current sentence text information based on the request ID to generate a reply text corresponding to the current sentence text information, and performing TTS audio broadcast on the reply text corresponding to the current sentence text information; during the TTS audio broadcast, performing a sentence attribution operation and TTS audio broadcast on the next sentence text information based on the request ID; until the received sentence text information contains a termination mark.

[0044] In one embodiment of the present invention, the sentence text information includes: a sentence timestamp, a sentence ID; the process of performing a sentence attribution operation on the sentence text information based on the request ID includes: extracting all sentence text information of the same round containing the same request ID, and performing a sentence attribution operation based on the sentence timestamps and sentence IDs of all sentence text information of the same round to generate a reply text corresponding to the current sentence text information. The sentence timestamp represents the time when each sentence is generated, so as to track the order and timeline of the conversation. The sentence ID assigns a unique identifier to each sentence for reference and management during the processing process.

[0045] In this embodiment, the round refers to a complete interaction process between user input and DM reply. The sentence text information of the same round refers to all sentences generated in the same round of dialogue. By focusing on the sentences of the same round, it can be ensured that the generated reply is associated with the user's latest input and maintain the coherence of the dialogue.

[0046] In this embodiment, the sentence attribution operation refers to the process of associating specific sentence text information with its corresponding request ID in order to generate a suitable reply text. The process of the sentence attribution operation includes: extracting all relevant sentence information based on the request ID, and sorting them according to the timestamps of the sentences to ensure that they are processed in the order in which the conversation occurs. Next, DM performs semantic analysis on each sentence to determine its main intention and emotion. Subsequently, DM classifies and categorizes the extracted sentences based on the results of the context analysis to clarify the relationship between the sentences. Subsequently, the classified and parsed sentence information is integrated to ensure that the reply text contains a full response to the user's latest input, and natural language generation technology is used to convert it into a fluent reply text. Finally, the generated reply text is broadcast to the user in TTS voice, and the conversation record is stored in the interaction log so that DM can continuously optimize and improve subsequent interactions.

[0047] It should be noted that the present invention extracts the sentence text information with the same request ID to ensure that in the same round of dialogue, all sentences are related and consistent. And by performing attribution operations on the sentences, it is ensured that the generated text remains at a logical level. DM will not arbitrarily split the information into multiple rounds of display, but will centrally process the relevant information to form a coherent reply. This reasonable induction of sentences effectively avoids the user's understanding obstacles caused by information splitting, making the dialogue more natural, thereby solving the discomfort in the user experience.

[0048] In one embodiment of the present invention, in addition to using speech synthesis technology (TTS) for broadcasting, DM will also display the reply text and the information input by the user to the user in text form after the interaction is completed. This display will include each input of the user and the corresponding DM reply to ensure that the user can clearly review the entire conversation process. This text display not only helps the user understand and review the interaction content, but also serves as a basis for subsequent information query, enhancing the user's sense of participation and satisfaction. In addition, text records can support users to conveniently view the historical information of the conversation when needed, further improving the user experience.

[0049] Step S13: receiving the next round of task request information again, establishing a next request ID corresponding to the next round of task request information, and sending the next round of task request information to the generative language model; receiving multiple sentence text information of the next round sent back by the generative language model; performing the following operations on the sentence text information one by one in the order of reception: performing a sentence attribution operation on the current sentence text information based on the next request ID to generate a reply text corresponding to the current sentence text information, and performing TTS audio broadcast on the reply text corresponding to the current sentence text information; during the TTS audio broadcast, simultaneously performing a sentence attribution operation and TTS audio broadcast on the next sentence text information based on the next request ID.

[0050] In one embodiment of the present invention, the method further includes: in response to an interruption instruction input by a user, stopping the TTS audio broadcast of the current round, and continuing to receive the remaining sentence text information of the current round sent back by the generative language model, and performing the sentence attribution operation on the remaining sentence text information of the current round to generate a reply text corresponding to the current sentence text information, and adding the reply text of the current sentence text information to the interaction log; receiving the next round of task request information input by the user again, sending the next round of task request information to the generative language model, and establishing a unique request ID corresponding to the next round of task request information; receiving multiple sentence text information corresponding to the next round of task request information sent back by the generative language model and performing a sentence attribution operation to generate the next round of reply text, performing TTS audio broadcast on the next round of reply text, and adding the next round of information to the interaction log.

[0051] In this embodiment, a more detailed processing flow is proposed for interruptions that may occur in the current round to ensure that the interaction between the user and the generative language model (Llama-bot) is smoother and more efficient. Llama-bot will continue to generate subsequent sentence content and send it to DM under any circumstances to ensure the continuity of information. DM records the sentence status of each request in JSON format, and uses request_id to dynamically manage the attribution and status of the sentence. If an interruption occurs, DM will immediately stop TTS broadcasting to prevent redundant information, and will no longer process timed requests. In the next round of requests, DM will perform sentence attribution operations on the task content based on the new request ID. For previously interrupted replies, DM will only record their content in JSON, and will not perform text-to-speech (TTS) broadcasting to ensure that the new round of processing will not affect the previous records.

[0052] In one embodiment of the present invention, the method further includes: in response to a rollback instruction input by a user, stopping the TTS audio broadcast of the current round, and continuing to receive the remaining sentence text information of the current round sent back by the generative language model, and extracting the valid information sentence before the rollback instruction is issued according to the sentence timestamp of the remaining sentence text information of the current round; receiving the next round of task request information input by the user, performing a merging operation on the task request information of the current round, the next round of task request information and the valid information sentence to generate rollback request information, sending the rollback request information to the generative language model, and establishing a unique request ID corresponding to the rollback request information; receiving multiple sentence text information corresponding to the rollback request information sent back by the generative language model; performing a sentence attribution operation to generate the next round of reply text, performing TTS audio broadcast on the next round of reply text, and adding the next round of information to the interaction log.

[0053] In one embodiment of the present invention, the steps performed by DM (dialogue manager) after receiving the user's fallback instruction include: first, DM immediately stops the current text-to-speech (TTS) audio broadcast, and DM continues to receive the remaining sentence text information of the current round returned by the generative language model (such as llama-bot). And the reply information for which TTS broadcast is not executed is discarded to ensure that the user will not be disturbed by redundant information during the fallback process. Subsequently, DM extracts the valid information sentence before the user issues the fallback instruction according to the timestamp, and merges the request information of the current round, the user's new request, and the extracted valid information sentence to generate a fallback request information, and assigns a unique request ID to this request. The merged request is then sent to the generative language model for processing.

[0054] Furthermore, when the generative language model completes processing, DM will receive multiple sentence text information corresponding to the rollback request information. To ensure the coherence of the information, DM performs sentence attribution operations and generates new reply text. The generated reply text will be re-broadcasted through TTS audio and recorded in the interaction log for subsequent user tracking. This design effectively improves the user's interactive experience in multiple rounds of dialogue, reduces the user's cognitive burden during the information review process, and ensures the integrity and effectiveness of information transmission.

[0055] The present invention allows users to interrupt or back off during a conversation by setting up corresponding interruption mechanisms and back off mechanisms. This flexible mechanism enables DM to continue processing unfinished sentences after the user issues an interruption command, thereby ensuring complete communication of information. When the user backs off, DM can merge valid information, thus avoiding the interaction fragmentation caused by user operations, making the conversation appear more coherent and natural. And through the anomaly detection mechanism, when an abnormality occurs in DM, a clear feedback is given to the user immediately, thereby reducing the frustration caused by the DM's failure to respond immediately. Even in abnormal situations, users can still obtain certain information to maintain basic communication, which greatly enhances the stability of the user experience.

[0056] It should be noted that in the embodiments of the present application, words such as "exemplary" or "for example" represent examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0057] In the embodiments of the present application, "at least one" refers to one or more, and "plurality" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can represent: a, b, c, ab, ac, bc or abc, where a, b, c can be single or multiple.

[0058] Figure 2 is a schematic block diagram of an audio interaction device based on a generative language model of a dialogue control module provided in an embodiment of the present application. Figure 2As shown, the device includes a task request module 201 and an interaction management module 202 .

[0059] Task request module 201: used to receive task request information input by a user, establish a unique request ID corresponding to the task request information, and send the task request information to a generative language model so that the generative language model generates corresponding multiple sentence text information based on the task request information.

[0060] Interaction management module 202: used to receive multiple sentence text information sent back by the generative language model; and perform the following operations on the sentence text information one by one in the order of reception: perform a sentence attribution operation on the current sentence text information based on the request ID to generate a reply text corresponding to the current sentence text information, and perform TTS audio broadcast on the reply text corresponding to the current sentence text information; during the TTS audio broadcast, perform a sentence attribution operation and TTS audio broadcast on the next sentence text information based on the request ID; until the received sentence text information contains a termination mark; receive the next round of task request information again, A next request ID corresponding to the next round of task request information is established, and the next round of task request information is sent to a generative language model; multiple sentence text information of the next round sent back by the generative language model is received; the following operations are performed on the sentence text information one by one in the order of reception: a sentence attribution operation is performed on the current sentence text information based on the next request ID to generate a reply text corresponding to the current sentence text information, and a TTS audio broadcast is performed on the reply text corresponding to the current sentence text information; during the TTS audio broadcast, a sentence attribution operation and a TTS audio broadcast are performed on the next sentence text information based on the next request ID.

[0061] It should be understood that the specific process of each module executing the above corresponding steps has been described in detail in the above method embodiment, and for the sake of brevity, it will not be repeated here.

[0062] It should also be understood that the division of modules in the embodiments of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation. In addition, each functional module in each embodiment of the present application may be integrated into a processor, or may exist physically separately, or two or more modules may be integrated into one module. The above-mentioned integrated modules may be implemented in the form of hardware or in the form of software functional modules.

[0063] Figure 3 is a schematic block diagram of an electronic terminal provided in an embodiment of the present application. Figure 3As shown, the electronic terminal includes: at least one processor 301, a memory 302, at least one network interface 303 and a user interface 305. The various components in the device are coupled together through a bus system 304. It can be understood that the bus system 304 is used to realize the connection and communication between these components. In addition to the data bus, the bus system 304 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, Figure 3 In the specification, various buses are labeled as bus systems.

[0064] The user interface 305 may include a display, a keyboard, a mouse, a trackball, a click gun, keys, buttons, a touch pad or a touch screen.

[0065] It is understood that the memory 302 can be a volatile memory or a non-volatile memory, and can also include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), which is used as an external cache. By way of example but not limitation, many forms of RAM are available, such as static random access memory (SRAM), synchronous static random access memory (SSRAM). The memory described in the embodiments of the present invention is intended to include but is not limited to these and any other suitable categories of memory.

[0066] The memory 302 in the embodiment of the present invention is used to store various categories of data to support the operation of the electronic terminal 300. Examples of these data include: any executable program for operating on the electronic terminal 300, such as an operating system 3021 and an application 3022; the operating system 3021 includes various system programs, such as a framework layer, a core library layer, a driver layer, etc., for implementing various basic services and processing hardware-based tasks. The application 3022 can include various applications, such as a media player (Media Player), a browser (Browser), etc., for implementing various application services. The 33 method provided by the embodiment of the present invention can be included in the application 3022.

[0067] The method disclosed in the above embodiment of the present invention can be applied to the processor 301, or implemented by the processor 301. The processor 301 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit in the processor 301 or the instruction in the form of software. The above processor 301 may be a general processor, a digital signal processor (DSP, Digital Signal Processor), or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The processor 301 can implement or execute the methods, steps and logic block diagrams disclosed in the embodiment of the present invention. The general processor 301 can be a microprocessor or any conventional processor, etc. In combination with the steps of the accessory optimization method provided in the embodiment of the present invention, it can be directly embodied as a hardware decoding processor to execute, or it can be executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium, which is located in a memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0068] In an exemplary embodiment, the electronic terminal 300 may be implemented by one or more application specific integrated circuits (ASIC), DSP, programmable logic device (PLD), complex programmable logic device (CPLD) to execute the aforementioned method.

[0069] According to the method provided in the embodiments of the present application, the present application also provides a computer program product, which includes: computer program code, when the computer program code is run on a computer, the computer executes the audio interaction method based on the generative language model of the dialogue control module in any of the embodiments shown above.

[0070] According to the method provided in the embodiments of the present application, the present application also provides a computer-readable storage medium, which stores a program code. When the program code runs on a computer, the computer executes the audio interaction method based on the generative language model of the dialogue control module in any of the embodiments shown above.

[0071] The terms "component", "module", "system", etc. used in this specification are used to represent computer-related entities, hardware, firmware, a combination of hardware and software, software, or software in execution. For example, a component can be, but is not limited to, a process running on a processor, a processor, an object, an executable file, an execution thread, a program and / or a computer. By way of illustration, both applications running on a computing device and a computing device can be components. One or more components may reside in a process and / or an execution thread, and a component may be located on a computer and / or distributed between two or more computers. In addition, these components may be executed from various computer-readable media having various data structures stored thereon. Components may, for example, communicate through local and / or remote processes according to signals having one or more data packets (e.g., data from two components interacting with another component between a local system, a distributed system and / or a network, such as the Internet interacting with other systems through signals).

[0072] Those of ordinary skill in the art will appreciate that the various illustrative logical blocks and steps described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0073] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0074] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0075] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0076] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0077] In the above embodiments, the functions of each functional unit can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions (programs). When loading and executing computer program instructions (programs) on a computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions can be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (Digital Subscriber Line, DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server, data center, etc. that contains one or more available media integrated. Available media may be magnetic media (e.g., floppy disks, hard disks, tapes), optical media (e.g., high-density digital video discs (DVDs), or semiconductor media (e.g., solid state disks (SSDs)).

[0078] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc., various media that can store program codes.

[0079] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0080] In summary, the present application provides an audio interaction method, device, medium, program product and terminal based on a generative language model of a dialogue control module. The present invention provides a method for reducing response delays and maintaining the integrity and flow of the audio interaction process, and adopts efficient sentence attribution operations and dynamic task request processing technical means to achieve natural and smooth communication between users and Llama-bot. By establishing a request ID to quickly integrate multiple sentence information, combining timestamps and sentence identifiers to ensure accurate attribution of information, the information delay and context incoherence problems in traditional pseudo-streaming interactions are effectively solved, the real-time and continuity of the interaction are improved, and the user satisfaction and the application effect of Llama-bot are significantly improved. Therefore, the present application effectively overcomes the various shortcomings in the prior art and has a high industrial utilization value.

[0081] The above embodiments are merely illustrative of the principles and effects of the present application and are not intended to limit the present application. Anyone familiar with the technology may modify or change the above embodiments without violating the spirit and scope of the present application. Therefore, all equivalent modifications or changes made by a person of ordinary skill in the art without departing from the spirit and technical ideas disclosed in the present application shall still be covered by the claims of the present application.

Claims

1. An audio interaction method based on a generative language model of a dialogue control module, characterized in that: The method is applied to a dialogue control module, comprising: Receive task request information input by a user, establish a request ID corresponding to the task request information, and send the task request information to a generative language model, so that the generative language model generates a plurality of corresponding sentence text information based on the task request information; Receive multiple sentence text information sent back by the generative language model; and perform the following operations on the sentence text information one by one in the order of receipt: perform a sentence attribution operation on the current sentence text information based on the request ID to generate a reply text corresponding to the current sentence text information, and perform TTS audio broadcast on the reply text corresponding to the current sentence text information; during the TTS audio broadcast, perform a sentence attribution operation and TTS audio broadcast on the next sentence text information based on the request ID; until the received sentence text information contains a termination mark; Receive the next round of task request information again, establish a next request ID corresponding to the next round of task request information, and send the next round of task request information to the generative language model; receive multiple sentence text information of the next round sent back by the generative language model; perform the following operations on the sentence text information one by one in the order of reception: perform a sentence attribution operation on the current sentence text information based on the next request ID to generate a reply text corresponding to the current sentence text information, and perform TTS audio broadcast on the reply text corresponding to the current sentence text information; during the TTS audio broadcast, simultaneously perform a sentence attribution operation and TTS audio broadcast on the next sentence text information based on the next request ID.

2. The audio interaction method based on the generative language model of the dialogue control module according to claim 1 is characterized in that: The sentence text information includes: a sentence timestamp and a sentence ID; and the process of performing a sentence attribution operation on the sentence text information based on the request ID includes: All sentence text information of the same round containing the same request ID is extracted, and a sentence attribution operation is performed based on the sentence timestamps and sentence IDs of all sentence text information of the same round to generate a reply text corresponding to the current sentence text information.

3. The audio interaction method based on the generative language model of the dialogue control module according to claim 2 is characterized in that: The method further comprises: In response to the interruption instruction input by the user, the TTS audio broadcast of the current round is stopped, and the remaining sentence text information of the current round sent back by the generative language model is continued to be received, and the sentence attribution operation is performed on the remaining sentence text information of the current round to generate a reply text corresponding to the current sentence text information, and the reply text of the current sentence text information is added to the interaction log; Receiving the next round of task request information input by the user again, sending the next round of task request information to the generative language model, and establishing a unique request ID corresponding to the next round of task request information; Receive multiple sentence text information corresponding to the next round of task request information sent back by the generative language model and perform sentence attribution operations to generate the next round of reply text, perform TTS audio broadcast on the next round of reply text, and add the next round of information to the interaction log.

4. The audio interaction method based on the generative language model of the dialogue control module according to claim 2 is characterized in that: The method further comprises: In response to the rollback instruction input by the user, the TTS audio broadcast of the current round is stopped, and the remaining sentence text information of the current round sent back by the generative language model is continued to be received, and the valid information sentence before the rollback instruction is issued is extracted according to the sentence timestamp of the remaining sentence text information of the current round; Receive the next round of task request information input by the user, perform a merging operation on the current round of task request information, the next round of task request information and the valid information clause to generate fallback request information, send the fallback request information to the generative language model, and establish a unique request ID corresponding to the fallback request information; Receive multiple sentence text information corresponding to the fallback request information sent back by the generative language model; perform sentence attribution operation to generate the next round of reply text, perform TTS audio broadcast on the next round of reply text, and add the next round of information to the interaction log.

5. The audio interaction method based on the generative language model of the dialogue control module according to claim 1 is characterized in that: The method further includes: if it is detected that the plurality of sentence text information sent back by the generative language model contains a preset end condition, ending the task request of the current round, and adding the task request information in the current round and the reply text to the interaction log.

6. The audio interaction method based on the generative language model of the dialogue control module according to claim 1 is characterized in that: The process of receiving the plurality of sentence text information sent back by the generative language model further includes: If the generative language model does not return the next frame of sentence text information within a preset time after returning the first frame of sentence text information, it is determined that the generative language model is abnormal, and the preset fallback speech is broadcast and the call is hung up; And / or, if the generative language model does not send back the first frame sentence text information within a preset time, it is determined that the generative language model is abnormal, a preset fallback speech is broadcasted, and the reply text is no longer broadcasted; And / or, if any sentence text information returned by the generative language model contains a speech synthesis markup language error, the model is determined to be abnormal, the preset fallback words are broadcast, and the reply text is no longer broadcast.

7. An audio interaction device based on a generative language model of a dialogue control module, characterized in that: The device comprises: The task request module is used to receive the task request information input by the user, establish a unique request ID corresponding to the task request information, and send the task request information to the generative language model, so that the generative language model generates a plurality of corresponding sentence text information based on the task request information; Interaction management module: used for receiving multiple sentence text information sent back by the generative language model; and performing the following operations on the sentence text information one by one in the order of receiving: performing sentence attribution operation on the current sentence text information based on the request ID to generate a reply text corresponding to the current sentence text information, and performing TTS audio broadcast on the reply text corresponding to the current sentence text information; during the TTS audio broadcast, performing sentence attribution operation and TTS audio broadcast on the next sentence text information based on the request ID; until the received sentence text information contains a termination mark; receiving the next round of task request information again, establishing A next request ID corresponding to the next round of task request information is generated, and the next round of task request information is sent to the generative language model; multiple sentence text information of the next round sent back by the generative language model is received; the following operations are performed on the sentence text information one by one in the order of reception: a sentence attribution operation is performed on the current sentence text information based on the next request ID to generate a reply text corresponding to the current sentence text information, and a TTS audio broadcast is performed on the reply text corresponding to the current sentence text information; during the TTS audio broadcast, a sentence attribution operation and a TTS audio broadcast are performed on the next sentence text information based on the next request ID.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the audio interaction method based on the generative language model of the dialogue control module as described in any one of claims 1 to 6 is implemented.

9. A computer program product, characterized in that The computer program product includes computer program code, and when the computer program code is executed on a computer, the computer implements the audio interaction method based on a generative language model of a dialogue control module as claimed in any one of claims 1 to 6.

10. An electronic terminal / electronic terminal, comprising a memory, a processor and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the audio interaction method based on the generative language model of the dialogue control module as described in any one of claims 1 to 6.

Images