An artificial intelligence-based voice interaction method

Abstract

The application relates to the technical field of voice interaction, and discloses a voice interaction method based on artificial intelligence, which comprises the following steps: acquiring a first voice input of a user and converting the first voice input into a first text, inputting the first text into an artificial intelligence dialogue model to generate a reply text, performing semantic segmentation on the reply text to obtain a plurality of reply semantic segments, converting the reply semantic segments into voice segments, and establishing a mapping relationship between the voice segments and a playing time interval; detecting a second voice input of the user in the process of playing the voice segments, acquiring a breaking time point, determining a target reply semantic segment according to the breaking time point and the mapping relationship; converting the second voice input of the user into a second text, determining a breaking type according to the second text and the target reply semantic segment, generating a continuation interaction strategy according to the breaking type, and outputting a subsequent voice reply. The application can perform associated processing on the voice inserted by the user in the voice playing process, and reduces repeated output and error processing conditions.

Description

Technical Field

[0001] This invention relates to the field of voice interaction technology, specifically to a voice interaction method based on artificial intelligence. Background Technology

[0002] With the development of speech recognition technology and artificial intelligence dialogue models, voice interaction devices are being applied to smart speakers, in-vehicle systems, mobile terminals, and intelligent customer service systems. Existing voice interaction methods typically acquire user voice input, convert the speech into text, generate response text based on an artificial intelligence dialogue model, and then convert the response text back into speech output to complete the voice interaction process.

[0003] In the aforementioned voice interaction process, when the system plays a long voice reply, the user inserts a voice message during the playback. Existing technologies typically only re-identify the user's inserted voice message and directly generate a new reply, without considering the current voice playback progress and the reply content. This results in the system being unable to accurately determine the original reply content corresponding to the user's inserted voice message, which can easily lead to duplicate output, content jump errors, or unplayed content being incorrectly discarded. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention provides an artificial intelligence-based voice interaction method that solves the problem in existing technologies where users cannot accurately associate their voice input with the original response content during voice response playback.

[0005] To achieve the above objectives, the present invention provides the following technical solution: a voice interaction method based on artificial intelligence, comprising: Obtain the user's first voice input and convert the user's first voice input into first text; Input the first text into the AI ​​dialogue model to generate a response text; The response text is semantically segmented to obtain multiple response semantic fragments, and a fragment identifier and semantic features are generated for each response semantic fragment; The multiple response semantic segments are converted into corresponding speech segments, and a mapping relationship is established between each response semantic segment and the playback time interval of the corresponding speech segment; During the playback of the audio segment, in response to the detection of the user's second voice input, the interruption time point corresponding to the user's second voice input is obtained; Based on the interruption time point and the mapping relationship, determine the target response semantic fragment corresponding to the interruption time point; The user's second voice input is converted into second text, and the interruption type is determined based on the second text and the target response semantic fragment; A continuation interaction strategy is generated based on the interruption type, and subsequent voice responses are output according to the continuation interaction strategy.

[0006] Preferably, the semantic segmentation of the reply text yields multiple reply semantic segments, including: The reply text is divided into multiple initial text segments based on at least one of sentence boundaries, topic change locations, and entity association relationships in the reply text. Multiple initial text fragments are merged or split to obtain multiple response semantic fragments.

[0007] Preferably, the semantic features include at least one of topic tags, keywords, entity information, importance level, contextual summary, and semantic vector.

[0008] Preferably, establishing the mapping relationship between each of the response semantic segments and the corresponding playback time interval of the speech segment includes: Obtain the playback start time and playback end time for each of the aforementioned audio segments; The playback time interval for each audio segment is determined based on the playback start time and the playback end time; The segment identifier of each response semantic segment is bound to the playback time interval of the corresponding audio segment to obtain the mapping relationship.

[0009] Preferably, determining the target response semantic fragment corresponding to the interruption time point based on the interruption time point and the mapping relationship includes: When the interruption time point is within the playback time interval of any of the audio segments, the response semantic segment corresponding to the playback time interval is determined as the target response semantic segment; When the interruption time point is located between the playback time intervals of two adjacent audio segments, the target response semantic segment is determined based on the semantic similarity between the second text and the response semantic segments corresponding to the two adjacent audio segments.

[0010] Preferably, after determining the target response semantic fragment corresponding to the interruption time point based on the interruption time point and the mapping relationship, the method further includes: Based on the interruption time point and the mapping relationship, determine the set of played semantic segments, the currently played semantic segments, and the set of unplayed semantic segments; The set of played semantic segments includes response semantic segments whose playback time interval is earlier than the interruption time point; the set of currently played semantic segments includes response semantic segments whose playback time interval includes the interruption time point; and the set of unplayed semantic segments includes response semantic segments whose playback time interval is later than the interruption time point.

[0011] Preferably, the interruption type includes at least one of the following: follow-up question interruption, error correction interruption, topic switching interruption, confirmation interruption, and termination interruption.

[0012] Preferably, the step of generating a continuation interaction strategy based on the interruption type includes: When the interruption type is a follow-up question interruption, a supplementary response strategy is generated based on the second text and the target response semantic fragment; When the interruption type is an error correction interruption, the target response semantic fragment is updated according to the second text, and a re-response strategy is generated; When the interruption type is a topic-switching interruption, a new topic response strategy is generated based on the second text; When the interruption type is a confirmation interruption, a skip reply strategy or a continue reply strategy is generated based on the target reply semantic fragment. When the interruption type is a termination interruption, an end-interaction strategy is generated.

[0013] Preferably, the step of outputting subsequent voice responses according to the continuation interaction strategy includes: When the continuation interaction strategy is the supplementary response strategy, a supplementary voice response for the target response semantic segment is output, and after the supplementary voice response is output, the voice segments corresponding to the set of unplayed semantic segments are output. When the continuation interaction strategy is the re-response strategy, the corrected voice response generated based on the updated target response semantic fragment is output. When the continuation interaction strategy is the new topic reply strategy, stop outputting the audio segments corresponding to the unplayed semantic segment set, and output the new topic audio reply generated based on the second text; When the continuation interaction strategy is the termination interaction strategy, the output of the voice segment is stopped.

[0014] Preferably, an artificial intelligence-based voice interaction device, according to the above-described artificial intelligence-based voice interaction method, includes: A speech recognition module is configured to acquire a user's first speech input and convert the user's first speech input into first text; and to acquire, in response to detecting a user's second speech input during the playback of a speech segment, acquire the interruption time point corresponding to the user's second speech input and convert the user's second speech input into second text. The response generation module is used to input the first text into the artificial intelligence dialogue model and generate response text. The semantic segmentation module is used to perform semantic segmentation on the response text to obtain multiple response semantic fragments, and generate a fragment identifier and semantic features for each response semantic fragment; The playback mapping module is used to convert the multiple response semantic segments into corresponding speech segments respectively, and to establish a mapping relationship between the playback time interval of each response semantic segment and the corresponding speech segment; The interruption and continuation module is used to determine the target response semantic fragment corresponding to the interruption time point based on the interruption time point and the mapping relationship, determine the interruption type based on the second text and the target response semantic fragment, generate a continuation interaction strategy based on the interruption type, and output the subsequent voice response according to the continuation interaction strategy.

[0015] This invention provides a voice interaction method based on artificial intelligence. It has the following beneficial effects: 1. This invention establishes a mapping relationship between the playback time interval of the response semantic segment and the voice segment, and determines the target response semantic segment based on the interruption time point when the user's second voice input is detected, so that the user's second voice input can be associated with the corresponding response semantic segment, thereby reducing the repeated output of played content and the incorrect processing of unplayed content.

[0016] 2. This invention segments the response text semantically and generates multiple response semantic fragments. In subsequent processing, the response semantic fragments are used as the smallest processing unit, so that each segment of the response content has a clear semantic boundary during voice interaction. Thus, when generating a continuation interaction strategy, it can be processed based on the target response semantic fragment, avoiding the processing errors caused by reconstructing the entire response text.

[0017] 3. This invention determines the interruption type based on the second text and the semantic fragment of the target response, and generates a corresponding continuation interaction strategy based on different interruption types, so that the output process of subsequent voice responses has a clear control path, thereby maintaining the continuity and consistency of the interaction content when the user continuously performs voice interaction. Attached Figure Description

[0018] Figure 1This is a flowchart illustrating an artificial intelligence-based voice interaction method according to the present invention. Detailed Implementation

[0019] The technical solution of the present invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] Please see the appendix Figure 1 This invention provides an artificial intelligence-based voice interaction method, comprising: Obtain the user's first voice input and convert the user's first voice input into first text; Specifically, when the voice interaction device is in voice interaction mode, the audio acquisition unit acquires the audio signal emitted by the user and sends the acquired audio signal to the speech recognition module. The speech recognition module performs voice activity detection on the audio signal, determines the start and end positions of the user's voice in the audio signal, and extracts the user's first voice input based on the start and end positions.

[0021] The speech recognition module preprocesses the user's initial speech input. Preprocessing includes echo cancellation, noise reduction, endpoint detection, and speech enhancement. Echo cancellation removes the playback audio component from the audio playback unit's output from the user's initial speech input. Noise reduction reduces ambient noise components in the user's initial speech input. Endpoint detection determines the effective speech range within the user's initial speech input. Speech enhancement adjusts the speech amplitude and spectral distribution within the effective speech range.

[0022] The speech recognition module inputs the preprocessed initial user speech input into the speech recognition engine. The speech recognition engine outputs the recognized text, recognition confidence score, word-level timestamps, and a list of candidate texts. The speech recognition module determines the first text from the candidate text list based on the recognition confidence score. The first text includes the text content corresponding to the user's initial speech input. The first text is also stored in association with the start time, end time, and recognition confidence score of the user's initial speech input.

[0023] When multiple candidate texts exist in the candidate text list, the speech recognition module arranges them in descending order of recognition confidence and designates the candidate text with the highest confidence as the first text. If the candidate text with the highest confidence contains an entity word from a preset business vocabulary, the speech recognition module retains the recognition result for that entity word. If the candidate text with the highest confidence contains a near-homophone that does not match the preset business vocabulary, the speech recognition module replaces the near-homophone according to the preset business vocabulary and designates the replaced text as the first text.

[0024] Furthermore, the first text is input into the artificial intelligence dialogue model to generate a response text; Specifically, the response generation module receives the first text and inputs it into the AI ​​dialogue model. The input data for the AI ​​dialogue model includes the first text, historical dialogue text, the current session identifier, and the current business scenario identifier. The historical dialogue text represents the interactions preceding the first text in the current session. The current session identifier distinguishes different user sessions. The current business scenario identifier limits the scope of business knowledge invoked by the AI ​​dialogue model when generating the response text.

[0025] The AI ​​dialogue model performs semantic parsing on the first text to obtain the user intent, entity parameters, and contextual information corresponding to it. The user intent represents the interaction purpose proposed by the user through their initial voice input. Entity parameters represent the object names, time information, location information, quantity information, and attribute information contained in the first text. Contextual information represents the referential and ellipsis relationships between the first text and previous dialogue texts.

[0026] The AI ​​dialogue model generates response text based on user intent, entity parameters, contextual information, and the current business scenario identifier. The response text is natural language text and includes at least one complete sentence. The sentences in the response text are arranged in the output order of the response content. The response generation module sends the response text to the semantic segmentation module and associates the response text with the current session identifier for storage.

[0027] When the AI ​​dialogue model generates multiple candidate response texts, the response generation module determines one candidate response text as the actual response text based on the semantic matching result between the candidate response text and the first text, the entity integrity result in the candidate response text, and the content constraint detection result of the candidate response text. The content constraint detection result indicates whether the candidate response text contains prohibited output content corresponding to the current business scenario identifier. The candidate response text determined as the actual response text is then sent to the semantic segmentation module.

[0028] Furthermore, the response text is semantically segmented to obtain multiple response semantic fragments, and a fragment identifier and semantic features are generated for each response semantic fragment; Semantic segmentation of the response text yields multiple semantic fragments, including: The response text is divided into multiple initial text fragments based on at least one of the following: sentence boundaries, topic change locations, and entity relationships. Multiple initial text fragments are merged or split to obtain multiple response semantic fragments.

[0029] Semantic features include at least one of topic tags, keywords, entity information, importance, contextual summaries, and semantic vectors.

[0030] Specifically, after receiving the response text, the semantic segmentation module identifies sentence boundaries. Sentence boundaries are determined by punctuation marks, pause markers, and grammatical boundaries in the response text. The semantic segmentation module then divides the response text into multiple initial text segments based on these sentence boundaries. Each initial text segment includes one or more semantically continuous sentences.

[0031] The semantic segmentation module performs topic identification on multiple initial text segments. The topic identification result is used to represent the topic label corresponding to each initial text segment. The semantic segmentation module determines the topic change position according to the position of the topic label change. When two adjacent initial text segments have different topic labels and there is no identical entity information between the two adjacent initial text segments, the semantic segmentation module determines the position between the two adjacent initial text segments as the topic change position.

[0032] The semantic segmentation module performs entity recognition on multiple initial text segments to obtain entity information for each segment. Entity information includes at least one of the following: object name, model name, function name, location name, time name, and numerical name. Based on this entity information, the semantic segmentation module determines the entity relationships between adjacent initial text segments. When two adjacent initial text segments contain the same entity information, or when a pronoun in the later initial text segment refers to entity information in the earlier initial text segment, the semantic segmentation module determines that an entity relationship exists between the two adjacent initial text segments.

[0033] The semantic segmentation module merges or splits multiple initial text segments based on sentence boundaries, topic change locations, and entity relationships. When two adjacent initial text segments have the same topic tag and there is an entity relationship between them, the semantic segmentation module merges them into a single response semantic segment. When an initial text segment contains more than two topic tags, and there is no entity relationship between the text content corresponding to different topic tags, the semantic segmentation module splits the initial text segment into two or more response semantic segments.

[0034] The semantic segmentation module generates segment identifiers for multiple semantic segments in the response text, following the text order within the response text. Each segment identifier includes the current session identifier and a segment number. The segment numbers increment sequentially according to the order in which the semantic segments are arranged in the response text. Different semantic segments have different segment identifiers.

[0035] The semantic segmentation module generates semantic features for each response semantic fragment. Semantic features include at least one of the following: topic tags, keywords, entity information, importance, context summary, and semantic vector. Topic tags are determined by topic recognition results. Keywords are determined by matching words from nouns, verbs, attribute words, and business terms in the response semantic fragment. Entity information is determined by entity recognition results. Importance is determined by the position of the response semantic fragment in the response text, whether it contains entity information, and whether it contains content corresponding to the user's intent. The context summary is generated from the text content of the current response semantic fragment, the text content of the previous response semantic fragment, and the text content of the next response semantic fragment. The semantic vector is generated by the vector generation model based on the text content of the response semantic fragment.

[0036] The semantic segmentation module associates and stores each response semantic segment, its segment identifier, and its semantic features. Multiple response semantic segments are arranged into a response semantic segment sequence according to their segment numbers. The semantic segmentation module sends the response semantic segment sequence to the playback mapping module and also to the interruption and continuation module.

[0037] Furthermore, multiple response semantic segments are converted into corresponding speech segments, and a mapping relationship is established between the playback time interval of each response semantic segment and the corresponding speech segment. Establish a mapping relationship between each response semantic segment and the playback time interval of the corresponding audio segment, including: Obtain the start and end times of playback for each audio segment; The playback time interval for each audio segment is determined based on the start and end times of playback. The segment identifier of each response semantic segment is bound to the playback time interval of the corresponding audio segment to obtain a mapping relationship.

[0038] Specifically, after receiving the sequence of response semantic segments, the playback mapping module sends multiple response semantic segments to the speech synthesis unit. The speech synthesis unit performs speech synthesis on each response semantic segment to obtain a speech segment corresponding to each response semantic segment. Each speech segment corresponds to a segment identifier. The speech segments are added to the playback queue according to their segment numbers.

[0039] The playback mapping module obtains the audio duration of each speech segment. The audio duration is determined by the sampling rate, number of sampling points, and number of channels of the speech segment. The playback mapping module determines the start and end times of each speech segment according to the order of the speech segments in the playback queue. The start time of the first speech segment is the time when the response voice begins playing. The end time of each speech segment is determined by its start time and audio duration. When a playback interval is set between two adjacent speech segments, the start time of the later speech segment is determined by the end time of the previous speech segment and the playback interval.

[0040] The playback mapping module determines the playback time interval for each audio segment based on its start and end times. The playback time interval includes both the start and end times. The module then binds the segment identifier of each response semantic segment to the corresponding playback time interval, thus establishing a mapping relationship.

[0041] The mapping relationship is stored as a set of records. Each record in the mapping relationship includes the current session identifier, segment identifier, text content of the response semantic segment, semantic features, speech segment identifier, playback start time, playback end time, and playback time interval. When the audio playback unit plays a speech segment, the playback mapping module sends the mapping relationship to the interruption and resumption module.

[0042] When the audio playback unit starts playing a speech segment, the playback mapping module records the actual playback start time. When the audio playback unit finishes playing the speech segment, the playback mapping module records the actual playback end time. If the actual playback start time and actual playback end time are inconsistent with the predetermined playback start time and playback end time, the playback mapping module updates the playback time interval with the actual playback start time and actual playback end time, and updates the mapping relationship according to the updated playback time interval.

[0043] Furthermore, during the playback of the audio segment, in response to the detection of the user's second voice input, the interruption time point corresponding to the user's second voice input is obtained; Specifically, during the playback of a speech segment, the audio acquisition unit continuously acquires ambient audio and sends it to the speech recognition module. The speech recognition module receives the playback reference signal corresponding to the currently output speech segment from the audio playback unit and performs echo cancellation processing on the ambient audio. The echo-cancelled ambient audio is then used to detect the user's second voice input.

[0044] The speech recognition module detects speech activity in the echo-cancelled ambient audio. When the energy value of consecutive speech frames in the ambient audio reaches a preset energy threshold, and the duration of the consecutive speech frames reaches a preset duration threshold, the speech recognition module determines that the user's second speech input has been detected. The speech recognition module determines the start time of the consecutive speech frames that meet the above conditions as the start time of the user's second speech input.

[0045] The speech recognition module obtains the corresponding interruption time point based on the start time of the user's second voice input. The interruption time point is the position of the start time of the user's second voice input on the current response voice playback timeline. The current response voice playback timeline starts from the start time of the first voice segment. The speech recognition module sends the interruption time point to the interruption continuation module.

[0046] The speech recognition module intercepts the user's second voice input and performs speech recognition on it to obtain the second text. The second text includes the text content corresponding to the user's second voice input. The speech recognition module associates the second text with the interruption time point and stores it, then sends the second text to the interruption and continuation module.

[0047] Furthermore, based on the interruption time point and mapping relationship, the target response semantic fragment corresponding to the interruption time point is determined; Based on the interruption time point and mapping relationship, determine the target response semantic fragment corresponding to the interruption time point, including: When the interruption point is within the playback time interval of any audio segment, the response semantic segment corresponding to the playback time interval is determined as the target response semantic segment; When the interruption point is located between the playback time intervals of two adjacent audio segments, the target response semantic segment is determined based on the semantic similarity between the second text and the response semantic segments corresponding to the two adjacent audio segments.

[0048] Specifically, the interruption and continuation module receives the interruption time point, the second text, and the mapping relationship. The interruption and continuation module queries the mapping relationship according to the playback time interval to determine the target response semantic fragment corresponding to the interruption time point.

[0049] When the interruption point falls within the playback time interval of any audio segment, the interruption and continuation module determines the corresponding response semantic segment as the target response semantic segment. The interruption point falling within the playback time interval means that the interruption point is no earlier than the start time of playback within that time interval and no later than the end time of playback within that time interval.

[0050] When the interruption point falls between the playback time intervals of two adjacent audio segments, the interruption and continuation module obtains the corresponding response semantic segments for each of the two adjacent audio segments. The interruption and continuation module converts the second text into a second text semantic vector and calculates the semantic similarity between the second text semantic vector and the semantic vectors of the two adjacent response semantic segments. The interruption and continuation module identifies the response semantic segment with the higher semantic similarity as the target response semantic segment. When two semantic similarities are equal, the interruption and continuation module identifies the response semantic segment that appears earlier in the sequence as the target response semantic segment.

[0051] When the interruption time is earlier than the start time of the first audio segment, the interruption and continuation module determines the response semantic segment corresponding to the first audio segment as the target response semantic segment. When the interruption time is later than the end time of the last audio segment, the interruption and continuation module determines the response semantic segment corresponding to the last audio segment as the target response semantic segment.

[0052] Furthermore, after determining the target response semantic fragment corresponding to the interruption time point based on the interruption time point and mapping relationship, the process also includes: Based on the interruption time and mapping relationship, determine the set of played semantic segments, the currently playing semantic segments, and the set of unplayed semantic segments; The set of played semantic segments includes response semantic segments whose playback time interval is earlier than the interruption time point; the set of currently played semantic segments includes response semantic segments whose playback time interval includes the interruption time point; and the set of unplayed semantic segments includes response semantic segments whose playback time interval is later than the interruption time point.

[0053] Specifically, after determining the target response semantic segment, the interruption and continuation module determines the set of played semantic segments, the currently playing semantic segments, and the set of unplayed semantic segments based on the interruption time and mapping relationship. The set of played semantic segments includes response semantic segments corresponding to audio segments whose playback end time is no later than the interruption time. The currently playing semantic segments include response semantic segments corresponding to audio segments whose playback time interval includes the interruption time. The set of unplayed semantic segments includes response semantic segments corresponding to audio segments whose playback start time is later than the interruption time.

[0054] When the interruption point is located between the playback time intervals of two adjacent audio segments, the currently playing semantic segment is set to null. The set of played semantic segments includes the response semantic segments corresponding to audio segments whose playback end time is earlier than the interruption point, and the set of unplayed semantic segments includes the response semantic segments corresponding to audio segments whose playback start time is later than the interruption point.

[0055] The interruption and resumption module associates and stores the set of played semantic segments, the currently playing semantic segment, the set of unplayed semantic segments, and the target response semantic segment. The set of played semantic segments is used to determine the content that has already been output. The currently playing semantic segment is used to determine whether the user's second voice input is directed at the currently playing content. The set of unplayed semantic segments is used to determine whether subsequent output should continue, be skipped, or stopped.

[0056] Furthermore, based on the interruption time point and mapping relationship, the target response semantic fragment corresponding to the interruption time point is determined; Interruption types include at least one of the following: follow-up question interruption, error correction interruption, topic switching interruption, confirmation interruption, and termination interruption.

[0057] Specifically, the interruption and continuation module determines the interruption type based on the second text and the target response semantic fragment. The interruption and continuation module constructs interruption type judgment data. This data includes the second text, the semantic vector of the second text, the text content of the target response semantic fragment, the semantic features of the target response semantic fragment, the set of played semantic fragments, the set of currently played semantic fragments, and the set of unplayed semantic fragments.

[0058] The interruption and continuation module determines whether the interruption is a follow-up question based on the interrogative words and questioning phrases in the second text and the semantic similarity between the second text and the target response semantic fragment. When the second text contains interrogative words or questioning phrases, and the semantic similarity between the second text and the target response semantic fragment reaches the first semantic similarity threshold, the interruption and continuation module determines the interruption type as a follow-up question interruption.

[0059] The interruption and continuation module determines whether the interruption is error-correction based on the negative words, correction words, and replacement entities in the second text and the entity information in the target response semantic fragment. When the second text contains negative words or correction words, and the replacement entities in the second text belong to the same entity category as the entity information in the target response semantic fragment, the interruption and continuation module determines the interruption type as error-correction.

[0060] The interruption and continuation module determines whether the interruption is a topic-switching interruption based on the semantic similarity between the second text and the target response semantic segment, the semantic similarity between the second text and the set of unplayed semantic segments, and the new task intent in the second text. When the semantic similarity between the second text and the target response semantic segment is lower than a second semantic similarity threshold, the semantic similarity between the second text and each response semantic segment in the set of unplayed semantic segments is lower than the second semantic similarity threshold, and the second text contains a new task intent, the interruption and continuation module determines the interruption type as a topic-switching interruption.

[0061] The interruption and continuation module determines whether an interruption is an acknowledgment interruption based on the confirmation words, continuation words, skipped words, and target response semantic fragments in the second text. When the second text contains both an acknowledgment word and a continuation word or a skipped word, the interruption and continuation module classifies the interruption as an acknowledgment interruption.

[0062] The interruption and continuation module determines whether the interruption is a termination type based on the stop words in the second text and the new task intent in the second text. When the second text contains a stop word and does not contain a new task intent, the interruption and continuation module determines the interruption type as a termination type.

[0063] When the second text simultaneously meets the judgment conditions for multiple interruption types, the interruption and continuation module determines an interruption type according to a preset priority. The preset priorities are, in order: termination interruption, topic switching interruption, error correction interruption, follow-up question interruption, and confirmation interruption. The interruption and continuation module associates and stores the determined interruption type with the interruption time point, the second text, and the target response semantic fragment.

[0064] Furthermore, the user's second voice input is converted into second text, and the interruption type is determined based on the second text and the semantic fragment of the target response; Generate a continuation interaction strategy based on the interruption type, including: When the interruption type is a follow-up question interruption, a supplementary response strategy is generated based on the second text and the semantic fragment of the target response. When the interruption type is error correction interruption, the semantic fragment of the target response is updated based on the second text, and a new response strategy is generated; When the interruption type is a topic-switching interruption, a new topic response strategy is generated based on the second text; When the interruption type is a confirmation interruption, a skip reply strategy or a continue reply strategy is generated based on the target reply semantic fragment. When the interruption type is a termination interruption, an end interaction strategy is generated.

[0065] Specifically, the interruption and continuation module generates a continuation interaction strategy based on the interruption type. The continuation interaction strategy includes a strategy type, a processing object, input data, output content, and playback control instructions. The strategy type represents a supplementary response strategy, a re-response strategy, a new topic response strategy, a skip response strategy, a continue response strategy, or an end-of-interaction strategy. The processing object represents the target response semantic segment, the set of unplayed semantic segments, or the current session. The input data represents the second text, the semantic features of the target response semantic segment, the set of played semantic segments, and the set of unplayed semantic segments. The output content represents the text content corresponding to the subsequent voice response. Playback control instructions control the audio playback unit to continue playback, stop playback, skip playback, or replay playback.

[0066] When the interruption type is a follow-up question, the interruption continuation module generates a supplementary response strategy based on the second text and the target response semantic fragment. The supplementary response strategy processes the target response semantic fragment. The input data for the supplementary response strategy includes the second text, the text content of the target response semantic fragment, the topic tags of the target response semantic fragment, the keywords of the target response semantic fragment, and the entity information of the target response semantic fragment. The output of the supplementary response strategy is supplementary explanatory text for the target response semantic fragment. The playback control command for the supplementary response strategy is to pause the current playback queue, output the supplementary voice response, and continue outputting the voice fragments corresponding to the unplayed semantic fragment set after the supplementary voice response is output.

[0067] When the interruption type is error correction, the interruption and continuation module updates the target response semantic fragment based on the second text and generates a re-response strategy. The interruption and continuation module extracts replacement entities and correction content from the second text and writes them into the semantic features of the target response semantic fragment. The re-response strategy processes the updated target response semantic fragment. The output of the re-response strategy is the correction description text generated based on the updated target response semantic fragment. The playback control command for the re-response strategy is to stop outputting audio fragments related to the target response semantic fragment in the original playback queue and output the corrected audio response.

[0068] When the interruption type is a topic-switching interruption, the interruption continuation module generates a new topic response strategy based on the second text. The processing object of the new topic response strategy is the new task intent corresponding to the second text. The input data of the new topic response strategy includes the second text and the current session identifier. The output content of the new topic response strategy is the new topic response text generated based on the second text. The playback control command of the new topic response strategy is to stop outputting the audio segments corresponding to the unplayed semantic segment set and output the new topic audio response.

[0069] When the interruption type is a confirmation interruption, the interruption continuation module generates a skip response strategy or a continue response strategy based on the second text. If the second text contains a skip word, the interruption continuation module generates a skip response strategy. The playback control instruction for the skip response strategy is to skip the remaining audio content corresponding to the target response semantic segment and output the corresponding audio segment starting from the first-order response semantic segment in the unplayed semantic segment set. If the second text contains a continue word, the interruption continuation module generates a continue response strategy. The playback control instruction for the continue response strategy is to continue outputting the audio segment after the interruption time point.

[0070] When the interruption type is a termination interruption, the interruption continuation module generates an end-interaction policy. The end-interaction policy processes the current reply text in the current session. The playback control command for the end-interaction policy is to stop outputting the audio segment and clear the playback queue corresponding to the current reply text.

[0071] Furthermore, a continuation interaction strategy is generated based on the interruption type, and subsequent voice responses are output according to the continuation interaction strategy.

[0072] The system will output subsequent voice responses according to the follow-up interaction strategy, including: When the follow-up interaction strategy is the supplementary response strategy, the supplementary voice response for the target response semantic segment is output, and after the supplementary voice response is output, the voice segments corresponding to the set of unplayed semantic segments are output. When the continuation interaction strategy is the re-response strategy, the output is a corrected voice response generated based on the updated target response semantic fragment; When the follow-up interaction strategy is the new topic reply strategy, stop outputting the audio segments corresponding to the unplayed semantic segments set, and output the new topic audio reply generated based on the second text; When the continuation interaction strategy is the end interaction strategy, the output of voice segments will stop.

[0073] Specifically, the interruption and continuation module outputs subsequent voice responses according to the continuation interaction strategy. Subsequent voice responses include at least one of the following: supplementary voice responses, corrected voice responses, voice responses on new topics, and voice segments output according to the continuation response strategy.

[0074] When the continuation interaction strategy is a supplementary response strategy, the interruption continuation module sends the supplementary explanatory text in the supplementary response strategy to the speech synthesis unit. The speech synthesis unit converts the supplementary explanatory text into a supplementary speech response. The audio playback unit outputs the supplementary speech response. After the supplementary speech response is output, the playback mapping module continues to send the speech segments corresponding to the unplayed semantic segments set to the audio playback unit according to the segment number of each response semantic segment in the unplayed semantic segment set. The playback mapping module updates the playback start time, playback end time, and playback time interval of the speech segments corresponding to the unplayed semantic segment set according to the actual playback duration of the supplementary speech response, and updates the mapping relationship.

[0075] When the continuation interaction strategy is a re-response strategy, the interruption and continuation module sends the correction description text to the speech synthesis unit. The speech synthesis unit converts the correction description text into a corrected speech response. The audio playback unit stops outputting speech segments related to the target response semantic segment in the original playback queue and outputs the corrected speech response. After the corrected speech response is output, the interruption and continuation module determines whether to continue outputting the speech segments corresponding to the unplayed semantic segment set based on the playback control instructions of the re-response strategy. If continued output is required, the playback mapping module updates the playback time interval and mapping relationship of the speech segments corresponding to the unplayed semantic segment set based on the actual playback duration of the corrected speech response.

[0076] When the follow-up interaction strategy is a new topic response strategy, the audio playback unit stops outputting the speech segments corresponding to the unplayed semantic segment set. The response generation module inputs the second text into the AI ​​dialogue model to generate the new topic response text. The semantic segmentation module performs semantic segmentation on the new topic response text, obtaining multiple response semantic segments corresponding to the new topic. The playback mapping module converts the multiple response semantic segments corresponding to the new topic into corresponding speech segments and establishes a mapping relationship for the new topic. The audio playback unit outputs the new topic speech response.

[0077] When the follow-up interaction strategy is to skip the response, the audio playback unit stops outputting the remaining audio content corresponding to the target response semantic segment. The playback mapping module starts from the response semantic segment that appears first in the unplayed semantic segment set and continues to output the corresponding audio segments according to the segment number. The playback mapping module updates the playback time interval and mapping relationship of the audio segments corresponding to the unplayed semantic segment set according to the actual playback order.

[0078] When the continuation interaction strategy is a continue-response strategy, the audio playback unit continues outputting speech segments from the interruption point. The playback mapping module updates the playback time interval of the speech segment corresponding to the currently playing semantic segment based on the actual playback start time when output resumes, and updates the mapping relationship accordingly.

[0079] When the continuation interaction strategy is set to end, the audio playback unit stops outputting audio segments. The interruption and continuation module sets the playback queue corresponding to the current reply text to the end state and stops sending the audio segment corresponding to the current reply text to the audio playback unit. The current session identifier, the set of played semantic segments, the currently playing semantic segment, the set of unplayed semantic segments, the interruption time point, the second text, the interruption type, and the continuation interaction strategy are stored together for current session recording.

[0080] An artificial intelligence-based voice interaction device, according to the aforementioned artificial intelligence-based voice interaction method, includes: The speech recognition module is used to acquire the user's first speech input and convert the user's first speech input into first text; and during the playback of a speech segment, in response to the detection of the user's second speech input, acquire the interruption time point corresponding to the user's second speech input and convert the user's second speech input into second text. The response generation module is used to input the first text into the artificial intelligence dialogue model and generate response text. The semantic segmentation module is used to perform semantic segmentation on the response text, obtain multiple response semantic fragments, and generate fragment identifiers and semantic features for each response semantic fragment; The playback mapping module is used to convert multiple response semantic segments into corresponding speech segments and establish a mapping relationship between the playback time interval of each response semantic segment and the corresponding speech segment. The interruption and continuation module is used to determine the target response semantic fragment corresponding to the interruption time point based on the interruption time point and mapping relationship, determine the interruption type based on the second text and the target response semantic fragment, generate a continuation interaction strategy based on the interruption type, and output the subsequent voice response according to the continuation interaction strategy.

[0081] Specifically, this invention also provides an artificial intelligence-based voice interaction device. This device is used to execute the aforementioned artificial intelligence-based voice interaction method. The device includes a voice recognition module, a response generation module, a semantic segmentation module, a playback mapping module, and an interruption / resumption module. The modules are connected via a data interface, which is used to transmit text data, semantic features, voice segments, playback time intervals, mapping relationships, interruption time points, interruption types, resumption interaction strategies, and subsequent voice responses.

[0082] In one embodiment, an AI-based voice interaction device is incorporated into a voice interaction device. The voice interaction device includes a processor, a memory, an audio acquisition unit, an audio playback unit, and a communication interface. The processor is electrically connected to the memory, the audio acquisition unit, the audio playback unit, and the communication interface. The memory stores computer programs, AI dialogue model execution data, response text, response semantic segments, voice segments, playback time intervals, and mapping relationships. The processor executes the computer programs stored in the memory to implement an AI-based voice interaction method. The audio acquisition unit acquires the user's first and second voice inputs. The audio playback unit plays voice segments and subsequent voice responses. The communication interface communicates with a cloud server, a business system, or an external storage device.

[0083] The speech recognition module is connected to the audio acquisition unit. The speech recognition module receives the user's first speech input from the audio acquisition unit and converts it into first text. The speech recognition module also detects the user's second speech input during the playback of a speech segment, obtains the interruption point corresponding to the second speech input, and converts the second speech input into second text.

[0084] The response generation module is connected to the speech recognition module. The response generation module receives the first text, inputs it into the AI ​​dialogue model, and generates the response text. When the AI ​​dialogue model is set locally on the voice interaction device, the response generation module calls the local model to generate the response text. When the AI ​​dialogue model is set on a cloud server, the response generation module sends the first text to the cloud server through a communication interface and receives the response text returned by the cloud server.

[0085] The semantic segmentation module is connected to the response generation module. The semantic segmentation module receives the response text, performs semantic segmentation on the response text to obtain multiple response semantic fragments, and generates a fragment identifier and semantic features for each response semantic fragment. The multiple response semantic fragments, fragment identifiers, and semantic features output by the semantic segmentation module are sent to the playback mapping module and the interruption / resumption module.

[0086] The playback mapping module is connected to the semantic segmentation module and the audio playback unit. The playback mapping module converts multiple response semantic segments into corresponding speech segments, obtains the start and end times of each speech segment, determines the playback time interval based on the start and end times, and establishes a mapping relationship between each response semantic segment and the corresponding speech segment's playback time interval. The playback mapping module sends the speech segments to the audio playback unit for playback and sends the mapping relationship to the interruption and continuation module.

[0087] The interruption and continuation module is connected to the speech recognition module, semantic segmentation module, and playback mapping module, respectively. The interruption and continuation module receives the interruption time point, the second text, semantic features, and the mapping relationship. Based on the interruption time point and the mapping relationship, the module determines the target response semantic segment corresponding to the interruption time point; determines the interruption type based on the second text and the target response semantic segment; generates a continuation interaction strategy based on the interruption type; and outputs the subsequent speech response according to the continuation interaction strategy.

[0088] In one implementation, the speech recognition module, response generation module, semantic segmentation module, playback mapping module, and interruption / resumption module are all stored in memory as computer programs and executed by the processor. In another implementation, one or more of the speech recognition module, response generation module, semantic segmentation module, playback mapping module, and interruption / resumption module are implemented using an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or a digital signal processor (DSP). These different implementations do not change the data transmission relationships and processing flow between the modules.

[0089] When an AI-based voice interaction device is running, the speech recognition module outputs the first text, the response generation module outputs the response text based on the first text, the semantic segmentation module outputs multiple semantic segments based on the response text, the playback mapping module outputs speech segments and mapping relationships based on the multiple semantic segments, and the interruption and continuation module outputs a continuation interaction strategy and subsequent voice responses based on the interruption time point, the second text, the target response semantic segment, and the interruption type. The above data processing order is consistent with the step order in AI-based voice interaction methods.

[0090] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A voice interaction method based on artificial intelligence, characterized in that, include: Obtain the user's first voice input and convert the user's first voice input into first text; Input the first text into the AI ​​dialogue model to generate a response text; The response text is semantically segmented to obtain multiple response semantic fragments, and a fragment identifier and semantic features are generated for each response semantic fragment; The multiple response semantic segments are converted into corresponding speech segments, and a mapping relationship is established between each response semantic segment and the playback time interval of the corresponding speech segment; During the playback of the audio segment, in response to the detection of the user's second voice input, the interruption time point corresponding to the user's second voice input is obtained; Based on the interruption time point and the mapping relationship, determine the target response semantic fragment corresponding to the interruption time point; The user's second voice input is converted into second text, and the interruption type is determined based on the second text and the target response semantic fragment; A continuation interaction strategy is generated based on the interruption type, and subsequent voice responses are output according to the continuation interaction strategy.

2. The voice interaction method based on artificial intelligence according to claim 1, characterized in that, The semantic segmentation of the reply text yields multiple reply semantic fragments, including: The reply text is divided into multiple initial text segments based on at least one of sentence boundaries, topic change locations, and entity association relationships in the reply text. Multiple initial text fragments are merged or split to obtain multiple response semantic fragments.

3. The voice interaction method based on artificial intelligence according to claim 1, characterized in that, The semantic features include at least one of topic tags, keywords, entity information, importance, contextual summaries, and semantic vectors.

4. The voice interaction method based on artificial intelligence according to claim 1, characterized in that, The step of establishing a mapping relationship between each of the response semantic segments and the corresponding playback time interval of the speech segment includes: Obtain the playback start time and playback end time for each of the aforementioned audio segments; The playback time interval for each audio segment is determined based on the playback start time and the playback end time; The segment identifier of each response semantic segment is bound to the playback time interval of the corresponding audio segment to obtain the mapping relationship.

5. The voice interaction method based on artificial intelligence according to claim 1, characterized in that, The step of determining the target response semantic fragment corresponding to the interruption time point based on the interruption time point and the mapping relationship includes: When the interruption time point is within the playback time interval of any of the audio segments, the response semantic segment corresponding to the playback time interval is determined as the target response semantic segment; When the interruption time point is located between the playback time intervals of two adjacent audio segments, the target response semantic segment is determined based on the semantic similarity between the second text and the response semantic segments corresponding to the two adjacent audio segments.

6. The voice interaction method based on artificial intelligence according to claim 1, characterized in that, After determining the target response semantic fragment corresponding to the interruption time point based on the interruption time point and the mapping relationship, the method further includes: Based on the interruption time point and the mapping relationship, determine the set of played semantic segments, the currently played semantic segments, and the set of unplayed semantic segments; The set of played semantic segments includes response semantic segments whose playback time interval is earlier than the interruption time point; the set of currently played semantic segments includes response semantic segments whose playback time interval includes the interruption time point; and the set of unplayed semantic segments includes response semantic segments whose playback time interval is later than the interruption time point.

7. The voice interaction method based on artificial intelligence according to claim 1, characterized in that, The interruption types include at least one of the following: follow-up question interruption, error correction interruption, topic switching interruption, confirmation interruption, and termination interruption.

8. The voice interaction method based on artificial intelligence according to claim 7, characterized in that, The step of generating a continuation interaction strategy based on the interruption type includes: When the interruption type is a follow-up question interruption, a supplementary response strategy is generated based on the second text and the target response semantic fragment; When the interruption type is an error correction interruption, the target response semantic fragment is updated according to the second text, and a re-response strategy is generated; When the interruption type is a topic-switching interruption, a new topic response strategy is generated based on the second text; When the interruption type is a confirmation interruption, a skip reply strategy or a continue reply strategy is generated based on the target reply semantic fragment. When the interruption type is a termination interruption, an end-interaction strategy is generated.

9. The voice interaction method based on artificial intelligence according to claim 8, characterized in that, The step of outputting subsequent voice responses according to the continuation interaction strategy includes: When the continuation interaction strategy is the supplementary response strategy, a supplementary voice response for the target response semantic segment is output, and after the supplementary voice response is output, the voice segments corresponding to the set of unplayed semantic segments are output. When the continuation interaction strategy is the re-response strategy, the corrected voice response generated based on the updated target response semantic fragment is output. When the continuation interaction strategy is the new topic reply strategy, stop outputting the audio segments corresponding to the unplayed semantic segment set, and output the new topic audio reply generated based on the second text; When the continuation interaction strategy is the termination interaction strategy, the output of the voice segment is stopped.

10. A voice interaction device based on artificial intelligence, characterized in that, A voice interaction method based on artificial intelligence according to any one of claims 1-9, comprising: A speech recognition module is configured to acquire a user's first speech input and convert the user's first speech input into first text; and to acquire, in response to detecting a user's second speech input during the playback of a speech segment, acquire the interruption time point corresponding to the user's second speech input and convert the user's second speech input into second text. The response generation module is used to input the first text into the artificial intelligence dialogue model and generate response text. The semantic segmentation module is used to perform semantic segmentation on the response text to obtain multiple response semantic fragments, and generate a fragment identifier and semantic features for each response semantic fragment; The playback mapping module is used to convert the multiple response semantic segments into corresponding speech segments respectively, and to establish a mapping relationship between the playback time interval of each response semantic segment and the corresponding speech segment; The interruption and continuation module is used to determine the target response semantic fragment corresponding to the interruption time point based on the interruption time point and the mapping relationship, determine the interruption type based on the second text and the target response semantic fragment, generate a continuation interaction strategy based on the interruption type, and output the subsequent voice response according to the continuation interaction strategy.

Images