A method for implementing an intelligent karaoke robot that supports multi-turn dialogue interaction

Abstract

This invention relates to the field of voice data processing technology, specifically to a method for implementing an intelligent karaoke robot that supports multi-turn dialogue interaction. First, a safe mute margin is determined based on the energy exceedance of the pre-read accompaniment waveform data to accurately predict when the microphone will fail during a future climax of the accompaniment. Simultaneously, the overlap of the pinyin sequences is determined based on the global alignment similarity of the pinyin sequences between the song versions matched in the first round of instructions. Based on this, the expected playback duration of the rhetorical question is dynamically deduced in conjunction with the candidate song names, effectively quantifying the effect of the ease of confusion of option pronunciation on the machine's rhetorical questioning time. Furthermore, the safe mute margin, representing the future level of environmental quietness, is compared with the expected playback duration of the rhetorical question, representing the required dialogue time cost, and then the corresponding karaoke robot interaction control is executed, resulting in a higher success rate and reliability for the karaoke robot's dialogue-based song selection interaction.

Description

Technical Field

[0001] This invention relates to the field of voice data processing technology, and specifically to a method for implementing an intelligent karaoke robot that supports multi-turn dialogue interaction. Background Technology

[0002] In high-sound-pressure environments such as KTVs, intelligent karaoke systems often experience multi-round dialogue interactions due to the drastic fluctuations in the accompaniment sound, which can severely interfere with the user's voice captured by the microphone array. This can easily mask the user's true commands, leading to interrupted interaction or garbled text recognition. Existing technologies typically employ a fixed timeout window after the user initiates their initial voice command to collect subsequent machine-generated responses and user feedback.

[0003] However, when performing multi-turn dialogues, existing technologies do not take into account the future energy change trends of the accompaniment audio to predict the exact moment when environmental noise will mask the microphone, nor do they consider the effect of the similarity of pronunciation of different candidate song versions on the machine's speaking speed and the total duration of the required questions. They cannot dynamically assess timing conflicts and switch interaction paths in time before the climax of the accompaniment causes the sound to fail. As a result, existing technologies have poor success rates and reliability in conducting dialogue and song selection interactions based on a fixed timeout waiting window. Summary of the Invention

[0004] To address the poor success rate and reliability of existing technologies that rely on fixed timeout windows for multi-turn dialogue-based song selection interactions, this invention aims to provide a method for implementing an intelligent song selection robot that supports multi-turn dialogue interactions. The specific technical solution adopted is as follows: The first aspect of this invention provides a method for implementing an intelligent karaoke robot that supports multi-turn dialogue interaction, comprising: During the accompaniment playback in the karaoke system, the system acquires the user's first voice command and the pre-read accompaniment waveform data after the user's first voice command ends; based on the energy exceedance of the pre-read accompaniment waveform data, it determines the safe mute margin. Based on the semantic information of the user's first voice command and the matching of song names in the local music library database, all song version distinguishing words are determined; based on the global alignment similarity of the pinyin sequences among the song version distinguishing words, the pinyin sequence overlap is determined; based on the pinyin sequence overlap and the corresponding candidate song names, the expected playback duration of the rhetorical question is determined. The interaction control of the karaoke robot is based on the relative size between the safety mute margin and the expected playback duration of the rhetorical question.

[0005] Furthermore, the process of obtaining the safety noise margin includes: In each audio data frame of the pre-read accompaniment waveform data, the corresponding short-time energy value is determined based on the overall size of all discrete digital transient amplitude data; the audio data frame corresponding to the first short-time energy value that is greater than the preset reception failure energy threshold is taken as the noise-affected data frame; and the safe mute margin is determined based on the time interval between the time corresponding to the noise-affected data frame and the time when the user's first round of voice commands ends.

[0006] Furthermore, the process of obtaining the song version distinguishing words includes: The user's initial voice command is converted into voice command text using a voice recognition engine; the voice command text is input into a trained named entity recognition model for sequence labeling to determine the entity label corresponding to each character; based on all characters whose entity labels are song name attributes, the search entity words are determined; based on the search entity words, a matching query is performed in the local music library database to determine all corresponding candidate songs; Based on the names of all candidate songs, pattern matching is performed using regular expressions to extract all song version distinguishing words and remove duplicates. The deduplicated song version distinguishing words are then used as the word segments for all candidate song names.

[0007] Furthermore, the process of obtaining the overlap degree of the pinyin sequence includes: Each candidate song name is segmented and converted into a sequence of pinyin characters using a text-to-pinyin component. Each candidate song name segment is used as the target name segment; the other candidate song names segmented outside the target name segment are used as reference name segmentations. The Pinyin character sequence of the target name segment is aligned with the Pinyin character sequence of each reference name segment based on the Niedermann-Onsch global sequence alignment algorithm to construct a similarity matrix; the elements in the cell at the lower right corner vertex of the similarity matrix are normalized to determine the single alignment score between the target name segment and each reference name segment. Calculate the single alignment scores between all candidate song title word segments, and take the maximum value of all single alignment scores as the pinyin sequence overlap.

[0008] Furthermore, the process of obtaining the expected playback duration of the rhetorical question includes: Based on the vocabulary distinctions of all song versions and the preset rhetorical question dialogue template structure, determine the rhetorical question prompt text sentence; The basic rhetorical question broadcast duration is determined by multiplying the total number of valid Chinese characters in the rhetorical question prompt text sentence by the preset single Chinese character broadcast time; the overlap of the pinyin sequence is positively correlated to determine the speech rate slowing weight. The weighted rhetorical question broadcast duration is determined by multiplying the speech rate slowing weight by the basic rhetorical question broadcast duration. The expected playback duration of the rhetorical question is determined by the sum of the weighted rhetorical question broadcast duration and the preset silent waiting duration.

[0009] Furthermore, the process of controlling the karaoke robot interaction based on the relative size between the safety mute margin and the expected playback duration of the questioning voice includes: If the safety mute margin is greater than or equal to the expected playback duration of the question, then the voice interaction state of the song request robot is maintained. If the safety mute margin is less than the expected playback duration of the question, a mute command is issued to shut down the microphone input channel of the karaoke robot, and the corresponding candidate song name is displayed on the display device for interaction.

[0010] Furthermore, the process of obtaining the short-time energy value includes: The corresponding transient amplitude data is determined by multiplying each discrete digital transient amplitude data with the prior global volume gain coefficient; in each audio data frame of the pre-read accompaniment waveform data, the sum of squares is performed on all transient amplitude data to determine the corresponding short-time energy value.

[0011] Furthermore, the process of obtaining the preset radio failure energy threshold includes: The short-time energy values ​​of all audio data frames in the historical data are statistically analyzed; cluster analysis is performed on the short-time energy values ​​of all audio data frames to determine at least two energy value clusters; the mean of all short-time energy values ​​in each energy value cluster is calculated to determine the corresponding energy reference value; the minimum short-time energy value in the energy cluster with the largest corresponding energy reference value is used as the preset radio reception failure energy threshold.

[0012] Furthermore, the preset single-character broadcast time is set to 0.25 seconds.

[0013] Furthermore, the preset silent waiting time is set to 3 seconds.

[0014] Secondly, the present invention provides a system for implementing an intelligent karaoke robot that supports multi-turn dialogue interaction, the system comprising: The data acquisition and preprocessing module is used to acquire the user's first round of voice commands and the pre-read accompaniment waveform data after the user's first round of voice commands ends during the accompaniment playback process of the karaoke system; and to determine the safe mute margin based on the energy exceedance of the pre-read accompaniment waveform data. The playback duration determination module is used to determine all song version distinguishing words based on the semantic information of the user's first voice command and the matching of song names in the local music library database; determine the pinyin sequence overlap based on the global alignment similarity of the pinyin sequences between the various song version distinguishing words; and determine the expected playback duration of the rhetorical question based on the pinyin sequence overlap and the corresponding candidate song names. The karaoke robot interaction module is used to control the karaoke robot interaction based on the relative size between the safety mute margin and the expected playback duration of the question voice.

[0015] Thirdly, the present invention provides a computer device including a memory and a processor. The memory is used to store computer program code, and the processor is used to call and run the computer program code from the memory to perform the method as described in the first aspect or any embodiment of the first aspect of the present invention.

[0016] Fourthly, the present invention provides a computer program product comprising computer program code, which, when executed, performs the method as described in the first aspect or any embodiment of the first aspect of the present invention.

[0017] Fifthly, the present invention provides a computer-readable storage medium storing computer program code that, when executed, performs the method as described in the first aspect or any embodiment of the first aspect of the present invention.

[0018] This application has the following beneficial effects: This invention acquires the pre-read accompaniment waveform data after the user's first round of voice commands during the accompaniment playback process in a karaoke system. It then determines a safe mute margin based on the energy exceedance of the pre-read accompaniment waveform data, enabling accurate prediction of when the microphone will fail during the climax of the accompaniment. Simultaneously, it determines the overlap of pinyin sequences based on the global alignment similarity of the pinyin sequences between the song versions matched in the first round of commands. Based on this, it dynamically infers the expected playback duration of the rhetorical question in conjunction with the candidate song names, effectively quantifying the effect of the ease of option pronunciation confusion on the machine's rhetorical questioning time. Furthermore, it compares the safe mute margin, representing the future level of environmental quietness, with the expected playback duration of the rhetorical question, representing the required dialogue time cost, and then executes corresponding karaoke robot interaction control. This allows it to break the rigid limitation of a fixed timeout waiting window before the noise of the accompaniment climax covers the user's voice, proactively deciding to execute the rhetorical question or switch the interaction path in advance. This effectively avoids getting stuck in a dead loop and receiving garbled characters, resulting in a higher success rate and reliability for the karaoke robot's dialogue-based song selection interaction. Attached Figure Description

[0019] To more clearly illustrate the technical solutions and advantages in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 A flowchart illustrating a method for implementing an intelligent karaoke robot that supports multi-turn dialogue interaction, as provided in one embodiment of the present invention; Figure 2 This is a structural diagram of an intelligent karaoke robot system supporting multi-turn dialogue interaction, provided as an embodiment of the present invention. Figure 3 This is a schematic diagram of a computer device structure provided in one embodiment of the present invention. Detailed Implementation

[0021] To further illustrate the technical means and effects adopted by the present invention to achieve its intended purpose, the following, in conjunction with the accompanying drawings and preferred embodiments, details the specific implementation, structure, features, and effects of a smart karaoke robot implementation method supporting multi-turn dialogue interaction proposed according to the present invention. In the following description, different "one embodiment" or "another embodiment" do not necessarily refer to the same embodiment, and specific features, structures, or characteristics in one or more embodiments can be combined in any suitable form. Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as implying or suggesting relative importance or implicitly indicating the number of indicated technical features. Thus, a feature defined with "first" or "second" may explicitly or implicitly include one or more of that feature.

[0022] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0023] The following description, in conjunction with the accompanying drawings, details a specific solution for an intelligent karaoke robot implementation method supporting multi-turn dialogue interaction provided by the present invention.

[0024] This invention provides a method for implementing an intelligent karaoke robot that supports multi-turn dialogue interaction. Please refer to [link / reference]. Figure 1 The diagram illustrates a flowchart of a method for implementing an intelligent karaoke robot supporting multi-turn dialogue interaction, according to an embodiment of the present invention. The method includes: Step S101: During the accompaniment playback process of the song selection system, acquire the user's first round of voice commands and the pre-read accompaniment waveform data after the user's first round of voice commands ends; determine the safe mute margin based on the energy exceedance of the pre-read accompaniment waveform data.

[0025] When the karaoke system is started and in idle standby mode, it first retrieves the low-level registers of the acoustic processing module on the karaoke machine's motherboard to read the current device's global volume gain coefficient. This coefficient is the scaling factor set by the user via a hardware knob or the system software UI. In high sound pressure level (SPL) accompaniment playback scenarios such as KTV rooms, the system establishes a real-time audio monitoring stream through its internal microphone array and activates an endpoint detection algorithm (VAD) to monitor ambient sound in real time. When a user initiates a voice request for song selection via microphone (e.g., saying "I want to select 'Song A'" or "Select 'Song B'"), the VAD algorithm captures the voice signal and continuously tracks it. When the algorithm detects that the duration of continuous silence exceeds the system's preset pause threshold (0.5 seconds in this embodiment), the system determines that the voice request has ended and encapsulates the detected audio signal into the user's first voice command. At this time, the system immediately records the system's absolute timestamp at the moment the first voice command ends, as the current baseline start time. It should be noted that the preset pause threshold can be adjusted according to the specific implementation environment, which will not be elaborated further here.

[0026] After obtaining the user's initial voice command and the baseline start time, the karaoke system uses this baseline start time as a trigger signal to immediately send a command to the accompaniment decoding engine on the karaoke machine's mainboard. The system obtains the absolute position of the current playback probe of the decoding engine, uses the accompaniment decoding time progress as the starting point of the time axis, and along the future playback direction of the audio track data stream, pre-extracts a segment of accompaniment decoding digital baseband signal with a preset fixed duration (8.0 seconds in this embodiment) from the system's memory audio buffer, and stores it in the look-ahead data pool. This allows the system to obtain the pre-read accompaniment waveform data after the user's initial voice command ends. Through this forward-looking track clipping and caching operation, this application can completely preserve the future accompaniment, which has not yet been released through the speaker of the karaoke room amplifier, in the form of a digital waveform within the current time node, providing a data support basis for subsequently predicting future short-term energy bursts and taking channel switching avoidance measures in advance. It should be noted that implementers can adjust the preset fixed time length according to the specific implementation environment. The preset fixed time length needs to be able to fully include the duration of the subsequent questioning voice broadcast to ensure that the time window of possible multi-round dialogue can be fully covered and to avoid missing noise out-of-bounds nodes. Further details will not be elaborated here.

[0027] After acquiring the user's initial voice commands and the pre-read accompaniment waveform data, existing technologies often use a fixed timeout window to receive subsequent user voice messages. This inevitably leads to problems such as the strong sound pressure of the pre-read accompaniment, with its drastic energy fluctuations, masking the user's voice and causing the recording equipment to malfunction due to amplitude distortion. Consequently, the system cannot distinguish between valid commands and background noise during the climax of the accompaniment. To accurately define the limit of the microphone's ability to maintain normal voice interaction during future accompaniment, this embodiment of the invention determines a safe silence margin in the current step by deeply analyzing the energy exceedance of the pre-read accompaniment waveform data. This serves as an objective physical measure for the system to determine the environmental noise immunity baseline. This helps the system know in advance how much safe silence time remains before the next climax of the accompaniment covers the user's voice, thus effectively avoiding garbled characters caused by forcing the user to speak in a noisy environment.

[0028] Preferably, in some possible implementations of the embodiments of the present invention, the process of obtaining the safety noise margin includes: In each audio data frame of the pre-read accompaniment waveform data, the corresponding short-time energy value is determined based on the overall magnitude of all discrete digital transient amplitude data. Specifically, the corresponding transient amplitude data is determined by multiplying each discrete digital transient amplitude data by the prior global volume gain coefficient. In each audio data frame of the pre-read accompaniment waveform data, the corresponding short-time energy value is determined by performing a sum-of-squares operation on all transient amplitude data. This is because the digital baseband signal output by the audio decoder cannot represent the actual physical sound pressure emitted by the booth speakers. Multiplying the discrete digital transient amplitude by the prior global volume gain coefficient for linear amplitude scaling restores the true acoustic impact intensity of the accompaniment on the microphone at the current amplifier volume. The short-time energy value obtained by summing the squares of the scaled amplitude quantifies the actual physical energy released in that audio data frame, thus providing an accurate energy basis for determining whether the acoustic echo cancellation algorithm is penetrated by extreme accompaniment sound pressure.

[0029] The audio data frame corresponding to the first short-term energy value exceeding the preset reception failure energy threshold is designated as the noise-affected data frame. The safe silence margin is determined based on the time interval between the moment the noise-affected data frame occurs and the moment the user's first voice command ends. This process allows for precise timing of when the future accompaniment sound pressure level will penetrate the acoustic echo cancellation algorithm's limit. This time interval visually represents the remaining usable quiet period before the system falls into a noise-masking state, providing a physical time scale for subsequent evaluation of whether the system has sufficient time to complete long-text voice responses.

[0030] It should be noted that if the first audio data frame in the pre-read accompaniment waveform data is determined to be a noise-affected data frame, it indicates that the environmental noise is at an extremely destructive state at this moment, and the karaoke system has completely lost the environmental tolerance to maintain voice interaction. At this time, a mute command is issued to shut down the microphone input channel of the karaoke robot, and the corresponding candidate song name is displayed on the display device for interaction. If there are no noise-affected data frames in all audio data frames of the pre-read accompaniment waveform data, that is, all the corresponding short-time energy values ​​are less than the preset sound failure energy threshold, it means that the sound environment is within an absolutely safe range within a preset fixed time length. At this time, the preset fixed time length is directly assigned to the safe mute margin.

[0031] In one specific implementation of this invention, the process of obtaining the preset radio failure energy threshold includes: The invention involves statistically analyzing the short-time energy values ​​of all audio data frames in historical data; performing cluster analysis on the short-time energy values ​​of all audio data frames to determine at least two energy value clusters; calculating the mean of all short-time energy values ​​in each energy value cluster to determine the corresponding energy reference value; and using the minimum short-time energy value in the energy cluster with the largest corresponding energy reference value as the preset sound pressure failure energy threshold. Historical data includes all audio data frames stored in the local database. The cluster analysis method in this embodiment uses the k-means clustering algorithm, and the K value is determined using the elbow method. Cluster analysis can adaptively separate normal accompaniment energy from extreme high sound pressure levels that cause microphone distortion based on actual operating data. The cluster with the largest energy reference value represents the set of extreme sound pressure levels that cause sound pressure algorithm failure. Selecting the minimum short-time energy value in this set as the threshold objectively defines the critical bottom line for microphone sound pressure failure under the current hardware environment, avoiding errors from manual experience. It should be noted that the k-means clustering algorithm and the elbow method are well-known techniques to those skilled in the art and will not be further limited or elaborated upon here.

[0032] Step S102: Based on the semantic information of the user's first voice command and the matching of song names in the local music library database, determine all song version distinguishing words; based on the global alignment similarity of the pinyin sequences among the distinguishing words of each song version, determine the pinyin sequence overlap; based on the pinyin sequence overlap and the corresponding candidate song names, determine the expected playback duration of the rhetorical question.

[0033] After obtaining the safety margin for muting in defining the future safe interaction window through the above steps, since the user's initial voice commands are usually colloquial and vague, this inevitably leads to the system hitting multiple candidate songs with slight version differences when searching the local music library. This makes it impossible for the system to provide the user with a clear and unique playback option in multiple rounds of dialogue. To solve the problem of redundant candidate options leading to a lack of targeted machine-generated questions, this embodiment of the invention delves into the semantic information of the user's initial voice commands in the current step and combines it with the matching of song names in the local music library database to extract and determine all song version distinguishing words. This helps the system accurately extract the most distinctive core option tags from the list of songs with the same name, so that the subsequent questioning language of the song request robot can directly address the differences in options and avoid ineffectively wasting safe dialogue time by playing useless, repetitive, and lengthy text.

[0034] Preferably, in some possible implementations of the embodiments of the present invention, the process of obtaining song version distinguishing words includes: The user's initial voice command is converted into voice command text by a voice recognition engine. This text is then input into a trained named entity recognition model for sequence labeling to determine the entity labels corresponding to each character. Based on all characters whose entity labels are song title attributes, a search entity word is determined (e.g., if the voice command text is "I want to request a song 'A'", the determined search entity word is "song A"). A matching query is performed in the local music library database based on the search entity word to determine all corresponding candidate songs. It should be noted that if no candidate song is found, it means the song requested in the current request is not found. In this case, an error message is played, and the user is asked to resubmit the voice command. If only one candidate song is found, it means the corresponding voice command has no ambiguity. In this case, the dialogue questioning mechanism is not triggered, and the song is directly added to the library. Furthermore, if the number of candidate songs exceeds a preset threshold (set to 10 in this embodiment, but adjustable), it indicates that the user's voice command is too broad, resulting in too many matching results. In this case, a mute command is issued to disable the microphone input channel of the song request robot, and the corresponding candidate song names are displayed on the display device for interaction.

[0035] Based on the names of all candidate songs, regular expressions are used for pattern matching to extract and deduplicate version-distinguishing words. These deduplicated version-distinguishing words are then used as segmentation words for all candidate song names (e.g., using regular expressions to match all candidate song names containing suffixes such as "live version," "tour concert version," or "DJ version" within parentheses, and removing duplicate words to obtain a unique set of version-distinguishing features). The named entity recognition model accurately extracts core song name attributes from the voice command text, significantly reducing retrieval noise caused by colloquial interjections. Regular expressions are used to automatically extract version-modifying characters such as "concert version" or "live version" within parentheses or suffixes from the full names of candidate songs, and deduplication is performed, refining complex raw database fields into unique version-distinguishing words. This process not only eliminates redundant broadcast information but also provides standardized candidate segmentation input for subsequent calculations of pinyin overlap, ensuring the simplicity and high discriminativeness of the system's questioning language. It should be noted that if, after extraction using regular expressions, no version-distinguishing words are found in the name of a candidate song (i.e., the candidate song name has no special version modifiers), the system will directly assign it the default character "standard version" to ensure that all candidate songs have the corresponding version-distinguishing features. In addition, it should be noted that string matching and extraction of text characters in a specific format using regular expressions are technical means well known to those skilled in the art, and will not be further limited or elaborated here.

[0036] After obtaining all the song version differentiation words through the above steps, when the system provides voice clarification to the user, there are often cases where the acoustic pronunciation structures of the alternative version words, such as "live version" and "concert version," highly overlap. This inevitably leads to the synthesized speech eliding or linking at normal speaking speed, making it easy for users in a private room to be confused by the sound. In order to effectively quantify the fuzzy interference caused by the similarity of the pronunciation of the alternative options to the user's auditory recognition, this embodiment of the invention calculates the global alignment similarity of the pinyin sequences between the differentiation words of each song version in the current step. This accurately determines the pinyin sequence overlap degree, which represents the limit probability of pronunciation overlap. This helps the system build a dynamic mathematical relationship between text attributes and speech engine time, thereby knowing the worst confusion penalty coefficient in the current round of questioning in advance and avoiding users mishearing and answering incorrectly due to unclear pronunciation of the alternative options.

[0037] Preferably, in some possible implementations of the embodiments of the present invention, the process of obtaining the overlap degree of the pinyin sequence includes: The text-to-pinyin component is used to segment each candidate song name into a pinyin character sequence; each candidate song name segment is then used as the target name segment; and the other candidate song names segmented other than the target name segment are used as reference name segment.

[0038] Based on the Niedermann-Onsch global sequence alignment algorithm, the pinyin character sequence of the target name segmentation is aligned with the pinyin character sequence of each reference name segmentation to construct a similarity matrix. Specifically: An initial matrix is ​​constructed between the target name segment and each reference name segment. The rows of the initial matrix represent the pinyin character sequence of the target name segment, and the columns represent the pinyin character sequence of each reference name segment. All elements in the initial matrix are set to 0. Then, the element values ​​of each cell are iterated and updated from top to bottom and left to right. When iterating through each cell, if the letters in the corresponding row and column are the same, the sum of the element value of the cell to its upper left and the preset matching score constant is calculated. The first score is the sum of the values ​​of the elements in the cell after traversal. If the letters in the row and column of the cell are different, the second score is the sum of the values ​​of the elements in the cell to the left of the cell and the preset mismatch penalty constant. The third score is the sum of the values ​​of the elements in the cell directly above the cell and the preset empty space penalty constant. The maximum value among the first, second, and third scores is taken as the value of the cell after traversal. After all cells have been traversed, the values ​​of the elements in each cell are filled into the corresponding positions to construct a similarity matrix. To ensure the normal startup of the above traversal and update process and provide boundary condition support, the system preprocesses the boundaries of the initial matrix before cell traversal and update: for each cell in the first row, the element value of its traversed element is set as the product of the column index of the cell and a preset empty space penalty constant; for each cell in the first column, the element value of its traversed element is set as the product of the row index of the cell and a preset empty space penalty constant, thus serving as the basis for the historical scores of subsequent internal cells using the values ​​of the upper left, upper right, and upper left cells for recursive calculations. It should be noted that the above process is a specific manifestation of the specific implementation process of the Niedermann-Onsch global sequence alignment algorithm in constructing the similarity matrix in this embodiment of the invention, and its significance will not be further elaborated here. In a specific implementation of this embodiment, the preset matching score constant is set to 1, and the preset empty space penalty constant and preset mismatch penalty constant are both set to -1. These three preset values ​​are all conventional definitions of the Niedermann-Onsch global sequence alignment algorithm and will not be further elaborated here.

[0039] The elements in the cell at the bottom right corner of the similarity matrix are normalized to determine the single alignment score between the target name segment and each reference name segment. All single alignment scores between all candidate song name segments are calculated, and the maximum value of all single alignment scores is taken as the pinyin sequence overlap. According to the principle of the Niedermann-Ongsch global sequence alignment algorithm, the larger the element in the cell at the bottom right corner of the similarity matrix, the more identical letter matches and the fewer gap mismatches there are between the pinyin sequences of the target name segment and the reference name segment. This indicates that the two name segments are extremely similar in pronunciation. Therefore, the maximum value of all single alignment scores obtained from pairwise matching of all candidate song name segments is used to determine the pinyin sequence overlap that represents the limit of the most easily confused pronunciation in the current candidate song list. In other words, the greater the pinyin sequence overlap, the higher the degree of overlap in word pronunciation between the current candidate versions, the greater the anti-confusion cost that the song request robot must pay when synthesizing speech, and the longer the subsequent playback time.

[0040] In one specific implementation of this invention, the process of obtaining the single alignment score is expressed by the formula: ;in, Target name segmentation Single alignment score between each reference name segmentation; Target name segmentation With the corresponding first The element value of the cell at the bottom right vertex of the similarity matrix between the word segments of the reference names; Target name segmentation With the corresponding first The maximum value between the number of rows and columns in the similarity matrix among the word segments of each reference name; Select a function to maximize the value; first, with As the denominator pair Normalization eliminates the scoring scale differences caused by words of different lengths (such as "live version" and "unplugged concert version"), transforming the absolute cumulative score into a relative score that reflects the character overlap ratio, making the results of different comparison combinations comparable. The maximum value selection function is introduced here because the Niedermann-Ongsch global sequence alignment algorithm includes negative penalty for mismatches and gaps. When the pinyin differences between two words are extremely large, the cumulative score may be negative. Comparing it with 0 and truncating it can strictly map the final pinyin sequence overlap to the non-negative effective probability interval of [0,1], avoiding negative time consumption that does not conform to physical meaning when extrapolating the time dilation weight in the later stage.

[0041] After accurately quantifying the overlap of the pinyin sequence reflecting the limit of pronunciation confusion through the above steps, machine-synthesized speech must slow down its pronunciation to ensure clarity when facing easily confused words. This inevitably leads to a significant non-linear increase in the time cost for the system to complete multiple rounds of dialogue. To accurately determine the complete physical time required for the system to actively slow down its speech to clarify options, this embodiment of the invention deeply integrates the overlap of the pinyin sequence and the corresponding candidate song names in the current step to calculate and determine the expected playback duration of the rhetorical question. This helps the system transform the abstract acoustic feature of text confusion into a concrete amount of time extension, thereby enabling the system to grasp the real time cost required to complete this round of rhetorical questioning and wait for a response in advance, avoiding the problem of inaccurate dialogue time estimation due to speech lengthening.

[0042] Preferably, in some possible implementations of the embodiments of the present invention, the process of obtaining the expected playback duration of the rhetorical question includes: Based on all song version distinguishing terms and a preset rhetorical question template structure, the rhetorical question prompt text sentence is determined. Since candidate songs without special version modifications have been assigned a default "Standard Version" label in previous steps, it ensures that the option labels involved in the splicing can fully cover the user's actual choices. For example, when the song version distinguishing terms involved in the splicing are "Standard Version" and "Live Version," the generated rhetorical question prompt text sentence is "Would you like to request the Standard Version or the Live Version?"; when the terms involved in the splicing are "Standard Version," "Live Version," and "Concert Version," the generated rhetorical question prompt text sentence is "Would you like to request the Standard Version, the Live Version, or the Concert Version?"; when the terms involved in the splicing are more than three versions such as "Standard Version," "Live Version," "Concert Version," and "DJ Version," the generated rhetorical question prompt text sentences are arranged in parallel, i.e., "Would you like to request the Standard Version, the Live Version, the Concert Version, or the DJ Version?". Through this template-based splicing, discrete option labels can be transformed into broadcast text that conforms to natural human-computer interaction habits.

[0043] The basic rhetorical question playback duration is determined by multiplying the total number of valid Chinese characters in the rhetorical question prompt text sentence by the preset single-character playback time. A positive correlation mapping is applied to the overlap of the pinyin sequence to determine the speech rate slowing weight. The weighted rhetorical question playback duration is determined by multiplying the speech rate slowing weight by the basic rhetorical question playback duration. The expected playback duration of the rhetorical question is determined by the sum of the weighted rhetorical question playback duration and the preset silence waiting time. Traditional voice interaction time estimation often relies solely on static calculations based on the number of characters, neglecting the forced constraint of rare or easily confused words on the synthesized speech rate. In multi-turn dialogue option confirmation scenarios, if the options have highly similar pronunciations (high pinyin sequence overlap), the system must actively slow down the speech rate to improve pronunciation differentiation; otherwise, the user will have to repeatedly ask questions if they cannot hear clearly. Through the linear positive correlation mapping and product weighting operations described above, this application transforms the pinyin sequence overlap, which represents the limit of pronunciation confusion for candidate words, into a speech rate slowing weight, which represents the multiplier of machine playback duration expansion. This is because a higher degree of overlap in the pinyin sequences indicates a very high probability of structural similarity and pronunciation overlap between two or more candidate word versions. When the system synthesizes and pronounces these words, auditory linking or elision is extremely likely to occur. Users in noisy environments like private rooms find it difficult to accurately discern subtle differences, thus necessitating a slower speaking speed. The dynamically compensated playback duration of the rhetorical question accurately and objectively reflects the actual physical time cost the system must expend to ensure clarification under the worst-case pronunciation conditions, completely resolving the interaction conflict problem caused by the rigid estimation of the fixed timeout window mechanism.

[0044] In one specific implementation of this invention, the process of obtaining the estimated playback duration of the rhetorical question is expressed by the following formula: ;in, The estimated playback duration for the rhetorical question; The total number of valid Chinese characters in the rhetorical question prompt text sentence; Preset the broadcast time for a single Chinese character; The basic question is the broadcast duration; The degree of overlap in the pinyin sequence; Adjust the preset speaking speed by a coefficient; Weighting for slowing down speech rate; Preset silent waiting time; The weighted question broadcast duration is specified. In this embodiment of the invention, the preset speech rate slowdown adjustment coefficient is set to 0.5, which can be adjusted according to the specific implementation environment. This parameter is used to adjust the maximum allowable lengthening ratio supported by the text-to-speech engine while maintaining the synthesized machine's pronunciation without distortion or cracking. This ensures that the change in speech rate slowdown weight can truly reflect the sound output limit of the karaoke machine hardware. Further details are omitted here.

[0045] In one specific implementation of this invention, the preset single-character broadcast time is set to 0.25 seconds, which represents the standard physical number of seconds required for the text-to-speech engine to synthesize and output a single Chinese character at a normal broadcast speed. This needs to be adjusted based on the default broadcast speed set in the implementation environment and can be adjusted according to the specific implementation environment. The preset silence waiting time is set to 3 seconds, which represents the sum of the network transmission latency constant including cloud semantic understanding and the preset fixed silence time left for the user to think, organize language and speak, so that the expected playback time of the rhetorical question can cover the complete interactive closed loop time from the machine's voice, the user's hearing to the system's completion of parsing.

[0046] Step S103: Perform interactive control of the song request robot based on the relative size between the safety margin for silence and the expected playback duration of the questioning voice.

[0047] After deriving the estimated playback duration of the rhetorical question reflecting the system's pronunciation time cost and the safe silence margin reflecting the tolerance of the external environment through the above steps, the existing technology always relies on a fixed timeout waiting window to passively receive the user's voice confirmation. This inevitably leads to the system still having its microphone open when the forcedly extended clarification speed has been exhausted, thus being completely overwhelmed by the subsequent high-energy accompaniment climax. In order to actively interrupt this auditory interaction channel that is bound to have timing collisions, the embodiment of the present invention, in the current step, deeply compares and determines the relative size between the estimated playback duration of the rhetorical question and the safe silence margin, and directly executes the underlying karaoke robot interaction control. This helps the system to preemptively downgrade and switch voice dialogues that are bound to fail to the visual display screen and the physical buttons of the remote control within one clock cycle, thereby accurately achieving seamless handover of cross-modal interaction and effectively ensuring the success rate and reliability of the karaoke machine's dialogue in extremely high-pressure music scenarios.

[0048] Preferably, in some possible implementations of the embodiments of the present invention, the process of controlling the interactive interaction of the karaoke robot based on the relative size between the safety mute margin and the expected playback duration of the questioning voice includes: If the safe mute margin is greater than or equal to the expected playback duration of the rhetorical question, the voice interaction state of the song-selection robot will be maintained; if the safe mute margin is less than the expected playback duration of the rhetorical question, a mute command will be issued to shut down the microphone input channel of the song-selection robot, and the corresponding candidate song name will be displayed on the display device for interaction.

[0049] When the safety margin of silence is sufficient to cover the expected playback duration of the rhetorical question, it indicates that even if the system actively slows down the speech rate to distinguish between highly similar options (such as the sung version and the live version), the energy of the pre-read accompaniment waveform will not exceed the threshold that would cause microphone distortion during the entire question-and-answer process. At this point, ambient noise is still within the safe processing range of the system's acoustic echo cancellation algorithm, allowing the system to continue calling the speech synthesis component to play the rhetorical question prompt and maintain microphone monitoring.

[0050] Conversely, if the safe mute margin is less than the expected playback duration of the rhetorical question, it indicates that during the system's slow reading of the rhetorical question options and waiting for the user's response, the climax of the accompaniment will inevitably erupt with high sound pressure, breaking through the system's microphone noise immunity threshold, causing the voice recognition module to receive garbled audio filled with noise. At this time, the system immediately sends a mute (MUTE) level command to the audio analog-to-digital converter (ADC) control register on the karaoke machine's motherboard, forcibly suspending and blocking the microphone's sound input; simultaneously, it extracts the popularity ranking field from the current candidate song database, sorts all candidate songs in descending order, calls the graphics processing component to render them into a two-dimensional card layer array, pushes it to the video output interface (such as the HDMI interface) of the karaoke room's display device, and transfers the operation focus to the user's remote control physical directional keys, thus completing the active relay from the doomed auditory channel to the interference-resistant visual button path; making the success rate and reliability of the karaoke robot's dialogue song selection interaction higher.

[0051] In summary, a method for implementing an intelligent karaoke robot supporting multi-turn dialogue interaction obtains the pre-read accompaniment waveform data after the user's first voice command during the karaoke system's accompaniment playback process. It determines a safe mute margin based on the energy exceedance of the pre-read accompaniment waveform data, enabling accurate prediction of when the microphone will fail during the climax of the accompaniment. Simultaneously, it determines the pinyin sequence overlap based on the global alignment similarity of the pinyin sequences between the song versions matched in the first round of commands. Based on this, it dynamically infers the expected playback duration of the rhetorical question in conjunction with the candidate song names, effectively quantifying the effect of the ease of option pronunciation confusion on the machine's rhetorical questioning time. Furthermore, it compares the safe mute margin, representing the future level of environmental quietness, with the expected playback duration of the rhetorical question, representing the required dialogue time cost, and then executes corresponding karaoke robot interaction control. This allows it to break the rigid limitation of a fixed timeout waiting window before the noise of the climax of the accompaniment covers the user's voice, proactively deciding to execute the rhetorical question or switch the interaction path in advance, effectively avoiding the situation of getting stuck in an infinite loop and receiving garbled characters. This results in a higher success rate and reliability for the karaoke robot's dialogue-based song selection interaction.

[0052] This invention also provides a system for implementing an intelligent karaoke robot that supports multi-turn dialogue interaction. Please refer to [link / reference]. Figure 2The diagram shows a structural diagram of an intelligent karaoke robot system that supports multi-turn dialogue interaction according to an embodiment of the present invention. The system includes: a data acquisition and preprocessing module 201, a playback duration determination module 202, and a karaoke robot interaction module 203.

[0053] The data acquisition and preprocessing module 201 is used to acquire the user's first round of voice commands and the pre-read accompaniment waveform data after the user's first round of voice commands ends during the accompaniment playback process of the karaoke system; and to determine the safe mute margin based on the energy out-of-bounds condition of the pre-read accompaniment waveform data. The playback duration determination module 202 is used to determine all song version distinguishing words based on the semantic information of the user's first round of voice commands and the matching of song names in the local music library database; determine the pinyin sequence overlap based on the global alignment similarity of the pinyin sequences between the various song version distinguishing words; and determine the expected playback duration of the rhetorical question based on the pinyin sequence overlap and the corresponding candidate song names. The karaoke robot interaction module 203 is used to control the karaoke robot interaction based on the relative size between the safety mute margin and the expected playback duration of the question voice.

[0054] It should be noted that the system provided in the above embodiments is only an example of the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the computer device can be divided into different functional modules to complete all or part of the functions described above. In addition, the intelligent karaoke robot implementation system supporting multi-turn dialogue interaction and the intelligent karaoke robot implementation method embodiment supporting multi-turn dialogue interaction provided in the above embodiments belong to the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0055] This invention also provides a computer device; please refer to [link / reference]. Figure 3 The illustration shows a schematic diagram of a computer device structure according to an embodiment of the present invention. The computer device includes a memory 301, a processor 302, and a computer program 303 stored in the memory 301 and running on the processor 302. When the processor 302 executes the computer program 303, the computer device can execute any of the aforementioned methods for implementing an intelligent karaoke robot that supports multi-turn dialogue interaction.

[0056] This invention also provides a computer program product that, when run on a computer device, enables the computer device to execute any of the aforementioned methods for implementing an intelligent karaoke robot that supports multi-turn dialogue interaction.

[0057] This invention also provides a computer-readable storage medium storing computer program code. When the computer program code is run on a computer device, the computer device can execute any of the aforementioned methods for implementing an intelligent karaoke robot that supports multi-turn dialogue interaction.

[0058] In the embodiments provided by the present invention, it should be understood that the computer device, computer program product and computer-readable storage medium provided are all used to execute the corresponding methods provided above, and therefore the beneficial effects they can achieve can be referred to the beneficial effects of the methods provided above, which will not be repeated here.

[0059] It should be noted that the order of the above embodiments of the present invention is merely for descriptive purposes and does not represent the superiority or inferiority of the embodiments. The processes depicted in the accompanying drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0060] The various embodiments in this specification are described in a progressive manner. The same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on describing the differences from other embodiments.

Claims

1. A method for implementing an intelligent karaoke robot that supports multi-turn dialogue interaction, characterized in that, The method includes: During the accompaniment playback in the karaoke system, the system acquires the user's first voice command and the pre-read accompaniment waveform data after the user's first voice command ends; based on the energy exceedance of the pre-read accompaniment waveform data, it determines the safe mute margin. Based on the semantic information of the user's first voice command and the matching of song names in the local music library database, all song version distinguishing words are determined; based on the global alignment similarity of the pinyin sequences among the song version distinguishing words, the pinyin sequence overlap is determined; based on the pinyin sequence overlap and the corresponding candidate song names, the expected playback duration of the rhetorical question is determined. The interaction control of the karaoke robot is based on the relative size between the safety mute margin and the expected playback duration of the rhetorical question.

2. The method for implementing an intelligent karaoke robot supporting multi-turn dialogue interaction according to claim 1, characterized in that, The process of obtaining the safety noise margin includes: In each audio data frame of the pre-read accompaniment waveform data, the corresponding short-time energy value is determined based on the overall size of all discrete digital transient amplitude data; the audio data frame corresponding to the first short-time energy value that is greater than the preset reception failure energy threshold is taken as the noise-affected data frame; and the safe mute margin is determined based on the time interval between the time corresponding to the noise-affected data frame and the time when the user's first round of voice commands ends.

3. The method for implementing an intelligent karaoke robot supporting multi-turn dialogue interaction according to claim 1, characterized in that, The process of obtaining the song version differentiation vocabulary includes: The user's initial voice command is converted into voice command text using a voice recognition engine; the voice command text is input into a trained named entity recognition model for sequence labeling to determine the entity label corresponding to each character; based on all characters whose entity labels are song name attributes, the search entity words are determined; based on the search entity words, a matching query is performed in the local music library database to determine all corresponding candidate songs; Based on the names of all candidate songs, pattern matching is performed using regular expressions to extract all song version distinguishing words and remove duplicates. The deduplicated song version distinguishing words are then used as the word segments for all candidate song names.

4. The method for implementing an intelligent karaoke robot supporting multi-turn dialogue interaction according to claim 1, characterized in that, The process of obtaining the overlap of the pinyin sequences includes: Each candidate song name is segmented and converted into a sequence of pinyin characters using a text-to-pinyin component. Each candidate song name segment is used as the target name segment; the other candidate song names segmented outside the target name segment are used as reference name segmentations. The Pinyin character sequence of the target name segment is aligned with the Pinyin character sequence of each reference name segment based on the Niedermann-Onsch global sequence alignment algorithm to construct a similarity matrix; the elements in the cell at the lower right corner vertex of the similarity matrix are normalized to determine the single alignment score between the target name segment and each reference name segment. Calculate the single alignment scores between all candidate song title word segments, and take the maximum value of all single alignment scores as the pinyin sequence overlap.

5. The method for implementing an intelligent karaoke robot supporting multi-turn dialogue interaction according to claim 1, characterized in that, The process for obtaining the expected playback duration of the rhetorical question includes: Based on the vocabulary distinctions of all song versions and the preset rhetorical question dialogue template structure, determine the rhetorical question prompt text sentence; The basic rhetorical question broadcast duration is determined by multiplying the total number of valid Chinese characters in the rhetorical question prompt text sentence by the preset single Chinese character broadcast time; the overlap of the pinyin sequence is positively correlated to determine the speech rate slowing weight. The weighted rhetorical question broadcast duration is determined by multiplying the speech rate slowing weight by the basic rhetorical question broadcast duration. The expected playback duration of the rhetorical question is determined by the sum of the weighted rhetorical question broadcast duration and the preset silent waiting duration.

6. The method for implementing an intelligent karaoke robot supporting multi-turn dialogue interaction according to claim 1, characterized in that, The process of controlling the karaoke robot interaction based on the relative size between the safety mute margin and the expected playback duration of the questioning voice includes: If the safety mute margin is greater than or equal to the expected playback duration of the question, then the voice interaction state of the song request robot is maintained. If the safety mute margin is less than the expected playback duration of the question, a mute command is issued to shut down the microphone input channel of the karaoke robot, and the corresponding candidate song name is displayed on the display device for interaction.

7. The method for implementing an intelligent karaoke robot supporting multi-turn dialogue interaction according to claim 2, characterized in that, The process of obtaining the short-time energy value includes: The corresponding transient amplitude data is determined by multiplying each discrete digital transient amplitude data with the prior global volume gain coefficient; in each audio data frame of the pre-read accompaniment waveform data, the sum of squares is performed on all transient amplitude data to determine the corresponding short-time energy value.

8. The method for implementing an intelligent karaoke robot supporting multi-turn dialogue interaction according to claim 2, characterized in that, The process of obtaining the preset radio failure energy threshold includes: The short-time energy values ​​of all audio data frames in the historical data are statistically analyzed; cluster analysis is performed on the short-time energy values ​​of all audio data frames to determine at least two energy value clusters; the mean of all short-time energy values ​​in each energy value cluster is calculated to determine the corresponding energy reference value; the minimum short-time energy value in the energy cluster with the largest corresponding energy reference value is used as the preset radio reception failure energy threshold.

9. The method for implementing an intelligent karaoke robot supporting multi-turn dialogue interaction according to claim 5, characterized in that, The preset single-character broadcast time is set to 0.25 seconds.

10. A method for implementing an intelligent karaoke robot supporting multi-turn dialogue interaction according to claim 5, characterized in that, The preset silent waiting time is set to 3 seconds.

Images