A Multi-role Voice Interaction Control Method Based on AI Calls

Abstract

This invention discloses a multi-role voice interaction control method based on AI calls, belonging to the field of voice interaction control technology. The method acquires a mixed audio stream and its attribute information after the AI ​​call is established, performs frame segmentation, endpoint detection, echo cancellation, speech enhancement, and noise reduction on the audio stream, and demixes and separates standardized speech segments in multi-person speaking scenarios. It performs two-level similarity matching between the MFCC-based voiceprint embedding vector and a preset role voiceprint library to bind the audio stream to the role identity. It performs speech recognition and semantic understanding on each audio stream to generate candidate commands. Then, based on role attribute information, it performs permission verification, conversation task relevance analysis, and conflict priority judgment to resolve conflicts in the execution command queue and generate the final execution command. Finally, it generates corresponding voice feedback based on the execution result, forming a closed loop for multi-role voice interaction control, improving the accuracy of voice command recognition in multi-person call scenarios.

Description

Technical Field

[0001] This invention relates to the field of interactive control technology, and more specifically, to a multi-role voice interactive control method based on AI calls. Background Technology

[0002] With the continuous evolution of AI speech recognition, natural language understanding, and intelligent communication technologies, the application paradigm of voice interaction has gradually expanded from the traditional single-user command control of a single device to complex scenarios including multi-person real-time calls, remote collaborative work, home device linkage, intelligent customer service conversations, and online meeting control. In such multi-participant environments, different speakers often issue voice commands synchronously or asynchronously, and each role has significant differences in identity attributes, control permissions, operation goals, and contextual semantics. However, existing voice interaction solutions generally focus on recognizing and linearly executing the semantic content of individual users in a single audio stream. They lack effective recognition and continuous tracking of multiple voice sources in a call channel. Relying solely on single-round voice recognition and simple command triggering methods, existing technologies struggle to accurately distinguish the control intentions of each role during concurrent or alternating speech by multiple people. This can lead to misidentification, misexecution, or logical conflicts between parallel instructions, resulting in unstable control results, chaotic interaction logic, and even erroneous changes in device status. Summary of the Invention

[0003] The main objective of this invention is to provide a multi-role voice interaction control method based on AI calls to overcome the problems mentioned in the background art.

[0004] To achieve the above objectives, a multi-role voice interaction control method based on AI calls is provided, comprising the following steps: 100. After an AI call is established, the mixed audio stream data and its attribute information are acquired. The mixed audio stream data is then processed by framing, speech segment detection, endpoint detection, echo cancellation, speech enhancement, and noise reduction to obtain standardized speech segments. In a multi-person speaking scenario, the standardized speech segments are subjected to blind source separation to obtain several independent audio streams. S200, extract voiceprint features from each audio stream, match the extracted voiceprint features with a preset role voiceprint database, determine the role identity of each audio stream, and establish a correspondence between audio streams and role identities; S300 performs speech recognition and semantic understanding on each audio stream that has been bound to a role identity, extracts the control object, control action and parameter value and calculates semantic confidence to generate candidate instructions; S400: Based on the role attribute information, the candidate instructions are verified for permissions and conflict resolution is performed to generate the final execution instruction, and voice feedback is output to the corresponding role according to the execution result.

[0005] Furthermore, in S100: The mixed audio stream is segmented according to a preset frame length and frame shift, and the short-time energy, zero-crossing rate, spectral entropy and signal-to-noise ratio of each frame are calculated. When the short-time energy of a frame is lower than a preset silence threshold, and the zero-crossing rate and spectral entropy are both lower than the corresponding noise threshold, the frame is determined to be a non-speech frame and discarded. When a preset number of consecutive frames meet the speech frame condition, they are merged into candidate speech segments.

[0006] Furthermore, S100 also includes: The candidate speech segments are processed by a speech endpoint detection algorithm to determine the speech start and end boundaries, and then by echo cancellation, speech enhancement and noise reduction to obtain the standardized speech segments. The blind source separation uses an algorithm based on independent component analysis to demix and separate standardized speech segments with multiple sound sources to obtain several statistically independent audio streams, each audio stream corresponding to an independent sound source.

[0007] Furthermore, the step of voiceprint feature extraction in S200 includes: After the audio stream is segmented into frames, the Mel frequency cepstral coefficients (MFCCs) and their first and second-order differences are extracted to obtain the MFCC feature sequence. The MFCC feature sequence is then mapped into a fixed-dimensional voiceprint embedding vector using a deep neural network-based voiceprint recognition model.

[0008] Furthermore, the step of voiceprint matching and character identification in S200 includes: The preset voiceprint database stores the account identifiers of each registered role and their corresponding historical voiceprint feature vectors; for each account identifier entering the current AI call, the center voiceprint vector is calculated based on its historical voiceprint feature vector. Calculate the center similarity between the current audio stream's voiceprint embedding vector and each of the central voiceprint vectors, and identify account identifiers with center similarity greater than a preset center similarity threshold as candidate accounts. The specific calculation process is as follows: A pre-defined voiceprint database is established, pre-storing the identity identifiers of each registered role and their corresponding historical voiceprint feature vectors. The account identifier entering the current AI call is extracted, and the historical voiceprint feature vectors corresponding to each account identifier in the voiceprint database are extracted. Then, the K-Means clustering algorithm is used to cluster the historical voiceprint feature vectors, or the arithmetic mean is directly calculated to obtain the center voiceprint vector of that account identifier. The cosine similarity is used to calculate the center similarity S between the voiceprint embedding vector of the current audio stream and the center voiceprint vector of each account identifier. centerThe specific calculation formula is as follows:

[0009] Where v represents the voiceprint embedding vector of the current audio stream, and c represents the center voiceprint vector of the account identifier; thus, the center similarity between the voiceprint embedding vector of the audio stream and all account identifiers entering the current AI call can be obtained; a preset center similarity threshold is set to 0.75 in this embodiment; if the center similarity is greater than the center similarity threshold, the corresponding account identifier is used as a candidate account of the current audio stream, so as to quickly filter out candidate accounts with high similarity, filter out obviously irrelevant accounts, and greatly reduce the amount of computation for subsequent fine comparison.

[0010] Furthermore, the step of voiceprint matching and character identification in S200 also includes: For each candidate account, calculate the cosine similarity between the current audio stream's voiceprint embedding vector and all historical voiceprint feature vectors under that candidate account, and take the maximum value as the maximum similarity of that candidate account; set the maximum similarity S... max and the center similarity S center The final similarity is calculated by applying a weighted harmonic average; the specific calculation method uses the following formula:

[0011] Where ε is a very small constant to prevent division by zero errors, it is set to 10 in this embodiment. -8 A preset recognition threshold is set; in this example, the recognition threshold is set to 0.85. The candidate account with the highest final similarity and greater than or equal to the preset recognition threshold is selected, and its corresponding role account identifier is determined as the final identity of the audio stream. The voiceprint embedding vector corresponding to the audio stream is written back to the role voiceprint library. If there is no candidate account with a final similarity greater than or equal to the preset recognition threshold, the current speaker is determined to be an unregistered user, and a temporary identity label is assigned to it.

[0012] Furthermore, S300 specifically includes: Speech recognition is performed on each audio stream to obtain transcribed text, and the text confidence score is calculated. The transcribed text is subjected to intent recognition to obtain the intent category and the corresponding intent probability; Entity extraction is performed on the transcribed text, the slots obtained from entity extraction are identified, and slot analysis is performed to calculate entity completeness. The semantic confidence of the transcribed text is obtained by weighted fusion of the text confidence, the intent probability, and the entity completeness. When the semantic confidence level is greater than or equal to the preset semantic confidence threshold, the transcribed text and its corresponding role account identifier, control object, control action, parameter value, generation timestamp, and semantic confidence level are encapsulated into a candidate instruction.

[0013] Furthermore, the calculation method for the entity integrity is as follows: Entity extraction is performed on the transcribed text. The extracted entities include at least the control object, control action, parameter value, time qualifier, space qualifier, and modifier. All semantic slots that should be identified in the preset instruction template are taken as the slots to be identified. The total number of slots to be identified is counted. The slots to be identified refer to the set of semantic elements pre-defined for the current control statement, which are used to represent the complete semantic structure of a control instruction. Specifically, they are configured as device name slots, action slots, value slots, time slots, location slots, and modifier slots, etc., according to different instruction types. Each entity type is assigned a weight coefficient, with the sum of all weight coefficients being 1. The extracted entity categories are used as valid slots, and the total number of valid slots is counted. Each valid slot is matched with the pre-defined weight coefficient corresponding to the slot to be identified, and these weight coefficients are summed to obtain the effective weight. Simultaneously, the weight coefficients corresponding to all slots to be identified are summed to obtain the identification weight. The total number of slots to be identified M, the total number of valid slots N, the identification weight f2, and the effective weight f1 are then used to calculate the entity completeness C using a formula. ent The specific calculation method uses the following formula three:

[0014] By comprehensively reflecting the completeness of target control information in the transcribed text based on both slot quantity coverage and slot weight coverage, the system avoids ignoring the differences in key slot weights by judging solely by the number of slots.

[0015] Furthermore, the steps in S400 for performing permission verification and conflict resolution on the candidate instructions include: Instructions that pass the permission verification are determined as execution instructions, and all execution instructions falling within the preset time window are grouped into the same execution group; If there are conflicting instructions within the same execution group that target the same device or share control resources, they constitute a conflict set. For each instruction in the conflict set, a conflict priority score is calculated based on its corresponding role permission level value, semantic confidence, time priority, session task relevance, and device state adaptability. The conflict set is resolved based on the conflict priority score to determine the final retained execution instructions.

[0016] Furthermore, for each instruction in the conflict set, a conflict priority score is calculated: For conflicting instructions in the conflict set, at least the role permission level value, semantic confidence, time priority, session task relevance and device state adaptability corresponding to the conflicting instructions are obtained. The role permission level is determined by the permission level of the corresponding role account in the preset permission table. The role type includes at least administrator role, authorized user role and temporary role. The permission levels corresponding to administrator role, authorized user role and temporary role are mapped to the corresponding permission level values ​​respectively. Time priority is determined based on the generation timestamp of the conflicting instruction within the same execution group, and is calculated using a normalized inverse time method; The conversation task relevance is used to characterize the semantic consistency between the conflicting instruction and the current conversation topic. The task keywords confirmed in the current call, the control objects and control actions of the instructions executed in the most recent round or several rounds are concatenated and input into a preset text encoding model to obtain the task vector corresponding to the current conversation topic. The instruction vector corresponding to the conflicting instruction is obtained by concatenating the transcribed text of the conflicting instruction, the control objects, control actions and parameter values, and inputting it into the same text encoding model. The cosine similarity is used to calculate the similarity between the task vector and the instruction vector, which is used as the conversation task relevance. Device state adaptability is used to characterize the degree of matching between the conflicting instruction and the current state of the target device. It is judged based on the current state of the target device, the type of target action, and the legal range of parameters: if the conflicting instruction is fully adapted to the current device state, the device state adaptability value is one; if there is redundancy but it can still be executed, the device state adaptability value is 0.5; if it obviously conflicts with the current state or exceeds the executable range of the device, the device state adaptability value is zero. The role permission level value, semantic confidence, time priority, session task relevance and device state adaptability are normalized and then linearly weighted and fused according to preset weights to obtain the conflict priority score, with the sum of the weights being one.

[0017] The beneficial effects of this invention are: (1) By collecting, framing, calculating features such as short-time energy and zero-crossing rate, endpoint detection, echo cancellation and noise reduction of the mixed audio stream of the call, the mixed speech in the multi-person call is first organized into standardized speech segments, and then further demixed into several independent audio streams, thereby reducing the interference of noise, overlapping speech and echo on subsequent recognition at the source, ensuring the input stability of subsequent voiceprint recognition and semantic analysis, and providing the prerequisite for multi-role recognition; (2) By extracting MFCC voiceprint embeddings from each audio stream and combining them with the character voiceprint library for coarse screening of center similarity, fine comparison of maximum similarity and double threshold judgment, the binding between the current audio stream and the character account identifier is effectively completed, avoiding confusion of the speaking subject in multi-person calls; by writing the newly identified high-confidence voiceprint samples back to the character voiceprint library, the historical sample space can be continuously expanded, improving the robustness and adaptability in long-term use, taking into account both fast screening efficiency and fine recognition accuracy, and improving the tolerance to voiceprint fluctuations under different devices, different emotions and different vocal states; (3) By further breaking down the speech recognition results into text confidence, intent probability and entity completeness, the natural language transcription results are transformed into structured instruction objects, thereby reducing the problem of recognizing text but being unable to execute it; when the semantic confidence is lower than the threshold, it can also send back supplementary prompts in a timely manner to prevent low-confidence statements from directly entering the execution link, and elevate it from the text layer to the executable semantic layer, so that subsequent permission verification and conflict handling are based on a quantifiable and comparable basis; (4) By performing role permission verification, session task correlation analysis, timing logic judgment and conflict priority scoring on the execution instructions, the system effectively filters out unauthorized instructions, irrelevant instructions and conflicting instructions, and avoids the device being triggered incorrectly or switched repeatedly. At the same time, the system generates differentiated voice feedback for different roles based on the execution results, so that the call interaction changes from one-way control to closed-loop response, enhances interpretability and user perception consistency, further compresses candidate instructions into final executable instructions, and realizes conflict resolution and result feedback in multi-person concurrent control scenarios. Attached Figure Description

[0018] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an undue limitation of the invention. In the drawings: Figure 1 This is a schematic diagram of the method flow of the present invention. Detailed Implementation

[0019] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0020] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0021] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate for the embodiments of the invention described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0022] To make the objectives and advantages of the present invention clearer, the present invention will be further described below with reference to embodiments; it should be understood that the specific embodiments described herein are merely for explaining the present invention and are not intended to limit the present invention.

[0023] Please see Figure 1 As shown, this invention is a multi-role voice interaction control method based on AI calls, comprising the following steps: S100, after the AI ​​call is established, the mixed audio stream data and attribute information of the mixed audio stream data are acquired. The attribute information includes the voice acquisition timestamp, call channel identifier, device identifier or account identifier, current session status data, and target device status data. The mixed audio stream data is either single-channel mixed speech or multi-channel separated speech. In this embodiment, the more challenging single-channel mixed speech scenario is the main description object. The acquired mixed audio stream is segmented according to the preset frame length and frame shift. Specifically, in this example, the mixed audio stream data is segmented according to a frame length of 20ms and a frame shift of 10ms. The short-time energy, zero-crossing rate, spectral entropy, and signal-to-noise ratio of each frame are calculated to determine the effective speech segment and the non-speech segment. The signal-to-noise ratio is calculated based on the preceding silence frame or noise estimation window. The specific judgment process is as follows: when the short-time energy of a frame is lower than the preset silence threshold, and the zero-crossing rate and spectral entropy are both lower than the noise threshold, the frame is judged as a non-speech frame and discarded. When 5-10 consecutive frames meet the speech conditions, they are merged into a candidate speech. For each candidate speech segment, a speech endpoint detection algorithm is used to further refine the identification of the true start and end boundaries of the speech, eliminating any remaining silent segments, background noise segments, and invalid interference segments at the beginning and end of the segment to ensure the purity and integrity of the speech content. Then, echo cancellation, speech enhancement, and noise reduction processing are applied to obtain a standardized speech segment. It should be noted that the standardized speech segments are temporally independent, and each segment ideally corresponds to an independent sound source, i.e., a single speech by a speaker. However, in complex scenarios such as multiple speakers speaking simultaneously, the standardized speech segment is very likely to still contain the superposition of multiple sound source signals. To solve this problem, this embodiment uses a blind source separation algorithm for demixing. When multi-channel observation conditions are available, a method based on independent component analysis is used to demix and separate the standardized speech segments. The standardized speech segments of multiple speakers are demixed and separated into several statistically independent audio streams, each audio stream corresponding to an independent sound source. If the standardized speech segment is a single speaker's speech, then the standardized speech segment retains only one audio stream after demixing and separation. S200 extracts voiceprint features from the various audio streams contained in a standardized speech segment and matches them with a pre-defined role voiceprint database to determine the role to which each audio stream belongs. Specifically, this includes: The audio stream is segmented into frames. In this embodiment, a frame length of 20ms and a frame shift of 10m are used for segmentation. Mel-frequency cepstral coefficients (MFCCs) are extracted from each frame. Typically, 12-13 dimensional MFCC coefficients and their first and second-order differences are taken to form a 39-dimensional feature vector, resulting in the acoustic MFCC feature sequence of the audio stream. MFCC features can effectively simulate the nonlinear perception characteristics of human ears to sound frequencies and are the most widely used feature representation in voiceprint recognition. Then, a voiceprint recognition model based on a deep neural network is used to map the MFCC feature sequence into a fixed-dimensional voiceprint embedding vector, denoted as v. In this embodiment, the x-vector model is used. The voiceprint embedding vector represents the speaker's acoustic identity features in a compact numerical form. A pre-defined voiceprint database is established, pre-storing the identity identifiers of each registered role and their corresponding historical voiceprint feature vectors. The account identifier entering the current AI call is extracted, and the historical voiceprint feature vectors corresponding to each account identifier in the voiceprint database are extracted. Then, the K-Means clustering algorithm is used to cluster the historical voiceprint feature vectors, or the arithmetic mean is directly calculated to obtain the center voiceprint vector of that account identifier. The cosine similarity is used to calculate the center similarity S between the voiceprint embedding vector of the current audio stream and the center voiceprint vector of each account identifier. center The specific calculation formula is as follows:

[0024] Where v represents the voiceprint embedding vector of the current audio stream, and c represents the center voiceprint vector of the account identifier; thus, the center similarity between the voiceprint embedding vector of the audio stream and all account identifiers entering the current AI call can be obtained; a preset center similarity threshold is set. In this embodiment, by statistically analyzing the sample similarity distribution within the same account and the similarity distribution between different accounts in historical voiceprint data, the boundary between the two distribution intervals is selected as the center similarity threshold, and preferably set to 0.75, so that most non-same-origin accounts are filtered out at this threshold, while retaining the vast majority of potential matching accounts for subsequent fine comparison stage; if the center similarity is greater than the center similarity threshold, the corresponding account identifier is used as the candidate account of the current audio stream, so as to quickly filter out candidate accounts with high similarity, filter out obviously irrelevant accounts, and greatly reduce the amount of computation for subsequent fine comparison; For candidate accounts obtained after coarse screening by center similarity, the cosine similarity between the current audio stream's voiceprint embedding vector and all historical voiceprint feature vectors under that candidate account is calculated, and the maximum value is taken as the maximum similarity of that account. Thus, the maximum similarity S between the current audio stream's voiceprint embedding vector and all candidate accounts can be obtained. maxThe higher the similarity, the better the match. Taking the maximum value finds the historical sample of the account that is closest to the current voice, which is used to encompass the voiceprint fluctuations of the same account under different devices and emotions; then the maximum similarity S max Similarity to center S center The final similarity score of the accounts is obtained through fusion. The fusion method can be either weighted harmonic mean or weighted arithmetic mean. In this embodiment, weighted harmonic mean is used because it is more sensitive to lower individual scores, avoiding misjudgments caused by one score being abnormally high while another is extremely low. The specific calculation method uses the following formula:

[0025] Where ε is a very small constant to prevent division by zero errors, it is set to 10 in this embodiment. -8 A preset recognition threshold is established. Based on the fluctuation range between different historical voiceprint samples of the same account, the minimum separable distance between different accounts, and the tolerance for false recognition, this threshold is set as the final judgment threshold that can simultaneously take into account the consistency within the same account and the distinguishability between different accounts. In this example, the recognition threshold is set to 0.85. Among all candidate accounts, the candidate account with the highest final similarity and the final similarity being greater than the preset recognition threshold is selected as the final identity of the audio stream. If the final similarity is lower than the recognition threshold, it is determined to be an unregistered user, a temporary identity label is assigned to it, and it is prompted to register its voiceprint. After completing the role identity recognition... Each audio stream is bound to a corresponding role identity identifier, establishing a correspondence between audio stream and role identity. Continuous tracking of the role is initiated until the role exits the call channel. Simultaneously, the voiceprint embedding vector corresponding to the audio stream in the established audio stream-role identity correspondence is used as the historical voiceprint feature vector of the account identifier corresponding to the role identity and saved to the role voiceprint database. This allows for a richer and more up-to-date sample space to improve the robustness and accuracy of recognition when calculating the central voiceprint vector of the account and performing maximum similarity matching. Otherwise, the saving operation is not performed to ensure the data quality of the role voiceprint database. The S300 performs speech recognition and semantic understanding on each audio signal, and binds each audio signal to a corresponding role account identifier. The specific process is as follows: After binding the audio stream with the role account identifier, speech recognition and semantic understanding processing are performed on each audio signal that has been assigned to a role. Specifically, each audio signal is re-confirmed for speech segments according to the same frame length and frame shift as S100, and the start and end boundaries of a single speaking round are determined in combination with the aforementioned endpoint detection results to avoid semantic crosstalk between sentences. For each speaking round, the Automatic Speech Recognition (ASR) model is input for text transcription, and the transcribed text and its corresponding word-by-word confidence sequence are output. The text confidence is calculated according to the average or geometric mean of the word-by-word confidence sequence of the output transcribed text. In this embodiment, the arithmetic mean is preferred to be used to calculate the text confidence of the speech recognition in this round. The transcribed text is subjected to intent recognition, which is used to determine whether the statement belongs to the control, query, confirmation, cancellation, or explanatory category. In this embodiment, the control statement is further divided into several subcategories such as power on, power off, adjustment, mode switching, and delayed execution. The transcribed text is input into a pre-trained intent classification model to obtain the probability distribution sequence of each intent category. The intent category with the highest probability is taken as the current intent, and the highest probability is taken as the intent probability of the current intent. Entity extraction is performed on the transcribed text. Specifically, the extracted entities include at least the control object, control action, parameter value, time qualifier, space qualifier, and modifier. All semantic slots that should be identified in the preset instruction template are designated as the slots to be identified, and the total number of slots to be identified is denoted as M. Specifically, the slots to be identified refer to a pre-defined set of semantic elements for the current control statement, used to represent the complete semantic structure of a control instruction. These slots can be configured as device name slots, action slots, value slots, time slots, etc., depending on the instruction type. Slots include room slots, location slots, and decoration slots; for example, when the statement is: "Set the conference room air conditioner to 26 degrees," the extracted control object is the conference room air conditioner, the control action is "set to," and the parameter value is 26 degrees; when the statement is: "Dim the lights a little," the extracted control object is the lights, the action type is "adjust," and the parameter value is "relative decrease." In this embodiment, such relative parameters are converted into standard control quantities through preset mapping rules. For example, dimming a little is mapped to a minimum step decrease in brightness; to reflect the degree of influence of different entity categories on the executableness of instructions, separate settings are set for each entity type. The weighting coefficients, control object, control action, and parameter value determine whether the instruction can form a clear control loop and belong to the high-contribution slots. Time, space, and modifiers are mainly used to enhance scene constraints and semantic refinement and belong to the auxiliary slots; therefore, their weights are set relatively low to avoid excessive interference with the overall completeness. Specifically, in this embodiment, the weighting coefficients can be set as follows: control object 0.25, control action 0.25, parameter value 0.20, time qualifier 0.10, space qualifier 0.10, and modifier qualifier 0.10. The sum of the coefficients is 1; then, the extracted entity categories are used as valid slots, and the total number of valid slots is counted as N. Each valid slot is then matched with the preset weight coefficient corresponding to the slot to be identified to obtain the weight coefficient for each valid slot, and these coefficients are summed to obtain the effective weight, denoted as f1. Simultaneously, the weight coefficients corresponding to all slots to be identified are summed to obtain the identification weight, denoted as f2. The total number of slots to be identified M, the total number of valid slots N, the identification weight f2, and the effective weight f1 are then used to calculate the entity completeness C using a formula. ent The specific calculation method uses the following formula three:

[0026] By comprehensively reflecting the completeness of target control information in the transcribed text based on both slot quantity coverage and slot weight coverage, the situation of ignoring the key slot weight differences is avoided by judging solely by the number of slots. Then, a proportional coefficient is assigned to each of the text confidence, intent probability, and entity completeness, and the sum of these coefficients is one. The text confidence, intent probability, and entity completeness are then weighted and fused according to the assigned proportional coefficients to obtain the semantic confidence of the transcribed text. In this embodiment, the proportional coefficients for text confidence, intent probability, and entity completeness are set to 0.40, 0.35, and 0.25, respectively, and can be adjusted according to actual application requirements. Specifically, text confidence characterizes the consistency between the speech recognition result and the original speech content; intent probability characterizes the credibility of the judgment of the control intent to which the current statement belongs; and entity completeness characterizes the completeness of the recognition of key slots such as the control object, control action, and parameter values. The semantic confidence obtained after the weighted fusion of these three factors comprehensively reflects the comprehensibility, decidability, and executability of the current transcribed text, avoiding misjudgments caused by relying on only a single indicator. A preset confidence threshold is used to determine whether the semantic information of the current transcribed text meets the requirements of subsequent generated instructions. In this embodiment, the confidence threshold is set to 0.80. If the semantic confidence of the transcribed text is greater than or equal to the confidence threshold, it means that the transcribed text has met the preset reliability requirements in terms of speech recognition, intent recognition, and entity extraction. The current statement can be determined to have complete control semantics. Then, the role account identifier, transcribed text, control object, control action, parameter value, generation timestamp, and semantic confidence are encapsulated into a candidate instruction for subsequent permission verification and execution. If the semantic confidence of the transcribed text is less than the confidence threshold, it indicates that the current statement is semantically incomplete, the intent is unclear, or key entities are missing. In this case, the semantic incompleteness prompt will be fed back to the corresponding role account identifier to guide the role to supplement the explanation, reconfirm, or continue to send the latest audio stream until the semantic confidence of the transcribed text corresponding to the collected subsequent audio stream is greater than or equal to the confidence threshold, or if the confidence threshold cannot be reached after a preset number of supplements, the current round of candidate instruction generation process will be terminated. In this way, the audio streams bound to each role can be converted into candidate instructions and sent to S400. S400: Based on role attribute information, instruction permission verification is performed, execution instructions are output, and an execution instruction queue is formed. The execution instruction queue is subjected to temporal logic judgment and conflict resolution. If there is no conflict, all execution instructions are executed directly; if a conflict exists, it is resolved. The final execution instruction is obtained according to the resolved execution instructions. After the instruction is complete, for each role that issued the instruction, corresponding voice response content is generated based on the execution result of the instruction. The execution result includes, but is not limited to, successful execution, suppression due to conflict, and rejection due to insufficient permissions. The response content is generated using a text generation model. In this embodiment, the text generation model is specifically a dialogue generation model based on a large language model. The specific process includes: Based on role attribute information, permission verification is performed on each candidate instruction. Specifically, the role type, permission level, allowed control device set, allowed action set, and allowed parameter range under the corresponding role account identifier are read. The control object, control action, and parameter value in the candidate instruction are matched with a preset permission table. If the control object belongs to the range of devices that the role can control, the control action belongs to the range of actions that the role can execute, and the parameter value is within the allowed parameter range, then the candidate instruction is determined to have execution permission. If any of the control object, control action, or parameter value does not meet the preset permission conditions, then the candidate instruction is determined to not have execution permission and is marked as an instruction with insufficient permission. To facilitate subsequent scheduling and processing, this embodiment sets a permission matching result identifier for each candidate instruction. When the permission matching result identifier is successful, the candidate instruction is converted into an execution instruction. When the permission matching result identifier is unsuccessful, it does not enter the execution stage but directly enters the feedback generation stage, outputting an insufficient permission prompt to the corresponding role. This ensures that all instructions entering the execution process meet the role authorization boundaries, avoiding erroneous operations caused by lack of permission control from the source. For execution instructions that pass the permission verification, the execution instruction queue is further subjected to timing logic judgment and conflict resolution. Specifically, the execution instruction queue is grouped by a preset time window. In this embodiment, a 5-second time window is preferred, and multiple execution instructions within the time window are regarded as the same execution group. Subsequently, the target object consistency analysis is performed on each instruction in the same execution group. If two or more instructions are for different target devices and there is no resource dependency between the devices, they are determined to be conflict-free instructions and can be executed in parallel. If two or more instructions are for the same target device or for a set of devices with shared control resources, they are determined to be conflicting instructions and are formed into a conflict set, which needs to be further resolved. The conflict types include at least the conflict of opposite actions of the same device, the conflict of different settings of the same parameter, the conflict of repeated occupation of the same resource, and the conflict of the order of dependencies. For example, if one instruction requires: turn on the projector, and another instruction requires: turn off the projector, then the two instructions constitute an opposite action conflict. If one instruction requires: adjust the temperature to 24 degrees Celsius, and another instruction requires: adjust the temperature to 28 degrees Celsius, then the parameter setting conflict exists. After a conflict is detected, conflict resolution is performed. The specific process is as follows: S401: If conflicting instructions in the conflict set can achieve a unified control result through parameter fusion, then fusion resolution is performed according to the device type. For example, temperature parameters are weighted averaged and rounded, brightness parameters are incremented or decremented by the minimum step size, and switch parameters are only allowed to retain one main execution instruction without numerical fusion. If the instructions in the conflict set cannot be executed in parallel or achieve a unified result through parameter fusion, then S402 is executed. S402, calculate the conflict priority score for each execution instruction in the conflict set within the same execution group. The priority score is determined by at least the role permission level value, semantic confidence, time order, and task relevance. The conflict priority score is used to characterize the reserved execution priority of the execution instruction in the conflict. The higher the score, the higher the priority of execution. The specific analysis and calculation process is as follows: For conflicting instructions in the conflict set, at least the corresponding role permission level value, semantic confidence, time priority, session task relevance, and device state adaptability are obtained. The role permission level is determined by the permission level of the corresponding role account identifier in the preset permission table. Role types include at least administrator roles, authorized user roles, and temporary roles. The role permission level values ​​corresponding to administrator roles, authorized user roles, and temporary roles are mapped to 1, 0.67, and 0.33, respectively, to achieve a quantitative expression of the control capabilities of different roles. The semantic confidence is taken from the semantic confidence obtained in S300 and is used to characterize the overall reliability of the instruction at the speech recognition, intent recognition, and entity extraction levels, with a value range of 0 to 1. The time priority is determined based on the generation timestamp of the conflicting instruction in the same execution group, preferably calculated using a normalized inverse time method. Specifically, the calculation method uses the following formula four:

[0027] Where t represents the timestamp of the conflicting instruction, tmin represents the timestamp of the earliest instruction in the execution group, tmax represents the timestamp of the latest instruction in the execution group, and ε is a very small constant to prevent division by zero, which is taken as 10 in this embodiment. 8 In this way, the earlier an instruction is issued, the higher its time priority; The conversation task relevance is used to characterize the semantic consistency between the conflicting instruction and the current conversation topic. Specifically, the task keywords confirmed in the current call, the control objects and control actions of the instructions executed in the most recent round or several rounds are concatenated and input into a preset text encoding model to obtain the task vector corresponding to the current conversation topic. The instruction vector corresponding to the conflicting instruction is obtained by concatenating the transcribed text of the conflicting instruction, the control object, the control action, and the parameter values, and inputting it into the same text encoding model. The specific text encoding model adopts BERT, RoBERTa, or sentence vector encoding model, and its output vector dimension is preferably 768 dimensions. Each dimension of the vector is used to characterize the semantic features of the text. The cosine similarity is used to calculate the similarity between the task vector and the instruction vector, which is used as the conversation task relevance. The higher the conversation task relevance, the more consistent the conflicting instruction is with the current conversation control objective, and the more it should be retained in the conflict resolution process. Device state adaptability characterizes the degree of matching between a conflicting command and the current state of the target device. Specifically, it is determined based on the current state of the target device, the type of the target action, and the legal range of parameters: if the conflicting command is perfectly compatible with the current device state, the device state adaptability is 1; if there is slight redundancy but it is still executable, the device state adaptability is 0.5; if it clearly conflicts with the current state or exceeds the device's executable range, the device state adaptability is 0. For example, if the air conditioner's current temperature is 24 degrees Celsius, issuing a command to adjust it to 24 degrees Celsius has a low state adaptability; while issuing a command to adjust it to 26 degrees Celsius has a high state adaptability.

[0028] After obtaining the above parameters, each parameter is normalized, and the conflict priority score is calculated by linearly weighting and fusing the role permission level value, semantic confidence, time priority, device state adaptability, and session task relevance according to preset weights, with the sum of the weights being one. In this embodiment, the weights of permission level, semantic confidence, time priority, session task relevance, and device state adaptability are set to 0.30, 0.20, 0.15, 0.20, and 0.15, respectively, to make the influence of role permissions and semantic credibility on conflict priority more prominent, while taking into account timing, task relevance, and state adaptability. Therefore, the conflict priority score is not determined solely by a single parameter, but is obtained by comprehensively considering the role authorization level, semantic recognition reliability, instruction generation order, current session task relevance, and target device state adaptability. S403: Sort the conflicting instructions in the conflict set according to their corresponding priority scores, select the highest priority score and subtract the second highest priority score to obtain the priority difference. If the priority difference is greater than or equal to the preset priority difference threshold, the instruction with the highest conflicting priority score is retained, and the remaining instructions are marked as suppressed instructions and do not enter the actual control execution flow. If the priority difference is less than the preset priority difference threshold, it is determined to be an instruction with similar priorities, and further triggers manual confirmation or voice confirmation to avoid erroneous execution due to similar priorities.

[0029] After conflict resolution, a final execution sequence is generated according to the resolved instructions. This sequence is then converted into control messages recognizable by the target device's protocol format and sent to the appropriate device for execution. Upon completion, the device returns an execution result, which includes at least the following types: successful execution, execution failure, parameter out of bounds, device busy, device offline, permission denied, and conflict suppressed. After all instructions are executed, a corresponding voice response is generated for each issuing role. Specifically, the response semantic elements are first constructed based on the execution result for that role. These elements include at least the role's identity, the original instruction content, the execution status, the cause of the conflict, the reason for the permission restriction, the current status of the target device, and suggested actions. These semantic elements are then input into a dialogue generation model based on a large language model, which generates a response text adapted to the current communication context. If a character's command is successfully executed, the response text informs the character that "the command has been completed" and the device's status result after execution. If a character's command is suppressed due to a conflict, the response text informs the character that "a conflict exists, your command has not yet been executed, please confirm whether to continue." If a character's command is rejected due to insufficient permissions, the response text informs the character that "there is currently no control permission for this item, please contact the authorized character for processing." If a character's command fails to execute due to out-of-bounds parameters or device malfunction, the response text further indicates the specific reason for the failure. After generating the response text, the speech synthesis module converts the response text into speech and outputs it to the corresponding character, thus forming a complete closed-loop interactive process of command input, permission verification, conflict resolution, and execution feedback, realizing multi-role voice interactive control.

[0030] The above are merely embodiments of the present invention and are not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principle of the present invention should be included within the scope of the claims of the present invention.

Claims

1. A multi-role voice interaction control method based on AI calls, characterized in that, Includes the following steps: S100, after the AI ​​call is established, the mixed audio stream data and its attribute information of the call are obtained, and the mixed audio stream data is processed by frame segmentation, speech segment detection, endpoint detection, echo cancellation, speech enhancement and noise reduction to obtain standardized speech segments. In a multi-person speaking scenario, blind source separation is performed on the standardized speech segments to obtain several independent audio streams; S200, extract voiceprint features from each audio stream, match the extracted voiceprint features with a preset role voiceprint database, determine the role identity of each audio stream, and establish a correspondence between audio streams and role identities; S300 performs speech recognition and semantic understanding on each audio stream that has been bound to a role identity, extracts the control object, control action and parameter value and calculates semantic confidence to generate candidate instructions; S400: Based on the role attribute information, the candidate instructions are verified for permissions and conflict resolution is performed to generate the final execution instruction, and voice feedback is output to the corresponding role according to the execution result.

2. The multi-role voice interaction control method based on AI calls according to claim 1, characterized in that, In S100: The mixed audio stream is segmented according to a preset frame length and frame shift, and the short-time energy, zero-crossing rate, spectral entropy and signal-to-noise ratio of each frame are calculated. When the short-time energy of a frame is lower than a preset silence threshold, and the zero-crossing rate and spectral entropy are both lower than the corresponding noise threshold, the frame is determined to be a non-speech frame and discarded. When a preset number of consecutive frames meet the speech frame condition, they are merged into candidate speech segments.

3. The multi-role voice interaction control method based on AI calls according to claim 2, characterized in that, The S100 further includes: The candidate speech segments are processed by a speech endpoint detection algorithm to determine the speech start and end boundaries, and then by echo cancellation, speech enhancement and noise reduction to obtain the standardized speech segments. The blind source separation uses an algorithm based on independent component analysis to demix and separate standardized speech segments with multiple sound sources to obtain several statistically independent audio streams, each audio stream corresponding to an independent sound source.

4. The multi-role voice interaction control method based on AI calls according to claim 3, characterized in that, The steps for voiceprint feature extraction in S200 include: After the audio stream is segmented into frames, the Mel frequency cepstral coefficients (MFCCs) and their first and second-order differences are extracted to obtain the MFCC feature sequence. The MFCC feature sequence is then mapped into a fixed-dimensional voiceprint embedding vector using a deep neural network-based voiceprint recognition model.

5. A multi-role voice interaction control method based on AI calls according to claim 4, characterized in that, The steps for voiceprint matching and role identification in S200 include: The preset voiceprint database stores the account identifiers of each registered role and their corresponding historical voiceprint feature vectors; for each account identifier entering the current AI call, the center voiceprint vector is calculated based on its historical voiceprint feature vector. Calculate the center similarity between the current audio stream's voiceprint embedding vector and each of the central voiceprint vectors, and identify the account identifiers whose center similarity is greater than a preset center similarity threshold as candidate accounts.

6. A multi-role voice interaction control method based on AI calls according to claim 5, characterized in that, The steps for voiceprint matching and character identification in S200 also include: For each candidate account, calculate the cosine similarity between the current audio stream's voiceprint embedding vector and all historical voiceprint feature vectors under that candidate account, and take the maximum value as the maximum similarity of that candidate account; calculate the final similarity by weighted harmonic average of the maximum similarity and the center similarity. The candidate account with the highest final similarity and greater than or equal to the preset recognition threshold is selected, and its corresponding role account identifier is determined as the final identity of the audio stream. The voiceprint embedding vector corresponding to the audio stream is written back to the role voiceprint library. If there is no candidate account with a final similarity greater than or equal to the preset recognition threshold, the current speaker is determined to be an unregistered user, and a temporary identity label is assigned to him / her.

7. A multi-role voice interaction control method based on AI calls according to claim 6, characterized in that, Specifically, S300 includes: Speech recognition is performed on each audio stream to obtain transcribed text, and the text confidence score is calculated. The transcribed text is subjected to intent recognition to obtain the intent category and the corresponding intent probability; Entity extraction is performed on the transcribed text, and slot analysis is conducted to calculate entity completeness. The semantic confidence of the transcribed text is obtained by weighted fusion of the text confidence, the intent probability, and the entity completeness. When the semantic confidence level is greater than or equal to the preset semantic confidence threshold, the transcribed text and its corresponding role account identifier, control object, control action, parameter value, generation timestamp, and semantic confidence level are encapsulated into a candidate instruction.

8. A multi-role voice interaction control method based on AI calls according to claim 7, characterized in that, The method for calculating the entity integrity is as follows: Entity extraction is performed on the transcribed text. The extracted entities include at least the control object, control action, parameter value, time qualifier, space qualifier, and modifier. All semantic slots that should be identified in the preset instruction template are taken as the slots to be identified. The total number of slots to be identified is counted. The slots to be identified refer to the set of semantic elements pre-defined for the current control statement, which are used to represent the complete semantic structure of a control instruction. Specifically, they are configured as device name slots, action slots, value slots, time slots, location slots, and modifier slots, etc., according to different instruction types. Each entity type is assigned a weight coefficient, and the sum of the weight coefficients is 1. The extracted entity categories are used as valid slots, and the total number of valid slots is counted. Each valid slot is matched with the weight coefficient corresponding to the preset slot to be identified to obtain the weight coefficient corresponding to each valid slot. The effective weight is obtained by summing the weight coefficients. Simultaneously, the weight coefficients corresponding to all slots to be identified are summed to obtain the identification weight. The total number of slots to be identified, the total number of valid slots, the identification weight, and the valid weight are then calculated using a formula to obtain the entity completeness.

9. A multi-role voice interaction control method based on AI calls according to claim 8, characterized in that, The steps of performing permission verification and conflict resolution on the candidate instructions in S400 include: Instructions that pass the permission verification are determined as execution instructions, and all execution instructions falling within the preset time window are grouped into the same execution group; If there are conflicting instructions within the same execution group that target the same device or share control resources, they constitute a conflict set. For each instruction in the conflict set, a conflict priority score is calculated based on its corresponding role permission level value, semantic confidence, time priority, session task relevance, and device state adaptability. The conflict set is resolved based on the conflict priority score to determine the final retained execution instructions.

10. A multi-role voice interaction control method based on AI calls according to claim 9, characterized in that, For each instruction in the conflict set, calculate the conflict priority score: For conflicting instructions in the conflict set, at least the role permission level value, semantic confidence, time priority, session task relevance and device state adaptability corresponding to the conflicting instructions should be obtained, and the permission levels corresponding to the administrator role, authorized user role and temporary role should be mapped to the corresponding permission level values ​​respectively. Time priority is determined based on the generation timestamp of the conflicting instruction within the same execution group, and is calculated using a normalized inverse time method; The conversation task relevance is used to characterize the semantic consistency between the conflicting instruction and the current conversation topic. The task keywords confirmed in the current call, the control objects and control actions of the instructions executed in the most recent round or several rounds are concatenated and input into a preset text encoding model to obtain the task vector corresponding to the current conversation topic. The instruction vector corresponding to the conflicting instruction is obtained by concatenating the transcribed text of the conflicting instruction, the control objects, control actions and parameter values, and inputting it into the same text encoding model. The cosine similarity is used to calculate the similarity between the task vector and the instruction vector, which is used as the conversation task relevance. Device state adaptability is used to characterize the degree of matching between the conflicting instruction and the current state of the target device. It is judged based on the current state of the target device, the target action type, and the legal range of parameters: if the conflicting instruction is fully adapted to the current device state, the device state adaptability value is one. If redundancy exists but execution is still possible, the device state adaptability is set to 0.

5. If there is a clear conflict with the current state or it exceeds the scope of the device's execution, the device state adaptability is set to zero. The role permission level value, semantic confidence, time priority, session task relevance and device state adaptability are normalized and then linearly weighted and fused according to preset weights to obtain the conflict priority score, with the sum of the weights being one.

Images