A method and apparatus for dynamically switching interaction rights

By using parallel multimodal data processing channels to dynamically adjust weights and generate interactive state events, the problems of incorrect switching of the main interactive person and missed identification of candidate interactive persons in human-computer interaction systems in noisy multi-person scenarios are solved, achieving natural, smooth switching of interactive rights and continuity.

CN123114407APending Publication Date: 2026-11-17AISPEECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611242840.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-08-17
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

Existing human-computer interaction solutions struggle to accurately identify and dynamically switch the main interactor in scenarios with multiple users, noise, and dynamic user changes, leading to problems such as missed identification of candidate interactors and poor interaction continuity.

Method used

By operating in parallel through the main interactive user data processing channel and the candidate interactive user intent monitoring channel, and combining multimodal data such as visual and audio information, interactive state events are generated, and weights are dynamically adjusted to achieve voice enhancement for the main interactive user and intent monitoring for candidate interactive users, ensuring the continuity and accuracy of the interaction.

Benefits of technology

It effectively solves the problem of missing candidate interaction person identification, realizes the ability to perceive potential interaction needs in advance without interrupting the current interaction, improves the accuracy and smoothness of interaction judgment, and enhances the robustness of the system in complex environments.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123114407A_ABST
    Figure CN123114407A_ABST
Patent Text Reader

Abstract

This invention discloses a method and apparatus for dynamically switching interaction rights. The method includes: acquiring interaction scene data and determining the primary interactor and candidate interactors; performing voice enhancement and intent monitoring through the primary interactor data processing channel to obtain enhanced voice and primary interactor confidence; monitoring the intent of candidate interactors through a parallel-running candidate interactor intent monitoring channel to obtain candidate interactor confidence; and generating and executing interaction state events based on the enhanced voice and confidence data, wherein the events include primary interactor hold, candidate interactor attention, or primary interactor switching. This invention, through a dual-channel parallel architecture, actively monitors the intent of candidate interactors while enhancing the primary interactor's voice, reducing missed recognition and false switching rates, and improving the continuity of interaction in noisy multi-person scenarios.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of human-computer interaction technology, and in particular to a method and apparatus for dynamic switching of multimodal interaction rights. Background Technology

[0002] With the rapid development of artificial intelligence technology, human-computer interaction systems have been widely used in service robots, intelligent conference terminals, intelligent customer service, and other scenarios. In real-world interaction scenarios involving multiple people, noise, and dynamic user changes, the system needs to accurately identify the main user currently interacting with and dynamically switch to other users with interactive intentions when appropriate to ensure the continuity and naturalness of the interaction.

[0003] Currently, existing human-computer interaction solutions mainly fall into the following categories. One category is a single-modal solution based on wake words and microphone array sound source localization. Users initiate interaction by uttering a specific wake word, and the system estimates the speaker's orientation using the microphone array to determine the interaction target and respond accordingly. Another category is a single-visual modal solution, which uses a camera to detect and track faces to identify the primary interactor, using the face tracking results as the basis for locking onto the interaction target. However, in real-world application scenarios with multiple users, noise, and dynamic user changes, these solutions generally suffer from the following shortcomings: Existing solutions often only focus on the current interactor or rely on wake-up triggers for interaction target determination, making it difficult to proactively discover and evaluate the access intentions of other candidate interactors while continuously enhancing the primary interactor's voice, leading to the problem of missed candidate interactors. When multiple users simultaneously have the intention to interact, relying solely on wake words or a single modality for judgment can easily lead to erroneous switching of the primary interactor. Wake-word-dependent solutions have poor wake-up performance in noisy environments and require users to actively initiate interaction, resulting in insufficient naturalness of the interaction. Single-visual solutions cannot simultaneously solve the problems of voice quality improvement and interaction intent judgment. Summary of the Invention To at least address the problems of erroneous switching of the main interactor, missed identification of candidate interactors, and poor continuity of multi-person interaction in the prior art, embodiments of the present invention provide a method and apparatus for dynamic switching of interaction rights, which are used to solve the above-mentioned technical problems, or at least partially solve the above-mentioned technical problems.

[0004] According to a first aspect of the present invention, a method for dynamically switching interaction rights is provided. The method involves parallel operation of a primary interactor data processing channel and a candidate interactor intent monitoring channel. While enhancing the voice of the current primary interactor, the method monitors the interaction intent of candidate interactors and generates an interaction state event based on the enhanced voice data, confidence data, and confidence data of the primary interactor and the candidate interactors, thereby realizing the dynamic switching of natural interaction rights in a multi-person scenario.

[0005] Specifically, the process includes: acquiring interaction scenario data; determining the current primary interactor and at least one candidate interactor based on the interaction scenario data; performing voice enhancement and interaction intent monitoring on the current primary interactor through the primary interactor data processing channel to obtain enhanced voice data and confidence data of the current primary interactor; monitoring the interaction intent of the at least one candidate interactor through the candidate interactor intent monitoring channel to obtain confidence data of the at least one candidate interactor; and obtaining and executing interaction state events based on the enhanced voice data of the current primary interactor, the confidence data of the primary interactor, and the confidence data of the at least one candidate interactor; wherein, the interaction state events include at least one of the following: primary interactor hold event, candidate interactor attention event, or primary interactor switch event. The primary interactor data processing channel and the candidate interactor intent monitoring channel operate in parallel.

[0006] In some possible embodiments, the interactive scene data includes arrayed voice data, as well as at least one of video data, distance information, touch interaction information, text interaction information, or historical interaction context.

[0007] In some possible embodiments, the main interactor data processing channel extracts visual feature data and audio feature data corresponding to the current main interactor from the interaction scene data, and based on the visual feature data and the audio feature data, separates and / or enhances the voice data of the current main interactor from the array voice data to obtain the enhanced voice data, and monitors the main interactor's intention to maintain interaction to obtain the confidence data of the main interactor.

[0008] In some possible embodiments, the visual feature data used to separate and / or enhance the speech of the current main interactive person includes the lip movement features of the current main interactive person, and the audio feature data includes the sound source location information and / or voiceprint information of the current main interactive person.

[0009] In some possible embodiments, the candidate interaction intent monitoring channel extracts candidate interaction intent feature data from the interaction scene data for each candidate interaction user, and determines the confidence level data of the corresponding candidate interaction user based on the candidate interaction intent feature data. The candidate interaction intent feature data includes at least one of facial identity information, gaze direction, gesture action, human posture, voice wake-up result, semantic intent result, lip movement state, dwell time or proximity trend.

[0010] In some possible embodiments, the confidence data of the main interactor and the confidence data of the candidate interactor can be generated by dynamic weight fusion, including: obtaining the normalized interaction intent score and reliability coefficient of each modality corresponding to the main interactor or the candidate interactor in the interaction scene data; determining the fusion weight value corresponding to each modality based on the basic weight value in the current scene and the reliability coefficient; and calculating the confidence data based on the normalized interaction intent score and the fusion weight value.

[0011] In some possible embodiments, the confidence data within a continuous time window can be time-smoothed, and the interactive state event can be generated based on the time-smoothed confidence data.

[0012] In some possible embodiments, the main interactor hold event is executed when the confidence data of the main interactor meets the main interactor hold condition and the confidence data of at least one candidate interactor does not meet the switch condition; the candidate interactor follow event is executed when the confidence data of at least one candidate interactor meets the follow condition but does not meet the switch condition; and the main interactor switch event is executed when the confidence data of the main interactor meets the main interactor exit condition and the confidence data of the target candidate interactor meets the switch condition.

[0013] In some possible embodiments, before generating the interaction state event, cross-modal consistency analysis and / or contextual reasoning may be performed on at least one of the video data, array voice data, distance information, touch interaction information, text interaction information, or historical interaction context in the interaction scene data, and auxiliary arbitration may be performed on the confidence data of the main interactor and the confidence data of the candidate interactor based on the cross-modal consistency analysis and / or the contextual reasoning.

[0014] In some possible embodiments, the method further includes: monitoring the quality of each modal signal corresponding to the interactive scene data, and reducing the fusion weight value of the corresponding modality when the quality of at least one modal signal meets the abnormal conditions, wherein the quality of the modal signal includes at least one of image sharpness, lighting conditions, occlusion state, lip visibility, speech signal-to-noise ratio, and speech endpoint stability.

[0015] In some possible embodiments, when a visual modality is detected to be unavailable for an extended period of time, the fusion weight value of the modal features obtained from video data is reduced, and the fusion weight of at least one of array speech, voiceprint information, sound source location information, or semantic information is increased.

[0016] According to a second aspect of the present invention, an interactive right dynamic switching device is provided, comprising: a data acquisition module for acquiring interactive scene data in an interactive scene; an object determination module for determining a current main interactor and at least one candidate interactor based on the interactive scene data; a main interactor data processing module for performing voice enhancement processing and maintaining interactive intent monitoring on the current main interactor to obtain enhanced voice data and confidence data of the current main interactor; a candidate interactor intent monitoring module for monitoring the interactive intent of the at least one candidate interactor to obtain confidence data of the at least one candidate interactor; and an event execution module for obtaining and executing an interactive state event based on the enhanced voice data of the current main interactor, the confidence data of the main interactor, and the confidence data of the at least one candidate interactor; wherein the interactive state event includes at least one of a main interactor hold event, a candidate interactor attention event, or a main interactor switch event. The main interactor data processing module and the candidate interactor intent monitoring module operate in parallel.

[0017] In some possible embodiments, the confidence data of the primary interactor and the confidence data of the candidate interactors are generated in the following manner: Obtain the normalized interaction intent score and reliability coefficient of each modality corresponding to the main interactor or the candidate interactor in the interaction scene data; Based on the base weight values ​​in the current scenario and the reliability coefficient, determine the fusion weight values ​​corresponding to each modality; Based on the normalized interaction intent score and the fusion weight value, the confidence data of the main interactor and the confidence data of the candidate interactors are calculated.

[0018] In some possible embodiments, it also includes: The confidence data of the main interactor and the confidence data of the candidate interactors within a continuous time window are subjected to time smoothing. The interaction state event is generated based on the confidence data of the main interactor and the confidence data of the candidate interactors after time smoothing. In some possible embodiments, the apparatus may further include an anomaly handling module for reducing the fusion weight value of the corresponding mode when the quality of at least one mode signal meets an anomaly condition.

[0019] According to a third aspect of the present invention, an electronic device is provided, comprising at least one processor and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the steps of the above-described dynamic switching method for interactive rights.

[0020] According to a fourth aspect of the present invention, a storage medium is provided having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the above-described dynamic switching method for interactive rights.

[0021] According to the present invention, the main interactor data processing channel and the candidate interactor intent monitoring channel operate in parallel, which can continuously enhance the current main interactor's voice and ensure the quality of the interaction while continuously monitoring and evaluating the interaction intent of candidate interactors. This allows the system to perceive potential interaction needs in advance without interrupting the current interaction, effectively solving the problem of missed identification of candidate interactors caused by focusing only on the current interactor in the prior art.

[0022] Furthermore, through a multimodal fusion confidence generation mechanism based on dynamic weight adjustment, the system can integrate multi-dimensional information such as facial identity, gaze direction, gestures, body posture, voice wake-up, semantic intent, and lip movement state to comprehensively judge the strength of the user's interaction intent. Compared with the judgment methods of single-modal or fixed-weight fusion in the prior art, the solution of this invention can more accurately distinguish between a user's brief passing by and a stop to prepare for interaction, reduce frequent false switching caused by momentary false detections, and improve the accuracy of interaction judgment.

[0023] Furthermore, through an interaction state event generation mechanism based on the confidence data of the main interactor and the candidate interactors, the system can output structured events such as main interactor maintaining contact, candidate interactor paying attention, or main interactor switching, rather than simple person identification results. This allows the decision to switch interaction rights to be based on a comprehensive analysis of the intensity of the interaction intentions of both parties, rather than a single triggering event, thereby achieving smooth and natural turn-taking transitions and solving the problems of abrupt switching and poor interaction continuity in existing technologies.

[0024] By using an anomaly handling mechanism, the quality of each modal signal is monitored in real time and the weights are dynamically adjusted. When a short-term anomaly occurs in a certain modality, it can be compensated by timing smoothing and dynamic weight reduction. When a certain modality is unavailable for a long time, the system can be continuously available through modality switching and rollback strategies, thereby improving the robustness and practicality of the system in complex real-world environments. Attached Figure Description

[0025] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0026] Figure 1 This is a flowchart of an embodiment of the interactive right dynamic switching method of the present invention; Figure 2 This is a detailed flowchart illustrating an example of voice enhancement processing in the main interactive human data processing channel according to an embodiment of the present invention. Figure 3 This is a flowchart illustrating an example of confidence data generation and dynamic weight adjustment according to an embodiment of the present invention; Figure 4 This is a flowchart illustrating an example of anomaly detection and adaptive processing according to an embodiment of the present invention; Figure 5 This is a schematic block diagram of an embodiment of the interactive right dynamic switching device of the present invention; Figure 6 This is a schematic block diagram of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0028] The method embodiments of the present invention can be implemented in software, hardware, or a combination of both. In the present invention, "module," "system," etc., refer to entities applied to a computer, including hardware, software, or a combination of both.

[0029] Finally, it should be noted that in this document, the terms "comprising" and "including" include not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0030] Figure 1 This is a flowchart of a dynamic switching method for interaction rights according to an embodiment of the present invention. Figure 1 As shown, the method includes the following steps.

[0031] In step S110, interactive scene data is acquired.

[0032] The interactive scene data refers to multimodal raw data synchronously collected by multiple sensors in an interactive scene. This data includes arrayed voice data, as well as at least one of video data, distance information, touch interaction information, text interaction information, or historical interaction context. Specifically, video data can be collected through at least one of an RGB camera, a depth camera, or a binocular camera, and used for subsequent visual tasks such as face detection, lip movement analysis, human pose estimation, and gesture recognition. Arrayed voice data can be synchronously collected through a multi-channel microphone array at a preset sampling rate. The microphone signals are converted from analog to digital to form a multi-channel time-domain audio stream, providing multi-channel acoustic information for sound source localization, beamforming, and target speech separation. Distance information can be collected through a distance sensor to assist in determining the spatial relationship between the user and the interactive device. The distance sensor can be, for example, an ultrasonic sensor, an infrared sensor, or a lidar sensor. Touch interaction information can be collected through a touch sensor or touchscreen, reflecting the user's interactive intent initiated through physical contact.

[0033] Furthermore, since the sampling frequencies, data formats, and transmission delays of different modal sensors vary, the system needs to perform time synchronization and spatial calibration on the interactive scene data. Regarding time synchronization, the system can align the data of each modality to a unified time reference using hardware synchronization signals, such as using a unified clock source to trigger synchronous acquisition by each sensor; or it can align the data of each modality to a unified time reference using software timestamp alignment, such as marking the acquisition time under a unified clock domain for each frame of data. Regarding spatial calibration, the system can obtain the intrinsic and extrinsic parameter matrices of the camera through camera calibration, obtain the spatial coordinates of each microphone unit through microphone array calibration, and establish a cross-modal spatial mapping relationship between the visual coordinate system and the acoustic coordinate system through camera-microphone joint calibration, so that the position of the person detected by vision can correspond to the sound source localization result in a unified spatial coordinate system.

[0034] In step S120, the current main interactor and at least one candidate interactor are determined based on the interaction scenario data.

[0035] The current primary interactor refers to the interaction object that the system currently locks onto and prioritizes, whose voice is enhanced and whose interaction intent is continuously tracked. The candidate interactor refers to a user in the interaction scenario who has potential interaction intent but has not yet been switched to the primary interactor by the system. The system can detect, track, and identify people in the scene based on multimodal perception results, and assign a unique tracking identifier to each detected person. Specifically, the system can detect facial regions in a scene from video data using a face detection model, extract facial feature vectors using a face recognition model, and match them with a registered identity database or establish temporary identities through online clustering. Simultaneously, the system can continuously locate and track people in the scene using a human detection and tracking model, such as a multi-target tracking framework based on object detection and Kalman filtering. Combining the sound source localization results, the system cross-modally associates the identities of visually detected people with the location of sound sources, establishing a joint audiovisual representation for each person. The sound source localization results can be obtained, for example, by estimating the azimuth angle of each sound source using the time difference of arrival of array speech data. Based on this, the system determines the current primary interactor according to a preset initial interaction strategy and identifies at least one of the remaining detected people as candidate interactors. The initial interaction strategy can be, for example, selecting the person closest to the device, the first to wake up, or the first to gaze at the device as the primary interactor.

[0036] In step S130, voice enhancement and interaction intent monitoring are performed on the current main interactor through the main interactor data processing channel to obtain the enhanced voice data and confidence data of the current main interactor.

[0037] The primary interactor data processing channel is one of the core processing channels of the system. Its main function is to separate and enhance the voice signal of the current primary interactor from the mixed speech by fusing visual and audio multimodal information, while continuously monitoring whether the current primary interactor maintains the interaction intent, and outputting the enhanced voice data and the primary interactor's confidence data. The primary interactor data processing channel runs in parallel with the candidate interactor intent monitoring channel described later, that is, the two channels process the same interaction scenario data concurrently within the same time window, rather than executing them serially and alternately.

[0038] Specifically, such as Figure 2 The processing procedure of the main interactive user data processing channel includes steps S1301 to S1303.

[0039] Step S1301: Audiovisual Feature Extraction. Extract the face region image corresponding to the current main interactive person from the video data, and locate the key points of the lip contour using a lip key point detection model to extract lip movement features. The lip key point detection model can be, for example, a facial key point regression model based on a convolutional neural network.

[0040] In some embodiments, the lip movement features may include: a temporal image patch sequence of the lip region, motion trajectory features of key points on the lips, and a temporal variation curve of the degree of lip opening and closing. Simultaneously, the visual end also extracts the facial identity features and occlusion status of the current main interacting person, the occlusion status including, for example, mask occlusion detection results.

[0041] The time-domain signals of each channel of the arrayed voice data are preprocessed, including pre-emphasis, framing, and windowing. The time-domain signals are then converted to the time-frequency domain using a short-time Fourier transform to obtain the complex spectral representation of each channel. Based on this, the location of the sound source corresponding to the current main interactor can be estimated using a sound source localization algorithm. The sound source localization algorithm is a commonly used algorithm in this field and will not be described in detail here.

[0042] In some embodiments, the audio terminal also extracts the online voiceprint features of the current main interactor. The online voiceprint features are speaker representation vectors extracted from short speech segments of the current main interactor collected during the current interaction, without relying on a pre-registered voiceprint database.

[0043] Step S1302: Sound source separation and / or speech enhancement. Construct an audiovisual fusion neural network to fuse the lip movement features, multi-channel audio features, sound source location information, and online speaker representation to generate a time-frequency mask or separation representation for target speech enhancement.

[0044] In some embodiments, the audiovisual fusion neural network may include a visual encoding branch, an audio encoding branch, and a fusion decoding branch. The visual encoding branch receives the lip movement feature sequence and extracts a temporal representation of the lip movement using a temporal convolutional network or a Transformer encoder. The audio encoding branch receives multi-channel time-frequency features of the arrayed speech data and extracts spatial-frequency acoustic representations using a complex convolutional network. The fusion decoding branch performs cross-modal fusion of the visual temporal representation, audio spatial-frequency representation, orientation coding, and speaker embedding vector, outputting a time-frequency mask for the target speaker. This cross-modal fusion can be achieved, for example, through a cross-modal attention mechanism, aligning and guiding visual and audio features in the temporal dimension. The time-frequency mask acts on the time-frequency representation of the arrayed speech data, separating the speech component of the current main interactor from the mixed speech and suppressing interfering voices and background noise to obtain the enhanced speech data.

[0045] In some embodiments, the audiovisual fusion neural network can utilize the temporal correspondence between lip movements and speech activity as a guiding signal when generating the time-frequency mask. When the visual end detects that the lips of the current main interactor are closed, the time-frequency mask of the corresponding time frame tends to zero, indicating that there is no speech component at that moment; when the lips are open, the time-frequency mask tends to a non-zero value, preserving the speech component at that moment. Compared with speech separation methods that rely solely on acoustic cues, this cross-modal separation method guided by visual lip movement features can introduce additional visual constraints in complex acoustic scenarios such as co-directional sound sources, close angles, or multiple speakers speaking simultaneously, improving the stability and accuracy of target speaker separation. The co-directional sound source refers to the situation where multiple speakers are located at the same or similar angles. Step S1303: Maintain interaction intent monitoring and confidence generation. While enhancing the voice of the current main interactor, the main interactor data processing channel continuously monitors whether the current main interactor maintains the interaction intent and generates the main interactor's confidence data. The main interactor's confidence data reflects the degree of credibility of the current main interactor as a continuous interaction object.

[0046] In some embodiments, the confidence data of the primary interactor is determined based on at least one of lip movement confidence data, location confidence data, or voiceprint confidence data.

[0047] When the lip movement features are obtained, the lip movement confidence data of the current main interactor is determined based on the lip movement features. For example, the lip movement confidence data is determined based on the continuity of lip movement detection and the consistency between lip movement and speech activity. When the sound source location information is obtained, the location confidence data of the current main interactive person is determined based on the sound source location information. For example, the location confidence data is determined based on the temporal stability of the sound source location and the consistency between the sound source location and the visual location of the current main interactive person. When the voiceprint information is obtained, the voiceprint confidence data of the current main interactor is determined based on the voiceprint information, for example, the voiceprint confidence data is determined based on the similarity between online voiceprint features and the voiceprint features of the current audio segment.

[0048] The main interactive person confidence data is determined by weighted fusion, product fusion, or consistency check fusion, using at least one of the aforementioned lip movement confidence data, location confidence data, or voiceprint confidence data.

[0049] Specifically, weighted fusion assigns weights to lip movement confidence data, location confidence data, and voiceprint confidence data based on the reliability of different modalities in the current scenario, and calculates a weighted sum to obtain the main interactive person confidence data.

[0050] Product fusion treats data with different confidence levels as independent or nearly independent data, enhances the confidence level when multiple modes are consistent through multiplication, and suppresses misjudgments caused by single mode anomalies.

[0051] The consistency verification fusion is based on whether different modalities are consistent with each other. When multiple modalities all point to the same candidate main interactor, the confidence data of the main interactor is increased; when there is a conflict between multiple modalities, the confidence data of the main interactor is decreased. Taking a service robot serving multiple users in a noisy exhibition setting as an example, the robot's microphone array collects mixed audio from the environment, including the target speaker's voice, surrounding conversations, and exhibition background noise. Simultaneously, the robot continuously captures video through a front-facing RGB camera. After the initial interaction is established, the system continuously locates the face region of the current main user through face tracking, while simultaneously detecting their lip movement state and extracting lip movement features. When the current main user continues to speak towards the robot and exhibits high consistency between lip movement and speech activity, the visually guided speech enhancement channel effectively separates and enhances their speech signal. Simultaneously, if the robot observes another user approaching from the side, looking at the robot, and remaining there, the system evaluates the user's interaction intent in the candidate user intent monitoring channel (see subsequent steps for details). Through the main user data processing channel in this embodiment, the system continuously outputs enhanced, clear speech data and confidence data reflecting the interaction state during the current main user's interaction process, ensuring a stable and uninterrupted interaction experience for the main user.

[0052] In step S140, the interaction intent of the at least one candidate interactor is monitored through the candidate interactor intent monitoring channel to obtain the confidence data of the at least one candidate interactor.

[0053] The candidate interaction intent monitoring channel operates in parallel with the main interaction data processing channel. Its main function is to continuously monitor the interaction intent of other potential interaction personnel in the scene while the system focuses on the current main interaction person, so as to provide the basis for evaluating the intent of the candidate parties for subsequent interaction right switching decisions.

[0054] Specifically, for each candidate interactor, the system extracts candidate interaction intent feature data from the interaction scenario data, and determines the confidence level data of the corresponding candidate interactor based on the candidate interaction intent feature data.

[0055] In one embodiment, the candidate interaction intent feature data comprehensively reflects the intensity of the candidate user's interaction intent from multiple dimensions, including facial identity. The identity information of the candidate user is determined through facial recognition, and a basic weight of the identity-related interaction intent is output based on a preset interaction priority. The interaction priority can be set, for example, higher for registered users than unregistered users, and higher for VIP users than regular users.

[0056] In one embodiment, the candidate interaction intent feature data comprehensively reflects the intensity of the candidate user's interaction intent from multiple dimensions, including the gaze direction dimension. Using eye keypoint detection and gaze estimation models, the angle between the candidate user's gaze direction vector and the main direction of the interaction device is calculated. When the angle is less than a preset threshold and the duration exceeds a preset time, it indicates that the candidate user is looking at the interaction device, and a higher gaze-related interaction intent score is output. For example, when the angle between the user's gaze direction and the robot's frontal direction is consistently less than a preset angle threshold for more than a preset time window, it can be determined as a gaze interaction intent; the preset angle threshold can be, for example, 15 degrees.

[0057] In one embodiment, the candidate interaction intent feature data comprehensively reflects the intensity of the candidate user's interaction intent from multiple dimensions, including the gesture dimension. A gesture detection and recognition model detects whether the candidate user performs an explicit interaction action, and outputs a gesture-related interaction intent score based on the detection results and recognition confidence. The explicit interaction action may include, for example, raising a hand, waving, pointing at the interactive device, or making a specific gesture. When a hand-raising or waving action is detected and the action lasts for multiple frames, a higher gesture intent score is output.

[0058] In one embodiment, the candidate interaction intent feature data comprehensively reflects the intensity of the candidate user's interaction intent from multiple dimensions, including human posture and motion trend dimensions. Through human posture estimation and motion trajectory analysis, the behavioral trend of the candidate user is determined. Specifically, the system can track the spatial position of the candidate user in consecutive frames, calculate their motion speed, direction of motion, and acceleration, and analyze their approach trend, dwelling state, or departure trend relative to the interactive device. When a candidate user is detected gradually approaching the interactive device from a distance and remaining within a preset distance for more than a preset time, a higher motion trend intent score is output; when a candidate user is detected quickly passing by the interactive device without their gaze directed towards the device, a lower intent score is output, thus distinguishing between "passing by" and "staying to prepare for interaction."

[0059] In one embodiment, the candidate interaction intent feature data comprehensively reflects the intensity of the candidate interactor's interaction intent from multiple dimensions, including the voice wake-up dimension. The system continuously monitors voice signals from the directions of each candidate interactor, and when a preset wake-up word is detected and the sound source localization result matches the candidate interactor's location, it outputs an interaction intent score related to voice wake-up.

[0060] In one embodiment, the candidate interaction intent feature data comprehensively reflects the intensity of the candidate interactor's interaction intent from multiple dimensions, including the semantic intent dimension. Through automatic speech recognition and natural language understanding technologies, the semantic intent intensity of speech content from the candidate interactor's direction is analyzed without a preset wake word. For example, when a candidate interactor is detected uttering a statement containing a clear interaction intent, a semantically relevant interaction intent score is output.

[0061] In one embodiment, the candidate interaction intent feature data comprehensively reflects the intensity of the candidate's interaction intent from multiple dimensions, including the lip movement state dimension. When the candidate is within the camera's field of view, their lip movement state is detected. Continuous lip movement, especially when the candidate is looking at the device, can serve as a valid indication that they are currently speaking or preparing to speak, and a corresponding lip movement intent score is output. For the feature data extracted from each of the above dimensions, the system calculates a normalized interaction intent score and a reliability coefficient for each dimension. The normalized interaction intent score reflects the strength of the dimension's indication of the interaction intent at the current moment, and the reliability coefficient reflects the credibility of the dimension under the current scene conditions. For example, in scenes with good lighting conditions and clearly visible faces, the reliability coefficients of gaze estimation and face recognition are high; in scenes with dim lighting or partially obscured faces, the reliability coefficients of visually related dimensions decrease accordingly.

[0062] In step S150, an interaction state event is obtained and executed based on the enhanced voice data of the current main interactor, the confidence data of the main interactor, and the confidence data of at least one candidate interactor.

[0063] The interactive state events include at least one of the following: main interactive person hold event, candidate interactive person attention event, or main interactive person switch event. These are structured decision outputs used by the system to control the interactive process, rather than simple person recognition results.

[0064] Furthermore, the generation of the interaction state events follows these rules: When the confidence data of the primary interactor meets the primary interactor hold condition, and the confidence data of at least one candidate interactor does not meet the switching condition, the primary interactor hold event is executed, and the system continues to perform voice enhancement and interaction response for the current primary interactor. When the confidence data of at least one candidate interactor meets the attention condition but does not meet the switching condition, the candidate interactor attention event is executed, the system records the interaction intent information of the candidate interactor and increases its attention level, but does not interrupt the response to the current primary interactor. When the confidence data of the primary interactor meets the primary interactor exit condition, and the confidence data of the target candidate interactor meets the switching condition, the primary interactor switch event is executed, the system switches the target candidate interactor to the new primary interactor, and begins voice enhancement processing and interaction response for that candidate interactor.

[0065] More specifically, the above conditions can be determined in the following ways. The main interactor retention condition includes: the main interactor's confidence data is higher than a preset retention threshold, and / or the main interactor's confidence data is higher than the candidate interactor's confidence data. The attention condition includes: the candidate interactor's confidence data is higher than a preset attention threshold but the switching condition is not met. The main interactor exit condition includes: the main interactor's confidence data is lower than a preset exit threshold, or the main interactor's enhanced voice data meets the interaction end state, such as detecting continuous silence of voice activity exceeding a preset duration, detecting a closing remark, or detecting the current main interactor turning away. The switching condition includes: the candidate interactor's confidence data is consistently higher than the current main interactor's confidence data, and exceeds a preset switching threshold and a preset duration threshold. In this embodiment, while continuously enhancing the voice of the current main interactor and ensuring the quality of interaction, the system continuously senses potential interaction needs through the parallel-running candidate interactor intent monitoring channel, and generates structured interaction state events based on the comprehensive relationship of the confidence data of both parties. This effectively solves the problems in the prior art that only focus on the current interactor, resulting in the omission of candidate interactors, and that only rely on a single trigger event, resulting in abrupt switching of interaction rights. It achieves natural and smooth turn-taking and interaction continuity in noisy multi-person scenarios.

[0066] Figure 3 A flowchart illustrating an example of confidence data generation and dynamic weight adjustment according to an embodiment of the present invention is provided. The confidence fusion method described in this embodiment can be applied to the generation of main interactor confidence data in step S130 and the generation of candidate interactor confidence data in step S140.

[0067] like Figure 3As shown, in step S310, the normalized interaction intent score S of each modality corresponding to the main interactor or the candidate interactor in the interaction scene data is obtained. i (t) and reliability coefficient R i (t).

[0068] For the primary interactor, N modalities include, but are not limited to, lip movement modality, sound source location modality, and voiceprint modality. For the candidate interactors, each modality includes, but is not limited to, facial identity modality, gaze direction modality, gesture action modality, body posture modality, voice wake-up modality, semantic intent modality, and lip movement state modality. Each modality outputs a normalized interaction intent score S. i (t) and a reliability coefficient R i (t). The normalized interaction intent score maps the original intent indicators of different dimensions to a unified numerical range, making the intent outputs of different modalities comparable. The unified numerical range can be, for example, between 0 and 1. The reliability coefficient reflects the credibility of the modal signal under the current scenario conditions. For example, it can be determined based on the historical stability of the modal signal and the current quality assessment results.

[0069] In step S320, based on the base weight values ​​in the current scenario and the reliability coefficient, the fusion weight values ​​corresponding to each modality are determined, and confidence data is calculated based on the normalized interaction intent score and the fusion weight values. The calculation formula is as follows:

[0070] Where, α i R represents the basic weights of the i-th modality in the current scenario. i (t) represents the reliability of the i-th mode in the current scenario, w i (t) represents the fusion weight value. Then, the confidence score data for user u is calculated:

[0071] In step S330, time smoothing is performed on the confidence data of the main interactor and the confidence data of the candidate interactors within a continuous time window. The calculation formula is as follows:

[0072] Where λ is the smoothing coefficient, the larger the value, the smoother the surface.

[0073] Because the detection results of each modality may have short-term jitter or occasional false detections, directly using single-frame confidence data for interactive state decisions may lead to frequent erroneous switching. To address this issue, the system performs time smoothing on the confidence data within a continuous time window t. The choice of the smoothing coefficient λ affects the degree of smoothing: a larger smoothing coefficient results in a smoother confidence sequence, but a slower response time; a smaller smoothing coefficient results in a faster response time, but a greater susceptibility to instantaneous jitter.

[0074] In step S340, the interaction state event is generated based on the confidence data of the main interactor and the confidence data of the candidate interactors after time smoothing.

[0075] The generation of the interactive state events is based on preset condition rules, specifically including: when the confidence data of the main interactor meets the main interactor retention condition, and the confidence data of at least one candidate interactor does not meet the switching condition, the main interactor retention event is generated and executed, and the system continues to perform voice enhancement and interactive response for the current main interactor. When the confidence data of at least one candidate interactor meets the attention condition but does not meet the switching condition, the candidate interactor attention event is generated and executed, and the system increases the attention level of the candidate interactor, which may include increasing the frequency and accuracy of intent monitoring, but without interrupting the response to the current main interactor. When the confidence data of the main interactor meets the main interactor exit condition, and the confidence data of the target candidate interactor meets the switching condition, the main interactor switching event is generated and executed, the system establishes the target candidate interactor as the new main interactor, starts voice enhancement processing and interactive response for it, and performs a smooth exit for the original main interactor.

[0076] The main interactor retention conditions include: the main interactor's confidence data is higher than a preset retention threshold, and / or the main interactor's confidence data is higher than the candidate interactor's confidence data. The attention conditions include: the candidate interactor's confidence data is higher than a preset attention threshold but the switching conditions are not met. The main interactor exit conditions include: the main interactor's confidence data is lower than a preset exit threshold, or the main interactor's enhanced voice data meets the interaction end state. The switching conditions include: the candidate interactor's confidence data is consistently higher than the current main interactor's confidence data, and exceeds a preset switching threshold and a duration threshold. It should be noted that the above thresholds can be preset according to the application scenario during system initialization, or dynamically adjusted according to the scenario during operation. For example, the preset threshold could be that the attention threshold is lower than the switching threshold, and the exit threshold is lower than the retention threshold.

[0077] Taking a multi-user interaction scenario with a community service robot as an example, when an elderly user, acting as the primary interactor, is consulting the robot about community services, the system continuously enhances their voice through the primary interactor data processing channel and monitors their confidence data regarding their intention to maintain interaction. Simultaneously, the candidate interactor intent monitoring channel detects another young user raising their hand, looking at the robot, and continuously approaching. This candidate interactor has high scores in gaze dimension, gesture dimension, and motion trend dimension. After dynamic weight fusion and time smoothing, their overall confidence data continuously increases. When the candidate interactor's confidence data exceeds the attention threshold but has not yet met the switching conditions, the system generates a candidate interactor attention event, increasing the attention level for that user. Subsequently, the primary interactor says "Thank you, I understand" and turns away. The system detects the end of the interaction through voice endpoint detection and human posture analysis, and the primary interactor's confidence data drops below the exit threshold. Simultaneously, the candidate interactor's confidence data has consistently been higher than the original primary interactor's confidence data and exceeds both the switching threshold and the preset duration threshold. The system then generates a primary interactor switching event, smoothly switching the interaction right to the younger user.

[0078] In this embodiment, by employing a dynamic weight fusion and multi-dimensional confidence comprehensive evaluation mechanism, the system can adaptively adjust the fusion weights of each interaction intent dimension based on the current scene conditions and the signal quality of each modality, achieving scene-adaptive confidence calculation. Combined with event generation rules based on temporal smoothing and conditional judgment, the system effectively suppresses frequent false switching caused by instantaneous false detections and short-term jitter, achieving stable and smooth dynamic switching of interaction weights.

[0079] Furthermore, before generating the interaction state event, the system may also perform cross-modal consistency analysis and / or contextual reasoning on at least one of the video data, array voice data, distance information, touch interaction information, text interaction information, or historical interaction context in the interaction scene data, and perform auxiliary arbitration on the confidence data of the main interactor and the confidence data of the candidate interactors based on the cross-modal consistency analysis and / or the contextual reasoning.

[0080] Specifically, the cross-modal consistency analysis includes cross-validating information from different modalities. For example, it checks whether visually detected lip movements and audio-detected speech activities are temporally consistent; whether the location indicated by the sound source localization result is consistent with the visually detected speaker's face location; and whether the gesture recognition result and semantic understanding result point to the same or similar interaction intent. When the analysis results from multiple modalities are consistent, the reliability of the corresponding confidence data can be improved; when the analysis results from some modalities are inconsistent with other modalities, the system can reduce the weighting of the confidence contribution of the inconsistent modalities.

[0081] The contextual reasoning involves utilizing historical interaction context information to assist current decision-making. For example, if the system records in recent interaction history that a candidate user has repeatedly attempted to initiate an interaction but failed, the system can appropriately increase the response sensitivity of its confidence data when its interaction intent is detected again. Furthermore, when the confidence data of multiple candidate users are simultaneously high, the system can perform contextual reasoning based on the interaction content of the current primary user. For instance, it can determine whether the primary user is performing a complex task requiring continuous focus, triggering a switch at an appropriate boundary point of the interaction task, rather than interrupting the interaction midway.

[0082] Figure 4 A flowchart illustrating an example of anomaly detection and adaptive processing according to an embodiment of the present invention is shown.

[0083] like Figure 4 As shown, in step S410, the quality of each modal signal corresponding to the interactive scene data is monitored.

[0084] Specifically, for the visual modality, monitoring metrics include, but are not limited to, image sharpness, lighting conditions, occlusion status, and face detection confidence. Image sharpness can be evaluated, for example, using Laplacian variance; lighting conditions can be analyzed, for example, using average image brightness and histograms; and occlusion status can be measured, for example, using face occlusion ratio and lip visibility score. For the audio modality, monitoring metrics include, but are not limited to, speech signal-to-noise ratio (SNR), speech endpoint stability, array orientation reliability, and consistency of signals across channels. SNR can be obtained, for example, by calculating the energy ratio after distinguishing speech segments from noise segments through speech activity detection; speech endpoint stability can be measured, for example, by the consistency of continuous speech endpoint detection results; and array orientation reliability can be evaluated, for example, by the temporal stability and spatial consistency of sound source localization results.

[0085] In step S420, it is determined whether there is at least one modal signal whose quality meets the abnormal condition.

[0086] Specifically, when one or more quality metrics for a certain modality are detected to be below the corresponding preset quality threshold for multiple consecutive frames, the quality of that modality signal is determined to meet an anomaly condition. The system distinguishes between two types of anomalies: short-term anomalies and long-term anomalies. Short-term anomalies refer to transient quality degradation that occurs within a short time window, such as lips becoming invisible due to a user briefly turning their head or transient noise in the environment. Long-term anomalies refer to quality degradation that persists within a longer time window, such as the overall unusability of the visual modality due to persistently low ambient light.

[0087] In step S430, when the quality of at least one modal signal meets the abnormal condition, the fusion weight value of the corresponding modality is reduced, and a corresponding compensation strategy is adopted according to the type of abnormality.

[0088] For short-term anomalies, the system compensates through methods such as temporal smoothing, confidence preservation, and short-term state prediction. Specifically, during a short-term anomaly, the system can maintain the confidence value or fusion weight value before the anomaly occurred, and rely on the temporal smoothing mechanism to transition naturally, avoiding drastic fluctuations in confidence and weight caused by instantaneous false detections.

[0089] In the event of prolonged anomalies, when the visual modality is detected to be unavailable for an extended period, the system reduces the fusion weight of modal features obtained from video data and increases the fusion weight of at least one of array speech, voiceprint information, sound source location information, or semantic information. For example, when persistently low ambient light or complete camera obstruction causes prolonged visual modality failure, the system reduces the fusion weights of visually related lip movement confidence, gaze confidence, gesture confidence, and facial identity confidence, while increasing the fusion weights of voiceprint confidence, sound source location confidence, and semantic confidence, switching the interactive system's operating mode from an audiovisual fusion mode to an audio-semantic-dominated mode. In extreme cases, when most modalities are unavailable, the system can revert to a basic array signal processing mode, using location information to index the speech content most relevant to the current main user's spatial location, ensuring uninterrupted basic interactive functions.

[0090] In step S440, after the anomaly is resolved, the fusion weight values ​​of each mode are smoothly restored.

[0091] Specifically, when the anomaly handling module detects that the anomaly condition has been resolved, and the decision-making module confirms this through multiple consecutive frames, the system executes a smooth recovery process. The resolution of the anomaly condition may include, for example, restoration of normal illumination, elimination of occlusion, or a return of the signal-to-noise ratio to a normal level. Specifically, the system does not immediately restore the modalities with reduced weights to their original weight values ​​before the anomaly. Instead, it gradually restores the fusion weight values ​​of each modality to normal levels within a preset transition time window using a gradual increase method. This gradual increase method can be, for example, linear interpolation or exponential smooth recovery, to avoid drastic changes in confidence calculation results and interactive state events caused by sudden weight abrupt changes. In the actual verification of this application's solution, when the visual modality is completely occluded, the system can smoothly transition to an audio-semantic-dominated processing mode within a preset time, maintaining uninterrupted core interactive functions.

[0092] Through the forward-looking anomaly detection and adaptive processing mechanism in this embodiment, the system possesses multimodal robustness and self-recovery capability in complex real-world environments. This mechanism is not equivalent to passive error correction after identification failure, but rather proactively intervenes in the early stages of input quality degradation. Through dynamic weight reduction, modality switching, and smooth recovery strategies, it ensures the system's continued availability under abnormal conditions, solving the problem in existing technologies where single-modal failure leads to a sharp decline in system performance or complete unavailability.

[0093] Figure 5 This is a schematic block diagram of an embodiment of the interactive right dynamic switching device of the present invention. Figure 5 As shown, the device may include the following modules.

[0094] The data acquisition module 510 is used to acquire interactive scene data in the interactive scene. The interactive scene data includes array voice data, as well as at least one of video data, distance information, touch interaction information, text interaction information, or historical interaction context.

[0095] The object determination module 520 is used to determine the current main interactor and at least one candidate interactor based on the interaction scenario data.

[0096] The main interactor data processing module 530 is used to perform voice enhancement processing and maintain interaction intent monitoring on the current main interactor, and obtain the enhanced voice data and confidence data of the current main interactor.

[0097] Specifically, the main user data processing module 530 is used to extract visual feature data and audio feature data corresponding to the current main user from the interaction scene data, separate and / or enhance the voice data of the current main user from the array voice data based on the visual feature data and the audio feature data, and monitor the main user's intention to maintain interaction. The visual feature data includes the lip movement features of the current main user, and the audio feature data includes the sound source location information and / or voiceprint information of the current main user.

[0098] The candidate interaction intent monitoring module 540 is used to monitor the interaction intent of the at least one candidate interaction user and obtain confidence data of the at least one candidate interaction user. Specifically, the candidate interaction intent monitoring module 540 is used to extract candidate interaction intent feature data from the interaction scenario data for each candidate interaction user, and determine the confidence data of the corresponding candidate interaction user based on the candidate interaction intent feature data.

[0099] Specifically, the candidate interaction intent feature data includes at least one of the following: facial identity information, gaze direction, gesture action, human posture, voice wake-up result, semantic intent result, lip movement state, dwell time, or proximity trend.

[0100] The event execution module 550 is used to obtain and execute interaction state events based on the enhanced voice data of the current main interactor, the confidence data of the main interactor, and the confidence data of at least one candidate interactor. The interaction state events include at least one of a main interactor hold event, a candidate interactor attention event, or a main interactor switch event.

[0101] In addition, the main interactive user data processing module 530 and the candidate interactive user intent monitoring module 540 operate in parallel, that is, they are both working during system operation and perform concurrent processing on the main interactive user and candidate interactive user in the same interactive scenario.

[0102] Furthermore, the device may also include a cross-modal arbitration module 560, which is used to perform cross-modal consistency analysis and / or contextual reasoning on at least one of the video data, array voice data, distance information, touch interaction information, text interaction information or historical interaction context in the interaction scene data, and to perform auxiliary arbitration on the confidence data of the main interactor and the confidence data of the candidate interactors based on the analysis results.

[0103] Furthermore, the device may also include an anomaly processing module 570, used to monitor the quality of each modal signal corresponding to the interactive scene data, and reduce the fusion weight value of the corresponding modality when the quality of at least one modal signal meets the anomaly condition. The modal signal quality includes at least one of image sharpness, lighting conditions, occlusion state, lip visibility, speech signal-to-noise ratio, and speech endpoint stability.

[0104] Furthermore, the anomaly handling module 570 is also used to reduce the fusion weight value of the modal features obtained based on video data and increase the fusion weight of at least one of array speech, voiceprint information, sound source location information or semantic information when the visual modality is detected to be unavailable for a long time.

[0105] The apparatus described in the above embodiments of the present invention can be used to execute the corresponding method embodiments of the present invention, and accordingly achieve the technical effects achieved by the above method embodiments of the present invention, which will not be elaborated further here. In the embodiments of the present invention, relevant functional modules can be implemented by combining software with a hardware processor.

[0106] In addition, such as Figure 6As shown, one embodiment of the present invention also provides an electronic device, including at least one processor 1 and a memory 2 communicatively connected to the at least one processor. The memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the steps of the dynamic switching method for interaction rights described in any of the above method embodiments.

[0107] An embodiment of the present invention also provides a storage medium on which a computer program is stored. When the computer program is executed by a processor, it implements the steps of the dynamic switching method for interaction rights described in any of the above method embodiments. The storage medium may include various media capable of storing program code, such as read-only memory, random access memory, magnetic disk, and optical disk.

[0108] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0109] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented using software plus a general-purpose hardware platform, or of course, using hardware. Based on this understanding, the above technical solutions, in essence or the parts that contribute to the related technology, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

Claims

1. A method for dynamically switching interaction rights, characterized in that, include: Acquire interaction scenario data; Based on the interaction scenario data, determine the current main interactor and at least one candidate interactor; The voice enhancement and interaction intent monitoring of the current main interactor are performed through the main interactor data processing channel to obtain the enhanced voice data and confidence data of the current main interactor. The interaction intent of the at least one candidate interactor is monitored through the candidate interactor intent monitoring channel to obtain the confidence data of the at least one candidate interactor. Based on the enhanced voice data of the current main interactor, the confidence data of the main interactor, and the confidence data of at least one candidate interactor, an interaction state event is obtained and executed; The interaction state events include at least one of the following: main user keep event, candidate user follow event, or main user switch event; The main interactive user data processing channel and the candidate interactive user intent monitoring channel operate in parallel.

2. The method according to claim 1, characterized in that, The interactive scenario data includes array voice data, as well as at least one of video data, distance information, touch interaction information, text interaction information, or historical interaction context.

3. The method according to claim 2, characterized in that, The process of performing voice enhancement processing and maintaining interaction intent monitoring on the current main interactor through the main interactor data processing channel to obtain the enhanced voice data and confidence data of the current main interactor includes: Extract visual and audio feature data corresponding to the current main interactive user from the interaction scene data; Based on the visual feature data and the audio feature data, the voice data of the current main interactor is separated and / or enhanced from the array voice data to obtain the enhanced voice data, and the main interactor's intention to maintain interaction is monitored to obtain the confidence data of the main interactor.

4. The method according to claim 3, characterized in that, The visual feature data of the current main interactive user includes the lip movement features of the current main interactive user; The audio feature data of the current main interactive user includes the sound source location information and / or voiceprint information of the current main interactive user; Based on the lip movement features, the sound source location information, and / or the voiceprint information, the target speech component corresponding to the current main interactive person is determined from the array speech data, and the target speech component is separated and / or enhanced to obtain the enhanced speech data of the current main interactive person.

5. The method according to claim 4, characterized in that, The confidence data of the main interactor is determined based on at least one of lip movement confidence data, location confidence data, or voiceprint confidence data, wherein: When the lip movement features are obtained, the lip movement confidence data of the current main interactor is determined based on the lip movement features; When the sound source location information is obtained, the location confidence data of the current main interactor is determined based on the sound source location information; When the voiceprint information is obtained, the voiceprint confidence data of the current main interactor is determined based on the voiceprint information; The confidence data of the main interactor is determined based on at least one of the established lip movement confidence data, the location confidence data, or the voiceprint confidence data.

6. The method according to claim 5, characterized in that, The confidence data of the main interactor is determined based on the weighted result of at least one known confidence data; or, if there are two or more known confidence data, the confidence data of the main interactor is determined based on the product result or consistency check result of the two or more confidence data.

7. The method according to claim 1, characterized in that, The step of monitoring the interaction intent of at least one candidate interactor through the candidate interactor intent monitoring channel and generating confidence data for the at least one candidate interactor includes: For each candidate interactor, candidate interaction intent feature data is extracted from the interaction scenario data; Based on the candidate interaction intent feature data, determine the confidence level data of the corresponding candidate interactor; The candidate interaction intent feature data includes at least one of the following: facial identity information, gaze direction, gesture action, human posture, voice wake-up result, semantic intent result, lip movement state, dwell time, or proximity trend.

8. The method according to claim 1, characterized in that, The confidence data of the main interactor and the confidence data of the candidate interactors are generated in the following way: Obtain the normalized interaction intent score and reliability coefficient of each modality corresponding to the main interactor or the candidate interactor in the interaction scene data; Based on the base weight values ​​in the current scenario and the reliability coefficient, determine the fusion weight values ​​corresponding to each modality; Based on the normalized interaction intent score and the fusion weight value, the confidence data of the main interactor and the confidence data of the candidate interactors are calculated.

9. The method according to claim 8, characterized in that, Also includes: The confidence data of the main interactor and the confidence data of the candidate interactors within a continuous time window are subjected to time smoothing. The interaction state event is generated based on the confidence data of the main interactor and the confidence data of the candidate interactors after time smoothing.

10. The method according to claim 1, characterized in that, The process of obtaining and executing an interaction state event based on the enhanced speech of the current primary interactor, the confidence data of the primary interactor, and the confidence data of at least one candidate interactor includes: The main interactor hold event is executed when the confidence data of the main interactor meets the main interactor hold condition and the confidence data of at least one candidate interactor does not meet the switch condition. When the confidence data of at least one candidate interactor meets the attention condition but does not meet the switching condition, the candidate interactor attention event is executed. When the confidence data of the main interactor meets the main interactor exit condition and the confidence data of the target candidate interactor meets the switching condition, the main interactor switching event is executed.

11. The method according to claim 10, characterized in that, The conditions for maintaining the main interactor, the conditions for following, the conditions for exiting the main interactor, and the conditions for switching are determined in the following ways: The main interaction person retention conditions include: the confidence data of the main interaction person is higher than the preset retention threshold, and / or the confidence data of the main interaction person is higher than the confidence data of the candidate interaction person; The attention conditions include: the confidence data of the candidate interactor is higher than the preset attention threshold but the switching conditions are not met; The main interaction user exit conditions include: the main interaction user's confidence data is lower than a preset exit threshold, or the main interaction user's enhanced voice data meets the interaction end state. The switching conditions include: the confidence data of the candidate interactor is consistently higher than the confidence data of the current main interactor, and exceeds the preset switching threshold and duration threshold.

12. The method according to claim 1, characterized in that, Before generating the interaction state event, it also includes: Perform cross-modal consistency analysis and / or contextual reasoning on at least one of video data, array audio data, distance information, touch interaction information, text interaction information, or historical interaction context; Based on the cross-modal consistency analysis and / or the contextual reasoning, auxiliary arbitration is performed on the confidence data of the main interactor and the confidence data of the candidate interactors.

13. The method according to claim 8, characterized in that, Also includes: Monitor the quality of each modal signal corresponding to the interactive scenario data; When the quality of at least one modal signal meets the abnormal condition, the fusion weight value of the corresponding modality is reduced; The modal signal quality includes at least one of image sharpness, lighting conditions, occlusion status, lip visibility, speech signal-to-noise ratio, and speech endpoint stability.

14. The method according to claim 13, characterized in that, Also includes: When visual modality is detected to be unavailable for an extended period of time, the fusion weight value of modal features obtained from video data is reduced, and the fusion weight of at least one of array speech, voiceprint information, sound source location information, or semantic information is increased.

15. A device for dynamically switching interaction rights, characterized in that, include: The data acquisition module is used to acquire interactive scene data in the interactive scenario; The object determination module is used to determine the current main interactor and at least one candidate interactor based on the interaction scenario data. The main interactor data processing module is used to perform voice enhancement processing and maintain interaction intent monitoring on the current main interactor, and obtain the enhanced voice data of the current main interactor and the confidence data of the main interactor. The candidate interaction intent monitoring module is used to monitor the interaction intent of the at least one candidate interaction and obtain the confidence data of the at least one candidate interaction. The event execution module is used to obtain and execute the interaction state event based on the enhanced voice data of the current main interactor, the confidence data of the main interactor, and the confidence data of at least one candidate interactor. The interaction state events include at least one of the following: main user keep event, candidate user follow event, or main user switch event; The main interactive user data processing module and the candidate interactive user intent monitoring module operate in parallel.

16. The apparatus according to claim 15, characterized in that, The confidence data of the main interactor and the confidence data of the candidate interactors are generated in the following way: Obtain the normalized interaction intent score and reliability coefficient of each modality corresponding to the main interactor or the candidate interactor in the interaction scene data; Based on the base weight values ​​in the current scenario and the reliability coefficient, determine the fusion weight values ​​corresponding to each modality; Based on the normalized interaction intent score and the fusion weight value, the confidence data of the main interactor and the confidence data of the candidate interactors are calculated.

17. The apparatus according to claim 15, characterized in that, Also includes: The confidence data of the main interactor and the confidence data of the candidate interactors within a continuous time window are subjected to time smoothing. The interaction state event is generated based on the confidence data of the main interactor and the confidence data of the candidate interactors after time smoothing.

18. The apparatus according to claim 17, characterized in that, It also includes an anomaly handling module, which reduces the fusion weight value of the corresponding mode when the quality of at least one mode signal meets the anomaly condition.

19. An electronic device comprising: At least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the steps of the method according to any one of claims 1 to 14.

20. A storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program implements the steps of the method according to any one of claims 1 to 14.