Intelligent man-machine interaction system and method based on multi-modal characteristics

Abstract

The invention discloses an intelligent man-machine interaction system and method based on multi-modal features, and relates to the field of man-machine interaction and artificial intelligence. The problems of low interaction efficiency and frequent errors caused by single intention understanding, feedback lagging and lack of continuous learning ability of a traditional interaction system are solved. According to the method, multi-modal data are collected, and uniform feature representation is generated through feature extraction and time sequence alignment fusion; generating a candidate intention list by using a lightweight encoder and an inference network based on the features; a final execution intention is determined by dynamically generating a multi-modal information probe and analyzing micro-feedback characteristics of a user in combination with iterative screening of a consensus degree calculation model; converting the determined intention into an equipment control instruction and executing the equipment control instruction; a reinforcement learning environment is constructed, a reasoning network and consensus degree model parameters are continuously optimized based on interaction process data and execution results, and accurate understanding of a multi-modal intention, self-adaptive optimization of an interaction process and continuous improvement of system performance are realized.

Description

Technical Field

[0001] This invention belongs to the field of human-computer interaction and artificial intelligence, specifically an intelligent human-computer interaction system and method based on multimodal features. Background Technology

[0002] Currently, human-computer interaction technology is undergoing a profound transformation from single-channel command-based interaction to multimodal natural interaction. With the rapid development of artificial intelligence, computer vision, and the Internet of Things (IoT), intelligent systems can now perceive user states and intentions by integrating multiple information channels such as voice, vision, gestures, and even physiological signals. Typical applications such as smartphones, smart speakers, and in-vehicle systems widely support hybrid interaction modes of voice + touch or vision + voice. However, multimodal fusion still faces many challenges. Technical bottlenecks exist in the temporal synchronization and semantic alignment of data from different modalities, resulting in information fusion remaining at a superficial level. Systems' understanding of user intentions is often limited to surface-level commands, making it difficult to capture implicit intentions and emotional states. Existing interaction modes lack dynamic adjustment capabilities and cannot be optimized in real time based on contextual changes and user feedback. Most systems remain in a passive response mode, lacking continuous learning and self-evolution capabilities, which restricts the in-depth development of interactive experiences towards personalization and intelligence.

[0003] The following problems exist in the existing technology:

[0004] Traditional multimodal systems typically process each modality's data independently, lacking a precise time synchronization mechanism. This results in information such as speech, vision, and text being misaligned on the timeline, creating information silos.

[0005] Existing interactive systems often output a single, highly confident intent. When user instructions are ambiguous or have multiple possible interpretations, misidentification is very likely. At the same time, the candidate intent list is easily monopolized by semantically similar high-probability intents, lacking diversity and failing to cover the user's potential needs.

[0006] Existing systems mostly adopt a linear interaction mode of recognition-execution, which cannot effectively capture and utilize the subtle behavioral feedback generated by users during the decision-making process, resulting in a lack of flexibility and adaptability in the interaction process.

[0007] Most interactive systems maintain fixed parameter configurations after deployment, making it difficult to learn from real-world usage scenarios, adapt to the personalized needs of different users, or improve interactive performance through continuous learning. Summary of the Invention

[0008] The present invention aims to solve at least one of the technical problems existing in the prior art; to this end, the present invention proposes an intelligent human-computer interaction system and method based on multimodal features to solve the above-mentioned technical problems.

[0009] The first aspect of the present invention provides an intelligent human-computer interaction system based on multimodal features, comprising the following modules:

[0010] Multimodal perception and feature extraction module: Acquires raw signals and generates multimodal feature data through a preprocessing process;

[0011] Candidate intent generation module: Based on multimodal feature data, using a lightweight model, it generates and outputs a list of candidate intents;

[0012] Intent Collaborative Calibration Module: Based on a candidate intent list, it iteratively filters and outputs consensus decisions by initiating interactive operations and parsing real-time user feedback from the candidate intent list.

[0013] Decision execution module: Receives the final execution intention and transforms it into specific control instructions to drive the corresponding terminal devices to complete the operation;

[0014] Feedback and Learning Module: By continuously monitoring the system's interaction process metrics and execution result feedback, the module optimizes the parameters and strategies for the candidate intent list and the final execution intent.

[0015] Preferably, the raw signal is acquired, and multimodal feature data is generated through a preprocessing procedure, including the following steps:

[0016] The system synchronously collects raw voice waveform data, visual image sequences, and raw physiological data from users through voice sensors, vision sensors, and physiological sensors, and also receives raw text data. A hardware-triggered synchronization mechanism is used, with the main control chip adding a unified timestamp to each frame of data.

[0017] Parallel processing and feature extraction are performed on the user's raw voice waveform data, visual image sequence, raw physiological data and raw text data to obtain voice text content, acoustic emotion feature vector, visual quantization feature vector, physiological feature vector and text semantic vector.

[0018] The central alignment and fusion engine is invoked to align all feature vectors within a time window based on a unified timestamp, and a lightweight cross-modal attention mechanism is used for fusion to generate multimodal feature data.

[0019] Preferably, based on multimodal feature data, a lightweight model is used to generate and output a list of candidate intentions, including the following steps:

[0020] Based on multimodal feature data, a unified multimodal context representation vector is generated through a lightweight Transformer encoder; simultaneously, a candidate intent library is predefined. ;

[0021] Based on the multimodal context representation vector, the initial probability of each candidate intent in the predefined intent library is calculated through a lightweight inference network; and the semantic similarity is calculated based on the embedding vector generated by the semantic representation of each candidate intent.

[0022] Based on the initial probability of the candidate intents and the semantic similarity between each candidate intent, the final ranking score of each candidate intent is calculated.

[0023] The formula for calculating the final ranking score is as follows: ;

[0024] in, Candidate intention The final sorting score, The initial probability of the candidate intent. Here, N represents the diversity weighting coefficient, and N is the total number of intents in the predefined candidate intent library. Candidate intention and semantic similarity;

[0025] The candidate intentions are sorted in descending order based on the final ranking score, and the top K candidate intentions are selected to form a candidate intention list. .

[0026] Preferably, based on a candidate intent list, by initiating interactive operations and parsing real-time user feedback, the candidate intent list is iteratively filtered and a consensus decision is output, including the following steps:

[0027] Based on the candidate intent list, the top M intents are selected as the probe targets, and information probes are dynamically generated for each selected intent and bound to the available output modalities of the system.

[0028] The probe sequence is executed sequentially according to the intended order, with each probe lasting for a fixed time window. During the probe execution, the user's physiological and behavioral response signals are collected in real time through voice sensors, visual sensors, and physiological sensors, and microfeedback features, including visual features, voice features, and physiological features, are extracted.

[0029] Based on the extracted micro-feedback features, a consensus degree calculation model is used to dynamically evaluate the user's level of acceptance of the current probing intention, and the consensus degree is calculated. ;

[0030] Real-time comparison of consensus Compared with the preset threshold θ: if If ≥θ, then immediately terminate the current probe sequence and confirm that the current probing intention is a consensus decision; if If <θ, the current probe effect is immediately withdrawn, and the probe for the next intent in the candidate intent list is executed.

[0031] If no consensus is reached after all the probe sequences have been executed, a downgrade mechanism is triggered, switching to a direct command receiving mode based on traditional speech recognition.

[0032] Preferably, the calculation of consensus includes the following steps:

[0033] The calculation formula based on the consensus calculation model: ;

[0034] Achieving consensus ,in, τ represents the microfeedback feature data; wk represents the weighting coefficient of each modal signal; τ represents the time decay factor; η represents the explicit feedback reward coefficient; and t represents the normalized probe process time. This is an explicit feedback indicator function.

[0035] Preferably, receiving the final execution intent and converting it into specific control commands to drive the corresponding terminal device to complete the operation includes the following steps:

[0036] Receive a consensus decision from the intent coordinating calibration module, the consensus decision containing a unique identifier for the intent selected from the candidate intent list;

[0037] Generate control instructions that drive the terminal device, corresponding to the unique identifier of the intent.

[0038] The control command is sent to the corresponding terminal device to execute the operation specified by the consensus decision.

[0039] Preferably, the step of continuously monitoring the interaction process indicators and execution result feedback of the system to optimize the parameters and strategies of the candidate intent list and the final execution intent includes the following steps:

[0040] By continuously monitoring the interaction process indicators and execution result feedback of the system, process data and result data for each interaction are collected; the process data includes a candidate intent list, information probe execution sequence and consensus decision achievement time, and the result data includes user feedback information.

[0041] A consensus efficiency evaluation index is calculated based on process data and outcome data.

[0042] A reinforcement learning environment is constructed, and a reward signal is generated based on the consensus efficiency evaluation index and user feedback information. The process of selecting trial intentions and adjusting the preset threshold θ is continuously optimized.

[0043] Based on the newly accumulated interaction process and result data, the parameters of the lightweight inference network in the candidate intent generation module and the parameters of the consensus calculation model in the intent collaboration calibration module are periodically updated.

[0044] A second aspect of the present invention provides an intelligent human-computer interaction method based on multimodal features, comprising the following steps:

[0045] S1: Acquire raw signals and generate multimodal feature data through a preprocessing process;

[0046] S2: Based on multimodal feature data, a lightweight model is used to generate and output a list of candidate intentions;

[0047] S3: Based on the candidate intent list, initiate interactive operations and parse the user's real-time feedback to iteratively filter from the candidate intent list and output a consensus decision;

[0048] S4: Receive the final execution intention and convert it into specific control instructions to drive the corresponding terminal device to complete the operation;

[0049] S5: Optimize parameters and strategies for the candidate intent list and the final execution intent by continuously monitoring the interaction process indicators and execution result feedback of the system.

[0050] Compared with the prior art, the beneficial effects of the present invention are:

[0051] This invention ensures the temporal consistency of multi-source data through a hardware-level synchronization mechanism, and generates high-quality feature representations by combining a cross-modal attention fusion method, providing a reliable data foundation for subsequent processing.

[0052] This invention employs a lightweight encoder architecture combined with a diverse ranking mechanism to generate a candidate intent list that combines accuracy and diversity, significantly enhancing the system's ability to understand complex and ambiguous intents.

[0053] This invention achieves iterative filtering of intent based on real-time user feedback by organically combining information probes and micro-feedback monitoring, transforming the interaction process from one-way command execution to two-way collaborative perception.

[0054] This invention constructs a complete feedback learning closed loop, continuously optimizing system parameters and decision-making strategies through a reinforcement learning mechanism, enabling the system to adapt to personalized needs and improve interaction efficiency. Attached Figure Description

[0055] Figure 1 This is a schematic diagram of the module flow of the present invention.

[0056] Figure 2 This is a schematic diagram of the method flow of the present invention. Detailed Implementation

[0057] The technical solution of the present invention will be clearly and completely described below with reference to the embodiments. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0058] Please see Figure 1 This invention relates to an intelligent human-computer interaction system based on multimodal features, comprising the following modules:

[0059] Multimodal perception and feature extraction module: Acquires raw signals and generates multimodal feature data through a preprocessing process;

[0060] Candidate intent generation module: Based on multimodal feature data, using a lightweight model, it generates and outputs a list of candidate intents;

[0061] Intent Collaborative Calibration Module: Based on a candidate intent list, it iteratively filters and outputs consensus decisions by initiating interactive operations and parsing real-time user feedback from the candidate intent list.

[0062] Decision execution module: Receives the final execution intention and transforms it into specific control instructions to drive the corresponding terminal devices to complete the operation;

[0063] Feedback and Learning Module: By continuously monitoring the system's interaction process metrics and execution result feedback, the module optimizes the parameters and strategies for the candidate intent list and the final execution intent.

[0064] Specifically, the system synchronously collects raw voice waveform data, visual image sequences, and raw physiological data from users through voice sensors, vision sensors, and physiological sensors, and also receives raw text data. A hardware-triggered synchronization mechanism is used, with the main control chip assigning a unified timestamp to each frame of data. The collected raw data is processed in parallel and features are extracted to obtain voice text content, acoustic emotion feature vectors, visual quantization feature vectors, physiological feature vectors, and text semantic vectors. A central alignment and fusion engine is invoked to align all feature vectors within a time window based on the unified timestamp, and a lightweight cross-modal attention mechanism is used for fusion to generate multimodal feature data.

[0065] Based on multimodal feature data, a unified multimodal context representation vector is generated through a lightweight Transformer encoder. Based on the multimodal context representation vector, an initial probability of each candidate intent in the predefined intent library is calculated through a lightweight inference network. Based on the initial probability of the candidate intent and the semantic similarity between each candidate intent, the final ranking score of each candidate intent is calculated. The candidate intents are sorted in descending order according to the final ranking score, and the top K candidate intents are selected to form a candidate intent list.

[0066] Based on the candidate intent list, the top M intents are selected as trial targets, and an information probe is dynamically generated for each selected intent and bound to the system's available output modality. The information probe sequence is executed sequentially according to the order of the intents, with each information probe lasting for a fixed time window. During the execution of the information probes, the user's physiological and behavioral response signals are collected in real time, and microfeedback features are extracted. Based on the extracted microfeedback features, the user's acceptance of the current trial intent is dynamically evaluated through a consensus calculation model, and the consensus score Cm is calculated. The consensus score Cm is compared with a preset threshold θ in real time to iteratively filter and output a consensus decision.

[0067] The system receives consensus decisions from the intent-based collaborative calibration step, generates control commands corresponding to these decisions, and drives the corresponding terminal devices to complete the operation. It collects process data and result data for each interaction; the process data includes a candidate intent list, an information probe execution sequence, and the consensus decision achievement time; the result data includes user feedback information. It calculates a consensus efficiency evaluation index using the process and result data; it constructs a reinforcement learning environment, generates reward signals based on the consensus efficiency evaluation index and user feedback information, and continuously optimizes the process of selecting trial intents and adjusting a preset threshold θ; based on newly accumulated interaction process and result data, it periodically updates the parameters of the lightweight inference network and the consensus degree calculation model.

[0068] In one embodiment of the present invention, the acquisition of the original signal and the generation of multimodal feature data through a preprocessing procedure include the following steps:

[0069] The system synchronously collects raw voice waveform data, visual image sequences, and raw physiological data from users through voice sensors, vision sensors, and physiological sensors, and also receives raw text data. A hardware-triggered synchronization mechanism is used, with the main control chip adding a unified timestamp to each frame of data.

[0070] Parallel processing and feature extraction are performed on the user's raw voice waveform data, visual image sequence, raw physiological data and raw text data to obtain voice text content, acoustic emotion feature vector, visual quantization feature vector, physiological feature vector and text semantic vector.

[0071] The central alignment and fusion engine is invoked to align all feature vectors within a time window based on a unified timestamp, and a lightweight cross-modal attention mechanism is used for fusion to generate multimodal feature data.

[0072] Specifically, the system acquires raw waveform data of user speech through a voice sensor; acquires visual image sequences and corresponding depth information streams through a vision sensor; simultaneously receives raw text data input by the user via a keyboard, touchscreen, or other input devices; and acquires the user's micro-motion characteristic signals through a physiological sensor. To ensure the time consistency of multimodal data, a hardware-triggered synchronization mechanism is adopted. The main control chip sends a unified synchronization signal to all sensors, assigning a microsecond-level timestamp to each frame of data, thus fundamentally solving the timing inaccuracy problem and ensuring the time alignment of multimodal data.

[0073] The raw data is then preprocessed and features extracted. The raw speech data stream undergoes noise reduction and sound source enhancement using beamforming technology, and is then fed into an end-to-end speech recognition model to be converted into timestamped text. Simultaneously, to overcome the semantic limitations of pure text information, a deep speech emotion model is run in parallel to extract acoustic emotion feature vectors representing emotional states from the acoustic characteristics of speech. The visual data stream uses facial landmark detection, gaze estimation, and posture recognition algorithms to extract visual quantitative feature vectors such as the user's facial action units, attention focus coordinates on the screen, and gestures and body postures. The physiological signal stream uses spectral analysis to extract physiological feature vectors representing the user's vital signs (such as respiratory rate and heart rate) from micro-motion feature signals. Finally, a word embedding model is used to convert the primary text data generated by speech recognition and the raw text data input by the user through other means into machine-understandable semantic vectors.

[0074] A central alignment and fusion engine is invoked to precisely align all the aforementioned feature data within a time window based on a unified timestamp. To deeply fuse these heterogeneous feature data, a lightweight cross-modal attention mechanism is used. This mechanism simulates the mutual query and response of different sensory information, generating a unified multimodal feature representation that comprehensively and contextually describes the user's state. For example, using a pronoun in the text (such as "this") to query the visual attention focus at the same moment. This unified multimodal feature representation is a core form of multimodal feature data as referred to in this invention. In this invention, the multimodal feature data generally refers to structured data obtained after preprocessing, feature extraction, and fusion of the original multimodal signals, which can be directly used by the computational model. Its most preferred form is the high-dimensional feature vector that comprehensively describes the user's state, generated through the aforementioned cross-modal attention mechanism fusion.

[0075] In one embodiment of the present invention, the step of generating and outputting a candidate intent list based on multimodal feature data and using a lightweight model includes the following steps:

[0076] Based on multimodal feature data, a unified multimodal context representation vector is generated through a lightweight Transformer encoder; simultaneously, a candidate intent library is predefined. ;

[0077] Based on the multimodal context representation vector, the initial probability of each candidate intent in the predefined intent library is calculated through a lightweight inference network; and the semantic similarity is calculated based on the embedding vector generated by the semantic representation of each candidate intent.

[0078] Based on the initial probability of the candidate intents and the semantic similarity between each candidate intent, the final ranking score of each candidate intent is calculated.

[0079] The formula for calculating the final ranking score is as follows: ;

[0080] in, Candidate intention The final sorting score, The initial probability of the candidate intent. Here, N represents the diversity weighting coefficient, and N is the total number of intents in the predefined candidate intent library. Candidate intention and semantic similarity;

[0081] The candidate intentions are sorted in descending order based on the final ranking score, and the top K candidate intentions are selected to form a candidate intention list. .

[0082] Specifically, based on multimodal feature data, the data includes, but is not limited to: text semantic vectors. Acoustic Emotion Feature Vector And a visual feature vector that integrates facial motion unit intensity, gaze coordinates (x, y) in screen coordinates, and gesture encoding. The received heterogeneous feature vectors are concatenated into a long vector. It is then fed into a lightweight Transformer encoder to generate a uniform, fixed-dimensional multimodal context representation vector H;

[0083] Predefine a candidate intent library Each candidate intent Each corresponds to an operation instruction described in natural language; and the description of each candidate intent is encoded through a pre-trained language model, resulting in a description for each candidate intent. Precompute a semantic embedding vector The vector is stored in a fixed candidate intent library; the multimodal context representation vector H is input into a lightweight inference network, namely a multilayer perceptron, and each candidate intent is calculated through the softmax function. initial probability score The calculation formula is: ,in, and Candidate intents in a multilayer perceptron The corresponding weights and biases.

[0084] Weights in a multilayer perceptron and bias The model is trained using a supervised learning paradigm. The source of the labeled dataset required for training is as follows: When an interaction reaches consensus through the intent co-calibration module and is successfully completed by the decision execution module, the multimodal context representation vector H during that interaction is automatically labeled as the final consensus decision intent label. This process utilizes the normal operation of the system to achieve automated collection of labeled data. During the system development and reinforcement phases, the virtual sandbox execution environment can be used to programmatically simulate massive user interaction scenarios. By changing input parameters such as voice commands and visual features, and automatically generating corresponding real intent labels according to predefined rules, simulated labeled data can be synthesized in batches for initial and reinforcement training of the model. After constructing the training set through one or more of the above methods, the labeled training dataset is used to minimize the cross-entropy loss of intent prediction, and the parameters are adjusted using the backpropagation algorithm. and Optimize to obtain its final value.

[0085] To prevent the candidate intent list from being monopolized by high-probability candidate intents with similar semantics, a diversity mechanism is introduced for filtering. This mechanism increases the diversity of the candidate set by reducing the final ranking score of candidate intents in the list that are too semantically similar to other candidate intents. For each candidate intent... The algorithm calculates the average semantic similarity between the candidate intent and other candidate intents in the current candidate set, and uses this average semantic similarity as a penalty to reduce the final ranking score of candidate intents in the list that have excessively high semantic similarity to other candidate intents. It also calculates pairwise candidate intents in the candidate intent library using the cosine similarity method. and semantic similarity between The specific formula is as follows .

[0086] Through calculation formula ; Obtain candidate intent Final sort score , where λ is the diversity weight, is the candidate intent and is the cosine similarity (negative value indicates difference), and N is the total number of intents in the predefined candidate intent library.

[0087] According to the final sorting score sort the intents in descending order, and select the top K (K < N) candidate intents to form the generated candidate intent list , and each entry in this list is a data structure, including the intent unique identifier, the natural language description of the intent, the final sorting score , the initial probability score and the intent embedding vector .

[0088] The value range of the diversity weight coefficient λ is usually set between [0.2, 0.8], and the optimal value needs to be determined through experiments; when λ is too small (λ < 0.2), the candidate list is easily monopolized by high-probability intents with similar semantics, reducing the system's ability to handle fuzzy intents; when λ is too large (λ > 0.8), the guiding role of the initial probability will be overly weakened, which may lead to the exclusion of key intents; the specific value should be determined in the system development stage by testing the impact of different λ values on the quality of the candidate list on the validation set, and selecting the optimal value that can maximize the comprehensive score.

[0089] In one embodiment of the present invention, based on the candidate intent list, by initiating an interactive operation and parsing the user's real-time feedback, iteratively filter and output a consensus decision from the candidate intent list, including the following steps:

[0090] Based on the candidate intent list, select the top M intents as the probing objects, and dynamically generate information probes bound to the system's available output modalities for each selected intent;

[0091] Execute the probe sequence in the order of the intent sorting, and each probe lasts for a fixed time window; during the probe execution, through the voice sensor, visual sensor and physiological sensor, collect the user's physiological and behavioral response signals in real time, and extract micro-feedback features, including visual features, voice features, and physiological features;

[0092] Based on the extracted micro-feedback features, dynamically evaluate the user's approval degree of the current probing intent through the consensus calculation model, and calculate the consensus degree ;

[0093] Compare the consensus degree with the preset threshold θ in real time: if ≥ θ, immediately terminate the current probe sequence and confirm the current probing intent as the consensus decision; if If <θ, immediately withdraw the current probe effect and continue to execute the probe for the next intention in the candidate intention list;

[0094] When the entire probe sequence is executed and consensus is still not reached, trigger the degradation processing mechanism and switch to the direct instruction receiving mode based on traditional speech recognition.

[0095] Specifically, the system connects to and drives multiple terminal devices, including but not limited to display screens, speakers, and hardware devices with haptic feedback capabilities. This module calibrates intentions by outputting information probes to the user and monitoring their feedback signals. The feedback signals are divided into two categories: micro-feedback, that is, the user's unintentional and subtle physiological and behavioral responses, including eye gaze deviation, facial expression changes, intonation fluctuations, and heart rate variability, etc.; explicit feedback, that is, the user's clear and active confirmation behaviors, such as speaking out equivalent keywords, nodding, or making confirmation gestures.

[0096] From the candidate intention list Select the top M intentions (M is a system configuration parameter and M < K, the typical value is 3) with the highest final sorting scores as the exploration objects for the current iteration cycle. For each selected intention Dynamically generate a low-cost and reversible information probe. The specific manifestation form of this probe is bound to the available output modalities of the system: the visual probe is to subtly highlight the UI elements related to this intention in the graphical user interface, for example, increase the color temperature of the button border by 100K; the voice probe is to play the keywords related to the intention through the text-to-speech engine with a tentative intonation of increasing the pitch by 5% and reducing the speech rate by 20%; the haptic probe is when the terminal device connected to the system supports haptic feedback, trigger a group of short pulses with a duration of 100ms through the linear motor of the terminal device, and set the vibration intensity to a weak level.

[0097] Execute the probe sequence in the order of intention sorting. The duration of each probe is a short and fixed time window . During the execution of the probe, simultaneously start the multi-modal micro-feedback monitoring pipeline to collect the physiological and behavioral response signals of the user in the time window of the probe execution Extract micro-feedback features using real-time processing capabilities: visual features include calculating the Euclidean distance offset of the user's eye gaze point relative to the reference position before the probe trigger , and calculating the average pixel motion vector modulus of the region of interest on the face through the optical flow method as the facial expression change intensity e; the voice feature is to detect whether there is a voice interruption during the execution of the probe and calculate the standard deviation of the voice fundamental frequency as the intonation change amount v; Physiological characteristics are derived by analyzing the time-domain features of heart rate variability using millimeter-wave radar signals from physiological sensors, and calculating the normalized heart rate variability. h.

[0098] Based on the extracted micro-feedback features, a consensus calculation model is used to dynamically evaluate the user's current probing intent. After calculating the consensus level Cm based on the degree of acceptance, the consensus level Cm is compared with a preset threshold θ in real time: if Cm ≥ θ, the current probe sequence is immediately terminated to confirm the intent. For consensus decision-making, the results are passed to the decision execution module; if Cm < θ, the current probe effect is immediately withdrawn, and the probe of the next intent in the candidate intent list is executed after a preset fixed delay; when all probe sequences have been executed and no consensus has been reached, a degradation processing mechanism is triggered: a prompt to clarify your needs is output through speech synthesis, and the system enters the direct command receiving mode based on traditional speech recognition.

[0099] The determination and optimization of the preset threshold θ rely on the historical interaction data collected by the feedback and learning module, especially the analysis of the distribution of consensus degree Cm in consensus achievement and consensus failure cases, so as to determine a threshold value that can effectively distinguish the user's intention recognition state; this threshold can be configured and adjusted according to the task criticality requirements of specific application scenarios.

[0100] In one embodiment of the present invention, the calculation of consensus includes the following steps:

[0101] The calculation formula based on the consensus calculation model: ;

[0102] Achieving consensus ,in, τ represents the microfeedback feature data; wk represents the weighting coefficient of each modal signal; τ represents the time decay factor; η represents the explicit feedback reward coefficient; and t represents the normalized probe process time. This is an explicit feedback indicator function.

[0103] Specifically, consensus The calculation formula is: ;

[0104] Among them, microfeedback eigenvalues This refers to four specific microfeedback feature data, specifically defined as follows: when k=1, represent g; when k=2 represent e; When k=3 represent v; When k=4 represent h;

[0105] t represents the normalized probe process time, the time from the start of execution of the current information probe. The time difference to the current calculation time, and the preset fixed duration window of the probe. The ratio, i.e. And normalize it, with the value range being [0,1];

[0106] Weight coefficient wk: During system initialization, initial weights are set based on prior knowledge of the reliability of each modality signal (e.g., visual 0.4, speech 0.3, physiological 0.3); at the same time, the weights are dynamically updated through an online learning mechanism. Specifically, the weights are adjusted using reinforcement learning algorithms (e.g., stochastic gradient descent) based on the final result (success or failure) of each interaction between the feedback and the learning module.

[0107] explicit feedback indicator function The value is 1 when a clear affirmation is detected, and 0 otherwise. The function takes the value 1 if and only if the system recognizes any of the following clear signals in real time: detecting a preset affirmative voice keyword (such as "yes") through the voice sensor; detecting a clear nodding action or a specific confirmation gesture (such as raising a thumb) through the visual sensor.

[0108] Time decay factor τ: This is a preset hyperparameter of the system. Its value is determined through pre-experiments during the system development phase. The specific determination process is as follows: using the historical interaction data collected by the feedback and learning module, the average consensus decision achievement time and consensus decision accuracy of the system under different τ values ​​are analyzed to find the optimal balance between the two. After multiple rounds of parameter tuning experiments, the τ value that can achieve the highest comprehensive consensus efficiency is taken as the final value and fixed in the system.

[0109] Explicit feedback reward coefficient η: This is a preset hyperparameter of the system. Its core design principle is to ensure that any explicit feedback can decisively improve the consensus level Cm, thereby being unconditionally prioritized by the system. To achieve this, the value of η must satisfy: η > 1 - min This means that even in the extreme case where all micro-feedback contributions are at their lowest levels, explicit feedback alone can still cause the consensus Cm to exceed a preset threshold θ. Its specific value is determined during the system development phase by testing the degree to which different η values ​​satisfy the above design principles and their impact on the smoothness of interaction, and is ultimately solidified in the system.

[0110] The four specific microfeedback feature data and their calculation methods are as follows:

[0111] Eye gaze deflection The calculation method is as follows: based on the gaze focus coordinates extracted by the multimodal perception and feature extraction module; at the probe triggering moment... Record the coordinates of the reference gaze point ( , ), calculate the Euclidean distance between the current gaze point (x, y) and the reference point at time t: ;right Perform normalization and map it to the [0,1] interval;

[0112] Intensity of facial expression changes The calculation method for e is as follows: extract the intensity of facial action units from visual features, calculate the intensity change of specific facial action units (such as eyebrows and corners of the mouth) before and after probe triggering, and calculate the weighted sum of their absolute values ​​to obtain e. e; for e is normalized and mapped to the [0,1] interval;

[0113] intonation variation The calculation method for v is as follows: Based on the acoustic emotion feature vector processing pipeline, during probe execution, the standard deviation of the user's voice fundamental frequency is calculated in real time and compared with the reference fundamental frequency of the user in a calm state; the magnitude of the change is the standard deviation. v; to v is normalized and mapped to the [0,1] interval;

[0114] Heart rate variability The h calculation method is as follows: Physiological feature vectors extracted from physiological sensors are used to perform spectral analysis on the acquired raw micro-motion signals, extracting energy changes in specific frequency bands. These changes are related to heart rate variability and are normalized to obtain h. h, and will h is mapped to the interval [0,1].

[0115] In one embodiment of the present invention, receiving the final execution intention and converting it into specific control instructions to drive the corresponding terminal device to complete the operation includes the following steps:

[0116] Receive a consensus decision from the intent coordinating calibration module, the consensus decision containing a unique identifier for the intent selected from the candidate intent list;

[0117] Generate control instructions that drive the terminal device, corresponding to the unique identifier of the intent.

[0118] The control command is sent to the corresponding terminal device to execute the operation specified by the consensus decision.

[0119] Specifically, it receives consensus decisions from the intent collaborative calibration module. These consensus decisions are the intent entries that are ultimately selected from the candidate intent list. Their data structure includes, but is not limited to, a unique identifier for the intent and a natural language description of the intent.

[0120] The system maintains a predefined instruction mapping table, which establishes a mapping relationship from the unique identifier of the intent to the specific device control instruction. This mapping relationship is predefined during the system development phase. For example, for a consensus decision with the intent identifier "volume up", the mapping table resolves it to an increase volume control instruction and parameters for the speaker device.

[0121] Based on the unique identifier of the intent in the consensus decision, the instruction mapping table is queried to parse out the specific operation sequence and operation object required to complete the intent; the device control instruction corresponding to the consensus decision is then generated. Simultaneously, the generated control instructions all conform to the underlying communication protocol recognizable by the terminal device, ensuring the executability of the instructions.

[0122] To ensure system security, control commands are validated in a virtual sandbox execution environment before being sent to physical devices. Only commands that pass the sandbox environment's security validation are approved for execution. This sandbox environment is an isolated environment built using containerization technology; the following operations are performed within this environment:

[0123] Command security verification: The system simulates the execution of control commands to detect whether there are any illegal operations or resource out-of-bounds behaviors;

[0124] Device status simulation: The state of each terminal device is simulated in the sandbox to predict the result after the command is executed and avoid conflicting commands (such as executing open and close at the same time).

[0125] Resource consumption assessment: Assess the computing, storage, and network resources required to execute this instruction, ensuring that they are within the limits allowed by the system load.

[0126] The system connects to and drives various terminal devices, including displays, speakers, and hardware devices with haptic feedback capabilities; it sends verified control commands to the following target devices via dedicated driver interfaces or communication buses:

[0127] Drive the display screen: Send graphical user interface update commands or content data streams to the display screen to control the display screen to complete interface rendering and display;

[0128] Drive the speaker: Send the audio stream synthesized by the text-to-speech engine, or the URL of the audio file to be played, to the speaker to control its playback, pause, or volume adjustment;

[0129] Driving haptic devices: Sending specific vibration patterns, intensities, and durations to the device's linear motor controller to trigger precise haptic feedback.

[0130] In one embodiment of the present invention, the step of optimizing the candidate intent list and the final execution intent by continuously monitoring the interaction process indicators and execution result feedback of the system includes the following steps:

[0131] By continuously monitoring the interaction process indicators and execution result feedback of the system, process data and result data for each interaction are collected; the process data includes a candidate intent list, information probe execution sequence and consensus decision achievement time, and the result data includes user feedback information.

[0132] A consensus efficiency evaluation index is calculated based on process data and outcome data.

[0133] A reinforcement learning environment is constructed, and a reward signal is generated based on the consensus efficiency evaluation index and user feedback information. The process of selecting trial intentions and adjusting the preset threshold θ is continuously optimized.

[0134] Based on the newly accumulated interaction process and result data, the parameters of the lightweight inference network in the candidate intent generation module and the parameters of the consensus calculation model in the intent collaboration calibration module are periodically updated.

[0135] Specifically, in each human-computer interaction session, process data and result data are collected from each stage of the system. The process data includes a complete record of the interaction path from candidate intent generation to consensus decision achievement, specifically including: the candidate intent list L for each round of interaction and its ranking score, the sequence and order of executed information probes, and the degree of consensus. Curves changing over time And the time taken from the start of the interaction to reaching a consensus decision. The results data records the final effectiveness of the interaction, including the number of explicit negative feedback responses from users after the results are presented. (e.g., corrections via voice or gestures), and success indicators based on the completion of task objectives.

[0136] Define a consensus efficiency evaluation index E, which aims to balance the speed (time consumption) of consensus decision-making with resource consumption (number of probes), so that the system will spontaneously tend to use fewer probes and reach a consensus decision in a shorter time during the optimization process. Its calculation formula is: ;in, The total number of information probes used to reach a consensus decision; γ is the penalty coefficient for the frequency of probe use; For a much smaller A smoothing factor is used to prevent the denominator from being zero. The probe count penalty coefficient γ is determined during the system development phase using hyperparameter tuning methods such as grid search or Bayesian optimization. Specifically, the historical interaction data collected by the feedback and learning module is used to optimize the overall mean of the consensus efficiency evaluation index E. Different candidate values ​​of γ are tested, and the candidate value of γ that maximizes E is determined as the final system configuration parameter.

[0137] A reinforcement learning environment is constructed to continuously optimize the decision-making strategy of the intention collaborative calibration module. The state space st is defined as the combination of the current multimodal context representation vector H and the current candidate intention list L, representing the instantaneous interaction context of the system.

[0138] The action space at encompasses two types of decisions: one is to select the intention of the next probe in the probe sequence, and the other is to fine-tune the preset threshold θ in the consensus model.

[0139] The reward function rt is calculated on the fly based on the result of each attempt, and the formula is: Where E is the consensus efficiency score of this attempt; The number of negative feedback received; The indicator function is 1 if no consensus is reached within the timeout period, and 0 otherwise; α, β, and δ are the weighting coefficients of each reward component;

[0140] The timeout here refers to the time starting from the first execution of the probe, and if it is within the system's preset maximum waiting time... If no consensus is reached within the time limit, it will be deemed as a timeout. The maximum time required to determine this is based on the potential duration of the probe sequence, and its calculation formula is as follows: =M ( +δ); where M is the number of candidate intentions selected in each round of probing (i.e., the upper limit of the number of probes). δ is the fixed duration window for a single probe; δ is the fixed interval required for the system to undo the current effect and load the next probe between two adjacent probes.

[0141] The weighting coefficients α, β, and δ are used to adjust the relative importance of consensus efficiency, user negative feedback, and timeout failure in the reward, respectively. The values ​​of these coefficients are determined based on the statistical characteristics of the historical interaction data collected by the feedback and learning modules. Specifically, during the system initialization or retraining phase, the variance of the consensus efficiency score E and the number of negative feedback events are calculated using the accumulated process and result data. The variance of E and the frequency of timeout events; α is the reciprocal of the variance of E, and β is... The reciprocal of the variance, δ, is the reciprocal of the frequency of timeout events. Based on the baseline weights calculated here, system administrators or developers can manually configure and adjust these baseline weights according to the needs of specific application scenarios, and finally complete the optimization through the reinforcement learning training process.

[0142] Training data is collected based on actual interactions with users during normal operation, and a proximal policy optimization algorithm is used to continuously optimize the decision logic of the intent-coordinated calibration module. This decision logic is implemented by a parameterized policy network π, which maps the current interaction state to the probability distribution of the next action. The optimization process is an iterative data-driven loop: First, the system interacts with the user using the current policy network, collecting a large amount of interaction trajectory data. Each trajectory includes the state, action, and corresponding immediate reward. Then, the algorithm evaluates the merits of the executed actions by calculating the advantage function, and adjusts the parameters of the policy network using the gradient ascent method under the constraint of limiting the single-step update magnitude. The optimization objective of this process is to maximize the expected discount cumulative reward, which is defined as the sum of discounts on all future immediate rewards, i.e. ,in This serves as the discount factor. By repeatedly executing this loop, the policy network is continuously optimized so that when it receives different multimodal interaction contexts (states st), the output action probability distribution can lead to a higher expected discount cumulative reward, thereby improving the speed and accuracy of consensus achievement for intent calibration.

[0143] Among them, discount factor To balance the importance of near-term and long-term returns, its value ranges from 0.9 to 0.99. This range is determined based on the fundamental characteristics of human-computer interaction tasks: a single intent calibration session can typically be completed within a finite number of steps (i.e., the complete execution of the probe sequence), and a value close to 1... A value (e.g., 0.99) ensures that the reinforcement learning agent, when making decisions, fully considers the impact of its current action on the outcome of interactions within several future steps, which matches the step size required to complete a full calibration session. The specific final value of this hyperparameter is determined during the system development phase by performing a grid search in the aforementioned virtual sandbox execution environment with the goal of maximizing the expected value of long-term cumulative rewards, and is then embedded into the system.

[0144] A periodic model update process is established to continuously optimize the core algorithm: In terms of consensus model weight optimization, the system uses newly accumulated interaction data with success or failure labels to retrain the micro-feedback feature weight wk in the consensus calculation formula, enabling the model to more accurately interpret users' nonverbal feedback; In terms of candidate intent generation model optimization, the system uses consensus decisions confirmed and successfully executed by the intent co-calibration module as new high-quality labeled data (multimodal context representation vector H, true intent label) to incrementally fine-tune the lightweight inference network in the candidate intent generation module, thereby continuously improving the ranking accuracy of initial candidate intents.

[0145] Please see Figure 2 As shown, this invention is an intelligent human-computer interaction method based on multimodal features, comprising the following steps:

[0146] S1: Acquire raw signals and generate multimodal feature data through a preprocessing process;

[0147] S2: Based on multimodal feature data, a lightweight model is used to generate and output a list of candidate intentions;

[0148] S3: Based on the candidate intent list, initiate interactive operations and parse the user's real-time feedback to iteratively filter from the candidate intent list and output a consensus decision;

[0149] S4: Receive the final execution intention and convert it into specific control instructions to drive the corresponding terminal device to complete the operation;

[0150] S5: Optimize parameters and strategies for the candidate intent list and the final execution intent by continuously monitoring the interaction process indicators and execution result feedback of the system.

[0151] The above embodiments are only used to illustrate the technical methods of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical methods of the present invention without departing from the spirit and scope of the technical methods of the present invention.

Claims

1. An intelligent human-computer interaction system based on multimodal features, characterized in that, Includes the following modules: Multimodal perception and feature extraction module: Acquires raw signals and generates multimodal feature data through a preprocessing process; Candidate intent generation module: Based on multimodal feature data, using a lightweight model, it generates and outputs a list of candidate intents; Intent Collaborative Calibration Module: Based on a candidate intent list, it iteratively filters and outputs consensus decisions by initiating interactive operations and parsing real-time user feedback from the candidate intent list. Decision execution module: Receives the final execution intention and transforms it into specific control instructions to drive the corresponding terminal devices to complete the operation; Feedback and Learning Module: By continuously monitoring the system's interaction process metrics and execution result feedback, the module optimizes the parameters and strategies for the candidate intent list and the final execution intent.

2. The intelligent human-computer interaction system based on multimodal features according to claim 1, characterized in that, The process of acquiring raw signals and generating multimodal feature data through a preprocessing workflow includes the following steps: The system synchronously collects raw voice waveform data, visual image sequences, and raw physiological data from users through voice sensors, vision sensors, and physiological sensors, and also receives raw text data. A hardware-triggered synchronization mechanism is used, with the main control chip adding a unified timestamp to each frame of data. Parallel processing and feature extraction are performed on the user's raw speech waveform data, visual image sequences, raw physiological data, and raw text data to obtain speech-text content, acoustic emotion feature vectors, visual quantization feature vectors, physiological feature vectors, and text semantic vectors. The central alignment and fusion engine is invoked to align all feature vectors within a time window based on a unified timestamp, and a lightweight cross-modal attention mechanism is used for fusion to generate multimodal feature data.

3. The intelligent human-computer interaction system based on multimodal features according to claim 1, characterized in that, The process of generating and outputting a candidate intent list based on multimodal feature data and using a lightweight model includes the following steps: Based on multimodal feature data, a unified multimodal context representation vector is generated through a lightweight Transformer encoder; simultaneously, a candidate intent library is predefined. ; Based on the multimodal context representation vector, the initial probability of each candidate intent in the predefined intent library is calculated through a lightweight inference network; and the semantic similarity is calculated based on the embedding vector generated by the semantic representation of each candidate intent. Based on the initial probability of the candidate intents and the semantic similarity between each candidate intent, the final ranking score of each candidate intent is calculated. The formula for calculating the final ranking score is as follows: ; in, Candidate intention The final sorting score, The initial probability of the candidate intent. Here, N represents the diversity weighting coefficient, and N is the total number of intents in the predefined candidate intent library. Candidate intention and semantic similarity; The candidate intentions are sorted in descending order based on the final ranking score, and the top K candidate intentions are selected to form a candidate intention list. .

4. The intelligent human-computer interaction system based on multimodal features according to claim 1, characterized in that, The process of iteratively filtering and outputting a consensus decision based on a candidate intent list by initiating interactive operations and parsing real-time user feedback includes the following steps: Based on the candidate intent list, the top M intents are selected as the probe targets, and information probes are dynamically generated for each selected intent and bound to the available output modalities of the system. The probe sequence is executed sequentially according to the intended order, with each probe lasting for a fixed time window. During the probe execution, the user's physiological and behavioral response signals are collected in real time through voice sensors, visual sensors, and physiological sensors, and microfeedback features, including visual features, voice features, and physiological features, are extracted. Based on the extracted micro-feedback features, a consensus degree calculation model is used to dynamically evaluate the user's level of acceptance of the current probing intention, and the consensus degree is calculated. ; Real-time comparison of consensus Compared with the preset threshold θ: if If ≥θ, then immediately terminate the current probe sequence and confirm that the current probing intention is a consensus decision; if If <θ, the current probe effect is immediately withdrawn, and the probe for the next intent in the candidate intent list is executed. If no consensus is reached after all the probe sequences have been executed, a downgrade mechanism is triggered, switching to a direct command receiving mode based on traditional speech recognition.

5. The intelligent human-computer interaction system based on multimodal features according to claim 4, characterized in that, The calculation of consensus includes the following steps: The calculation formula based on the consensus calculation model: ; Achieving consensus ,in, τ represents the microfeedback feature data; wk represents the weighting coefficient of each modal signal; τ represents the time decay factor; η represents the explicit feedback reward coefficient; and t represents the normalized probe process time. This is an explicit feedback indicator function.

6. The intelligent human-computer interaction system based on multimodal features according to claim 1, characterized in that, The process of receiving the final execution intent and converting it into specific control commands to drive the corresponding terminal device to complete the operation includes the following steps: Receive a consensus decision from the intent coordinating calibration module, the consensus decision containing a unique identifier for the intent selected from the candidate intent list; Generate control instructions that drive the terminal device, corresponding to the unique identifier of the intent. The control command is sent to the corresponding terminal device to execute the operation specified by the consensus decision.

7. The intelligent human-computer interaction system based on multimodal features according to claim 1, characterized in that, The process of continuously monitoring the interaction process metrics and execution result feedback of the system to optimize the parameters and strategies of the candidate intent list and the final execution intent includes the following steps: By continuously monitoring the interaction process indicators and execution result feedback of the system, process data and result data for each interaction are collected; the process data includes a candidate intent list, information probe execution sequence and consensus decision achievement time, and the result data includes user feedback information. A consensus efficiency evaluation index is calculated based on process data and outcome data. A reinforcement learning environment is constructed, and a reward signal is generated based on the consensus efficiency evaluation index and user feedback information. The process of selecting trial intentions and adjusting the preset threshold θ is continuously optimized. Based on the newly accumulated interaction process and result data, the parameters of the lightweight inference network in the candidate intent generation module and the parameters of the consensus calculation model in the intent collaboration calibration module are periodically updated.

8. A method for intelligent human-computer interaction based on multimodal features, characterized in that, Includes the following steps: S1: Acquire raw signals and generate multimodal feature data through a preprocessing process; S2: Based on multimodal feature data, a lightweight model is used to generate and output a list of candidate intentions; S3: Based on the candidate intent list, initiate interactive operations and parse the user's real-time feedback to iteratively filter from the candidate intent list and output a consensus decision; S4: Receive the final execution intention and convert it into specific control instructions to drive the corresponding terminal device to complete the operation; S5: Optimize parameters and strategies for the candidate intent list and the final execution intent by continuously monitoring the interaction process indicators and execution result feedback of the system.

Images