Dialogue Agent interaction method based on multimodal intention understanding

Abstract

The invention relates to the technical field of man-machine interaction, in particular to a dialogue Agent interaction method based on multi-modal intention understanding, which comprises the following steps: S1, collecting multi-modal data in a user interaction process in real time, and calculating a time synchronization deviation value of each modal data source; s2, constructing a space-time fusion feature vector; s3, analyzing a dominant action instruction and a recessive behavior clue in the space-time fusion feature vector; s4, generating a multi-level intention analysis tree; s5, when the corrected confidence coefficient of any node in the intention analysis tree is lower than a set threshold value, activating a targeted sensor to complementarily collect data; and S6, analyzing a tree drive response decision according to the finally confirmed intention. According to the method, high-precision identification and response control of the dialogue Agent on the user intention in a complex scene are realized by constructing a multi-modal interaction method with space-time consistency fusion capability, an explicit and implicit intention analysis mechanism and an adaptive modal clarification strategy.

Description

Technical Field

[0001] This invention relates to the field of human-computer interaction technology, and in particular to a dialogue agent interaction method based on multimodal intent understanding. Background Technology

[0002] Currently, intelligent interaction systems are widely used in smart homes, vehicle control, service robots, and virtual assistants. Their core lies in the system's ability to accurately understand user intent and generate effective responses accordingly. With the development of multimodal perception technology, the information provided by users during interaction is no longer limited to text or voice, but also includes spatiotemporal information such as video images and sensor data. To improve interaction accuracy, researchers generally employ multimodal fusion methods to jointly model different data sources, extracting rich semantic cues to enhance the dialogue agent's ability to understand user intent. However, existing methods generally face problems such as temporal asynchrony between modalities, lack of spatial relationships, and difficulty in identifying implicit needs, making it difficult to meet the requirements for accurate parsing of user intent in complex and ever-changing scenarios.

[0003] In practical applications, user intent often manifests as a hybrid structure of explicit actions and implicit cues. Traditional methods often fail to form a stable and reliable intent structure, especially when user expression is ambiguous, behavior is vague, or modal data is incomplete. Furthermore, the dynamic alignment mechanism between multimodal data is imperfect, and the spatiotemporal mapping structure lacks a unified model, leading to conflicts or weak correlations in the fusion results. Simultaneously, the lack of an adaptive clarification mechanism based on confidence levels prevents the system from making effective judgments and providing feedback at critical nodes. Therefore, a dialogue agent interaction method based on multimodal intent understanding is urgently needed to address these issues. Summary of the Invention

[0004] To achieve the above objectives, the present invention provides a dialogue agent interaction method based on multimodal intent understanding.

[0005] A dialogue agent interaction method based on multimodal intent understanding includes the following steps:

[0006] S1: Real-time acquisition of text streams, voice waveforms, video frame sequences, and sensor timing data during user interaction, and calculation of time synchronization deviation values ​​for each modal data source;

[0007] S2: Dynamically compensate and align multimodal data based on time synchronization deviation value, establish cross-modal correlation mapping through physical spatiotemporal constraints, and generate spatiotemporal fusion feature vectors;

[0008] S3: Analyze the explicit action instructions and implicit behavioral cues in the spatiotemporal fusion feature vector, and output a triple containing explicit intention, implicit need and corresponding confidence score;

[0009] S4: Combine the user's historical behavior profile with the current dialogue state to dynamically adjust the confidence score in the triple and generate a multi-level intent parsing tree;

[0010] S5: When the corrected confidence of any node in the intent parsing tree is lower than the set threshold, activate the targeted sensor to supplement the collected data, generate a multimodal verification question, and output the intent clarification instruction set;

[0011] S6: Drive response decisions based on the final confirmed intent parsing tree, and simultaneously execute voice feedback content, visual interface operations, and environmental control signals.

[0012] Optionally, S1 specifically includes:

[0013] S11: Receive real-time text stream data input by the user through the text input interface;

[0014] S12: Acquire user voice waveform data via microphone array;

[0015] S13: Capture video frame sequences during user interaction using a camera;

[0016] S14: Collect time-series sensing data of user actions or physiological states through wearable sensing devices;

[0017] S15: Based on text stream timestamps, speech segment timestamps, video frame timestamps, and sensor sampling timestamps, a reference clock is used for unified calibration to map the timestamps of various modal data to the same time base, and the synchronization deviation value between any two modal data is calculated accordingly.

[0018] Optionally, S2 specifically includes:

[0019] S21: Based on the synchronization deviation value calculated by S1, the time axis of the speech waveform data is linearly interpolated to align the speech waveform with the character input time sequence of the text stream.

[0020] S22: By interpolating or dropping frames, the time axis of the video frame sequence is adjusted according to the synchronization deviation value to ensure that the shooting time of the video frame matches the aligned speech waveform data within millisecond precision.

[0021] S23: Based on the synchronization deviation value, the sliding window method is used to truncate or expand the sensor time series data to ensure that the sampling window of the sensor data corresponds one-to-one with the time domain boundary of the speech waveform and video frame data window.

[0022] S24: Based on the spatial distance between the user and the Agent and the physical space constraints of the interactive interface layout, determine the spatial relationship between text streams, voice waveforms, video frames and sensor data, and establish a spatial association mapping between cross-modal data with spatial location as nodes and data type as attributes;

[0023] S25: Based on the fused time-aligned modal data and cross-modal spatial correlation mapping, they are combined to construct a spatiotemporal fusion feature vector.

[0024] Optionally, S24 specifically includes:

[0025] S241: Obtain the spatial coordinates of each modal acquisition device in the user interaction environment. Let the spatial positions of the text input device, voice acquisition microphone, camera, and sensor be P, respectively. T =(x T ,y T , z T ),P S =(x S ,y S , z S ),P V =(x V ,y V , z V ) and P D =(x D ,y D , z D ), where x, y, z are three-dimensional spatial coordinates;

[0026] S242: Calculate the spatial distance D between any two modal devices based on their spatial coordinates. i,j ;

[0027] S243: Spatial distance D i,j For edge weights, construct an undirected graph G = (N, E), where the node set N represents the data sources of each modality, and the edge set E represents the spatial association between modalities, where the association strength is inversely proportional to the edge weight, i.e.:

[0028] Where w i,j The spatial correlation weights between mode i and mode j are given, where ∈ is a non-zero small constant.

[0029] S244: Normalize the graph structure G and use a graph embedding algorithm to convert the spatial structure representation into a fixed-dimensional embedding vector as a spatial association feature.

[0030] Optionally, S25 specifically includes:

[0031] S251: Extract features from the time-aligned text stream, speech waveform, video frame sequence, and sensor data to obtain the corresponding modal feature vectors;

[0032] S252: For the spatial embedding vectors of each modal node in the S24 graph structure, the modal feature vectors and spatial embedding vectors are combined by vector concatenation to generate spatiotemporal modal feature vectors.

[0033] S253: Spatial correlation weights w constructed based on S24 i,j The spatiotemporal feature vectors between modes are weighted and fused to form a unified spatiotemporal fusion feature vector F. ST ;

[0034] S254: Spatiotemporal fusion feature vector F ST Normalization is performed to obtain the final spatiotemporal fusion feature vector used for intent parsing.

[0035] Optionally, S3 specifically includes:

[0036] S31: The spatiotemporal fusion feature vector F obtained in S25 ST The input is fed into a multi-layer dual-branch decoding network structure, which includes an explicit intent recognition branch and an implicit demand recognition branch.

[0037] S32: In the explicit intent recognition branch, a multi-class neural network is used to classify F. ST Perform category discrimination and output the intent type vector I of the explicit action command. exp and its corresponding confidence score vector C exp Each component corresponds to a dominant operation category and its probability of occurrence;

[0038] S33: In the implicit demand identification branch, a graph attention network combined with the user's historical interaction behavior graph is used to analyze F. ST Perform context graph embedding and output the implicit requirement vector I. imp and its corresponding confidence score C imp ;

[0039] S34: Perform soft maximum normalization on the results of the dominant and recessive branches respectively to obtain the final dominant intention triplet (i exp , l exp c exp ) and implicit demand triples (i imp ,l imp ,c imp ), where i represents the identification number, l represents the semantic label, and c represents the normalized confidence score;

[0040] S35: Combine the two triples mentioned above to form a complete intent recognition output structure, which serves as the result of multimodal intent parsing.

[0041] Optionally, S33 specifically includes:

[0042] S331: Constructing a user history interaction behavior graph G U =(N U E U ), where the set of nodes N U For user history interaction events, the edge set E U This refers to the interaction relationships between events;

[0043] S332: The spatiotemporal fusion feature vector F obtained in S25 ST This is used as a feature of the current interaction node and added to the historical interaction behavior graph G. U In this process, an enhanced contextual interaction graph G is formed. C =(N U ∪{n cur}, E C ), where n cur The current interaction node;

[0044] S333: With the enhanced spectrum G C As input, use a graph attention network to compute the attention weights 'a' between nodes. ij ;

[0045] S334: Based on the calculated attention weight a ij Then, perform node feature aggregation and update to obtain the updated latent requirement embedding vector h. cur ′;

[0046] S335: Embed the updated implicit requirement into vector h cur The input is fed into a single-layer fully connected network and activated by the Sigmoid function, outputting a latent demand vector I. imp and the corresponding confidence score vector C imp .

[0047] Optionally, S4 specifically includes:

[0048] S41: Based on user historical behavior profile data, extract confidence records of past user interaction events and statistical information on user behavior preferences to form a historical confidence reference table;

[0049] S42: Match the confidence scores in the explicit intention and implicit need triples obtained in S3 with the historical confidence reference table, and adjust the confidence scores according to the matching degree. The adjusted confidence score is defined as: c′=α1·c+(1-α1)·c histWhere c is the initial confidence score, c hist The average confidence score for the corresponding event in the historical confidence reference table, where α1 is the fusion factor;

[0050] S43: Determine the stage and context of the user's interaction with the Agent based on the current dialogue state, set the dialogue state correction weight β, and combine it with the adjusted confidence score to perform state correction, and obtain the second-corrected confidence score c″, the expression of which is: c″=c′·β;

[0051] S44: Using the semantic categories of explicit intent and implicit needs as the root node of the tree, and the confidence score after secondary correction as the node weight, a multi-level intent parsing tree is constructed by expanding level by level. Each level in the tree represents a semantic granularity.

[0052] Optionally, S5 specifically includes:

[0053] S51: Traverse all nodes in the multi-level intent parsing tree constructed in S4, extract the corrected confidence score c″ corresponding to each node, and compare it with the preset confidence threshold c″. th Perform a comparison; if any node satisfies c″ <c th If the intention of the node is deemed uncertain, a clarification process will be initiated.

[0054] S52: Based on the intent type, modality dependency information, and contextual semantic structure corresponding to the uncertain node, call the predefined modality-intent association library, select the supplementary perception modality type that is highly associated with the corresponding intent, and determine the set of targeted sensors to be activated;

[0055] S53: Send a supplementary acquisition command to the selected sensor via the command bus, set the sampling window to a continuous data segment within Δt milliseconds after the current time, and acquire enhanced sensing data as the intention completion input;

[0056] S54: Based on the intent semantic labels of uncertain nodes in the parse tree and the data features of the current supplementary modality, call the corresponding entries in the dialogue template library to generate a multimodal verification question with a clear objective;

[0057] S55: Construct a structured intent clarification instruction set, which includes verification question text, trigger modality type, supplementary acquisition time window, and a set of semantic intent candidates to be matched.

[0058] Optionally, S6 specifically includes:

[0059] S61: Extract the explicit intent node with the highest confidence score and its associated implicit demand node from the final confirmed intent parsing tree obtained from S5 to form the main response intent structure. The main response intent structure includes semantic labels, execution instruction types and target object parameters.

[0060] S62: Based on the type of execution instruction in the main response intent structure, map the intent semantic label to system control instructions, including voice feedback instructions, visual interface instructions, and environmental control instructions;

[0061] S63: The three types of sub-instructions generated by S62 are uniformly scheduled according to the timestamp synchronization mechanism.

[0062] The beneficial effects of this invention are:

[0063] This invention, by introducing multimodal data acquisition, time synchronization deviation compensation, physical spatiotemporal constraint mapping, and graph structure embedding techniques, achieves dynamic fusion of information from different modalities under a unified time reference and spatial reference. It constructs a fusion feature vector with strong spatiotemporal consistency and complete semantic expression, significantly improving the ability of the dialogue agent to parse explicit action commands and implicit behavioral cues. By constructing a dual-branch intent recognition network, explicit intent and implicit needs are extracted separately, and the confidence score is dynamically corrected by combining historical behavior profiles and contextual states. This effectively constructs a multi-level intent parsing tree, realizing semantic hierarchical and structured expression of intent understanding.

[0064] This invention proposes an adaptive modal compensation mechanism based on confidence threshold triggering. When the confidence of any node in the parse tree is insufficient, it can accurately call the most relevant perceptual modality for supplementary collection and generate a structured intent clarification instruction set to guide user interaction feedback and enhance the system's intent correction capability in ambiguous scenarios. Attached Figure Description

[0065] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only for this invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0066] Figure 1 This is a schematic diagram of the dialogue agent interaction method according to an embodiment of the present invention;

[0067] Figure 2 This is a schematic diagram illustrating the process of constructing a spatiotemporal fusion feature vector according to an embodiment of the present invention. Detailed Implementation

[0068] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. It should also be noted that, to make the embodiments more comprehensive, the following embodiments are the best and preferred embodiments, and those skilled in the art can use other alternative methods to implement some well-known technologies; moreover, the accompanying drawings are only for more specific description of the embodiments and are not intended to specifically limit the present invention.

[0069] It should be noted that the use of terms such as "an embodiment," "an embodiment," "an exemplary embodiment," and "some embodiments" in the specification indicates that the described embodiment may include a specific feature, structure, or characteristic, but not every embodiment necessarily includes that specific feature, structure, or characteristic. Furthermore, when a specific feature, structure, or characteristic is described in connection with an embodiment, implementing such a feature, structure, or characteristic in conjunction with other embodiments (whether explicitly described or not) should be within the knowledge of those skilled in the art.

[0070] In general, terms can be understood, at least in part, from their use in context. For example, depending at least in part on the context, the term "one or more" as used herein can be used to describe any feature, structure, or characteristic in the singular sense, or can be used to describe a combination of features, structures, or characteristics in the plural sense. Additionally, the term "based on" can be understood as not necessarily intended to convey an exclusive set of factors, but can instead, depending at least in part on the context, allow for the presence of other factors that are not necessarily explicitly described.

[0071] like Figure 1-Figure 2 As shown, a dialogue agent interaction method based on multimodal intent understanding includes the following steps:

[0072] S1: Real-time acquisition of text streams, voice waveforms, video frame sequences, and sensor timing data during user interaction, and calculation of time synchronization deviation values ​​for each modal data source;

[0073] S2: Dynamically compensate and align multimodal data based on time synchronization deviation value, establish cross-modal correlation mapping through physical spatiotemporal constraints, and generate spatiotemporal fusion feature vectors;

[0074] S3: Analyze the explicit action instructions and implicit behavioral cues in the spatiotemporal fusion feature vector, and output a triple containing explicit intention, implicit need and corresponding confidence score;

[0075] S4: Combine the user's historical behavior profile with the current dialogue state to dynamically adjust the confidence score in the triple and generate a multi-level intent parsing tree;

[0076] S5: When the corrected confidence of any node in the intent parsing tree is lower than the set threshold, activate the targeted sensor to supplement the collected data, generate a multimodal verification question, and output the intent clarification instruction set;

[0077] S6: Drive response decisions based on the final confirmed intent parsing tree, and simultaneously execute voice feedback content, visual interface operations, and environmental control signals.

[0078] S1 specifically includes:

[0079] S11: Receive real-time text stream data input by the user through the text input interface. The text stream data is transmitted in character encoding form through the natural language input device, and the original timestamp of the text stream is recorded according to the character reception timestamp.

[0080] S12: Acquire user speech waveform data through a microphone array, and preprocess the acquired speech, including signal noise reduction, speech activity detection and energy normalization, extract the start and end time information of each speech segment, and generate the corresponding timestamp marker sequence.

[0081] S13: Capture video frame sequences during user interaction using a camera, set a constant frame rate for sampling, and add a system timestamp to each video frame to ensure that the video frame time recording has millisecond-level resolution.

[0082] S14: Collect time-series sensing data of user actions or physiological states through wearable sensing devices. The sensing channel includes at least one accelerometer and one skin conductance sensor. The sampling frequency is a fixed value, and a corresponding timestamp is generated for each sampling by a built-in real-time clock.

[0083] S15: Based on text stream timestamps, speech segment timestamps, video frame timestamps, and sensor sampling timestamps, a unified reference clock is used for calibration to map the timestamps of various modal data to the same time base. Based on this, the synchronization deviation value between any two modal data is calculated. The synchronization deviation value is defined as the time difference between the recording times of each modal data within the same interactive event window. The above steps clarify the time recording mechanism for collecting various modal data and use a unified reference clock to perform synchronization calibration on the collected data, so that each modal data has a unified time base. This provides high-precision time support for subsequent multimodal alignment and feature fusion, and improves the accuracy of cross-modal intent understanding.

[0084] S2 specifically includes:

[0085] S21: Based on the synchronization deviation value calculated by S1, the time axis of the speech waveform data is linearly interpolated to align the speech waveform with the character input time sequence of the text stream.

[0086] S22: By interpolating or dropping frames, the time axis of the video frame sequence is adjusted according to the synchronization deviation value to ensure that the shooting time of the video frame matches the aligned speech waveform data within millisecond precision.

[0087] S23: Based on the synchronization deviation value, the sliding window method is used to truncate or expand the sensor time series data to ensure that the sampling window of the sensor data corresponds one-to-one with the time domain boundary of the speech waveform and video frame data window.

[0088] S24: Based on the spatial distance between the user and the Agent and the physical space constraints of the interactive interface layout, determine the spatial relationship between text streams, voice waveforms, video frames and sensor data, and establish a spatial association mapping between cross-modal data with spatial location as nodes and data type as attributes;

[0089] S25: Based on the fused time-aligned modal data and cross-modal spatial association mapping, they are combined to construct a spatiotemporal fusion feature vector. The above steps achieve a high degree of consistency between different modal data in the temporal and spatial domains by dynamically adjusting and accurately aligning the time axis of each modal data and comprehensively considering the spatial layout constraints of the data source. This provides a complete and high-precision data foundation for accurately analyzing user intent and behavior, and improves the accuracy of subsequent intent analysis.

[0090] S24 specifically includes:

[0091] S241: Obtain the spatial coordinates of each modal acquisition device in the user interaction environment. Let the spatial positions of the text input device, voice acquisition microphone, camera, and sensor be P, respectively. T =(x T ,y T , z T ),P S =(x S ,y S , z S ),P V =(x V ,y V , z V ) and P D =(x D ,y D , z D ), where x, y, z are three-dimensional spatial coordinates;

[0092] S242: Calculate the spatial distance D between any two modal devices based on their spatial coordinates. i,j Spatial distance is defined as: Where i,j∈{T,S,V,D}, and i≠j;

[0093] S243: Spatial distance D i,j For edge weights, construct an undirected graph G = (N, E), where the node set N represents the data sources of each modality, and the edge set E represents the spatial association between modalities, where the association strength is inversely proportional to the edge weight, i.e.:

[0094] Where w i,j ∈ represents the spatial correlation weight between mode i and mode j, where ∈ is a non-zero small constant to prevent division by zero errors;

[0095] S244: The graph structure G is normalized, and a graph embedding algorithm is used to convert the spatial structure representation into a fixed-dimensional embedding vector as a spatial association feature to participate in the subsequent spatiotemporal fusion feature construction. The above steps establish a spatial graph structure between modalities by calculating the actual three-dimensional spatial position of the device, and introduce a graph weight mechanism to represent the modal association strength, so that the spatial relationship between different modalities is quantifiable and embeddable, providing structured spatial dependency information for subsequent spatiotemporal feature vector fusion, which helps to improve the accuracy and robustness of multimodal semantic modeling.

[0096] S25 specifically includes:

[0097] S251: Extract features from the time-aligned text stream, speech waveform, video frame sequence, and sensor data to obtain corresponding modal feature vectors, denoted as text modal feature vector F. T Speech modal feature vector F S Video modal feature vector F V and sensor modal eigenvectors F D ;

[0098] S252: For the spatial embedding vector E of each modal node in the graph structure of S24 T E S E V E D The modal feature vector and the spatial embedding vector are combined using a vector concatenation method to generate the spatiotemporal modal feature vector H. i Defined as: H i =[F i E i ], where i∈{T,S,V,D}, symbol; denotes vector concatenation operation;

[0099] S253: Spatial correlation weights w constructed based on S24 i,j The spatiotemporal feature vectors between modes are weighted and fused to form a unified spatiotemporal fusion feature vector F. ST Specifically defined as:

[0100] F ST =∑ i∈{T，S，V，D} ∑ j∈{T，S，V，D},j≠i w i,j ·(H i ⊙H j), where ⊙ represents element-wise multiplication between eigenvectors;

[0101] S254: Spatiotemporal fusion feature vector F ST Normalization is performed to obtain the final spatiotemporal fusion feature vector used for intent parsing. The above steps, by clearly defining the fusion process of temporal alignment features and spatial association embedding, and using cross-modal spatial association weights for weighted fusion, effectively capture the rich spatial associations and collaborative relationships between cross-modal data. This makes the final spatiotemporal fusion feature vector more comprehensively and accurately reflect user interaction intent and behavioral information, and enhances the accuracy and stability of the subsequent intent parsing process.

[0102] S3 specifically includes:

[0103] S31: The spatiotemporal fusion feature vector F obtained in S25 ST The input is fed into a multi-layer dual-branch decoding network structure. The decoding network includes an explicit intent recognition branch and an implicit demand recognition branch. The two branches share the underlying feature extraction parameters and output corresponding vector representations respectively.

[0104] S32: In the explicit intent recognition branch, a multi-class neural network is used to classify F. ST Perform category discrimination and output the intent type vector I of the explicit action command. exp and its corresponding confidence score vector C exp Each component corresponds to a dominant operation category and its probability of occurrence;

[0105] The above-mentioned S32 specifically includes:

[0106] S321: The spatiotemporal fusion feature vector F obtained in S25 ST ∈R d The input is fed into a predefined multi-class neural network structure, which consists of a set of fully connected layers. The network ends with a Softmax activation function, which is used to map high-dimensional feature vectors into class probability distributions of explicit action commands.

[0107] S322: Define a predefined set of explicit action categories as {l1, l2, ..., l...} n The network output vector P = [p1, p2, ..., p...] n ], where each component p i Represents the eigenvector F ST It was determined to be in the i-th category l i The probability satisfies: Where z i Output the raw score for the corresponding category to the fully connected layer;

[0108] S323: Denote the index of the category with the highest probability in the above category probability distribution vector P as i. exp =argmax(p i ), will correspond to semantic tags The maximum probability value is defined as the confidence score of the explicit action instruction.

[0109] S324: Encapsulate the output as an explicit intent triple (i exp , l exp c exp ), which serves as the structured recognition output for explicit action instructions.

[0110] S33: In the implicit demand identification branch, a graph attention network (GAT) is used in conjunction with the user's historical interaction behavior graph to identify F. ST Perform context graph embedding and output the implicit requirement vector I. imp and its corresponding confidence score C imp Among them, implicit needs are mainly triggered by low-significance features, reflecting potential preferences or unspoken goals;

[0111] S34: Perform soft maximum normalization on the results of the dominant and recessive branches respectively to obtain the final dominant intention triplet (i exp , l exp c exp ) and implicit demand triples (i imp ,l imp ,c imp ), where i represents the identification number, l represents the semantic label, and c represents the normalized confidence score;

[0112] S35: Combine the two triples mentioned above to form a complete intent recognition output structure, which serves as the result of multimodal intent parsing and is used in subsequent intent correction and decision generation processes. The above steps use a structured dual-branch decoding mechanism to parse explicit actions and implicit behaviors respectively, while introducing context graph modeling and confidence assessment, which achieves fine differentiation and quantitative expression of users' immediate expressions and potential needs, significantly improving the granularity and accuracy of multimodal intent recognition.

[0113] S33 specifically includes:

[0114] S331: Constructing a user history interaction behavior graph G U =(N U E U ), where the set of nodes N U For user history interaction events, the edge set E U To represent the interaction relationships between events, each node has a semantic feature vector representation of the corresponding historical events;

[0115] S332: The spatiotemporal fusion feature vector F obtained in S25 ST This is used as a feature of the current interaction node and added to the historical interaction behavior graph G. U In this process, an enhanced contextual interaction graph G is formed. C =(N U ∪{n cur}, E C ), where n cur The current interaction node;

[0116] S333: With the enhanced spectrum G C As input, use a graph attention network (GAT) to compute the attention weights a between nodes. ij The formula is: Among them, a ij Let be the attention weights between node i and its neighboring node j, W be the learnable weight matrix, a be the weight vector of the attention mechanism, σ be the non-linear activation function, and h be the weights. i ,h j For node features, || denotes vector concatenation, N i Let i be the set of neighboring nodes of node i;

[0117] S334: Based on the calculated attention weight a ij Then, perform node feature aggregation and update to obtain the updated latent requirement embedding vector h. cur ′ is defined as: Here, node cur represents the currently interacting node;

[0118] S335: Embed the updated implicit requirement into vector h cur The input is fed into a single-layer fully connected network and activated by the Sigmoid function, outputting a latent demand vector I. imp and the corresponding confidence score vector C imp The specific calculation is as follows: I imp =σ(W I h cur ′+b I ); C imp =σ(W C h cur ′+b C ), where W I and W C The weight matrix for implicit demand and confidence level output; b I and b C This is the corresponding bias vector.

[0119] S4 specifically includes:

[0120] S41: Based on user historical behavior profile data, extract confidence records of past user interaction events and statistical information on user behavior preferences to form a historical confidence reference table;

[0121] Table 1 Historical Confidence Reference

[0122]

[0123] In Table 1 above, the historical event number uniquely identifies a user's complete past interaction and serves as an index key; the intent category label represents the explicit or implicit category label corresponding to the user's intent in that interaction, such as "play music" or "check the weather"; the context semantic type describes the linguistic structural features of the interaction context, such as whether it is a question, statement, command, or expression containing implicit emotions; the original modality combination represents the modality data combination involved in intent judgment during the interaction process, and the marker is used to identify the modality source; the average confidence score c hist This represents the average historical confidence score of the corresponding category when the system identifies the same type of intent multiple times, and is used as a reference indicator to correct the current prediction; the interaction frequency represents the cumulative number of times the intent category appears in the user's historical interactions, and can be used as an important reference factor for weighting or confidence correction credibility.

[0124] S42: Match the confidence scores in the explicit intention and implicit need triples obtained in S3 with the historical confidence reference table, and adjust the confidence scores according to the matching degree. The adjusted confidence score is defined as: c′=α1·c+(1-α1)·c hist Where c is the initial confidence score, c hist The average confidence score for the corresponding event in the historical confidence reference table is α1, which is the fusion factor and ranges from 0 to 1.

[0125] S43: Determine the stage and context of the user's interaction with the Agent based on the current dialogue state, set the dialogue state correction weight β, and combine it with the adjusted confidence score to correct the state, and obtain the second-corrected confidence score c″, the expression of which is: c″=c′·β, where β is a weight coefficient of different stages preset according to the current dialogue state.

[0126] S44: Using the semantic categories of explicit intent and implicit needs as the root node of the tree, and the confidence score after secondary correction as the node weight, a multi-level intent parsing tree is constructed by expanding level by level. Each level in the tree represents a semantic granularity, and the parent-child hierarchical relationship between nodes is determined according to the confidence score. The above steps combine user historical behavior information and the current dialogue context state to perform hierarchical dynamic correction of the initial confidence score, and generate a multi-level tree structure based on the corrected confidence score. This significantly improves the stability and accuracy of multimodal intent parsing results, thereby effectively supporting the efficiency and accuracy of subsequent decision-making.

[0127] S5 specifically includes:

[0128] S51: Traverse all nodes in the multi-level intent parsing tree constructed in S4, extract the corrected confidence score c″ corresponding to each node, and compare it with the preset confidence threshold c″. th Perform a comparison; if any node satisfies c″ <c th If the intention of the node is deemed uncertain, a clarification process will be initiated.

[0129] S52: Based on the intent type, modal dependency information and contextual semantic structure corresponding to the uncertain node, call the predefined modal-intent association library, select the supplementary perception modal type that is highly associated with the corresponding intent, and determine the set of targeted sensors to be activated. The sensors include, but are not limited to, microphones, thermal infrared cameras, and physiological monitoring devices.

[0130] Table 2 Modality-Intent Association Library

[0131]

[0132] In Table 2 above, the intent category label corresponds to the explicit or implicit semantic labels that may appear in the intent parsing tree, such as "turn on the lights" or "express anxiety," and is the core field of the index. The semantic type indicates the semantic category to which the intent belongs, including explicit instruction, interrogation, negation, and implicit emotion, which are used to match the semantic context in which the system is currently located. The priority order of key modalities indicates the priority of the modal features on which the intent depends under normal recognition conditions. From left to right, it indicates the priority order from high to low, which is used to determine the risk of current modality missing. The supplementary perception trigger condition is used to define the trigger condition when the system needs to call the entry in this table. It is usually when the confidence of the current intent node is lower than the set threshold and the key modality is missing or the recognition is abnormal. The recommended supplementary acquisition modality is used when the trigger condition is met. The system recommends non-repeating modal channels to clarify the current intent as a data source for dynamic perception completion.

[0133] This modality-intent association library is used in the sensor selection logic of S52. When the system finds that the confidence of the parsing tree node is lower than the set threshold, it will call this table and use "intent category + current semantic type + modality missing status" as the query conditions to quickly locate the recommended supplementary perception modality and initiate the data collection and clarification question generation for the corresponding modality.

[0134] S53: Send a supplementary acquisition command to the selected sensor via the command bus, set the sampling window to a continuous data segment within Δt milliseconds after the current time, and acquire enhanced sensing data as the intention completion input;

[0135] S54: Based on the semantic labels of the uncertain nodes in the parse tree and the data features of the current supplementary modality, call the corresponding entries in the dialogue template library to generate a multimodal verification question with a clear objective. This question is semantically bound to the core variables of the required supplementary modality, such as "whether the person looked at the location" or "whether a negative tone was uttered".

[0136] S55: Construct a structured intent clarification instruction set, which includes verification question text, trigger modality type, supplementary acquisition time window, and a set of semantic intent candidates to be matched. The above steps automatically activate the most relevant supplementary perception channel when the intent confidence is insufficient and generate targeted multimodal verification questions, realizing a real-time adaptive clarification mechanism in the intent recognition process. This improves the system's responsiveness to ambiguous expressions and potential misidentification, and effectively enhances the robustness and user trust in the human-computer interaction process.

[0137] S6 specifically includes:

[0138] S61: Extract the explicit intent node with the highest confidence score and its associated implicit demand node from the final confirmed intent parsing tree obtained from S5 to form the main response intent structure. The main response intent structure includes semantic labels, execution instruction types and target object parameters.

[0139] S62: Based on the execution instruction type in the main response intent structure, map the intent semantic tags to system control instructions, including voice feedback instructions, visual interface instructions, and environmental control instructions; specifically:

[0140] Voice feedback command: Generate semantically consistent voice response text, input it into the text-to-speech (TTS) module, synthesize the voice signal and play it through the output channel;

[0141] Visual interface instructions: parse the target parameters and operation types of the interface, generate interface behavior scripts, and control the interface components to perform operations such as highlighting, jumping, and pop-up prompts;

[0142] Environmental control commands: Based on the intent target object parameters, the corresponding device control protocol is called to generate control codes, driving the smart device to perform external environmental adjustment behaviors such as lighting, temperature control, and sound.

[0143] S63: The three types of sub-instructions generated in S62 are uniformly scheduled according to the timestamp synchronization mechanism to ensure that voice output, visual interface changes and environmental response behaviors are completed in a coordinated manner within a preset time delay, thus ensuring the consistency and real-time performance of human-computer interaction responses. The above steps clearly distinguish between the three types of sub-response instructions (voice, vision and environment) and use a unified scheduling strategy to complete the synchronous execution across channels, so that multimodal response actions have temporal consistency and scene adaptability, thereby improving the system's response efficiency and perception consistency to complex human-computer interaction instructions.

[0144] This invention encompasses any substitutions, modifications, equivalent methods, and solutions made within the spirit and scope of this invention. To provide the public with a thorough understanding of this invention, specific details are described in detail in the following preferred embodiments; however, those skilled in the art will fully understand the invention even without these details. Furthermore, to avoid unnecessary misunderstanding of the essence of this invention, well-known methods, processes, procedures, components, and circuits are not described in detail.

[0145] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as within the scope of protection of the present invention.

Claims

1. A dialogue agent interaction method based on multimodal intent understanding, characterized in that, Includes the following steps: S1: Real-time acquisition of text streams, voice waveforms, video frame sequences, and sensor timing data during user interaction, and calculation of time synchronization deviation values ​​for each modal data source; S2: Dynamically compensate and align multimodal data based on time synchronization deviation value, establish cross-modal correlation mapping through physical spatiotemporal constraints, and generate spatiotemporal fusion feature vectors; S3: Analyze the explicit action instructions and implicit behavioral cues in the spatiotemporal fusion feature vector, and output a triple containing explicit intention, implicit need and corresponding confidence score; S4: Combine the user's historical behavior profile with the current dialogue state to dynamically adjust the confidence score in the triple and generate a multi-level intent parsing tree; S5: When the corrected confidence of any node in the intent parsing tree is lower than the set threshold, activate the targeted sensor to supplement the collected data, generate a multimodal verification question, and output the intent clarification instruction set; S6: Drive response decisions based on the final confirmed intent parsing tree, and simultaneously execute voice feedback content, visual interface operations, and environmental control signals.

2. The dialogue agent interaction method based on multimodal intent understanding according to claim 1, characterized in that, S1 specifically includes: S11: Receive real-time text stream data input by the user through the text input interface; S12: Acquire user voice waveform data via microphone array; S13: Capture video frame sequences during user interaction using a camera; S14: Collect time-series sensing data of user actions or physiological states through wearable sensing devices; S15: Based on text stream timestamps, speech segment timestamps, video frame timestamps, and sensor sampling timestamps, a reference clock is used for unified calibration to map the timestamps of various modal data to the same time base, and the synchronization deviation value between any two modal data is calculated accordingly.

3. The dialogue agent interaction method based on multimodal intent understanding according to claim 1, characterized in that, S2 specifically includes: S21: Based on the synchronization deviation value calculated by S1, the time axis of the speech waveform data is linearly interpolated to align the speech waveform with the character input time sequence of the text stream. S22: By interpolating or dropping frames, the time axis of the video frame sequence is adjusted according to the synchronization deviation value to ensure that the shooting time of the video frame matches the aligned speech waveform data within millisecond precision. S23: Based on the synchronization deviation value, the sliding window method is used to truncate or expand the sensor time series data to ensure that the sampling window of the sensor data corresponds one-to-one with the time domain boundary of the speech waveform and video frame data window. S24: Based on the spatial distance between the user and the agent and the physical space constraints of the interactive interface layout, determine the spatial relationship between text streams, voice waveforms, video frames and sensor data, and establish a spatial association mapping between cross-modal data with spatial location as nodes and data type as attributes; S25: Based on the fused time-aligned modal data and cross-modal spatial correlation mapping, they are combined to construct a spatiotemporal fusion feature vector.

4. The dialogue agent interaction method based on multimodal intent understanding according to claim 3, characterized in that, S24 specifically includes: S241: Obtain the spatial coordinates of each modal acquisition device in the user interaction environment. Let the spatial positions of the text input device, voice acquisition microphone, camera, and sensor be P, respectively. T =(x T y T , z T ),P S =(x S y S , z S ),P V =(x V y V , z V ) and P D =(x D y D , z D ), where x, y, z are three-dimensional spatial coordinates; S242: Calculate the spatial distance D between any two modal devices based on their spatial coordinates. i,j ; S243: Spatial distance D i,j For edge weights, construct an undirected graph G = (N, E), where the node set N represents the data sources of each modality, and the edge set E represents the spatial association between modalities, where the association strength is inversely proportional to the edge weight, i.e.: Where w i,j The spatial correlation weights between mode i and mode j are given by ∈, where ∈ is a non-zero small constant. S244: Normalize the graph structure G and use a graph embedding algorithm to convert the spatial structure representation into a fixed-dimensional embedding vector as a spatial association feature.

5. The dialogue agent interaction method based on multimodal intent understanding according to claim 4, characterized in that, Specifically, S25 includes: S251: Extract features from the time-aligned text stream, speech waveform, video frame sequence, and sensor data to obtain the corresponding modal feature vectors; S252: For the spatial embedding vectors of each modal node in the S24 graph structure, the modal feature vectors and spatial embedding vectors are combined by vector concatenation to generate spatiotemporal modal feature vectors. S253: Spatial correlation weights w constructed based on S24 i,j The spatiotemporal feature vectors between modes are weighted and fused to form a unified spatiotemporal fusion feature vector F. ST ; S254: Spatiotemporal fusion feature vector F ST Normalization is performed to obtain the final spatiotemporal fusion feature vector used for intent parsing.

6. The dialogue agent interaction method based on multimodal intent understanding according to claim 1, characterized in that, S3 specifically includes: S31: The spatiotemporal fusion feature vector F obtained in S25 ST The input is fed into a multi-layer dual-branch decoding network structure, which includes an explicit intent recognition branch and an implicit demand recognition branch. S32: In the explicit intent recognition branch, a multi-class neural network is used to classify F. ST Perform category discrimination and output the intent type vector I of the explicit action command. exp and its corresponding confidence score vector C exp Each component corresponds to a dominant operation category and its probability of occurrence; S33: In the implicit demand identification branch, a graph attention network combined with the user's historical interaction behavior graph is used to analyze F. ST Perform context graph embedding and output the implicit requirement vector I. imp and its corresponding confidence score C imp ; S34: Perform soft maximum normalization on the results of the dominant and recessive branches respectively to obtain the final dominant intention triplet (i exp , l exp c exp ) and implicit demand triples (i imp ,l imp ,c imp ), where i represents the identification number, l represents the semantic label, and c represents the normalized confidence score; S35: Combine the two triples mentioned above to form a complete intent recognition output structure, which serves as the result of multimodal intent parsing.

7. The dialogue agent interaction method based on multimodal intent understanding according to claim 6, characterized in that, Specifically, S33 includes: S331: Constructing a user history interaction behavior graph G U =(N U E U ), where the set of nodes N U For user historical interaction events, the edge set E U This refers to the interaction relationships between events; S332: The spatiotemporal fusion feature vector F obtained in S25 ST This is used as a feature of the current interaction node and added to the historical interaction behavior graph G. U In this process, an enhanced contextual interaction graph G is formed. C =(N U ∪{n cur }, E C ), where n cur This is the current interaction node; S333: With the enhanced map G C As input, use a graph attention network to compute the attention weights 'a' between nodes. ij ; S334: Based on the calculated attention weight a ij Then, perform node feature aggregation and update to obtain the updated latent requirement embedding vector h. cur ′; S335: Embed the updated implicit requirement into vector h cur The input is fed into a single-layer fully connected network and activated by the Sigmoid function, outputting a latent demand vector I. imp and the corresponding confidence score vector C imp .

8. The dialogue agent interaction method based on multimodal intent understanding according to claim 1, characterized in that, S4 specifically includes: S41: Based on user historical behavior profile data, extract confidence records of past user interaction events and statistical information on user behavior preferences to form a historical confidence reference table; S42: Match the confidence scores in the explicit intention and implicit need triples obtained in S3 with the historical confidence reference table, and adjust the confidence scores according to the matching degree. The adjusted confidence score is defined as: c′=α1·c+(1-α1)·c hist Where c is the initial confidence score, c hist The average confidence score for the corresponding event in the historical confidence reference table, where α1 is the fusion factor; S43: Determine the stage and context of the user's interaction with the Agent based on the current dialogue state, set the dialogue state correction weight β, and combine it with the adjusted confidence score to perform state correction, and obtain the second-corrected confidence score c″, the expression of which is: c″=c′·β; S44: Using the semantic categories of explicit intent and implicit needs as the root node of the tree, and the confidence score after secondary correction as the node weight, a multi-level intent parsing tree is constructed by expanding level by level. Each level in the tree represents a semantic granularity.

9. The dialogue agent interaction method based on multimodal intent understanding according to claim 8, characterized in that, S5 specifically includes: S51: Traverse all nodes in the multi-level intent parsing tree constructed in S4, extract the corrected confidence score c″ corresponding to each node, and compare it with the preset confidence threshold c″. th Perform a comparison; if any node satisfies c″ <c th If the intention of the node is deemed uncertain, a clarification process will be initiated. S52: Based on the intent type, modality dependency information, and contextual semantic structure corresponding to the uncertain node, call the predefined modality-intent association library, select the supplementary perception modality type that is highly associated with the corresponding intent, and determine the set of targeted sensors to be activated; S53: Send a supplementary acquisition command to the selected sensor via the command bus, set the sampling window to a continuous data segment within Δt milliseconds after the current time, and acquire enhanced sensing data as the intention completion input; S54: Based on the intent semantic labels of uncertain nodes in the parse tree and the data features of the current supplementary modality, call the corresponding entries in the dialogue template library to generate a multimodal verification question with a clear objective; S55: Construct a structured intent clarification instruction set, which includes verification question text, trigger modality type, supplementary acquisition time window, and a set of semantic intent candidates to be matched.

10. The dialogue agent interaction method based on multimodal intent understanding according to claim 1, characterized in that, S6 specifically includes: S61: Extract the explicit intent node with the highest confidence score and its associated implicit demand node from the final confirmed intent parsing tree obtained from S5 to form the main response intent structure. The main response intent structure includes semantic labels, execution instruction types and target object parameters. S62: Based on the type of execution instruction in the main response intent structure, map the intent semantic label to system control instructions, including voice feedback instructions, visual interface instructions, and environmental control instructions; S63: The three types of sub-instructions generated by S62 are uniformly scheduled according to the timestamp synchronization mechanism.

Images