Intelligent interaction intent understanding method and system based on multi-modal fusion

Abstract

The application provides a kind of intelligent interaction intention understanding method and system based on multi-modal fusion, it is related to the technical field of intelligent interaction, when presenting ambiguity intention resolution option set in user interaction interface, the line of sight free path sequence and touch pressure fluctuation sequence of user are synchronously collected.Path fractal dimension calculation is carried out to the line of sight sequence to obtain line of sight intention certainty index, and the touch sequence is subjected to fluctuation singular spectrum analysis to obtain touch intention stability index.The intention understanding model is called to process the index competition game of the two indexes, and the user real intention identifier is generated.The interactive response behavior chain is extracted from the intention response topological network according to the user real intention identifier, and the instruction is generated after compilation and sent to the target application system to trigger feedback operation.The application improves the accuracy of intelligent interaction and user experience.

Description

Technical Field

[0001] This application relates to the field of intelligent interaction technology, and more specifically, to a method and system for understanding intelligent interaction intent based on multimodal fusion. Background Technology

[0002] In the field of intelligent interaction, accurately understanding the user's interaction intent is key to achieving efficient and natural human-computer interaction. Most existing methods for understanding intelligent interaction intent rely on single-modal data, such as inferring user intent solely from voice input, text input, or simple touch operations. However, single-modal data often has limitations and cannot comprehensively and accurately reflect the user's true thoughts.

[0003] Taking voice interaction as an example, a user's voice may be interfered with by environmental noise, accents, and other factors, leading to deviations in semantic understanding. Simple touch operations, such as clicking and swiping, may have multiple meanings in different scenarios, making it difficult to accurately determine the user's intent. Furthermore, when user input is ambiguous, existing methods typically only require the user to re-enter or select a preset option, lacking in-depth exploration and analysis of the user's potential intent. This fails to effectively resolve ambiguity, thus affecting the smoothness of the interaction and the user experience. Summary of the Invention

[0004] In view of this, the purpose of this application is to provide a method and system for intelligent interactive intent understanding based on multimodal fusion.

[0005] According to a first aspect of this application, a method for intelligent interactive intent understanding based on multimodal fusion is provided, the method comprising: The user interface presents a set of ambiguous intent resolution options corresponding to the current interaction round, and simultaneously collects the user's gaze wandering path sequence and touch pressure fluctuation sequence for each ambiguous intent resolution option in the set of ambiguous intent resolution options; The path fractal dimension calculation is performed on the gaze-free path sequence to obtain the gaze intent determinism index, and the wave singular spectrum analysis is performed on the touch pressure fluctuation sequence to obtain the touch intent stability index. The pre-built intent understanding model is invoked to perform an index competition game on the gaze intent certainty index and the touch intent stability index to generate the user's true intent identifier corresponding to the current interaction round. Based on the user's true intent identifier, the corresponding interactive response behavior chain is dynamically extracted from the preset intent response topology network, and the interactive response behavior chain is compiled to generate an interactive response instruction and sent to the target application system to trigger the corresponding interactive feedback operation.

[0006] According to a second aspect of this application, a multimodal fusion-based intelligent interactive intent understanding system is provided. The multimodal fusion-based intelligent interactive intent understanding system includes a machine-readable storage medium and a processor. The machine-readable storage medium stores machine-executable instructions. When the processor executes the machine-executable instructions, the multimodal fusion-based intelligent interactive intent understanding system implements the aforementioned multimodal fusion-based intelligent interactive intent understanding method.

[0007] Based on any of the above aspects, the technical effect of this application is as follows: By presenting a set of ambiguous intent resolution options in the user interface and simultaneously collecting the user's gaze-wandering path sequence and touch pressure fluctuation sequence, this method fully utilizes multimodal information from the user's vision and touch. A gaze-wandering intent determinism index is obtained by calculating the path fractal dimension of the gaze-wandering path sequence, and a touch intent stability index is obtained by performing fluctuation singular spectrum analysis on the touch pressure fluctuation sequence. This allows for the quantification of user intent characteristics from different perspectives. The intent understanding model is then used to perform an index competition game between the two indices, simulating the complex thinking involved in human decision-making, resulting in more accurate and reliable user intent identifiers. Based on the user's true intent identifiers, interactive response behavior chains are dynamically extracted and interactive response commands are generated, achieving personalized interactive feedback and significantly improving the accuracy, fluency, and user experience of intelligent interaction. Attached Figure Description

[0008] Figure 1 A flowchart illustrating the intelligent interactive intent understanding method based on multimodal fusion provided in an embodiment of this application is shown. Figure 2 The diagram shows a component structure of an intelligent interactive intent understanding system based on multimodal fusion provided in an embodiment of this application. Detailed Implementation

[0009] Figure 1 This paper illustrates a flowchart of an intelligent interactive intent understanding method and system based on multimodal fusion provided in an embodiment of this application. The detailed steps include: Step S110: Present the set of ambiguous intent resolution options corresponding to the current interaction round on the user interaction interface, and simultaneously collect the user's gaze wandering path sequence and touch pressure fluctuation sequence for each ambiguous intent resolution option in the set of ambiguous intent resolution options.

[0010] On the interactive interface of the smart home control screen, a set of ambiguous intent resolution options O={o1, o2, o3} corresponding to the current interaction round is presented in the form of floating cards. In a specific example, the user's voice command "adjust the lights" is ambiguous due to the existence of multiple adjustable lighting devices; therefore, o1 corresponds to "living room ceiling light," o2 corresponds to "dining room chandelier," and o3 corresponds to "hallway wall light." A description of the current position status of the light is displayed below each option. Eye-tracking sensors and touch pressure sensors are activated simultaneously. The eye-tracking sensor collects the coordinates of the user's gaze focus at a sampling rate of 60 Hz, generating a gaze-free path sequence G={g1, g2, ..., gN}, where each element gi is a quintuple gi=(xi, yi, ti, idi, fi). xi and yi are the x and y coordinates of the gaze focus in the screen coordinate system, with the top left corner of the screen as the origin, in pixels; ti is the timestamp of the sampling time, in milliseconds; idi is the index of the ambiguous intent resolution option to which the current gaze point belongs, ranging from 1 to 3, with idi being 0 if the gaze point is not in any option area; fi is the gaze-wandering status flag, with a value of 0 or 1, where 1 indicates that the point is the starting or ending point of gaze-wandering. The touch pressure sensor collects the pressure value of the user operating on the screen at a sampling rate of 100 Hz, generating a touch pressure fluctuation sequence P={p1, p2, ..., pM}, where each element pj is a quadruple pj=(fj, tj, xj, yj). fj is the pressure value, in Newtons; tj is the timestamp of the sampling time, in milliseconds; xj and yj are the screen coordinates of the touch point.

[0011] The raw data collected is immediately encrypted when stored locally, using the Advanced Encryption Standard (AES-256) algorithm. The encryption key is generated by the device's unique identifier and the temporary interactive password set by the user through the password-based key derivation function PBKDF2. The key length is 256 bits, and the number of iterations is 10,000 to prevent data leakage when the storage medium is illegally accessed.

[0012] Step S120: Perform path fractal dimension calculation on the gaze-free path sequence to obtain the gaze intent determinism index, and simultaneously perform wave singular spectrum analysis on the touch pressure fluctuation sequence to obtain the touch intent stability index.

[0013] The gaze-free path sequence G and the touch pressure fluctuation sequence P are quantized independently to obtain the gaze intent determinism index vector C_gaze=[C1, C2, C3] and the touch intent stability index T_touch.

[0014] Step S121: Perform gaze start point calibration processing on the gaze wandering path sequence, identify the projection start point coordinates of the user's gaze first projected onto the screen area corresponding to each ambiguous intent resolution option from the gaze wandering path sequence, and record the gaze projection start time corresponding to each ambiguous intent resolution option.

[0015] Traverse the gaze-wandering path sequence G. For each ambiguous intention resolution option ok, where k takes the values ​​1, 2, or 3, find the first gaze point gi that satisfies idi = k and fi = 1, starting from the beginning of the sequence. Record the coordinates (xi, yi) of this point as the gaze projection starting point coordinates S_k = (x_start_k, y_start_k) for option ok. Record the timestamp ti of this point as the gaze projection starting time t_start_k for option ok. For example, for option o1, if the first point that satisfies the conditions is g5, then S1 = (x5, y5), and t_start1 = t5.

[0016] Step S122: Perform gaze-free trajectory tracking processing on the gaze-free path sequence, and extract the gaze-free trajectory point coordinate sequence formed by the continuous gaze-free movement of the user's gaze on the screen after the projection starting point coordinates are extracted from the gaze-free path sequence.

[0017] For each option "ok", starting from the gaze projection origin S_k, all subsequent gaze points are continuously extracted in ascending order of timestamps until the moment the user makes a touch selection t_select, forming a continuous gaze-free trajectory point coordinate sequence T_k. Each element in T_k is a two-dimensional coordinate (x, y). The sequence T_k records the detailed path of the user's gaze moving across various areas of the screen from the first glance at option "ok" to the final decision. The length of T_k is L_k, which is the total number of gaze points belonging to this trajectory.

[0018] Step S123: Perform grid coverage counting processing on the coordinate sequence of the line-of-sight free trajectory points, map the coordinate sequence of the line-of-sight free trajectory points to a grid coordinate system of a preset scale, and count the number of grid cells covered by the coordinate sequence of the line-of-sight free trajectory points as the first coverage counting parameter.

[0019] Construct a grid coordinate system covering the entire screen area, setting the initial grid cell side length to r0 in pixels. Map each point (x, y) in the gaze-wander trajectory point coordinate sequence T_k to the grid coordinate system, determining the grid cell to which each point belongs. The row index of the cell is floor(y / r0), and the column index is floor(x / r0). Count the total number of grid cells covered by at least one trajectory point, denoted as N_k(r0). This is the first coverage count parameter.

[0020] Step S124: Perform grid scaling processing on the coordinate sequence of the line-of-sight wandering trajectory points, reduce the size of the grid cells of the grid coordinate system according to a preset scaling ratio, generate a scaled grid coordinate system, and recount the number of grid cells covered by the line-of-sight wandering trajectory point coordinate sequence in the scaled grid coordinate system as the second coverage count parameter.

[0021] The grid cell side length is reduced to half of its original size, i.e., the scaled grid cell side length r1 = r0 / 2. The grid coordinate system is reconstructed using the new grid side length r1. The coordinate sequence T_k of the line-of-sight trajectory points is mapped to the scaled grid coordinate system again, and the total number of grid cells covered by at least one trajectory point is counted, denoted as N_k(r1). This is the second coverage counting parameter.

[0022] Step S125: Calculate the fractal dimension approximation of the line-of-sight free trajectory point coordinate sequence based on the first coverage count parameter, the second coverage count parameter, and the preset scaling ratio, and use the fractal dimension approximation as the line-of-sight free trajectory fractal dimension parameter corresponding to the line-of-sight free path sequence.

[0023] Based on the box counting principle of fractal dimension, the approximate fractal dimension D_k of the coordinate sequence T_k of the line-of-sight trajectory is calculated. The calculation formula is D_k=log(N_k(r1) / N_k(r0)) / log(r0 / r1). Since r0 / r1=2, D_k=log(N_k(r1) / N_k(r0)) / log(2). Calculate the corresponding line-of-sight trajectory fractal dimension parameters D1, D2, and D3 for each option ok. The larger the value of D_k, the more complex and tortuous the line-of-sight trajectory, and the stronger its space-filling ability.

[0024] Step S126: Perform negative mapping processing on the fractal dimension parameter of the gaze-free trajectory, substitute the fractal dimension parameter of the gaze-free trajectory into the negative correlation mapping function to calculate the gaze intent certainty index, and the gaze intent certainty index is negatively correlated with the fractal dimension parameter of the gaze-free trajectory.

[0025] A negative correlation mapping function f(D) = 1 / (D + ε) is constructed, where ε is a preset minimum positive number, taking the value 0.001, to prevent the denominator from being zero. For each option "ok", its gaze-wandering trajectory fractal dimension parameter D_k is substituted into the above function to obtain the gaze intent certainty index C_k = 1 / (D_k + 0.001). The larger the value of C_k, the simpler and more direct the user's gaze-wandering trajectory for that option, and the higher the intent certainty. The gaze intent certainty indices of all options form a vector C_gaze = [C1, C2, C3].

[0026] Step S127: Perform pressure start point detection processing on the touch pressure fluctuation sequence, extract the touch pressure start value generated when the user's finger first touches the screen from the touch pressure fluctuation sequence, and record the pressure start time corresponding to the touch pressure start value.

[0027] Traverse the touch pressure fluctuation sequence P, starting from the first point p1, and check whether the pressure value fj at each point exceeds the preset contact pressure threshold δ_contact, where δ_contact is 0.15 Newtons. When the first point pj that satisfies fj greater than 0.15 is found, record the pressure value at that point as the touch pressure start value F_start=fj, and record the timestamp of that point as the pressure start time T_start=tj. This point marks the beginning of the user's finger contacting the screen.

[0028] Step S128: Perform pressure fluctuation interval division processing on the touch pressure fluctuation sequence, and divide the touch pressure fluctuation sequence into pressure rising fluctuation subsequence, pressure maintaining fluctuation subsequence and pressure falling fluctuation subsequence according to the initial value of touch pressure.

[0029] Starting from the pressure initiation point T_start, the pressure fluctuation sequence P is traversed sequentially. First, the pressure rise phase is identified: the moment T_peak occurs when the pressure value reaches its first local maximum and the pressure values ​​of the next three consecutive points begin to decrease. All pressure points within the interval [T_start, T_peak] constitute the pressure rise fluctuation subsequence P_up. Then, the pressure maintenance phase is identified: starting after T_peak, the moment T_hold_end occurs when the pressure value begins to continuously decrease and falls below the preset maintenance pressure threshold δ_hold. δ_hold is 0.8 multiplied by F_start. All pressure points within the interval [T_peak, T_hold_end] constitute the pressure maintenance fluctuation subsequence P_hold. Finally, the pressure fall phase is identified: starting after T_hold_end, until the pressure value drops to near zero or the user's finger leaves the screen at T_end, all pressure points within the interval [T_hold_end, T_end] constitute the pressure fall fluctuation subsequence P_down. P_up, P_hold, and P_down together completely cover the entire touch process from finger pressing to finger lifting.

[0030] Step S129: Perform upward trend fitting processing on the pressure rise fluctuation subsequence, calculate the difference value of adjacent touch pressure values ​​in the pressure rise fluctuation subsequence, and sum all the difference values ​​and divide by the total time span of the pressure rise fluctuation subsequence to generate the average pressure rise rate parameter.

[0031] For a pressure rise fluctuation subsequence P_up, assuming it contains L_up pressure points arranged in chronological order as p_up1, p_up2, ..., p_upL_up, each point p_upi contains a pressure value f_upi and a timestamp t_upi. The pressure difference between adjacent points is calculated sequentially as Δf_upi = f_up(i+1) - f_upi, where i ranges from 1 to L_up-1. All Δf_upi values ​​are summed to obtain the total pressure rise change ΔF_up = ΣΔf_upi. The total time span of the pressure rise is calculated as Δt_up = t_upL_up - t_up1. The average pressure rise rate parameter R_up = ΔF_up / Δt_up, in Newtons per second, is generated.

[0032] Step S1210: Perform maintenance stability calculation on the pressure maintenance fluctuation subsequence, calculate the standard deviation of all touch pressure values ​​in the pressure maintenance fluctuation subsequence, and use the reciprocal of the standard deviation as the pressure maintenance stability parameter.

[0033] For a pressure maintenance fluctuation subsequence P_hold, assuming it contains L_hold pressure points, with pressure value sequences f_hold1, f_hold2, ..., f_holdL_hold, first calculate the arithmetic mean of these pressure values: μ_hold = (1 / L_hold) * Σf_holdi. Then, calculate the square of the difference between each pressure value and the mean, and sum them to obtain the variance σ²_hold = (1 / (L_hold-1)) * Σ(f_holdi-μ_hold)². Take the square root to obtain the standard deviation σ_hold = sqrt(σ²_hold). The pressure maintenance stability parameter S_hold = 1 / (σ_hold + ε), where ε is 0.001. The larger S_hold is, the smaller the fluctuation during the pressure maintenance phase, and the smoother the operation.

[0034] Step S1211: Perform downward trend fitting processing on the pressure drop fluctuation subsequence, calculate the absolute value of the difference between adjacent touch pressure values ​​in the pressure drop fluctuation subsequence, and sum all the absolute values ​​of the difference and divide by the total time span of the pressure drop fluctuation subsequence to generate the average pressure drop rate parameter.

[0035] For a pressure drop fluctuation subsequence P_down, assuming it contains L_down pressure points arranged in chronological order as p_down1, p_down2, ..., p_downL_down, each point p_downi contains a pressure value f_downi and a timestamp t_downi. Calculate the absolute value of the pressure difference between adjacent points Δf_downi = |f_down(i+1) - f_downi|, where i ranges from 1 to L_down-1. Summate all Δf_downi values ​​to obtain the total pressure drop change ΔF_down = ΣΔf_downi. Calculate the total pressure drop time span Δt_down = t_downL_down - t_down1. Generate the average pressure drop rate parameter R_down = ΔF_down / Δt_down, in Newtons per second.

[0036] Step S1212: Construct a touch pressure fluctuation feature vector based on the average pressure rise rate parameter, the pressure stability parameter, and the average pressure fall rate parameter, and input the touch pressure fluctuation feature vector into a preset singular spectrum analysis model for singular value decomposition to generate at least one singular value component.

[0037] The average rate of pressure rise (R_up), the average rate of pressure maintenance (S_hold), and the average rate of pressure fall (R_down) are combined into a three-dimensional touch pressure fluctuation feature vector V_touch = [R_up, S_hold, R_down]. V_touch is then input into a pre-defined singular spectrum analysis model. This model first embeds the one-dimensional touch pressure fluctuation feature vector into a trajectory matrix. Choosing an embedding window length of L=2, V_touch is converted into a 2x2 trajectory matrix X. The first row of matrix X is [V_touch1, V_touch2], and the second row is [V_touch2, V_touch3]. Then, singular value decomposition is performed on matrix X, i.e., X = UΣV^T, where U is the left singular matrix, Σ is the diagonal matrix, and the diagonal elements σ1 and σ2 are the singular values. V is the right singular matrix. The singular values ​​σ1 and σ2 are the two singular value components, each representing the projection intensity of the original pressure fluctuation feature onto different modes.

[0038] Step S1213: Extract the principal singular component with the largest energy proportion from the at least one singular component, and use the amplitude of the principal singular component as the stability index of the touch intent.

[0039] Calculate the energy of each singular value component, i.e., the square of the singular value E1 = σ1², E2 = σ2². Calculate the total energy E_total = E1 + E2. Calculate the energy proportion of each component p1 = E1 / E_total, p2 = E2 / E_total. Compare p1 and p2, and select the singular value component with the larger energy proportion as the principal singular value component, denoted as σ_main. Use σ_main as the touch intent stability index T_touch. The larger T_touch is, the more concentrated the energy of the touch pressure fluctuation is on a primary pattern, and the more stable and clear the operation intent is.

[0040] Step S130: Call the pre-built intent understanding model to perform index competition game processing on the gaze intent certainty index and the touch intent stability index to generate the user's true intent identifier corresponding to the current interaction round.

[0041] A pre-trained intent understanding model is invoked. This model is designed based on game theory and is used to fuse the exponents of gaze modality and touch modality to determine the final true intent of the user through competitive game.

[0042] Step S131: Input the gaze intent certainty index into the gaze index encoder of the intent understanding model for index feature embedding processing to generate a gaze certainty feature vector. The gaze certainty feature vector includes a numerical code corresponding to the gaze intent certainty index and a path morphology code corresponding to the gaze free path sequence.

[0043] The gaze intent deterministic index vector C_gaze=[C1, C2, C3] is input into the gaze index encoder of the intent understanding model. The gaze index encoder consists of two sub-modules: a fully connected embedding layer and a long short-term memory network layer. First, C_gaze passes through the fully connected embedding layer, which has 3 input nodes and 64 output nodes. Through a linear transformation Y=W_gaze*C_gaze+b_gaze, the 3D vector is mapped to a 64-dimensional embedding space, generating an initial numerical encoding vector E_gaze_num, where W_gaze is a 64×3 weight matrix and b_gaze is a 64-dimensional bias vector. Simultaneously, the coordinate sequences T1, T2, and T3 of the three gaze-free trajectory points generated in step S122 are input into the long short-term memory network layer. This long short-term memory network layer contains 128 hidden units, which extract temporal features for each sequence. For sequence T1, the Long Short-Term Memory (LSTM) network sequentially inputs the coordinates (x, y) of each point at each time step. After passing through the input gate, forget gate, output gate, and cell state update, it finally outputs a 128-dimensional path morphology encoding vector P1. Similarly, P2 and P3 are obtained. P1, P2, and P3 are concatenated with E_gaze_num along the feature dimension to form a gaze deterministic feature vector F_gaze with a dimension of 64 + 128 × 3 = 448.

[0044] Step S132: Input the touch intent stability index into the touch index encoder of the intent understanding model for index feature embedding processing to generate a touch stability feature vector. The touch stability feature vector includes a numerical code corresponding to the touch intent stability index and a wave pattern code corresponding to the touch pressure wave sequence.

[0045] The touch intent stability index T_touch (scalar) is input into the touch index encoder of the intent understanding model. The touch index encoder consists of a fully connected embedding layer and a convolutional neural network layer. First, T_touch passes through the fully connected embedding layer, which has 1 input node and 32 output nodes. Through the linear transformation Z = W_touch * T_touch + b_touch, the scalar is mapped to a 32-dimensional embedding space, generating a preliminary numerical encoding vector E_touch_num, where W_touch is a 32×1 weight matrix and b_touch is a 32-dimensional bias vector. Simultaneously, the original touch pressure fluctuation sequence P collected in step S110 is input into the convolutional neural network layer. This convolutional neural network layer consists of two one-dimensional convolutional layers and a global max pooling layer. The first one-dimensional convolutional layer has 32 convolutional kernels, each with a size of 3 and a stride of 1. It performs convolution operations on the input pressure sequence P (of length M), outputting 32 feature maps of length M-2. The second one-dimensional convolutional layer has 64 kernels, each with a size of 3 and a stride of 1. It performs convolution operations on the aforementioned 32 feature maps, outputting 64 feature maps of length M-4. Then, a global max-pooling layer takes the maximum value along the length dimension of each feature map, compressing each feature map into a scalar, ultimately resulting in a 64-dimensional wave pattern encoding vector Q. Q is then concatenated with E_touch_num along the feature dimensions to form a touch stability feature vector F_touch with dimensions 64+32=96.

[0046] Step S133: Call the exponential competitive game module of the intent understanding model to perform zero-sum game calculation on the gaze deterministic feature vector and the touch stability feature vector to generate a gaze-touch game payoff matrix. The gaze-touch game payoff matrix is ​​used to represent the payoff distribution relationship between the gaze deterministic feature vector and the touch stability feature vector under different strategy combinations.

[0047] The gaze deterministic feature vector F_gaze (448-dimensional) and the touch stability feature vector F_touch (96-dimensional) are input into the exponential competitive game module. This module first constructs a two-player zero-sum game framework, treating the gaze modality as player A and the touch modality as player B. The policy space size is equal to the number of ambiguity resolution options, i.e., 3. This module includes a policy mapping layer that maps F_gaze to the gaze modality's policy space through a fully connected network with 448 input nodes and 3 output nodes, outputting a 3-dimensional vector π_gaze, representing the gaze modality's initial tendency to choose various policies. Similarly, F_touch is mapped to the touch modality's policy space through another fully connected network (96 inputs, 3 outputs), outputting a 3-dimensional vector π_touch. Then, a 3×3 gaze-touch game payoff matrix M_payoff is constructed. Each element m_ij in the matrix represents the payoff obtained by the gaze modality when the gaze modality chooses policy i and the touch modality chooses policy j. m_ij is calculated using a bilinear function: m_ij = F_gaze^T * W_ij * F_touch, where W_ij is a 448×96 learnable weight matrix obtained through model training. For i and j ranging from 1 to 3, a total of nine of these weight matrices are required.

[0048] Step S134: Perform a first round of strategy adjustment on the gaze deterministic feature vector according to the gaze touch game payoff matrix. The feature dimensions in the gaze deterministic feature vector that have strategy conflicts with the touch stability feature vector are corrected according to the preset first game coefficient to generate the gaze feature vector after the initial game.

[0049] Based on the payoff matrix M_payoff, the expected payoff vector R_gaze = M_payoff * π_touch for the gaze modality is calculated under the given touch modality policy π_touch. R_gaze is a 3-dimensional vector. The gradient of each feature dimension of F_gaze with respect to the expected payoff is calculated, resulting in the gradient vector ∇R_gaze, which has the same 448 dimensions as F_gaze. The magnitude of the gradient reflects the sensitivity of that feature dimension to the payoff. Feature dimensions that identify policy conflicts: when the gradient direction of a certain feature dimension is opposite to the projection direction of F_touch on that dimension, a policy conflict is considered to exist. According to the preset first game coefficient α1 (α1 is 0.1), the feature dimensions with conflicts are corrected: F_gaze_p1_i = F_gaze_i - α1 * sign(conflict) * F_gaze_i, where sign(conflict) is +1 or -1 to indicate the direction of conflict. The above processing is performed on all 448 dimensions to obtain the gaze feature vector F_gaze_p1 after the initial game.

[0050] Step S135: Perform a first round of strategy adjustment on the touch stability feature vector according to the gaze-touch game payoff matrix. Correct the feature dimensions in the touch stability feature vector that conflict with the gaze deterministic feature vector according to the preset second game coefficient to generate the touch feature vector after the initial game.

[0051] Similarly, based on the payoff matrix M_payoff, the expected payoff vector R_touch for the touch mode is calculated as R_touch = -M_payoff^T*π_gaze, given the gaze modality policy π_gaze. R_touch is a 3-dimensional vector. The gradient of each feature dimension of F_touch with respect to the expected payoff is calculated, resulting in the gradient vector ∇R_touch (96 dimensions). Feature dimensions in F_touch that conflict with F_gaze are identified. According to the preset second game coefficient α2 (α2 is 0.1), the conflicting feature dimensions are corrected: F_touch_p1_j = F_touch_j - α2*sign(conflict)*F_touch_j. The above processing is performed on all 96 dimensions to obtain the touch feature vector F_touch_p1 after the initial game.

[0052] Step S136: Input the post-preliminary game gaze feature vector and the post-preliminary game touch feature vector into the Nash equilibrium solution layer of the exponential competitive game module for equilibrium strategy solution processing, and generate a hybrid strategy Nash equilibrium vector, wherein the hybrid strategy Nash equilibrium vector contains the equilibrium strategy probability value corresponding to each of the ambiguous intention resolution options.

[0053] F_gaze_p1 and F_touch_p1 are input into the Nash equilibrium solution layer. This layer uses an iterative optimization algorithm to solve the mixed-policy Nash equilibrium of the two-person zero-sum game. First, the policy π_gaze1 for the gaze mode is recalculated based on F_gaze_p1, and the policy π_touch1 for the touch mode is recalculated based on F_touch_p1. Then, the mixed policy vectors p_gaze = π_gaze1 and p_touch = π_touch1 are initialized. Entering the iterative loop, in each iteration t, the optimal response policy p_gaze_br = argmax_{p}(p^T*M_payoff*p_touch) for the gaze mode under the current p_touch is calculated, and p_gaze = (1-β)*p_gaze + β*p_gaze_br is updated, where β is the learning rate and is set to 0.1. Then, calculate the optimal response policy p_touch_br = argmin_{p}(p_gaze^T*M_payoff*p) for the touch modality under the current p_gaze, and update p_touch = (1-β)*p_touch + β*p_touch_br. Repeat the iteration until ||p_gaze-p_gaze_prev|| < 10e-6 or the maximum number of iterations of 100 is reached. The finally converged p_gaze is the hybrid policy Nash equilibrium vector P_nash, which is a 3-dimensional probability vector whose elements p1, p2, and p3 correspond to the equilibrium policy probability values ​​of the three ambiguity intention resolution options, respectively.

[0054] Step S137: Perform maximum value indexing on the hybrid strategy Nash equilibrium vector, extract the index position corresponding to the element with the largest value from the hybrid strategy Nash equilibrium vector, and determine the corresponding ambiguous intent resolution option from the ambiguous intent resolution option set based on the index position as the user's true intent identifier.

[0055] Compare the three probability values ​​p1, p2, and p3 in P_nash, and find the index k* = argmax(p_k) corresponding to the maximum value. Select the option with index k* from the ambiguous intent resolution option set O = {o1, o2, o3} as the user's true intent identifier, denoted as intent_id = o_k*. For example, if P_nash = [0.2, 0.7, 0.1], then k* = 2, and intent_id = o2, which is "restaurant chandelier".

[0056] Step S140: Dynamically extract the corresponding interactive response behavior chain from the preset intent response topology network according to the user's real intent identifier, compile the interactive response behavior chain to generate an interactive response instruction and send it to the target application system to trigger the corresponding interactive feedback operation.

[0057] Based on the determined user's true intent identifier intent_id, the specific interactive response behavior chain is extracted from a pre-built intent-response topology network G_net and compiled into executable instructions to be sent to the target device.

[0058] Step S141: Parse the intent node code contained in the user's real intent identifier, and locate the starting intent node corresponding to the intent node code from the preset intent response topology network.

[0059] The intent response topology network G_net is a directed graph, denoted as G_net=(V, E), where V is the set of intent nodes, and each node v∈V contains a node code code(v) and a set of operation attributes; E is the set of directed edges, representing the flow relationships between intents. The user's true intent identifier intent_id is parsed. For example, intent_id "o2" means "dining light", and its corresponding intent node code is "dining_light_on". The node v_start in G_net with the node code equal to "dining_light_on" is located as the starting intent node.

[0060] Step S142: Starting from the starting intent node, perform a depth-first traversal according to the preset directed edge connection relationship in the intent response topology network to generate at least one candidate response path from the starting intent node to the ending intent node.

[0061] Starting from the initial intent node v_start, a depth-first traversal algorithm is used in the intent-response topology network G_net to explore all possible paths along the direction of directed edges E until one or more terminating intent nodes v_terminal are reached. Terminating intent nodes are those nodes with no outgoing edges or whose outgoing edges point to themselves, indicating that the operation is complete. Each complete path from v_start to v_terminal constitutes a candidate response behavior chain Chain_m = [v_start, v2, v3, ..., v_terminal]. Traversing all possible paths generates a set of candidate response behavior chains Chains = {Chain_1, Chain_2, ..., Chain_M}.

[0062] Step S143: Obtain the system resource occupancy status parameters corresponding to the current interaction round. The system resource occupancy status parameters include the CPU idle rate, available memory capacity, and remaining network bandwidth.

[0063] Before executing the intent response, the system interface of the operating system is invoked to obtain the current system resource usage status in real time. The CPU idle rate (cpu_idle) is obtained, a floating-point number between 0 and 1 representing the proportion of CPU time currently idle. The available memory capacity (mem_avail) is obtained, in megabytes, representing the size of currently available random access memory. The remaining network bandwidth (bw_avail) is obtained, in megabits per second, representing the remaining available bandwidth of the current network connection.

[0064] Step S144: Based on the CPU idle rate, available memory capacity, and remaining network bandwidth, perform resource consumption estimation processing on each candidate response path to calculate the total amount of system resources expected to be consumed by each candidate response path during execution.

[0065] For each candidate response behavior chain Chain_m, each intent node v has a preset resource consumption coefficient. For node v, its CPU consumption coefficient is c_cpu(v), memory consumption coefficient is c_mem(v) (in megabytes), and network consumption coefficient is c_bw(v) (in megabits). The total consumption is calculated by summing the resource consumption coefficients of all nodes on Chain_m: CPU_cost_m = Σc_cpu(v), MEM_cost_m = Σc_mem(v), BW_cost_m = Σc_bw(v). Then, based on the current system resource status, the estimated total system resource consumption Resource_total_m for each path is calculated. The calculation formula is: Resource_total_m = w_cpu * (CPU_cost_m / (cpu_idle + δ)) + w_mem * (MEM_cost_m / (mem_avail + δ)) + w_bw * (BW_cost_m / (bw_avail + δ)). Where w_cpu, w_mem, and w_bw are preset weights, taken as 0.4, 0.3, and 0.3 respectively, and δ is a very small positive number of 0.001 to prevent division by zero errors. The denominator reflects the current resource scarcity; the more scarce the resources, the greater the total value generated by consuming the same amount of resources.

[0066] Step S145: Select the candidate response path with the smallest expected total system resource consumption from at least one candidate response path as the target interactive response behavior chain.

[0067] Compare the Resource_total_m values ​​of all candidate response behavior chains, and select the path with the smallest value as the target interaction response behavior chain: Chain_target = argmin_{Chain_m}Resource_total_m. This ensures that, under resource constraints, the response scheme with the lowest resource consumption is selected first.

[0068] Step S146: Perform node operation parsing processing on each intent node contained in the target interaction response behavior chain, and extract the corresponding operation type code and operation parameter list from each intent node.

[0069] Iterate through each intent node v in the target interaction response behavior chain Chain_target. For each node v, parse the operation type code op_code(v) from its attributes, such as "DEVICE_CONTROL" or "STATE_QUERY". Simultaneously parse the list of operation parameters params(v) required for the operation; for example, for "DEVICE_CONTROL", the parameter list might include the device identifier device_id and the operation command. Save the (op_code, params) pairs parsed from each node.

[0070] Step S147: Arrange the operation type code and operation parameter list corresponding to each intent node in the order of appearance of the intent nodes in the target interaction response behavior chain to generate an operation instruction sequence.

[0071] According to the order of the nodes in Chain_target, the (op_code, params) pairs parsed in step S146 are arranged sequentially to form an ordered sequence of operation instructions Inst_seq=[(op_code1, params1), (op_code2, params2), ..., (op_codeL, paramsL)], where L is the number of nodes in Chain_target.

[0072] Step S148: Input the operation instruction sequence into a preset instruction compiler for instruction compilation processing, convert the operation instruction sequence into a binary instruction code stream that can be recognized by the target application system, and generate the instruction code stream part of the interactive response instruction.

[0073] The operation instruction sequence `Inst_seq` is input into the preset instruction compiler. The instruction compiler maintains an opcode mapping table, mapping `op_code` to fixed-length binary opcodes; for example, "DEVICE_CONTROL" is mapped to 0x01, and "STATE_QUERY" is mapped to 0x02. For each operation instruction, the compiler first converts the opcode to binary, and then, according to the target application system's interface definition language, encodes each parameter in the parameter list `params` into binary data of a specific format (such as integers, string length plus content, etc.). Finally, the binary encodings of all instructions are concatenated sequentially to form a continuous binary instruction stream `Instruction_stream`.

[0074] Step S149: Extract the service discovery protocol identifier and service access endpoint address of the target application system from the preset network configuration parameters, and establish a remote procedure call connection with the target application system based on the service discovery protocol identifier and the service access endpoint address.

[0075] The system reads the Service Discovery Protocol (SIP) identifier of the target application system from the local network configuration file, such as "mDNS" or "SSDP," and the service access endpoint address, such as "192.168.1.100:8080." Based on the SIP identifier, it resolves the address within the local network. For example, if the protocol is "mDNS," it queries whether the device corresponding to the IP address is online using the multicast DNS protocol. After confirming the device's reachability, it initiates a connection request to the IP address and port based on the Transmission Control Protocol (TCP). After a three-way handshake, a reliable remote procedure call (RPC) connection is established with the target application system.

[0076] Step S1410: Encapsulate the instruction code stream portion of the interactive response instruction into the message body payload area of ​​the remote procedure call protocol, and add a call identifier field and a call timeout field to the header of the message body to generate a remote procedure call request message to be sent.

[0077] The binary instruction stream `Instruction_stream` generated in step S148 is used as the message payload and placed into the message body of the Remote Procedure Call (RPC) protocol. Two fields are added to the message header: a unique call identifier `call_id`, composed of the current timestamp (in milliseconds) and an auto-incrementing sequence number, for example, `call_id = (timestamp << 16) | sequence_num`; and a call timeout field `timeout_ms`, set to 5000, indicating that the maximum time to wait for a response for this call is 5000 milliseconds. After assembly, a complete Remote Procedure Call (RPC) request message `RPC_req` is generated.

[0078] Step S1411: Send the remote procedure call request message to the target application system through the established remote procedure call connection, and start the call timeout timer to wait for the remote procedure call response message returned by the target application system.

[0079] The RPC_req message is sent to the target application system via the TCP connection established in step S149. Immediately after sending, a call timeout timer is started, with a duration of 5000 milliseconds. Within 5000 milliseconds, the calling thread blocks and waits asynchronously for the target application system to return the remote procedure call response message RPC_resp.

[0080] Step S1412: Receive the remote procedure call response message returned by the target application system, parse the call return status code and call return data from the remote procedure call response message, and write the call return status code and the call return data into the interaction log record.

[0081] Before the timeout expires, an RPC_resp message is received from the target application system. The message header is parsed to obtain the call return status code (status_code), for example, 0x00 indicates success, 0x01 indicates failure, and 0x02 indicates timeout. The message body is parsed to obtain the call return data (return_data), such as the result or status information of the operation. The call_id, status_code, return_data, and current timestamp are combined into an interaction log record and written to the local persistent storage system.

[0082] Step S210: Obtain the user input voice stream corresponding to the current interaction round, and perform speech recognition processing on the user input voice stream to generate a user input text sequence.

[0083] Before step S110, the user's voice signal is acquired via a microphone array, forming a user input voice stream, audio_stream. This audio_stream is then input into an automatic speech recognition model. This model employs an end-to-end Conformer architecture, first dividing the voice stream into frames of 25 milliseconds each with a 10-millisecond frame shift, and extracting 80-dimensional Mel filter bank features. The feature sequence is then input into a Conformer encoder, which consists of 12 stacked Conformer blocks, each containing a multi-head self-attention module, a convolutional module, and a feedforward network module. The encoder output is fed into a 6-layer Transformer decoder, which decodes word-by-word using an autoregressive approach to generate the most probable text sequence. The final output is the user input text sequence, text_input, for example, "Turn on that light".

[0084] Step S220: Perform semantic slot filling processing on the user input text sequence, and identify at least one semantic slot contained in the user input text sequence and the slot filling value corresponding to each semantic slot.

[0085] The user-input text sequence `text_input` is fed into a BERT-based semantic slot filling model. A `[CLS]` marker is added before the text, and a `[SEP]` marker is added after it before inputting into the BERT model. The BERT model consists of a 12-layer Transformer encoder, each layer containing 768 hidden units and 12 attention heads. The output vector corresponding to each token in the last layer of the BERT model is taken and input into a Conditional Random Field (CRF) layer. The CRF layer learns the transition probabilities between labels and outputs the optimal label sequence for each token. The labels use the format "B-slot type", "I-slot type", and "O". For example, for "turn on that light", it might identify "that" as "B-reference" and "light" as "B-device type". Based on the label sequence, semantic slots and filling values ​​are extracted, such as the slot "device type" being filled with "light" and the slot "reference" being filled with "that".

[0086] Step S230: Perform slot conflict detection processing on the at least one semantic slot to detect whether there is a slot conflict event in the at least one semantic slot where the slot fill values ​​of at least two semantic slots are contradictory.

[0087] The identified semantic slots and their fill values ​​are input into the slot conflict detection module. This module maintains a domain knowledge base that records the compatibility relationships between different slot values. For example, the knowledge base records multiple instances under the "lamp" type: "living room lamp", "dining room lamp", and "bedroom lamp". When the "device type" is "lamp" and the "reference" is "that", the specific instance cannot be uniquely identified, constituting a "reference ambiguity" conflict. After detecting such a conflict, the module outputs a conflict event flag conflict_flag=true, and a list of slots involved in the conflict conflict_slots=["device type", "reference"].

[0088] Step S240: If the slot conflict event is detected, generate slot conflict description information based on at least two semantic slots involved in the slot conflict event.

[0089] Since conflict_flag=true, conflict_desc is generated based on the slots and their fill values ​​in conflict_slots. conflict_desc is a structured data that contains the conflict type "referential ambiguity", the device type "light", and a list of all candidate devices retrieved from the device registry: candidates=[{"id":light1", "name":living room light"}, {"id":light2", "name":dining room light}, {"id":light3", "name":bedroom light"}].

[0090] Step S250: Query at least one ambiguous intent node associated with the slot conflict description information from the preset ambiguous intent knowledge graph, and each ambiguous intent node corresponds to an ambiguous intent to be resolved.

[0091] Use `conflict_desc` as the query condition and input a predefined ambiguous intent knowledge graph. This graph is stored in a graph database, with node types including "ambiguous intent," "device," and "action," and edges representing semantic relationships. The query matches ambiguous intent nodes related to "light" and "referencing resolution," returning three nodes: node1, node2, and node3, corresponding to the three ambiguous intents to be resolved: "select living room light," "select dining room light," and "select bedroom light," respectively.

[0092] Step S260: Perform node attribute parsing processing on the at least one ambiguous intent node, and extract the corresponding ambiguous intent description text and ambiguous intent resolution prompt information from each ambiguous intent node.

[0093] Perform attribute parsing on the three returned ambiguous intent nodes. Extract the descriptive text desc1="turn on the living room light" and the hint information hint1="located next to the sofa" from node1. Extract desc2="turn on the dining room light" and hint2="located above the dining table" from node2. Extract desc3="turn on the bedroom light" and hint3="located on the bedside table" from node3.

[0094] Step S270: Generate ambiguous intent resolution options for each ambiguous intent node based on the ambiguous intent description text and ambiguous intent resolution prompt information corresponding to each ambiguous intent node, and combine all ambiguous intent resolution options into the ambiguous intent resolution option set.

[0095] Based on the extracted descriptive text and hint information, three ambiguous intent resolution options are generated: button o1 displays the text desc1 with hint1 in smaller text below; button o2 displays the text desc2 with hint2 in smaller text below; button o3 displays the text desc3 with hint3 in smaller text below. o1, o2, and o3 are combined into an ambiguous intent resolution option set O = {o1, o2, o3}.

[0096] Step S280: The set of ambiguous intent resolution options is presented in the form of a floating window on the user interface, and the corresponding ambiguous intent resolution prompt information is displayed next to each ambiguous intent resolution option.

[0097] Finally, on the smart home control screen, a set of ambiguous intent resolution options O is rendered as a floating window. Each option button and its prompt information are clearly visible, waiting for the user to select via sight and touch. This step marks the beginning of step S110.

[0098] Step S310: Perform gaze regression detection processing on the gaze wandering path sequence, identify the regression events in the gaze wandering path sequence where the user's gaze wanders from the screen area corresponding to the current ambiguity intention resolution option to the screen area corresponding to other ambiguity intention resolution options and then returns to the screen area corresponding to the current ambiguity intention resolution option, and record the number of gaze regressions corresponding to each ambiguity intention resolution option.

[0099] During the processing of the gaze-wandering path sequence in step S120, gaze regression detection is performed in parallel. For each option 'ok', its corresponding screen region is Region_k. The gaze-wandering path sequence G is traversed, and the current gaze region prev_region is maintained. When the idi of the gaze point gi changes from equal to k to not equal to k (i.e., leaving region k), and then becomes equal to k again (i.e., returning to region k), and the idi is never equal to k in between, it is counted as a gaze regression event for option 'ok'. After traversing the entire sequence, the total number of regressions for each option 'ok' is counted and denoted as R_cnt_k.

[0100] Step S320: Perform regression attenuation adjustment processing on the gaze intent certainty index corresponding to each ambiguous intent resolution option based on the gaze regression number corresponding to each ambiguous intent resolution option. Multiply the gaze intent certainty index corresponding to each ambiguous intent resolution option by a regression attenuation coefficient that is negatively correlated with the gaze regression number to generate the regression-adjusted gaze intent certainty index corresponding to each ambiguous intent resolution option.

[0101] Construct the regression attenuation coefficient β_k = 1 / (1 + R_cnt_k). Multiply the gaze intent certainty index C_k calculated in step S126 by β_k to obtain the regression-adjusted gaze intent certainty index C_k_adjusted = C_k * β_k. The more gaze regressions R_cnt_k, the smaller the attenuation coefficient β_k, and the smaller the adjusted C_k_adjusted, indicating that the user's certainty about the option decreases due to repeated wandering.

[0102] Step S330: Update the gaze intent certainty index according to the regression-adjusted gaze intent certainty index corresponding to each of the ambiguous intent resolution options.

[0103] Replace the original C_k with C_k_adjusted, and the updated gaze intent determinism index vector is C_gaze_adjusted = [C1_adjusted, C2_adjusted, C3_adjusted]. This updated vector will be used in subsequent intent understanding models.

[0104] Step S340: Perform pressure mutation detection processing on the touch pressure fluctuation sequence, identify pressure mutation points in the touch pressure fluctuation sequence where the change in touch pressure value exceeds a preset mutation threshold within a unit time, and record the mutation time and mutation direction corresponding to each pressure mutation point.

[0105] Traverse the touch pressure fluctuation sequence P. For each point pj (j from 2 to M-1), calculate the pressure difference Δf_prev = fj - f(j-1) with the previous point and the pressure difference Δf_next = f(j+1) - fj with the next point. If |Δf_prev| > δ_sudden and |Δf_next| > δ_sudden, where δ_sudden is the mutation threshold of 0.3 Newtons, then identify the point as a pressure mutation point. Record the timestamp tj of this point, and determine the mutation direction based on the sign of Δf_prev: if Δf_prev > 0, it is a positive mutation; otherwise, it is a negative mutation.

[0106] Step S350: Divide the pressure mutation points into positive pressure mutation points and negative pressure mutation points according to the mutation direction corresponding to the pressure mutation points.

[0107] Based on the mutation direction information recorded in step S340, all identified stress mutation points are divided into two categories. All points with positive mutation directions are assigned to the positive stress mutation point set Sudden_pos, and all points with negative mutation directions are assigned to the negative stress mutation point set Sudden_neg.

[0108] Step S360: Count the number of positive pressure mutation points and the number of negative pressure mutation points in the touch pressure fluctuation sequence, calculate the absolute value of the difference between the number of positive pressure mutation points and the number of negative pressure mutation points, and divide the absolute value of the difference by the total number of positive and negative pressure mutation points to generate a pressure mutation direction imbalance parameter.

[0109] Count the number of elements in `Sudden_pos` (N_pos) and `Sudden_neg` (N_neg). Calculate the total number of elements (N_total = N_pos + N_neg). Calculate the absolute value of the difference (|N_pos - N_neg|). Generate the imbalance parameter U_imbalance = |N_pos - N_neg| / (N_total + ε), where ε is 0.001. The closer U_imbalance is to 1, the more unidirectional the pressure change, potentially reflecting user agitation or rough handling.

[0110] Step S370: Adjust the touch intent stability index by imbalance penalty according to the imbalance parameter of the pressure change direction, and multiply the touch intent stability index by the imbalance penalty coefficient that is negatively correlated with the imbalance parameter of the pressure change direction to generate the imbalance-adjusted touch intent stability index.

[0111] Construct an imbalance penalty coefficient γ = 1 - U_imbalance. Multiply the touch intent stability index T_touch calculated in step S1213 by γ to obtain the imbalance-adjusted touch intent stability index T_touch_adjusted = T_touch * (1 - U_imbalance). The larger U_imbalance is, the smaller the penalty coefficient γ is, and the smaller the adjusted T_touch_adjusted is, indicating that the touch stability is reduced due to pressure change imbalance.

[0112] Step S380: Update the touch intent stability index according to the imbalance-adjusted touch intent stability index.

[0113] Replace the original T_touch with T_touch_adjusted. The updated touch intent stability index will be used in the subsequent intent understanding model.

[0114] Step S410: Obtain the historical interaction trust parameter corresponding to the current interaction round. The historical interaction trust parameter includes the number of times the user and the target application system successfully interacted and the number of times they failed to interact in the historical interaction rounds before the current interaction round.

[0115] During step S130, when the intent understanding model is invoked, the local database is queried to retrieve all historical interaction records between the current user and the smart home system prior to the current interaction round. The total number of interaction rounds in the historical records that successfully fulfilled the user's intent, S_success, and the total number of interaction rounds that failed or were canceled by the user, F_fail, are counted.

[0116] Step S420: Calculate the historical interaction success rate parameter based on the ratio of the number of successful interactions to the sum of the number of successful interactions and the number of failed interactions.

[0117] The historical interaction success rate parameter R_success is calculated as R_success = S_success / (S_success + F_fail + ε), where ε is 0.001. R_success, ranging from 0 to 1, reflects the level of trust established between the user and the system over a long period.

[0118] Step S430: Calculate the trust weighting coefficient based on the historical interaction success rate parameter, and adjust the first game coefficient and the second game coefficient using the trust weighting coefficient to generate the adjusted first game coefficient and the adjusted second game coefficient.

[0119] Calculate the trust weighting coefficient λ = 1 + R_success. Multiply the first game coefficient α1 (original value 0.1) used in step S134 by λ to obtain the adjusted first game coefficient α1_adjusted = α1 * (1 + R_success). Multiply the second game coefficient α2 (original value 0.1) used in step S135 by λ to obtain the adjusted second game coefficient α2_adjusted = α2 * (1 + R_success). The higher the historical interaction success rate, the larger λ is, and the larger the game adjustment coefficient is, meaning that there is more trust in historical successful experiences.

[0120] Step S440: Using the adjusted first game coefficient, perform a second round of strategy adjustment processing on the gaze feature vector after the initial game according to the gaze touch game payoff matrix. Correct the strategy of the feature dimensions in the gaze feature vector after the initial game that have strategy conflicts with the touch stability feature vector to generate the gaze feature vector after the second game.

[0121] Based on the payoff matrix M_payoff generated in step S133 and the touch stability feature vector F_touch generated in step S132, the same method as in step S134 is used to identify the feature dimensions in the post-initial game gaze feature vector F_gaze_p1 that conflict with F_touch. A second round of correction is performed using the adjusted first game coefficient α1_adjusted: F_gaze_p2_i = F_gaze_p1_i - α1_adjusted * sign (conflict) * F_gaze_p1_i. After processing all dimensions, the post-secondary game gaze feature vector F_gaze_p2 is obtained.

[0122] Step S450: Using the adjusted second game coefficients, perform a second round of strategy adjustment processing on the touch feature vector after the initial game according to the gaze-touch game payoff matrix. Correct the strategy of the feature dimensions in the touch feature vector after the initial game that conflict with the gaze deterministic feature vector to generate the touch feature vector after the second game.

[0123] Similarly, based on M_payoff and F_gaze, we identify the feature dimensions in the touch feature vector F_touch_p1 that conflict with F_gaze after the initial game. We then use the adjusted second game coefficient α2_adjusted for a second round of correction: F_touch_p2_j = F_touch_p1_j - α2_adjusted * sign (conflict) * F_touch_p1_j. This yields the touch feature vector F_touch_p2 after the second game.

[0124] Step S460: Input the post-secondary game gaze feature vector and the post-secondary game touch feature vector into the Nash equilibrium solution layer of the exponential competition game module for secondary equilibrium strategy solution processing to generate a secondary mixed strategy Nash equilibrium vector.

[0125] Input F_gaze_p2 and F_touch_p2 back into the Nash equilibrium solution layer described in step S136, repeat the iterative optimization process, solve the new hybrid policy Nash equilibrium, and generate the secondary hybrid policy Nash equilibrium vector P_nash_2.

[0126] Step S470: Perform maximum value indexing on the quadratic mixing strategy Nash equilibrium vector, extract the index position corresponding to the element with the largest value from the quadratic mixing strategy Nash equilibrium vector, and determine the corresponding ambiguous intent resolution option from the ambiguous intent resolution option set based on the index position as the user's true intent identifier.

[0127] Compare the three probability values ​​in P_nash_2, find the index k*_2 corresponding to the maximum value, and select the option with index k*_2 from the ambiguous intent resolution option set O as the user's true intent identifier intent_id_2. This step updates or replaces the result of step S137, incorporating historical trust information into the intent decision.

[0128] Step S510: Associate and store the user's true intent identifier with the gaze-free path sequence and touch pressure fluctuation sequence corresponding to the current interaction round to generate the game record of the current interaction round.

[0129] After each interaction, the final determined user intent_id (which may be the result of step S137 or S470) is associated with the original gaze-free path sequence G and the original touch pressure fluctuation sequence P collected in step S110, as well as all intermediate data generated in this round of interaction (such as C_gaze, T_touch, P_nash, etc.), through the unique interaction round identifier session_id, and packaged into a structured game record of the current interaction round, which is then stored in the historical database.

[0130] Step S520: Add the current round of game record to the historical game record set, and update the historical game record set.

[0131] The game record of the current round of interaction is appended to the set of historical game records with the user identifier user_id as the key, thus completing the update of the set of historical game records.

[0132] Step S530: Extract the historical gaze-free path sequence and historical touch pressure fluctuation sequence corresponding to all historical interaction rounds that are identical to the user's true intent identifier from the updated historical game record set.

[0133] When model calibration is required, all historical interaction round records that are completely consistent with the current user's true intent identifier intent_id are selected from the updated historical game record set. From these records, the historical gaze-free path sequence G_hist_i and the historical touch pressure fluctuation sequence P_hist_i corresponding to each interaction are extracted to form a positive sample dataset D_hist={(G_hist_1, P_hist_1), (G_hist_2, P_hist_2), ..., (G_hist_H, P_hist_H)}, where H is the number of historical interaction rounds.

[0134] Step S540: Perform path fractal dimension averaging on the historical gaze-free path sequences corresponding to all historical interaction rounds, calculate the arithmetic mean of the fractal dimension parameters of the historical gaze-free trajectory corresponding to each of the ambiguous intent resolution options, and process the arithmetic mean using the negative correlation mapping function to generate the historical average gaze intent determinism index.

[0135] For each historical interaction round in D_hist, steps S121 to S125 are repeated for its gaze-free path sequence G_hist_i to calculate the fractal dimension parameter D_k_hist_i of the historical gaze-free trajectory for each option ok. Assuming there are H historical rounds, H D_k_hist values ​​are obtained for option ok. The arithmetic mean D_k_hist_avg = (1 / H) * ΣD_k_hist_i is calculated. Then, D_k_hist_avg is substituted into the negative correlation mapping function in step S126 to obtain the historical average gaze intent determinism index C_k_hist_avg = 1 / (D_k_hist_avg + ε) for option ok. After calculation for all options, the historical average gaze intent determinism index vector C_gaze_hist_avg = [C1_hist_avg, C2_hist_avg, C3_hist_avg].

[0136] Step S550: Perform fluctuation singular spectrum averaging analysis on the historical touch pressure fluctuation sequence corresponding to all historical interaction rounds, calculate the arithmetic mean of the amplitude of the historical principal singular value component corresponding to each of the ambiguous intent resolution options, and generate the historical average touch intent stability index.

[0137] For each historical interaction round in D_hist, steps S127 to S1213 are repeated for its touch pressure fluctuation sequence P_hist_i to calculate the historical touch intent stability index T_touch_hist_i. H T_touch_hist values ​​are obtained. Their arithmetic mean T_touch_hist_avg = (1 / H) * ΣT_touch_hist_i is calculated.

[0138] Step S560: Input the historical average gaze intent certainty index and the historical average touch intent stability index into the index competition game module of the intent understanding model for index competition game processing to generate the user's true intent identifier after historical game.

[0139] Using C_gaze_hist_avg and T_touch_hist_avg as input, and following the processing flow from S131 to S137, the input is used to understand the intent model, and a complete exponential competitive game is performed to finally obtain a historical game-related user intent identifier intent_id_hist_pred.

[0140] Step S570: Perform a consistency comparison between the historical post-game user true intent identifier and the user true intent identifier. If the historical post-game user true intent identifier and the user true intent identifier are inconsistent, generate a game model calibration trigger signal.

[0141] The intent_id_hist_pred is compared with the original user's true intent identifier intent_id, which is used as the filtering criterion. If the two are inconsistent, for example, intent_id_hist_pred is o1 while intent_id is o2, it indicates that the current intent understanding model has a bias when reviewing historical data, and a game model calibration trigger signal calibrate_signal=true is generated.

[0142] Step S580: Based on the game model calibration trigger signal, call the parameter update module of the intent understanding model, and use the historical gaze-free path sequence and historical touch pressure fluctuation sequence corresponding to all historical interaction rounds as training data to fine-tune and update the model parameters of the exponential competition game module of the intent understanding model.

[0143] In response to `calibrate_signal=true`, the parameter update module of the intent understanding model is invoked. Historical gaze-free path sequences and historical touch pressure fluctuation sequences from `D_hist` are used as training data, with their corresponding historical real user intent identifiers `intent_id` as labels. The Adam optimizer is employed, with a learning rate of 0.0001 and a batch size of 8, using the cross-entropy loss function as the optimization objective to fine-tune and update the learnable parameters in the exponential competitive game module (including W_ij in the payoff matrix, weights in the policy mapping layer, etc.). Five training epochs are set to prevent overfitting. Through this online learning method, the model can self-correct based on historical experience.

[0144] Step S610: Extract at least two candidate interactive response behavior chains associated with the user's real intent identifier from the preset intent response topology network, each candidate interactive response behavior chain corresponding to a different response complexity level.

[0145] In step S140, when extracting the interaction response behavior chain, all possible response paths associated with the user's true intent identifier intent_id are searched from the intent response topology network G_net. The paths are divided into different complexity levels based on the number and type of nodes on each path. For example, three candidate chains are extracted: Chain_A=[v_start, v_execute], corresponding to low complexity; Chain_B=[v_start, v_confirm, v_execute], corresponding to medium complexity; and Chain_C=[v_start, v_query, v_display, v_adjust, v_execute], corresponding to high complexity.

[0146] Step S620: Obtain the user cognitive load parameters corresponding to the current interaction round. The user cognitive load parameters include the continuous interaction duration of the user before the current interaction round, the number of operation errors of the user in the current interaction round, and the response delay duration of the user in the current interaction round.

[0147] Three cognitive load parameters are obtained from the interaction log: the total duration of continuous user interaction with the system before the current interaction round, T_continuous, in seconds; the number of operational errors Err_count that occur between the presentation of the option and the user's effective touch selection in the current interaction round, such as the number of times the user immediately corrects an accidental touch on a non-option area; and the response delay T_delay between the presentation of the option and the user's first effective touch operation in the current interaction round, in milliseconds.

[0148] Step S630: Construct a user cognitive load feature vector based on the continuous interaction duration, the number of operation errors, and the response delay duration, and input the user cognitive load feature vector into a preset cognitive load assessment model for load level prediction processing to generate a user's current cognitive load level identifier.

[0149] Normalize T_continuous, Err_count, and T_delay. The normalization method uses min-max normalization, employing the maximum and minimum values ​​from historical data. For example, T_continuous_norm = (T_continuous - min_T) / (max_T - min_T). Construct a three-dimensional cognitive load feature vector V_cog = [T_continuous_norm, Err_count_norm, T_delay_norm]. Input V_cog into a pre-defined cognitive load assessment model. This model is a support vector machine classifier using radial basis function kernels, pre-trained on data labeled with cognitive load levels (low, medium, high). The model outputs a user's current cognitive load level identifier L_cog, with values ​​of "low," "medium," or "high."

[0150] Step S640: Select the candidate interaction response behavior chain that matches the user's current cognitive load level identifier from the at least two candidate interaction response behavior chains as the target interaction response behavior chain.

[0151] According to the preset matching rules: if L_cog is "high", then the Chain_A with the lowest complexity is selected; if L_cog is "medium", then the Chain_B with medium complexity is selected; if L_cog is "low", then the Chain_C with the highest complexity is selected. The selected chain is then used as the target interactive response chain Chain_target.

[0152] Step S650: Perform behavior chain decoupling processing on the target interaction response behavior chain, and split the target interaction response behavior chain into at least two sub-behavior chains that can be executed in parallel.

[0153] Analyze the data dependencies between nodes in the target interaction response behavior chain Chain_target. If there is no data dependency between two nodes, they can be executed in parallel. For example, for Chain_C, there is no data dependency between the node v_query (querying device status) and the node v_display (displaying the control interface), so they can be split. Therefore, Chain_C is split into two sub-behavior chains: SubChain_1 = [v_query], SubChain_2 = [v_display, v_adjust, v_execute]. SubChain_1 and SubChain_2 can be executed in parallel.

[0154] Step S660: Perform resource contention analysis on the at least two parallel executable sub-behavior chains to detect whether there is a resource contention conflict during the execution of the at least two parallel executable sub-behavior chains.

[0155] Analyze the system resource usage of the decomposed sub-behavior chains. For example, SubChain_1 mainly uses network resources for querying, while SubChain_2 mainly uses graphics processor resources and display cache for rendering the interface. Detect whether there is resource contention between the two, such as whether they are simultaneously competing for the lock of the same system service (such as the window manager). By querying the resource usage mapping table, it was found that the two use different system services and there is no resource contention.

[0156] Step S670: If the resource contention conflict is detected, the execution order of the at least two parallel sub-behavior chains is adjusted to a serial execution order, and the adjusted sub-behavior chain execution sequence is generated.

[0157] If a resource contention conflict is detected (e.g., both subchains need exclusive access to the same hardware device), the conflicting subchains are adjusted to execute sequentially. For example, the execution order of SubChain_1 and SubChain_2 is adjusted so that SubChain_1 is executed first, and SubChain_2 is executed after it releases its resources, generating the adjusted execution sequence.

[0158] Step S680: If no resource contention conflict is detected, mark the at least two parallel executable sub-behavior chains as a set of parallel executable sub-behavior chains.

[0159] In this example, since no resource contention conflict was detected, SubChain_1 and SubChain_2 are marked as the set of parallel execution sub-behavior chains Parallel_set={SubChain_1, SubChain_2}.

[0160] Step S690: Perform node operation parsing processing on each intent node contained in the target interactive response behavior chain according to the adjusted sub-behavior chain execution sequence or the parallel execution sub-behavior chain set, and extract the corresponding operation type code and operation parameter list from each intent node.

[0161] According to the determined execution plan (parallel or serial), each intent node in Chain_target is parsed. For the parallel plan, nodes in the two sub-chains are parsed separately. For example, the operation type code "STATE_QUERY" and parameter list [device_id] are extracted from the v_query node of SubChain_1. "UI_RENDER" and parameter [template_id] are extracted from the v_display node of SubChain_2, "PARAM_ADJUST" and parameter [param_name] are extracted from the v_adjust node, and "DEVICE_CONTROL" and parameter [device_id, command] are extracted from the v_execute node.

[0162] Step S6100: According to the execution order of the intent nodes in the adjusted sub-behavior chain execution sequence or the parallel execution sub-behavior chain set, arrange the operation type code and operation parameter list corresponding to each intent node in sequence to generate an optimized operation instruction sequence.

[0163] Following the order in the parallel execution set Parallel_set, the operation instructions in each subchain are arranged according to their internal order within the subchain, generating an optimized operation instruction sequence Inst_seq_opt. Because it is parallel, the operating system can schedule instructions from two subchains simultaneously. Finally, Inst_seq_opt will be used as input to step S146, replacing the original instruction sequence.

[0164] Step S710: Within a preset intent verification time window, continuously collect the system response output stream generated by the target application system after executing the interactive response command. The system response output stream includes at least one response output unit and an output timestamp corresponding to each response output unit.

[0165] After successfully sending the command and receiving the response in step S1412, an intent verification time window with a preset length of T_verify = 3000 milliseconds is initiated. Through a message queue listening mechanism, all output data generated by the target application system during and after command execution is continuously collected, forming the system response output stream S_out. S_out is an ordered sequence, where each element is a response output unit u, represented as u = (t_out, type_out, content_out). t_out is the output timestamp, type_out is the output type (e.g., "status change", "operation result"), and content_out is the output content (e.g., "device status: ON").

[0166] Step S720: Perform semantic parsing processing on the system response output stream, extract the response output semantic vector from each response output unit, and concatenate all response output semantic vectors along the time dimension according to the output timestamp to generate a response output semantic time sequence matrix.

[0167] The system response output stream S_out is input into a pre-trained semantic parsing model. This model uses a BERT-based architecture, segmenting the content_out of each response output unit and adding [CLS] and [SEP] tags before inputting it into the BERT model. The output vector (768-dimensional) corresponding to the [CLS] tag in the last layer of BERT is taken as the response output semantic vector v_out_i for that unit. Following the increasing order of timestamp t_out_i, all v_out_i are concatenated vertically as row vectors to form an N×768 two-dimensional matrix M_semantic, where N is the total number of response output units collected within the T_verify time period. M_semantic is the temporal matrix of the response output semantics.

[0168] Step S730: Input the response output semantic time sequence matrix into a preset intent satisfaction evaluation model for satisfaction scoring processing, and generate an intent satisfaction score curve corresponding to the system response output stream. The intent satisfaction score curve includes the instantaneous satisfaction score value corresponding to each output timestamp.

[0169] The M_semantic is input into a predefined intent satisfaction evaluation model. This model consists of a one-dimensional temporal convolutional layer and gated recurrent units (ROUs). First, the M_semantic passes through a one-dimensional temporal convolutional layer with 64 kernels, each with a size of 3 and a stride of 1, outputting an N×64 feature map. This feature map is then input into a gated recurrent unit (ROU) layer containing 128 hidden units. The output of the RNU at each time step is connected to a fully connected layer with one node, using the sigmoid activation function to output a value between 0 and 1, which is the instantaneous satisfaction score s_sat_i corresponding to that timestamp. Each timestamp t_out_i and its corresponding s_sat_i are paired into ordered pairs (t_out_i, s_sat_i) to form the intent satisfaction score curve.

[0170] Step S740: Perform peak detection processing on the intent satisfaction score curve, identify the satisfaction peak points in the intent satisfaction score curve where the instantaneous satisfaction score value exceeds the preset satisfaction threshold, and record the peak time and peak score value corresponding to each satisfaction peak point.

[0171] Iterate through all points on the intention satisfaction rating curve and detect local maxima of the instantaneous satisfaction rating value s_sat_i. For each local maximum, if its s_sat_i value exceeds the preset satisfaction threshold T_sat=0.8, it is identified as a satisfaction peak. Record the timestamp t_peak_j and the peak rating value s_peak_j of this point.

[0172] Step S750: Calculate the average satisfaction peak parameter based on the arithmetic mean of the peak scores of all satisfaction peak points, and calculate the satisfaction peak time dispersion parameter based on the standard deviation of the peak times of all satisfaction peak points.

[0173] Suppose M satisfaction peak points are detected. Calculate the average satisfaction peak parameter P_peak_avg = (1 / M)*Σs_peak_j. Calculate the mean μ_t_peak = (1 / M)*Σt_peak_j of the peak time t_peak_j. Calculate the variance σ²_t_peak = (1 / (M-1))*Σ(t_peak_j-μ_t_peak)², and then take the square root to obtain the standard deviation σ_t_peak, which is the satisfaction peak time dispersion parameter.

[0174] Step S760: Perform integral processing on the intent satisfaction score curve, and calculate the area enclosed by the intent satisfaction score curve and the time axis within the intent verification time window as the cumulative total satisfaction parameter.

[0175] Numerical integration is performed on the intention satisfaction rating curve over the time interval [0, T_verify]. Using the trapezoidal rule, the area of ​​the trapezoid between adjacent time points (t_i, s_i) and (t_{i+1}, s_{i+1}) is approximated as (s_i + s_{i+1}) * (t_{i+1} - t_i) / 2. The cumulative total satisfaction parameter A_sat_total is obtained by summing all the trapezoidal areas.

[0176] Step S770: Construct an intention satisfaction comprehensive evaluation vector based on the average satisfaction peak parameter, the satisfaction peak time dispersion parameter, and the cumulative satisfaction total parameter, and input the intention satisfaction comprehensive evaluation vector into a preset feedback intensity prediction model for feedback intensity prediction processing to generate an intention feedback intensity prediction value for subsequent interaction rounds.

[0177] Construct a three-dimensional intent satisfaction comprehensive evaluation vector V_sat_eval = [P_peak_avg, σ_t_peak, A_sat_total]. Input V_sat_eval into a pre-defined feedback intensity prediction model. This model is a multilayer perceptron regression model, containing an input layer (3 nodes), two hidden layers (64 and 32 nodes respectively, with ReLU activation function), and an output layer (1 node, no activation function). The model outputs a scalar value, which is the predicted value F_pred for the intent feedback intensity in subsequent interaction rounds.

[0178] Step S780: If the estimated value of the intention feedback intensity in the subsequent interaction rounds is lower than the preset feedback intensity threshold, then extract the low satisfaction time interval from the intention satisfaction score curve where the instantaneous satisfaction score value is continuously lower than the preset low satisfaction threshold.

[0179] Compare F_pred with the preset feedback intensity threshold F_threshold=0.6. If F_pred<0.6, then find all time intervals from the intention satisfaction rating curve where the instantaneous satisfaction rating value s_sat_i is continuously lower than the preset low satisfaction threshold L_sat=0.3. Merge adjacent or overlapping intervals to obtain the low satisfaction time interval list Intervals_low=[(t_low_start_1, t_low_end_1), ...].

[0180] Step S790: Filter out the low satisfaction response output units corresponding to the low satisfaction time interval from the system response output stream, and perform output content error pattern recognition processing on the low satisfaction response output units to generate at least one error pattern label.

[0181] Based on `Intervals_low`, response output units whose output timestamps `t_out` fall within any low satisfaction time interval are selected from the system response output stream `S_out`, forming a low satisfaction response output unit set `U_low`. The `content_out` of each unit in `U_low` is input into a predefined error pattern recognition model. This model is a TextCNN classifier containing convolutional layers (kernel sizes of 2, 3, and 4), pooling layers, and fully connected layers, outputting the probability of belonging to a predefined error pattern (such as "device unresponsive" or "operation timeout"). The top 1-2 labels with the highest probabilities are taken as the error pattern label `L_err`.

[0182] Step S7100: Query the corresponding intent correction policy code from the preset intent correction policy library according to the at least one error mode label, and generate an intent correction instruction according to the intent correction policy code.

[0183] Using the error mode label L_err as the key, the system queries the preset intent correction policy library. For example, if L_err contains "Device Not Responding", the query returns the correction policy code "RETRY_WITH_BACKOFF"; if it contains "Operation Timeout", the query returns "INCREASE_TIMEOUT". Based on the retrieved policy code, a structured intent correction instruction C_inst is generated. For example, C_inst = {"Policy 1": "RETRY_WITH_BACKOFF", "Parameters": {"Maximum Retry Count": 3, "Initial Backoff": 200}, "Policy 2": "INCREASE_TIMEOUT", "Parameters": {"New Timeout": 10000}}.

[0184] Step S7110: Send the intent correction instruction to the target application system to trigger the target application system to correct the execution result of the interaction response instruction.

[0185] The intent to correct the instruction C_inst is encapsulated into a new remote procedure call request message according to the remote procedure call process described in steps S148 to S1412, and sent to the target application system through the established connection. After parsing the instruction, the target application system corrects the execution result of the previous interactive response instruction according to the strategy in the instruction, such as retrying to turn on the device or adjusting the timeout settings for subsequent operations.

[0186] Step S810: Real-time acquisition of the user's facial micro-expression image sequence corresponding to the current interaction round. The user's facial micro-expression image sequence includes continuous facial frame images from the time the ambiguous intent resolution option set is presented until the generation of the user's true intent identifier.

[0187] While presenting the ambiguous intent resolution option set in step S110, the infrared face camera built into the smart home control panel is activated to begin real-time acquisition of the user's facial video stream. The acquisition time window starts from the moment the floating window appears in step S110 and ends at the moment the user's true intent identifier is generated in step S137 or S470. The camera captures facial images at a rate of 30 frames per second, generating a sequence of micro-expression images of the user's face I_face={img_1, img_2, ..., img_F}, where each frame image img_k is a 640×480 pixel grayscale image, associated with the acquisition timestamp t_img_k.

[0188] Step S820: Perform facial action unit encoding processing on the user's facial micro-expression image sequence, extract the set of facial action unit activation states corresponding to each facial frame image from the continuous facial frame images, and the set of facial action unit activation states includes the orbicularis oculi muscle contraction state encoding, the corrugator supercilii muscle contraction state encoding, and the orbicularis oris muscle contraction state encoding.

[0189] Each frame image, img_k, is input into a pre-trained facial action unit encoding model. This model is based on a VGG-16 convolutional neural network, with the final classification layer removed and three parallel binary classification sub-networks added. Each sub-network consists of two fully connected layers, outputting a value between 0 and 1. The model outputs three encoded values: orbicularis oculi muscle contraction state encoding AU6_k, corrugator supercilii muscle contraction state encoding AU4_k, and orbicularis oris muscle contraction state encoding AU23_k. These three values ​​constitute the facial action unit activation state set V_au_k = (AU6_k, AU4_k, AU23_k) for that frame. Processing all frames yields the activation state sequence V_au_list.

[0190] Step S830: Construct a facial micro-expression state time sequence matrix based on the set of facial action unit activation states, wherein the rows of the facial micro-expression state time sequence matrix correspond to the time dimension, and the columns of the facial micro-expression state time sequence matrix correspond to the facial action unit encoding dimension.

[0191] Each 3D vector V_au_k in V_au_list is treated as a row and stacked vertically in ascending order of timestamp t_img_k to form an F×3 2D matrix M_au. M_au is the temporal matrix of facial micro-expression states, and each column represents the activation intensity curve of a facial action unit over time.

[0192] Step S840: Input the facial micro-expression state time sequence matrix into a preset micro-expression recognition model for micro-expression category recognition processing, and generate at least one micro-expression category label that appears in the current interaction round and the occurrence time corresponding to each micro-expression category label.

[0193] The input M_au is a pre-defined micro-expression recognition model. This model consists of a temporal convolutional network and a bidirectional long short-term memory (LSTM) network. The temporal convolutional network contains multiple residual blocks to capture features at different time scales. The output is fed into a 256-unit bidirectional LSTM network layer. At each time step, the output of the bidirectional LSTM network layer is connected to a fully connected classification layer, which uses the Softmax activation function to output the probability of belonging to a predefined micro-expression category ("confused", "pleased", "surprised", "anxious"). The output sequence is post-processed to identify at least one micro-expression category label, such as "confused", and its start time t_micro_start and end time t_micro_end are recorded.

[0194] Step S850: Select micro-expression category labels associated with the intentional confused state from the at least one micro-expression category label as confused micro-expression labels, wherein the confused micro-expression label includes the start time of the confused micro-expression and the end time of the confused micro-expression.

[0195] Based on a pre-defined association mapping table, tags related to the "confused intent" state, such as "confused," are selected from the identified micro-expression category tags. If the "confused" tag is selected, its start time is recorded as t_confuse_start, and its end time as t_confuse_end.

[0196] Step S860: Calculate the duration parameter of the confused micro-expression based on the start time and end time of the confused micro-expression, and generate a confusion intensity coefficient based on the duration parameter of the confused micro-expression.

[0197] Calculate the duration of the confused micro-expression: T_confuse_dur = t_confuse_end - t_confuse_start. The confusion intensity coefficient α_confuse = min(1, T_confuse_dur / T_base), where T_base is the baseline duration, taken as 1000 milliseconds. If T_confuse_dur ≥ 1000, then α_confuse = 1; if T_confuse_dur < 1000, then α_confuse = T_confuse_dur / 1000.

[0198] Step S870: Extract the gaze wandering sequence within the time window corresponding to the start time of the confused microexpression to the end time of the confused microexpression from the gaze wandering path sequence.

[0199] Based on the time window [t_confuse_start, t_confuse_end], all line-of-sight points whose timestamps ti fall within this window are selected from the complete line-of-sight free path sequence G collected in step S110, and rearranged in chronological order to form the line-of-sight free ion sequence G_confuse during the confusion period.

[0200] Step S880: Perform path fractal dimension calculation on the confused period free ion sequence to generate the fractal dimension parameter of the confused period free trajectory, and process the fractal dimension parameter of the confused period free trajectory using the negative correlation mapping function to generate the confusion period line of intent certainty index; calculate the line of sight confusion contribution parameter based on the absolute value of the difference between the confusion period line of intent certainty index and the line of sight intent certainty index.

[0201] For G_confuse, repeat steps S121 to S125 to calculate the fractal dimension parameter D_confuse of the line-of-sight trajectory during the confusion period. Substitute this into the negative correlation mapping function to obtain the line-of-sight intent certainty index C_confuse = 1 / (D_confuse + ε). From the line-of-sight intent certainty indices calculated in step S126, select C_selected, which corresponds to the final intent_id. Calculate the line-of-sight confusion contribution parameter Δ_gaze = |C_confuse - C_selected|.

[0202] Step S890: Extract the touch pressure subsequence during the confusion period from the touch pressure fluctuation sequence, within the time window corresponding to the start time of the confused micro-expression to the end time of the confused micro-expression.

[0203] Similarly, from the touch pressure fluctuation sequence P, all pressure points with timestamps tj falling within [t_confuse_start, t_confuse_end] are selected and rearranged in chronological order to form the touch pressure subsequence P_confuse during the confusion period.

[0204] Step S8100: Perform confusion period fluctuation singular spectrum analysis on the confusion period touch pressure subsequence to generate the amplitude of the principal singular value component of the confusion period touch pressure, and use the amplitude of the principal singular value component of the confusion period touch pressure as the touch intention stability index during the confusion period; calculate the touch confusion contribution parameter based on the absolute value of the difference between the touch intention stability index during the confusion period and the touch intention stability index.

[0205] For P_confuse, repeat steps S127 to S1213 to calculate the touch intent stability index T_confuse during the confusion period. Select T_touch from the touch intent stability indices calculated in step S1213. Calculate the touch confusion contribution parameter Δ_touch = |T_confuse - T_touch|.

[0206] Step S8110: Normalize the gaze confusion contribution parameter and the touch confusion contribution parameter to generate normalized gaze confusion contribution parameters and normalized touch confusion contribution parameters. Use the normalized gaze confusion contribution parameter as the weight of gaze mode in the multimodal confusion fusion weight coefficient, and use the normalized touch confusion contribution parameter as the weight of touch mode in the multimodal confusion fusion weight coefficient.

[0207] Calculate the sum of Δ_gaze and Δ_touch: Sum_delta = Δ_gaze + Δ_touch + ε. Calculate the normalized gaze confusion contribution parameter W_gaze = Δ_gaze / Sum_delta. Calculate the normalized touch confusion contribution parameter W_touch = Δ_touch / Sum_delta. W_gaze and W_touch are the multimodal confusion fusion weight coefficients.

[0208] Step S8120: Calculate a first confusion adjustment factor and a second confusion adjustment factor based on the confusion intensity coefficient, the normalized gaze confusion contribution parameter, and the normalized touch confusion contribution parameter; use the first confusion adjustment factor to weight and adjust the gaze intent certainty index to generate a confusion-adjusted gaze intent certainty index; use the second confusion adjustment factor to weight and adjust the touch intent stability index to generate a confusion-adjusted touch intent stability index.

[0209] Calculate the first confusion adjustment factor β_gaze = 1 + α_confuse * W_gaze. Calculate the second confusion adjustment factor β_touch = 1 + α_confuse * W_touch. Calculate the confusion-adjusted gaze intent certainty index C_adjusted = C_selected * β_gaze. Calculate the confusion-adjusted touch intent stability index T_adjusted = T_touch * β_touch.

[0210] Step S8130: Input the confusion-adjusted gaze intent certainty index and the confusion-adjusted touch intent stability index into the index competition game module of the intent understanding model for secondary index competition game processing to generate the user's true intent identifier after confusion calibration.

[0211] Following steps S131 to S137, C_adjusted (as a three-dimensional vector, corresponding to the three options) and T_adjusted (scalar) are re-inputted into the intent understanding model to perform a complete exponential competitive game process, resulting in the confused-calibrated user's true intent identifier intent_id_calibrated.

[0212] Step S8140: Perform a consistency comparison between the confused user's true intent identifier after calibration and the user's true intent identifier. If the confused user's true intent identifier after calibration is inconsistent with the user's true intent identifier, then update the user's true intent identifier according to the confused user's true intent identifier after calibration.

[0213] Compare intent_id_calibrated with the original intent_id. If they match, no update is made. If they do not match, for example, intent_id_calibrated is o1 while intent_id is o2, then intent_id is updated with intent_id_calibrated, and subsequent steps S140 will be performed based on the updated intent identifier.

[0214] Step S910: Start the interaction feedback effect monitoring timer, and continuously collect the interaction feedback effect data returned by the target application system within the preset effect monitoring time window. The interaction feedback effect data includes the interface change image sequence after the interaction response command is successfully executed and the user's subsequent operation trajectory sequence.

[0215] After successfully sending the command in step S1412, start the interaction feedback effect monitoring timer and set the monitoring time window T_monitor=5000 milliseconds. Start two acquisition threads: the screen recording thread captures interface images at 10 frames / second and generates an interface change image sequence I_ui={img_ui_1, ..., img_ui_U}; the operation listening thread listens for subsequent user operations at 50 times / second and generates a user subsequent operation trajectory sequence Op={op_1, ..., op_V}, where each operation point op_n=(t_op_n, type_op_n, x_op_n, y_op_n, p_op_n).

[0216] Step S920: Perform image difference processing on the interface change image sequence, calculate the pixel difference between adjacent interface change images, and generate an interface change severity parameter based on the pixel difference.

[0217] Convert all images in I_ui to grayscale. Starting from the second frame, calculate the pixel difference between each frame and the previous frame: D_k = Σ|gray_k(x,y) - gray_{k-1}(x,y)|, summing over all pixels. This yields a difference sequence D_list. Calculate the mean D_avg of D_list. Calculate the normalized average difference I_change = D_avg / (W*H*255), where W and H are the image width and height. I_change represents the degree of drastic change in the interface.

[0218] Step S930: Perform trajectory smoothness calculation processing on the user's subsequent operation trajectory sequence, calculate the directional change angle between adjacent operation trajectory points in the user's subsequent operation trajectory sequence, and generate user operation smoothness parameters based on the standard deviation of all directional change angles.

[0219] Points with the type "sliding movement" are selected from the Op sequence to form the trajectory point sequence Track. For the i-th point in Track, the vectors V_i = (x_i - x_{i-1}, y_i - y_{i-1}) and V_{i+1} = (x_{i+1} - x_i, y_{i+1} - y_i) are calculated. The included angle θ_i = arccos((V_i·V_{i+1}) / (|V_i|·|V_{i+1}|+ε) is calculated. For all θ_i, the standard deviation σ_θ is calculated. The user operation smoothness parameter S_smooth = 1 / (σ_θ+ε).

[0220] Step S940: Construct an interaction feedback effect evaluation feature vector based on the interface change severity parameter and the user operation smoothness parameter, and input the interaction feedback effect evaluation feature vector into a preset feedback effect evaluation model for effect level prediction processing to generate an interaction feedback effect level identifier.

[0221] Construct a two-dimensional feature vector V_fb_eval = [I_change, S_smooth]. Input a pre-defined feedback effect evaluation model, which is a gradient boosting decision tree classifier with 100 decision trees, each with a depth of 3. The model outputs the probability of belonging to one of four levels: "Excellent", "Good", "Average", and "Poor". The level with the highest probability is taken as the interaction feedback effect level label L_fb.

[0222] Step S950: If the interactive feedback effect level indicator is lower than the preset effect level threshold, then extract the possible cause features of the interactive response instruction execution failure from the interactive feedback effect data.

[0223] The preset effect level threshold is "Good". If L_fb is "Medium" or "Poor", then extract possible cause features: detect error pop-ups from interface change images and extract keywords from the pop-up text; count the number of repeated clicks from the operation trajectory; extract error codes from the system log. Summarize these into a possible cause feature set F_failure_reason.

[0224] Step S960: Based on the possible cause characteristics, query the corresponding failure cause entity from the preset failure cause knowledge base, and extract the failure cause description text and optimization suggestion information from the failure cause entity.

[0225] Using F_failure_reason as the query condition, query the knowledge base graph database of failure reasons. Match the corresponding failure reason entity, such as "unstable network connection". Extract the failure reason description text "weak Wi-Fi signal of device and gateway" and the optimization suggestion information "suggest restarting the gateway or moving closer to the device" from the attributes of this entity.

[0226] Step S970: Based on the optimization suggestion information, perform instruction optimization and adjustment processing on the interactive response instruction to generate an optimized interactive response instruction, and resend the optimized interactive response instruction to the target application system.

[0227] Based on the optimization suggestions, the original instruction sequence was modified by inserting a "RETRY_CONTROL" instruction at the instruction header, with parameters of a maximum of 3 retries and a retry interval of 1000 milliseconds. The optimized instruction code stream was generated, repackaged, and sent to the target application system via RPC.

[0228] Step S980: If the interaction feedback effect level identifier is higher than or equal to the preset effect level threshold, then the interaction feedback effect level identifier is associated with the user's true intent identifier and stored to generate a successful interaction feedback effect record.

[0229] If L_fb is "Excellent" or "Good", then L_fb is associated with intent_id, and packaged with all data of the current interaction round into a record of successful interaction feedback effect, stored in the historical database, and used as a positive sample for subsequent model optimization.

[0230] Figure 2This illustration shows a multimodal fusion-based intelligent interactive intent understanding system 100, including a processor 1001, a memory 1003, and program code stored in the memory 1003. The processor 1001 executes the program code to implement the steps of the multimodal fusion-based intelligent interactive intent understanding method. The processor 1001 and the memory 1003 are connected, for example, via a bus 1002. Optionally, the multimodal fusion-based intelligent interactive intent understanding system 100 may further include a transceiver 1004, which can be used for data interaction between this multimodal fusion-based intelligent interactive intent understanding system and other multimodal fusion-based intelligent interactive intent understanding systems, such as sending and / or receiving data. It should be noted that in actual scheduling, the transceiver 1004 is not limited to one, and the structure of this multimodal fusion-based intelligent interactive intent understanding system 100 does not constitute a limitation on the embodiments of this application. The memory 1003 is used to store the program code executing the embodiments of this application, and its execution is controlled by the processor 1001. The processor 1001 is used to execute program code stored in the memory 1003 to implement the steps shown in the foregoing method embodiments.

[0231] This application provides a computer-readable storage medium storing program code, which, when executed by a processor, can implement the steps and corresponding content of the aforementioned method embodiments.

[0232] The above description is only an optional implementation method for some implementation scenarios of this application. It should be noted that for those skilled in the art, other similar implementation methods based on the technical concept of this application, without departing from the technical concept of this application, also fall within the protection scope of the embodiments of this application.

Claims

1. A method for understanding intelligent interactive intent based on multimodal fusion, characterized in that, The method includes: The user interface presents a set of ambiguous intent resolution options corresponding to the current interaction round, and simultaneously collects the user's gaze-free path sequence and touch pressure fluctuation sequence for each ambiguous intent resolution option in the set. The method for determining the ambiguous intent resolution option set includes: acquiring the user's input speech stream corresponding to the current interaction round; performing speech recognition processing on the user's input speech stream to generate a user input text sequence; performing semantic slot filling processing on the user input text sequence to identify at least one semantic slot and its corresponding slot filling value; performing slot conflict detection processing on the at least one semantic slot to detect whether there is a slot conflict event where at least two semantic slots have contradictory slot filling values; if the slot conflict event is detected, generating a slot conflict resolution mechanism based on the at least two semantic slots involved in the slot conflict event. The system retrieves and queries at least one ambiguous intent node associated with the slot conflict description information from a preset ambiguous intent knowledge graph. It then performs node attribute parsing on the at least one ambiguous intent node, extracts the ambiguous intent description text and ambiguous intent resolution prompts, generates ambiguous intent resolution options, and combines all ambiguous intent resolution options into an ambiguous intent resolution option set. The gaze-free path sequence is composed of user gaze focus coordinates collected by an eye-tracking sensor at a preset sampling rate, arranged in chronological order. Each element in the gaze-free path sequence includes gaze focus coordinates, a sampling timestamp, the index of the ambiguous intent resolution option to which the current gaze point belongs, and a gaze-free status flag. The touch pressure fluctuation sequence is composed of pressure values ​​collected by a touch pressure sensor at a preset sampling rate, arranged in chronological order. Each element in the touch pressure fluctuation sequence includes pressure values, a sampling timestamp, and the screen coordinates of the touch point. The path fractal dimension calculation is performed on the gaze-free path sequence to obtain the gaze intent determinism index, and the wave singular spectrum analysis is performed on the touch pressure fluctuation sequence to obtain the touch intent stability index. The pre-built intent understanding model is invoked to perform an index competition game on the gaze intent certainty index and the touch intent stability index to generate the user's true intent identifier corresponding to the current interaction round. Based on the user's true intent identifier, the corresponding interactive response behavior chain is dynamically extracted from the preset intent response topology network, and the interactive response behavior chain is compiled to generate an interactive response instruction and sent to the target application system to trigger the corresponding interactive feedback operation. The process of calculating the path fractal dimension of the gaze-free path sequence to obtain the gaze intent determinism index, and the process of performing fluctuation singular spectrum analysis on the touch pressure fluctuation sequence to obtain the touch intent stability index, includes: The gaze-free path sequence is processed to perform gaze-start point calibration, and the coordinates of the projection start point from the gaze-free path sequence are identified so that the user's gaze is first projected onto the screen area corresponding to each ambiguous intent resolution option, and the gaze projection start time corresponding to each ambiguous intent resolution option is recorded. The gaze-free path sequence is processed by gaze-free trajectory tracking, and the gaze-free trajectory point coordinate sequence formed by the continuous gaze-free movement of the user's gaze on the screen after the projection start point coordinates are extracted from the gaze-free path sequence. The coordinate sequence of the line-of-sight wandering trajectory points is processed by grid coverage counting, and the coordinate sequence of the line-of-sight wandering trajectory points is mapped to a grid coordinate system of a preset scale. The number of grid cells covered by the coordinate sequence of the line-of-sight wandering trajectory points is counted as the first coverage counting parameter. The coordinate sequence of the line-of-sight wandering trajectory points is subjected to grid scaling processing. The size of the grid cell of the grid coordinate system is reduced according to a preset scaling ratio to generate a scaled grid coordinate system. The number of grid cells covered by the line-of-sight wandering trajectory point coordinate sequence in the scaled grid coordinate system is recounted as the second coverage count parameter. The fractal dimension approximation of the line-of-sight free trajectory point coordinate sequence is calculated based on the first coverage count parameter, the second coverage count parameter, and the preset scaling ratio, and the fractal dimension approximation is used as the line-of-sight free trajectory fractal dimension parameter corresponding to the line-of-sight free path sequence. The fractal dimension parameter of the gaze-free trajectory is negatively mapped, and the gaze intent certainty index is calculated by substituting the fractal dimension parameter of the gaze-free trajectory into the negative correlation mapping function. The gaze intent certainty index is negatively correlated with the fractal dimension parameter of the gaze-free trajectory. The touch pressure fluctuation sequence is processed by pressure start point detection. The touch pressure start value generated when the user's finger first touches the screen is extracted from the touch pressure fluctuation sequence, and the pressure start time corresponding to the touch pressure start value is recorded. The touch pressure fluctuation sequence is divided into pressure fluctuation intervals according to the initial value of the touch pressure. The touch pressure fluctuation sequence is divided into pressure rising fluctuation subsequence, pressure maintaining fluctuation subsequence, and pressure falling fluctuation subsequence. An upward trend fitting process is performed on the pressure rise fluctuation subsequence, the difference between adjacent touch pressure values ​​in the pressure rise fluctuation subsequence is calculated, and all difference values ​​are summed and then divided by the total time span of the pressure rise fluctuation subsequence to generate the average pressure rise rate parameter. The pressure maintenance fluctuation subsequence is processed by maintenance stability calculation. The standard deviation of all touch pressure values ​​in the pressure maintenance fluctuation subsequence is calculated, and the reciprocal of the standard deviation is used as the pressure maintenance stability parameter. The pressure drop fluctuation subsequence is fitted with a downward trend. The absolute value of the difference between adjacent touch pressure values ​​in the pressure drop fluctuation subsequence is calculated. All absolute values ​​of the difference are summed and then divided by the total time span of the pressure drop fluctuation subsequence to generate the average pressure drop rate parameter. A touch pressure fluctuation feature vector is constructed based on the average rate of pressure rise, the pressure stability parameter, and the average rate of pressure fall parameter. The touch pressure fluctuation feature vector is then input into a preset singular spectrum analysis model for singular value decomposition to generate at least one singular value component. Extract the principal singular component with the largest energy proportion from the at least one singular component, and use the amplitude of the principal singular component as the stability index of the touch intent; The process of invoking a pre-built intent understanding model to perform an index competition game on the gaze intent certainty index and the touch intent stability index generates the user's true intent identifier corresponding to the current interaction round, including: The gaze intent certainty index is input into the gaze index encoder of the intent understanding model for index feature embedding processing to generate a gaze certainty feature vector. The gaze certainty feature vector includes a numerical code corresponding to the gaze intent certainty index and a path morphology code corresponding to the gaze free path sequence. The touch intent stability index is input into the touch index encoder of the intent understanding model for index feature embedding processing to generate a touch stability feature vector. The touch stability feature vector includes a numerical code corresponding to the touch intent stability index and a wave pattern code corresponding to the touch pressure wave sequence. The exponential competitive game module of the intent understanding model is invoked to perform zero-sum game calculation on the gaze deterministic feature vector and the touch stability feature vector to generate a gaze-touch game payoff matrix. The gaze-touch game payoff matrix is ​​used to represent the payoff distribution relationship between the gaze deterministic feature vector and the touch stability feature vector under different strategy combinations. The gaze deterministic feature vector is subjected to a first round of strategy adjustment based on the gaze touch game payoff matrix. The feature dimensions in the gaze deterministic feature vector that conflict with the touch stability feature vector are corrected according to the preset first game coefficient to generate the gaze feature vector after the initial game. The touch stability feature vector is subjected to a first round of strategy adjustment based on the gaze-touch game payoff matrix. The feature dimensions in the touch stability feature vector that conflict with the gaze deterministic feature vector are corrected according to the preset second game coefficient to generate the touch feature vector after the initial game. The visual feature vector and the touch feature vector after the initial game are input into the Nash equilibrium solution layer of the exponential competitive game module for equilibrium strategy solution processing, generating a hybrid strategy Nash equilibrium vector, which contains the equilibrium strategy probability value corresponding to each of the ambiguity intention resolution options. The hybrid strategy Nash equilibrium vector is subjected to maximum value indexing and positioning processing. The index position corresponding to the element with the largest value is extracted from the hybrid strategy Nash equilibrium vector. Based on the index position, the corresponding ambiguous intent resolution option is determined from the ambiguous intent resolution option set as the user's true intent identifier.

2. The intelligent interactive intent understanding method based on multimodal fusion according to claim 1, characterized in that, The step of dynamically extracting the corresponding interactive response behavior chain from the preset intent response topology network based on the user's true intent identifier, compiling the interactive response behavior chain to generate an interactive response instruction and sending it to the target application system to trigger the corresponding interactive feedback operation includes: Parse the intent node code contained in the user's true intent identifier, and locate the starting intent node corresponding to the intent node code from the preset intent response topology network; Starting from the initial intent node, a depth-first traversal is performed according to the preset directed edge connection relationship in the intent response topology network to generate at least one candidate response path from the initial intent node to the final intent node. Obtain the system resource usage status parameters corresponding to the current interaction round, including the CPU idle rate, available memory capacity, and remaining network bandwidth. Based on the CPU idle rate, available memory capacity, and remaining network bandwidth, resource consumption is estimated for each candidate response path to calculate the total system resources expected to be consumed during the execution of each candidate response path. Select the candidate response path with the minimum expected total system resource consumption from at least one candidate response path as the target interactive response behavior chain; Perform node operation parsing processing on each intent node in the target interactive response behavior chain, and extract the corresponding operation type code and operation parameter list from each intent node; According to the order of appearance of intent nodes in the target interaction response behavior chain, the operation type code and operation parameter list corresponding to each intent node are arranged sequentially to generate an operation instruction sequence; The operation instruction sequence is input into a preset instruction compiler for instruction compilation processing, and the operation instruction sequence is converted into a binary instruction code stream that can be recognized by the target application system, generating the instruction code stream part of the interactive response instruction; Extract the service discovery protocol identifier and service access endpoint address of the target application system from the preset network configuration parameters, and establish a remote procedure call connection with the target application system based on the service discovery protocol identifier and the service access endpoint address; The instruction code stream portion of the interactive response command is encapsulated into the message body payload area of ​​the remote procedure call protocol, and a call identifier field and a call timeout field are added to the header of the message body to generate a remote procedure call request message to be sent. The remote procedure call request message is sent to the target application system through the established remote procedure call connection, and a call timeout timer is started to wait for the remote procedure call response message returned by the target application system; Receive the remote procedure call response message returned by the target application system, parse the call return status code and call return data from the remote procedure call response message, and write the call return status code and call return data into the interaction log record.

3. The intelligent interactive intent understanding method based on multimodal fusion according to claim 1, characterized in that, The step of calculating the path fractal dimension of the gaze-free path sequence to obtain the gaze intent determinism index, and performing fluctuation singular spectrum analysis on the touch pressure fluctuation sequence to obtain the touch intent stability index, further includes: The gaze-wandering path sequence is subjected to gaze regression detection processing to identify regression events in the gaze-wandering path sequence in which the user's gaze wanders from the screen area corresponding to the current ambiguous intent resolution option to the screen area corresponding to other ambiguous intent resolution options and then returns to the screen area corresponding to the current ambiguous intent resolution option. The number of gaze regressions corresponding to each ambiguous intent resolution option is recorded. Based on the number of gaze regressions corresponding to each of the ambiguous intent resolution options, the gaze intent certainty index corresponding to each of the ambiguous intent resolution options is subjected to regression decay adjustment processing. The gaze intent certainty index corresponding to each of the ambiguous intent resolution options is multiplied by a regression decay coefficient that is negatively correlated with the number of gaze regressions to generate the regression-adjusted gaze intent certainty index corresponding to each of the ambiguous intent resolution options. The gaze intent certainty index is updated based on the regression-adjusted gaze intent certainty index corresponding to each of the aforementioned ambiguity intent resolution options; The touch pressure fluctuation sequence is subjected to pressure mutation detection processing to identify pressure mutation points in the touch pressure fluctuation sequence where the change in touch pressure value exceeds a preset mutation threshold per unit time, and the mutation time and mutation direction corresponding to each pressure mutation point are recorded. Based on the direction of the change corresponding to the pressure change point, the pressure change point is divided into positive pressure change point and negative pressure change point; The number of positive pressure mutation points and the number of negative pressure mutation points in the touch pressure fluctuation sequence are counted. The absolute value of the difference between the number of positive pressure mutation points and the number of negative pressure mutation points is calculated. The absolute value of the difference is divided by the total number of positive and negative pressure mutation points to generate a pressure mutation direction imbalance parameter. The touch intent stability index is adjusted by applying an imbalance penalty based on the imbalance parameter of the pressure change direction. The touch intent stability index is then multiplied by an imbalance penalty coefficient that is negatively correlated with the imbalance parameter of the pressure change direction to generate the imbalance-adjusted touch intent stability index. The touch intent stability index is updated based on the imbalance-adjusted touch intent stability index.

4. The intelligent interactive intent understanding method based on multimodal fusion according to claim 1, characterized in that, The step of invoking a pre-built intent understanding model to perform an index competition game on the gaze intent certainty index and the touch intent stability index to generate the user's true intent identifier corresponding to the current interaction round also includes: Obtain the historical interaction trust parameter corresponding to the current interaction round, wherein the historical interaction trust parameter includes the number of times the user and the target application system successfully interacted and the number of times they failed in the historical interaction rounds before the current interaction round; The historical interaction success rate parameter is calculated based on the ratio of the number of successful interactions to the sum of the number of successful interactions and the number of failed interactions. The trust weighting coefficient is calculated based on the historical interaction success rate parameter, and the first game coefficient and the second game coefficient are adjusted using the trust weighting coefficient to generate the adjusted first game coefficient and the adjusted second game coefficient. Using the adjusted first game coefficient, the gaze feature vector after the initial game is subjected to a second round of strategy adjustment processing according to the gaze touch game payoff matrix. The feature dimensions in the gaze feature vector after the initial game that have strategy conflicts with the touch stability feature vector are subjected to strategy correction processing to generate the gaze feature vector after the second game. Using the adjusted second game coefficient, the second round of strategy adjustment processing is performed on the touch feature vector after the initial game according to the gaze-touch game payoff matrix. The feature dimensions in the touch feature vector after the initial game that have strategy conflicts with the gaze deterministic feature vector are corrected by strategy processing to generate the touch feature vector after the second game. The post-secondary game gaze feature vector and the post-secondary game touch feature vector are input into the Nash equilibrium solution layer of the exponential competition game module for secondary equilibrium strategy solution processing to generate a secondary mixed strategy Nash equilibrium vector. The maximum value indexing process is performed on the Nash equilibrium vector of the quadratic mixing strategy. The index position corresponding to the element with the largest value is extracted from the Nash equilibrium vector of the quadratic mixing strategy. Based on the index position, the corresponding ambiguous intent resolution option is determined from the ambiguous intent resolution option set as the user's true intent identifier.

5. The intelligent interactive intent understanding method based on multimodal fusion according to claim 1, characterized in that, After generating the user's true intent identifier corresponding to the current interaction round, the method further includes: The user's true intent identifier is associated with the gaze-free path sequence and touch pressure fluctuation sequence corresponding to the current interaction round and stored to generate the game record of the current interaction round. Add the current round of game record to the historical game record set, and update the historical game record set; Extract the historical gaze-free path sequence and historical touch pressure fluctuation sequence corresponding to all historical interaction rounds that are identical to the user's true intent identifier from the updated historical game record set; The path fractal dimension average calculation is performed on the historical gaze-free path sequence corresponding to all historical interaction rounds. The arithmetic mean of the fractal dimension parameter of the historical gaze-free trajectory corresponding to each of the ambiguous intent resolution options is calculated, and the arithmetic mean is processed by the negative correlation mapping function to generate the historical average gaze intent determinism index. Perform fluctuation singular spectrum averaging analysis on the historical touch pressure fluctuation sequence corresponding to all historical interaction rounds, calculate the arithmetic mean of the amplitude of the historical principal singular value component corresponding to each of the ambiguous intent resolution options, and generate the historical average touch intent stability index. The historical average gaze intent certainty index and the historical average touch intent stability index are input into the index competition game module of the intent understanding model for index competition game processing to generate the user's true intent identifier after historical game. The user's true intention identifier after the historical game is compared with the user's true intention identifier. If the user's true intention identifier after the historical game is inconsistent with the user's true intention identifier, a game model calibration trigger signal is generated. The game model calibration trigger signal calls the parameter update module of the intent understanding model. The historical gaze-free path sequence and historical touch pressure fluctuation sequence corresponding to all historical interaction rounds are used as training data to fine-tune and update the model parameters of the exponential competition game module of the intent understanding model.

6. The intelligent interactive intent understanding method based on multimodal fusion according to claim 1, characterized in that, The step of dynamically extracting the corresponding interactive response behavior chain from the preset intent response topology network based on the user's real intent identifier, compiling the interactive response behavior chain to generate an interactive response instruction and sending it to the target application system to trigger the corresponding interactive feedback operation, further includes: Extract at least two candidate interactive response behavior chains associated with the user's true intent identifier from the preset intent response topology network, with each candidate interactive response behavior chain corresponding to a different response complexity level; Obtain the user cognitive load parameters corresponding to the current interaction round. The user cognitive load parameters include the continuous interaction duration of the user before the current interaction round, the number of operation errors of the user in the current interaction round, and the response delay duration of the user in the current interaction round. A user cognitive load feature vector is constructed based on the continuous interaction duration, the number of operation errors, and the response delay duration. The user cognitive load feature vector is then input into a preset cognitive load assessment model for load level prediction processing to generate a user's current cognitive load level identifier. Select the candidate interaction response behavior chain that matches the user's current cognitive load level identifier from the at least two candidate interaction response behavior chains as the target interaction response behavior chain; The target interactive response behavior chain is decoupled by splitting it into at least two sub-behavior chains that can be executed in parallel. Resource contention analysis is performed on the at least two parallel executable sub-behavior chains to detect whether there is a resource contention conflict during the execution of the at least two parallel executable sub-behavior chains; If the resource contention conflict is detected, the execution order of the at least two parallel executable sub-behavior chains is adjusted to a serial execution order, and the adjusted sub-behavior chain execution sequence is generated. If no resource contention conflict is detected, the at least two parallel executable sub-behavior chains are marked as a set of parallel executable sub-behavior chains; According to the adjusted sub-behavior chain execution sequence or the parallel execution sub-behavior chain set, each intent node in the target interactive response behavior chain is parsed, and the corresponding operation type code and operation parameter list are extracted from each intent node. According to the execution order of the intent nodes in the adjusted sub-behavior chain execution sequence or the parallel execution sub-behavior chain set, the operation type code and operation parameter list corresponding to each intent node are arranged sequentially to generate an optimized operation instruction sequence.

7. A smart interactive intent understanding system based on multimodal fusion, characterized in that, The method includes a processor and a computer-readable storage medium storing machine-executable instructions that, when executed by the processor, implement the intelligent interactive intent understanding method based on multimodal fusion as described in any one of claims 1-6.

Images