Intelligent display terminal multi-modal interaction method and system

Abstract

The application discloses a kind of wisdom display terminal multimodal interaction method and system, it is related to wisdom display terminal interaction technical field, including the sensor array of touch sensor, voice collector, image sensor and eye movement tracker composition, real-time acquisition touch position data, voice instruction signal, continuous hand action video sequence and user line of sight coordinate sequence;Each kind of data is converted and extracted processing with pertinence;With terminal current application state and scene classification model determine interaction scene category and corresponding modal priority rule;With the data after processing and modal priority rule synchronous input multimodal fusion intent recognition model, generate final interaction intent instruction;Control terminal execution operation and update application state.The method improves the accuracy of interaction intent recognition and interaction continuity, applicable to wisdom display terminal interaction scene.

Description

Technical Field

[0001] This invention belongs to the field of smart display terminal interaction technology, specifically a multimodal interaction method and system for smart display terminals. Background Technology

[0002] Currently, most multimodal interaction technologies for smart display terminals employ a combination of single or a few input modes. After collecting user input data through corresponding sensors, these modes are directly fused to identify the user's interaction intent. Common solutions utilize touch sensors to collect touch data and voice sensors to collect voice signals. Some solutions also add image sensors to collect hand gestures. This data is then simply superimposed and input into a recognition model to generate interaction commands and control the terminal to execute corresponding operations.

[0003] Existing technologies lack targeted transformation and extraction processing for the collected multi-dimensional input data, performing only basic collection and simple integration. This results in insufficient accuracy and structure of the input data, affecting the accuracy of subsequent interaction intent recognition. Furthermore, existing technologies do not differentiate between scenarios based on the current application state of the terminal, nor do they set corresponding modal priority rules. They simply fuse various types of input data, leading to a low degree of matching between the intent recognition results and the user's actual needs. In addition, existing technologies do not synchronously update the terminal's current application state information after controlling the terminal to perform an operation, causing subsequent interactions to fail to adapt to the latest state, resulting in interaction gaps and inconsistent responses. Summary of the Invention

[0004] This invention aims to solve at least one of the technical problems existing in the prior art;

[0005] Therefore, this invention proposes a multimodal interaction method for a smart display terminal, comprising:

[0006] Multi-dimensional user input data is collected in real time by a sensor array set on a smart display terminal. The sensor array includes a touch sensor, a voice collector, an image sensor, and an eye tracker. The multi-dimensional user input data includes touch position data, voice command signals, continuous hand movement video sequences, and user gaze coordinate sequences.

[0007] The touch position data is converted into a set of screen coordinates, the voice command signal is converted into voice text information, dynamic gesture trajectory is extracted from the continuous hand action video sequence, and the gaze area and gaze dwell time are identified from the user gaze coordinate sequence.

[0008] The system obtains the current application status information of the smart display terminal, determines the interaction scenario category of the current user based on the current application status information and a preset scenario classification model, and queries the modal priority rule corresponding to the interaction scenario category.

[0009] The screen coordinate set, voice and text information, dynamic gesture trajectory, gaze area, gaze dwell time, and the determined modal priority rules are synchronously input into the pre-trained multimodal fusion intent recognition model to generate the current user's final interaction intent command.

[0010] Based on the final interaction intent instruction, the smart display terminal is controlled to perform the corresponding operation and update the current application status information.

[0011] Further, extracting dynamic gesture trajectories from the continuous hand movement video sequence includes:

[0012] Hand key point detection is performed on each frame of the continuous hand movement video sequence to locate the pixel position coordinates of multiple hand joints in each frame;

[0013] The pixel position coordinates of the same hand joint in multiple consecutive frames are tracked over time to form the movement path of each hand joint.

[0014] Based on the movement paths of all hand joints, the motion vector sequence of the hand as a whole in three-dimensional space is calculated, and the motion vector sequence includes translational motion components and rotational motion components.

[0015] Identify continuous motion segments that conform to predetermined start point, end point, and path characteristics from the motion vector sequence; the continuous motion segments constitute the dynamic gesture trajectory.

[0016] Add timestamp information and confidence score to the dynamic gesture trajectory.

[0017] Further, identifying the gaze region and gaze duration from the user's gaze coordinate sequence includes:

[0018] The user's gaze coordinate sequence is denoised and smoothed to eliminate physiological tremors and coordinate jumps caused by blinking during eye movements;

[0019] The movement of the gaze coordinates within a predetermined time window is calculated. When the movement is less than the stationary determination threshold, the user's gaze is determined to enter a fixation state, and the moment the fixation begins is recorded.

[0020] Calculate the average of all gaze coordinates under the gaze state to obtain the coordinates of the gaze center point;

[0021] The coordinates of the gaze center point are mapped to the screen coordinate system of the smart display terminal, and combined with the layout of the interface elements on the screen, the interface functional area in which the coordinates of the gaze center point fall is determined, and the interface functional area is identified as the gaze area.

[0022] The timer continues from the start of the gaze state until the movement of the gaze exceeds the stationary threshold. The end time is recorded, and the time from the start time to the end time is calculated to obtain the gaze dwell time.

[0023] Furthermore, based on the current application state information and a preset scene classification model, the current user's interaction scene category is determined, and the modal priority rule corresponding to the interaction scene category is queried, including:

[0024] The current application status information includes the identifier of the application running in the foreground, the current layout of the application's interface, and the set of operation instructions allowed by the application.

[0025] Extract feature vectors from the current application state information. The feature vectors include application identifier encoding, current operation interface layout encoding, active window position information, current interface element interaction hot zone distribution information, and system background noise level.

[0026] The feature vector is input into the scene classification model, and the scene classification model outputs the category probability distribution corresponding to the user interaction scene. The category with the highest probability is selected from the category probability distribution as the interaction scene category. The interaction scene categories include meeting presentation scene, audio-visual entertainment scene, document editing scene, system settings scene, and leisure browsing scene.

[0027] Query the modality priority rule library pre-bound to the interaction scenario category to obtain the corresponding modality priority rule. The modality priority rule defines the confidence weight coefficient and activation condition for different input modalities under the corresponding interaction scenario category.

[0028] Furthermore, the modality priority rule defines the confidence weight coefficients and activation conditions corresponding to different input modalities under the corresponding interaction scenario category, including:

[0029] For conference presentation scenarios, the modal priority rule assigns the highest confidence weight coefficient to the voice command signal modality and the dynamic gesture trajectory modality, sets the lowest confidence weight coefficient to the touch position data modality, and sets the activation condition of eye tracking data to be used only for auxiliary selection when a predetermined wake-up gesture is detected.

[0030] The eye-tracking data refers to the gaze area and gaze duration identified and processed from the user's gaze coordinate sequence collected by the eye tracker.

[0031] For document editing scenarios, the modal priority rule assigns the highest confidence weight coefficient to the touch position data modality and the eye tracking data modality, and the activation condition for the voice command signal modality is set to be recognized only when a voice command is input within a specific functional area of ​​the document editing application;

[0032] For audio-visual entertainment scenarios, the modal priority rule assigns the highest confidence weight coefficient to the dynamic gesture trajectory modality and the voice command signal modality, and the activation condition of the touch position data modality is set to only accept touch operations within a specific control bar area of ​​the screen.

[0033] The confidence weight coefficient and the activation condition retrieved from the modality priority rule base will be passed to the multimodal fusion intent recognition model.

[0034] Furthermore, the screen coordinate set, speech-text information, dynamic gesture trajectory, gaze region, gaze dwell time, and the determined modal priority rules are synchronously input into a pre-trained multimodal fusion intent recognition model to generate the current user's final interaction intent command, including:

[0035] Based on the activation conditions in the modal priority rules, the screen coordinate set, voice and text information, dynamic gesture trajectory, gaze area, and gaze dwell time are filtered to block input data that does not meet the activation conditions.

[0036] For input data that meets the activation conditions, a corresponding fusion weight is assigned to the input data of each mode according to the confidence weight coefficient in the modality priority rule.

[0037] The multimodal input data, after being assigned fusion weights, is input to the feature encoding layer of the multimodal fusion intent recognition model. The feature encoding layer encodes the input data of each modality into a feature vector of a uniform dimension.

[0038] The feature vectors of the same dimension from different modalities are weighted and concatenated according to their corresponding fusion weights to form a multimodal fusion feature vector;

[0039] The multimodal fusion feature vector is input into the intent decision layer of the multimodal fusion intent recognition model. The intent decision layer outputs the intent recognition result corresponding to the current moment. The intent recognition result is mapped to the final interactive intent instruction that the smart display terminal can execute.

[0040] Furthermore, for the input data that meets the activation conditions, according to the confidence weight coefficient in the modality priority rule, a corresponding fusion weight is assigned to the input data of each modality, including:

[0041] From the modality priority rules, read the preset baseline confidence weight coefficient for each input modality;

[0042] The environmental interference assessment value of the current environment is obtained, which is calculated based on the real-time environmental noise level and light intensity collected by the sensor array;

[0043] Based on the environmental interference assessment value, the baseline confidence weight coefficient is dynamically adjusted. For input modes affected by the current environmental interference assessment value, the baseline confidence weight coefficient is lowered to form the adjusted real-time confidence weight coefficient.

[0044] The real-time confidence weight coefficients are normalized so that the sum of the weight coefficients of all activated modes is a constant value. The normalized coefficients are used as the fusion weights for feature fusion.

[0045] Further, the multimodal input data, after being assigned fusion weights, is input to the feature encoding layer of the multimodal fusion intent recognition model. The feature encoding layer encodes the input data of each modality into a feature vector of a uniform dimension, including:

[0046] For the set of screen coordinates of the touch position data mode, a position sequence encoder is used to process the data, extract the spatial distribution pattern of the touch points and the click time interval features, and output the first feature vector.

[0047] For the voice text information of the voice command signal mode, a natural language semantic encoder is used to process it, extract the semantic vector of the text and the command keywords, and output the second feature vector;

[0048] For the dynamic gesture trajectory modality, the dynamic gesture trajectory is processed by an action sequence encoder to extract the kinematic features and trajectory shape features of the gesture and output a third feature vector;

[0049] For the gaze region and gaze dwell time of the eye-tracking data modality, a visual attention encoder is used to process them, extract the semantic importance of the gaze region and the interest feature based on the dwell time, and output a fourth feature vector;

[0050] The first feature vector, the second feature vector, the third feature vector, and the fourth feature vector have the same dimension.

[0051] Furthermore, it also includes the step of online optimization of the multimodal fusion intent recognition model after generating the final interactive intent command:

[0052] After executing the final interactive intent instruction, obtain the user's feedback information, which includes whether the user has undone the operation or repeated the same intent operation through other input modalities within a preset time.

[0053] If the feedback information indicates that the user has undone or repeated the operation, the intent recognition result is determined to be inaccurate, and the multimodal input data, the assigned fusion weights, the current application state information, and the intent recognition result are marked as training samples to be corrected.

[0054] A batch of training samples to be corrected is periodically collected, compared with the correct intention instruction annotations, and the model prediction loss is calculated.

[0055] The model predicts the loss, and the parameters of the multimodal fusion intent recognition model are fine-tuned through backpropagation. The weight parameters of the feature encoding layer and the intent decision layer are updated to achieve online self-optimization of the model.

[0056] Furthermore, the present invention also includes a smart display terminal multimodal interaction system, the system including a memory, a processor, and a computer program stored in the memory and running on the processor, wherein when the processor executes the computer program, it implements the steps of the smart display terminal multimodal interaction method described above.

[0057] Compared with the prior art, the beneficial effects of the present invention are:

[0058] This system performs targeted transformation and extraction processing on various multi-dimensional user input data collected by sensor arrays. It converts touch position data into a set of screen coordinates, converts voice command signals into voice text information, extracts dynamic gesture trajectories from continuous hand movement video sequences, and identifies the gaze area and gaze dwell time from the user's gaze coordinate sequence. This achieves accurate and structured processing of input data, avoiding the data chaos and insufficient accuracy caused by conventional multimodal interaction that only performs simple collection of input data without specific transformation and extraction. It makes the input data more consistent with the needs of subsequent intent recognition, improves the accuracy of interactive intent recognition, and reduces recognition bias.

[0059] Based on the current application status information of the smart display terminal and a preset scene classification model, the user's interaction scene category is determined and the corresponding modal priority rule is matched. Then, the processed input data of each type and the modal priority rule are synchronously input into a pre-trained multimodal fusion intent recognition model to generate the final interaction intent command. Simultaneously, the terminal's current application status information is updated after the corresponding operation is executed. This approach avoids the problems of low intent recognition matching with user needs caused by conventional multimodal interactions that do not distinguish between scenes, lack modal priorities, and simply fuse data. It makes intent recognition more closely aligned with the current interaction scene, while forming a complete interaction loop, avoiding gaps in subsequent interactions, and improving the continuity and smoothness of the interaction. Attached Figure Description

[0060] Figure 1 This is a flowchart illustrating the steps of a multimodal interaction method for a smart display terminal according to the present invention.

[0061] Figure 2 A flowchart for identifying fixation regions and fixation duration;

[0062] Figure 3 Contribution analysis diagram of feature vectors for interactive scene classification;

[0063] Figure 4 Heatmaps of modal priority weights for different interaction scenarios;

[0064] Figure 5 This is a time-series graph showing the change in the number of training samples to be corrected. Detailed Implementation

[0065] The technical solution of the present invention will be clearly and completely described below with reference to the embodiments. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0066] See Figure 1The smart display terminal is equipped with a sensor array, including touch sensors, a voice acquisition unit, an image sensor, and an eye tracker. These sensors operate continuously, collecting touch position data, voice command signals, continuous hand movement video sequences, and user gaze coordinate sequences in real time, forming multi-dimensional user input data. Touch position data is converted into a set of coordinates in the screen coordinate system. Voice command signals are processed by a speech recognition engine and converted into speech-text information. Dynamic gesture trajectories representing user hand movements are extracted from the continuous hand movement video sequences. The user's current primary gaze area and the gaze dwell time in that area are identified from the user gaze coordinate sequence. Simultaneously, the system acquires the current application status information of the smart display terminal, including foreground applications and interface layout. Based on the current application status information and a pre-set scene classification model, the system determines the user's current interaction scene category and then queries the modal priority rule corresponding to that interaction scene category. Subsequently, the screen coordinate set, speech-text information, dynamic gesture trajectory, gaze area, gaze dwell time, and the queried modal priority rule are simultaneously input into a pre-trained multi-modal fusion intent recognition model. The model comprehensively analyzes all input information and outputs an instruction corresponding to the user's final interaction intent. Based on this final interaction intent instruction, the system controls the smart display terminal to perform the corresponding operation, and updates the current application status information after the operation is completed to start the next round of interaction.

[0067] In one embodiment of the present invention, when extracting a dynamic gesture trajectory from a continuous video sequence of hand movements, hand keypoint detection is performed on each frame of the video sequence to locate the pixel coordinates of multiple hand joints in each frame. The pixel coordinates of the same hand joint in multiple consecutive frames are tracked over time to form the movement path of each hand joint. Based on the movement paths of all hand joints, a motion vector sequence of the hand as a whole in three-dimensional space is calculated. This motion vector sequence includes translational and rotational motion components. From the calculated motion vector sequence, a continuous action segment that conforms to predetermined start-point, end-point, and path characteristics is identified. This identified continuous action segment constitutes the desired dynamic gesture trajectory. Finally, timestamp information and a confidence score are added to this dynamic gesture trajectory.

[0068] In practical implementation, hand keypoint detection captures continuous hand movement video sequences using image sensors deployed on smart display terminals. These sequences consist of frames arranged chronologically. A hand keypoint detection algorithm is run on each frame of the sequence to locate the pixel coordinates of multiple hand joints within each frame. These pixel coordinates include, but are not limited to, the two-dimensional coordinates of the wrist and fingertips. Temporal tracking is then performed on the pixel coordinates of the same hand joint across multiple frames. This tracking is achieved using optical flow or association matching algorithms to form the movement path of each hand joint. The movement path of a hand joint is a time-varying coordinate sequence of the hand joint in the image coordinate system, representing its local motion in three-dimensional space. The image coordinate system is established with the top-left corner of a single frame in a continuous hand movement video sequence as the origin, the horizontal axis pointing to the right as the positive x-axis, and the vertical axis pointing downwards as the positive y-axis, using image pixels as the basic unit. This system is used to locate the hand joints within the video frame. The screen coordinate system is established with the top-left corner of the smart display terminal screen as the origin, the horizontal axis pointing to the right as the positive x-axis, and the vertical axis pointing downwards as the positive y-axis, using screen pixels as the basic unit. The image coordinate system and the screen coordinate system are transformed using the image sensor calibration parameters of the smart display terminal. The motion vector sequence of the hand as a whole in three-dimensional space is calculated based on the movement paths of all hand joints. This calculation involves mapping from two-dimensional image coordinates to three-dimensional spatial coordinates. One implementation method involves recovering the three-dimensional spatial position of the hand joints through multi-view visual geometry or by combining depth sensor data. The translational motion component in the motion vector sequence is defined by the difference between the three-dimensional spatial positions of the hand joints at adjacent time stamps, and the rotational motion component is defined by the Euler angle changes of the rigid skeleton formed by the hand keypoints at adjacent time stamps. The formula for calculating the motion vector sequence is:

[0069]

[0070] in: Indicates at time The overall translational motion component of the hand. Indicates at time The overall rotational motion component of the hand. It is the total number of hand joints involved in the calculation. It is the first Each hand joint point at time Three-dimensional spatial coordinates, function This represents a function that calculates rotational changes from two sets of hand joint points;

[0071] Two sets of hand joints are defined as follows: each set consists of the three-dimensional spatial coordinates of all hand joints detected at adjacent time points t and t-1. All joints of the same hand are grouped into one set at the same time point, and the two sets at adjacent time points are matched one-to-one. The function f is a rotation matrix solution function based on rigid skeleton transformation. By performing singular value decomposition on the two sets of hand joints, the rotation angle and direction of the hand as a whole in three-dimensional space are obtained, which are used to characterize the rotational motion components of the hand as a whole.

[0072] The system identifies continuous motion segments from motion vector sequences that conform to predetermined start, end, and path features. These features are loaded from a predefined gesture template library during system initialization. The recognition process involves dynamically time-warping and matching the real-time calculated motion vector sequence with reference motion sequences in the gesture template library. When the similarity exceeds a set threshold, the successfully matched motion vector subsequence is identified as a continuous motion segment, constituting a dynamic gesture trajectory. A timestamp and confidence score are appended to the dynamic gesture trajectory. The timestamp records the start and end times of the continuous motion segment, and the confidence score is derived from the similarity score obtained through the aforementioned dynamic time-warping matching. The dynamic gesture trajectory, along with its accompanying timestamp and confidence score, will be output as part of the multimodal input data for subsequent multimodal fusion intent recognition.

[0073] In one embodiment of the present invention, when identifying the gaze region and gaze duration from the user's gaze coordinate sequence, see [reference needed]. Figure 2 First, the original user gaze coordinate sequence is denoised and smoothed to eliminate coordinate jumps caused by physiological tremors and blinking during eye movements. The movement amplitude of the processed gaze coordinates within a predetermined time window is calculated. When this movement amplitude is less than a preset static determination threshold, the user's gaze is determined to have entered a fixation state, and the start time of fixation is recorded. The average value of all gaze coordinates in the fixation state is calculated to obtain the coordinates of the fixation center point. The coordinates of the fixation center point are mapped to the screen coordinate system of the smart display terminal, and combined with the current layout of interface elements on the screen, the specific interface functional area where the fixation center point falls is determined, and this interface functional area is marked as the fixation area. Timing continues from the start of the fixation state until the gaze movement amplitude exceeds the static determination threshold, at which point the end time is recorded. The duration from the start time to the end time is calculated; this duration is the fixation dwell time.

[0074] In practical implementation, the user's gaze coordinate sequence undergoes denoising and smoothing processing. This sequence consists of the original gaze point coordinates output by the eye tracker at a fixed sampling frequency. The denoising and smoothing process employs a time-window-based moving average filter or a Kalman filter. The moving average filter takes the arithmetic mean of multiple original gaze point coordinates within the window, while the Kalman filter predicts and corrects coordinates based on a gaze motion model. The filter processing eliminates physiological tremors and coordinate jumps caused by blinking during eye movements, outputting a smoothed gaze coordinate sequence. The movement amplitude of the processed gaze coordinates within a predetermined time window is calculated; the length of this predetermined time window is configurable. The movement amplitude is obtained by calculating the sum of the Euclidean distances between all consecutive gaze point coordinates within the predetermined time window. When the calculated movement amplitude is less than a pre-set stillness determination threshold, the system determines that the user's gaze has entered a fixation state and records the determination time as the fixation start time. The stillness determination threshold is calibrated based on the physical size of the display screen and the viewing distance. The average value of all gaze coordinates in the fixation state is calculated to obtain the gaze center point coordinates, which are two-dimensional planar coordinates. The formula is:

[0075]

[0076] in: Indicates the coordinates of the gaze center point. This represents the total number of valid gaze coordinates collected within the time period determined to be a gaze state. and They represent the first The system calculates the x and y coordinates of each valid gaze center point. The gaze center point coordinates are then mapped to the screen coordinate system of the smart display terminal, defined in pixels. The mapping process combines the screen resolution of the smart display terminal and the calibration parameters of the eye tracker for coordinate transformation. The transformed gaze center point coordinates are compared with the layout of interface elements on the screen. This comparison is performed by traversing the bounding box coordinates of all interactive functional areas on the current screen interface. When the gaze center point coordinates fall within the bounding box of a certain interface functional area, that interface functional area is identified as the gaze area. A timer is maintained internally from the start of the gaze state. When the real-time calculated gaze movement exceeds the stationary threshold, the gaze state is considered to have ended, and this moment is recorded as the gaze end time. The duration from the start of the gaze to the end of the gaze is calculated in milliseconds; this calculated duration is the gaze persistence time. It is understandable that if the user's gaze does not leave the current gaze area during the timing process but a brief, minor movement occurs with an amplitude less than the static determination threshold, the coordinates during the minor movement will still participate in the iterative update of the gaze center point coordinates, but the timer will not be reset.

[0077] In some embodiments, the denoising and smoothing process employs a Kalman filter, where the state vector of the Kalman filter includes the gaze coordinates and their velocity, and the observation vector is the original output coordinates of the eye tracker. In other embodiments, the predetermined time window length is set to 200 to 300 milliseconds, and the stillness determination threshold is set to one to two percent of the screen width. Optionally, the gaze center point coordinates can be calculated using a weighted average method after removing the largest and smallest outliers. Optionally, after the gaze center point coordinates are mapped to the screen coordinate system, if the coordinate point does not fall within any predefined interface functional area bounding box, the gaze area is identified as the screen background area. It can be understood that the length of gaze dwell time is an input feature for inferring the user's interest in the interface functional area.

[0078] In one embodiment of the present invention, when determining the interaction scenario category based on the current application state information and a preset scenario classification model, the current application state information includes the application identifier running in the foreground, the current operation interface layout of the application, and the set of operation instructions allowed by the application. The preset scenario classification model is constructed by collecting application state data, interface layout data, user interaction behavior data, and environmental noise data from a smart display terminal in five typical interaction scenarios: conference presentation, audio-visual entertainment, document editing, system settings, and leisure browsing. A labeled sample set is constructed and trained using a multilayer perceptron network under supervised conditions. After training, the model is solidified into a static model file that can be directly accessed and stored in the local storage area of ​​the smart display terminal. Feature vectors are extracted from this current application state information. These feature vectors include the application identifier code, the current operation interface layout code, active window position information, the distribution information of the current interface element interaction hotspots, and the system background noise level. This feature vector is input into the scenario classification model, which outputs a probability distribution of the corresponding user interaction scenario category. The category with the highest probability is selected from this probability distribution as the final determined interaction scenario category. The interaction scenario categories include conference presentation, audio-visual entertainment, document editing, system settings, and leisure browsing. The modal priority rule library, pre-bound to the determined interaction scenario category, is then queried to obtain the corresponding modal priority rules. These rules define the confidence weight coefficients and activation conditions for different input modalities within the corresponding interaction scenario category. For conference presentation scenarios, the modal priority rules assign the highest confidence weight coefficients to the voice command signal modality and dynamic gesture trajectory modality, while the confidence weight coefficient for the touch position data modality is set to the lowest. Furthermore, the activation condition for eye-tracking data is set to be used only when a predetermined wake-up gesture is detected. For document editing scenarios, the modal priority rules assign the highest confidence weight coefficients to the touch position data modality and eye-tracking data modality, and the activation condition for the voice command signal modality is set to be recognized only when voice commands are input within a specific functional area of ​​the document editing application. For audio-visual entertainment scenarios, the modal priority rules assign the highest confidence weight coefficients to the dynamic gesture trajectory modality and voice command signal modality, and the activation condition for the touch position data modality is set to be accepted only when touch operations are performed within a specific control bar area of ​​the screen. The confidence weight coefficients and activation conditions retrieved from the modality priority rule base will be passed to the multimodal fusion intent recognition model.

[0079] In practical implementation, the current application status information of the smart display terminal is obtained. This current application status information is a dynamic data set during system runtime. It includes the foreground application identifier, the current user interface layout, and the allowed operation command set. The application identifier is the application's unique package name or process ID. The current user interface layout is view tree data describing the position and hierarchy of elements such as buttons, text boxes, and images on the interface. The allowed operation command set is a list of user operations supported by the current interface, read from the application programming interface or a predefined configuration file.

[0080] Feature vectors are extracted from the current application state information. These feature vectors are numerical representations used as input to the scene classification model. The extracted feature vectors include application identifier encoding, current interface layout encoding, active window position information, current interface element interaction hotspot distribution information, and system background noise level. The application identifier encoding maps the identifier to a fixed-length one-hot encoded vector by looking up a pre-defined application dictionary. The current interface layout encoding recursively traverses the view tree and transforms element type and position parameters into a one-dimensional sequence. The active window position information is the coordinate range of the foreground application window on the screen. The current interface element interaction hotspot distribution information is a spatial heatmap of the areas where clickable or focusable elements cluster on the screen. The system background noise level is a decibel value calculated from the audio signal collected by a voice collector. The feature vectors are then input to the scene classification model, which is a classification neural network pre-trained using labeled application state data and scene category data. After receiving the feature vectors, the scene classification model outputs a category probability distribution for the corresponding user interaction scene at the output layer. This category probability distribution is a vector, where each element represents the probability that the input belongs to a predefined interaction scene category. The category with the highest probability is selected as the interaction scenario category from the category probability distribution. The interaction scenario categories include meeting presentation scenarios, audio-visual entertainment scenarios, document editing scenarios, system settings scenarios, and leisure browsing scenarios. The selection process is completed by taking the maximum value using the argmax operation.

[0081] The system retrieves the corresponding modal priority rules from a pre-bound modal priority rule library for each interaction scenario category. This library is a database or configuration file stored locally on the device or in the cloud. Modal priority rules define the confidence weight coefficients and activation conditions for different input modalities within the corresponding interaction scenario category. The confidence weight coefficients are weighted values ​​used for subsequent multimodal fusion, and the activation conditions are Boolean logic expressions that determine whether the corresponding modal input data participates in the current intent recognition. In practice, the modal priority rules differ across interaction scenario categories. For example, in a conference presentation scenario, the highest confidence weight coefficients are assigned to the voice command signal modality and the dynamic gesture trajectory modality, while the lowest confidence weight coefficient is set for the touch position data modality. Furthermore, the activation condition for eye-tracking data is set to be used only when a predetermined wake-up gesture is detected. In a document editing scenario, the highest confidence weight coefficients are assigned to the touch position data modality and the eye-tracking data modality, and the activation condition for the voice command signal modality is set to be recognized only when voice commands are input within a specific functional area of ​​the document editing application. In audio-visual entertainment scenarios, the modal priority rule assigns the highest confidence weight coefficients to dynamic gesture trajectory modalities and voice command signal modalities, and the activation condition for touch position data modalities is set to only accept touch operations within a specific control bar area on the screen. In some embodiments, the generation of feature vectors includes an encoding process, the formula of which is expressed as:

[0082]

[0083] in: This represents the final generated feature vector. Represents the feature encoding function. to These represent different sub-items of current application state information, such as application identifier and current interface layout. Encoding function This can be a multilayer perceptron or a convolutional neural network. The confidence weight coefficients and activation conditions retrieved from the modality priority rule base are passed to the multimodal fusion intent recognition model via an internal message passing mechanism or shared memory variables. Modality priority rules can be understood as a combination of static configuration and dynamic scene classification results, providing a basis for scene adaptation in subsequent fusion recognition. Optionally, the modality priority rule base is a pre-built configuration database based on human-computer interaction experiments and scene adaptation requirements. Each rule in the base is bound to a corresponding interaction scene category, specifying the confidence weight coefficients and activation conditions for each input modality. The rule base is pre-installed in the smart display terminal system partition in the form of a structured file, directly read and called during use, and can be modified through system updates or user customization. Optionally, the scene classification model uses a hybrid structure combining convolutional neural networks and long short-term memory networks to process sequential interface layout encoding. In some embodiments, the set of operation instructions allowed by the application is also included as part of the feature vector in the encoding, used to distinguish the interaction scenes of different functional pages within the same application. It is understandable that the system background noise level, as an environmental feature, helps to distinguish between scenarios requiring high speech recognition confidence and scenarios in noisy environments.

[0084] See Figure 3 This is a feature vector contribution analysis chart for interactive scenarios, showing the contribution distribution of five types of interactive scenarios across five core feature dimensions. The values ​​range from 0 to 0.3, with higher values ​​indicating a stronger influence of the feature on scenario classification. The meeting presentation scenario shows prominent application identifiers and strong active window position features, indicating that meeting scenarios are primarily distinguished by application type and window layout. The audio-visual entertainment scenario shows the most significant application identifiers and environmental noise features, consistent with the audio-visual scenario's sensitivity to the audio environment. The document editing scenario has the highest proportion of interface layout and window position features, reflecting the unique interface structure of the document editing scenario. The system settings scenario shows prominent window position and interactive hotspot features, strongly correlated with the fixed layout and interactive areas of the system settings interface. The casual browsing scenario shows the most significant interactive hotspot and window position features, consistent with the user's frequent clicking behavior in the browsing scenario.

[0085] In one embodiment of the present invention, when screen coordinate sets, voice and text information, dynamic gesture trajectories, gaze regions, gaze dwell times, and modality priority rules are synchronously input into a multimodal fusion intent recognition model, the input data of each modality is first filtered according to the activation conditions in the modality priority rules, blocking input data that does not meet the activation conditions. For input data that meets the activation conditions, a corresponding fusion weight is assigned to the input data of each modality according to the confidence weight coefficient in the modality priority rules. When assigning fusion weights to each modality, the preset baseline confidence weight coefficient for each input modality is read from the modality priority rules, and the current environmental interference assessment value calculated based on the real-time environmental noise level and illumination intensity collected by the sensor array is obtained. According to the environmental interference assessment value, the baseline confidence weight coefficient is dynamically adjusted. For input modalities affected by the current environmental interference assessment value, their baseline confidence weight coefficient is lowered, forming the adjusted real-time confidence weight coefficient. The real-time confidence weight coefficients are normalized to ensure that the sum of the weight coefficients for all activated modalities is a constant. These normalized coefficients serve as the fusion weights for feature fusion. The multimodal input data, weighted by fusion, is then fed into the feature encoding layer of the multimodal fusion intent recognition model. The feature encoding layer encodes the input data for each modality into a feature vector of a uniform dimension. For the screen coordinate set of the touch position data modality, a position sequence encoder is used to extract the spatial distribution pattern of the touch points and the click time interval features, outputting the first feature vector. For the speech text information of the voice command signal modality, a natural language semantic encoder is used to extract the semantic vector of the text and command keywords, outputting the second feature vector. For the dynamic gesture trajectory modality, an action sequence encoder is used to extract the kinematic features and trajectory shape features of the gesture, outputting the third feature vector. For the gaze region and gaze dwell time of the eye-tracking data modality, a visual attention encoder is used to extract the semantic importance of the gaze region and the interest feature based on dwell time, outputting the fourth feature vector. The first, second, third, and fourth feature vectors have the same dimension. These feature vectors with the same dimension from different modalities are weighted and concatenated according to their corresponding fusion weights to form a multimodal fusion feature vector. This multimodal fusion feature vector is input into the intent decision layer of the multimodal fusion intent recognition model. The intent decision layer outputs the intent recognition result corresponding to the current moment, which is mapped to the final interactive intent command that the smart display terminal can execute.

[0086] In practice, the screen coordinate set, speech-text information, dynamic gesture trajectory, gaze region, and gaze dwell time are filtered according to the activation conditions in the modality priority rules. The activation conditions are Boolean logic judgment statements from the modality priority rules. The system parses the activation conditions defined for each input modality in the modality priority rules. The activation conditions may contain logical judgments on the attributes of the input data itself or the current application state. When the input data of a certain modality meets its corresponding activation condition, the input data is marked as valid and passed to subsequent steps. Input data that does not meet the activation condition is masked, meaning that the input data of that modality will not participate in subsequent fusion and decision-making in this recognition cycle.

[0087] For input data that meets the activation conditions, a corresponding fusion weight is assigned to the input data of each modality according to the confidence weight coefficient in the modality priority rules. The preset baseline confidence weight coefficient for each input modality is read from the modality priority rules; this baseline confidence weight coefficient is the initial weight value set for each modality in different scenarios in the rule base. The environmental interference assessment value of the current environment is obtained. This assessment value is calculated based on the real-time environmental noise level and illumination intensity collected by the sensor array. The environmental noise level is determined by the background volume of the voice acquisition unit during periods without user voice input, and the illumination intensity is determined by the ambient brightness detected by the image sensor or ambient light sensor. The baseline confidence weight coefficient is dynamically adjusted according to the environmental interference assessment value, following a preset interference-weight mapping relationship. For input modalities affected by the current environmental interference assessment value, their baseline confidence weight coefficient is lowered. For example, high environmental noise may lead to a lower baseline confidence weight coefficient for the voice command signal modality, and low illumination conditions may lead to a lower baseline confidence weight coefficient for the vision-based dynamic gesture trajectory modality, resulting in the adjusted real-time confidence weight coefficient. The real-time confidence weight coefficients are normalized so that the sum of the weight coefficients for all activated modes is a constant value. The normalized coefficients are then used as the fusion weights for feature fusion. Table 1 shows an example of adjusting the baseline confidence weight coefficients based on different environmental interference evaluation values ​​in a conference presentation scenario.

[0088] Table 1: Dynamic Adjustment Table of Fusion Weights under Different Environmental Conditions in Conference Presentation Scenarios

[0089]

[0090] In practice, the formula for normalization is:

[0091]

[0092] in: It is the first Normalized fusion weights for each activation mode, It is the first Real-time confidence weight coefficients for each activated mode after environmental interference assessment. It is the total number of currently active input modes. This is the sum of the real-time confidence weight coefficients of all activated modalities. The multimodal input data, weighted by fusion, is then fed into the feature encoding layer of the pre-trained multimodal fusion intent recognition model. This layer contains multiple parallel sub-encoder networks. The pre-trained multimodal fusion intent recognition model uses labeled interaction data from four modalities as training samples. It combines scene category labels and modal priority weights output by a scene classification model, employing an eye-tracking data modality architecture and a fully connected layer network for pre-training. After pre-training, the data is stored locally on the terminal and can be directly loaded and run. For the screen coordinate set of the touch position data modality, a position sequence encoder is used. This encoder is a recurrent neural network or a self-attention network, extracting the spatial distribution pattern of touch points and click time interval features, and outputting the first feature vector. For the speech text information of the speech command signal modality, a natural language semantic encoder is used. This encoder is a pre-trained language model, such as BERT or a Transformer-like model, extracting the semantic vector of the text and command keywords, and outputting the second feature vector. For dynamic gesture trajectories in the dynamic gesture trajectory modality, an action sequence encoder (ASE), which is a temporal convolutional network or a long short-term memory network, is used to extract the kinematic features and trajectory shape features of the gesture, outputting a third feature vector. For the gaze region and gaze dwell time in the eye-tracking data modality, a visual attention encoder (VAE), which is a fully connected neural network, is used to extract the semantic importance of the gaze region and interest features based on dwell time, outputting a fourth feature vector. The first, second, third, and fourth feature vectors are mapped to the same dimension through the projection layer at the end of the encoder network. The feature vectors from different modalities with the same dimension are weighted and concatenated according to their corresponding fusion weights to form a multimodal fusion feature vector. The weighted concatenation is performed by multiplying each feature vector by its corresponding normalized fusion weight and then performing a vector concatenation operation. The multimodal fusion feature vector is input into the intent decision layer of the multimodal fusion intent recognition model. The intent decision layer consists of one or more fully connected layers and a Softmax classifier. The intent decision layer outputs the intent recognition result corresponding to the current moment. The intent recognition result is a probability distribution vector. The category with the highest probability is mapped to the operation instruction code that the smart display terminal can execute. This operation instruction code is the final interactive intent instruction.

[0093] In some embodiments, the environmental interference assessment value is calculated using a linear combination formula, which sums the normalized environmental noise level and illumination intensity value multiplied by a preset interference coefficient. Optionally, the sub-encoders in the feature encoding layer are jointly trained during the model pre-training phase to learn the intermodal feature representations. It is understood that the weighted concatenation operation makes the modal features with higher confidence weights have a greater impact on the final decision during subsequent intent decision-making. In some embodiments, if the input data of all modalities are masked by activation conditions, the system maintains the previous valid intent state or outputs an empty instruction. Optionally, the normalized fusion weights are applied before the input feature vector, or equivalently scaled on the weighted concatenated fusion feature vector. It is understood that the operation instruction encoding output by the intent decision layer needs to match the instruction set defined by the application programming interface of the smart display terminal system.

[0094] See Figure 4 This is a heatmap showing the modal priority weights across different interaction scenarios. It visually illustrates the priority weight allocation of four input modalities across five interaction scenarios. Darker colors represent higher weights, reflecting a scenario-adaptive multimodal interaction strategy. In meeting presentations and audio-visual entertainment scenarios, voice and gestures are central, meeting the needs of contactless interaction and avoiding frequent screen contact. In document editing scenarios, touch and eye tracking dominate, ensuring operational accuracy and cursor positioning efficiency. In system settings and casual browsing scenarios, modal weights are more balanced, adapting to complex or diverse interactive behaviors. This heatmap clearly presents the scenario-modal adaptation relationship, providing an intuitive basis for weight allocation in multimodal fusion intent recognition models. It can be directly used to guide modal weighting logic in model training and real-time inference, improving the scenario adaptability and accuracy of interactive intent recognition.

[0095] In one embodiment of the present invention, the method includes the step of online optimization of a multimodal fusion intent recognition model after generating a final interactive intent instruction. After executing the final interactive intent instruction, user feedback information is obtained, including whether the user has undone an operation or repeated the same intent operation through other input modalities within a preset time. If the feedback information indicates that the user has undone an operation or repeated an operation, the intent recognition result is determined to be inaccurate, and the current multimodal input data, assigned fusion weights, current application state information, and intent recognition result are marked as training samples to be corrected. A batch of training samples to be corrected is periodically collected and compared with the correct intent instruction annotations to calculate the model prediction loss. Using the calculated model prediction loss, the parameters of the multimodal fusion intent recognition model are fine-tuned through backpropagation, updating the weight parameters of the feature encoding layer and the intent decision layer, thereby achieving online self-optimization of the model.

[0096] In practice, the online optimization step begins after the final interactive intent command is executed, and the system enters a feedback monitoring cycle. User feedback is obtained by monitoring subsequent user input operations received by the smart display terminal within a preset time window. Feedback includes whether the user has undone an operation or repeated the same intent operation using other input modalities within the preset time. The length of the preset time window is configurable, for example, set to 2 seconds or 5 seconds after the final interactive intent command is executed. An undone operation is a clear cancellation or backtracking command input by the user via touch, voice, or gesture, such as clicking the undone button, saying "cancel," or making a specific undone gesture. A repeated operation is when the user issues the same or semantically equivalent operation command again within the preset time using an input modality different from the one that triggered the final interactive intent command. For example, the system generates and executes the final interactive intent command based on the voice command "Open File A," and then the user clicks the "Open File A" icon again via touch within the preset time. If the feedback indicates that the user has undone or repeated an operation, the system determines that the intent recognition result is inaccurate. The determination logic is rule-based; detecting any feedback condition triggers an inaccuracy determination. The multimodal input data, assigned fusion weights, current application state information, and intent recognition result are labeled as training samples to be corrected. The multimodal input data includes the original screen coordinate set, speech and text information, dynamic gesture trajectories, gaze region, and gaze dwell time. The assigned fusion weights are normalized weights actually used for feature fusion. The current application state information includes the application identifier that triggered this recognition, the interface layout, and other information. The intent recognition result is the intent category that the multimodal fusion intent recognition model outputs in this instance and is judged as inaccurate. These data are packaged into a single data sample, labeled with a tag indicating an error in this recognition, and stored in a local temporary cache queue.

[0097] A batch of training samples to be corrected is collected periodically. This periodic collection is triggered by a fixed sample size threshold or a fixed time interval, such as whenever 100 training samples accumulate in the cache queue, or every hour. These samples are then compared with the correct intent instruction annotation, which needs to be inferred indirectly from subsequent user actions or explicitly specified. In the case of a user performing an undo operation, the correct intent instruction annotation can be inferred as "no operation" or maintaining the previous state. When the user repeatedly performs an operation through other input modalities, the correct intent instruction annotation is the explicit instruction corresponding to this repeated operation. The model prediction loss is calculated using a loss function, which measures the difference between the multimodal fusion intent recognition model's predicted output for this batch of training samples to be corrected and the correct intent instruction annotation. Loss function. The calculation formula is:

[0098]

[0099] in: This represents the model prediction loss for the current batch. This indicates the size of the currently collected batch of training samples to be corrected. It is the index of the sample in the batch. This indicates the first inference based on user feedback. The correct intent command annotation for each sample This indicates that the multimodal fusion intent recognition model is for the first... Each sample is predicted as follows The probability of each category. Loss function. The larger the value, the greater the deviation between the model's current prediction and the correct intent indicated by user feedback.

[0100] The parameters of the multimodal fusion intent recognition model are fine-tuned using backpropagation based on the gradient descent algorithm, utilizing the model prediction loss. (Model prediction loss) The gradients of the trainable parameters of the multimodal fusion intent recognition model are calculated, and the calculated gradients are used to update the model weights. Specifically, the update operation applies to the weight parameters of the feature encoding layer and the intent decision layer of the multimodal fusion intent recognition model. The feature encoding layer includes the network parameters of the position sequence encoder, natural language semantic encoder, action sequence encoder, and visual attention encoder. The intent decision layer includes the network parameters of its fully connected layers and classifier. Parameter updates are performed at a relatively small learning rate to achieve minor adjustments to the model parameters and avoid disrupting the model's original pre-trained knowledge. The online self-optimization process does not require prolonged service downtime and can be performed incrementally in the background. After the model parameters are updated, the new multimodal fusion intent recognition model will immediately take effect in subsequent interactive recognition.

[0101] In some embodiments, the length of the preset time window can be dynamically adjusted according to different final interaction intent command types. Shorter windows can be set for operations with clear follow-up feedback, such as opening files and sending messages, while longer windows can be set for continuous operations such as scrolling pages and adjusting volume. Optionally, when marking training samples to be corrected, the system simultaneously records environmental interference evaluation values, which can serve as reference data for subsequent analysis of the model's performance in specific environments. It is understood that periodic batch processing helps balance real-time learning and computational overhead, avoiding significant impacts on system real-time performance. In some embodiments, the loss function can employ cross-entropy loss combined with label smoothing techniques to mitigate potential noise in the correct intent command annotations inferred from user feedback. Optionally, backpropagation fine-tuning uses a stochastic gradient descent optimizer with momentum to stabilize the online learning process. It is understood that online self-optimization enables the multimodal fusion intent recognition model to adapt to users' personal habits and interaction patterns in specific environments.

[0102] See Figure 5 This is a time-series graph showing the fluctuation of the number of training samples to be corrected over 24 hours, reflecting the distribution characteristics of user feedback samples to be corrected over time during the online optimization of the multimodal interaction model. The data exhibits a clear 5-hour cycle, with the number of samples first rapidly rising to a peak within each cycle, then falling back to a trough, consistent with the time-series patterns of user interaction behavior. The highest number occurs around 23 hours, approximately 92; peaks in other cycles remain stable between 67 and 86. The lowest number occurs around 9 hours, with 3; troughs in other cycles range from 8 to 17. The number of samples oscillates between 3 and 92 within 24 hours, without a significant long-term upward or downward trend, indicating that the number of samples to be corrected during online model optimization is in a stable fluctuation state. The absence of a sustained surge in the number of samples suggests that the online optimization strategy is effective, promptly correcting erroneous intent recognition and preventing the accumulation of incorrect samples.

[0103] The above embodiments are only used to illustrate the technical methods of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical methods of the present invention without departing from the spirit and scope of the technical methods of the present invention.

Claims

1. A multimodal interaction method for a smart display terminal, characterized in that, include: Multi-dimensional user input data is collected in real time by a sensor array set on a smart display terminal. The sensor array includes a touch sensor, a voice collector, an image sensor, and an eye tracker. The multi-dimensional user input data includes touch position data, voice command signals, continuous hand movement video sequences, and user gaze coordinate sequences. The touch position data is converted into a set of screen coordinates, the voice command signal is converted into voice text information, dynamic gesture trajectory is extracted from the continuous hand action video sequence, and the gaze area and gaze dwell time are identified from the user gaze coordinate sequence. The system acquires the current application status information of the smart display terminal. Based on the current application status information and a preset scene classification model, it determines the current user's interaction scene category and queries the modal priority rules corresponding to the interaction scene category. This includes: the current application status information including the application identifier running in the foreground, the current operation interface layout of the application, and the allowed operation instruction set of the application; extracting feature vectors from the current application status information, including application identifier encoding, current operation interface layout encoding, active window position information, current interface element interaction hotspot distribution information, and system background noise level; inputting the feature vectors into the scene classification model, which outputs a category probability distribution corresponding to the user's interaction scene; selecting the category with the highest probability from the category probability distribution as the interaction scene category, including meeting presentation scene, audio-visual entertainment scene, document editing scene, system settings scene, and leisure browsing scene; and querying the modal priority rule library pre-bound to the interaction scene category to obtain the corresponding modal priority rules, which define the confidence weight coefficients and activation conditions corresponding to different input modalities under the corresponding interaction scene category. The screen coordinate set, voice and text information, dynamic gesture trajectory, gaze area, gaze dwell time, and the determined modal priority rules are synchronously input into the pre-trained multimodal fusion intent recognition model to generate the current user's final interaction intent command. Based on the final interaction intent instruction, the smart display terminal is controlled to perform the corresponding operation and update the current application status information.

2. The multimodal interaction method for a smart display terminal according to claim 1, characterized in that, Extracting dynamic gesture trajectories from the continuous hand movement video sequence includes: Hand key point detection is performed on each frame of the continuous hand movement video sequence to locate the pixel position coordinates of multiple hand joints in each frame; The pixel position coordinates of the same hand joint in multiple consecutive frames are tracked over time to form the movement path of each hand joint. Based on the movement paths of all hand joints, the motion vector sequence of the hand as a whole in three-dimensional space is calculated, and the motion vector sequence includes translational motion components and rotational motion components. Identify continuous motion segments that conform to predetermined start point, end point, and path characteristics from the motion vector sequence; the continuous motion segments constitute the dynamic gesture trajectory. Add timestamp information and confidence score to the dynamic gesture trajectory.

3. The multimodal interaction method for a smart display terminal according to claim 2, characterized in that, Identifying the gaze region and gaze duration from the user's gaze coordinate sequence includes: The user's gaze coordinate sequence is denoised and smoothed to eliminate physiological tremors and coordinate jumps caused by blinking during eye movements; The movement of the gaze coordinates within a predetermined time window is calculated. When the movement is less than the stationary determination threshold, the user's gaze is determined to enter a fixation state, and the moment the fixation begins is recorded. Calculate the average of all gaze coordinates under the gaze state to obtain the coordinates of the gaze center point; The coordinates of the gaze center point are mapped to the screen coordinate system of the smart display terminal, and combined with the layout of the interface elements on the screen, the interface functional area in which the coordinates of the gaze center point fall is determined, and the interface functional area is identified as the gaze area. The timer continues from the start of the gaze state until the movement of the gaze exceeds the stationary threshold. The end time is recorded, and the time from the start time to the end time is calculated to obtain the gaze dwell time.

4. The multimodal interaction method for a smart display terminal according to claim 3, characterized in that, The modality priority rule defines the confidence weight coefficients and activation conditions for different input modalities under the corresponding interaction scenario category, including: For conference presentation scenarios, the modal priority rule assigns the highest confidence weight coefficient to the voice command signal modality and the dynamic gesture trajectory modality, sets the lowest confidence weight coefficient to the touch position data modality, and sets the activation condition of eye tracking data to be used only for auxiliary selection when a predetermined wake-up gesture is detected. The eye-tracking data refers to the gaze area and gaze duration identified and processed from the user's gaze coordinate sequence collected by the eye tracker. For document editing scenarios, the modal priority rule assigns the highest confidence weight coefficient to the touch position data modality and the eye tracking data modality, and the activation condition for the voice command signal modality is set to be recognized only when a voice command is input within a specific functional area of ​​the document editing application; For audio-visual entertainment scenarios, the modal priority rule assigns the highest confidence weight coefficient to the dynamic gesture trajectory modality and the voice command signal modality, and the activation condition of the touch position data modality is set to only accept touch operations within a specific control bar area of ​​the screen. The confidence weight coefficient and the activation condition retrieved from the modality priority rule base will be passed to the multimodal fusion intent recognition model.

5. The multimodal interaction method for a smart display terminal according to claim 4, characterized in that, The screen coordinate set, speech and text information, dynamic gesture trajectory, gaze region, gaze dwell time, and the determined modal priority rules are synchronously input into a pre-trained multimodal fusion intent recognition model to generate the current user's final interaction intent command, including: Based on the activation conditions in the modal priority rules, the screen coordinate set, voice and text information, dynamic gesture trajectory, gaze area, and gaze dwell time are filtered to block input data that does not meet the activation conditions. For input data that meets the activation conditions, a corresponding fusion weight is assigned to the input data of each mode according to the confidence weight coefficient in the modality priority rule. The multimodal input data, after being assigned fusion weights, is input to the feature encoding layer of the multimodal fusion intent recognition model. The feature encoding layer encodes the input data of each modality into a feature vector of a uniform dimension. The feature vectors of the same dimension from different modalities are weighted and concatenated according to their corresponding fusion weights to form a multimodal fusion feature vector; The multimodal fusion feature vector is input into the intent decision layer of the multimodal fusion intent recognition model. The intent decision layer outputs the intent recognition result corresponding to the current moment. The intent recognition result is mapped to the final interactive intent instruction that the smart display terminal can execute.

6. The multimodal interaction method for a smart display terminal according to claim 5, characterized in that, For input data that meets the activation conditions, according to the confidence weight coefficient in the modality priority rule, a corresponding fusion weight is assigned to the input data of each modality, including: From the modality priority rules, read the preset baseline confidence weight coefficient for each input modality; The environmental interference assessment value of the current environment is obtained, which is calculated based on the real-time environmental noise level and light intensity collected by the sensor array; Based on the environmental interference assessment value, the baseline confidence weight coefficient is dynamically adjusted. For input modes affected by the current environmental interference assessment value, the baseline confidence weight coefficient is lowered to form the adjusted real-time confidence weight coefficient. The real-time confidence weight coefficients are normalized so that the sum of the weight coefficients of all activated modes is a constant value. The normalized coefficients are used as the fusion weights for feature fusion.

7. The multimodal interaction method for a smart display terminal according to claim 6, characterized in that, The multimodal input data, after being assigned fusion weights, is input to the feature encoding layer of the multimodal fusion intent recognition model. The feature encoding layer encodes the input data of each modality into a feature vector of a uniform dimension, including: For the set of screen coordinates of the touch position data mode, a position sequence encoder is used to process the data, extract the spatial distribution pattern of the touch points and the click time interval features, and output the first feature vector. For the voice text information of the voice command signal mode, a natural language semantic encoder is used to process it, extract the semantic vector of the text and the command keywords, and output the second feature vector; For the dynamic gesture trajectory modality, the dynamic gesture trajectory is processed by an action sequence encoder to extract the kinematic features and trajectory shape features of the gesture and output a third feature vector; For the gaze region and gaze dwell time of the eye-tracking data modality, a visual attention encoder is used to process them, extract the semantic importance of the gaze region and the interest feature based on the dwell time, and output a fourth feature vector; The first feature vector, the second feature vector, the third feature vector, and the fourth feature vector have the same dimension.

8. The multimodal interaction method for a smart display terminal according to claim 7, characterized in that, It also includes the step of online optimization of the multimodal fusion intent recognition model after generating the final interactive intent command: After executing the final interactive intent instruction, obtain the user's feedback information, which includes whether the user has undone the operation or repeated the same intent operation through other input modalities within a preset time. If the feedback information indicates that the user has undone or repeated the operation, the intent recognition result is determined to be inaccurate, and the multimodal input data, the assigned fusion weights, the current application state information, and the intent recognition result are marked as training samples to be corrected. A batch of training samples to be corrected is periodically collected, compared with the correct intention instruction annotations, and the model prediction loss is calculated. The model predicts the loss, and the parameters of the multimodal fusion intent recognition model are fine-tuned through backpropagation. The weight parameters of the feature encoding layer and the intent decision layer are updated to achieve online self-optimization of the model.

9. A multimodal interactive system for a smart display terminal, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the multimodal interaction method for a smart display terminal as described in any one of claims 1 to 8.

Images