Multi-modal task interaction assisting method and system and computer equipment

Abstract

The invention relates to the technical field of man-machine interaction, and discloses a multi-modal task interaction assisting method and system and computer equipment, and the method comprises the steps: collecting multi-modal interaction data of a user executing a multi-dimensional interaction task; determining a user behavior feature vector based on the multi-modal interaction data, and constructing a task completion quantitative model; determining the weight of each operation step by adopting a dynamic weight distribution strategy based on the multi-dimensional interaction task completion degree quantification model, and dynamically adjusting a preset adaptive coefficient by adopting a preset adjustment mechanism based on the multi-modal interaction data and the weight of each operation step; generating a dynamic task guiding strategy by adopting a dynamic path planning algorithm based on the weight of each operation step and a dynamically adjusted preset adaptive coefficient; and executing the dynamic task guiding strategy to obtain a multi-modal execution result. According to the method, a complete closed-loop process is formed from data acquisition, analysis, decision making to execution, intelligent services can be provided for middle-aged and elderly users, and the user experience and the task execution efficiency are improved.

Description

Technical Field

[0001] The present invention relates to the field of human-computer interaction technology, and in particular to a multimodal task interaction assistance method, system and computer equipment. Background Art

[0002] With the development of mobile internet, various applications (APPs) have gradually become popular among users of all ages. However, there are relatively few intelligent services for middle-aged and elderly users, especially in terms of task guidance. Specifically, there are:

[0003] Static task guidance: The task guidance of traditional apps is usually a fixed process and lacks dynamic adjustment capabilities.

[0004] Single feedback mode: It relies only on visual cues and does not combine multimodal feedback such as hearing and touch, which makes it difficult to adapt to the cognitive characteristics of middle-aged and elderly users.

[0005] Lack of age-friendly features: Age-friendly features such as font scaling and operational tolerance are not considered. The aforementioned shortcomings and lack of intelligent services lead to a poor user experience and low efficiency in completing interactive tasks for middle-aged and elderly users when interacting with the app. Summary of the Invention

[0006] In view of this, the present invention provides a multimodal task interaction assistance method, system and computer device to solve the problem that the lack of intelligent services leads to insufficient experience and low efficiency in completing interactive tasks for middle-aged and elderly people when interacting with APPs.

[0007] In a first aspect, the present invention provides a multimodal task interaction assistance method, the method comprising:

[0008] Collect multimodal interaction data of users performing multi-dimensional interaction tasks;

[0009] Determine user behavior feature vectors based on multimodal interaction data, and build a multi-dimensional interaction task completion quantification model based on the behavior feature vectors;

[0010] Based on the multi-dimensional interactive task completion quantification model, a dynamic weight allocation strategy is adopted to determine the weight of each operation step in the multi-dimensional interactive task, and a preset adjustment mechanism is used to dynamically adjust the preset adaptive coefficient based on the multimodal interaction data and the weight of each operation step;

[0011] Based on the weight of each operation step and the dynamically adjusted preset adaptive coefficient, a dynamic path planning algorithm is used to generate a dynamic task guidance strategy;

[0012] Execute dynamic task guidance strategies to obtain multimodal execution results.

[0013] In the present invention, by collecting multimodal interaction data, user information can be obtained from multiple dimensions, covering the user's operational behavior, environmental status, and interaction intentions. Compared with single-modal data collection, multimodal data collection more comprehensively reflects the user's true state when performing tasks, avoiding misjudgments caused by missing information. Based on multimodal interaction data, user behavior feature vectors are determined, and a multi-dimensional interactive task completion quantification model is constructed, achieving in-depth analysis of user behavior. The use of a dynamic weight allocation strategy and a preset adjustment mechanism gives the system dynamic optimization capabilities, allowing the system to flexibly adapt to different user ability levels, task scenarios, and environmental conditions based on changes during user operation. Based on the dynamically adjusted weights and adaptive coefficients, a dynamic path planning algorithm is used to generate a dynamic task guidance strategy, and the strategy is executed to obtain multimodal execution results, effectively improving task completion efficiency. The present invention forms a complete closed-loop process from data collection, analysis, decision-making to execution, which can provide intelligent services for middle-aged and elderly users, improve user experience and task execution efficiency, and solve the problem that the lack of intelligent services leads to insufficient experience and low efficiency of interactive task completion when interacting with apps for middle-aged and elderly people.

[0014] In an optional embodiment, the multimodal interaction data includes touch gesture recognition data, voice recognition data, and sensor data;

[0015] Determine user behavior feature vectors based on multimodal interaction data, including:

[0016] Based on touch gesture recognition data, voice recognition data and sensor data, a deep learning algorithm is used to extract user operation behavior patterns and habits, identify user operation preferences and abnormal operation behaviors, and obtain user behavior feature vectors. User behavior feature vectors include user operation habit features, operation proficiency features and error tendency features.

[0017] In the present invention, multimodal interaction data covers touch gesture recognition data, voice recognition data and sensor data, collecting user information from different dimensions and breaking the limitations of a single data source. The use of deep learning algorithms to process multimodal interaction data can mine the user behavior patterns hidden behind the data. The algorithm can automatically learn user operation behavior patterns and habits, and accurately identify user operation preferences through large-scale data training. The determined user behavior feature vector contains user operation habit features, operation proficiency features and error tendency features. Accurate feature extraction enables the system to provide personalized services for different users. This process can process multimodal interaction data in real time and dynamically update user behavior feature vectors. The system continuously monitors user operations and can respond quickly once changes in behavior patterns or abnormal operations are detected, providing a data basis for the subsequent construction of a hybrid habit strength model and a multi-dimensional interaction task completion quantification model.

[0018] In an optional embodiment, the multi-dimensional interactive task includes multiple operation steps;

[0019] A multi-dimensional interactive task completion quantification model is constructed based on user behavior feature vectors, including:

[0020] Based on the user behavior feature vector and multiple operation steps, a multi-dimensional interactive task completion metric model is constructed using a preset multi-dimensional task completion metric formula;

[0021] The formula for the quantification model of multi-dimensional interactive task completion is expressed as:

[0022] Completion_Index=Σ(Step_Weight[i]×Success_Rate[i])+λ×log(Retry_Count+1);

[0023] Among them, Step_Weight[i] represents the i-th weight, Success_Rate[i] represents the task completion success rate, λ×log(Retry_Count+1) represents the retry correction, λ represents the preset adaptive coefficient, and Retry_Count represents the retry behavior.

[0024] In this paper, a multidimensional interactive task completion metric model is constructed based on user behavior feature vectors and multiple operational steps using a pre-defined multidimensional task completion metric formula. The model's lambda (λ) can be dynamically adjusted to avoid over-penalizing user attempts while also rationally reflecting the impact of retries on task completion. By incorporating three key factors—task step weights, completion success rates, and retry behavior—the multidimensional interactive task completion metric model can more accurately and comprehensively quantify task completion, avoiding evaluation bias.

[0025] In an optional embodiment, the multimodal task interaction assistance method further includes:

[0026] Extract time features, frequency features and quality features from user behavior feature vectors;

[0027] The pre-trained model is trained based on time features, frequency features, and quality features until the preset convergence conditions are met to obtain a hybrid habit strength model;

[0028] The user behavior pattern is predicted based on the hybrid habit strength model to obtain a prediction result, and the hybrid habit strength model is dynamically optimized based on the relationship between the prediction result and a preset threshold.

[0029] In this invention, time, frequency, and quality features are extracted from user behavior feature vectors to analyze user behavior from a more diverse perspective. A pre-trained model is trained based on the extracted time, frequency, and quality features to produce a hybrid habit strength model. This targeted training approach enables the model to fully learn the key elements of user behavior habits. The trained model can more accurately assess user habit strength, providing more reliable results for determining user proficiency in a particular operation and predicting future behavior trends. User behavior patterns are predicted based on the hybrid habit strength model, and the model is dynamically optimized based on the relationship between the predicted results and a preset threshold. When the user habit strength assessment result exceeds the preset threshold, indicating a significant change in user behavior, the system promptly adjusts and optimizes the model. By assessing user habit strength using the hybrid habit strength model, the system can predict potential operational risks or changes in user needs. Risk prediction and proactive services based on habit strength assessment transform the system from a passive response to a proactive service, improving user convenience and security while also strengthening user trust in the system. From feature extraction to model training, evaluation, and dynamic optimization, a data-driven closed-loop process is formed.

[0030] In an optional embodiment, a dynamic weight allocation strategy is adopted based on a multi-dimensional interactive task completion quantification model to determine the weight of each operation step in the multi-dimensional interactive task, including:

[0031] Calculate the operation complexity score of each operation step based on the number of operations, the number of interaction types, and cognitive load of each operation step;

[0032] Calculate the overall task importance score of each operation step based on the user's task process dependency and failure impact of each operation step;

[0033] Based on the operational complexity score of each operation step and the importance score of the overall task, a dynamic weight allocation strategy is used to determine the weight of each operation step in the multi-dimensional interactive task;

[0034] The formula of dynamic weight allocation strategy is expressed as:

[0035] W[i]=(Complexity[i]×Importance[i]) / Σ(Complexity[j]×Importance[j]);

[0036] Where W[i] represents the weight of the i-th operation step, Complexity[i] represents the operation complexity score of the i-th operation step, and Importance[i] represents the importance score of the i-th operation step to the overall task.

[0037] In this invention, a dynamic weight allocation strategy comprehensively and scientifically determines the weight of each operation step based on the two core dimensions of operation complexity and importance. When calculating the operation complexity score, the number of user operations on each operation step, the number of interaction types, and cognitive load are comprehensively considered. The importance score is based on the user's dependence on the task flow of each operation step and the degree of impact of failure. Through this comprehensive evaluation of these two dimensions, the one-sidedness of weight determination is avoided, and the weight results are more consistent with the actual situation of the task, providing a scientific basis for subsequent task completion quantification and system decision-making.

[0038] In an optional embodiment, a preset adaptive coefficient is dynamically adjusted using a preset adjustment mechanism based on the multimodal interaction data and the weight of each operation step, including:

[0039] Calculate historical task completion rate, cognitive load index, and environmental adaptability index based on multimodal interaction data;

[0040] Calculate the adaptability factor based on historical task completion rate, cognitive load index and environmental adaptability index;

[0041] Generate a preset adaptive coefficient calculation model based on the adaptive factor and the weight of each operation step;

[0042] The preset adaptive coefficient is dynamically adjusted using a preset adjustment mechanism based on a preset adaptive coefficient calculation model; wherein the expression of the preset adaptive coefficient calculation model is:

[0043] λ=λ_base+λ_range×(1-exp(-k×Adaptive_Factor));

[0044] Among them, λ_base represents the basic penalty coefficient, λ_range represents the dynamic adjustment range, k represents the sensitivity coefficient, and Adaptive_Factor represents the adaptive factor. Its expression is:

[0045] AdaptiveFactor=w1×Success_Rate+w2×Cognitive_Load+w3×Environmental_Index;

[0046] Among them, Success_Rate represents the task completion success rate, Cognitive_Load represents the cognitive load index, Environmental_Index represents the environmental adaptability index, w1, w2 and w3 represent the task completion success rate weight, cognitive load index weight and environmental adaptability index weight respectively.

[0047] In this invention, a preset adjustment mechanism calculates historical task completion rates, cognitive load index, and environmental adaptability index based on multimodal interaction data. Leveraging the advantages of multimodal data, the adaptive factor is calculated by integrating historical task completion rates, cognitive load index, and environmental adaptability index. This is then combined with the weights of each operation step to generate a preset adaptive coefficient calculation model, enabling a multi-factor, accurate assessment of user task execution. The preset adjustment mechanism dynamically adjusts the preset adaptive coefficient based on real-time data during user operations, giving the system strong adaptability.

[0048] In an optional embodiment, a dynamic task guidance strategy is generated using a dynamic path planning algorithm based on the weight of each operation step and a dynamically adjusted preset adaptive coefficient, including:

[0049] Obtain the calculation results of the multi-dimensional interactive task completion quantification model and the preset requirements of the multi-dimensional interactive task;

[0050] Based on the calculation results of the multi-dimensional interactive task completion quantification model, the preset requirements of the multi-dimensional interactive task, the weight of each operation step and the dynamically adjusted preset adaptive coefficient, a dynamic path planning algorithm is used to generate a dynamic task guidance strategy.

[0051] In this invention, when generating a dynamic task guidance strategy, the calculation results of a multi-dimensional interactive task completion quantification model, the pre-set requirements for the multi-dimensional interactive task, the weights of each operation step, and the dynamically adjusted pre-set adaptive coefficients are comprehensively considered. Multi-source data complements and verifies each other. The introduction of operation step weights enables the dynamic task guidance strategy to focus on the key aspects of the task. By focusing on and optimizing key aspects, the efficiency and quality of task completion are significantly improved, and the risk of task execution is reduced.

[0052] In an optional implementation, executing a dynamic task guidance strategy to obtain a multimodal execution result includes:

[0053] Generate a user visual guidance enhancement plan based on the calculation results of the dynamic task guidance strategy and the multi-dimensional interactive task completion quantification model. The user visual guidance enhancement plan includes dynamic adjustment of the user device's interface elements, spacing, layout, interface contrast and brightness;

[0054] Based on the calculation results of the dynamic task guidance strategy and the multi-dimensional interactive task completion quantification model, task guidance prompts are generated when the user performs the multi-dimensional interactive task, and a task progress tracking plan is generated based on the task guidance prompts;

[0055] Generate a multimodal feedback control scheme after the user performs each operation step in the multidimensional interactive task;

[0056] The user visual guidance enhancement scheme, task progress tracking scheme and multimodal feedback control scheme are taken as multimodal execution results.

[0057] In the present invention, a user visual guidance enhancement scheme is generated based on the dynamic task guidance strategy and the calculation results of the multi-dimensional interactive task completion quantification model, which realizes the precise optimization of the user device interface, reduces the operational errors caused by visual impairments, and significantly improves the comfort and accuracy of user visual interaction. According to the dynamic task guidance strategy and the calculation results of the quantitative model, a task guidance prompt and task progress tracking scheme are generated to provide users with real-time and effective task execution support. After the user completes each operation step, a multimodal feedback control scheme is generated, and the user is informed of the operation results in a timely manner through various feedback forms such as vision, hearing, and touch, forming a complete interactive closed loop. The user visual guidance enhancement scheme, the task progress tracking scheme and the multimodal feedback control scheme are integrated into a multimodal execution result, which realizes the synergy and complementarity of each scheme and provides users with all-round task execution support.

[0058] In a second aspect, the present invention provides a multimodal task interaction assistance system, the system comprising:

[0059] The perception layer is used to collect multimodal interaction data of users performing multi-dimensional interactive tasks;

[0060] The decision layer is used to determine the user behavior feature vector based on the multimodal interaction data, and generate a multi-dimensional interactive task completion quantification model based on the behavior feature vector; based on the multi-dimensional interactive task completion quantification model, a dynamic weight allocation strategy is used to determine the weight of each operation step in the multi-dimensional interactive task, and a dynamic adjustment mechanism is used to dynamically adjust the preset retry correction coefficient based on the multimodal interaction data and the weight of each operation step; and a dynamic task guidance strategy is generated based on the weight of each operation step and the dynamically adjusted retry correction coefficient;

[0061] The execution layer is used to execute dynamic task guidance strategies and obtain multimodal execution results.

[0062] In a third aspect, the present invention provides a computer device comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the multimodal task interaction assistance method of the first aspect or any corresponding embodiment thereof by executing the computer instructions.

[0063] In a fourth aspect, the present invention provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the multimodal task interaction assistance method of the first aspect or any corresponding embodiment thereof.

[0064] In a fifth aspect, the present invention provides a computer program product comprising computer instructions for causing a computer to execute the multimodal task interaction assistance method of the first aspect or any corresponding embodiment thereof. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the specific embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0066] Figure 1 is a flowchart of a multimodal task interaction assistance method according to an embodiment of the present invention;

[0067] Figure 2 is a flowchart of another multimodal task interaction assistance method according to an embodiment of the present invention;

[0068] Figure 3 is a flowchart of another multimodal task interaction assistance method according to an embodiment of the present invention;

[0069] Figure 4 is a structural block diagram of a multimodal task interaction assistance system according to an embodiment of the present invention;

[0070] Figure 5 is a flow chart for constructing a hybrid habit strength model in a multimodal task interaction assistance method according to an embodiment of the present invention;

[0071] Figure 6 is a flow chart of generating a dynamic task guidance strategy in a multimodal task interaction assistance method according to an embodiment of the present invention;

[0072] Figure 7 is a flowchart of generating a user visual guidance enhancement scheme in a multimodal task interaction assistance method according to an embodiment of the present invention;

[0073] Figure 8 is a flowchart of a task progress tracking solution generated in a multimodal task interaction assistance method according to an embodiment of the present invention;

[0074] Figure 9 is a flow chart of generating a multimodal feedback control scheme in a multimodal task interaction assistance method according to an embodiment of the present invention;

[0075] Figure 10 is a structural block diagram of another multimodal task interaction assistance system according to an embodiment of the present invention;

[0076] Figure 11 Schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0077] To make the purpose, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making creative efforts shall fall within the scope of protection of the present invention.

[0078] According to an embodiment of the present invention, an embodiment of a multimodal task interaction assistance method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0079] In this embodiment, a multimodal task interaction assistance method is provided, which can be used in a multimodal task interaction assistance system, such as Figure 4 As shown in Figure 2, the multimodal task interaction assistance system includes a perception layer, a decision layer, and an execution layer. Figure 1 is a flow chart of a multimodal task interaction assistance method according to an embodiment of the present invention. Figure 1 As shown, the process includes the following steps:

[0080] Step S101: Collect multimodal interaction data of users performing multidimensional interaction tasks.

[0081] Specifically, the perception layer is used to collect multimodal interaction data including user physiological data, environmental perception data, and interaction behavior data when users perform multi-dimensional interaction tasks. Figure 4 As shown, the perception layer includes a gyroscope sensor, a touch gesture recognition module, and a speech recognition module.

[0082] Among them, the gyroscope sensor is used to monitor the posture changes of the user interactive device in real time, monitor the user's hand-held stability indicators, identify unexpected jitters and abnormal movements, and provide device motion status data. Before the gyroscope sensor works, set the raw data of the gyroscope sensor (the raw data of the gyroscope sensor refers to the physical quantity measurement value directly output by the sensor without any calibration or processing. This data reflects the angular velocity (rotation rate) of the device in three-dimensional space and is the benchmark for subsequent posture calculation, stability analysis and other applications), sampling rate and data timestamp. After the settings are completed, the interactive device direction information, stability score information and action type judgment information are output through the gyroscope sensor.

[0083] The touch gesture recognition module is used to provide users with a touch gesture interface, collect touch gesture sequences, gesture types, touch pressure values, and touch time, identify and classify user touch gesture sequences, analyze touch pressure distribution and contact area, evaluate the accuracy of touch gesture operations, and generate touch gesture adaptability optimization suggestions.

[0084] The speech recognition module is used to collect user voice input information, including voice audio stream, ambient noise level, user speaking rate and user voice characteristics, identify and understand the voice command content in the voice input information, evaluate the clarity of the voice input, intelligently filter out ambient noise interference, and provide optimization suggestions for voice interaction.

[0085] The user's multimodal interaction data is collected through the above-mentioned perception layer and transmitted to the decision-making layer.

[0086] Step S102: determining a user behavior feature vector based on the multimodal interaction data, and constructing a multi-dimensional interaction task completion quantification model based on the behavior feature vector.

[0087] Specifically, if Figure 4 As shown in Figure 2, the decision layer includes a behavior analyzer and a state predictor.

[0088] The behavior analyzer is used to receive the touch gesture operation sequence, data timestamp, touch mode and error mode collected in the perception layer (wherein, the touch mode refers to determining whether the current operation is a single-point operation or a multi-finger trigger. If a multi-finger trigger is an error mode, it is not considered valid data), and analyze user operation behavior patterns and habits in real time, identify and quantify user operation preferences, detect abnormal operation behavior and issue warnings, and generate user behavior for prediction, including user behavior feature vectors, habit strength scores, error probability predictions, etc.

[0089] The state predictor uses a hybrid habit strength modeling algorithm to construct a multi-dimensional interactive task completion quantification model based on the behavioral feature vector. Specifically, the state predictor is used to predict the user's next possible operation steps based on the user behavior output by the behavior analyzer, evaluate the risks and task completion success rates of different operation paths, predict changes in user cognitive load, and generate multi-path prediction results for strategy selection. The input parameters of the state analyzer are the current state of the user's task execution, user behavior feature vector, task vector, and historical task completion success rate. The output data are the predicted behavior sequence, predicted task completion success rate, predicted cognitive load, and confidence.

[0090] Step S103: Based on the multi-dimensional interactive task completion quantification model, a dynamic weight allocation strategy is adopted to determine the weight of each operation step in the multi-dimensional interactive task, and a preset adaptive coefficient is dynamically adjusted using a preset adjustment mechanism based on the multimodal interaction data and the weight of each operation step.

[0091] Specifically, the state predictor is also used to determine the weights of each operation step in a multidimensional interactive task by adopting a dynamic weight allocation strategy based on a multidimensional interactive task completion quantification model, and dynamically adjust the preset adaptive coefficient by adopting a preset adjustment mechanism based on multimodal interaction data and the weights of each operation step.

[0092] Step S104 : Based on the weight of each operation step and the dynamically adjusted preset adaptive coefficient, a dynamic path planning algorithm is used to generate a dynamic task guidance strategy.

[0093] Specifically, if Figure 4 As shown, the decision layer also includes a strategy generator, which uses a dynamic weight allocation strategy to determine the weight of each operation step in the multidimensional interactive task based on the multidimensional interactive task completion quantification model, and dynamically adjusts the preset adaptive coefficient based on the multimodal interaction data and the weight of each operation step using a preset adjustment mechanism.

[0094] Furthermore, the strategy generator generates personalized task guidance strategies based on the state prediction results output by the state predictor, optimizing interaction paths, reducing cognitive load, and specifying error resolution and recovery strategies for task failures. The strategy generator's inputs are the state prediction results, user behavior output by the behavior analyzer, multimodal interaction task requirements, and user capability assessment results (weights of each operation step and dynamically adjusted adaptive coefficients). Its outputs are task guidance strategies, intervention levels, alternative solutions, and optimized paths.

[0095] Step S105: Execute the dynamic task guidance strategy to obtain a multimodal execution result.

[0096] Specifically, the execution layer is used to execute dynamic task guidance strategies to obtain multimodal execution results.

[0097] The execution layer includes the interface reconstruction module, the prompt generation module, and the feedback collaboration module, among which:

[0098] The task guidance strategy is the input of the execution layer. The essence of the task guidance strategy is a dynamically generated decision instruction set that guides the execution layer on how to respond to multimodal task requirements. Its contents include:

[0099] Mission objectives: Specific problems that need to be solved at present (such as "stabilizing handheld devices" and "identifying user intentions").

[0100] Execution constraints: priority, timeliness, resource limitations, etc. (e.g., "real-time requirements > accuracy").

[0101] Contextual information: user status, environment parameters, historical interaction data, etc.

[0102] Modal allocation logic: determines which modules (interface reconstruction module / prompt generation module / feedback collaboration module) need to intervene and how they collaborate.

[0103] The interface reconstruction module is used to dynamically adjust the size, spacing and layout of UI elements to ensure visual friendliness for elderly users. Based on the user's visual perception ability, it adaptively adjusts the interface contrast and brightness, realizes intelligent anti-mistouch area division, reduces the probability of misoperation, and provides adaptive font scaling function to ensure that the text is clear and readable. The input of the interface reconstruction module is user visual perception ability indicators, touch gesture accuracy data, device information (screen size, brightness, direction), etc., and the output is a new layout configuration, scaling ratio, safe operation area and font adjustment plan.

[0104] The prompt generation module generates multi-level task guidance prompts, adapting to different cognitive abilities, intelligently determining prompt timing to avoid interrupting normal operations, dynamically adjusting prompt complexity based on the user's real-time cognitive load, and supporting intelligent switching between multiple prompt formats, including text, voice, and animation. The prompt generation module takes as input the current task context, user cognitive load, attention level, and historical error records, and outputs prompt type selection, prompt content, triggering timing, and duration.

[0105] Among them, the current task context refers to the real-time environment and status information required by the system when performing the task, which is used to dynamically adjust the behavior strategy, that is, the "surrounding situation" that needs to be paid attention to during the task execution, such as the current task progress, user feedback status, etc.

[0106] Prompt types include text, voice broadcast, animation demonstration and other types.

[0107] Trigger timing refers to the time or scenario in which necessary prompts need to be given during the task process. For example, when the default text prompt appears for a certain period of time and the user does not respond, other prompt types can be selected.

[0108] Trigger timing refers to the time or scenario in which necessary prompts need to be given during the task process. For example, when the default text prompt appears for a certain period of time and the user does not respond, other prompt types can be selected.

[0109] The duration includes both the length of time the trigger condition is met and the length of time the current prompt content is given to the user.

[0110] The feedback coordination module coordinates and manages the three feedback channels of vision, hearing, and touch. It selects the optimal feedback combination based on the environment and user status, precisely controls feedback intensity and duration, and handles conflicts and priorities among multi-channel feedback. The feedback coordination module includes event types, environmental context (noise, lighting, etc.), and user preferences, and outputs specific parameter settings for visual, auditory, and tactile feedback. Event types include, but are not limited to, task status change events (task completion, failure, etc.) and events where no feedback is given for an extended period of time after a prompt.

[0111] In summary, steps S101 to S105 are all implemented by the multimodal task interaction assistance system, in which the perception layer, decision layer, and execution layer form a complete data closed loop, specifically:

[0112] 1. Perception layer → Decision layer: The perception layer continuously transmits raw user interaction data, provides environment and device status information, and transmits multimodal interaction features to the decision layer in real time.

[0113] 2. Decision-making layer → Execution layer: The decision-making layer issues interaction strategies and optimization plans and provides predictive decision support. The execution layer is responsible for adjusting the interface and feedback parameters.

[0114] 3. Execution layer → Perception layer: The execution layer feeds back execution effect data to the perception layer, updates interaction status information, and adjusts perception module parameters.

[0115] Through this closed-loop collaborative mechanism, the system can continuously optimize the interactive experience for elderly users and provide more precise assistance and support. Each module is equipped with corresponding fault tolerance mechanisms and degradation strategies to ensure stable and reliable system operation.

[0116] The multimodal task interaction assistance method provided in this embodiment, by collecting multimodal interaction data, can obtain user information from multiple dimensions, covering the user's operational behavior, environmental status, and interaction intentions. Compared with single-modal data collection, multimodal data collection more comprehensively reflects the user's true state when performing tasks, avoiding misjudgments caused by missing information. Based on multimodal interaction data, user behavior feature vectors are determined and a multi-dimensional interactive task completion quantification model is constructed, achieving in-depth analysis of user behavior. A dynamic weight allocation strategy and preset adjustment mechanism are adopted to enable the system to have dynamic optimization capabilities, allowing the system to flexibly adapt to different user ability levels, task scenarios, and environmental conditions based on changes during user operation. Based on the dynamically adjusted weights and adaptive coefficients, a dynamic path planning algorithm is used to generate a dynamic task guidance strategy, and the strategy is executed to obtain multimodal execution results, effectively improving task completion efficiency. The present invention forms a complete closed-loop process from data collection, analysis, decision-making, to execution, and can provide intelligent services for middle-aged and elderly users, improving user experience and task execution efficiency. This solves the problem that the lack of intelligent services leads to insufficient user experience and low interactive task completion efficiency when middle-aged and elderly users interact with apps.

[0117] In this embodiment, a multimodal task interaction assistance method is provided, which can be used in a multimodal task interaction assistance system, such as Figure 4 As shown in Figure 2, the multimodal task interaction assistance system includes a perception layer, a decision layer, and an execution layer. Figure 2 is a flow chart of a multimodal task interaction assistance method according to an embodiment of the present invention. Figure 2 As shown, the process includes the following steps:

[0118] Step S201: Collect multimodal interaction data of users performing multi-dimensional interaction tasks. Figure 1 Step S101 of the illustrated embodiment will not be described in detail here.

[0119] Step S202 : determining a user behavior feature vector based on the multimodal interaction data, and constructing a multi-dimensional interaction task completion quantification model based on the behavior feature vector.

[0120] Specifically, the multimodal interaction data includes touch gesture recognition data, voice recognition data, and sensor data; the multi-dimensional interaction task includes multiple operation steps; the above step S202 includes:

[0121] Step S2021: Based on touch gesture recognition data, voice recognition data, and sensor data, a deep learning algorithm is used to extract user operation behavior patterns and habits, identify user operation preferences and abnormal operation behaviors, and obtain a user behavior feature vector. The user behavior feature vector includes user operation habit features, operation proficiency features, and error tendency features.

[0122] Specifically, if Figure 4 As shown, the behavior analyzer uses a deep learning algorithm to extract user operation behavior patterns and habits based on touch gesture recognition data, voice recognition data and sensor data, identify user operation preferences and abnormal operation behaviors, and obtain a user behavior feature vector. The user behavior feature vector includes user operation habit features, operation proficiency features and error tendency features, and transmits the user behavior feature vector to the state predictor.

[0123] Step S2022: Based on the user behavior feature vector and multiple operation steps, a preset multi-dimensional task completion metricization formula is used to construct a multi-dimensional interactive task completion metricization model.

[0124] The formula for the quantification model of multi-dimensional interactive task completion is expressed as:

[0125] Completion_Index=Σ(Step_Weight[i]×Success_Rate[i])+λ×log(Retry_Count+1);

[0126] Among them, Step_Weight[i] represents the i-th weight, Success_Rate[i] represents the task completion success rate, λ×log(Retry_Count+1) represents the retry correction, λ represents the preset adaptive coefficient, and Retry_Count represents the retry behavior.

[0127] Specifically, in view of the characteristics of middle-aged and elderly users when using smart devices, traditional task completion assessment methods have the following problems: they only consider the final completion status, ignore the attempt behavior in the process, do not take into account the difficulty differences of task steps, lack dynamic assessment of user cognitive load, and cannot reflect the synergistic effect of multimodal interaction.

[0128] Based on the above problems, this embodiment provides a formula for a multi-dimensional interactive task completion quantification model that comprehensively considers multi-dimensional factors:

[0129] Completion_Index=Σ(Step_Weight[i]×Success_Rate[i])+λ×log(Retry_Count+1)(1);

[0130] 1. The components of the formula are as follows:

[0131] 1. Step weight item Step_Weight[i]: The step weight reflects the importance of each step to the task, taking into account factors such as user cognitive load and operation complexity, and dynamically adjusts the weight distribution through the deep learning model.

[0132] 2. Task completion success rate item Success_Rate[i]: The task completion success rate measures the execution quality of each step, combining operation accuracy and time efficiency, and introducing collaborative scoring of multimodal interaction in the calculation.

[0133] 3. Retry correction term λ×log(Retry_Count+1): Retry correction applies nonlinear penalties to user retry behavior, where λ is an adaptive coefficient in the range [0.1, 0.5]. The adaptive coefficient uses a logarithmic function to avoid excessive penalties.

[0134] 2. The correlation between the multi-dimensional interactive task completion quantification model formula and each layer in the multimodal task interaction assistance system is explained as follows:

[0135] 1. Associated with the perception layer: Gyroscope sensors are used to collect user operation data, monitor operation accuracy and completion status in real time, and provide identification and classification of retry behaviors.

[0136] 2. Association with the decision-making layer: The behavior analyzer calculates the success rate of each step task completion Success_Rate, the state predictor dynamically adjusts the weight of each step Step_Weight, and the strategy generator optimizes the task guidance strategy based on the multimodal task completion.

[0137] The multimodal task completion degree includes the completion success rate and weight. The completion degree calculation formula is as follows:

[0138] Basic formula: Task_Completion_Score=Σ(Success_Rate_i×Step_Weight_i) / ΣStep_Weight_i;

[0139] Among them, Success_Rate_i: the historical completion success rate of step i (provided by the behavior analyzer); Step_Weight_i: the real-time weight of step i (dynamically adjusted by the state predictor).

[0140] The dynamic weight adjustment logic is as follows:

[0141] The state predictor modifies the weights in real time based on the context. For example, if the user is stuck, the weight of the current step is used (priority guidance is required); if the environment is disturbed, the weight of the step that relies on voice is used (switching to visual prompts); if the task is urgent, the weight of the subsequent step is used (resources are preloaded in advance).

[0142] Example:

[0143] If the user frequently fails at step 2 (Success_Rate_2 = 0.3), and this step becomes critical due to device tilt (Step_Weight_2 from 0.4 → 0.8):

[0144] Task_Completion_Score=(0.9×0.2+0.3×0.8+0.6×0.5) / (0.2+0.8+0.5)=0.53.

[0145] 3. Associated with the execution layer: The interface reconstruction module adjusts the UI elements according to the completion degree, the prompt generation module adjusts the prompt level according to the task completion score, and the feedback collaboration module selects the feedback intensity according to the multimodal task completion degree.

[0146] 3. The application of the multi-dimensional interactive task completion quantification model formula in the multimodal task interaction assistance system is as follows:

[0147] 1. Real-time evaluation application: dynamically calculate the current task progress, predict the difficulty of task completion, and promptly identify operational obstacles.

[0148] 2. Strategy optimization application: guide the generation of personalized task guidance strategies, optimize multimodal feedback combinations (including visual, auditory, and tactile feedback), and adjust interface adaptation parameters.

[0149] 3. Long-term application optimization: Build user capability profiles, identify common operational obstacles, and optimize the overall interaction process.

[0150] 4. Verification and Effect:

[0151] This was verified through an experiment involving 3,000 users aged 60 and over, with a 50 / 50 split between an experimental and control group. Data was collected based on the overall completion rate of new users' tasks within the app. Using both static and multimodal task guidance within the app, the two groups of data were compared, measuring the completion and accuracy of each step within the task series, as well as the overall completion time.

[0152] This quantitative model has the following advantages:

[0153] 1. Improved accuracy: The accuracy of task completion prediction increased by 35%, the accuracy of operational obstacle identification increased by 42%, and the accuracy of user capability assessment increased by 28%.

[0154] 2. Enhanced adaptability: Supports more than 90% of common task scenarios, adapts to more than 95% of user operating habits, and covers more than 85% of abnormal operating conditions.

[0155] 3. Practicality verification: average task completion time is reduced by 40%, operation error rate is reduced by 45%, and user satisfaction is increased by 52%.

[0156] These data show that the quantitative model formula can effectively support the system's aging-friendly intelligent interaction and significantly improve the user experience of middle-aged and elderly users.

[0157] In this paper, a multidimensional interactive task completion metric model is constructed based on user behavior feature vectors and multiple operational steps using a pre-defined multidimensional task completion metric formula. The model's lambda (λ) can be dynamically adjusted to avoid over-penalizing user attempts while also rationally reflecting the impact of retries on task completion. By incorporating three key factors—task step weights, completion success rates, and retry behavior—the multidimensional interactive task completion metric model can more accurately and comprehensively quantify task completion, avoiding evaluation bias.

[0158] Step S203: extract the time features, frequency features, and quality features from the user behavior feature vector; train the pre-trained model based on the time features, frequency features, and quality features until the preset convergence conditions are met, thereby obtaining a hybrid habit strength model; predict the user behavior pattern based on the hybrid habit strength model to obtain a prediction result, and dynamically optimize the hybrid habit strength model based on the relationship between the prediction result and the preset threshold.

[0159] Specifically, based on the formation and reinforcement characteristics of operating habits of middle-aged and elderly users, a hybrid habit strength model was designed: combining frequency and time dimensions, considering operating stability indicators, introducing forgetting curve correction, and supporting incremental learning.

[0160] like Figure 5 As shown, this is a flow chart for constructing a hybrid habit strength model in a multimodal task interaction assistance method. Multimodal data is collected through the perception layer, including touch gesture operation sequences, data timestamps, and task completion quality (task completion quality is mainly collected based on the effectiveness of task completion, combined with the effectiveness association judgment of the touch gesture operation sequence, such as whether the user touches the valid option with a single finger or with multiple fingers. Multi-finger touch will be evaluated as ineffective completion) and other user behavior data. Based on the multimodal data, time features, frequency features, and quality features are extracted. The pre-trained model is trained based on the time features, frequency features, and quality features until the preset convergence conditions are met to obtain a hybrid habit strength model. The user behavior pattern is predicted based on the hybrid habit strength model to obtain the prediction result, and the hybrid habit strength model is dynamically optimized based on the relationship between the prediction result and the preset threshold. Among them:

[0161] 1. Habit strength calculation: Frequency impact: using exponential decay function; Time impact: based on operational stability assessment; Weight adaptation: through gradient descent optimization.

[0162] 2. Training data requirements: Sample size: no less than 10,000 records for each type of behavior; Feature dimensions: 20-dimensional features such as time, frequency, and accuracy; Labeling standards: expert scores are used as training labels.

[0163] 3. Convergence condition setting: loss threshold: less than 0.01, number of iterations: more than 1000 rounds; verification accuracy: reach more than 95%.

[0164] Step S204: Based on the multi-dimensional interactive task completion quantification model, a dynamic weight allocation strategy is used to determine the weight of each operation step in the multi-dimensional interactive task, and a preset adjustment mechanism is used to dynamically adjust the preset adaptive coefficient based on the multimodal interaction data and the weight of each operation step. Figure 1 Step S103 of the illustrated embodiment will not be described in detail here.

[0165] Step S205: Based on the weight of each operation step and the dynamically adjusted preset adaptive coefficient, a dynamic path planning algorithm is used to generate a dynamic task guidance strategy. Figure 1 Step S104 of the illustrated embodiment will not be described in detail here.

[0166] Step S206: Execute the dynamic task guidance strategy to obtain the multi-modal execution result. Figure 1 Step S105 of the illustrated embodiment will not be described in detail here.

[0167] The multimodal task interaction assistance method provided in this embodiment uses multimodal interaction data covering touch gesture recognition data, voice recognition data, and sensor data, collecting user information from different dimensions and breaking the limitations of a single data source. The multimodal interaction data is processed using a deep learning algorithm, which can mine the user behavior patterns hidden behind the data. The algorithm can automatically learn user operation behavior patterns and habits, and accurately identify user operation preferences through large-scale data training. The determined user behavior feature vector includes user operation habit features, operation proficiency features, and error tendency features. Accurate feature extraction enables the system to provide personalized services for different users. This process can process multimodal interaction data in real time and dynamically update user behavior feature vectors. The system continuously monitors user operations and can respond quickly once changes in behavior patterns or abnormal operations are detected, providing a data foundation for the subsequent construction of a hybrid habit strength model and a multi-dimensional interaction task completion quantification model.

[0168] In this embodiment, a multimodal task interaction assistance method is provided, which can be used in a multimodal task interaction assistance system, such as Figure 4 As shown in Figure 2, the multimodal task interaction assistance system includes a perception layer, a decision layer, and an execution layer. Figure 3 is a flow chart of a multimodal task interaction assistance method according to an embodiment of the present invention. Figure 3 As shown, the process includes the following steps:

[0169] Step S301: Collect multimodal interaction data of users performing multi-dimensional interaction tasks. Figure 2Step S201 of the illustrated embodiment will not be described in detail here.

[0170] Step S302: Determine the user behavior feature vector based on the multimodal interaction data, and construct a multi-dimensional interaction task completion quantification model based on the behavior feature vector. Figure 2 Step S202 of the illustrated embodiment will not be described in detail here.

[0171] Step S303: Based on the multi-dimensional interactive task completion quantification model, a dynamic weight allocation strategy is adopted to determine the weight of each operation step in the multi-dimensional interactive task, and a preset adaptive coefficient is dynamically adjusted using a preset adjustment mechanism based on the multimodal interaction data and the weight of each operation step.

[0172] Specifically, the above step S303 includes:

[0173] Step S3031: Calculate the operation complexity score of each operation step based on the number of operations, the number of interaction types, and the cognitive load of each operation step; calculate the overall task importance score of each operation step based on the user's task flow dependency and the impact of failure of each operation step; and use a dynamic weight allocation strategy to determine the weight of each operation step in the multi-dimensional interactive task based on the operation complexity score of each operation step and the overall task importance score.

[0174] The formula of dynamic weight allocation strategy is expressed as:

[0175] W[i] = (Complexity[i] × Importance[i]) / Σ(Complexity[j] ×Importance[j]) (2);

[0176] Where W[i] represents the weight of the i-th operation step, Complexity[i] represents the operation complexity score of the i-th operation step, and Importance[i] represents the importance score of the i-th operation step to the overall task.

[0177] Specifically, the value range of Complexity[i] is [1,5]. The comprehensive score is based on the number of operations, the number of interaction types, and the cognitive load included in the operation steps. An example is shown in Table 1 below:

[0178] Table 1 Example of operation complexity scoring

[0179]

[0180] Importance[i] ranges from 1 to 5 and is scored by business experts based on task process dependencies and the impact of failure. An example of importance scoring is shown in Table 2 below:

[0181] Table 2 Importance scoring examples

[0182]

[0183]

[0184] The denominator Σ(Complexity[j]×Importance[j]) in formula (2) ensures that the sum of all weights is 1. The verification example is as follows:

[0185] Assume that a task consists of 3 steps, and its complexity and importance scores are as follows:

[0186] Step1:Complexity=2,Importance=3→2×3=6;

[0187] Step2:Complexity=4,Importance=5→4×5=20;

[0188] Step3:Complexity=1,Importance=1→1×1=1;

[0189] Total weight sum = 6 + 20 + 1 = 27 then:

[0190] w[1]=6 / 27≈0.222, w[2]=20 / 27≈0.741, w[3]=1 / 27≈0.037.

[0191] Step S3032: Calculate the historical task completion rate, cognitive load index, and environmental adaptability index based on the multimodal interaction data; calculate the adaptive factor based on the historical task completion rate, cognitive load index, and environmental adaptability index; generate a preset adaptive coefficient calculation model based on the adaptive factor and the weight of each operation step; and dynamically adjust the preset adaptive coefficient using a preset adjustment mechanism based on the preset adaptive coefficient calculation model.

[0192] Among them, the expression of the preset adaptive coefficient calculation model is:

[0193] λ=λ_base+λ_range×(1-exp(-k×Adaptive_Factor));

[0194] Among them, λ_base represents the basic penalty coefficient, λ_range represents the dynamic adjustment range, k represents the sensitivity coefficient, and Adaptive_Factor represents the adaptive factor. Its expression is:

[0195] AdaptiveFactor=w1×Success_Rate+w2×Cognitive_Load+w3×Environmental_Index;

[0196] Among them, Success_Rate represents the task completion success rate, Cognitive_Load represents the cognitive load index, Environmental_Index represents the environmental adaptability index, w1, w2 and w3 represent the task completion success rate weight, cognitive load index weight and environmental adaptability index weight respectively.

[0197] Specifically, considering the characteristics of middle-aged and elderly users during the interaction process, the traditional fixed penalty coefficient has the following problems: it cannot adapt to the learning curves of different users, ignores the impact of environmental factors on operations, does not consider the differences in task types, and lacks dynamic evaluation of cognitive load.

[0198] 1. Based on the above problems, this embodiment provides an adaptive dynamic adjustment mechanism for the λ value:

[0199] λ= λ_base+λ_range× (1-exp(-k×Adaptive_Factor)) (3);

[0200] in:

[0201] Adaptive_Factor=w1×Success_Rate+w2×Cognitive_Load+w3×Environmental_Index(4);

[0202] The definitions and calculation rules of the parameters in formula (3) and formula (4) are shown in Table 3 below:

[0203] Table 3 Parameter definitions and calculation rules

[0204]

[0205] 2. An example of the dynamic weight adjustment mechanism is as follows:

[0206] 1. The initial weight configuration is: w1 = 0.4 / / task completion success rate weight; w2 = 0.4 / / cognitive load weight; w3 = 0.2 / / environmental factor weight.

[0207] 2. Dynamically adjust rules: In high cognitive load scenarios, increase the weight of w2; in complex environmental conditions, increase the weight of w3; in the early stages of learning, increase the weight of w1.

[0208] 3. This embodiment also analyzes aging-friendly features, including:

[0209] 1. Cognitive load assessment: operation response time analysis, error pattern recognition, attention level monitoring, and task switching cost calculation.

[0210] 2. Environmental adaptability analysis: lighting condition assessment, sound environment detection, device grip stability, and operating posture analysis.

[0211] 3. Historical data processing: time decay weighting, outlier filtering, trend analysis, and periodic pattern recognition.

[0212] 4. The application effect of dynamically adjusting the preset adaptive coefficient λ using a preset adjustment mechanism based on multimodal interaction data and the weight of each operation step is verified as follows:

[0213] 1. Improved adaptability: User operation fluency increased by 38%, task completion efficiency increased by 42%, and error recovery speed increased by 45%.

[0214] 2. Personalization level: supports more than 85% of personalized scenarios, adapts to more than 90% of user habits, and covers more than 95% of environmental changes.

[0215] 3. Stability performance: System response delay is reduced by 35%, parameter fluctuation range is reduced by 40%, and prediction accuracy is improved by 47%.

[0216] 5. The synergistic relationship between dynamic adjustment of the λ value and the three-tier architecture is as follows:

[0217] 1. Perception layer collaboration: real-time collection of multimodal interaction data, dynamic monitoring of environmental status, and recognition of user behavior patterns.

[0218] 2. Decision-making layer collaboration: real-time assessment of cognitive load, environmental adaptability analysis, and user status prediction updates.

[0219] 3. Execution layer collaboration: dynamic adjustment of interface parameters, adaptive feedback intensity, and optimization of guidance strategies.

[0220] Through a multi-level collaborative dynamic adjustment mechanism of the lambda value, the system can provide more accurate aging-friendly interaction support for middle-aged and elderly users, significantly improving the personalization and adaptability of the interactive experience.

[0221] It should be noted that based on the cognitive characteristics and multimodal interaction needs of middle-aged and elderly users, the following decision table was designed at the decision-making level, as shown in Table 4:

[0222] Table 4 Decision table

[0223] Trigger Conditions Visual feedback Auditory feedback haptic feedback Priority Operational Error Highlight Tips Short beep Short vibration 1 Mission accomplished animation effects Voice praise Long vibration 2 Help Request Steps Voice guidance none 3 Timeout reminder Flashing prompt Reminder sound Double vibration 4

[0224] The decision rules are explained based on Table 4:

[0225] 1. Priority principle: High-priority events can interrupt low-priority feedback, events of the same priority level are processed in chronological order, and priorities can be dynamically adjusted in special scenarios.

[0226] 2. Feedback selection rules: Select the auditory feedback intensity based on the ambient noise level, adjust the visual feedback method according to the ambient lighting conditions, and determine the tactile feedback mode based on the device holding state.

[0227] 3. Synergistic effect evaluation: Calculate the combined impact of multimodal feedback, evaluate the cognitive load of feedback combinations, monitor feedback effects, and make dynamic adjustments.

[0228] A decision model is constructed based on the decision table and decision rules. The application of the decision model is described as follows;

[0229] 1. Real-time scenario adaptation: Dynamically select the optimal feedback combination, adaptively adjust feedback intensity, and intelligently handle feedback conflicts.

[0230] 2. Personalization: Learn user feedback preferences, establish a personalized feedback model, and optimize feedback timing.

[0231] 3. Effect evaluation: track feedback response data, analyze feedback effectiveness, and continuously optimize decision-making rules.

[0232] Step S304 : Based on the weight of each operation step and the dynamically adjusted preset adaptive coefficient, a dynamic path planning algorithm is used to generate a dynamic task guidance strategy.

[0233] Specifically, the above step S304 includes:

[0234] Step S3041: Obtain the calculation results of the multi-dimensional interactive task completion quantification model and the multi-dimensional interactive task preset requirements.

[0235] Specifically, the calculation results of the multi-dimensional interactive task completion quantification model include user cognitive load, task completion success rate, environmental adaptation index, error rate, etc. When designing tasks, hard requirements and optimization goals must be predefined. Generally, multi-dimensional interactive task requirements include the following dimensions:

[0236] 1. User dimension requirements:

[0237] The maximum permissible cognitive load value is set for the specific content of the requirement type (e.g. eye tracking index ≤ 0.6);

[0238] Accessibility compatibility supports voice control / high contrast mode, etc. (e.g., compliant with WCAG 2.1AA standards);

[0239] Personalized adaptation allows the difficulty to be adjusted based on user historical data (for example, the novice mode defaults to a step-by-step guide).

[0240] 2. Task process dimension requirements:

[0241] Requirement type, specific content, completion time threshold, and maximum time required for a single step (e.g., key operations must be completed within 30 seconds).

[0242] The fault tolerance mechanism allows an upper limit on the number of errors (for example, three consecutive errors trigger a help video);

[0243] Step dependencies clarify the order of steps and allow for skipping conditions (e.g., step B must be unlocked after step A succeeds 80%).

[0244] 3. Environmental dimension requirements:

[0245] Requirement type: Specific content: Multimodal degradation rules define feedback alternatives when the environment is abnormal (e.g., disable voice when noise > 65dB);

[0246] Resource usage limits the maximum CPU / memory usage (e.g. animation rendering does not exceed 15% of the CPU);

[0247] Cross-platform consistency across different devices (mobile phones / AR glasses);

[0248] The core task flow must be consistent (for example, the layout of payment process buttons must be aligned).

[0249] 4. System dimension requirements:

[0250] Requirement type: Specific content: Real-time requirement: Maximum delay from sensor input to feedback output (e.g., anti-shake prompt delay ≤ 80ms);

[0251] Data security threshold user behavior data desensitization rules (for example: location information accuracy is blurred to a range of 100 meters);

[0252] Policy update frequency dynamically adjusts the minimum time interval for the policy (for example, the weight is recalculated every 5 seconds).

[0253] Step S3042 , based on the calculation results of the multi-dimensional interactive task completion quantification model, the preset requirements of the multi-dimensional interactive task, the weights of each operation step and the dynamically adjusted preset adaptive coefficient, a dynamic path planning algorithm is used to generate a dynamic task guidance strategy.

[0254] Specifically, based on the cognitive characteristics of middle-aged and elderly users when completing complex tasks, a state-space-based dynamic path planning algorithm was designed to reduce the cognitive load of the task, optimize the length of the operation path, provide real-time task guidance strategies, and adapt to individual differences. The decision-making layer's strategy generator uses the dynamic path planning algorithm to generate a flowchart of the dynamic task guidance strategy, as shown in the figure below. Figure 6 As shown, the following steps are included:

[0255] 1. State space construction: Dimension selection: cognitive load, operation speed, error rate; value range: each dimension is normalized to the interval [0,1]; granularity adjustment: dynamically set according to the characteristics of the user group.

[0256] 2. User state tree generation: Historical data analysis: Extracting operation patterns and habits; Current state assessment: Real-time monitoring of multimodal interaction data Depth control: Limiting to 5 layers to reduce computational complexity.

[0257] 3. Reward matrix calculation: User feature matching: Consider individual differences; Task difficulty assessment: Dynamically adjust reward weights; Migration cost calculation: Optimize cognitive load distribution.

[0258] 4. Heuristic pruning strategy: Rule definition: based on the operating characteristics of elderly users; threshold setting: dynamically adjust pruning intensity; path retention: ensure the integrity of key operation paths, and ultimately generate a dynamic task guidance strategy.

[0259] Step S305: Execute the dynamic task guidance strategy to obtain a multimodal execution result.

[0260] Specifically, the above step S305 includes:

[0261] Step S3051: Generate a user visual guidance enhancement plan based on the calculation results of the dynamic task guidance strategy and the multi-dimensional interactive task completion quantification model. The user visual guidance enhancement plan includes dynamic adjustment of the interface elements, spacing, layout, interface contrast and brightness of the user device.

[0262] Specifically, the user visual guidance enhancement scheme generation in the multimodal task interaction assistance method is implemented by the interface reconstruction module, and the generated flowchart is as follows: Figure 7 shown.

[0263] The design principles of user visual guidance enhancement solutions include: clarity: ensuring that visual elements are easy to identify; consistency: maintaining the uniformity of interactive feedback; adaptability: dynamic adjustment based on user capabilities; and anti-interference: reducing the impact of environmental factors.

[0264] The interface reconstruction module generates visual guidance enhancement solutions including:

[0265] 1. Interface adaptive adjustment: dynamically calculate the UI scaling ratio of interface elements, intelligently adjust element spacing, and optimize touch response area.

[0266] 2. User history analysis: record operation error patterns, calculate response time distribution, and evaluate the success rate of interactive task completion.

[0267] Step S3052 : generating a task guidance prompt when the user performs the multi-dimensional interactive task based on the dynamic task guidance strategy and the calculation result of the multi-dimensional interactive task completion quantification model, and generating a task progress tracking plan based on the task guidance prompt.

[0268] Specifically, the prompt generation module of the execution layer generates task guidance prompts when the user performs a multi-dimensional interactive task based on the calculation results of the dynamic task guidance strategy and the multi-dimensional interactive task completion quantification model, and generates a task progress tracking plan based on the task guidance prompts.

[0269] The tracking dimensions of the task progress tracking solution include: video viewing progress, attention score, interaction completion, and task completion rate. The flowchart of the task progress tracking solution generated in the multimodal task interaction auxiliary method is as follows: Figure 8 As shown, the implementation process includes:

[0270] Collect multi-dimensional data, including user viewing time, number of interactions and task completion status (task completion success rate, etc.), calculate task completion progress based on multi-dimensional data, and generate a task progress tracking plan to evaluate user task progress. The evaluation content includes: progress evaluation: real-time calculation of task completion, dynamic adjustment of multi-dimensional weights and intelligent identification of abnormal situations.

[0271] When generating a task progress tracking scheme, the prompt generation module generates task guidance prompts, including visual progress indications, voice prompts, and tactile feedback controls.

[0272] Step S3053: Generate a multimodal feedback control solution after the user performs each operation step in the multi-dimensional interactive task.

[0273] Specifically, a feedback collaboration module is used to generate a multimodal feedback control scheme after the user performs each operation step in the multi-dimensional interactive task. The design goals include: reducing cognitive load, improving feedback efficiency, ensuring smooth interaction, and adapting to environmental changes. The implementation flow chart of the generated multimodal feedback control scheme is as follows: Figure 9 As shown, it includes: task status detection, feedback strategy selection, multimodal collaboration, and effect evaluation.

[0274] Feedback strategies include:

[0275] Success feedback: animation + voice + vibration; warning feedback: highlight + prompt sound; guidance feedback: step display + voice.

[0276] Environmental adaptation: Noisy environment: enhance visual feedback; strong light environment: enhance tactile feedback; motion state: simplify feedback method.

[0277] Step S3054: taking the user visual guidance enhancement scheme, the task progress tracking scheme, and the multimodal feedback control scheme as the multimodal execution result.

[0278] The multimodal task interaction assistance method provided in this embodiment comprehensively considers the calculation results of a multi-dimensional interactive task completion quantification model, the preset requirements of the multi-dimensional interactive task, the weights of each operation step, and the dynamically adjusted preset adaptive coefficients when generating a dynamic task guidance strategy. Multi-source data complements and verifies each other. The introduction of operation step weights enables the dynamic task guidance strategy to focus on the key aspects of the task. By focusing on and optimizing key aspects, the efficiency and quality of task completion are significantly improved, and the risk of task execution is reduced. Based on the dynamic task guidance strategy and the calculation results of the multi-dimensional interactive task completion quantification model, a user visual guidance enhancement solution is generated. This achieves precise optimization of the user device interface, reduces operational errors caused by visual impairments, and significantly improves the comfort and accuracy of user visual interaction. Based on the dynamic task guidance strategy and the calculation results of the quantification model, task guidance prompts and task progress tracking solutions are generated, providing users with real-time and effective task execution support. A multimodal feedback control solution is generated after the user completes each operation step, promptly informing the user of the operation results through multiple feedback forms such as vision, hearing, and touch, forming a complete interactive closed loop. The user visual guidance enhancement scheme, task progress tracking scheme and multimodal feedback control scheme are integrated into a multimodal execution result, achieving the synergy and complementarity of each scheme and providing users with comprehensive task execution support.

[0279] In this embodiment, a multimodal task interaction assistance system is also provided, which is used to implement the above-mentioned embodiments and preferred embodiments. The details that have been described will not be repeated here. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the system described in the following embodiments is preferably implemented in software, implementation using hardware, or a combination of software and hardware, is also possible and contemplated.

[0280] This embodiment provides a multimodal task interaction assistance system, such as Figure 10 Shown, including:

[0281] The perception layer 1001 is used to collect multimodal interaction data of users performing multi-dimensional interaction tasks.

[0282] The decision layer 1002 is used to determine the user behavior feature vector based on the multimodal interaction data, and generate a multidimensional interaction task completion quantification model based on the behavior feature vector; based on the multidimensional interaction task completion quantification model, a dynamic weight allocation strategy is used to determine the weight of each operation step in the multidimensional interaction task, and a dynamic adjustment mechanism is used to dynamically adjust the preset retry correction coefficient based on the multimodal interaction data and the weight of each operation step; and a dynamic task guidance strategy is generated based on the weight of each operation step and the dynamically adjusted retry correction coefficient.

[0283] The execution layer 1003 is used to execute the dynamic task guidance strategy and obtain multi-modal execution results.

[0284] In some optional implementations, the multimodal interaction data includes touch gesture recognition data, voice recognition data, and sensor data; the multi-dimensional interaction task includes multiple operation steps; and the decision layer 1002 includes:

[0285] The feature vector extraction unit is used to extract user operation behavior patterns and habits based on touch gesture recognition data, voice recognition data and sensor data using a deep learning algorithm, identify user operation preferences and abnormal operation behaviors, and obtain a user behavior feature vector. The user behavior feature vector includes user operation habit features, operation proficiency features and error tendency features.

[0286] A multi-dimensional interactive task completion metric model construction unit is used to construct a multi-dimensional interactive task completion metric model based on a user behavior feature vector and multiple operation steps using a preset multi-dimensional task completion metric formula;

[0287] The formula for the quantification model of multi-dimensional interactive task completion is expressed as:

[0288] Completion_Index=Σ(Step_Weight[i]×Success_Rate[i])+λ×log(Retry_Count+1);

[0289] Among them, Step_Weight[i] represents the i-th weight, Success_Rate[i] represents the task completion success rate, λ×log(Retry_Count+1) represents the retry correction, λ represents the preset adaptive coefficient, and Retry_Count represents the retry behavior.

[0290] In some optional implementations, the multimodal task interaction assistance system further includes:

[0291] The hybrid habit strength model construction module is used to extract the time features, frequency features and quality features from the user behavior feature vector; the pre-trained model is trained based on the time features, frequency features and quality features until the preset convergence conditions are met to obtain a hybrid habit strength model; the user behavior pattern is predicted based on the hybrid habit strength model to obtain the prediction result, and the hybrid habit strength model is dynamically optimized based on the relationship between the prediction result and the preset threshold.

[0292] In some optional implementations, the decision layer 1002 further includes:

[0293] A step weight dynamic adjustment unit is used to calculate the operation complexity score of each operation step based on the number of user operations, the number of interaction types, and cognitive load of each operation step; calculate the overall task importance score of each operation step based on the user's task flow dependency and the impact of failure of each operation step; and use a dynamic weight allocation strategy to determine the weight of each operation step in a multi-dimensional interactive task based on the operation complexity score of each operation step and the importance score of the overall task;

[0294] The formula of dynamic weight allocation strategy is expressed as:

[0295] W[i]=(Complexity[i]×Importance[i]) / Σ(Complexity[j]×Importance[j]);

[0296] Where W[i] represents the weight of the i-th operation step, Complexity[i] represents the operation complexity score of the i-th operation step, and Importance[i] represents the importance score of the i-th operation step to the overall task.

[0297] In some optional implementations, the decision layer 1002 further includes:

[0298] The adaptive coefficient dynamic adjustment unit is used to calculate the historical task completion rate, cognitive load index and environmental adaptability index based on multimodal interaction data; calculate the adaptive factor based on the historical task completion rate, cognitive load index and environmental adaptability index; generate a preset adaptive coefficient calculation model based on the adaptive factor and the weight of each operation step; and dynamically adjust the preset adaptive coefficient using a preset adjustment mechanism based on the preset adaptive coefficient calculation model; wherein the expression of the preset adaptive coefficient calculation model is:

[0299] λ=λ_base+λ_range×(1-exp(-k×Adaptive_Factor));

[0300] Among them, λ_base represents the basic penalty coefficient, λ_range represents the dynamic adjustment range, k represents the sensitivity coefficient, and Adaptive_Factor represents the adaptive factor. Its expression is:

[0301] AdaptiveFactor=w1×Success_Rate+w2×Cognitive_Load+w3×Environmental_Index;

[0302] Among them, Success_Rate represents the task completion success rate, Cognitive_Load represents the cognitive load index, Environmental_Index represents the environmental adaptability index, w1, w2 and w3 represent the task completion success rate weight, cognitive load index weight and environmental adaptability index weight respectively.

[0303] In some optional implementations, the decision layer 1002 further includes:

[0304] The data acquisition unit is used to obtain the calculation results of the multi-dimensional interactive task completion quantification model and the preset requirements of the multi-dimensional interactive task.

[0305] The dynamic task guidance strategy generation unit is used to generate a dynamic task guidance strategy using a dynamic path planning algorithm based on the calculation results of the multi-dimensional interactive task completion quantification model, the preset requirements of the multi-dimensional interactive task, the weights of each operation step, and the dynamically adjusted preset adaptive coefficient.

[0306] In some optional implementations, the execution layer 1003 includes:

[0307] The visual guidance enhancement scheme generation unit is used to generate a user visual guidance enhancement scheme based on the calculation results of the dynamic task guidance strategy and the multi-dimensional interactive task completion quantification model. The user visual guidance enhancement scheme includes dynamic adjustment of the interface elements, spacing, layout, interface contrast and brightness of the user device.

[0308] The task progress tracking scheme generating unit is used to generate task guidance prompts when the user performs a multi-dimensional interactive task based on the calculation results of the dynamic task guidance strategy and the multi-dimensional interactive task completion quantification model, and generate a task progress tracking scheme based on the task guidance prompts.

[0309] The feedback control scheme generating unit is used to generate a multimodal feedback control scheme after the user performs each operation step in the multi-dimensional interactive task.

[0310] The multimodal execution result determination unit is used to take the user visual guidance enhancement scheme, the task progress tracking scheme and the multimodal feedback control scheme as the multimodal execution result.

[0311] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.

[0312] The multimodal task interaction assistance system in this embodiment is presented in the form of a functional unit, where the unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0313] The embodiment of the present invention also provides a computer device having the above Figure 10 The multimodal task interaction assistance system shown.

[0314] See also Figure 11 , Figure 11 is a structural diagram of a computer device provided by an optional embodiment of the present invention, such as Figure 11 As shown, the computer device includes: one or more processors 10, memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components utilize different buses to communicate with each other and can be installed on a common mainboard or installed in other ways as needed. The processor can process the instructions executed in the computer device, including instructions stored in the memory or on the memory to display the graphical information of the GUI on an external input / output device (such as, a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Equally, multiple computer devices can be connected, and each device provides part of the necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 11 A processor 10 is taken as an example.

[0315] The processor 10 may be a central processing unit, a network processor, or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be an application-specific integrated circuit, a programmable logic device, or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic, or any combination thereof.

[0316] The memory 20 stores instructions that can be executed by at least one processor 10, so that the at least one processor 10 executes the method shown in the above embodiment.

[0317] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and application programs required for at least one function; the data storage area may store data created based on the use of the computer device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely located relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0318] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid-state drive; the memory 20 may also include a combination of the above types of memory.

[0319] The computer device further includes an input device 30 and an output device 40. The processor 10, the memory 20, the input device 30 and the output device 40 may be connected via a bus or other means. Figure 11 The bus connection is taken as an example.

[0320] The input device 30 can receive input digital or character information and generate key signal input related to user settings and function control of the computer device, such as a touch screen, a keypad, a mouse, a trackpad, a touch pad, an indicator stick, one or more mouse buttons, a trackball, a joystick, etc. The output device 40 can include a display device, an auxiliary lighting device (e.g., an LED), and a tactile feedback device (e.g., a vibration motor). The above-mentioned display device includes but is not limited to a liquid crystal display, a light emitting diode, a display, and a plasma display. In some optional embodiments, the display device can be a touch screen.

[0321] The embodiment of the present invention also provides a computer-readable storage medium. The above-mentioned method according to the embodiment of the present invention can be implemented in hardware, firmware, or implemented as a computer code that can be recorded in a storage medium, or implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and downloaded through a network and will be stored in a local storage medium, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state drive, etc.; further, the storage medium can also include a combination of the above-mentioned types of memory. It can be understood that a computer, a processor, a microprocessor controller or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor or hardware, the method shown in the above embodiment is implemented.

[0322] A portion of the present invention may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present invention through the operation of the computer. Those skilled in the art should understand that the form in which the computer program instruction exists in a computer-readable medium includes, but is not limited to, a source file, an executable file, an installation package file, etc. Accordingly, the way in which the computer program instruction is executed by the computer includes, but is not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Here, the computer-readable medium may be any available computer-readable storage medium or communication medium that can be accessed by the computer.

[0323] Although the embodiments of the present invention have been described with reference to the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention. Such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A multimodal task interaction assistance method, characterized in that: The method comprises: Collect multimodal interaction data of users performing multi-dimensional interaction tasks; Determining a user behavior feature vector based on the multimodal interaction data, and constructing a multi-dimensional interaction task completion quantification model based on the behavior feature vector; Determine the weight of each operation step in the multi-dimensional interactive task using a dynamic weight allocation strategy based on the multi-dimensional interactive task completion quantification model, and dynamically adjust the preset adaptive coefficient using a preset adjustment mechanism based on the multimodal interaction data and the weight of each operation step; Based on the weight of each operation step and the dynamically adjusted preset adaptive coefficient, a dynamic path planning algorithm is used to generate a dynamic task guidance strategy; The dynamic task guidance strategy is executed to obtain a multimodal execution result.

2. The method according to claim 1, characterized in that The multimodal interaction data includes touch gesture recognition data, voice recognition data and sensor data; Determining the user behavior feature vector based on the multimodal interaction data includes: Based on the touch gesture recognition data, voice recognition data and sensor data, a deep learning algorithm is used to extract user operation behavior patterns and habits, identify user operation preferences and abnormal operation behaviors, and obtain a user behavior feature vector. The user behavior feature vector includes user operation habit features, operation proficiency features and error tendency features.

3. The method according to claim 1, characterized in that The multi-dimensional interactive task includes multiple operation steps; A multi-dimensional interactive task completion quantification model is constructed based on the user behavior feature vector, including: Based on the user behavior feature vector and multiple operation steps, a multi-dimensional interactive task completion metric model is constructed using a preset multi-dimensional task completion metric formula; The formula of the multi-dimensional interactive task completion quantification model is expressed as: Completion_Index=Σ(Step_Weight[i]×Success_Rate[i])+λ×log(Retry_Count+1); Among them, Step_Weight[i] represents the i-th weight, Success_Rate[i] represents the task completion success rate, λ×log(Retry_Count+1) represents the retry correction, λ represents the preset adaptive coefficient, and Retry_Count represents the retry behavior.

4. The method according to claim 2, characterized in that The method further comprises: Extracting time features, frequency features, and quality features from the user behavior feature vector; The pre-trained model is trained based on the time feature, frequency feature, and quality feature until a preset convergence condition is met, thereby obtaining a hybrid habit strength model; The user behavior pattern is predicted based on the hybrid habit strength model to obtain a prediction result, and the hybrid habit strength model is dynamically optimized based on the relationship between the prediction result and a preset threshold.

5. The method according to claim 1, wherein The method of determining the weight of each operation step in the multi-dimensional interactive task by using a dynamic weight allocation strategy based on the multi-dimensional interactive task completion quantification model includes: Calculate the operation complexity score of each operation step based on the number of operations, the number of interaction types, and cognitive load of each operation step; Calculate the overall task importance score of each operation step based on the user's task process dependency and failure impact of each operation step; Based on the operational complexity score of each operation step and the importance score of the overall task, a dynamic weight allocation strategy is used to determine the weight of each operation step in the multi-dimensional interactive task; The formula of the dynamic weight allocation strategy is expressed as: W[i]=(Complexity[i]×Importance[i]) / Σ(Complexity[j]×Importance[j]); Where W[i] represents the weight of the i-th operation step, Complexity[i] represents the operation complexity score of the i-th operation step, and Importance[i] represents the importance score of the i-th operation step to the overall task.

6. The method according to claim 2, characterized in that The dynamically adjusting the preset adaptive coefficient using a preset adjustment mechanism based on the multimodal interaction data and the weight of each operation step includes: Calculating a historical task completion rate, a cognitive load index, and an environmental adaptability index based on the multimodal interaction data; Calculating an adaptability factor based on the historical task completion rate, the cognitive load index, and the environmental adaptability index; Generate a preset adaptive coefficient calculation model based on the adaptive factor and the weight of each operation step; The preset adaptive coefficient is dynamically adjusted using a preset adjustment mechanism based on a preset adaptive coefficient calculation model; wherein the expression of the preset adaptive coefficient calculation model is: λ=λ_base+λ_range×(1-exp(-k×Adaptive_Factor)); Among them, λ_base represents the basic penalty coefficient, λ_range represents the dynamic adjustment range, k represents the sensitivity coefficient, and Adaptive_Factor represents the adaptive factor. Its expression is: AdaptiveFactor=w1×Success_Rate+w2×Cognitive_Load+w3×Environmental_Index; Among them, Success_Rate represents the task completion success rate, Cognitive_Load represents the cognitive load index, Environmental_Index represents the environmental adaptability index, w1, w2 and w3 represent the task completion success rate weight, cognitive load index weight and environmental adaptability index weight respectively.

7. The method according to claim 1, characterized in that The dynamic task guidance strategy is generated by using a dynamic path planning algorithm based on the weight of each operation step and a dynamically adjusted preset adaptive coefficient, including: Obtain the calculation results of the multi-dimensional interactive task completion quantification model and the preset requirements of the multi-dimensional interactive task; Based on the calculation results of the multi-dimensional interactive task completion quantification model, the preset requirements of the multi-dimensional interactive task, the weight of each operation step and the dynamically adjusted preset adaptive coefficient, a dynamic path planning algorithm is used to generate a dynamic task guidance strategy.

8. The method according to claim 1, characterized in that The executing the dynamic task guidance strategy to obtain a multimodal execution result includes: Generating a user visual guidance enhancement scheme based on the calculation results of the dynamic task guidance strategy and the multi-dimensional interactive task completion quantification model, wherein the user visual guidance enhancement scheme includes dynamic adjustment of interface elements, spacing, layout, interface contrast, and brightness of the user device; Generating a task guidance prompt when a user performs a multi-dimensional interactive task based on the calculation results of the dynamic task guidance strategy and the multi-dimensional interactive task completion quantification model, and generating a task progress tracking plan based on the task guidance prompt; Generate a multimodal feedback control scheme after the user performs each operation step in the multidimensional interactive task; The user visual guidance enhancement scheme, task progress tracking scheme and multimodal feedback control scheme are taken as multimodal execution results.

9. A multimodal task interaction assistance system, characterized in that: The system comprises: The perception layer is used to collect multimodal interaction data of users performing multi-dimensional interactive tasks; A decision layer, configured to determine a user behavior feature vector based on the multimodal interaction data, and generate a multidimensional interactive task completion quantification model based on the behavior feature vector; determine the weight of each operation step in the multidimensional interactive task using a dynamic weight allocation strategy based on the multidimensional interactive task completion quantification model, and dynamically adjust a preset retry correction coefficient using a dynamic adjustment mechanism based on the multimodal interaction data and the weight of each operation step; and generate a dynamic task guidance strategy using a dynamic path planning algorithm based on the weight of each operation step and the dynamically adjusted retry correction coefficient; The execution layer is used to execute the dynamic task guidance strategy to obtain a multimodal execution result.

10. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the multimodal task interaction assistance method according to any one of claims 1 to 8 by executing the computer instructions.

Images