Digital skill intelligent learning platform based on multi-modal interaction

Abstract

This invention discloses a digital skills intelligent learning platform based on multimodal interaction, comprising a multimodal input receiving unit, a cross-modal intent fusion engine, a multidimensional adaptive feedback system, and a digital skills knowledge graph. The platform collects and preprocesses various modal signals from the user through the multimodal input receiving unit; the cross-modal intent fusion engine dynamically allocates modal weights and identifies operational intentions; the multidimensional adaptive feedback system provides corresponding visual, tactile, and auditory feedback based on the user's error rate; and the digital skills knowledge graph generates personalized learning trajectories based on user interaction data. This invention achieves accurate recognition of learning intentions under multimodal interaction, provides an immersive learning experience, and improves the efficiency and user satisfaction of digital skills learning through data-driven personalized learning path planning, effectively solving problems such as single interaction, delayed feedback, and fixed learning paths in traditional digital skills learning.

Description

Technical Field

[0001] This invention relates to the field of digital skills learning technology, and in particular to a digital skills intelligent learning platform based on multimodal interaction. Background Technology

[0002] Although significant progress has been made in technologies such as artificial intelligence, multimodal interaction, and knowledge graphs, their comprehensive application in the field of digital skills learning is still in its early stages. Existing technical solutions are mostly focused on the application of single technologies and lack a systematic integration of multimodal perception, cross-modal fusion, adaptive feedback, and knowledge structured representation, thus failing to fully solve the complex problems in digital skills learning.

[0003] Traditional learning platforms lack intelligent intent understanding mechanisms, making it difficult to accurately identify user intentions. This is especially problematic in complex task scenarios, where users often need to perform cumbersome steps, increasing learning difficulty and frustration. When users use multiple input methods simultaneously, the system struggles to effectively resolve intermodal conflicts, leading to a higher rate of misjudgment. Existing platforms often employ pre-set, fixed learning paths, neglecting differences in user skill levels and learning characteristics. They fail to intelligently adjust learning content and difficulty based on real-time user performance, resulting in low learning efficiency.

[0004] Therefore, there is an urgent need for a digital skills intelligent learning platform that can comprehensively utilize multimodal interaction, cross-modal fusion, adaptive feedback and knowledge graph technologies to provide a personalized, efficient and socially recognized digital skills learning experience. Summary of the Invention

[0005] This invention provides a digital skills intelligent learning platform based on multimodal interaction, which addresses the problems of existing digital skills learning platforms, such as single interaction methods, lagging feedback mechanisms, rigid learning paths, and insufficient modal conflict recognition capabilities. It achieves intelligent fusion of multimodal inputs, real-time adaptive feedback, and data-driven personalized learning path planning, thereby improving the efficiency, experience, and effectiveness of digital skills learning.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] A digital skills intelligent learning platform based on multimodal interaction, comprising: a multimodal input receiving unit, a cross-modal intent fusion engine, a multidimensional adaptive feedback system, and a digital skills knowledge graph.

[0008] The multimodal input receiving unit is used to acquire and preprocess at least two modal signals from the user's voice command signal, gesture operation signal, eye tracking signal, and tactile input signal.

[0009] The cross-modal intent fusion engine includes a weight allocation unit and an operation intent recognition unit.

[0010] The weight allocation unit assigns dynamic weights to each input modality under the current learning task based on the user's historical interaction data, wherein the sum of the weights of each modality is 1 and all are non-negative values; the weights are calculated by comprehensively considering the historical accuracy of each modality, the task complexity parameter and the modality priority coefficient.

[0011] The operation intent recognition unit calculates the composite operation command based on each modal signal and its corresponding weight.

[0012] The multidimensional adaptive feedback system includes a visual feedback unit, a tactile feedback unit, and a voice feedback unit, and adjusts the combination of feedback modes according to the real-time error rate.

[0013] The digital skills knowledge graph includes a set of skill points, a set of edges connecting skill points, and a weight matrix of connection strength, which is used to generate personalized learning trajectories by combining user interaction behavior data.

[0014] Furthermore, the cross-modal intent fusion engine includes a modal conflict detection unit, used to identify contradictions between input commands from different modalities within the same time window, and to determine whether the distance between expected operation vectors of different modalities exceeds a preset conflict threshold by calculating the distance between them.

[0015] When there is a conflict between different modalities, the corresponding instructions are executed based on the priority of the modalities; the priority of the modalities is sorted from largest to smallest according to the user's historical usage preference for the modalities.

[0016] Furthermore, by calculating the distance D between the expected operating vectors of different modes... c (S i ,S j The specific method for determining whether a preset conflict threshold has been exceeded is as follows: using the function: D c (S i ,S j )=||O(S i )-O(S j ||, quantifies the distance between expected operation vectors of different modes within the same time window Δt, O(S i ) and O(S j ) represent the expected operation vectors for different modes i and j, respectively, when D c 9S i ,S j 0>θ c At that time, it is determined that there is a conflict between different modes.

[0017] θ c For the collision threshold parameter:

[0018] in, w represents the root mean square value of historical conflicts between mode i and mode j over the past n interactions; i and w j C represents the current weights of mode i and mode j, respectively; task C is a task complexity factor used to increase threshold tolerance in complex tasks. task Task complexity factor: E task B represents the estimated number of steps to complete the current learning task, ranging from 1 to 10. task This represents the task branching factor, i.e., the average number of operation choices per step, with a value ranging from 1 to 5; C task Taking into account both the length and breadth complexity of the task, the range is [0.1, 5].

[0019] α is the historical fluctuation coefficient, β is the weight influence coefficient, and γ is the task complexity coefficient;

[0020]

[0021] Where, λ α =0.05, which is the learning rate parameter; N user This represents the number of times a user interacts with the system, increasing with the user's experience. The value ranges from [0.5, 0.8]. New users start with a value of 0.5, and as user experience grows, the system gradually increases its reliance on the user's historical behavior patterns. N user The value gradually increases; ACC avg This represents the average recognition accuracy across all modalities, with a value ranging from [0.2, 0.4]; D task This represents the depth level of the current task in the digital skills knowledge graph. This coefficient increases with the task complexity and ranges from [0, 0.3].

[0022] Furthermore, the weight allocation unit dynamically allocates the weights of each input mode in the following manner:

[0023] For the input mode set M = m1, m2, ..., m k The corresponding weight vector is W = w1, w2, ..., w k Satisfying the constraints And w s ≥0, k is the total number of input modes; the weight calculation model is defined as: Wherein, φ(m) s ) is mode m s The comprehensive scoring function.

[0024] Among them, ACC s Represents mode ms The historical accuracy rate, with values ​​ranging from [0,1]; P s Represents mode m s The priority coefficient is determined by user preferences and modality applicability, and its value ranges from [0,1]; E s Represents mode m s The error rate is obtained by analyzing the most recent 100 input modalities; the system also sets the weight adjustment frequency: Indicates each execution of f a The weight allocation is updated once after each operation.

[0025] Furthermore, the operation intent recognition unit calculates the composite operation command in the following manner:

[0026] D c (S i ,S j )≤θ c When it is determined that there is no conflict between different modes, the signal set corresponding to the input mode set M is: S = S1, S2, ..., S k .

[0027] For each modal signal S s Through the feature extraction function: Extract n s 3D feature vector;

[0028] Through the mode-specific mapping function: Φ s :F s →O s , will mode m s eigenvector F s Mapping to the operation vector space O s , where Φ s (F s )=σ(W s ·F s +b s ), W s and b s These are the mapping weight matrix and the bias vector, respectively, and σ is the activation function.

[0029] Unified operation vector space transformation function T s :O s →O u This transforms operation vectors from different modalities into a unified operation vector space, ensuring dimensionality consistency.

[0030] Compound operation instruction O co The calculation formula is: Through the decision function D(O) co → A, will perform a compound operation instruction O coMapped to the final operation instruction A, where the decision function D(O) co ) is represented as: Among them, V A Let A be the standard operation vector. <O co V A > represents the inner product operation of two vectors; This refers to the set of all operation commands supported by the platform.

[0031] Furthermore, the visual feedback unit displays dynamic prompts and error operation markers on the screen, providing graphical guidance when the user's operation deviates from the expected path, including color change prompts, trajectory comparison display, and visualization of step completion.

[0032] The tactile feedback unit transmits information on the correctness of operation through vibration and force feedback devices, adjusts the vibration frequency and intensity according to the accuracy of operation, provides tactile confirmation at key operation points, and simulates the correct operation feel through force feedback.

[0033] The voice feedback unit provides real-time guidance and error correction suggestions through voice commands, uses natural language to analyze user confusion points, provides targeted voice prompts according to error types, and provides encouraging voice feedback when completing phased tasks;

[0034] The system continuously monitors the user's operation error rate. When the error rate exceeds the preset threshold, it automatically increases the feedback intensity and dynamically adjusts the proportion of each modal feedback according to different task types and operation stages to obtain the adjusted feedback modal combination.

[0035] Furthermore, the adjustment model for the combination of feedback modes in the multidimensional adaptive feedback system is as follows: Let the current error rate be E(t), the system calculates the feedback weight vector W(t) = [wv(t), wt(t), ws(t)] for each mode, where wv(t), wt(t), and ws(t) are the weight values ​​of visual, tactile, and speech feedback, respectively; when E(t) > θE, the weight vector W(t+1) = W(t) + h·(E(t) - θE)·ΔW is updated, where θE is the error rate threshold, h = 0.05, and ΔW is the gradient vector.

[0036] Furthermore, the method for generating personalized learning trajectories by combining the digital skill knowledge graph with user interaction behavior data is as follows: the system records user interaction behavior data during skill point practice, including completion time, number of errors, operation fluency, and modal usage preferences; analyzes the user's mastery level of different skill points and identifies weak points; calculates the optimal learning path based on the correlation strength weight matrix between skill points to ensure a reasonable learning order for relevant skill points; dynamically adjusts the difficulty of learning content and recommended practice methods according to the user's skill mastery characteristics; and predicts skill development trends by combining the user's historical learning trajectory and proactively recommends learning goals for the next stage.

[0037] Furthermore, the model for generating the personalized learning trajectory is as follows:

[0038] Define a user skill state vector H(u) = [h1, h2, ...], where h i It represents the user's mastery of skill point i, with a value range of [0,1]. Based on the user interaction behavior feature set F(u)=v1,v2,v3,v4, the skill state H is predicted by a deep neural network function.

[0039] The user interaction behavior feature set F(u) = v1, v2, v3, v4; v1 is the operation speed index, v2 is the operation accuracy rate, v3 is the number of repeated attempts, and v4 is the frequency of asking for help.

[0040] The set of optional learning tasks is represented as: B = b1, b2, ..., and the transition function is P(H′|H, b), which represents the probability that the user will transition to state H′ after performing action b from skill state H; the reward function is R(H, b) = ΔH; ΔH is the skill improvement amount; the optimization strategy is achieved through iteration: Where H′ represents the new state after performing action b. This represents the maximum Q value under the new state.

[0041] Personalized learning trajectories are generated based on the knowledge graph structure and user interaction data.

[0042] Where, π * (H) represents the optimal personalized learning trajectory strategy, i.e., the action to be selected in state H; E b Let w_{ij} be the set of skill points involved in action b; w_{ij} is the association strength between skill point i and skill point j; μ is the knowledge graph structure influence factor.

[0043] Furthermore, the platform sets up a skills certification and social incentive mechanism, issuing skills NFT certificates after users complete a specific skills learning path.

[0044] The method for generating skill NFT credentials is as follows: The platform uses an anti-tampering skill proof generator, through the function N(u,s,a)=Ha(E(I u ||T s ||D a ||r)) Generates skill NFT credentials, where I u For user identification identifier, I u For the skills assessment result tensor, D a Let r be the hash value of the operation data, r be a random number, and E be the asymmetric encryption function E(m) = m. e modn, Ha is the SHA-256 hash function, and || represents the concatenation operation.

[0045] Compared with the prior art, the beneficial effects of this invention are:

[0046] This invention can dynamically adjust the weights of different modalities based on users' historical interaction data, and resolve potential contradictions in multimodal input through a precise conflict detection and processing mechanism, ensuring the accuracy of operation intent recognition. It intelligently adjusts the combination of feedback modalities according to the user's real-time error rate, providing immediate and accurate operation guidance and enhancing the immersive learning experience. It achieves intelligent fusion of multimodal input, real-time adaptive feedback, and data-driven personalized learning path planning, thereby improving the efficiency of digital skills learning and has significant promotional value. Attached Figure Description

[0047] Figure 1 This is a schematic diagram of the components of the digital skills intelligent learning platform based on multimodal interaction according to the present invention. Detailed Implementation

[0048] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention are described clearly and completely below. Obviously, the described embodiments are only some embodiments of this invention, not all embodiments. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0049] It should be noted that the multimodal interaction paradigm of this invention is not only applicable to digital skills learning, but can also be extended to multiple fields such as professional skills training, distance education, and assisted learning for special groups. The technical solution in this invention relies on modern artificial intelligence, knowledge graphs, reinforcement learning, and multimodal interaction technologies to achieve an intelligent, personalized, and immersive learning experience.

[0050] The components of the present invention and their working principles will now be described in detail with reference to the accompanying drawings.

[0051] like Figure 1As shown, this invention presents a digital skills intelligent learning platform based on multimodal interaction. The platform includes: a multimodal input receiving unit, a cross-modal intent fusion engine, a multidimensional adaptive feedback system, and a digital skills knowledge graph.

[0052] The multimodal input receiving unit is used to acquire and preprocess at least two modal signals from the user's voice command signal, gesture operation signal, eye tracking signal, and tactile input signal.

[0053] The multimodal input receiving unit adopts a multi-channel parallel processing architecture to acquire and preprocess different modal signals of user input. Voice command signals are acquired through a microphone array and then processed for noise reduction, signal enhancement, and voice activity detection; gesture operation signals are acquired through a depth camera and then processed for skeletal point extraction and motion recognition; eye-tracking signals are acquired through an infrared eye tracker to acquire the user's gaze point and pupil changes; tactile input signals are detected through pressure sensors and a touchscreen to detect the user's physical contact operations; each modal signal uses an independent preprocessing pipeline to ensure signal quality and real-time processing; for mobile device scenarios, the system automatically trims and adjusts the types of received modalities based on available sensors.

[0054] The cross-modal intent fusion engine includes a weight allocation unit and an operation intent recognition unit. The system maintains a user modality usage history database, recording the usage frequency, accuracy, and operation efficiency of different modalities in various learning tasks. The weight allocation adopts a sliding window method, which considers both long-term historical performance and recent interaction trends, and ensures the smoothness and dynamic adaptability of weight adjustment through weighted averaging. For new users, the system adopts a cold start strategy, which is initialized based on the average modality preferences of demographically similar users.

[0055] The weight allocation unit assigns dynamic weights to each input modality under the current learning task based on the user's historical interaction data, wherein the sum of the weights of each modality is 1 and all are non-negative values; the weights are calculated by comprehensively considering the historical accuracy of each modality, the task complexity parameter and the modality priority coefficient.

[0056] The weight allocation unit dynamically allocates the weights of each input mode in the following manner:

[0057] For the input mode set M = m1, m2, ..., m k The corresponding weight vector is W = w1, w2, ..., w k Satisfying the constraints And w s ≥0, k is the total number of input modes; the weight calculation model is defined as: Wherein, φ(m) s ) is mode m sA comprehensive scoring function;

[0058] Among them, ACC s Represents mode m s The historical accuracy rate, with values ​​ranging from [0,1]; P s Represents mode m s The priority coefficient is determined by user preferences and modality applicability, and its value ranges from [0,1]; E s Represents mode m s The error rate is obtained by analyzing the most recent 100 input modalities; the system also sets the weight adjustment frequency: Indicates each execution of f a The weight allocation is updated once after each operation.

[0059] The operation intent recognition unit calculates the composite operation command based on each modal signal and its corresponding weight.

[0060] The operation intent recognition unit calculates the composite operation command in the following manner:

[0061] D c (S i ,S j )≤θ c When it is determined that there is no conflict between different modes, the signal set corresponding to the input mode set M is: S = S1, S2, ..., S k ;

[0062] For each modal signal S s Through the feature extraction function: Extract n s 3D feature vector;

[0063] Through the mode-specific mapping function: Φ s :F s →O s , will mode m s eigenvector F s Mapping to the operation vector space O s , where Φ s (F s )=σ(W s ·F s +b s ), W s and b s These are the mapping weight matrix and the bias vector, respectively, and σ is the activation function;

[0064] Unified operation vector space transformation function T s :O s →O uThis transforms operation vectors from different modalities into a unified operation vector space, ensuring dimensionality consistency.

[0065] Compound operation instruction O co The calculation formula is: Through the decision function D(O) co → A, will perform a compound operation instruction O co Mapped to the final operation instruction A, where the decision function D(O) co ) is represented as: Among them, V A Let A be the standard operation vector. <O co V A > represents the inner product operation of two vectors; This refers to the set of all operation commands supported by the platform.

[0066] The cross-modal intent fusion engine includes a modal conflict detection unit, which is used to identify contradictions between input commands of different modalities within the same time window and to determine whether the distance between expected operation vectors of different modalities exceeds a preset conflict threshold by calculating the distance between them.

[0067] The modal conflict detection unit adopts a two-level detection architecture.

[0068] The first level is timing detection, which identifies concurrent modal commands through a configurable time window; the second level is semantic detection, which analyzes the operational intent of different modal commands. Figure 1 To the point of being responsive.

[0069] The conflict threshold parameter is dynamically adjusted based on the user's proficiency. Expert users set a lower threshold to accurately capture subtle conflicts, while beginners set a higher threshold to improve fault tolerance. When a modal conflict is detected, the system not only executes the dominant modal instruction based on priority, but also explains the conflict situation to the user through a multi-dimensional feedback system to help the user understand and adjust their operation.

[0070] When there is a conflict between different modalities, the corresponding instructions are executed based on the priority of the modalities; the priority of the modalities is sorted from largest to smallest according to the user's historical usage preference for the modalities.

[0071] By calculating the distance D between the expected operation vectors of different modalities c (S i ,S j The specific method for determining whether a preset conflict threshold has been exceeded is as follows: using the function: D c (S i ,S j )=||O(S i )-O(S j ||, quantifies the distance between expected operation vectors of different modes within the same time window Δt, O(Si ) and O(S j ) represent the expected operation vectors for different modes i and j, respectively, when D c (S i ,S j )>θ c At that time, it is determined that there is a conflict between different modes;

[0072] θ c For the collision threshold parameter:

[0073] in, w represents the root mean square value of historical conflicts between mode i and mode j over the past n interactions; i and w j C represents the current weights of mode i and mode j, respectively; task C is a task complexity factor used to increase threshold tolerance in complex tasks. task Task complexity factor: E task B represents the estimated number of steps to complete the current learning task, ranging from 1 to 10. task This represents the task branching factor, i.e., the average number of operation choices per step, with a value ranging from 1 to 5; C task Taking into account both the length and breadth complexity of the task, the range is [0.1, 5];

[0074] α is the historical fluctuation coefficient, β is the weight influence coefficient, and γ is the task complexity coefficient;

[0075]

[0076] Where, λ α =0.05, which is the learning rate parameter; N user This represents the number of times a user interacts with the system, increasing with the user's experience. The value ranges from [0.5, 0.8]. New users start with a value of 0.5, and as user experience grows, the system gradually increases its reliance on the user's historical behavior patterns. N user The value gradually increases; ACC avg This represents the average recognition accuracy across all modalities, with a value ranging from [0.2, 0.4]; D task This represents the depth level of the current task in the digital skills knowledge graph. This coefficient increases with the task complexity and ranges from [0, 0.3].

[0077] The multidimensional adaptive feedback system includes a visual feedback unit, a tactile feedback unit, and a voice feedback unit, adjusting the combination of feedback modes based on the real-time error rate. The visual feedback unit displays dynamic prompts and error operation markers on the screen, providing graphical guidance when the user's operation deviates from the expected path, including color-changing prompts, trajectory comparison displays, and visualization of step completion. The tactile feedback unit transmits operation correctness information through vibration and force feedback devices, adjusting the vibration frequency and intensity according to the operation accuracy, providing tactile confirmation at key operation points, and simulating the correct operation feel through force feedback. The voice feedback unit provides real-time guidance and error correction suggestions through voice commands, uses natural language to analyze user confusion points, provides targeted voice prompts based on error type, and provides encouraging voice feedback when completing phased tasks.

[0078] The speech modality is processed using a dual-path approach, combining acoustic and semantic features; the gesture modality combines static pose features and dynamic trajectory features; the eye-tracking modality analyzes gaze point distribution and shift patterns; and the tactile modality extracts pressure, orientation, and duration features. The feature vectors are transformed into a unified operation vector space via a pre-trained modality-specific mapping network, achieving cross-modal semantic alignment.

[0079] The system continuously monitors the user's operation error rate. When the error rate exceeds the preset threshold, it automatically increases the feedback intensity and dynamically adjusts the proportion of each modal feedback according to different task types and operation stages to obtain the adjusted feedback modal combination.

[0080] The adjustment model for the combination of feedback modes in a multidimensional adaptive feedback system is as follows: Let the current error rate be E(t). The system calculates the feedback weight vector W(t) = [wv(t), wt(t), ws(t)] for each mode, where wv(t), wt(t), and ws(t) are the weight values ​​of visual, tactile, and speech feedback, respectively. When E(t) > θE, the weight vector W(t+1) = W(t) + h·(E(t) - θE)·ΔW is updated, where θE is the error rate threshold, h = 0.05, and ΔW is the gradient vector.

[0081] The digital skills knowledge graph includes a set of skill points, a set of edges connecting skill points, and a weight matrix of connection strength, which is used to generate personalized learning trajectories by combining user interaction behavior data.

[0082] The method for generating personalized learning trajectories by combining the digital skills knowledge graph with user interaction behavior data is as follows: The system records user interaction behavior data during skill point practice, including completion time, number of errors, operation fluency, and modal usage preferences; analyzes the user's mastery level of different skill points and identifies weak points; calculates the optimal learning path based on the correlation strength weight matrix between skill points to ensure a reasonable learning order for related skill points; dynamically adjusts the difficulty of learning content and recommended practice methods according to the user's skill mastery characteristics; and predicts skill development trends by combining the user's historical learning trajectory and proactively recommends learning goals for the next stage.

[0083] The model for generating the personalized learning trajectory is:

[0084] Define a user skill state vector H(u) = [h1, h2, ...], where h i It represents the user's mastery of skill point i, with a value range of [0,1]. Based on the user interaction behavior feature set F(u)=v1,v2,v3,v4, the skill state H is predicted by a deep neural network function.

[0085] The user interaction behavior feature set F(u) = v1, v2, v3, v4; v1 is the operation speed index, v2 is the operation accuracy rate, v3 is the number of repeated attempts, and v4 is the frequency of asking for help.

[0086] The set of optional learning tasks is represented as: B = b1, b2, ..., and the transition function is P(H′|H, b), which represents the probability that the user will transition to state H′ after performing action b from skill state H; the reward function is R(H, b) = ΔH; ΔH is the skill improvement amount; the optimization strategy is achieved through iteration: Where H′ represents the new state after performing action b. The maximum Q value in the new state;

[0087] Personalized learning trajectories are generated based on the knowledge graph structure and user interaction data.

[0088] Where, π * (H) represents the optimal personalized learning trajectory strategy, i.e., the action to be selected in state H; E b Let w_{ij} be the set of skill points involved in action b; w_{ij} is the association strength between skill point i and skill point j; μ is the knowledge graph structure influence factor.

[0089] The platform sets up a skills certification and social incentive mechanism, and issues skills NFT certificates after users complete a specific skills learning path;

[0090] The method for generating skill NFT credentials is as follows: The platform uses an anti-tampering skill proof generator, through the function N(u,s,a)=Ha(E(I u ||T s ||D a ||r)) Generates skill NFT credentials, where I u For user identification identifier, I u For the skills assessment result tensor, D a Let r be the hash value of the operation data, r be a random number, and E be the asymmetric encryption function E(m) = m. e modn, Ha is the SHA-256 hash function, and || represents the concatenation operation.

[0091] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A digital skill intelligent learning platform based on multi-modal interaction, characterized in that, The platform comprises a multi-modal input receiving unit, a cross-modal intention fusion engine, a multi-dimensional adaptive feedback system and a digital skill knowledge graph; The multi-modal input receiving unit is used for collecting and pre-processing at least two modal signals among voice instruction signals, gesture operation signals, eye movement tracking signals and tactile input signals of a user; The cross-modal intention fusion engine comprises a weight distribution unit and an operation intention recognition unit; The weight distribution unit assigns dynamic weights to each input modality for a current learning task based on user historical interaction data, wherein the sum of the weights of each modality is 1 and each weight is non-negative; the weights are calculated by comprehensively considering the historical accuracy rate, task complexity parameter and modality priority coefficient of each modality; The operation intention recognition unit calculates a composite operation instruction according to each modality signal and its corresponding weight; The cross-modal intention fusion engine comprises a modality conflict detection unit for identifying conflicts between different modality input instructions within the same time window, and determining whether the distance between different modality expected operation vectors exceeds a preset conflict threshold; by calculating the distance between the expected operation vectors of different modalities The specific method for determining whether the preset conflict threshold is exceeded is: by a function: , quantifying the distance between the expected operation vectors of different modalities in the same time window, and respectively represent the expected operation vectors of different modalities and , and when , it is determined that there is a conflict between different modalities;​ to a preset conflict threshold: ; wherein, represents the modality and the modality In the past the historical conflict root mean square value in the interaction; and are the current weights of the modalities and the modality respectively; is a task complexity factor for increasing threshold tolerance in complex tasks, , represents the estimated number of steps to complete the current learning task, with a value range of 1-10, represents the task branch factor, i.e., the average number of operation choices per step, with a value range of 1-5; comprehensively considers the length and width complexity of the task, with a range of [0.1, 5]; is a historical fluctuation coefficient, is a weight influence coefficient, is a task complexity coefficient; The multi-dimensional adaptive feedback system comprises a visual feedback unit, a tactile feedback unit and a voice feedback unit, which adjusts the feedback modality combination according to the real-time error rate; The digital skill knowledge graph contains a skill point set, a set of associated edges between skill points and an associated strength weight matrix, which is used to generate a personalized learning track in combination with user interaction behavior data.

2. The multi-modal interaction based digital skill intelligent learning platform as claimed in claim 1, wherein, When there is a conflict between different modalities, the corresponding instructions are executed based on the priority order of the modalities; the priority order of the modalities is sorted in descending order according to the historical use preference of the user for the modalities.

3. The multi-modal interaction based digital skill intelligent learning platform according to claim 2, wherein, , , Specifically: ； ； ； wherein, = 0.05, is a learning rate parameter; represents the number of interactions of the user in the system, which increases with the user's experience in using the system, and the value range is [0.5, 0.8]; the initial value of a new user is 0.5, and the system gradually increases the dependence on the historical behavior pattern of the user as the user's experience increases, the value gradually increases; represents the average recognition accuracy of all modalities, and the value range is [0.2, 0.4]; represents the depth level of the current task in the digital skill knowledge graph, and the coefficient increases with the increase of the task complexity, and the range is [0, 0.3].

4. The multi-modal interaction based digital skill intelligent learning platform as claimed in claim 3, wherein, The weight distribution unit dynamically assigns weights to each input modality by the following method: For a set of input modalities , the corresponding weight vector is , satisfying the constraint and ; is the total number of input modalities; The weight calculation model is defined as: wherein, is a combined score function for the modality ; wherein, denotes the history accuracy rate of the modality , which ranges from 0 to 1; denotes the priority coefficient of the modality , which is determined by user preference and modality applicability, and ranges from 0 to 1; denotes the error rate of the modality , which is calculated based on the last 100 inputs; the system also sets the weight adjustment frequency: , which means that the weight distribution is updated once every operations.

5. The multi-modal interaction based digital skill intelligent learning platform as claimed in claim 4, wherein, The operation intention recognition unit calculates a composite operation instruction by the following method: At this time, it is determined that there is no conflict between different modalities, and the input modality set The corresponding signal set is: ; for each modality signal by a feature extraction function: extracts a d-dimensional feature vector; By a modality-specific mapping function: the feature vector of a modality is mapped to an action vector space where , and are a mapping weight matrix and a bias vector, respectively, is an activation function; Unified operation vector space conversion function Converting operation vectors of different modalities to a unified operation vector space ensures dimensional consistency. Composite operation instruction The calculation formula is: ; through the decision function , the composite operation instruction is mapped to the final operation instruction , wherein the decision function is expressed as: , wherein is the operation corresponding to the standard operation vector, is the inner product operation of two vectors; , is the set of all operation instructions supported by the platform.

6. The multi-modal interaction based digital skill intelligent learning platform according to claim 5, wherein, The visual feedback unit provides dynamic prompt information and error operation markers through screen display, and provides graphical guidance when the user's operation deviates from the expected path, including color transformation prompts, trajectory comparison displays and step completion visualization; The tactile feedback unit transmits operation correctness information through vibration and force feedback devices, adjusts the vibration frequency and intensity according to the operation accuracy, provides tactile confirmation at key operation points, and simulates the correct operation feel through force feedback; The voice feedback unit provides real-time guidance and error correction suggestions through voice instructions, uses natural language to analyze user confusion points, provides targeted voice prompts according to error types, and gives encouraging voice feedback when completing a phased task; The system continuously monitors the user operation error rate, automatically enhances the feedback intensity when the error rate exceeds a preset threshold, dynamically adjusts the feedback proportion of each modality according to different task types and operation stages, and obtains an adjusted feedback modality combination.

7. The multi-modal interaction based digital skill intelligent learning platform according to claim 6, wherein, The adjustment model of feedback mode combination in multi-dimensional adaptive feedback system is: let the current error rate be , the system calculates the feedback weight vector of each mode , wherein , , are the weight values of visual, tactile and voice feedback respectively; when , the weight vector is updated , wherein θE is the error rate threshold, , is the gradient vector.

8. The multi-modal interaction based digital skill intelligent learning platform as claimed in claim 1, wherein, The method for the digital skill knowledge graph to generate a personalized learning track in combination with user interaction behavior data is as follows: the system records user interaction behavior data during skill point practice, including completion time, error times, operation fluency and modality use preference; Analyze the user's mastery of different skill points and identify weak skill points; Based on the associated strength weight matrix between skill points, calculate the optimal learning path to ensure that the learning order of related skill points is reasonable; Adjust the difficulty of learning content and recommend exercise methods dynamically according to the skill mastering characteristics of users; predict the skill development trend by combining the historical learning track of users, and actively recommend the next stage learning goal.

9. The multi-modal interaction based digital skill intelligent learning platform according to claim 8, wherein, The model for generating the personalized learning track is: Defining a user skill state vector wherein, represents the user's mastery of a skill point , with a value range [0, 1]; based on the user interaction behavior feature set , the skill state is predicted by a deep neural network function ; the set of household interaction behavior features ; for an operation speed index, for an operation accuracy rate, for a number of repeated attempts, for a help-seeking frequency; The set of optional learning tasks is denoted as , and the transition function is , representing the probability of the user transitioning from skill state to state after performing action ; the reward function ; is the amount of skill improvement; the policy is optimized iteratively as , where is the new state after performing action , and is the maximum value for the new state. Generate a personalized learning track according to the knowledge graph structure and user interaction behavior data: ; wherein, is the optimal personalized learning trajectory policy, i.e., the action that should be chosen at state ; is the skill point involved in the action ; is the association strength of the skill point to the skill point is the knowledge graph structure influence factor.

10. The multi-modal interaction based digital skill intelligent learning platform as claimed in claim 1, wherein, The platform sets up a skill authentication and social incentive mechanism, and issues a skill NFT certificate after the user completes a specific skill learning path; The generation method of the skill NFT certificate is: the platform generates the skill NFT certificate based on the tamper-proof skill proof generator, through the function , is the user identity identifier, is the skill evaluation result tensor, is the operation behavior data hash value, is a random number, is an asymmetric encryption function , is a SHA-256 hash function, and || represents a concatenation operation.

Images