Learning Module Multimodal Data Processing Method and Device
By extracting the embedded features of learners' multimodal data and performing quality assessment and dynamic weight allocation, the problem of inaccurate learning state prediction is solved, enabling accurate identification and real-time intervention of learners' states.
Patent Information
- Application Number
- CN202611432045.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-09-15
- Publication Date
- 2026-11-17
AI Technical Summary
Existing technologies struggle to accurately identify the contribution ratio of learners' multimodal data, leading to inaccurate predictions of learning status and difficulty in providing a basis for real-time intervention.
By extracting the embedding features of multimodal data and performing quality assessment, calculating individual baseline offset features and confidence levels, constructing a dynamic modality conflict map, dynamically assigning weights and performing fusion feature recognition, and using a hierarchical reinforcement learning model for weight constraints and adjustments.
It enables accurate identification of learners' learning status, improves the accuracy of learning status prediction results, reduces interference from low-quality data, and provides a basis for real-time intervention.
Smart Images

Figure CN123115872A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of modal recognition technology, and in particular to a method and apparatus for processing multimodal data in a learning cabin. Background Technology
[0002] In the current learning environment, the analysis of learners' status during the learning process is no longer limited to outcome data such as academic performance, correct answer rate, learning time, and task completion rate. Traditional learning evaluation methods usually judge learners' learning effectiveness based on test scores, homework completion, or teachers' subjective observations after the learning activity. This method is difficult to detect problems such as decreased attention, emotional fluctuations, anxiety, fatigue, and excessive cognitive load that occur during the learning process, and it is also difficult to provide a basis for real-time intervention for the system.
[0003] In related technologies, multimodal learning state recognition methods typically output learning state, attentional state, cognitive load, or learning ability assessment results. These methods struggle to explain the contribution ratio of different modalities, making it difficult to determine which type of data is primarily relied upon for current learning. Teachers or learning systems also find it difficult to understand the reasons behind a particular state judgment. Furthermore, multimodal fusion methods using feature concatenation, deep neural network fusion, or recurrent neural networks usually set fixed weights to construct fused features. Even when using Transformers or attention mechanisms, the features are not explicitly constructed with dynamic weights that integrate constraints such as individual baselines, data quality, and modal conflicts. This lack of detail fails to reflect the learner's true learning situation, leading to inaccurate learning state predictions. Summary of the Invention
[0004] This invention provides a multimodal data processing method and apparatus for a learning cabin, which solves the shortcomings of existing technologies in multimodal learning state identification, such as difficulty in explaining the contribution ratio of different modal data and difficulty in representing the learner's actual learning situation, leading to inaccurate learning state prediction results.
[0005] This invention provides a method for processing multimodal data in a learning cabin, comprising: The embedding features corresponding to each modality are extracted from the multimodal data generated by learners performing learning tasks in the learning pod, and the quality of the multimodal data is evaluated to obtain the evaluation results. Based on the embedded features and the individual baselines associated with the corresponding modalities, individual baseline offset features corresponding to different modalities are obtained; the learning states of the learners are identified using the embedded features corresponding to each modality, resulting in multiple single-modal recognition results, and the confidence level corresponding to each single-modal recognition result is calculated; Based on the multiple single-modal recognition results and their corresponding confidence levels, the evaluation results corresponding to different modal data, and individual baseline offset features, a logical relationship between different modal data is established to obtain a dynamic modal conflict map and a multimodal relationship representation; The first weight is assigned to the multimodal data based on the individual baseline offset features and the multimodal relationship representation, and the first weight is constrained by the evaluation results of the corresponding modal data and the dynamic modal conflict diagram to obtain the second weight; The fusion features are calculated based on the embedding features and second weights corresponding to different modal data, and the learning state of the learner is identified using the fusion features to obtain the fusion identification result.
[0006] According to the multimodal data processing method for a learning cabin provided by the present invention, the step of establishing a logical relationship between different modal data based on the multiple single-modal recognition results and corresponding confidence levels, the evaluation results corresponding to different modal data, and individual baseline offset features includes: Calculate the similarity and prediction difference between any two single-modal recognition results in the state dimension among the multiple single-modal recognition results; If the similarity is greater than the first threshold, and the evaluation results and confidence scores of the modal data corresponding to any two single-modal recognition results meet the preset conditions, the logical relationship between the modal data corresponding to any two single-modal recognition results is determined to be a support relationship. When the prediction difference is greater than the second threshold, and the confidence level of at least one of the modal data corresponding to any two single-modal recognition results satisfies the preset condition, the logical relationship between the modal data corresponding to any two single-modal recognition results is determined to be a conflict relationship.
[0007] According to a learning cabin multimodal data processing method provided by the present invention, the step of assigning a first weight to the multimodal data based on the individual baseline offset features and the multimodal relationship representation includes: A state space is constructed based on the individual baseline offset features, multimodal relationship representation, and current learning task context information; The target fusion pattern for the current time window is determined using the upper-layer policy network of the hierarchical reinforcement learning model and the state space. The original importance scores of each modality data are calculated using the lower-level weight allocation network of the hierarchical reinforcement learning model, the state space, and the target fusion mode. The multiple original importance scores are then normalized to obtain the first weight.
[0008] According to a learning cabin multimodal data processing method provided by the present invention, the step of constraining the first weight using the evaluation results of the corresponding modal data and the dynamic modal conflict graph includes: If the evaluation result of one of the modal data is lower than a preset quality threshold, the upper limit of the first weight corresponding to the one modal data shall be adjusted not to exceed the weight threshold. If a modality is marked as missing, the first weight corresponding to that modality is set to zero. When a modal data has a conflict relationship with multiple modal data in the dynamic modal conflict graph, and the evaluation result of the modal data is lower than a preset quality threshold, the first weight corresponding to the modal data is reduced to a preset weight value.
[0009] According to the multimodal data processing method for a learning cabin provided by the present invention, the first weight is determined by a hierarchical reinforcement learning model; After obtaining the fusion recognition result, the method further includes: The target modal data in the multimodal data is masked, zeroed out, or replaced with the baseline to obtain the multimodal data after intervention. The multimodal data after the intervention is used to re-fuse features and identify them to obtain a comparative identification result. Based on the change in identification loss between the fused identification result and the comparative identification result, the counterfactual contribution of the target modality data in the current identification is calculated. A reinforcement learning reward function is constructed based on the counterfactual contribution and delayed feedback information corresponding to each modality data; the delayed feedback information is determined by the learner's answer performance, task completion time, or evaluation feedback in subsequent learning tasks; The reward value is calculated using the reinforcement learning reward function, and the policy parameters of the hierarchical reinforcement learning model are updated based on the reward value.
[0010] According to the present invention, a multimodal data processing method for a learning cabin includes at least two of the following: electrocardiogram data, eye-tracking data, video data, voice data, and learning behavior data; the logical relationships include at least one of the following: support relationships, conflict relationships, complementary relationships, and missing relationships. After obtaining the fusion recognition result, the method further includes: Obtain modal contribution descriptions; the modal contribution descriptions include the fusion recognition results, the second weight percentages corresponding to each modal data, and the logical relationships in the dynamic modal conflict graph.
[0011] The present invention also provides a learning cabin multimodal data processing device, comprising: The feature extraction and evaluation module is used to extract the embedded features corresponding to each modality from the multimodal data generated by learners performing learning tasks in the learning cabin, and to evaluate the quality of the multimodal data to obtain the evaluation results. The single-modality recognition module is used to obtain individual baseline offset features corresponding to different modalities based on the embedded features and the individual baselines associated with the corresponding modalities; to identify the learning state of the learner using the embedded features corresponding to each modality, to obtain multiple single-modality recognition results, and to calculate the confidence level corresponding to each single-modality recognition result. The multimodal relationship establishment module is used to establish logical relationships between different modal data based on the multiple single-modal recognition results and corresponding confidence levels, the evaluation results corresponding to different modal data, and individual baseline offset features, to obtain a dynamic modal conflict map and a multimodal relationship representation; The weight calculation module is used to assign a first weight to the multimodal data based on the individual baseline offset features and the multimodal relationship representation, and to constrain the first weight using the evaluation results of the corresponding modal data and the dynamic modal conflict diagram to obtain a second weight; The comprehensive recognition module is used to calculate fusion features based on the embedding features and second weights corresponding to different modal data, and to use the fusion features to identify the learner's learning state to obtain the fusion recognition result.
[0012] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the learning cabin multimodal data processing method as described above.
[0013] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the learning cabin multimodal data processing method as described above.
[0014] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the learning cabin multimodal data processing method as described above.
[0015] The learning cabin multimodal data processing method and apparatus provided by this invention extracts the embedding features of multimodal data and performs quality assessment, providing data support for preventing inferior data from participating in fusion decisions. It eliminates the risk of misjudgment caused by individual learner differences by calculating individual baseline offset features, and clarifies the judgment tendency of each sensor at the current moment by calculating the single-modal recognition results and confidence levels. By constructing a dynamic modal conflict graph to represent the logical relationship between different modal data, and by dynamically allocating the first weight and generating the second weight using rule constraints, it reduces the interference of low-quality data or superficial deceptive data, achieving dynamic weight decision-making. Finally, it accurately identifies the learner's learning state through fusion features, realizing the dynamic integration and state assessment of multimodal data in the learning cabin, and improving the accuracy of learning state prediction results. Attached Figure Description
[0016] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.
[0017] Figure 1 This is one of the flowcharts of the learning cabin multimodal data processing method provided by the present invention.
[0018] Figure 2 This is a schematic diagram of dynamic modal conflict relationships provided by the present invention.
[0019] Figure 3 This is a flowchart of the hierarchical reinforcement learning weight allocation process provided by the present invention.
[0020] Figure 4 This is a flowchart of the counterfactual contribution and delayed feedback update provided by the present invention.
[0021] Figure 5 This is the second flowchart of the learning cabin multimodal data processing method provided by the present invention.
[0022] Figure 6 This is one of the structural schematic diagrams of the learning cabin multimodal data processing device provided by the present invention.
[0023] Figure 7 This is the second structural schematic diagram of the learning cabin multimodal data processing device provided by the present invention.
[0024] Figure 8 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation
[0025] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.
[0026] The following is combined Figures 1-7 The present invention describes a learning cabin multimodal data processing method and apparatus.
[0027] Figure 1 This is one of the flowcharts illustrating the multimodal data processing method for the learning cabin provided by the present invention, such as... Figure 1 As shown, the method includes the following steps: Step 110: Extract the embedding features corresponding to each modality from the multimodal data generated by learners performing learning tasks in the learning pod, and evaluate the quality of the multimodal data to obtain the evaluation results.
[0028] In this step, learners generate multimodal data simultaneously, including electrocardiogram, eye movement, video, voice, and learning behavior, while performing tasks in the learning pod.
[0029] In this step, the original data can be converted into a feature vector of uniform dimension using an encoding network to obtain the corresponding embedded features. This process eliminates the structural differences between different modalities of data, enabling heterogeneous data to be computed in the same mathematical space.
[0030] For example, the aforementioned encoding network can be a one-dimensional convolutional neural network, an image convolutional neural network, etc.
[0031] In this step, scores can be assigned based on data completeness, noise level, or missing data to achieve quality assessment. This provides a reliable basis for subsequent weight allocation and avoids low-quality features interfering with the final judgment.
[0032] In this embodiment, the system (the method execution entity) temporarily stores various modal data generated by the learner in the learning cabin in chronological order, such as electrocardiogram data, eye-tracking data, video data, voice data, and learning behavior data; then it records the start and end times, window number, learning task number, learning stage number, and the correspondence between the modal data in each analysis time window; finally, it records the completeness rate, noise level, missing data, outlier ratio, effective acquisition time, and quality score of each modal data in each time window.
[0033] Specifically, the embedded features and evaluation results are obtained through the following steps: (1) Obtain multimodal learning status data in the learning cabin.
[0034] In this embodiment, after entering the smart education learning cabin, learners can perform reading, answering questions, oral practice, interview training, experimental training, interactive Q&A, or other learning tasks within the cabin. During the execution of learning tasks, the system acquires multimodal learning state data generated by the learner.
[0035] The multimodal learning state data includes at least two of the following: ① Electrocardiogram (ECG) data, including heart rate, heart rate variability, ECG waveform characteristics, heart rate change trends, and physiological arousal level; ② Eye movement data, including fixation point location, fixation area, fixation duration, saccade trajectory, blink frequency, pupil changes, and degree of gaze deviation; ③ Video data, including facial expressions, head posture, body posture, range of motion, gaze direction, micro-expression changes, and sitting posture changes; ④ Voice data, including speech rate, volume, pitch, pause duration, fluency of response, emotional characteristics of speech, and sentence completeness; ⑤ Learning behavior data, including answer accuracy, answer time, mouse clicks, keyboard input, page dwell time, task completion progress, number of interactions, etc.
[0036] This embodiment does not limit the technical focus to specific acquisition devices, but rather focuses on the dynamic integration method of multi-source heterogeneous data in the learning cabin; that is, as long as the above-mentioned types of learning state data can be obtained, they can be used as input data in this embodiment.
[0037] (2) Perform time windowing and time alignment on multimodal data.
[0038] Since the sampling frequency, data structure and data length of the above-mentioned different modal data are different, if they are directly fused, it is easy to cause the problem that the different modal data do not belong to the same learning time or the same learning task segment; therefore, this embodiment first slices the multimodal data according to a unified time window.
[0039] Specifically, setting the first t The time windows are: ; in, The time window length can be set to 5 seconds, 10 seconds, or 30 seconds depending on the learning task type, or it can be adaptively determined based on question switching, voice pauses, changes in learning stages, or interactive events.
[0040] In this embodiment, the system records the window number, start and end time, learning task number, learning stage number, and the correspondence between each modality of data in the time window index area. For data with different sampling frequencies, the system uses resampling, interpolation, fragment aggregation, or event alignment to ensure that different modality data can be mapped to the same time window.
[0041] For example, video data can be aggregated into the same time window according to the sequence of image frames; electrocardiogram data can be represented according to the heart rate change sequence and statistical characteristics within the time window; eye movement data can be represented according to the gaze trajectory, gaze area and blink frequency within the time window; speech data can be represented according to the speech rate, pauses, volume and pitch changes within the time window; and learning behavior data can be represented according to behavioral events such as answering, clicking, inputting, and pausing within the time window.
[0042] In this embodiment, the system converts the originally asynchronously generated multimodal data into multimodal data fragments within the same time window, providing a foundation for subsequent unified processing.
[0043] (3) Construct the embedded representation of each modality data, i.e., the embedded features.
[0044] In order to input data of different types, dimensions and sampling frequencies into a unified model, this embodiment sets up corresponding modal encoders for different modalities to convert the raw data or preprocessed data into computable embedding vectors.
[0045] Specifically, the first i Class modality in the first t The embedding within a time window is represented as follows: ; in, Indicates the first i The encoder corresponding to the class modality, Indicates the first i Class modality in the first t Embedded feature vectors within a time window; Indicates the first i Class modality in the first t Raw or preprocessed data within a time window.
[0046] In this embodiment, different modes can use different encoding methods: ① ECG data encoders can employ one-dimensional convolutional networks, recurrent neural networks, temporal statistical feature networks, or heart rate variability feature encoding methods; ②Eye-tracking data encoders can employ trajectory sequence coding networks, fixation region transfer matrix coding methods, or eye-tracking event statistical coding methods; ③ The video data encoder can use a convolutional neural network, a face key point encoding network, a pose key point encoding network, or a temporal visual encoding network; ④ The speech data encoder can employ an acoustic feature coding network, a speech emotion feature coding network, or a speech temporal feature extraction network; ⑤ Learning behavior data encoders can employ behavior sequence coding networks, event statistical coding methods, or time series models.
[0047] In this embodiment, in order to reduce the dimensionality differences between different modalities, the embedding vectors of each modality can be further processed by linear mapping, autoencoder, principal component analysis, contrastive learning encoder or low-dimensional projection network to obtain a modal representation with a unified dimension or that can be mapped to the same space.
[0048] (4) Preprocess and quality assessment of each modal data.
[0049] In this embodiment, within each time window, the system preprocesses the data for each modality, specifically including: The ECG data is filtered, outlier removal is performed, heart rate is calculated, heart rate variability is calculated, and waveform features are extracted. Invalid fixation point removal, fixation region mapping, trajectory smoothing, blink event recognition, and pupil change calculation are performed on eye-tracking data. The system performs image frame extraction, facial landmark detection, expression recognition, head pose estimation, body pose estimation, and motion amplitude calculation on video data. The speech data is subjected to noise reduction, framing, endpoint detection, speech rate calculation, volume calculation, pitch change extraction, pause detection, and speech emotion feature extraction. The system performs missing value processing, abnormal behavior removal, behavior serialization, answer time calculation, and accuracy statistics on learning behavior data.
[0050] After preprocessing, the system calculates data quality indices for each modality; the data quality of the i-th modality within the t-th time window is represented as: ; in, This represents a data quality assessment function. This indicates the data quality score for this modality.
[0051] In this embodiment, the data quality score can be determined by the following factors: (1) data integrity rate, which indicates whether the modality data is complete within the time window; (2) noise level, which indicates whether the modality data is affected by environmental interference or abnormal acquisition; (3) outlier ratio, which indicates the proportion of outlier points in the modality data; (4) effective acquisition duration, which indicates the duration for which the modality is effective within the time window; (5) signal stability, which indicates whether the modality data experiences severe abnormal fluctuations in a short period of time; and (6) device or algorithm confidence, which indicates whether the modality feature extraction result is reliable.
[0052] For example, when eye-tracking data has a large number of missing points within a certain time window, the system lowers the eye-tracking modality quality score; when the video image is occluded, the system lowers the video modality quality score; when the speech data is empty, the system records the missing speech modality; when the ECG data has high noise, the system lowers the ECG modality quality score. Subsequent reinforcement learning models will use these quality scores as an important basis when allocating weights to avoid low-quality data having an excessive impact on the final judgment.
[0053] Step 120: Based on the embedded features and the individual baselines associated with the corresponding modalities, obtain the individual baseline offset features corresponding to different modalities; use the embedded features corresponding to each modality to identify the learner's learning state, obtain multiple single-modal recognition results, and calculate the confidence level corresponding to each single-modal recognition result.
[0054] In this step, the individual baseline can be the historical stationary data characteristics of the learner under normal conditions, recorded by the system over a long period of time.
[0055] Individual baselines may include multiple baselines, each corresponding to a modality type. For example, they may record the learner's electrocardiogram baseline, eye movement baseline, video behavior baseline, speech expression baseline, and learning behavior baseline under different learning tasks and normal learning states.
[0056] In this step, the individual baseline offset feature can be determined by the difference between the current embedded feature and the individual baseline. This offset feature is used to eliminate learners’ inherent habits, such as naturally high heart rate.
[0057] In this step, the learner's learning status is initially judged by the features of each modality data, and the corresponding single-modality recognition result is obtained. Then, the confidence of the recognition result is calculated, which can provide a basis for subsequent discovery of contradictions or support relationships between multimodalities.
[0058] In this embodiment, the learner's individual baseline is established and the baseline offset feature is calculated through the following steps.
[0059] (1) Since learning pods are typically relatively fixed learning spaces for individual learners or a small number of learners, learners may enter the same learning pod multiple times to complete different types of learning tasks; therefore, the system can accumulate historical multimodal data of the same learner under normal learning conditions and establish an individual baseline; for the first i For class modalities, the individual learner baseline is represented as: ; in, This indicates the first step in the learner's history learning process. i Modal data, Indicates the baseline generation function. This indicates that the learner was in the first... i Individual baselines on class modalities.
[0060] In this embodiment, the individual baseline may include: an electrocardiogram baseline, such as average heart rate, normal range of heart rate variability, and range of heart rate fluctuations; wherein, the eye movement baseline is determined by average fixation duration, normal blink frequency, distribution of normal fixation area, and range of pupil changes; the video behavior baseline is determined by normal facial expression distribution, head posture habits, body movement amplitude, and sitting posture change habits; the speech expression baseline is determined by normal speech rate, volume range, pause habits, and fluency of expression; and the learning behavior baseline is determined by average answer time, normal accuracy range, page dwell habits, and interaction frequency.
[0061] (2) Within the current time window, the system compares the current features of each modality with the individual baseline to obtain the baseline offset features: ; in, Indicates the first i Embedded features within the current time window of the class modality This represents the individual baseline corresponding to this modality. This indicates the degree of deviation of the modality from the learner's normal state, i.e., the baseline deviation feature mentioned above.
[0062] In this embodiment, the system does not simply determine whether a high heart rate indicates anxiety or that less movement indicates focus, but rather determines whether the current state deviates abnormally from the learner's normal state.
[0063] For example, if a learner's heart rate is usually high, the system will not directly judge their anxiety based solely on their high heart rate; only when the current heart rate is significantly higher than their personal historical baseline, and there are also abnormalities in eye movement or behavioral data, will the system increase the contribution of that physiological abnormality to the final state judgment.
[0064] (3) Generate preliminary judgment results for each mode based on the single-mode recognizer.
[0065] After completing the embedding representation of each modality, the system inputs the embedding vector of each modality into the corresponding single-modality recognizer to obtain the preliminary judgment result of the modality on the current learning state.
[0066] No. i Preliminary judgment result of the modality on the current learning state Represented as: ; in, Indicates the first i A single-modal recognizer corresponding to a class modality.
[0067] In this embodiment, the single-modal recognition results may include: the learner's attention level, anxiety level, fatigue level, cognitive load, learning engagement, and learning risk status.
[0068] (4) The system records the confidence level of the preliminary judgment result of the modality (i.e., the recognition result of each single modality), which is expressed by the following formula: ; in, Indicates the first i The confidence level of the current judgment result for the modality. This is the confidence score calculation function.
[0069] For example, within a certain time window, the video modality might output "stable state, confidence 0.72"; the ECG modality might output "high level of tension, confidence 0.84"; the eye-tracking modality might output "unstable attention, confidence 0.81"; the speech modality might output "missing" due to the lack of speech input; and the learning behavior modality might output "slow response, confidence 0.76". These single-modal recognition results are not directly used as the final result, but are used for subsequent construction of modality conflict graphs and dynamic weight allocation.
[0070] Step 130: Based on the multiple single-modal recognition results and their corresponding confidence levels, the evaluation results corresponding to different modal data and individual baseline offset features, establish the logical relationship between different modal data to obtain the dynamic modal conflict map and multimodal relationship representation.
[0071] In this embodiment, the logical relationship may include one or more of the following: support relationship, conflict relationship, complement relationship and missing relationship. The logical relationship is used to explicitly characterize the consistency or contradiction of each modality's judgment of the current state.
[0072] In this embodiment, the dynamic modal conflict graph can be a topological graph composed of each modality as nodes and logical relationships as edges. Correspondingly, the comprehensive relationship feature vector generated by encoding the dynamic modal conflict graph through a graph network is a multimodal relationship representation, which can serve as a key basis for subsequent reinforcement learning models to perform mode selection and weight allocation.
[0073] Furthermore, based on multiple single-modal recognition results and their corresponding confidence levels, the evaluation results corresponding to different modal data, and individual baseline offset features, logical relationships between different modal data are established, including: (1) Calculate the similarity and prediction difference of any two single-modal recognition results in the state dimension among multiple single-modal recognition results; (2) When the similarity is greater than the first threshold and the evaluation results and confidence of the modal data corresponding to any two single modal recognition results meet the preset conditions, the logical relationship between the modal data corresponding to any two single modal recognition results is determined as a support relationship; (3) When the prediction difference is greater than the second threshold, and the confidence of at least one of the modal data corresponding to any two single-modal recognition results meets the preset condition, the logical relationship between the modal data corresponding to any two single-modal recognition results is determined to be a conflict relationship.
[0074] In this embodiment, to avoid simply splicing or weighting all modal features, the system first determines the relationship between different modalities before fusion. The system uses each modality as a graph node and the support, conflict, complementarity or missing relationships between different modalities as graph edges to construct a dynamic modal conflict graph under the current time window.
[0075] In this embodiment, the dynamic modal conflict graph Represented as: ; in, Represents a set of modal nodes. This represents the set of modal relation edges.
[0076] In this embodiment, modal nodes may include ECG nodes, eye-tracking nodes, video nodes, speech nodes, and learning behavior nodes; for any two modal nodes and The system determines the type of relationship between the two based on their preliminary judgment results, confidence level, data quality, and baseline offset direction.
[0077] In this embodiment, if the two modalities have the same judgment direction on the current learning state, that is, the similarity between the two modalities is greater than a preset first threshold, then a support edge is established.
[0078] For example, if both ECG and eye-tracking data indicate a high level of anxiety, a support relationship is established between the ECG and eye-tracking nodes.
[0079] In this embodiment, if the two modalities have opposite judgment directions on the current learning state or the difference exceeds a preset second threshold, a conflict edge is established.
[0080] For example, if video data shows that the learner's movements are stable and their expression is calm, while ECG data shows that the learner is in a state of obvious tension, then a conflict relationship is established between the video data and the ECG data.
[0081] In this embodiment, if one modality can explain or supplement information that another modality cannot cover, then a complementary edge is established.
[0082] For example, if learning behavior data shows prolonged response time and electrocardiogram data shows increased anxiety, both of which indicate that the learner may be under high cognitive load, then the two can be considered complementary.
[0083] In this embodiment, if the data quality of a certain modality is lower than a threshold or data is missing within the current time window, a missing edge is established or the modality is marked as a missing node.
[0084] For example, if speech data is absent in a silent reading task, the system will mark the speech modality as a missing or low-contribution candidate modality.
[0085] In this embodiment, for any two modes i and j The strength of their relationship can be expressed as: ; in, Indicates the first i Class modality and the first j The strength of the relationship between modalities Represents relational computation functions. and These represent the preliminary judgment results for the two modes, and These represent the confidence levels of the two modes, respectively. and These represent the data quality of the two modalities, and These represent the individual baseline offset features of the two modes, respectively.
[0086] In this embodiment, the system explicitly expresses the relationships between multiple modalities, rather than relying on an automatic fusion model. This modality conflict graph provides a structured basis for subsequent reinforcement learning weight allocation.
[0087] In this embodiment, the dynamic modal conflict graph is input into a graph encoding network to generate a multimodal relationship representation under the current time window. Unlike directly concatenating the features of each modality and inputting them into the fusion model, this embodiment explicitly calculates and encodes the relationship between different modalities before fusion.
[0088] Specifically, in the t Within the first time window, for the first... i Modality, the system has already obtained the embedding features of this modality. Baseline offset characteristics Data quality score Confidence level and single-modal learning state prediction vector Among them, the first i The learning state prediction vector for each modality It can be represented as: ; in, Indicates the first i The probability of a modality predicting an attention state. Indicates the first i The predictive probability of a modality for anxieties. Indicates the first i The predictive probability of fatigue state by modal class. Indicates the first i The predictive probability of cognitive load state by modality.
[0089] For any two modal nodes and The system first calculates the similarity between the two prediction results. : ; in, For the first j The learned state prediction vector for each modality; The system represents cosine similarity; it also calculates the maximum difference between the two in the same learning state dimension. The difference is expressed as: ; in, It represents state dimensions such as attention, anxiety, fatigue, and cognitive load.
[0090] In this embodiment, when the prediction directions of two modalities are consistent, and both their data quality scores and recognition confidence levels are higher than a first threshold, the system establishes a support relationship between them; the judgment condition for the support relationship can be expressed as: ; in, Indicates the similarity threshold. Indicates the data quality threshold. This indicates the confidence threshold for identification.
[0091] In this embodiment, when the prediction difference between two modalities in the same learning state dimension exceeds a second threshold, and at least one of the modalities has high data quality and recognition confidence, the system establishes a conflict relationship between them; the judgment condition for the conflict relationship can be expressed using the above-mentioned difference as follows: ; in, This indicates the predicted difference threshold.
[0092] For example, if the video modality predicts that the learner's state is stable, while the ECG modality predicts that the learner's anxiety level is high, and the difference in the prediction of the anxiety state dimension between the two exceeds a threshold, the system will mark the video modality and the ECG modality as having a conflict relationship.
[0093] In this embodiment, when the prediction results of two modalities do not constitute a clear support relationship or conflict relationship, but their combined use can supplement the explanation of the learning state, the system establishes a complementary relationship between the two.
[0094] In this embodiment, the complementarity relationship can be determined by the joint contribution gain in historical training data or by a complementarity relationship predictor.
[0095] Specifically, during the training phase, the removal of the first... i Modal class, first j Modal class and simultaneous removal of the first i Class and First j The change in recognition loss after removing two modalities: If the increase in recognition loss caused by removing both modalities simultaneously is significantly higher than removing either modality alone, then the two modalities are considered to have a complementary relationship. Strength of complementarity. It can be represented as: ; in, Indicates feedback label, Indicates the removal of the first i The recognition results after class modality Indicates the removal of the first j The recognition results after class modality This means removing the first one at the same time. i Class and First j The recognition result after class modality; when When the value exceeds a preset threshold, the system considers the two to have a complementary relationship; This represents the loss function.
[0096] In this embodiment, when the data quality score of a certain modality is lower than a preset threshold, or when the modality is missing within the current time window, the system marks the modality node as a missing node or a low-quality node; the judgment condition can be expressed as: .
[0097] In this embodiment, for missing or low-quality nodes, the system limits their upper weight limit during subsequent weight allocation to prevent low-quality data from having an excessive impact on the final judgment; thus, the dynamic modal conflict graph constructed by the system is represented as follows: ; in, Represents a set of modal nodes. Represents the set of modal relation edges. This represents the support relation adjacency matrix. Represents the adjacency matrix of conflict relationships. Represents the adjacency matrix of complementary relationships. This represents a matrix of missing or low-quality markers.
[0098] In this embodiment, the relationships in the modal conflict graph are not manually specified, but are automatically calculated based on single-modal prediction results, data quality scores, identification confidence levels, and individual baseline offsets. Furthermore, each edge records not only the relationship type but also the relationship strength. Taking supportive and conflicting relationships as examples, their relationship strength can be expressed as: ; ; in, Indicates the strength of the support relationship. This indicates the strength of the conflict relationship. As can be seen from the above calculation, if the data quality of a certain modality is low or the identification confidence is low, even if its prediction results differ from those of other modalities, the strength of the conflict relationship formed will be reduced, thereby improving the reliability of the conflict graph relationship.
[0099] In this embodiment, after obtaining the dynamic modal conflict graph, the system sets the initial node representation of each modal node as follows: ; in, Indicates the first i The initial node representation of a modal node.
[0100] In this embodiment, the graph coding network performs message passing and updating of node representations according to different types of relationships; i Modal nodes in the first l +1 layer graph coding It can be represented as: ; in, For learnable parameters, Indicates the strength of the support relationship. Indicates the intensity of the conflict relationship. Indicates the strength of the complementary relationship. Indicates the first i Is the class modality missing or of low quality? Represents a non-linear activation function; Indicates the first i Modal nodes in the first l Layer graph encoding.
[0101] In this embodiment, after several layers of graph encoding, the system obtains the first... i Enhanced Relationship Representation of Modal Nodes Subsequently, the system obtains the multimodal relation representation within the current time window using attention pooling: ; ; in, This represents a vector of learnable attention parameters. This represents the transpose of the attention parameter vector. This represents the learnable attention transformation matrix. Indicates the first t Within the first time window i Normalized attention weights corresponding to modal nodes It is the hyperbolic tangent function. and They represent the first i Modal nodes and the first j Modal nodes after the first L The relationship-enhanced representation obtained after layer graph encoding; This represents the multimodal relationship vector within the current time window. This relationship vector is used to characterize whether the current modalities are mutually supportive, conflicting, complementary, or partially missing, providing structured input for subsequent reinforcement learning models to select fusion modes and allocate modal weights.
[0102] Figure 2 This is a schematic diagram of dynamic modal conflict relationships provided by the present invention. Figure 2In the illustrated embodiment, ECG, eye movement, video, speech, and learning behavior are represented as graph nodes, connected by supporting edges, conflicting edges, complementary edges, or missing edges. A supporting edge is established between the ECG and eye movement when both indicate the learner is in a state of tension or attentional instability; a conflicting edge is established between the video and ECG when the video shows the learner's movements are smooth while the ECG shows tension; a complementary edge is established between the learning behavior and ECG when the learning behavior shows slow responses and the ECG shows physiological tension; and a missing speech node is marked as a missing node when speech data is absent or of low quality.
[0103] This embodiment uses similarity and difference comparisons, combined with data quality and confidence, to determine the association between different modalities, effectively eliminating false interference caused by low-quality sensors and providing algorithmic support for constructing highly reliable dynamic modal conflict maps.
[0104] Step 140: Assign a first weight to the multimodal data based on the individual baseline offset characteristics and multimodal relationship representation, and constrain the first weight using the evaluation results of the corresponding modal data and the dynamic modal conflict diagram to obtain the second weight.
[0105] In this step, the first weight represents the proportion of importance of each modality, which is used to match the current learning task pattern.
[0106] In this step, constraints may include limiting the upper limit if the data quality is low, and reducing the weight if there are serious conflicts. The first weight is modified by the constraints and used for subsequent feature fusion, which can ensure the stability of weight allocation under extreme data conditions.
[0107] Furthermore, a first weight is assigned to the multimodal data based on individual baseline offset features and multimodal relationship representations, including: constructing a state space based on individual baseline offset features, multimodal relationship representations, and current learning task context information; determining the target fusion mode for the current time window using the upper-layer policy network and state space of the hierarchical reinforcement learning model; calculating the original importance score of each modality data using the lower-layer weight allocation network, state space, and target fusion mode of the hierarchical reinforcement learning model, and normalizing multiple original importance scores to obtain the first weight.
[0108] In this embodiment, the modality embedding features, baseline offset features, data quality indicators, single modality recognition confidence, modality relation vectors, and learning task context are collectively constructed into a hierarchical reinforcement learning state space.
[0109] state space Represented as: ; in, For each modality embedding feature set, including 1-n Embedded features: ; For each modality, there is a set of individual baseline offset features, including 1- n Individual baseline offset characteristics: ; For each modal data quality set, including 1- n Data quality: ; For each modality, the confidence set for single-modal recognition includes 1- n Confidence level: ; For each modality, the set of state prediction vectors for single-modal learning includes 1- n Each learning state prediction vector: Each set is represented by the following formula: ; ; ; ; ; in, This represents the multimodal relation vector generated by the dynamic modal conflict graph; This indicates the context information of the learning task.
[0110] In this embodiment, the learning task context information includes the learning task type, learning stage, question difficulty, current interaction event, historical learning performance, task objective, and current learning duration. The system can encode the learning task type, learning stage, and task objective into discrete vectors, and encode the question difficulty, historical learning performance, and learning duration into continuous numerical vectors, ultimately forming a context feature representation.
[0111] In this embodiment, through this state space design, the reinforcement learning model can not only obtain the characteristics of each modality's data itself, but also the following information: whether each modality's data is reliable, whether each modality deviates from the learner's normal state, whether there are supportive or conflicting relationships between different modalities, what learning task scenario is currently in, and the learning performance in similar scenarios in the past. The above information collectively constitutes the basis for selecting the upper-layer fusion mode and allocating the lower-layer weights.
[0112] This embodiment employs a hierarchical reinforcement learning model to dynamically weight multimodal data; the hierarchical reinforcement learning model includes an upper-layer policy network and a lower-layer weight allocation network. The role of the upper-layer policy network is to select the fusion mode to be used in the current time window based on the current state space.
[0113] In this embodiment, the upper-layer policy network receives the current state space. Output the selection probability of each fusion mode: ; in, This represents the upper-layer policy network. Indicates the parameters of the upper-layer policy network. This represents a candidate fusion mode. This indicates that the first [number] is selected in the current state space. The probability of a fusion mode.
[0114] In this embodiment, the system can sample fusion modes according to the above probability distribution during the training phase, and select the fusion mode with the highest probability as the current fusion mode during the application phase. : .
[0115] In this embodiment, the fusion modes include: (1) Physiology-dominated fusion mode: applicable to scenarios where physiological data such as ECG is of high quality, baseline shift is significant, and physiological modalities form a support relationship with other high-quality modalities; (2) Eye-tracking-dominated fusion mode: applicable to scenarios where eye-tracking data is of high quality, eye-tracking baseline shift is significant, and eye-tracking modalities contribute significantly to state judgment in tasks such as reading, observation, and answering questions; (3) Speech-dominated fusion mode: applicable to scenarios where speech data is of high quality and speech expression features are strongly correlated with task objectives in tasks such as oral training, interview training, reading aloud training, and question-and-answer training; (4) Behavior Dominant fusion mode: applicable to scenarios where learning behavior data can directly reflect task performance in tasks such as answering questions, testing, and operation training, and the quality of behavior data is high; (5) Balanced fusion mode: applicable to scenarios where the quality of data in each modality is high, the judgment results of single modalities support each other, and there is no obvious conflict; (6) Conflict suppression fusion mode: applicable to scenarios where there is obvious conflict between different modalities, and it is necessary to improve the influence of high-quality, high-confidence modalities and reduce the influence of low-quality or low-confidence modalities; (7) Missing compensation fusion mode: applicable to scenarios where a certain modality is missing or of low quality, and it is necessary to compensate for the judgment by other modalities.
[0116] To prevent the upper-layer policy network from selecting a fusion mode that is significantly inconsistent with the current data state, this embodiment can also set a fusion mode constraint mask. For example, when the speech modality is missing in the current time window, the probability of selecting the speech-dominant fusion mode is reduced; when the eye-tracking modality quality is below a threshold, the probability of selecting the eye-tracking-dominant fusion mode is reduced; when there are multiple high-intensity conflict edges in the dynamic modality conflict graph, the prior probability of the conflict suppression fusion mode is increased; when multiple modalities have balanced quality and many support relationships, the prior probability of the balanced fusion mode is increased.
[0117] In this embodiment, the current fusion mode is not manually selected, but is automatically output by the upper-layer policy network based on the state space. The selection process is based on the embedding features of each modality, individual baseline offset, data quality, recognition confidence, dynamic modality conflict graph relationship representation, and learning task context. Furthermore, the fusion mode selection strategy is continuously updated based on reward feedback during reinforcement learning training, enabling the system to gradually learn more reasonable fusion modes under different learning tasks and different modal relationships.
[0118] In this embodiment, the lower-level weight allocation network calculates the dynamic weights of each modality in the current learning state judgment based on the current state space and the fusion mode output by the upper-level policy network. Unlike manually setting fixed weights, the modal weights in this embodiment are calculated in real time by the model based on the current learning context.
[0119] Specifically, upon obtaining the current fusion mode Then, the system first encodes the fusion pattern into a pattern vector. The original importance scores of each modality are then input into the lower-level weight allocation network along with the current state space; the lower-level weight allocation network outputs the original importance scores of each modality. , represented as: ; in, This indicates the lower-level weight allocation network. This indicates the network parameters for assigning weights to lower layers. Indicates the first i The original importance score of the modality within the current time window.
[0120] In this embodiment, to ensure that weight allocation simultaneously considers data quality, identification confidence, baseline offset, modal relationships, and fusion mode priors, this embodiment can... i Original importance score of class modality Further expressed as: ; in, Indicates the first i Modal data quality scoring Indicates the first i Confidence of class modality Indicates the first i The intensity of the shift of the modality relative to the individual baseline. Indicates the first i The sum of the strengths of the support relationships formed between a modality and other modalities. Indicates the first i The sum of the intensity of conflict relationships between a modality and other modalities. Indicates the first iIs the class modality missing or of low quality? Indicates the current fusion mode for the first i Prior biases of class modality, - These are learnable or adjustable parameters.
[0121] in, , They are represented as follows: ; .
[0122] In other words, if a modality has high data quality, high identification confidence, significant deviation from the individual baseline, and forms a supporting relationship with multiple high-quality modalities, then the importance score of that modality increases; if a modality has missing data, low quality, or forms a conflicting relationship with multiple high-quality modalities, then the importance score of that modality decreases.
[0123] In this embodiment, after obtaining the original importance scores of each modality, the system performs normalization processing using the following formula to obtain the initial weights. : ; in, Indicates the first j The original importance scores of each modality are used to obtain the initial weight vector, i.e., the first weight. : ; The following conditions must be met: ; ; in, Indicates the first i The proportion of influence of the modality on the final learning state judgment within the current time window; i The value range is 1- n .
[0124] This embodiment introduces a global state space containing the task context and adopts a hierarchical reinforcement learning architecture with upper-layer fusion mode and lower-layer specific weights. This hierarchical decision-making mechanism improves the rationality of weight allocation under complex learning tasks (such as task switching and severe modal contradictions) and enhances the adaptive evolution capability of the dynamic integration mechanism.
[0125] Furthermore, the first weight is constrained using the evaluation results of the corresponding modal data and the dynamic modal conflict graph, including: when the evaluation result of one modal data in multiple modal data is lower than a preset quality threshold, adjusting the upper limit of the first weight corresponding to the modal data to not exceed the weight threshold; when a modal data is marked as missing, setting the first weight corresponding to the modal data to zero; when a modal data has conflicting relationships with multiple modal data in the dynamic modal conflict graph, and the evaluation result of one modal data is lower than the preset quality threshold, reducing the first weight corresponding to the modal data to the preset weight value.
[0126] In this embodiment, the system then corrects the initial weights through a weight safety constraint layer. The corrected weight vector... Represented as: ; in, This represents the weighted safety constraint function. This represents the set of data quality for each modality. This represents the set of confidence levels for each modality. Represents a dynamic modal conflict diagram. This indicates the context of the learning task.
[0127] In this embodiment, the weighted security constraints include the following rules: (1) When the data quality of a certain modality is lower than a preset threshold, the upper limit of the weight of that modality is restricted; (2) When data for a certain modality is missing, reduce the weight of that modality to zero; In some embodiments, when data for a certain modality is missing, the modality weight can be adjusted to a value close to zero, for example, the difference between the weight value and 0 is lower than a preset threshold.
[0128] (3) When a certain modality is strongly correlated with the current learning task and the data quality is high, increase the lower limit of the weight of that modality; (4) When a mode conflicts with multiple high-quality, high-confidence modes and its own confidence is low, reduce the weight of that mode; (5) When the change of a certain modality weight exceeds the preset threshold within an adjacent time window, the modality weight is smoothed.
[0129] In this embodiment, after applying weight safety constraints, the system renormalizes the weights to obtain new normalized weights. : .
[0130] in, For weighted constraints , For weighted constraints , Indicates the first j The proportion of influence of each modality on the final learning state judgment within the current time window; then, the weight vector used for multimodal fusion, i.e., the second weight, is obtained through the following formula. : .
[0131] This embodiment sets rule constraints on the first weight of the reinforcement learning output through data quality assessment results and modal conflict graphs. This method of combining reinforcement learning with rule constraints can prevent the weight allocation distortion caused by sudden physical conditions such as sensor obstruction, loose wearing, and environmental noise in the learning cabin, and improve the robustness and decision-making fault tolerance of the multimodal data dynamic integration mechanism in complex continuous learning environments.
[0132] Step 150: Calculate the fusion features based on the embedding features and second weights corresponding to different modal data, and use the fusion features to identify the learner's learning state to obtain the fusion identification result.
[0133] In this step, the embedded features of each modality are weighted and summed (or weighted concatenated) with their corresponding second weights to obtain the fused features, which are used to represent the most valuable and realistic learning state information at present.
[0134] In this step, the fused features are input into the final learning state assessment model, which outputs specific evaluation conclusions such as the learner's current attention level, anxiety level, or cognitive load.
[0135] Specifically, the system performs weighted fusion of the embedding features of each modality based on the second weight, as follows: ; in, This represents the comprehensive learning state vector within the current time window; subsequently, the system inputs the comprehensive learning state vector into the learning state evaluation model to obtain the learning state evaluation result. : ; in, It indicates the current learning status outcome, including one or more of the following: attention level, anxiety level, fatigue level, cognitive load, learning engagement, or learning risk warning.
[0136] The multimodal data processing method for learning cabins provided in this invention provides data support for preventing inferior data from participating in fusion decisions by extracting the embedding features of multimodal data and performing quality assessment. It eliminates the risk of misjudgment caused by individual learner differences by calculating individual baseline offset features, and clarifies the judgment tendency of each sensor at the current moment by calculating the single-modal recognition results and confidence levels. By constructing a dynamic modal conflict graph to represent the logical relationship between different modal data, and by dynamically allocating the first weight and generating the second weight using rule constraints, it reduces the interference of low-quality data or superficial deceptive data, achieving dynamic weight decision-making. Finally, it accurately identifies the learner's learning state through fusion features, realizing the dynamic integration and state assessment of multimodal data in the learning cabin, and improving the accuracy of learning state prediction results.
[0137] In some embodiments, the first weight is determined by a hierarchical reinforcement learning model; after obtaining the fused recognition result, the learning cabin multimodal data processing method further includes: (1) Masking, zeroing or replacing the target modal data in the multimodal data with the baseline is performed to obtain the multimodal data after intervention; (2) Use the multimodal data after intervention to perform feature fusion and recognition again, obtain the comparison recognition results, and calculate the counterfactual contribution of the target modality data in the current recognition based on the change in recognition loss between the fused recognition results and the comparison recognition results; (3) Construct a reinforcement learning reward function based on the counterfactual contribution and delayed feedback information corresponding to each modality data; the delayed feedback information is determined by the learner's answer performance, task completion time or evaluation feedback in subsequent learning tasks; (4) Calculate the reward value using the reinforcement learning reward function, and update the policy parameters of the hierarchical reinforcement learning model based on the reward value.
[0138] To avoid the reinforcement learning model assigning weights solely based on surface relevance, this embodiment further introduces a counterfactual contribution calculation mechanism, the process of which is as follows: ① Within the current time window, the system first obtains the fusion recognition result based on all modal data. , represented as: .
[0139] in, To represent the mapping function corresponding to the learning state evaluation model, used to determine the fused features. Obtain learning status assessment results .
[0140] ② By masking, zeroing, perturbing, or replacing a specific modality with the learner's historical baseline, the fusion recognition result after removing that modality is obtained: ; in, Indicates the removal of the first i The learning state recognition result after modality classification.
[0141] ③No. i The counterfactual contribution of the modality to the current judgment result Represented as: ; in, These are feedback tags obtained through teacher annotations, learner self-assessments, subsequent learning performance, or comprehensive system evaluation. Loss This represents the loss function.
[0142] In this embodiment, if the recognition loss increases significantly after removing a certain modality, it indicates that the modality makes a high contribution to the current judgment; if the recognition loss changes little after removing a certain modality, it indicates that the modality makes a low contribution within the current time window. This counterfactual contribution is used for subsequent reinforcement learning reward feedback, enabling the model to learn the modalities that truly contribute in the current scenario, rather than relying solely on fixed experience or the black-box attention within the model.
[0143] In this embodiment, the reinforcement learning reward function considers not only the accuracy of the current learning state recognition, but also the counterfactual contribution, subsequent learning feedback, data quality, modality conflict handling effect, and weight stability.
[0144] In this embodiment, the reward function Represented as: ; in, This indicates the reward for recognizing the current learning state. This indicates a reward for contributions that contradict facts. This indicates a reward for subsequent learning performance. This indicates a data quality reward; Indicates rewards for modal conflict resolution; Indicates a penalty for weight jumps; This is the reward adjustment coefficient.
[0145] In this embodiment, the current learning state recognition reward is used to measure whether the current learning state result output by the system is accurate; the counterfactual contribution reward is used to measure whether the model output weights are consistent with the actual contribution of each modality; the subsequent learning performance feedback reward is used to measure the predictive effect of the current state judgment and weight allocation on the subsequent learning performance; the data quality reward is used to encourage the model to assign reasonable weights to high-quality data and reduce the weights to low-quality data; the modality conflict handling reward is used to encourage the model to increase the weights of high-confidence modalities and decrease the weights of low-confidence modalities when there is a modality conflict; and the weight jump penalty is used to avoid frequent and large fluctuations in weights in a short period of time.
[0146] In this embodiment, the reward value includes a counterfactual contribution reward and a data quality reward; wherein, the counterfactual contribution reward... It can be represented as: ; in, Represents the normalized i-th i Modal counterfactual contribution; this formula indicates that the higher the reward, the closer the modal weights output by the model are to the true counterfactual contribution of that modality.
[0147] In this embodiment, data quality reward It can be represented as: .
[0148] The above formula means that when the model assigns higher weights to high-quality data, the reward is higher; if the model assigns too high weights to low-quality data, the reward is lower.
[0149] In this embodiment, weight jump penalty It can be represented as: ; The above formula means that if the weight changes too much within adjacent time windows, a penalty is imposed to improve the stability of the fusion result.
[0150] In this embodiment, subsequent learning performance feedback rewards can be determined based on changes in subsequent answer accuracy, task completion status, teacher feedback, learner self-assessment, learning interruptions, or performance in the next stage of learning. For example, if the system determines that the learner's cognitive load is high in the current time window, and the learner subsequently experiences a decrease in answer accuracy, a significant increase in answer time, or a self-assessment of tension, then this determination receives positive feedback; conversely, if subsequent performance is inconsistent with the current determination, the corresponding reward is reduced.
[0151] In this embodiment, through the aforementioned reward function, the reinforcement learning model continuously updates the upper-layer fusion mode selection strategy and the lower-layer weight allocation strategy, enabling the system to gradually learn the appropriate weights that various types of data should occupy under different learning tasks, different learners, and different modal conflict situations.
[0152] Figure 3 This is a flowchart of the hierarchical reinforcement learning weight allocation process provided by the present invention. Figure 3 In the illustrated embodiment, the current state space is... (Constructed based on features / offsets / quality / confidence / prediction vectors / conflict graphs / task context) First, input the upper-layer policy network. (Same as the upper-level policy network above) (Same), the upper-layer policy network selects the current fusion mode. The current state space and fusion mode are jointly input into the lower-level weight allocation network. (Same as the lower-level policy network mentioned above) (Same), the lower-level weight allocation network outputs the initial weights for each mode. The initial weights are then corrected by a weight safety constraint layer (specifically, based on data quality, task requirements, and conflict relationships) to obtain the final dynamic weights used for fusion. .
[0153] The learning cabin multimodal data processing method provided in this invention obtains the actual contribution of each modality data within the black box model through counterfactual inference, and constructs a reinforcement learning reward and punishment mechanism in combination with delayed learning performance verification, so that the multimodal integrated model can dynamically output the current weight, and can also continuously reflect on itself and evolve based on the subsequent real learning effect.
[0154] In some embodiments, the multimodal data includes at least two of electrocardiogram data, eye-tracking data, video data, speech data, and learning behavior data; the logical relationships include at least one of supporting relationships, conflicting relationships, complementary relationships, and missing relationships; after obtaining the fusion recognition result, the method further includes: obtaining a modality contribution description; the modality contribution description includes the fusion recognition result, the second weight ratio corresponding to each modality data, and the logical relationships in the dynamic modality conflict graph.
[0155] In this embodiment, after completing the dynamic weight fusion, the system not only outputs the final learning state result, but also outputs the modal contribution description corresponding to the result.
[0156] Specifically, the output modal contribution description includes the following: current learning state results, current fusion mode, dynamic weight ratio of each modality; data quality score of each modality, the degree of deviation of each modality relative to the individual baseline, the main supporting and conflicting relationships in the dynamic modality conflict graph, the counterfactual contribution of each modality, the main basis modality for the final judgment, and learning intervention suggestions.
[0157] For example, within a specific learning task time window, the system outputs the following results: The learner is currently in a state of high anxiety; the current fusion mode is the physiological-eye-tracking dominant fusion mode; the weight of ECG data is 0.41, eye-tracking data is 0.34, video data is 0.09, speech data is 0.06, and learning behavior data is 0.10; ECG data is significantly elevated relative to the individual baseline, and eye-tracking data shows fixation instability and increased blinking frequency, with a supportive relationship between ECG and eye-tracking nodes; although video data shows that the learner's movements are fewer and expressions are stable, there is a conflict between video nodes and ECG and eye-tracking nodes, and the video data contributes little to the counterfactual judgment. Therefore, the system reduces the weight of the video modality and increases the weights of the ECG and eye-tracking modalities. The system suggests reducing the difficulty of the current task, giving the learner a short buffer time, or providing emotional reassurance.
[0158] Figure 4 This is a flowchart of the counterfactual contribution and delayed feedback update provided by the present invention. Figure 4 In the illustrated embodiment, after the system outputs the learning state result and the weights of each modality (multimodal fusion and learning state output) in the current time window, it performs counterfactual contribution calculation (specifically, it is achieved by masking / perturbing / replacing a certain modality); the system compares the above feedback with the current recognition result to form a delayed feedback reward, specifically by collecting subsequent learning performance, teacher feedback, and learner self-evaluation; finally, the reward is fed back to the hierarchical reinforcement learning model to update the fusion mode selection strategy and weight allocation strategy.
[0159] The learning cabin multimodal data processing method provided in this invention defines the types of multimodal input sources and logical relationships, and outputs a description of modal contributions including weight proportions and conflict relationships. This realizes the transformation of the multimodal fusion model from black-box decision-making to white-box interpretation. Through this output method, teachers or learning systems can not only know the current state of learners, but also know why the system made this judgment, and which modal data the current judgment mainly relies on.
[0160] Figure 5 This is the second flowchart of the learning cabin multimodal data processing method provided by the present invention. Figure 5 In the illustrated embodiment, the method includes the following steps: (1) Acquire multimodal learning status data in the learning cabin, including ECG / eye movement / video / speech / learning behavior; (2) Perform time windowing and time alignment on multimodal data; (3) Perform preprocessing and quality assessment on each modal data; (4) Construct the embedding representation of each modality; (5) Establish individual baselines and calculate baseline offset characteristics; (6) Perform preliminary single-modal state judgment; (7) Construct a dynamic modal conflict diagram; (8) Execution level reinforcement learning weight allocation; (9) Perform weighted safety constraint correction; (10) Perform multimodal weighted fusion; (11) Output the learning status results and contribution description; (12) Perform counterfactual contribution calculation and delayed feedback update.
[0161] In one feasible embodiment, learners practice answering math questions within a smart education learning pod. The system acquires learner status data in 10-second time windows. Within a given time window, video data shows minimal body movement and minimal facial expression changes; audio data is empty; ECG data shows a significantly higher heart rate than the learner's baseline; eye-tracking data shows frequent deviations of the learner's gaze from the question area and a higher blinking frequency than the baseline; learning behavior data shows that the learner's current answer time is significantly longer than the historical average answer time. The system first performs time alignment, preprocessing, and quality assessment on the above data, obtaining quality scores for each modality: ECG, eye-tracking, video, audio, and learning behavior. Since the audio data is empty, the audio modality is marked as missing. Subsequently, the system inputs the data of each modality into the corresponding encoder to obtain the embedding vector. Based on the learner's historical data, it calculates the baseline offset of each modality and finds that the ECG modality and eye-tracking modality have significant offsets, while the video modality has a smaller offset. A preliminary judgment is made through the single-modality recognizer: the video modality is judged to be in a stable state, the ECG modality is judged to be in a high level of tension, the eye-tracking modality is judged to be inattentive, and the learning behavior modality is judged to be slow to respond. Based on the preliminary judgment results, the system constructs a dynamic modal conflict graph: ECG nodes and eye-tracking nodes have a supporting relationship, ECG nodes and video nodes have a conflicting relationship, eye-tracking nodes and learning behavior nodes have a complementary relationship, and speech nodes are missing nodes. After receiving the embedding features of each modality, baseline offset, quality score, recognition confidence, modal conflict graph relationship vector, and learning task context, the hierarchical reinforcement learning model selects either the "physiological-eye-tracking dominant fusion mode" or the "conflict suppression fusion mode" in the upper-layer policy network. The lower-layer weight allocation network outputs the initial weights. The weight safety constraint layer corrects the weights based on information such as missing speech, video and ECG / eye-tracking conflict, and high quality of ECG / eye-tracking, ultimately obtaining an ECG weight of 0.41, an eye-tracking weight of 0.34, a video weight of 0.09, a speech weight of 0.06, and a learning behavior weight of 0.10. The system fuses the multimodal embedding features based on the above weights and ultimately determines that the learner is in a state of high anxiety or high cognitive load. Subsequently, the system outputs an explanation: This judgment was primarily based on ECG and eye-tracking data; although the video data showed fewer learner movements, it conflicted with the ECG and eye-tracking results, and therefore its weight was reduced; the audio data was also reduced in weight due to its absence; learning behavior data provided supplementary support for the cognitive load assessment. If the learner's accuracy rate subsequently decreases, the response time continues to lengthen, or the learner self-reports nervousness, the system will use this feedback as a positive reward to reinforce the current weight allocation strategy; if subsequent performance is inconsistent with this judgment, the system will reduce the relevant strategy rewards and update the model parameters.
[0162] This embodiment achieves a complete closed-loop processing of multimodal data within the learning cabin, from acquisition, alignment, quality assessment, individual baseline offset, modal conflict modeling, hierarchical reinforcement learning weighting, weight constraint correction, multimodal fusion, counterfactual contribution calculation, to delayed feedback updates. Unlike traditional direct splicing or fixed-weighted multimodal fusion methods, this embodiment can dynamically calculate the influence ratio of each modality data on the final learning state judgment under different learning tasks, different learner individual characteristics, and different modal conflicts, and output interpretable modal contribution descriptions.
[0163] Figure 6 This is one of the structural schematic diagrams of the learning cabin multimodal data processing device provided by the present invention. Figure 6 In the illustrated embodiment, the device may include 1 a multimodal data acquisition module, 2 a time window alignment module, 3 a data preprocessing and quality assessment module, 4 an individual baseline modeling module, 5 a modality embedding and encoding module, 6 a unimodal state recognition module, 7 a dynamic modality conflict graph construction module, 8 a hierarchical reinforcement learning weight allocation module, 9 a weight security constraint module, 10 a multimodal fusion evaluation module, 11 a counterfactual contribution calculation module, 12 a delay feedback update module, and 13 a result interpretation and output module.
[0164] The system comprises the following modules: a multimodal data acquisition module for acquiring ECG, eye-tracking, video, speech, and learning behavior data within the learning cabin; a time window alignment module for slicing and synchronizing data from different modalities within a unified time window; a data preprocessing and quality assessment module for cleaning, filtering, denoising, and scoring the data for each modality; an individual baseline modeling module for establishing individual baselines for each modality based on learners' historical data; a modality embedding and encoding module for converting data from each modality into embedding vectors; a single-modality state recognition module for generating preliminary judgments and confidence levels for each modality regarding the current learning state; and a dynamic modality conflict graph construction module for calculating the inter-modality conflicts between different modalities. The system includes modules for supporting, conflicting, complementary, or missing modal relationships; a hierarchical reinforcement learning weight allocation module for selecting fusion modes and outputting dynamic weights for each modality; a weight safety constraint module for adjusting weights based on data quality, task context, and modal conflict relationships; a multimodal fusion evaluation module for fusing modal features based on dynamic weights and outputting learning state results; a counterfactual contribution calculation module for calculating the actual contribution of each modality to the current judgment result; a delayed feedback update module for updating reinforcement learning strategies based on subsequent learning performance, teacher feedback, or learner self-evaluation; and a result interpretation output module for outputting learning state results, modal weights, major contributing modalities, and modal conflict explanations.
[0165] The learning cabin multimodal data processing device provided by the present invention will be described below. The learning cabin multimodal data processing device described below and the learning cabin multimodal data processing method described above can be referred to in correspondence.
[0166] Figure 7 This is a second structural schematic diagram of the learning cabin multimodal data processing device provided by the present invention, as shown below. Figure 7 As shown, the device includes: a feature extraction and evaluation module 710, a single-modal recognition module 720, a multimodal relationship establishment module 730, a weight calculation module 740, and a comprehensive recognition module 750.
[0167] The feature extraction and evaluation module 710 is used to extract the embedded features corresponding to each modality from the multimodal data generated by learners performing learning tasks in the learning cabin, and to evaluate the quality of the multimodal data to obtain the evaluation results. The single-modality recognition module 720 is used to obtain individual baseline offset features corresponding to different modalities based on the embedded features and the individual baselines associated with the corresponding modalities; to identify the learner's learning state using the embedded features corresponding to each modality, to obtain multiple single-modality recognition results, and to calculate the confidence level corresponding to each single-modality recognition result; The multimodal relationship establishment module 730 is used to establish logical relationships between different modal data based on multiple single-modal recognition results and corresponding confidence levels, evaluation results corresponding to different modal data, and individual baseline offset features, to obtain a dynamic modal conflict map and multimodal relationship representation; The weight calculation module 740 is used to assign a first weight to the multimodal data based on the individual baseline offset characteristics and multimodal relationship representation, and to constrain the first weight using the evaluation results of the corresponding modal data and the dynamic modal conflict diagram to obtain a second weight; The integrated recognition module 750 is used to calculate fusion features based on the embedded features and second weights corresponding to different modal data, and to use the fusion features to identify the learner's learning state to obtain the fusion recognition result.
[0168] The multimodal data processing device for the learning cabin provided by this invention extracts the embedding features of multimodal data and performs quality assessment, providing data support for preventing inferior data from participating in fusion decisions. It eliminates the risk of misjudgment caused by individual learner differences by calculating individual baseline offset features, and clarifies the judgment tendency of each sensor at the current moment by calculating the single-modal recognition results and confidence levels. By constructing a dynamic modal conflict graph to represent the logical relationship between different modal data, and by dynamically allocating the first weight and generating the second weight using rule constraints, it reduces the interference of low-quality data or superficial deceptive data, achieving dynamic weight decision-making. Finally, it accurately identifies the learner's learning state through fusion features, realizing the dynamic integration and state assessment of multimodal data in the learning cabin, and improving the accuracy of learning state prediction results.
[0169] Figure 8 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 8As shown, the electronic device may include a processor 810, a communications interface 820, a memory 830, and a communication bus 840. The processor 810, communications interface 820, and memory 830 communicate with each other via the communication bus 840. The processor 810 can call logical instructions in the memory 830 to execute a multimodal data processing method for the learning pod. This method includes: extracting embedding features corresponding to each modality from the multimodal data generated by the learner performing learning tasks in the learning pod, and performing quality assessment on the multimodal data to obtain assessment results; obtaining individual baseline offset features corresponding to different modalities based on the individual baselines associated with the embedding features and corresponding modalities; identifying the learner's learning state using the embedding features corresponding to each modality to obtain multiple single-modal recognition results, and calculating the confidence level corresponding to each single-modal recognition result. Based on multiple unimodal recognition results and their corresponding confidence levels, evaluation results corresponding to different modal data, and individual baseline offset features, logical relationships between different modal data are established to obtain a dynamic modal conflict map and a multimodal relationship representation. A first weight is assigned to the multimodal data based on the individual baseline offset features and the multimodal relationship representation, and the first weight is constrained using the evaluation results of the corresponding modal data and the dynamic modal conflict map to obtain a second weight. Fusion features are calculated based on the embedding features corresponding to different modal data and the second weight, and the fusion features are used to identify the learner's learning state to obtain a fusion recognition result.
[0170] Furthermore, the logical instructions in the aforementioned memory 830 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0171] On the other hand, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program is implemented to perform the learning cabin multimodal data processing method provided by the above methods. The method includes: extracting embedding features corresponding to each modality from multimodal data generated by learners performing learning tasks in the learning cabin, and performing quality assessment on the multimodal data to obtain assessment results; obtaining individual baseline offset features corresponding to different modal data based on the individual baselines associated with the embedding features and the corresponding modalities; and identifying the learner's learning state using the embedding features corresponding to each modality to obtain multiple single-modal recognition results. The system calculates the confidence level corresponding to each single-modal recognition result; establishes logical relationships between different modal data based on multiple single-modal recognition results and their corresponding confidence levels, evaluation results corresponding to different modal data, and individual baseline offset features, to obtain a dynamic modal conflict map and a multimodal relationship representation; assigns a first weight to the multimodal data based on the individual baseline offset features and the multimodal relationship representation, and constrains the first weight using the evaluation results of the corresponding modal data and the dynamic modal conflict map to obtain a second weight; calculates fusion features based on the embedding features and the second weight corresponding to different modal data, and uses the fusion features to identify the learner's learning state to obtain a fusion recognition result.
[0172] Furthermore, the present invention also provides a computer program product, comprising a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer is able to execute the learning cabin multimodal data processing method provided by the above methods. This method includes: extracting embedding features corresponding to each modality from multimodal data generated by learners performing learning tasks in the learning cabin, and performing quality assessment on the multimodal data to obtain assessment results; obtaining individual baseline offset features corresponding to different modalities based on the individual baselines associated with the embedding features and the corresponding modalities; and using the embedding features corresponding to each modality to analyze the learner's learning state. The learning process involves line recognition to obtain multiple unimodal recognition results and calculate the confidence level corresponding to each unimodal recognition result. Based on the multiple unimodal recognition results and their corresponding confidence levels, the evaluation results corresponding to different modal data, and individual baseline offset features, logical relationships between different modal data are established, resulting in a dynamic modality conflict map and a multimodal relationship representation. A first weight is assigned to the multimodal data based on the individual baseline offset features and the multimodal relationship representation, and the first weight is constrained using the evaluation results of the corresponding modal data and the dynamic modality conflict map to obtain a second weight. A fusion feature is calculated based on the embedding features corresponding to different modal data and the second weight, and the fusion feature is used to identify the learner's learning state, resulting in a fusion recognition result.
[0173] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.
[0174] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.
[0175] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A method for processing multimodal data in a learning cabin, characterized in that, include: The embedding features corresponding to each modality are extracted from the multimodal data generated by learners performing learning tasks in the learning pod, and the quality of the multimodal data is evaluated to obtain the evaluation results. Based on the embedded features and the individual baselines associated with the corresponding modalities, individual baseline offset features corresponding to different modalities are obtained; the learning states of the learners are identified using the embedded features corresponding to each modality, resulting in multiple single-modal recognition results, and the confidence level corresponding to each single-modal recognition result is calculated; Based on the multiple single-modal recognition results and their corresponding confidence levels, the evaluation results corresponding to different modal data, and individual baseline offset features, a logical relationship between different modal data is established to obtain a dynamic modal conflict map and a multimodal relationship representation; The first weight is assigned to the multimodal data based on the individual baseline offset features and the multimodal relationship representation, and the first weight is constrained by the evaluation results of the corresponding modal data and the dynamic modal conflict diagram to obtain the second weight; The fusion features are calculated based on the embedding features and second weights corresponding to different modal data, and the learning state of the learner is identified using the fusion features to obtain the fusion identification result.
2. The learning cabin multimodal data processing method according to claim 1, characterized in that, The step of establishing logical relationships between different modal data based on the multiple single-modal recognition results and their corresponding confidence levels, the evaluation results corresponding to different modal data, and individual baseline offset features includes: Calculate the similarity and prediction difference between any two single-modal recognition results in the state dimension among the multiple single-modal recognition results; If the similarity is greater than the first threshold, and the evaluation results and confidence scores of the modal data corresponding to any two single-modal recognition results meet the preset conditions, the logical relationship between the modal data corresponding to any two single-modal recognition results is determined to be a support relationship. When the prediction difference is greater than the second threshold, and the confidence level of at least one of the modal data corresponding to any two single-modal recognition results satisfies the preset condition, the logical relationship between the modal data corresponding to any two single-modal recognition results is determined to be a conflict relationship.
3. The learning cabin multimodal data processing method according to claim 1, characterized in that, The step of assigning a first weight to the multimodal data based on the individual baseline offset features and the multimodal relationship representation includes: A state space is constructed based on the individual baseline offset features, multimodal relationship representation, and current learning task context information; The target fusion pattern for the current time window is determined using the upper-layer policy network of the hierarchical reinforcement learning model and the state space. The original importance scores of each modality data are calculated using the lower-level weight allocation network of the hierarchical reinforcement learning model, the state space, and the target fusion mode. The multiple original importance scores are then normalized to obtain the first weight.
4. The learning cabin multimodal data processing method according to claim 1, characterized in that, The step of constraining the first weight using the evaluation results of the corresponding modal data and the dynamic modal conflict graph includes: If the evaluation result of one of the modal data is lower than a preset quality threshold, the upper limit of the first weight corresponding to the one modal data shall be adjusted not to exceed the weight threshold. If a modality is marked as missing, the first weight corresponding to that modality is set to zero. When a modal data has a conflict relationship with multiple modal data in the dynamic modal conflict graph, and the evaluation result of the modal data is lower than a preset quality threshold, the first weight corresponding to the modal data is reduced to a preset weight value.
5. The learning cabin multimodal data processing method according to claim 1, characterized in that, The first weight is determined through a hierarchical reinforcement learning model; After obtaining the fusion recognition result, the method further includes: The target modal data in the multimodal data is masked, zeroed out, or replaced with the baseline to obtain the multimodal data after intervention. The multimodal data after the intervention is used to re-fuse features and identify them to obtain a comparative identification result. Based on the change in identification loss between the fused identification result and the comparative identification result, the counterfactual contribution of the target modality data in the current identification is calculated. A reinforcement learning reward function is constructed based on the counterfactual contribution and delayed feedback information corresponding to each modality data; the delayed feedback information is determined by the learner's answer performance, task completion time, or evaluation feedback in subsequent learning tasks; The reward value is calculated using the reinforcement learning reward function, and the policy parameters of the hierarchical reinforcement learning model are updated based on the reward value.
6. The learning cabin multimodal data processing method according to claim 1, wherein the multimodal data includes at least two of electrocardiogram data, eye-tracking data, video data, voice data, and learning behavior data; and the logical relationship includes at least one of support relationship, conflict relationship, complement relationship, and missing relationship; After obtaining the fusion recognition result, the method further includes: Obtain modal contribution descriptions; The modal contribution description includes the fusion recognition result, the second weight ratio corresponding to each modal data, and the logical relationship in the dynamic modal conflict graph.
7. A learning cabin multimodal data processing device, characterized in that, include: The feature extraction and evaluation module is used to extract the embedded features corresponding to each modality from the multimodal data generated by learners performing learning tasks in the learning cabin, and to evaluate the quality of the multimodal data to obtain the evaluation results. The single-modality recognition module is used to obtain individual baseline offset features corresponding to different modalities based on the embedded features and the individual baselines associated with the corresponding modalities; to identify the learning state of the learner using the embedded features corresponding to each modality, to obtain multiple single-modality recognition results, and to calculate the confidence level corresponding to each single-modality recognition result. The multimodal relationship establishment module is used to establish logical relationships between different modal data based on the multiple single-modal recognition results and corresponding confidence levels, the evaluation results corresponding to different modal data, and individual baseline offset features, to obtain a dynamic modal conflict map and a multimodal relationship representation; The weight calculation module is used to assign a first weight to the multimodal data based on the individual baseline offset features and the multimodal relationship representation, and to constrain the first weight using the evaluation results of the corresponding modal data and the dynamic modal conflict diagram to obtain a second weight; The comprehensive recognition module is used to calculate fusion features based on the embedding features and second weights corresponding to different modal data, and to use the fusion features to identify the learner's learning state to obtain the fusion recognition result.
8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the learning cabin multimodal data processing method as described in any one of claims 1 to 6.
9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the learning cabin multimodal data processing method as described in any one of claims 1 to 6.
10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the learning cabin multimodal data processing method as described in any one of claims 1 to 6.
