Multi-modal data fusion evaluation algorithm method for ideological and political education

Abstract

The invention relates to the technical field of ideological and political education evaluation, and discloses an ideological and political education-oriented multi-modal data fusion evaluation algorithm method. The method comprises the steps that text data, voice data, video data and behavior log data in the ideological and political education process are acquired through a multi-source acquisition module, and a multi-mode ideological and political education data warehouse is generated through standardized cleaning; performing semantic feature extraction on the data warehouse based on a graph modeling module to obtain an ideological and political education feature tensor, and constructing an ideological and political education multi-modal association graph in combination with each modal association graph; an ideological and political feature vector of each mode is extracted through a core quantification module, an ideological and political mode entropy is calculated in combination with the association map, and an ideological and political core feature set is screened according to an entropy value; the dynamic evaluation module is used for generating a real-time evaluation result of the ideological and political education process according to the core feature set, multi-modal data can be comprehensively integrated, ideological and political related features can be deeply mined, and the comprehensiveness and timeliness of evaluation are improved.

Description

Technical Field

[0001] This invention relates to the field of ideological and political education evaluation technology, specifically to a multimodal data fusion evaluation algorithm method for ideological and political education. Background Technology

[0002] Currently, the evaluation methods for ideological and political education still have significant limitations. Traditional evaluations often rely on single data sources, such as students' exam scores, written assignments, or class attendance records. This type of data can only reflect a partial aspect of the effectiveness of ideological and political education and is insufficient to comprehensively capture students' true state in multiple dimensions, including their ideological cognition, values, and behavioral performance. For example, grading assignments solely in text form cannot accurately determine students' true thoughts during class discussions, nor can it assess whether students' behavior in practical activities aligns with the goals of ideological and political education.

[0003] The diverse data generated during ideological and political education is often scattered, with text, audio, video, and behavioral log data stored in different systems or platforms, lacking an effective integration mechanism. This data fragmentation prevents the full utilization of the complementarity of multi-source data during evaluation, leading to biased and subjective evaluation results. For example, when evaluating students' ideological and political learning, relying solely on text assignments may overlook the depth of thought demonstrated in classroom discussions or the sense of social responsibility shown in practical activities, thus affecting the accuracy and comprehensiveness of the evaluation results.

[0004] Most existing evaluation methods lack in-depth analysis and semantic mining capabilities for multimodal data. For collected audio and video data, they often only perform simple format conversions or storage, failing to extract the ideological and political characteristics they contain, such as students' emotional tendencies and value judgments. Behavioral log data can only be used to statistically analyze superficial information such as students' study time and login frequency, making it difficult to uncover the underlying patterns of students' ideological changes and behavioral motivations. This insufficient utilization of multimodal data prevents ideological and political education evaluations from reflecting problems and shortcomings in the educational process in a timely and accurate manner, and also hinders the provision of effective guidance for optimizing ideological and political education based on evaluation results. Summary of the Invention

[0005] The purpose of this invention is to provide a multimodal data fusion evaluation algorithm method for ideological and political education, so as to solve the problems mentioned in the background art.

[0006] To achieve the above objectives, this invention provides a multimodal data fusion evaluation algorithm method for ideological and political education, the method comprising:

[0007] The text data, voice data, video data, and behavior log data generated during the ideological and political education process are acquired through a multi-source acquisition module. The text data, voice data, video data, and behavior log data are then standardized and cleaned to generate a multimodal ideological and political education data warehouse.

[0008] Based on the graph modeling module, semantic features are extracted from the multimodal ideological and political education data warehouse to obtain the ideological and political education feature tensor. Based on the modal association graph of each modality in the ideological and political education feature tensor, a multimodal association graph of ideological and political education is constructed.

[0009] The core quantization module extracts the ideological and political feature vectors of each modality from the ideological and political education feature tensor, calculates the ideological and political modal entropy of each modality by combining the multimodal association graph of ideological and political education, and selects the core feature set of ideological and political education based on all the ideological and political modal entropies.

[0010] The dynamic evaluation module generates real-time evaluation results of the ideological and political education process based on the core feature set of ideological and political education.

[0011] Preferably, the text data includes classroom discussion records and learning reflection texts, the voice data includes audio of teacher-student interactions, the video data includes videos of teaching scenarios, and the behavior log data includes operation records of the online learning platform.

[0012] Preferably, the standardization and cleaning of the text data, voice data, video data, and behavior log data specifically includes:

[0013] The speech data is subjected to noise reduction and speech-to-text processing to generate speech-text data;

[0014] Keyframe extraction and scene annotation are performed on the video data to generate video semantic data;

[0015] The behavior log data is timestamped and categorized by event type to generate structured behavior data;

[0016] The aforementioned voice text data, video semantic data, and structured behavioral data are integrated with the original text data along a timeline to form a multimodal ideological and political education data warehouse.

[0017] Preferably, semantic feature extraction of the multimodal ideological and political education data warehouse specifically includes:

[0018] Deep semantic encoding is performed on the text data, speech text data, video semantic data, and structured behavioral data in the multimodal ideological and political education data warehouse to generate initial feature vectors for each modality.

[0019] The initial feature vectors are aligned and fused using a cross-modal attention mechanism to generate a feature tensor for ideological and political education.

[0020] Preferably, constructing the multimodal association graph of ideological and political education specifically includes:

[0021] Select a target mode from the ideological and political education feature tensor, and calculate the cross-modal similarity between the initial feature vector of the target mode and the initial feature vectors of other modes.

[0022] Based on the cross-modal similarity, a modal association graph of the target modality is generated. After traversing all modalities, the modal association graphs are merged to form a multimodal association graph of ideological and political education.

[0023] Preferably, calculating the ideological and political mode entropy for each mode specifically includes:

[0024] The node centrality of each mode in the multimodal association graph of ideological and political education is obtained as the mode weight;

[0025] The ideological and political feature vectors are sorted by feature importance, and the ideological and political modal entropy of each modality is calculated in combination with the modal weights.

[0026] The preferred selection of the core feature set for ideological and political education specifically includes:

[0027] Candidate features with feature importance higher than a preset threshold are extracted from the ideological and political education feature tensor;

[0028] A comprehensive feature score is generated by weighted summation of the ideological and political modal entropy and cross-modal similarity corresponding to the candidate features;

[0029] The top N features, selected in descending order of their comprehensive scores, constitute the core feature set for ideological and political education.

[0030] Preferably, the real-time evaluation results of the ideological and political education process include:

[0031] The core feature set of ideological and political education is input into a temporal convolutional network to extract the temporal dynamic features of the ideological and political education process.

[0032] Based on the aforementioned time dynamic features and the topological structure of the multimodal association graph of ideological and political education, an evaluation score sequence for each time slice is generated.

[0033] Preferably, the multimodal association graph of ideological and political education is updated synchronously when extracting time-dynamic features, specifically as follows:

[0034] The cross-modal similarity weights are adjusted based on the evaluation score of the current time slice, and the edge connections of the political education multimodal association graph are reconstructed based on the adjusted weights.

[0035] Preferably, the deep semantic encoding adopts a Transformer-based multi-task joint training model, and the cross-modal attention mechanism adopts a gated bidirectional attention mechanism.

[0036] Compared with the prior art, the beneficial effects of the present invention are:

[0037] By employing a multi-source acquisition module, comprehensive data acquisition of text, audio, video, and behavioral logs generated during ideological and political education is achieved. Standardized cleaning is then performed to generate a multimodal ideological and political education data warehouse, effectively addressing the problem of single and fragmented data sources in traditional ideological and political education evaluation. The integration of multi-source data allows the evaluation process to fully utilize the characteristics of different data types, capturing relevant information from multiple dimensions such as text, audio, video, and behavior. This breaks the limitations of relying solely on single data sources for evaluation, making the information used in the evaluation more comprehensive and richer, and more completely reflecting the ideological and political education process and students' learning status.

[0038] In the data processing stage, semantic features were extracted from the multimodal ideological and political education data warehouse based on the graph modeling module, and a multimodal association graph of ideological and political education was constructed, realizing deep semantic mining of multimodal data. By extracting ideological and political education feature tensors and constructing modal association graphs, the originally scattered and disordered multimodal data can be transformed into structured information with clear semantic relationships, clearly presenting the inherent connections between different modalities, such as the correlation between students' opinions in text data and emotional tendencies in voice data, and the correlation between students' behavioral performance in video data and learning trajectories in behavioral log data. This semantic-level integration and association analysis enables the evaluation process to delve deeper into the ideological and political characteristics hidden behind the data, rather than merely utilizing the surface information, further enhancing the depth of understanding of the ideological and political education process.

[0039] The core quantification module extracts the ideological and political feature vectors of each modality and calculates the modal entropy of ideological and political education, thereby selecting the core feature set of ideological and political education and providing a precise quantitative basis for evaluation. The calculation of modal entropy effectively measures the richness and importance of ideological and political information in each modality's data. The core feature set obtained through selection focuses on information that is crucial to the evaluation of ideological and political education, avoiding the interference of redundant data on the evaluation results. This quantitative selection process makes the evaluation criteria more objective and clear, reduces the influence of subjective factors in traditional evaluation, and makes the evaluation results more credible and persuasive, more accurately reflecting the actual effect of ideological and political education.

[0040] The dynamic evaluation module generates real-time evaluation results based on the core characteristics of ideological and political education, enabling dynamic monitoring and feedback of the ideological and political education process. Real-time evaluation can promptly capture changes in the ideological and political education process, whether it's dynamic adjustments in students' ideological understanding or problems arising in the teaching and learning process, all of which can be quickly presented through real-time evaluation results. Educators can use the real-time evaluation results to promptly grasp the progress of ideological and political education, understand students' needs and confusions in the learning process, and then adjust teaching strategies in a timely manner, optimizing teaching content and methods. This allows ideological and political education to better adapt to students' learning pace and ideological development patterns, enhancing its relevance and effectiveness, and promoting a virtuous cycle of "evaluation-feedback-optimization" in the ideological and political education process. Attached Figure Description

[0041] Figure 1 This is a schematic diagram illustrating the working principle of the multimodal data fusion evaluation algorithm method for ideological and political education described in this invention.

[0042] Figure 2 A flowchart for standardized cleaning of multimodal data;

[0043] Figure 3 A flowchart for constructing a multimodal association graph for ideological and political education. Detailed Implementation

[0044] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0045] Please see Figure 1 This invention provides a multimodal data fusion evaluation algorithm method for ideological and political education, the method comprising: the collaborative operation of a multi-source acquisition module, a graph modeling module, a core quantification module and a dynamic evaluation module.

[0046] The multi-source acquisition module is responsible for acquiring text, voice, video, and behavioral log data in real time from ideological and political education scenarios. It then integrates this heterogeneous data into a unified-format multimodal ideological and political education data warehouse through a standardized cleaning process. The graph modeling module extracts semantic features from this data warehouse, generates an ideological and political education feature tensor, and further constructs a multimodal association graph to represent the semantic relationships between modalities. The core quantification module extracts ideological and political feature vectors for each modality from the feature tensor, calculates the modal entropy of ideological and political education based on the multimodal association graph, and selects the core feature set. The dynamic evaluation module uses the core feature set to analyze the dynamic changes in the ideological and political education process in real time and generates evaluation results. The entire method, through multimodal fusion and dynamic graph updates, achieves a comprehensive and objective evaluation of the effectiveness of students' ideological and political education.

[0047] Example 1: See Figure 2 The multi-source acquisition module in ideological and political education scenarios encompasses the orderly acquisition and processing of various data types. Classroom discussion records and learning reflections, as the main components of text data, are collected through the education platform interface or manual input. Audio data, such as teacher-student interaction audio, is collected using microphone arrays deployed in the classroom or mobile recording equipment. Teaching scene videos are recorded by fixed cameras or handheld video devices and stored as high-definition files. Behavioral log data, such as online learning platform operation records, is obtained through the system's backend log export function. After the raw data is acquired, it immediately enters a standardized cleaning process to eliminate noise and inconsistencies. The entire processing flow adopts a pipeline architecture to achieve automated data preprocessing. The audio data processing adopts a staged noise reduction strategy. First, a filter based on spectrum analysis is applied to suppress environmental background noise. Then, dynamic threshold detection is used to remove sudden interference sounds. The processed audio segments are input into the speech recognition engine for conversion. The speech recognition model is trained based on a deep learning architecture and can adapt to the diverse pronunciations and vocabulary in educational scenarios. The converted audio-text data is accompanied by timestamps and speaker tags. Video data processing begins with keyframe extraction. An adaptive sampling algorithm dynamically adjusts the extraction frequency based on the magnitude of image changes. The extracted keyframes are then input into a pre-trained image recognition model to generate scene description labels. These labels cover multi-dimensional information, including object recognition, action detection, and sentiment analysis. The resulting video semantic data is stored in a structured format. Behavioral log data cleaning focuses on time series alignment. Time information in the original logs is converted to a unified time zone and format. Operation events are categorized into types such as login, browsing, and interaction based on a predefined rule base. The categorized data is then reconstructed into structured behavioral data with time-series tags.

[0048] Multimodal data integration relies on a unified timeline. Timestamps of all data segments are converted to millisecond precision and sorted. Text data retains its original content and formatting marks; audio-text data is supplemented with audio quality indicators; video semantic data is associated with keyframe coordinate information; and structured behavioral data is embedded with event type codes. A cross-modal indexing mechanism is established during the integration process, enabling rapid retrieval and association of data at any given time. The resulting multimodal ideological and political education data warehouse adopts a layered storage architecture. The raw data layer retains unmodified initial data; the cleaned data layer stores processed standardized data; and the index layer maintains the mapping relationship between timestamps and modality types. The data warehouse supports multi-dimensional queries by time range, modality type, content keywords, and other dimensions. Data quality control is implemented throughout the entire process. After speech-to-text conversion, manual sampling verification is performed to correct recognition errors. Video annotation results are improved in accuracy through multi-model cross-validation. Behavioral log classification rules are updated regularly to adapt to platform function changes. Data warehouse integrity checks are performed automatically daily, and missing or abnormal data triggers a re-collection process. The maintenance of multimodal data correlation relies on a time synchronization mechanism. All acquisition devices' clocks are periodically calibrated, and the acquisition device number and geographical location information are automatically marked when data flows in, providing contextual support for subsequent cross-modal analysis. Fault tolerance measures during implementation include activating a backup recognition engine when speech recognition fails, recovering from the most recent keyframe when video processing is interrupted, isolating abnormal segments and recording error logs when log parsing errors occur, and designing redundant backup nodes in the data warehouse to prevent single points of failure. The entire multi-source acquisition and cleaning process operates automatically, with manual intervention only occurring during quality checks and anomaly handling, thus ensuring the efficiency and consistency of ideological and political education data processing.

[0049] The data acquisition process employs differentiated strategies based on the characteristics of different modalities. Text data undergoes initial cleaning using natural language processing (NLP) techniques to remove irrelevant symbols and formatting marks. Speech data retains paralinguistic features such as intonation and speech rate during conversion. Video data analysis focuses not only on visual content but also records audio channel information. Behavioral log parsing preserves the complete trajectory of user operation sequences. During standardization, all data is converted to UTF-8 encoding format, timestamps are uniformly standardized using ISO 8601, numerical data is normalized to the same units, and categorical data is mapped to a unified encoding system. The architecture of the multimodal data warehouse considers subsequent analysis needs, establishing a subject domain division mechanism to categorize ideological and political education-related content into different subject domains. Data quality monitoring points are set up to continuously track completeness, accuracy, and consistency indicators, and data lineage tracing records the processing history of each data point. The data access interface provides multiple query methods, supports time-window-based segment extraction and modality-based conditional filtering, and offers both batch export and streaming data delivery modes. The data processing pipeline adopts a modular design, with each functional module capable of independent upgrades and expansions. The speech recognition module supports multiple dialects and customized terminology databases, the video analysis module can identify specific teaching scenarios and teacher-student interaction patterns, and the behavior log parsing module adapts to the data formats of different learning platforms. Detailed operation logs and performance metrics are generated throughout the system's operation, facilitating monitoring of each stage's processing status and troubleshooting anomalies. Data security measures are implemented consistently. Sensitive information is anonymized during the collection phase, data transmission uses encrypted channels, data storage implements hierarchical access control, and all access operations leave audit trails. The system design considers the specificities of educational scenarios, supporting offline processing to cope with unstable network environments, providing data compression to reduce storage and transmission pressure, and possessing horizontal scalability to meet large-scale data processing needs. During standardized cleaning, special attention is paid to maintaining the semantic integrity of the data. Emotional expression and rhetorical features are preserved during text data cleaning; differentiating speech segments from different speakers is considered during speech conversion; nonverbal communication information is accurately captured during video analysis; and the operational context is maintained during behavior log processing. In the data integration phase, cross-modal association rules are established, time tolerance thresholds are defined to handle clock deviations between different devices, and content matching algorithms are set to discover semantic associations between different modalities. The resulting multimodal ideological and political education data warehouse not only stores raw and cleaned data but also includes rich metadata descriptions. Each data segment comes with a quality score and confidence index, providing a reliable data foundation for subsequent analysis modules. The warehouse maintains data version management functions, supporting historical data backtracking and processing process reproduction, meeting the requirements of ideological and political education evaluation for data credibility and traceability.

[0050] Example 2: The semantic feature extraction process is based on cleaned text, speech text, video semantics, and structured behavior data from the multimodal ideological and political education data warehouse. Deep semantic encoding employs a Transformer-based multi-task joint training model architecture, which includes a dual-path design with shared encoding layers and modality-specific encoding layers. The text data processing path is initialized using a variant of the BERT model, and continuous pre-training is performed on the ideological and political education corpus to adapt to professional terminology and expression styles. Speech text data is processed through the same text encoding pipeline but with an additional phoneme embedding layer to preserve pronunciation features. Video semantic data uses the VisionTransformer architecture to extract spatial features and integrates a temporal attention mechanism. Structured behavior data uses a multilayer perceptron encoder to process mixed numerical and categorical inputs. The initial feature vectors for each modality are uniformly normalized to 768 dimensions to ensure consistency in subsequent processing. During feature extraction, the attention weight distribution is preserved for interpretability analysis.

[0051] The cross-modal attention mechanism implements a gated bidirectional attention mechanism, which comprises two core stages: feature alignment and information fusion. In the feature alignment stage, the cross-attention weight matrix between any two modal feature vectors is calculated. A learnable query-key-value pair mechanism captures fine-grained associations between modalities, and the gating unit uses a sigmoid activation function to generate dynamic weights to control the information flow intensity. In the information fusion stage, the weighted features are concatenated and then dimensionality reduced to the target dimension through a linear transformation layer. Residual connections are used during the fusion process to preserve the original feature information, resulting in a final output ideological and political education feature tensor containing a unified 512-dimensional feature representation. The entire processing flow employs an end-to-end training approach, and the loss function combines intra-modal reconstruction loss and inter-modal contrastive loss to improve feature quality.

[0052] The multi-task joint training model is designed to balance modality specificity and cross-modal consistency. A shared encoding layer learns a general cross-modal representation, while modality-specific encoding layers focus on mining unique features for each modality. Training data is randomly sampled from a multimodal ideological and political education data warehouse. Model optimization employs an adaptive learning rate adjustment strategy, emphasizing intra-modal feature learning in the early stages and enhancing cross-modal alignment in later stages. Regularization techniques include random feature masking and random modality discarding to prevent overfitting. The model inference phase supports both single-modal and multimodal joint input modes, and all output feature vectors are mapped to the same semantic space for easy subsequent correlation analysis.

[0053] The specific implementation of the gated bidirectional attention mechanism includes a multi-head attention design, where each attention head focuses on cross-modal interactions at different levels, and the head outputs are weighted and aggregated by the gating network. The gating network parameters are dynamically generated from the input features, ensuring adaptive adjustment of the fusion strategy for different input content. Attention calculation uses a scaled dot product method to avoid the gradient vanishing problem. During implementation, the stability of the attention distribution is continuously monitored, and abnormal attention weights are smoothed to maintain the reliability of the feature fusion process. After feature tensor generation, quality verification is performed. Feature quality is evaluated by calculating feature reconstruction error and inter-modal consistency indices; unqualified samples trigger a reprocessing process. Tensor storage employs a hierarchical compression strategy, storing original features and fused features separately for subsequent analysis and traceability. A bidirectional mapping relationship is established between the feature index and the metadata of the multimodal data warehouse. The entire semantic feature extraction process supports incremental learning, enabling the gradual integration of new data without complete retraining, adapting to the dynamic evolution of ideological and political education content. The pipeline design considers a balance between computational efficiency and accuracy. For text and speech data, a hierarchical feature extraction strategy is employed: the lower layer captures local features, while the higher layer integrates global semantics. For video data processing, spatial pyramid pooling is introduced to adapt to inputs at different resolutions. The behavioral data encoder integrates a feature importance evaluation mechanism to automatically filter low-value features. Cross-modal attention computation uses a block-based processing strategy to handle long input sequences, and memory usage optimization is achieved through gradient checkpointing and dynamic memory allocation.

[0054] The model training environment is configured with a distributed computing framework, supporting multi-GPU parallel training and data parallel processing. The training process records complete hyperparameters and performance metrics for easy experimental reproduction. The inference service deployment provides a high-concurrency interface, supporting both real-time feature extraction and batch processing modes. Service monitoring includes key metrics such as resource utilization and processing latency. A feature quality assurance system establishes an automatic detection mechanism, regularly running feature consistency checks and new data adaptability tests to ensure the continuous reliability of the feature extraction process. The feature storage format adopts a standardized protocol, including meta-information such as feature vectors, confidence scores, timestamps, and modality types, supporting fast retrieval and cross-system exchange. Feature version management records model versions and parameter configurations, facilitating feature traceability and performance comparison. The feature update mechanism supports both full and incremental updates to meet update needs in different scenarios. The entire implementation process emphasizes feature interpretability; visualization of attention weights and gating values ​​helps understand cross-modal interaction patterns, providing insights for subsequent analysis. The final output of the ideological and political education feature tensor serves as the foundation for subsequent graph construction and feature selection. Its design fully considers the specific characteristics of ideological and political education content, achieving cross-modal semantic alignment while maintaining the richness of features across modalities, thus providing high-quality feature representations for multimodal fusion evaluation. The tensor generation process is automated and streamlined, minimizing the need for manual intervention and ensuring the objectivity and consistency of feature extraction.

[0055] Taking a university ideological and political education class as an example, the teacher is conducting a special lecture. The class adopts a blended teaching model, including multiple stages such as teacher explanation, group discussion, video presentation, and student interaction. A multimodal data acquisition system synchronously records the entire teaching process: text data includes students' digital notes and text statements in the online discussion area; audio data is collected through array microphones deployed in the classroom to capture teacher-student dialogues; video data records the live classroom teaching through a panoramic camera; and behavioral log data comes from the backend records of the online learning platform, including students' page browsing, Q&A interactions, and resource downloads. Text data processing first involves preprocessing to remove irrelevant symbols and stop words, and marking professional terms. The BERT model loads pre-trained weights from the ideological and political education domain to encode the cleaned text, generating a 768-dimensional text feature vector. This vector captures the semantic content and sentiment of the text. After noise reduction, the speech recognition engine converts the teacher's explanation and students' statements into text. This text is also input into the BERT model to generate a speech-text feature vector, but unlike pure text processing, the speech-text retains the speaker's prosodic markers and pause information. Video data is sampled one frame per second using a keyframe extraction algorithm, and features are extracted from each frame using the VisionTransformer model. The model pays particular attention to the identification of subjects in the scene: such as the teacher's gestures and expressions, the students' reactions, and visual elements such as the PPT content, while also analyzing textual information in the video (such as courseware subtitles). The extracted visual feature vectors contain spatial and temporal features, and are finally aggregated into a 512-dimensional video semantic feature vector. Behavioral log data is structured, classifying students' operational behaviors into types such as "listening," "interacting," "thinking," and "note-taking," and encoded into a 128-dimensional behavioral feature vector using a multilayer perceptron. The feature vectors of each modality are input into a gated bidirectional attention mechanism for fusion. This mechanism first calculates the attention weight between text features and speech text features. The attention weight distribution shows that when the speech text contains emphasis, its association weight with the text features is automatically increased; when key content appears in the video, the attention connection between visual features and text features is strengthened. The gating unit dynamically adjusts the information flow according to feature quality; for example, when the speech recognition confidence is low, the influence of speech features on the fusion result is reduced. During cross-modal fusion, the mechanism detects frequent occurrences of keywords in text features, while in speech features, the teacher amplifies the pronunciation intensity of the word; in video features, relevant case studies are displayed on corresponding PPT slides; and in behavioral features, students interact frequently during this period. The mechanism calculates the correlation between these cross-modal signals through multi-head attention, ultimately generating a comprehensive 512-dimensional feature tensor. This tensor integrates the teaching representations of keyword concepts from multimodal signals. The entire processing is performed in real-time, with the feature tensor continuously updated as classroom teaching progresses, forming a time-series multimodal feature representation. The feature tensor is stored with timestamps and modality source information for easy subsequent analysis and traceability.At the end of the class, the feature intensity of keywords increased in the text and speech modalities, the frequency of student interaction increased in the video modalities, and the downloading of materials occurred frequently in the behavioral modalities. These multimodal signals were uniformly represented in the feature tensor. The final generated feature tensor contains multimodal fusion information of the entire class, preserving the characteristics of each modality and reflecting cross-modal correlations, providing high-quality feature input for subsequent graph construction and evaluation. The entire processing is completed automatically at the end of the class without manual intervention, realizing intelligent feature extraction and fusion of the ideological and political education classroom teaching process.

[0056] Example 3: See Figure 3 The construction of the multimodal correlation graph of ideological and political education is based on the initial feature vectors of each modality extracted from the feature tensor of ideological and political education. First, a target modality needs to be selected as the benchmark for analysis. The selection of the target modality is usually determined according to specific analytical needs; it may focus on text modality or video modality as the core analytical dimension. After selection, the cross-modal similarity measure between the feature vector of the target modality and the feature vectors of all other modalities is calculated. The similarity calculation adopts an improved cosine similarity method, which can effectively handle the comparison of high-dimensional sparse feature vectors while overcoming the evaluation bias problem of traditional cosine similarity when the vector magnitudes differ significantly.

[0057] A modality-specific adjustment factor is introduced during cross-modal similarity calculation. This factor is dynamically generated based on the data distribution characteristics of each modality feature and is used to eliminate evaluation bias caused by inherent differences between different modalities. For any two modality feature vectors, the adjusted similarity calculation formula is:

[0058] ;

[0059] in: This represents the adjusted cross-modal similarity value, which is a scalar value within the range of [-1, 1]. and These represent the feature vectors of the two modes, respectively. and The corresponding components of the vector, and It is the mean of the eigenvectors. and The standard deviation of the eigenvectors is represented by... The smoothing hyperparameter is used to control the adjustment strength of distribution differences. This calculation process is performed independently for the target mode and each other mode, generating a symmetric similarity matrix that records the correlation strength between all modes.

[0060] Based on the calculated similarity matrix, a modality association graph is constructed for the target modality. Nodes in the graph represent different modalities, and edge weights are the calculated adjusted similarity values. Node attributes include meta-information such as modality type, feature dimension, and data quality metrics. Edge attributes, in addition to storing similarity values, also record confidence metrics and timestamps generated during the calculation process. Each modality association graph is stored using an adjacency list structure, while also preserving the complete adjacency matrix for subsequent graph algorithm processing. After constructing the association graph for one target modality, the above process is repeated for all modalities, generating independent association graphs for text, speech, video, and behavioral modalities respectively. Each association graph describes the intrinsic connections of the entire multimodal system from the perspective of a specific modality. The text modality association graph focuses on semantic-level connections, the video modality association graph highlights the correspondence of visual features, the speech modality association graph focuses on the degree of matching of audio features, and the behavioral modality association graph emphasizes the similarity of operational patterns. The merging of the independent modal association graphs adopts a fusion strategy based on the maximum connected subgraph. First, the nodes in each graph are uniformly identified and their metadata is aligned. Then, the edge weights are weighted and averaged, with the weight allocation depending on the importance of each modality in the current analysis. During the merging process, potential conflicts are detected and resolved. For example, if there are edge connections with excessively different similarity values, the weight value of the edge is recalculated through a negotiation mechanism. The final multimodal association graph of ideological and political education contains all modal nodes and rich edge connections, forming a complete and complex network structure.

[0061] The graph storage utilizes graph database technology, with full-text indexes for node and edge attributes supporting complex queries. It also preserves a complete historical record of graph construction for easy version backtracking. The graph visualization interface offers various layout algorithms, intuitively displaying the association strength and clustering features between different modalities, and supports interactive exploratory analysis. Graph quality assessment is conducted through two dimensions: internal consistency testing and external validity verification. Internal consistency testing measures the self-consistency and stability of the graph structure, while external validity verification compares the graph output with domain knowledge. Significant anomalies discovered during the assessment process trigger a recalculation process to ensure the accuracy and reliability of the final graph. The completed multimodal association graph for ideological and political education serves as the foundation for subsequent feature selection and dynamic evaluation. Its rich topological information supports a deeper understanding of the intrinsic connections between multimodal data. The graph maintenance mechanism includes regular updates and incremental expansion functions, adapting to the dynamic changes in data during ideological and political education, ensuring the timeliness and accuracy of analysis results. The entire construction process is automated and streamlined, requiring no manual intervention from feature input to graph output, ensuring the objectivity and repeatability of graph construction.

[0062] Example 4: Core Feature Selection from Multimodal Association Graphs and Feature Vectors in Ideological and Political Education. First, the node centrality index for each modality needs to be extracted from the graph structure. The node centrality is calculated using the eigenvector centrality algorithm, which considers the relative importance of nodes in the graph and the strength of their connections with other nodes. The text modality, speech modality, video modality, and behavioral modality receive centrality weights of 0.35, 0.28, 0.22, and 0.15, respectively. These weights reflect the importance of different modalities in the overall ideological and political education data. The text modality receives the highest weight due to its rich semantic information, while the behavioral modality has a relatively lower weight due to data sparsity. Feature importance ranking uses an interpretability method based on gradient boosting decision trees to evaluate the contribution of each feature vector in the ideological and political education feature tensor. Taking the text modality feature vector as an example, it includes sub-features such as lexical complexity, sentiment polarity, and topic relevance. The importance score of each sub-feature is calculated using the information gain during decision tree splitting. Voice modality features include dimensions such as intonation variation, speech rate stability, and emotional intensity; video modality features cover indicators such as attention concentration, interaction frequency, and facial expression richness; and behavioral modality features include parameters such as operation frequency, learning duration, and number of interactions. The importance scores of all features are normalized to form a unified and comparable importance ranking.

[0063] The calculation of modal entropy for ideological and political education integrates two dimensions: modal weight and feature importance. The modal entropy value of each feature is calculated using a weighted information entropy formula. In the text modality, the importance score of the feature "frequency of political terminology" is 0.92, and combined with the text modal weight of 0.35, the modal entropy value of this feature is calculated to be 0.322. In the video modality, the importance score of the feature "attention concentration" is 0.85, and combined with the video modal weight of 0.22, the modal entropy value is 0.187. After the modal entropy values ​​of all features are calculated, they are arranged in descending order of numerical value to form a feature importance spectrum. A dynamic threshold mechanism is set for candidate feature selection. The threshold is determined based on the 75th percentile of the historical data distribution, and the threshold in the current analysis period is 0.18. Features with importance scores higher than this threshold are included in the candidate set, which includes 5 features from the text modality, 3 features from the speech modality, 4 features from the video modality, and 2 features from the behavioral modality, for a total of 14 candidate features. Each candidate feature records complete metadata information, including feature ID, modality, original importance score, modality entropy value, and other attributes. Cross-modal similarity is introduced as a weighting factor in the calculation of the feature composite score; this similarity data is extracted from the multimodal association graph constructed in Example 3. For each candidate feature, the mean similarity with all other modal features is calculated as the cross-modal influence index of that feature. The composite score is obtained by weighted summation of modality entropy value and cross-modal similarity, with weight coefficients determined to be 0.6 and 0.4 through grid search optimization. Taking the "frequency of political term usage" feature of the text modality as an example, its modality entropy value is 0.322, its cross-modal similarity is 0.86, and its composite score is 0.322×0.6+0.86×0.4=0.545.

[0064] Table 1: Calculation of Comprehensive Score for Candidate Features

[0065]

[0066] In the final selection process, the top N features are chosen in descending order of their comprehensive scores to form the core feature set for ideological and political education. The value of N is dynamically adjusted based on the needs of the specific application scenario. In the current teaching scenario, N is set to 8. The features selected for the core feature set include 3 features from the text modality, 2 features from the video modality, 2 features from the speech modality, and 1 feature from the behavioral modality. The core feature set is arranged in descending order of comprehensive scores, while retaining the source modality and calculation process data for each feature to facilitate subsequent analysis, traceability, and result interpretation. The core feature set is updated using a sliding window strategy. After processing each new batch of data, the feature importance is recalculated and the ranking is updated to maintain the timeliness and adaptability of the feature set. Feature set version management records detailed logs of each update, including feature changes, reasons for score fluctuations, and related parameter adjustments, forming a complete feature evolution trajectory. The final output core feature set for ideological and political education serves as input to the dynamic evaluation module, providing feature support for the real-time evaluation of the ideological and political education process.

[0067] Example 5: The dynamic evaluation module receives the core feature set of ideological and political education from the core quantification module. This feature set contains eight of the most representative feature dimensions selected from multimodal data. These features are organized according to time series, with each feature corresponding to a numerical representation in each time slice, forming a feature sequence of length 256 time steps. The temporal convolutional network adopts a multi-layer causal convolutional architecture. The first layer uses 128 filters, a kernel size of 5, and a dilation coefficient of 1; the second layer uses 64 filters, a kernel size of 3, and a dilation coefficient of 2; and the third layer uses 32 filters, a kernel size of 3, and a dilation coefficient of 4. After processing by a global pooling layer and a fully connected layer, the network output generates a dynamic feature representation for each time slice. These representations capture the evolution of patterns in ideological and political education over time. The extraction process of temporal dynamic features adopts a sliding window mechanism, with a window size of 16 time steps and a stride of 8 time steps, ensuring a smooth transition of features between consecutive time slices. The feature data within each window is standardized and then input into the network. The network output includes not only the feature representation of the current time slice but also the predicted features for the next three time slices, forming a dynamic feature sequence with temporal foresight. These dynamic features are fused with the topological structure of the multimodal association graph of ideological and political education. The graph node features and temporal features interact through a graph attention mechanism to generate a comprehensive evaluation score for each time slice.

[0068] The evaluation score sequence is generated based on a neural network regression model, which consists of two components: a graph convolutional layer and a temporal fully connected layer. The graph convolutional layer processes the node relationships in the multimodal association graph, encoding the topological structure information into graph embedding vectors. The temporal fully connected layer concatenates the dynamic features with the graph embedding vectors to calculate the final score. The score output uses a continuous numerical range of 0-100, with one score result generated every 8 time slices. A total of 32 evaluation scores are generated for the entire 256 time slice sequence, and these scores are arranged in chronological order to form the evaluation score sequence. The update of the multimodal association graph is performed synchronously with the extraction of temporal dynamic features. The evaluation score of the current time slice is used as a feedback signal to adjust the edge connection weights of the graph. Time slices with higher evaluation scores receive a positive boost to the cross-modal similarity weights, while time slices with lower scores trigger a similarity weight downsizing mechanism. The weight adjustment adopts a gradient-based optimization method, with each update amplitude controlled within ±15% of the initial weights to avoid drastic changes in the graph structure. The updated graph edge weights recalculate node centrality, which in turn affects the feature fusion method of subsequent time slices, forming a dynamic feedback loop.

[0069] During the update process, the deviation between the current evaluation score and the historical average score is first calculated, and this deviation is mapped to the weight adjustment coefficient. For text-video modal edge connections, the weight is increased by 8% when the score deviation is +5 points, and decreased by 7% when the score deviation is -5 points. After the edge connection weights between all modalities are adjusted synchronously, the topological attributes of the graph are recalculated, including indicators such as node degree distribution, clustering coefficient, and path length, to ensure the stability of the graph structure. The updated graph is immediately used for feature fusion in the next time slice, achieving real-time adaptive adjustment. The evaluation results output includes detailed time series data and graph change records. Each evaluation score is accompanied by a generated timestamp, a list of participating features, and a confidence index. The system records detailed logs for each weight adjustment of the graph, including the weight before adjustment, the adjustment magnitude, the reason for adjustment, and the impact assessment. This data is used for subsequent analysis and system optimization. The evaluation interface provides a visualization function, which can simultaneously display the dynamic changes of the evaluation score curve and the multimodal association graph, and supports tracing back to detailed scoring basis at any time point. The entire implementation process employs a distributed computing architecture, with temporal convolutional network inference and graph update operations executed in parallel to ensure the system's efficiency in processing real-time data streams. The network model undergoes incremental training every 24 hours, incorporating the latest evaluation feedback data to maintain predictive accuracy. During system operation, the stability of the evaluation score distribution is continuously monitored, and abnormal fluctuations are automatically detected and alerted to ensure the reliability and consistency of the evaluation results. The final output evaluation score sequence serves as the basis for quality assessment of the ideological and political education process, providing data support for educational improvement.

[0070] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0071] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A multimodal data fusion evaluation algorithm method for ideological and political education, characterized in that, Includes the following steps: The text data, voice data, video data, and behavior log data generated during the ideological and political education process are acquired through a multi-source acquisition module. The text data, voice data, video data, and behavior log data are then standardized and cleaned to generate a multimodal ideological and political education data warehouse. Based on the graph modeling module, semantic features are extracted from the multimodal ideological and political education data warehouse to obtain the ideological and political education feature tensor. Based on the modal association graph of each modality in the ideological and political education feature tensor, a multimodal association graph of ideological and political education is constructed. The core quantization module extracts the ideological and political feature vectors of each modality from the ideological and political education feature tensor, calculates the ideological and political modal entropy of each modality by combining the multimodal association graph of ideological and political education, and selects the core feature set of ideological and political education based on all the ideological and political modal entropies. The dynamic evaluation module generates real-time evaluation results of the ideological and political education process based on the core feature set of ideological and political education.

2. The multimodal data fusion evaluation algorithm method for ideological and political education according to claim 1, characterized in that, The text data includes classroom discussion records and learning reflections; the audio data includes audio of teacher-student interactions; the video data includes videos of teaching scenarios; and the behavior log data includes operation records of the online learning platform.

3. The multimodal data fusion evaluation algorithm method for ideological and political education according to claim 1, characterized in that, The standardization and cleaning of the text data, voice data, video data, and behavior log data specifically includes: The speech data is subjected to noise reduction and speech-to-text processing to generate speech-text data; Keyframe extraction and scene annotation are performed on the video data to generate video semantic data; The behavior log data is timestamped and categorized by event type to generate structured behavior data; The aforementioned voice text data, video semantic data, and structured behavioral data are integrated with the original text data along a timeline to form a multimodal ideological and political education data warehouse.

4. The multimodal data fusion evaluation algorithm method for ideological and political education according to claim 1, characterized in that, Semantic feature extraction for the multimodal ideological and political education data warehouse specifically includes: Deep semantic encoding is performed on the text data, speech text data, video semantic data, and structured behavioral data in the multimodal ideological and political education data warehouse to generate initial feature vectors for each modality. The initial feature vectors are aligned and fused using a cross-modal attention mechanism to generate a feature tensor for ideological and political education.

5. The multimodal data fusion evaluation algorithm method for ideological and political education according to claim 1, characterized in that, The construction of the aforementioned multimodal association graph of ideological and political education specifically includes: Select a target mode from the ideological and political education feature tensor, and calculate the cross-modal similarity between the initial feature vector of the target mode and the initial feature vectors of other modes. Based on the cross-modal similarity, a modal association graph of the target modality is generated. After traversing all modalities, the modal association graphs are merged to form a multimodal association graph of ideological and political education.

6. The multimodal data fusion evaluation algorithm method for ideological and political education according to claim 1, characterized in that, The calculation of the ideological and political modal entropy for each modality specifically includes: The node centrality of each mode in the multimodal association graph of ideological and political education is obtained as the mode weight; The ideological and political feature vectors are sorted by feature importance, and the ideological and political modal entropy of each modality is calculated in combination with the modal weights.

7. The multimodal data fusion evaluation algorithm method for ideological and political education according to claim 1, characterized in that, The specific features for selecting the core characteristics of ideological and political education include: Candidate features with feature importance higher than a preset threshold are extracted from the ideological and political education feature tensor; A comprehensive feature score is generated by weighted summation of the ideological and political modal entropy and cross-modal similarity corresponding to the candidate features; The top N features, selected in descending order of their comprehensive scores, constitute the core feature set for ideological and political education.

8. The multimodal data fusion evaluation algorithm method for ideological and political education according to claim 1, characterized in that, The real-time evaluation results of the ideological and political education process specifically include: The core feature set of ideological and political education is input into a temporal convolutional network to extract the temporal dynamic features of the ideological and political education process. Based on the aforementioned time dynamic features and the topological structure of the multimodal association graph of ideological and political education, an evaluation score sequence for each time slice is generated.

9. The multimodal data fusion evaluation algorithm method for ideological and political education according to claim 8, characterized in that, When extracting time-dynamic features, the multimodal association graph of ideological and political education is updated synchronously, specifically as follows: The cross-modal similarity weights are adjusted based on the evaluation score of the current time slice, and the edge connections of the political education multimodal association graph are reconstructed based on the adjusted weights.

10. The multimodal data fusion evaluation algorithm method for ideological and political education according to claim 4, characterized in that, The deep semantic encoding adopts a Transformer-based multi-task joint training model, and the cross-modal attention mechanism adopts a gated bidirectional attention mechanism.

Images