Online resource adaptive recommendation method for multi-modal learning behavior analysis

Abstract

The invention discloses an online resource adaptive recommendation method based on multi-modal learning behavior analysis, and relates to the technical field of resource recommendation. The method comprises the following steps: firstly, dynamically collecting multi-modal data by using a heterogeneous sensor array, and carrying out noise reduction, probability distribution matching normalization and time-space alignment preprocessing; features are extracted through a hierarchical network, modeling learning behaviors such as a variational auto-encoder are combined, and the learning state is evaluated from multiple dimensions; recommendation decisions are generated based on reinforcement learning, recommendation is optimized in combination with personalized presentation and multi-source feedback analysis, meanwhile, the system has the functions of dynamic strategy adjustment, intelligent resource creation, cross-scene migration recommendation and the like, and accurate self-adaptive recommendation is achieved. According to the method, multi-modal data are comprehensively collected and deeply processed, learning behaviors and evaluation states are accurately analyzed, personalized resource recommendation is provided through intelligent recommendation and dynamic optimization strategies, recommendation accuracy and learning effects can be improved, user experience can be enhanced, and the utilization rate and competitiveness of platform resources can be improved.

Description

Technical Field

[0001] The present invention relates to the technical field of resource recommendation, and in particular to an online resource adaptive recommendation method based on multimodal learning behavior analysis. Background Art

[0002] With the rapid development of Internet technology, online learning has become an important way for people to acquire knowledge and skills. The vast amount of online learning resources provides learners with a wealth of choices, but it also brings the problem of difficulty in resource screening. Traditional online resource recommendation methods are mostly based on single-modal data. For example, recommendations are made based solely on text data such as learners' historical click records and search keywords. This fails to fully and accurately understand learners' learning behaviors and needs. During online learning, learners' learning behaviors and status are presented through multi-modal information, including operational behaviors during the learning process, physiological reactions during learning, and interactions with others. The limitations of single-modal data lead to deviations between recommendation results and learners' actual needs, making it difficult to meet personalized learning needs.

[0003] Existing multimodal learning behavior analysis and resource recommendation technologies still have many shortcomings. In terms of data collection, the collection of multimodal data lacks systematicity and dynamism, and the collection strategy cannot be adjusted according to the learner's real-time status and environmental changes, resulting in uneven quality of the collected data and the presence of a large amount of redundant or invalid data. In the data processing and analysis stage, the fusion method of multimodal data is relatively simple, failing to fully explore the potential correlation between different modal data. The feature extraction is not deep enough, making it difficult to accurately portray the learner's learning behavior pattern. At the same time, the learning status assessment system is not perfect, often focusing only on single indicators such as academic performance, ignoring the impact of learners' emotional state, social interaction, etc. on learning outcomes.

[0004] Furthermore, existing recommendation models and strategies lack adaptability and dynamic optimization capabilities. Once trained, recommendation models struggle to adapt quickly to the dynamic changes in learners' learning behaviors and the constant updating of learning resources, leading to a gradual decline in recommendation effectiveness. Recommendation strategies are also relatively rigid and cannot be flexibly adjusted to suit learners' different learning stages, learning goals, and learning scenarios. This makes it impossible to provide learners with accurate and efficient resource recommendation services, hindering the effectiveness of online learning and improving user experience. Summary of the Invention

[0005] The present invention proposes an online resource adaptive recommendation method based on multimodal learning behavior analysis to solve the problems mentioned in the above-mentioned prior art.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: an online resource adaptive recommendation method based on multimodal learning behavior analysis, comprising:

[0007] Data collection and dynamic perception steps: A heterogeneous sensor array is used to collaboratively collect learners' data. In addition to video viewing, text reading, interaction and physiological data, wearable devices are used to collect skin electrical response and body acceleration data. Environmental sensors are used to obtain the light intensity and noise decibel environment data of the learning space to build a dynamic perception model. According to the learner's learning stage, device performance, and network status, the dynamic perception model is constructed through the formula Dynamically adjust the data collection frequency, where f rate is the acquisition frequency, S load is the current load of the device, S total is the total load capacity of the equipment, α and β are adjustment coefficients, C comp Score network connectivity;

[0008] Data fusion preprocessing steps: First, use the Transformer-based denoising autoencoder to denoise the original data, and use the attention mechanism to capture data dependencies to remove noise; in the normalization stage, use the normalization method based on probability distribution matching, and for data x, use the formula Mapping to the target probability distribution, where p(x) is the probability density of data x, using a spatiotemporal alignment algorithm to learn the event trigger time as an anchor point, combined with dynamic time warping technology to solve the time offset problem of different modal data;

[0009] Feature layer extraction and enhancement steps: Construct a layered feature extraction network. The bottom layer uses a dual-stream 3D-CNN to extract spatial and temporal features for video data, and a bidirectional gated recurrent unit Bi-GRU combined with an attention mechanism to extract semantic features for text data. The middle layer uses a graph convolutional network to construct a learner relationship graph for interaction data and calculate the node influence score. Where N(i) is the set of neighbor nodes of node i, w ij is the edge weight, f j is the neighbor node feature; the high-level layer uses the self-attention mechanism to perform cross-modal enhancement on the bottom-level and middle-level features, through the formula Generate enhanced feature vector;

[0010] Steps for dynamic modeling of learning behavior: Establish a learning behavior generation model based on variational autoencoder VAE, map features to latent space through encoder, and reconstruct learning behavior data from latent space through decoder; introduce time series prediction module, use long short-term memory network-conditional random field LSTM-CRF structure to predict learners' future behavior trends, and use loss function L=L recon +λ·L kl +μ·L pred Perform joint optimization, where L recon is the reconstruction loss, L kl is the KL divergence loss, Lpred is the prediction loss, λ and μ are weight coefficients;

[0011] Learning status assessment steps: Construct a four-dimensional assessment system including cognition, emotion, sociality and environment. In the cognitive dimension, the knowledge mastery is assessed through knowledge graph reasoning; in the emotional dimension, the emotional entropy is calculated by integrating facial expressions, voice intonation and text emotion to assess emotional stability; in the social dimension, the social activity is assessed by analyzing the centrality index of learners in the interactive network; in the environmental dimension, the environmental suitability is assessed by combining environmental data with learning behavior correlation analysis; finally, the weighted fusion formula is used to calculate the emotional entropy value. Get the comprehensive learning status score, where S i Score each dimension, w i is the weight;

[0012] Resource recommendation decision-making steps: Construct a recommendation decision-making framework based on reinforcement learning, using the learning state evaluation results as the environment state, the recommended resource action as the agent behavior, and the learner feedback as the reward signal; using the dual deep Q network DDQN combined with the priority experience replay mechanism, through the formula Update the Q value function, where θ is the network parameter, θ - is the target network parameter, r is the immediate reward, and γ is the discount factor. The resource combination value is calculated by the resource co-evolution model considering the resource ecological relationship, and resource sets with complementary and synergistic effects are recommended first.

[0013] Personalized recommendation presentation steps: Build a personalized recommendation interface based on user profiles, predict learners' attention distribution through eye movement prediction models, and dynamically adjust the display layout and visual prominence of recommended resources. Adopt a progressive recommendation approach, first displaying core recommended resources, and then, based on learners' initial interactive feedback, adding related extended resources in real time to achieve layered recommendations of primary and extended resources.

[0014] Recommendation effect feedback analysis steps: In addition to collecting learners' explicit feedback, we also use tracking technology to collect implicit behavioral data on learners during resource usage. We then use causal analysis models to explore the causal relationship between recommended resources and learning outcomes. We also introduce social network analysis methods to analyze the dissemination paths and influence of recommended resources within the learning community, and evaluate the recommendation effects at the individual, group, and community levels.

[0015] Steps for continuous evolution of the recommendation model: Establish a dynamic model evolution mechanism to trigger local model updates when changes in learner behavior patterns are detected; regularly use federated learning technology to aggregate model parameters on the learning platform for optimization while protecting user privacy; use concept drift detection algorithms to identify changes in data distribution, and use knowledge distillation technology to migrate old model knowledge to the new model to achieve smooth model evolution;

[0016] Adaptive adjustment steps for recommendation strategies: Build an adaptive decision engine, dynamically adjust recommendation strategy parameters through a fuzzy decision tree algorithm based on learning state change trends, recommendation effect evaluation results, and resource ecological changes; introduce an adversarial learning mechanism to simulate learner behavior responses under different recommendation strategies, generate recommendation strategies through strategy game optimization, and achieve dynamic adaptation of recommendation strategies.

[0017] Furthermore, it also includes:

[0018] Learning resource creation steps: Using a model that combines generative adversarial networks and Transformers, input keywords and knowledge structure conditions to generate learning resources in the form of pictures, texts, and videos based on the common needs and individual preferences of the learner group; score the generated resources through a quality assessment model, with scoring indicators including content accuracy, fun, and educational value, and select high-quality generated resources for inclusion in the recommendation library.

[0019] Cross-scenario learning transfer recommendation steps: Analyze the behavioral differences of learners in different learning scenarios to build a scenario mapping model; when learners switch learning scenarios, use transfer learning technology to transfer the learning preferences and behavior patterns in the original scenario to the new scenario, and adjust the recommendation strategy based on the characteristics of the new scenario to achieve cross-scenario recommendations.

[0020] Furthermore, in the data collection and dynamic perception steps, edge computing devices are used to pre-process some data in real time, and the edge node selection algorithm is used to select the edge node. Select the optimal edge node for data processing to reduce data transmission delay, where C speed Calculate the speed for the node, S cap is the storage capacity, D distance is the distance from the learner's device, ω i is the weight coefficient.

[0021] Furthermore, in the data fusion preprocessing step, the local linear embedding (LLE) algorithm is used to reduce the dimension of high-dimensional physiological data. i =∑ j∈N(i) w ij y j Reconstruct the data, where y i is the data point after dimensionality reduction, w ij is the weight, preserving the local manifold structure of the data.

[0022] Furthermore, in the feature layer extraction and enhancement step, the dynamic word vector generation method is used to extract text features, and the formula v is used according to the context. word =GRU([v prev , v context ]) Dynamically update word vectors to improve semantic understanding accuracy, where vprev is the previous word vector, v context is the context vector.

[0023] Furthermore, in the learning behavior dynamic modeling step, the attention flow mechanism AFM is introduced to highlight the learning behavior characteristics by calculating the attention weights of the features at different time steps. The formula is a t =Softmax(W2·tanh(W1·h t +b1)), where h t is the hidden state at time step t, W1 and W2 are weight matrices, and b1 is the bias vector.

[0024] Furthermore, in the learning status assessment step, the social dimension assessment introduces social capital theory, and evaluates the learner's resource acquisition and influence in the learning social network by calculating the learner's structural hole index and trust index. The formula is: Among them S k is the score of each social capital indicator, α k is the weight coefficient.

[0025] Furthermore, in the resource recommendation decision step, the life cycle of the resource is considered and the timeliness value V of the resource is calculated through the resource aging model. age =V0·e -λ·t Dynamically adjust the recommendation priority, where V0 is the initial value of the resource, λ is the aging coefficient, and t is the resource release time.

[0026] Furthermore, in the step of adaptive adjustment of recommendation strategy, genetic algorithm is used to optimize the recommendation strategy parameters, and the fitness function Evaluate the quality of the strategy and generate a better strategy through selection, crossover, and mutation operations, where R i is the score of different evaluation indicators, β i is the weight coefficient.

[0027] Compared with the existing technology, the beneficial effects of the present invention are:

[0028] In terms of data acquisition and processing, a heterogeneous sensor array collects multimodal data and dynamically adjusts the acquisition frequency based on the learner's state and environment, ensuring that the acquired data is comprehensive, accurate, and real-time. Advanced data preprocessing and deep fusion technologies are used to effectively remove noise, achieve precise alignment of multimodal data, and enhance features, laying a solid foundation for subsequent analysis.

[0029] In terms of learning behavior analysis and status assessment, the constructed hierarchical feature extraction network and dynamic learning behavior model can deeply explore learners' behavioral patterns and potential needs. Combined with a multi-dimensional assessment system, it can comprehensively and accurately assess learners' learning status, focusing not only on the degree of knowledge mastery, but also fully considering emotional, social and other factors.

[0030] In the resource recommendation phase, a reinforcement learning-based intelligent recommendation decision-making framework and adaptive adjustment strategy enable the recommendation system to dynamically optimize recommendation strategies based on learner status and recommendation effectiveness. Combined with intelligent resource creation and cross-scenario migration recommendations, it provides precise and personalized resource recommendation services. Whether recommending new resources or adjusting recommendations based on learner scenario changes, it can meet learners' needs.

[0031] This method can effectively improve the accuracy and effectiveness of online learning resource recommendations, enhance learners' learning interest and enthusiasm, enhance learning effects and user experience, and also help improve the resource utilization and competitiveness of online learning platforms, providing strong support for the development of online education. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 This is a schematic diagram of the online resource adaptive recommendation method based on multimodal learning behavior analysis proposed by the present invention;

[0033] Figure 2 This is a schematic diagram of the dynamic adjustment and comparison of data collection frequency for online resource adaptive recommendation based on multimodal learning behavior analysis proposed by the present invention;

[0034] Figure 3 This is a schematic diagram comparing scores in various dimensions of learning status assessment for the online resource adaptive recommendation method based on multimodal learning behavior analysis proposed in the present invention;

[0035] Figure 4 This is a schematic diagram showing the time-varying trend of the accuracy of adaptive online resource recommendations based on the multimodal learning behavior analysis proposed in the present invention. DETAILED DESCRIPTION

[0036] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0037] In the description of the present invention, it should be understood that the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", "clockwise", "counterclockwise" and the like to indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as limiting the present invention.

[0038] In addition, the terms "first" and "second" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the said features. In the description of the present invention, the meaning of "multiple" is two or more, unless otherwise clearly and specifically defined. In addition, the terms "installed", "connected" and "connected" should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be a connection between the two elements. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances. The present invention will be further described in detail below with reference to the accompanying drawings.

[0039] Reference Figure 1 and Figure 4 :A specific implementation method of an online resource adaptive recommendation method based on multimodal learning behavior analysis

[0040] In actual application scenarios, this online resource adaptive recommendation method based on multimodal learning behavior analysis will be run in collaboration between the server side of the online learning platform and the user terminal. The specific implementation process is as follows:

[0041] Multimodal data collection and dynamic perception steps: At the user terminal device level, deploy corresponding sensors and data collection programs for different types of devices. On the smartphone side, call the system API to collect accelerometer data (sampling frequency is 50Hz) and gyroscope data (sampling frequency is 100Hz) to monitor the learner's body movement; use the mobile phone camera (resolution is 1920×1080, frame rate is 30fps) to collect facial expression data, and use the depth sensor of the front camera to obtain the three-dimensional position information of the eye, combined with a dedicated SDK to achieve eye gaze tracking (accuracy can reach 1° viewing angle). On the PC side, an external professional eye tracker (sampling frequency is 120Hz) accurately records the gaze trajectory; collect voice data through the microphone (sampling rate is 44.1kHz, 16-bit quantization); at the same time, deploy temperature and humidity sensors, light intensity sensors, and noise decibel sensors in the learning environment to collect environmental data in real time (collection interval is 1 minute).

[0042] In terms of dynamic perception, the edge computing node obtains device performance data every 5 seconds, including CPU usage, memory usage, network bandwidth and other information. total When a device is first connected to the platform, a benchmark test is run to determine the comprehensive score covering multiple dimensions such as computing power, storage capacity, and network transmission capacity. load The network connectivity score is calculated by real-time monitoring of the CPU, memory, disk I / O, and network I / O usage, using a weighted summation method. The weights are dynamically adjusted according to different learning scenarios. For example, in a video learning scenario, the network I / O weight is set to 0.4 and the CPU weight is set to 0.3; in a text reading scenario, the memory weight is set to 0.4 and the disk I / O weight is set to 0.3. comp Based on the three indicators of network delay, packet loss rate, and throughput, a score of 0-10 is obtained through fuzzy comprehensive evaluation method. The adjustment coefficients α and β are dynamically adjusted according to the learning scenario and learner profile. For novice learners, α = 0.7 and β = 0.8 in the theoretical learning scenario; for advanced learners, α = 0.85 and β = 0.75 in the practical operation scenario. For example, when a learner is doing online programming practice, if the device CPU usage reaches 80% and the network delay exceeds 200ms, according to the formula Calculate the acquisition frequency f rate , increasing the frequency of collecting physiological data from once per minute to five times per minute, ensuring the integrity and real-time nature of key data.

[0043] Preprocessing steps for deep fusion of multimodal data: After the original multimodal data is transmitted to the cloud server through an encrypted channel, it first enters the Transformer-based denoising autoencoder for cleaning. Taking video viewing behavior data as an example, if there is an abnormal record of continuous fast-forwarding exceeding 50% of the video length without pause, the autoencoder uses a multi-head attention mechanism to analyze the playback records, timestamps, and other modal data before and after the operation (such as facial expressions and mouse operations during the same period) to determine whether it is a valid operation. If the analysis finds that the learner's facial expression during this period shows confusion, and the mouse frequently clicks on other interfaces, the fast-forward operation is deemed invalid data and is eliminated.

[0044] In the normalization process, for numerical data such as learning time and reading speed, we first calculate its probability density distribution in the entire data set \(p(x)\). Taking the learning time data of an online course as an example, we construct a probability density function using the kernel density estimation method. For the learning time x of a single learner, according to the formula It is mapped to the interval [0, 1]. For non-numeric data, such as facial expressions (happy, sad, focused, etc.), an improved one-hot encoding method is used, introducing semantic similarity weights. Taking the expressions "focused" and "serious" as examples, the pre-trained sentiment analysis model calculates a semantic similarity of 0.8 between the two. During encoding, the weights of the corresponding positions are set to 1 and 0.8, respectively, to better reflect the semantic differences in the encoding.

[0045] Spatiotemporal alignment uses learning events as anchors, such as course chapter switches, video playback keyframes, and text paragraph jumps. When the system detects a learner switching from video learning to text reading, it records the switch timestamp t0. Using dynamic time warping technology, the system precisely matches the last 5 seconds of video playback and the first 5 seconds of text reading on the timeline. By minimizing the distance between the time series, the time deviation between the two modal data is controlled within ±0.3 seconds, ensuring data consistency and coherence.

[0046] Feature layer extraction and enhancement steps: In the underlying feature extraction stage, a two-stream 3D-CNN network structure is used for video data. The spatial stream network receives video frame images and extracts spatial features from each frame through multiple convolutional layers (convolution kernel size is 3×3×3) and pooling layers (pooling kernel size is 2×2×2). These features include the layout of interface elements, the location of operation buttons, and character gestures. The temporal stream network receives a continuous sequence of video frames and uses 3D convolution operations to capture the temporal characteristics of action changes, such as the temporal patterns of operations such as fast forwarding, pausing, and dragging the progress bar. For text data, a bidirectional gated recurrent unit (Bi-GRU) combined with an attention mechanism is used. The Bi-GRU's hidden layer dimension is set to 256. Two forward and backward GRU units are used to extract features from the beginning and end of the text, respectively. The attention mechanism then focuses on key content such as key paragraphs, keywords, and code comments, highlighting text features that are important for learners to understand and master knowledge.

[0047] The middle-level feature extraction uses a graph convolutional network (GNN) to construct a learner interaction graph. Taking the question-answer interaction in an online learning community as an example, each learner is regarded as a node in the graph. The node features include the learner's basic information, historical learning records, community activity, etc. The edges between nodes represent the interactive relationships between learners, such as questions, replies, likes, and collections. The weight of the edge is calculated based on factors such as the frequency, quality, and time of the interaction. Through the formula Calculate the node influence score, where N(i) is the set of neighbor nodes of node i, w ij is the weight of the edge between node i and node j, f j is the eigenvector of neighbor node j. Through this calculation, the core knowledge users and influence propagation paths in the community are discovered, providing a basis for subsequent recommendation and social interaction analysis.

[0048] High-level feature fusion uses a self-attention mechanism to integrate the different modal features extracted from the bottom and middle layers. By calculating the attention weights between different modal features, it highlights the more critical features in learning behavior analysis and learning status assessment. For example, in a programming learning scenario, the self-attention mechanism will assign higher weights to features related to code execution results and error debugging based on the current learning task and goal, considering the video features of code writing operations, the code text features, and the learner's interaction features in the community about programming problems. This generates an enhanced comprehensive feature vector for subsequent learning behavior modeling and status assessment.

[0049] Dynamic modeling steps for learning behavior: A learning behavior model is constructed based on a variational autoencoder (VAE). The encoder part consists of a multi-layer fully connected neural network, which compresses multimodal features into a 128-dimensional latent space vector. The decoder also uses a fully connected neural network and attempts to reconstruct the original learning behavior data from the latent space vector. In order to make the distribution of the latent space conform to the standard normal distribution, the KL divergence loss term is introduced. At the same time, the long short-term memory network-conditional random field (LSTM-CRF) module is combined to predict the learner's future learning behavior. Taking the example of predicting whether a learner will debug code during programming learning, the LSTM network receives the learning behavior features of the past seven time steps (including code writing time, number of error prompts, frequency of document consultation, etc.), memorizes and updates this information through a gating mechanism, and the CRF layer predicts the behavior label (debugging or continuing writing) for the next time step based on the LSTM output.

[0050] During the training process, the reconstruction loss L is defined recon is the mean square error between the reconstructed data and the original data, and the KL divergence loss L kl Measure the difference between the latent space distribution and the standard normal distribution, predict the loss L pred Cross entropy loss is used. The weight coefficients λ = 0.1, μ = 0.2, and the parameters are updated through the Adam optimizer. The learning rate is set to 0.001 and iterate 100 rounds to make the total loss function L = L recon +λ·L kl +μ·L pred Convergence. During training, the model's performance on the test set is regularly verified. When the validation loss stops decreasing after five consecutive rounds, training is stopped and the optimal model parameters are saved.

[0051] Multi-dimensional evaluation steps of learning status: The cognitive dimension is evaluated by constructing a knowledge graph. Taking programming language learning as an example, the knowledge graph covers knowledge points such as variable declaration, function definition, class inheritance, algorithm implementation, and the association relationship between knowledge points (such as prior knowledge, application scenarios, similar concepts, etc.). By analyzing the learners' mastery of each knowledge point in learning activities such as programming homework, online tests, and code submission (number of correct uses, error types, review frequency, etc.), the graph reasoning algorithm is used to calculate the learners' node scores on the knowledge graph, and then evaluate their knowledge mastery. For example, if a learner frequently encounters parameter type mismatch errors in the function call knowledge point, the score of the knowledge point will be lowered, and based on the association relationship of the knowledge graph, the subsequent knowledge points that may be affected will be analyzed to provide a basis for personalized learning path planning.

[0052] The emotional dimension is assessed by integrating facial expression recognition, speech intonation analysis, and text sentiment classification. Facial expression recognition uses a pre-trained convolutional neural network model to classify facial expressions (such as happiness, sadness, anger, concentration, and confusion) from facial images captured by the camera. Speech intonation analysis extracts features such as the fundamental frequency, energy, and speaking rate of the speech signal and uses a support vector machine model to determine emotional tendencies. Text sentiment classification uses a pre-trained Transformer-based model (such as BERT) to perform sentiment analysis on learners' comments, questions, code annotations, and other text. The sentiment analysis results of the three modalities are weighted and fused, and the weights are dynamically adjusted according to different learning scenarios. In the video explanation scenario, the weight of facial expression is set to 0.4, the weight of speech intonation is set to 0.3, and the weight of text sentiment is set to 0.3. In the programming practice scenario, the weight of text sentiment is set to 0.4, the weight of facial expression is set to 0.3, and the weight of speech intonation is set to 0.3. Ultimately, the learner's emotional state score and emotional entropy value are obtained to assess emotional stability.

[0053] The social dimension is evaluated by calculating the learner's structural hole index, trust index, interactive activity index, etc. in the learning community. The structural hole index calculates the learner's ability to act as an information intermediary by analyzing the learner's position in the social network. For example, if the learner is connected to multiple different learning groups, and there are few connections between these groups, then the learner's structural hole index is higher; the trust index is calculated based on the learner's historical interaction records (response quality, commitment fulfillment, whether effective help is provided, etc.); the interactive activity index counts the number of questions, replies, likes, and shares asked by the learner within a certain period of time. Through weighted summation, the formula is: Among them S k is the score of each social capital indicator, α k is the weight coefficient, and the comprehensive score of the learner in the social dimension is obtained to evaluate his / her resource acquisition and influence in the learning social network.

[0054] The environmental dimension analyzes the relationship between environmental data and learning behavior. Through statistical analysis methods, the correlation between light intensity and learning concentration is studied. For example, it is found that when the light intensity is 300-500 lux, the average concentration time of learners is the longest and the error rate is the lowest; the relationship between noise decibels and learning efficiency is that when the noise exceeds 60 decibels, the learner's code writing speed decreases by 15% and the error rate increases by 20%. Based on these correlations, environmental optimization suggestions are provided to learners, and environmental factors are taken into account when recommending resources. For example, in a noisy environment, video courses with subtitles or learning materials with both pictures and texts are recommended. Finally, through the weighted fusion formula Get the comprehensive learning status score, where S i Score each dimension, w iAs weight coefficients, cognitive dimension w1 = 0.4, emotional dimension w2 = 0.2, social dimension w3 = 0.2, and environmental dimension w4 = 0.2, the learners' learning status is divided into four levels: efficient learning, normal learning, need for attention, and learning difficulty.

[0055] Intelligent Resource Recommendation Decision-Making Steps: The reinforcement learning-based recommendation decision-making framework uses the learner's learning state (including cognitive, emotional, social, and environmental assessments) as the environment state and recommends different types of learning resources (such as basic tutorials, advanced courses, practical projects, learning tools, and community activities) as the agent's action. Training is performed using the Dual Deep Q-Network (DDQN) algorithm. The experience replay pool stores 100,000 experience data points. A prioritized experience replay mechanism prioritizes experience samples based on temporal difference error (TD-error), prioritizing high-value samples for replay to improve learning efficiency. A discount factor γ = 0.9 is used, and the target network parameters are updated every 1,000 steps to prevent the algorithm from falling into local optima.

[0056] In the resource recommendation process, the ecological relationship of resources is considered and the resource combination value is calculated through the resource co-evolution model. Taking online programming learning as an example, the data storage relationship between Python crawler courses and database operation courses is analyzed. The value of the course combination is evaluated by calculating indicators such as the similarity of course content, the complementarity of knowledge points, and the common needs of learners. At the same time, according to the resource aging model V age =V0·e -λ·t , where V0 is the initial value of the resource (determined based on the resource's quality score, user reviews, expert recommendations, etc.), λ is the aging coefficient (λ = 0.001 for technical resources and λ = 0.0005 for theoretical resources), and t is the resource release time. The recommendation priority of resources is dynamically adjusted. For old tutorials published more than three months ago, their recommendation priority is reduced by 30%. Newly released resources with content more relevant to current technological developments are preferred.

[0057] Personalized recommendation presentation steps: Build a personalized recommendation interface based on user portraits. The user portraits include information on multiple dimensions, such as the learner's basic information (age, gender, learning foundation), learning preferences (favorite subjects, learning methods, resource types), historical learning records (courses learned, learning time, performance), and social relationships (learning groups, followed users, and people with whom they frequently interact). Use an eye movement prediction model to analyze the learner's attention distribution on the learning interface. The model learns the degree to which different interface elements attract learners' attention by training a large amount of eye movement data and interface layout features. For example, for a programming learning interface, elements such as the code editing area, the run button, and the error prompt window have a higher attention weight. Based on the attention prediction results, important resources such as recommended popular open source project code examples and related learning tools are placed in the center of the screen or in high-attention distribution areas to improve the visibility and click-through rate of the resources.

[0058] A progressive recommendation approach is used, first displaying core recommended resources, such as a basic Python grammar course for beginners. Once a learner clicks to view, the app analyzes their learning behavior (such as viewing progress, pause counts, and note-taking). If the learner repeatedly views loop statements, additional resources such as practical examples and common error resolution are added to the bottom of the page. This creates a layered recommendation system of "primary resources + additional resources" to meet the learning needs of learners at different stages.

[0059] Multi-source feedback analysis steps for recommendation effectiveness: A large number of tracking points are deployed on the learning platform's front-end pages and back-end services to record learners' explicit feedback on recommended resources, including whether they click to view the recommended resource, viewing / reading time, whether they bookmark or share the resource, resource rating (1-5 points), and comment content. At the same time, by monitoring learners' implicit feedback data, such as changes in learning performance (such as programming homework scores and online test scores), improvements in learning efficiency (such as coding speed and problem-solving time), and changes in learning interests (such as exploration of new fields and depth of resource browsing), we can comprehensively evaluate the effectiveness of recommendations.

[0060] Causal analysis models, using propensity score matching and difference-in-difference methods, analyze the causal relationship between recommended resources and learning outcomes. For example, to evaluate the impact of a recommended algorithm optimization course on learners' programming skills, learners who took the recommended course served as an experimental group, while a control group of learners who did not take the course but had similar characteristics served as a comparison. By comparing the changes in programming skills between the two groups over time, the actual effectiveness of the course was determined. Social network analysis was used to track the dissemination and influence of recommended resources within the learning community, analyzing how the resources spread through sharing and discussion among learners, assessing the social value and dissemination of the resources, and comprehensively evaluating the recommendation effect from multiple levels: individual, group, and community.

[0061] Continuous evolution of the recommendation model: The recommendation model is deployed on a distributed server cluster, and is set to perform pattern detection on learners' behavior data every hour. Learners' learning behavior data is clustered using a clustering algorithm (such as DBSCAN). If a certain type of learning behavior pattern (such as the distribution of error types or learning time patterns in a specific programming language) is found to have changed significantly, and the degree of change exceeds a preset threshold (such as the offset of the cluster center being greater than 0.5), a local update of the model is triggered. Local updates use an incremental learning algorithm to adjust only the affected model parameters, avoiding the waste of computing resources and time consumption caused by retraining the entire model.

[0062] Federated learning is conducted once a month, and models are jointly trained with multiple cooperating online learning platforms under the premise of protecting user privacy. Each platform retains user data locally and only uploads the updated amount of model parameters. The model parameters of multiple platforms are aggregated through a secure aggregation protocol (such as the federated averaging algorithm) to update the global model. At the same time, concept drift detection algorithms (such as the DDM algorithm) are used to check changes in data distribution every 30 minutes. When new learning trends are detected (such as a sudden increase in learning demand for an emerging technology) or significant changes in data patterns, useful knowledge in the old model is migrated to the new model through knowledge distillation technology, so that the new model can quickly adapt to changes, achieve smooth evolution of the model, and maintain the effectiveness and advancement of the recommendation system.

[0063] Adaptive adjustment steps for recommendation strategies: Build an adaptive decision engine to monitor changes in learners' learning status, recommendation effectiveness evaluation indicators (such as recommendation accuracy, resource click-through rate, and learning effect improvement rate), and dynamic changes in the resource ecosystem (such as the release of new resources, updates to existing resources, and changes in resource popularity) in real time. A fuzzy decision tree algorithm is used to process uncertain and imprecise data. Factors such as the learner's programming project completion progress (fast, medium, slow), code quality score (high, medium, low), and learning status change trend (increasing, stable, declining) are used as input variables and mapped to fuzzy sets through fuzzification.

[0064] The above are only preferred specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any technician familiar with this technical field, within the technical scope disclosed by the present invention, who makes equivalent replacements or changes based on the technical solutions and inventive concepts of the present invention, should be covered by the scope of protection of the present invention.

Claims

1. A method for adaptively recommending online resources based on multimodal learning behavior analysis, characterized in that: include: Data collection and dynamic perception steps: A heterogeneous sensor array is used to collaboratively collect learners' data. In addition to video viewing, text reading, interaction and physiological data, wearable devices are used to collect skin electrical response and body acceleration data. Environmental sensors are used to obtain the light intensity and noise decibel environment data of the learning space to build a dynamic perception model. According to the learner's learning stage, device performance, and network status, the dynamic perception model is constructed through the formula Dynamically adjust the data collection frequency, where f rate is the acquisition frequency, S load is the current load of the device, S total is the total load capacity of the equipment, α and β are adjustment coefficients, C comp Score network connectivity; Data fusion preprocessing steps: First, use the Transformer-based denoising autoencoder to denoise the original data, and use the attention mechanism to capture data dependencies and remove noise; In the normalization stage, a normalization method based on probability distribution matching is used. For data x, the formula Mapping to the target probability distribution, where p(x) is the probability density of data x, using a spatiotemporal alignment algorithm to learn the event trigger time as an anchor point, combined with dynamic time warping technology to solve the time offset problem of different modal data; Feature layer extraction and enhancement steps: Build a layered feature extraction network. The bottom layer uses a two-stream 3D-CNN to extract spatial and temporal features for video data, and a bidirectional gated recurrent unit (Bi-GRU) combined with an attention mechanism to extract semantic features for text data. The middle layer uses graph convolutional networks to construct learner relationship graphs based on interaction data and calculate node influence scores. Where N(i) is the set of neighbor nodes of node i, w ij is the edge weight, f j is the neighbor node feature; The high-level layer uses the self-attention mechanism to perform cross-modal enhancement on the low-level and middle-level features, and the formula Generate enhanced feature vector.

2. The online resource adaptive recommendation method based on multimodal learning behavior analysis according to claim 1 is characterized in that: Also includes: Steps for dynamic modeling of learning behavior: Establish a learning behavior generation model based on variational autoencoder VAE, map features to latent space through encoder, and reconstruct learning behavior data from latent space through decoder; introduce time series prediction module, use long short-term memory network-conditional random field LSTM-CRF structure to predict learners' future behavior trends, and use loss function L=L recon +λ·L kl +μ·L pred Perform joint optimization, where L recon is the reconstruction loss, L kl is the KL divergence loss, L pred is the prediction loss, λ and μ are weight coefficients; Learning status assessment steps: Construct a four-dimensional assessment system covering cognition, emotion, sociality, and environment. The cognitive dimension assesses knowledge mastery through knowledge graph reasoning; the emotional dimension uses an emotion analysis model that integrates facial expressions, voice intonation, and text emotion to calculate the emotional entropy value to assess emotional stability; the social dimension assesses social activity by analyzing the learner's centrality index in the interaction network; The environmental dimension combines environmental data with learning behavior correlation analysis to evaluate environmental suitability; finally, the weighted fusion formula is used to Get the comprehensive learning status score, where S i Score each dimension, w i is the weight; Resource recommendation decision-making steps: Construct a recommendation decision-making framework based on reinforcement learning, using the learning state evaluation results as the environment state, the recommended resource action as the agent behavior, and the learner feedback as the reward signal; using the dual deep Q network DDQN combined with the priority experience replay mechanism, through the formula Update the Q value function, where θ is the network parameter, θ - is the target network parameter, r is the immediate reward, and γ is the discount factor. The resource combination value is calculated by the resource co-evolution model considering the resource ecological relationship, and resource sets with complementary and synergistic effects are recommended first. Personalized recommendation presentation steps: Build a personalized recommendation interface based on user profiles, predict learners' attention distribution through eye movement prediction models, and dynamically adjust the display layout and visual prominence of recommended resources. Adopt a progressive recommendation approach, first displaying core recommended resources, and then, based on learners' initial interactive feedback, adding related extended resources in real time to achieve layered recommendations of primary and extended resources. Recommendation effect feedback analysis steps: In addition to collecting learners' explicit feedback, we also use tracking technology to collect implicit behavioral data on learners during resource usage. We then use causal analysis models to explore the causal relationship between recommended resources and learning outcomes. We also introduce social network analysis methods to analyze the dissemination paths and influence of recommended resources within the learning community, and evaluate the recommendation effects at the individual, group, and community levels. Steps for continuous evolution of the recommendation model: Establish a dynamic model evolution mechanism to trigger local model updates when changes in learner behavior patterns are detected; regularly use federated learning technology to aggregate model parameters on the learning platform for optimization while protecting user privacy; use concept drift detection algorithms to identify changes in data distribution, and use knowledge distillation technology to migrate old model knowledge to the new model to achieve smooth model evolution; Adaptive adjustment steps for recommendation strategies: Build an adaptive decision engine to dynamically adjust recommendation strategy parameters using a fuzzy decision tree algorithm based on learning state change trends, recommendation effect evaluation results, and resource ecosystem changes. Introduce an adversarial learning mechanism to simulate learner behavior responses under different recommendation strategies, and generate recommendation strategies through strategic game optimization to achieve dynamic adaptation of recommendation strategies. Learning resource creation steps: Using a model that combines generative adversarial networks and Transformers, input keywords and knowledge structure conditions to generate learning resources in the form of pictures, texts, and videos based on the common needs and individual preferences of the learner group; score the generated resources through a quality assessment model, with scoring indicators including content accuracy, fun, and educational value, and select high-quality generated resources for inclusion in the recommendation library.

3. The online resource adaptive recommendation method based on multimodal learning behavior analysis according to claim 1 is characterized in that: Also includes: Recommended steps for cross-scenario learning transfer: Analyze learners' behavioral differences in different learning scenarios and build a scenario mapping model; When learners switch learning scenarios, the learning preferences and behavior patterns in the original scenario are transferred to the new scenario through transfer learning technology, and the recommendation strategy is adjusted according to the characteristics of the new scenario to achieve cross-scenario recommendations.

4. The online resource adaptive recommendation method based on multimodal learning behavior analysis according to claim 1, characterized in that: In the data collection and dynamic perception steps, edge computing devices are used to pre-process some data in real time, and the edge node selection algorithm is used to select the edge node. Select the optimal edge node for data processing to reduce data transmission delay, where C speed Calculate the speed for the node, S cap is the storage capacity, D distance is the distance from the learner's device, ω i is the weight coefficient.

5. The online resource adaptive recommendation method based on multimodal learning behavior analysis according to claim 1 is characterized in that: In the data fusion preprocessing step, the local linear embedding (LLE) algorithm is used to reduce the dimension of high-dimensional physiological data. i =∑j∈N(i)w ij y j Reconstruct the data, where y i is the data point after dimensionality reduction, w ij is the weight, preserving the local manifold structure of the data.

6. The online resource adaptive recommendation method based on multimodal learning behavior analysis according to claim 1, characterized in that: In the feature layer extraction and enhancement step, the dynamic word vector generation method is used to extract text features, and the formula v is used according to the context. word =GRU([v prev , v context ]) Dynamically update word vectors to improve semantic understanding accuracy, where v prev is the previous word vector, v context is the context vector.

7. The online resource adaptive recommendation method based on multimodal learning behavior analysis according to claim 2, characterized in that: In the learning behavior dynamic modeling step, the attention flow mechanism AFM is introduced to highlight the learning behavior characteristics by calculating the attention weights of the features at different time steps. The formula is a t =Softmax(W2·tanh(W1·h t +b1)), where h t is the hidden state at time step t, W1 and W2 are weight matrices, and b1 is the bias vector.

8. The online resource adaptive recommendation method based on multimodal learning behavior analysis according to claim 2, characterized in that: In the learning status assessment step, the social dimension assessment introduces social capital theory. By calculating the learner's structural hole index and trust index, the learner's resource acquisition and influence in the learning social network are evaluated. The formula is: Among them S k is the score of each social capital indicator, α k is the weight coefficient.

9. The online resource adaptive recommendation method based on multimodal learning behavior analysis according to claim 2, characterized in that: In the resource recommendation decision step, the life cycle of the resource is considered and the timeliness value V of the resource is calculated through the resource aging model. age =V0·e -λ·t Dynamically adjust the recommendation priority, where V0 is the initial value of the resource, λ is the aging coefficient, and t is the resource release time.

10. The online resource adaptive recommendation method based on multimodal learning behavior analysis according to claim 2, characterized in that: In the step of adaptive adjustment of recommendation strategy, genetic algorithm is used to optimize the recommendation strategy parameters, and the fitness function Evaluate the quality of the strategy and generate a better strategy through selection, crossover, and mutation operations, where R i is the score of different evaluation indicators, β i is the weight coefficient.

Images