An online multi-modal resource hybrid recommendation method fusing classroom behavior sequences

Abstract

The application discloses an online multi-modal resource mixed recommendation method fusing a classroom behavior sequence, and comprises the following steps: step 1, frame images are collected from a video, and classroom behaviors in the frame images are labeled; step 2, multi-modal knowledge ontology modeling is carried out; step 3, a student classroom behavior is recognized and represented based on a YOLO-V5 network model; step 4, a reconstructed behavior feature sequence is trained through a Baum-welch algorithm to obtain a multi-state GMM-HMM classification model based on a classroom behavior; and step 5, a mixed recommendation model is established by introducing three types of data, namely, classroom behaviors, cognitive levels and learning styles, and a resource recommendation list TOP-N is given in combination with a collaborative filtering algorithm. The application introduces classroom behavior data to enrich learner representation information, so that more accurate recommendation of online course resources is realized.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of online education, and particularly relates to an online multi-modal resource hybrid recommendation method fusing a classroom behavior sequence. BACKGROUND

[0002] With the deep application of information technology in the field of education, the rapid growth of the total amount of online course resources and the continuous acceleration of their update speed not only make it difficult to meet the in-depth learning needs of learners, but also still face problems such as a continuously low completion rate, repeated development of course resources, difficulty in cross-platform resource retrieval and fusion, and low online learning efficiency. How to mine and recommend multi-modal learning resources that best meet the in-depth learning needs of learners from massive online course resources, and then promote the in-depth learning of learners by providing accurate personalized learning services, has become a key problem to be solved in current education informatization.

[0003] However, although various intelligent technology terminals expand the learning space-time of students, the classroom environment is still the main position for students to conduct formal learning. Under the classroom environment, the performance of individual behavior of students has a direct impact on the learning effect. If students fail to concentrate on listening and learning some knowledge points in the classroom, more education resources related to the knowledge points will be needed in the subsequent online education process. However, the current recommendation of online education resources can only be selected and recommended according to the explicit requirements provided by human beings, cannot be combined with the classroom monitoring system, and cannot be coded and analyzed according to the learning behavior in the classroom environment, so that the system cannot determine the recommendation strategy according to the performance of individual behavior of students in the classroom environment, resulting in that the effective integration and utilization of online course resources cannot be realized for specific students. SUMMARY

[0004] The purpose of the application is to provide an online multi-modal resource hybrid recommendation method fusing a classroom behavior sequence to solve the technical problem in the prior art that the learning behavior of students in the classroom environment cannot be coded and analyzed to construct a learning behavior sequence, so that online course resources that meet the learning needs of learners cannot be accurately recommended, and the effective integration and utilization of online course resources cannot be realized for specific students, resulting in that the recommendation of online education resources lacks accuracy and timeliness.

[0005] The online multi-modal resource hybrid recommendation method fusing a classroom behavior sequence comprises the following steps:

[0006] Step 1: frame images are collected from a video, and classroom behaviors in the frame images are labeled;

[0007] Step 2: multi-modal knowledge ontology modeling is performed;

[0008] Step 3: Based on the YOLO-V5 network model, the student classroom behavior is recognized and represented modeling;

[0009] Step 4: The reconstructed behavior feature sequence is trained by the Baum-welch algorithm to obtain a multi-state GMM-HMM classification model based on classroom behavior;

[0010] Step 5: Introducing classroom behavior, cognitive level and learning style three kinds of data to establish a hybrid recommendation model, and combining with the collaborative filtering algorithm, a resource recommendation list TOP-N is given.

[0011] Preferably, in step 2, for a certain heterogeneous multi-source course resource, the corresponding multi-modal knowledge point j set can be represented as K j =[k j1 ,k j2 ,…,k jn ] where k j is the same knowledge point j of different sources of resources, and the knowledge ontology KOM specifically contains the following information:

[0012] KOM=(K_id,K_name,K_content,K_style,K_level,K_OR),

[0013] K_style∈{t,p,a,v},

[0014] K_OR i，j =(K_id i ,K_id j ,r_style,r_weight),

[0015] Wherein, K_id represents the unique identifier of the knowledge object, K_name represents the knowledge point name, K_content represents the knowledge point specific content, K_style represents the style type of the knowledge point; t represents the text format resource, p represents the picture format resource, a represents the audio format resource, v represents the video format resource, K_level represents the difficulty of the knowledge point, the larger the value means the greater the difficulty; K_OR represents the relationship set type to which the knowledge point belongs, K_id i , K_id j represent two knowledge point object identifiers, r_style represents the relationship type between two knowledge point object identifiers, mainly including dependency, parent-child, parallel, selection, expansion five kinds of relationships; r_weight represents the weight corresponding to the relationship, r_weight takes value in [0, 1].

[0016] Preferably, in step 3, the recorded classroom video is used as a basic unit to automatically extract the classroom behavior sequence of each student according to the timeline, and form a behavior feature sequence reconstructed based on knowledge points; as shown in the following formula:

[0017] Stu i ={(k1,O(b)1),(k2,O(b)2),...,(k n , O(b) n )}, n∈(1,2,...,N),

[0018] S n =(k n , O(b) n ),

[0019] Among them Stu i This represents the sequence of classroom behavior state values ​​corresponding to each student's knowledge point, where i is the unique identifier of the individual student, and k... n Let n represent a knowledge point, where n is the unique identifier corresponding to the knowledge point, and the number of knowledge points is N. O(b) represents the state value of the classroom behavior sequence when a student learns knowledge point n, and b represents the classroom behavior sequence. n , O(b) n ) represents the set of relationships between the reconstructed classroom behavior data and knowledge points.

[0020] Preferably, in step 3, without considering the weight coefficients of various behaviors, the classroom behavior sequence is reconstructed. The reconstructed feature data takes values ​​between [0, 1]. Classroom behaviors include positive and negative behaviors, with a state value of 1 for positive behaviors and a state value of 0 for negative behaviors. For consecutive negative behaviors, a penalty coefficient is introduced, with data values ​​between [0, 1]. The specific formula for the behavior sequence within a single knowledge point is as follows:

[0021] b n ={(b η ,t1),(b η ,t2),...,(b η , t T )},η∈[1,2,3,4,5,6,7]

[0022] Among them, b n This is a sequence of behaviors within a single knowledge point, representing classroom behavior at knowledge point n, where η represents the specific behavior identifier, and T represents the time length based on the knowledge point as the basic unit. Based on the above sequence of behaviors within a single knowledge point, student behavior sequences can be reconstructed according to knowledge points as basic units to determine the set of relationships between classroom behavior data and knowledge points. The specific formula is as follows:

[0023] Sn = (k n , O(b) n ),

[0024]

[0025]

[0026] wherein, (k n , O(b) n ) is the reconstructed behavior data and the relationship set of knowledge points, wherein b n (value) is the classroom behavior when the knowledge point n; the formula also defines the penalty coefficient w, Neg_b represents the number of continuous negative behaviors in the sequence, count represents the count, max represents the maximum value from the count result, and w takes a value between [0, 1].

[0027] Preferably, the step 4 specifically comprises the following steps:

[0028] Step 4.1: training the GMM-HMM model based on the Baum-welch algorithm, first constructing a multi-state model λ based on GMM-HMM;

[0029] Step 4.2: initializing π, A and B;

[0030] Step 4.3: training parameters π and A, taking the difference between adjacent two times in iteration satisfying certain requirements as a condition to obtain updated π and A, and parameter B is estimated by the Gaussian distribution parameter formula through the knowledge point mastery state;

[0031] Step 4.4: using the Viterbi method to decode the real-time collected behavior sequence data to obtain the observation state sequence O n , so as to judge the state sequence S n corresponding to the knowledge point mastery degree;

[0032] Step 4.5: calculating parameters μ(S n ), σ(S n ) and S final ;

[0033] Step 4.6: setting the recommended item adjustment coefficient to determine the number of recommended knowledge point resources in the recommended items.

[0034] Preferably, the formula of the multi-model λ in the step 4.1 is as follows:

[0035] λ = (π, A, B, μ(S n ), σ(S n ), S final ),

[0036] where λ is a GMM-HMM based multi-state model, and the right side of the equation contains 6 parameters of the model: π is the initial state probability distribution; A is the state transition probability matrix; B is the observation state probability distribution matrix; μ(S n ) is the GMM function mean value, representing the average value of the mastery level of the nth knowledge point; σ(S n ) is the variance in the GMM function; S final is the knowledge point mastery state, which is determined according to the behavior sequence, S n represents the state sequence of the knowledge point mastery level.

[0037] In step 4.2, π, A and B are initialized, the observation sequence O(b) is clustered into M classes using a clustering method, and each class is numbered according to the Gaussian mixture number (1, 2,..., M). At the same time, the behavior sequence state value distribution probability O n (n = 1, 2,..., N) is obtained according to the knowledge point division.

[0038] In step 4.3, the parameter B is obtained by the Gaussian distribution parameter estimation formula according to the knowledge point mastery state, and the formula is as follows:

[0039]

[0040]

[0041] where b p (O n ) is the specific element value in the parameter B, p is the element index value in B, P is the number of elements in B, M is the number of weighting coefficients, which is the same as the number of clusters, ξ(O n , μ p，m , σ p，m ) is the mixed Gaussian function, C p，m is the weighting coefficient.

[0042] In step 4.5, the corresponding calculation formulas of μ(S n ), σ(S n ) and S final are as follows:

[0043]

[0044]

[0045] S final = S n .

[0046] Preferably, in step 4.6, after the GMM-HMM model training is completed, an effective recommendation strategy is formed according to the different degrees of knowledge point mastery, and the adjustment coefficient of the recommended items in the resource recommendation list TOP-N is set:

[0047]

[0048] Wherein, theta is the adjustment coefficient of the recommended items in TOP-N, the number of recommended items required to recommend the knowledge point resources is calculated according to the adjustment coefficient theta, and the specific formula is Here, N represents the number of knowledge points, Indicates rounding down.

[0049] Preferably, in step 5, the user modeling is combined with the basic cognitive level U_basic and the learning style U_style of the student, and the recommended sequence U_behavior generated by the student behavior is integrated, and the specific formula is as follows:

[0050] User={alphaU_behavior, betaU_basic, gammaU_style}

[0051] Wherein, three-dimensional dynamic weight coefficients alpha, beta and gamma are set, which correspond to the recommended sequence U_behavior, the basic cognitive level U_basic and the learning style U_style respectively; for new users, the beta weight is higher than the other two, and the setting of the maximum weight value is affected by the accuracy and recall rate of the TOP-N recommendation result of the recommendation system, and satisfies alpha+beta+gamma=1, and the weight value interval is [0, 1], and the specific formula is as follows:

[0052]

[0053]

[0054]

[0055] The formula includes the accuracy and recall rate calculation formula, which is used to test the precision and recall rate of the system, wherein N u is the number of recommended lists; T u Then, it represents the number of lists that are actually needed; it also includes the calculation formula of the harmonic mean F1 of the accuracy and recall rate; then, the maximum weight value is corrected according to the above-mentioned precision, recall rate and harmonic mean.

[0056] The application has the following advantages:

[0057] 1、The application realizes the fusion of the behavior data generated by the students in the classroom into the online self-study after class by fusing the behavior data of the students in the classroom, and connects the classroom teaching and after-school learning, and forms an effective closed-loop system.

[0058] 2、The application integrates classroom behavior, cognitive level and learning style to establish a hybrid recommendation model. According to the result, the weights of the three dimensions are optimized, the cold start problem is effectively solved, and the generalization ability and accuracy of the model are improved.

[0059] 3、The application uses classroom student behavior data to perceive the mastery degree of the current knowledge point content, and simultaneously takes into account the student cognitive basis and learning style, selects resources of different styles for recommendation. By clustering student groups with similar learning characteristics, precise and personalized learning services are provided for them.

[0060] 4、The application designs a multi-modal resource model, which can effectively integrate existing online course resources, improve online course completion rate and resource adaptation degree, and promote deep learning. BRIEF DESCRIPTION OF DRAWINGS

[0061] Figure 1 The model diagram of the online multi-modal resource hybrid recommendation method of the application fusing classroom behavior sequence.

[0062] Figure 2 The network structure diagram of the GMM-HMM multi-state model proposed by the application.

[0063] Figure 3 The recognition result diagram of the main classroom behavior in the application. DETAILED DESCRIPTION

[0064] The specific embodiments of the application will be further described in detail below with reference to the drawings, and by describing the embodiments, to help those skilled in the art to have a more complete, accurate and in-depth understanding of the inventive concept and technical solutions of the application.

[0065] As shown in Figures 1-2 The application provides an online multi-modal resource hybrid recommendation method fusing classroom behavior sequence, comprising the following steps.

[0066] Step 1: frame images are collected from the video, and classroom behaviors in the frame images are labeled, including positive behaviors and negative behaviors.

[0067] Relying on the education collection equipment and the recording and broadcasting system, the video is segmented according to the course session, frame images are selected by frame extraction, and the obtained frame images are labeled according to seven main classroom behaviors (mainly including four positive behaviors of writing, raising hands, standing up, and watching the blackboard (serious listening) and three negative behaviors of sleeping, playing mobile phones, and eating).

[0068] Step 2: multi-modal knowledge ontology modeling.

[0069] The structured representation and storage of domain-specific educational resources are constructed using different granularities, such as courses, chapters, sections, and knowledge points. Smaller granularities imply more refined descriptions of attributes such as correlation and hierarchy. For example, for a course resource originating from multiple MOOC courses (representing heterogeneous, multi-source course resources), the set of a certain multimodal knowledge point j can be represented as K. j =[k j1 k j2 , ..., k jn ], where k j For the same knowledge point j from different resource sources, the Knowledge Ontology (KOM) specifically contains the following information:

[0070] KOM=(K_id,K_name,K_content,K_style,K_level,K_OR) (1)

[0071] K_style∈{t,p,a,v} (2)

[0072] K_OR i，j =(K_id i K_id j ,r_style,r_weight) (3)

[0073] In equation (1), K_id represents the unique identifier of the knowledge object, K_name represents the name of the knowledge point, K_content represents the specific content of the knowledge point, and K_style represents the style type of the knowledge point; in equation (2), t represents the text format resource, p represents the image format resource, a represents the audio format resource, and v represents the video format resource. K_level represents the difficulty level of the knowledge point, K_level∈[1,2,3,4,5], and the larger the value, the greater the difficulty. K_OR represents the relation set type to which the knowledge point belongs, where K_id in equation (3) i K_id j This represents two knowledge point objects. `r_style` indicates the relationship type, primarily including five types: dependency, parent-child, parallel, selection, and extension. `r_weight` represents the weight corresponding to the relationship, with values ​​ranging from [0, 1]. See Table 1 for details on the relationship types between knowledge points.

[0074] Table 1: Types of Relationships Between Knowledge Points

[0075]

[0076] Step 3: Model the recognition and representation of student classroom behavior based on the YOLO-V5 network model.

[0077] After the YOLO-V5 network model achieves good training results, the recorded classroom videos are used as basic units based on knowledge points. To ensure consistency in the number of frames extracted per unit time, the classroom behavior sequence of each student is automatically extracted along the timeline, forming a behavioral feature sequence reconstructed based on knowledge points, as shown in the following formula.

[0078] Stu i ={(k1,O(b)1),(k2,O(b)2),...,(k n , O(b) n )}, n∈(1, 2,...,N) (4)

[0079] S n =(k n , O(b) n (5)

[0080] Among them Stu i This represents the sequence of classroom behavior state values ​​corresponding to each student's knowledge point, where i is the unique identifier of the individual student, and k... n Let N represent a knowledge point, where n represents the unique identifier corresponding to the knowledge point. Taking "Object-Oriented Programming" as an example, the number of knowledge points N = 97. O(b) represents the state value of the classroom behavior sequence when a student learns knowledge point n, and b represents the classroom behavior sequence. n , O(b) n ) represents the set of relationships between the reconstructed classroom behavior data and knowledge points.

[0081] Assuming an ideal state, without considering the weighting coefficients of various behaviors, the classroom behavior sequence is reconstructed, and the reconstructed feature data takes values ​​between [0, 1]. Furthermore, considering that continuous negative behaviors have a positive correlation with learning outcomes, a penalty coefficient is introduced here, with data values ​​between [0, 1]. The specific formula for the behavior sequence within a single knowledge point is as follows.

[0082] b n ={(b η ,t1),(b η ,t2),...,(b η , t T )},η∈[1,2,3,4,5,6,7] (6)

[0083] Among them, b n This is a sequence of behaviors within a single knowledge point, representing classroom behaviors at knowledge point n, where η represents the specific behavior identifier, and T represents the time length based on the knowledge point as the basic unit. The definitions and identifiers of the main classroom behavior types are shown in Table 2, and the identification results for each main classroom behavior are shown in the figure below. Figure 3 .

[0084] Table 2: Definition and identification of main classroom behavior types

[0085]

[0086] Based on the above behavior sequence in a single knowledge point, the student behavior sequence can be reconstructed according to the knowledge point as the basic unit, and the relationship set of classroom behavior data and knowledge point is determined, and the specific formula is as follows:

[0087] S n =(k n ,O(b) n ) (7)

[0088]

[0089]

[0090] wherein (k n , O(b) n ) is the relationship set of reconstructed behavior data and knowledge point, wherein b n (value) is the classroom behavior when the knowledge point n. In formula (9), the penalty coefficient w is defined, Neg_b represents the number of consecutive negative behaviors in the sequence, count represents the count, max represents the maximum value from the count result, and w takes a value between [0, 1].

[0091] Step 4: The reconstructed behavior feature sequence is trained by Baum-welch algorithm to obtain a multi-state GMM-HMM classification model based on classroom behavior.

[0092] Step 4.1: Train the GMM-HMM model based on the Baum-welch algorithm, first construct a multi-state model λ based on GMM-HMM, and the formula is as follows:

[0093] λ = (π, A, B, μ(S n ), σ(S n ), S final ). (10)

[0094] wherein λ is a multi-state model based on GMM-HMM, and the right side of the equation contains 6 parameters of the model: π is the initial state probability distribution; A is the state transition probability matrix; B is the observation state probability distribution matrix; μ(S n ) is the GMM function mean, representing the average value of the mastery degree of the nth knowledge point; σ(S n ) is the variance in the GMM function; S final is the knowledge point mastery state, which is determined according to the behavior sequence, and S n represents the state sequence of the knowledge point mastery degree.

[0095] Step 4.2: initialize π, A and B, cluster the observation sequence O(b) into M classes by clustering method, and number each class according to the Gaussian mixture number (1, 2,..., M), and get the behavior sequence state value distribution probability O n (n = 1, 2,..., N), in this embodiment, M = 5 and N = 97 are set.

[0096] Step 4.3: train parameters π and A, and obtain updated π and A when the difference between two adjacent iterations meets certain requirements. Parameter B is obtained by estimating the Gaussian distribution parameter according to the knowledge point mastery state, and the formula is as follows:

[0097]

[0098]

[0099] Wherein, b p (O n ) is a specific element value in parameter B, p is an element index value in B, P is the number of elements in B, M is the number of weighted coefficients, which is the same as the number of classes in clustering, ξ(O n , μ p，m , σ p，m ) is a mixed Gaussian function, C p，m is a weighted coefficient.

[0100] Step 4.4: use Viterbi method to decode the real-time collected behavior sequence data to obtain the observation state sequence O n , so as to determine the state sequence S n of the corresponding knowledge point mastery degree.

[0101] Step 4.5: calculate parameters μ(S n ), σ(S n ) and S final , and the corresponding calculation formula is as follows:

[0102]

[0103]

[0104] S final = S n . (15)

[0105] Step 4.6: set the recommended item adjustment coefficient. After the GMM-HMM model is trained, the observation state sequence O nThus, the state sequence S of the mastery degree of the corresponding knowledge point is determined n An effective recommendation strategy is formed according to different mastery degrees of the knowledge points, and an adjustment coefficient of a recommended item in a resource recommendation list TOP-N is set.

[0106]

[0107] wherein θ is the adjustment coefficient of the recommended item in the TOP-N, the number of recommended items occupied by the resource of the knowledge point to be recommended is calculated according to the adjustment coefficient θ, and the calculation is specifically wherein N represents the number of knowledge points, represents a floor function.

[0108] Step 5: A hybrid recommendation model is established by introducing three types of data of classroom behavior, cognitive level and learning style, and a resource recommendation list TOP-N (a multi-modal resource object set sequence of hybrid recommendation) is given in combination with a collaborative filtering algorithm.

[0109] This step specifically includes: user modeling is performed in combination with the basic cognitive level U_basic and the learning style U_style of the student, and a recommendation sequence U_behavior generated by the student behavior is integrated, and the specific formula is as follows:

[0110] user = {aU_behavior, bU_basic, gU_style} (17)

[0111] wherein three dynamic weight coefficients a, b and g of the three dimensions are set, and correspond to the recommendation sequence U_behavior, the basic cognitive level U_basic and the learning style U_style respectively. The basic cognitive level and the learning style can be obtained from data analysis in a learning record database and a user mode database. For a new user, the weight b is higher than the other two, and after the data of the classroom behavior and the online learning process of the platform of the user are generated, the related values will also change. The setting of the maximum weight value is affected by the accuracy and recall rate of the TOP-N recommendation result of the recommendation system, and satisfies a+b+g=1, and the weight value interval is [0, 1].

[0112]

[0113]

[0114]

[0115] The formulas (18) and (19) are respectively the accuracy and recall rate calculation formulas for testing the precision and recall rate of the system, wherein N u is the number of recommended lists; T uThis indicates the actual number of lists required. Equation (20) represents the harmonic mean F1 of precision and recall. The final weight value is adjusted based on the precision, recall, and harmonic mean mentioned above.

[0116] The resources recommended in the above process originate from the set of resource objects mapped from the educational resource database. Based on user modeling, feature attributes are extracted, and after collaborative filtering, the set of resource objects is selected to obtain the final resource recommendation list.

[0117] The actual effect of this solution will be explained below with reference to specific experiments in this embodiment.

[0118] Data preparation: The data used is the recorded data from the "Object-Oriented Programming" course in the smart classroom.

[0119] (1) The raw data collected consisted of approximately 675 minutes of class videos, mainly for 15 classroom scenarios (45 minutes each) of the pilot course "Object-Oriented Programming". The real classroom scenarios collected in this study were smart classrooms equipped with three cameras, distributed at the front and back of the classroom (one at the front and two at the back). Since the experiment required the identification of students' classroom behavior, the experimental data consisted of frontal information of the students in class, but there was a characteristic of partial occlusion of learning behavior.

[0120] (2) This method processes the acquired videos sequentially as follows: ① Screening and cropping / segmentation. First, the entire long recorded video is screened, and frames are extracted. OpenCVTutorial is used to generate JPG format frame images, resulting in approximately 16,875 frame images. Then, a similarity comparison algorithm based on ImageHash is constructed to further remove highly similar images from the extracted frame data. ② Data annotation. Based on the selected 1,752 images, behavioral annotations are performed according to seven categories. The annotation boxes are rectangular boxes, and some annotated frame images are shown below. Figure 3 As shown, each frame image is labeled with a rectangle indicating the student's spatial location, and the top left corner of the rectangle indicates the student's behavior category. Finally, the exported data label file is saved as a JSON file, consisting of image file information, bounding box coordinates, and corresponding labels.

[0121] Experimental Procedure and Results: The data was then processed according to the methods described in the previous embodiments, and a resource recommendation list was output. Based on the sample data model training, the model achieved an average accuracy of 91% in recognizing seven learning behaviors. Even under interference from irrelevant factors such as student attire, partial occlusion, and classroom background in real classroom scenarios, it still demonstrated good recognition performance. It is suitable for analyzing relatively complex real classroom scenarios and can effectively identify student behavioral sequence characteristics.

[0122] The application is described above by way of example with reference to the accompanying drawings without limitation to the specifically described embodiments. Any modifications of the application concept and the technical solutions described above, which do not depart from the scope of the application, or direct application of the application concept and the technical solutions described above to other fields, shall fall within the scope of the application.

Claims

1. A hybrid online multimodal resource recommendation method integrating classroom behavior sequences, characterized in that: Includes the following steps: Step 1: Select and capture frame images from the video, and label the classroom behaviors in the frame images, including positive and negative behaviors; Step 2: Perform multimodal knowledge ontology modeling; To achieve structured representation and storage of educational resources, forming a knowledge ontology that includes multimodal knowledge points; Step 3: Model student classroom behavior based on the YOLO-V5 network model; In step 3, the classroom recording video is automatically extracted into a sequence of classroom behaviors for each student, using knowledge points as the basic unit, and a sequence of behavioral features is formed based on the knowledge points. As shown in the following formula: ， ， in This represents the state values ​​of the student's classroom behavior sequence, broken down by knowledge point. As a unique identifier for each student, This represents knowledge points, among which This represents a unique identifier corresponding to a knowledge point; the number of knowledge points is [number missing]. , This indicates the knowledge points that students learn. b represents the state value of the classroom behavior sequence at a given time. This is a set of relationships between reconstructed classroom behavior data and knowledge points; Without considering the weighting coefficients of various behaviors, the classroom behavior sequence is reconstructed. The reconstructed feature data takes values ​​between [0,1]. Classroom behaviors include positive and negative behaviors, with a state value of 1 for positive behaviors and a state value of 0 for negative behaviors. For consecutive negative behaviors, a penalty coefficient is introduced, with data values ​​between [0,1]. The specific formula for the behavior sequence within a single knowledge point is as follows: in, It is a sequence of behaviors within a single knowledge point, representing classroom behaviors when knowledge point n is reached, where Indicates specific behavior identifiers, This represents the time length based on a knowledge point as the basic unit. Based on the behavioral sequence within a single knowledge point, the student behavioral sequence is reconstructed according to the knowledge point as the basic unit to determine the set of relationships between classroom behavioral data and knowledge points. The specific formula is as follows: ， ， ， in, This is the set of relationships between the reconstructed behavioral data and knowledge points, where Classroom behavior is defined for knowledge point n; the formula also defines a penalty coefficient. , This indicates the number of consecutive negative behaviors in the sequence. count This represents counting, and `max` represents taking the maximum value between the count result and 1. The value is between [0, 1]. Step 4: Train the reconstructed behavioral feature sequence using the Baum-Welch algorithm to obtain a multi-state GMM-HMM classification model based on classroom behavior; Step 4 includes: setting the adjustment coefficient for the recommendation item; after the GMM-HMM model is trained, the observed state sequence is obtained by decoding the student classroom behavior sequence. This allows us to determine the state sequence of the level of mastery of the corresponding knowledge points. Effective recommendation strategies are developed based on varying levels of knowledge mastery, and adjustment coefficients are set for the recommended items in the TOP-N resource recommendation list. Step 5: Introduce three types of data—classroom behavior, cognitive level, and learning style—to build a hybrid recommendation model, and combine it with a collaborative filtering algorithm to provide a TOP-N list of recommended resources.

2. The online multimodal resource hybrid recommendation method integrating classroom behavior sequences according to claim 1, characterized in that: In step 2, for a specific heterogeneous, multi-source course resource, the corresponding multimodal knowledge points... The set is represented as in The same knowledge point from different sources Knowledge Ontology The specific information included is as follows: ， ， ， in, This serves as a unique identifier for a knowledge object. Indicates the name of the knowledge point. This indicates the specific content of the knowledge point. The style type used to represent knowledge points; Represented as a text-formatted resource. This is represented as an image format resource. Represented as an audio format resource, Represented as a video format resource, This indicates the difficulty level of the knowledge point; the larger the value, the greater the difficulty. This indicates the type of relation set to which the knowledge point belongs. This is represented as two knowledge point object identifiers. The relationship type between two knowledge point objects is mainly divided into five types: dependency, parent-child, parallel, selection, and extension. This represents the weight corresponding to the relation. The value is between [0, 1].

3. The online multimodal resource hybrid recommendation method integrating classroom behavior sequences according to claim 1, characterized in that: Step 4 specifically includes the following steps: Step 4.1: Train the GMM-HMM model based on the Baum-Welch algorithm. First, construct a multi-state model based on GMM-HMM. ; Step 4.2: Initialization , and ; Step 4.3: Training Parameters and The updated value is obtained by taking the condition that the difference between two adjacent iterations meets certain requirements. and ,parameter The mastery status of knowledge points is obtained using the Gaussian distribution parameter estimation formula; Step 4.4: Use the Viterbi method to perform state decoding on the real-time acquired behavioral sequence data to obtain the observed state sequence. This allows us to determine the state sequence of the level of mastery of the corresponding knowledge points. ; Step 4.5: Calculate parameters , and ; Step 4.6: Set the recommendation item adjustment coefficient to determine the number of recommended items that should be the knowledge point resources to be recommended.

4. The online multimodal resource hybrid recommendation method integrating classroom behavior sequences according to claim 3, characterized in that: The multi-model in step 4.1 The specific formula is as follows: ， in For a multi-state model based on GMM-HMM, the right side of the equation contains the model's six parameters: The initial state probability distribution; This is the state transition probability matrix; The probability distribution matrix of the observed state; Let be the mean of the GMM function, representing the th The average level of mastery of each knowledge point; The variance in the GMM function; This represents the mastery status of knowledge points, which is determined based on a sequence of behaviors. A sequence of states representing the degree of mastery of knowledge points.

5. The online multimodal resource hybrid recommendation method integrating classroom behavior sequences according to claim 4, characterized in that: In step 4.2, initialization , and , observation sequence Clustering methods are used to cluster the data into groups. Classify and sort each class according to Gaussian mixture index. Number the data and, based on the knowledge points, obtain the probability distribution of the behavioral sequence state values. ; In step 4.3, the parameters The mastery status of knowledge points is obtained using the Gaussian distribution parameter estimation formula, as follows: ， ， in, It is a parameter The specific element values ​​in the data. for The element index value in the middle, for The number of elements in The number of weighting coefficients is the same as the number of clusters. It is a Gaussian mixture function. These are the weighting coefficients.

6. The online multimodal resource hybrid recommendation method integrating classroom behavior sequences according to claim 5, characterized in that: In step 4.5, with , and The corresponding calculation formula is as follows: ， ， 。 7. The online multimodal resource hybrid recommendation method integrating classroom behavior sequences according to claim 6, characterized in that: In step 4.6, after the GMM-HMM model is trained, an effective recommendation strategy is formed based on the different levels of knowledge mastery, and the adjustment coefficients of the recommended items in the TOP-N resource recommendation list are set: = Among them The adjustment factor for the recommended items in TOP-N, based on the adjustment factor Calculate the number of recommended items that require recommending resources for this knowledge point, specifically: Here, N represents the number of knowledge points. This indicates rounding down to the nearest integer.

8. The online multimodal resource hybrid recommendation method integrating classroom behavior sequences according to claim 1, characterized in that: In step 5, the students' basic cognitive level is taken into account. and learning style Recommendation sequences generated by student behavior The specific formula for user modeling is as follows: The dynamic weight coefficients are set for three dimensions. , respectively corresponding to the recommended sequences Basic cognitive level and learning style For new users, The weight should be higher than the other two. The final weight value is affected by the accuracy and recall of the TOP-N recommendation results of the recommendation system, and satisfies the following conditions: The weights take values ​​in the range [0,1], and the specific formulas are as follows: The formula includes formulas for calculating precision and recall, used to verify the precision and recall of the system. The number of recommended lists; This indicates the actual number of items in the list; it also includes the harmonic mean of precision and recall. The calculation formula is then used to calculate the final weight value based on the precision, recall, and harmonic mean mentioned above.

Images