Personalized course recommendation method and system based on multi-modal data analysis

Abstract

The invention belongs to the field of education data mining methods, and provides a personalized course recommendation method and system based on multi-modal data analysis, and the technical scheme is as follows: obtaining learning data of students in different courses and multi-modal data of different courses; extracting multi-modal features of the students based on the learning data of the students; extracting corresponding modal features from the multi-modal data of each course, mapping different modal features to the same space to obtain the mapped modal features of each course, and splicing the mapped features to obtain the multi-modal representation of each course; and in combination with the multi-modal representation of the student, the multi-modal representation of each course and the trained course recommendation model, calculating to obtain a matching degree score of the course and the student, and recommending a corresponding course to the student according to the matching degree score. And the accuracy of course prediction is improved.

Description

Technical Field

[0001] The present invention belongs to the field of educational data mining methods, and in particular to a personalized course recommendation method and system based on multimodal data analysis. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] Traditional teaching methods usually use uniform teaching content and progress, which makes it difficult to meet the personalized learning needs of different students. With the development of artificial intelligence technology, how to use AI technology to analyze students' learning situation and provide personalized feedback has become an important research direction in the field of education.

[0004] Although the current course recommendation method has shifted from single-dimensional modeling to multi-dimensional modeling, taking into account the influencing factors of the students themselves and the courses themselves, modeling the students' cognitive abilities and online courses respectively, and then using the characteristics of online courses to predict students' interest in courses, this solution has the following technical problems: on the one hand, the students' learning status and interest cognitive abilities may not be well learned, resulting in unreliable course prediction results; on the other hand, when matching courses for students through supervised learning, the hidden modal characteristics of students' non-interactive courses are not considered, and some modal features are missed, resulting in inaccurate course recommendations. Summary of the invention

[0005] In order to solve at least one of the technical problems existing in the above-mentioned background technology, the present invention provides a personalized course recommendation method and system based on multimodal data analysis, which improves the accuracy of course recommendation by learning the multi-dimensional characteristics of users for courses and learning the characteristics of hidden states.

[0006] In order to achieve the above object, the present invention adopts the following technical solution: A first aspect of the present invention provides a personalized course recommendation method based on multimodal data analysis, comprising the following steps: Obtain students’ learning data in different courses and multimodal data of different courses; Extract students' multimodal features based on their learning data; Extract the corresponding modal features from the multimodal data of each course, map different modal features to the same space, obtain the mapped modal features of each course, and concatenate the mapped features to obtain the multimodal representation of each course; The matching score between the course and the student is calculated by combining the multimodal representation of the student, the multimodal representation of each course and the trained course recommendation model, and the corresponding courses are recommended to the students based on the matching score.

[0007] Furthermore, the extracting of multimodal features of students based on their learning data includes: The learning ability vector is constructed based on learning data and the established multi-level learning ability evaluation index system; The text vectors are extracted based on the acquired text data of each course and the trained language model, and the text vectors are input into the bidirectional GRU network for training to obtain the hidden memory information in the text; The learning ability vector is used as a priori knowledge representation, combined with the hidden memory information carrying deep emotional characteristics for attention interaction, to obtain the student's learning interest feature vector; Locate the face in the image data, estimate the head posture in the face area, judge the student's attention state based on the head posture, and extract the learning state feature vector; The learning ability vector, learning interest feature vector and learning status feature vector are fused through a fully connected layer to obtain the final multimodal representation of the student.

[0008] Furthermore, the multi-level learning ability evaluation index system includes primary indicators and secondary indicators, and each primary indicator includes multiple secondary indicators; Among them, the first-level indicators include knowledge mastery and problem solving , learning efficiency , Innovation capability ; Knowledge acquisition and problem solving Including knowledge point coverage , Depth of knowledge points and error rate , learning efficiency Including learning amount per unit time , the correlation between study time and grades , innovation capability Include scores or comments on open-ended tasks , the number of unique solutions proposed in the task , the number of times students actively searched for additional resources .

[0009] Furthermore, the corresponding modal features are extracted from the multimodal data of each course, and different modal features are mapped to the same space to obtain the modal features of each course after mapping, including: Set the quantity modal size to N The sample pairs consist of: ,in, From modal U , From modal V , …, From modal L , the corresponding codes are expressed as , where each is taken as a positive example, and all other samples are It is considered a negative example; Each two modal data is treated as a group for migration mapping, and the corresponding loss function is: , , in, Representing modality V middle and The intra-modal similarity weight between is the temperature parameter, Representing modality V Middle k The encoding representation corresponding to each modal feature.

[0010] Furthermore, the matching score between the course and the student is calculated by combining the multimodal representation of the student, the multimodal representation of each course and the trained course recommendation model, including: Construct a calculation formula for the student's match score for the course; The multimodal representation of each course determines the student's matching score for the positive and negative courses; Determine the hidden mode of the course, combine the student's multimodal representation and the student's matching scores for the positive and negative courses, and calculate the student's matching score for the hidden negative course.

[0011] Furthermore, the loss function of the course recommendation model is , , , in, is the overall loss function, is the recommendation loss, is the hidden mode regularization, is the set of technical parameters, To adjust the balance parameter of importance, is the weight of parameter regularization to prevent overfitting, For Students Negative Example Course k The matching score of For Students The regular course of For Students Negative example courses, is the training data set, For Students For regular courses The matching score of For Students Negative Example Course k The matching score of For Students For hidden negative examples The matching score of .

[0012] A second aspect of the present invention provides a personalized course recommendation system based on multimodal data analysis, comprising: A data acquisition module, which is used to acquire students' learning data in different courses and multimodal data of different courses; A student feature extraction module, which is used to extract multimodal features of students based on their learning data; The course feature extraction module is used to extract the corresponding modal features from the multimodal data of each course, map different modal features to the same space, obtain the mapped modal features of each course, and concatenate the mapped features to obtain the multimodal representation of each course; The course recommendation module is used to combine the multimodal representation of students, the multimodal representation of each course and the trained course recommendation model to calculate the matching score between courses and students, and recommend corresponding courses to students based on the matching score.

[0013] A third aspect of the present invention provides a computer-readable storage medium.

[0014] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps in the personalized course recommendation method based on multimodal data analysis as described above.

[0015] A fourth aspect of the present invention provides a computer device.

[0016] A computer device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps in the personalized course recommendation method based on multimodal data analysis as described above are implemented.

[0017] A fifth aspect of the present invention provides a program product.

[0018] A program product, which is a computer program product, includes a computer program, and when the computer program is executed by a processor, the steps in the personalized course recommendation method based on multimodal data analysis as described above are implemented.

[0019] Compared with the prior art, the present invention has the following beneficial effects: 1. The present invention mines the correlation between students and courses from the multi-dimensional learning characteristics of students in the learning process of different courses, and well learns the interest feature vectors, learning state feature vectors, and learning ability vectors of students in the learning process of different courses, thereby enhancing the credibility of course prediction results. At the same time, the present invention considers the characteristics of the hidden modes of students for non-interactive courses, mines the hidden partial modal features, and improves the accuracy of course matching.

[0020] 2. The present invention extracts corresponding modal features from the multimodal data of each course, and when mapping different modal features to the same space, uses a continuous similarity metric to align the embedding spaces of any two different modalities in the continuous space, and measures the similarity based on the degree of attraction between all samples. This can achieve more fine-grained modal gap calculations and avoid completely ignoring samples with continuity similarity in the training data.

[0021] Advantages of additional aspects of the present invention will be given in part in the following description, and in part will become obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0023] Figure 1 is a flow chart of a personalized course recommendation method based on multimodal data analysis provided by an embodiment of the present invention; Figure 2 is a schematic diagram of a multimodal feature extraction process for students provided by an embodiment of the present invention; Figure 3 is a comparison result of the accuracy of different course recommendation algorithms provided by the embodiments of the present invention; Figure 4 is a comparison result of the recall rates of different course recommendation algorithms provided by the embodiments of the present invention; Figure 5 is the F1 score comparison result of different course recommendation algorithms provided by the embodiments of the present invention; Figure 6 This is a comparison result of average response times of different course recommendation algorithms provided by the embodiments of the present invention. DETAILED DESCRIPTION

[0024] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0025] It should be noted that the following detailed descriptions are all illustrative and intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0026] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0027] As mentioned in the background technology, although the current course recommendation method has the following technical problems: on the one hand, the students' learning status and interest cognitive ability may not be well learned, resulting in unreliable course prediction results; on the other hand, when matching courses for students through supervised learning, the hidden modal features of students' non-interactive courses are not considered, and some modal features are missed, resulting in inaccurate course matching. The present invention models the personalized characteristics of different students by combining the comprehensive characteristics of students and the prompts of different modalities of different courses, and designs user learning under hidden modalities, which can effectively and efficiently learn students' cognitive abilities, thereby improving the accuracy of course recommendations.

[0028] Embodiment 1 like Figure 1 As shown, this embodiment provides a personalized course recommendation method based on multimodal data analysis, including the following steps: Step 1: Obtain students’ learning data in different courses and multimodal data of different courses; Given a set of students , Course Collection , course mode collection , each mode in the course mode set has its corresponding mode feature, and each student There is a collection of interactive history courses ; The learning data of students in different courses include homework data, examination data, online learning behavior data of each course, and classroom scene image data in class; Among them, the homework data of each course includes homework grades, such as the score of each homework, homework difficulty, homework difficulty level, whether it matches the course progress, homework feedback data including teachers' and parents' comments and suggestions on homework, etc.; Examination data for each course includes examination results, examination scores, examination time, etc. For example, the scores of large examinations such as midterm and final examinations, the scores of each question, and the time taken by students to complete the examinations; Online learning behavior data includes login frequency, learning time, resource usage, interaction, learning path and completion progress.

[0029] Through this step, we obtain students’ learning data in different courses and multimodal data of different courses, and we can make full use of relevant data to improve the accuracy of prediction.

[0030] Step 2: Extract students’ multimodal features based on their learning data in different courses, including learning ability features, learning preference features, and learning status features; like Figure 2 As shown, the specific steps include: Step 201: Based on the learning data and the established multi-level learning ability evaluation index system, a learning ability vector is constructed. ; In this embodiment, the multi-level learning ability evaluation index includes a primary index and a secondary index, and each primary index includes a plurality of secondary indexes; Among them, the first-level indicators include knowledge mastery and problem solving , learning efficiency , Innovation capability ; Knowledge acquisition and problem solving Including knowledge point coverage , Depth of knowledge points and error rate , learning efficiency Including learning amount per unit time , the correlation between study time and grades , innovation capability Include scores or comments on open-ended tasks , the number of unique solutions proposed in the task , the number of times students actively searched for additional resources ; Specifically, knowledge acquisition and problem solving The data acquisition of each indicator includes: Knowledge point coverage The ratio of the number of knowledge points learned to the total number of knowledge points can be used for evaluation. For example, if the mathematics course contains 50 knowledge points and the number of knowledge points learned covers 40, the coverage rate is 80%; Depth of knowledge , can be distinguished by questions of different difficulty levels, including multiple-choice questions and application questions, and can be scored using rubrics, such as 1 to 5 points corresponding to different levels; for example, if a student can recite the formula (memory) but cannot solve the variant problem (application), then the depth of mastery is shallow; Error rate You can use the ratio of the number of incorrectly answered questions to the total number of questions. For example, if a certain knowledge point is answered incorrectly 6 times out of 10 questions, the error rate is 60%, then it needs to be paid special attention to.

[0031] Learning efficiency The data acquisition of each indicator includes: Learning volume per unit time The amount of knowledge that students have mastered or the amount of learning tasks that they have completed within a fixed period of time can be measured by the ratio of learning outcomes (such as the number of knowledge points or exercises) to learning time (hours). When obtaining specific data, learning software such as Pomodoro Todo can be used to automatically record the effective time. Correlation between study time and grades To determine the statistical relationship between time spent on learning and grades, and to judge the effectiveness of time spent, Pearson correlation coefficient analysis can be used. The larger the absolute value of the Pearson correlation coefficient, the stronger the correlation. Innovation capability The data acquisition of each indicator includes: Scores on open-ended tasks : Traditional standardized scoring methods often fail to fully reflect students' comprehensive abilities. In this embodiment, based on the acquired knowledge, a set of evaluation systems that take into account objectivity and flexibility is established in open tasks (such as project-based learning, exploratory topics, creative design, etc.). A structured rubric is used to decompose evaluation indicators from multiple dimensions. Each dimension is given a clear grade description, and the teacher gives the score, as shown in Table 1: Table 1 Multi-dimensional scoring criteria

[0032] The number of unique solutions proposed in the task The number of non-standardized and differentiated solutions proposed by students when solving problems. For example, in the brainstorming stage, if the repetition rate between the solutions submitted by students in the solution design stage and the existing solutions is less than a set value, such as 10%, it is considered a unique solution; Number of times students actively searched for additional resources To measure the frequency with which students independently obtain resources outside the textbook (such as literature, tools, and expert consultation) during the task process, by recording the number of books and websites consulted.

[0033] Step 202: extracting text vectors based on the acquired text data of each course and the trained language model, and inputting the text vectors into a bidirectional GRU network for training to obtain hidden memory information in the text; In this embodiment, the text data of each course obtained is, for example, the text content of students in the discussion area or feedback; Use pre-trained word vector models to embed unstructured text into structured high-dimensional vectors To represent the semantic information of the input text, represents the embedding representation of each word, k Represents the length of the text.

[0034] Then use the bidirectional GRU memory network to model the contextual dependency information of the text, combined with the height vector And GRU memory network to obtain the text memory matrix with deep interest features ; Step 203: Using the learning ability vector as a prior knowledge representation, combined with the text memory matrix carrying deep interest features to perform attention interaction, and obtain the student's learning preference features; In order to realize the text memory matrix with deep interest features and learning ability vector The attention interaction fusion includes the following steps: Step 2031: The learning ability vector Mapped to the text memory matrix through a fully connected layer (FC) Same dimensions: , in, , t is the hidden dimension of the text memory matrix; Step 2032: Using the learning capability vector As a query, the text memory matrix As the key and value, the attention weight matrix A is obtained by interacting through the attention mechanism; , Step 2033: Use the attention weight matrix A to calculate the text memory matrix Perform weighted summation to obtain the learning preference feature vector : ; Step 204: Extract the student learning state features based on the acquired image data and the constructed attention state learning network ; By monitoring whether the students' eyes stay in areas unrelated to the events in class for a long time (under the desk, outside the window, etc.), it can be judged whether the students are distracted. The specific steps include: Step 2041, preprocessing the acquired image data to obtain a face image; Based on the camera in the classroom, the current learning image of the students is obtained; the obtained image data is preprocessed, specifically including: using existing cropping tools such as Dockerface to crop the information irrelevant to the face in the image, and finally obtaining the face image; Step 2042: extracting head posture features based on facial key point coordinates; The coordinates of the facial contour, eyes, eyebrows, nose and lips of the face detected from the cropped area are: , is the number of data points; In this embodiment, based on the existing 3D head general model and 2D facial key point coordinates The corresponding transformation relationship between them is used to obtain the 3D head vector, and then the 3D head vector is normalized into a one-dimensional vector; Specifically, the 3D head general model can adopt the FLAME model; Step 2043: The preprocessed face image and the trained initial feature detection network are used to obtain an initial face detection result. ; In this example, SqueezeNet, which was originally trained on the ImageNet dataset, is used as the backbone network for fine-tuning, and combined with the BN layer and the convolutional layer for joint training to obtain the initial face detection results. ; Step 2044: Initial face detection result Perform weighted coding of the eye area, enhance the information of the eye area, and output facial coding feature maps with different weight ratios ; Specifically, the head posture features are weightedly coded in the eye region, including weighted coding of the eye region using an eye region weighted coding module, wherein the eye region weighted coding module includes three additional 2×1 convolutional layers with a kernel size of 7, followed by a sigmoid nonlinear activation function; the input of the eye region weighted coding module is the initial face detection result , and finally generate a spatial weight matrix, expressed as: , in, and To output the width and height of the special diagnosis; Step 2045: Facial encoding feature map Compared with the initial face detection result Perform element-by-element multiplication along the channel direction to obtain the final encoding feature matrix S; Step 2046: The head posture feature extracted in step 2042 and the encoding feature matrix output in step 2045 are fused to obtain the final learning state feature, which is expressed as: , in, represents the Kronecker inner product.

[0035] Step 205: The learning ability vector , learning preference feature vector and learning status characteristics The final student multimodal representation is obtained by fusion of the fully connected layers .

[0036] The present invention fully extracts the multi-dimensional learning characteristics of students in the learning process of different courses, which is conducive to exploring the correlation between students and courses.

[0037] Step 3: Extract the corresponding modal features from the multimodal data of each course, map the different modal features to the same space, obtain the mapped modal features of each course, and concatenate the mapped features to obtain the multimodal representation of each course; Traditional contrastive loss functions are usually used for unimodal self-supervised learning and multimodal alignment, and neither considers that there may be highly similar samples of the same type in the same batch of training. For example, some categories may have the same attributes and are considered to be counterexamples to each other. Secondly, the existing contrastive loss simply defines similarity as a binary attribute. All "positive examples" are equally drawn, while all "negative examples" are equally excluded, which is actually inconsistent with the actual situation. This embodiment proposes to use a continuous similarity metric that can align the embedding spaces of any two different modalities in a continuous space, and measure the similarity based on the degree of attraction between all samples, so that a more fine-grained modal gap calculation can be achieved.

[0038] make Represents a batch of training data pairs, which consists of a set number of modal sizes N The sample pairs consist of: ,in, From modal U , From modal V , …, From modal L , the corresponding codes are expressed as , where each is taken as a positive example, and all other samples are It is considered a negative example; When mapping different modal data, each two modal data are used as a group for migration mapping, for example , Migrate With other modes For alignment, the corresponding loss function is: , in, Indicates the modal V and The intra-modal similarity weight between them can be used to impose targeted constraints on different samples during comparative optimization. is the temperature parameter, Representing modality V Middle k The encoding representation corresponding to each modal feature; When calculating, the weights need to be standardized as The principle of weight adjustment is: similar samples in the same modality have higher weights, and different samples should have lower weights. In this embodiment, the modality V The similarity weight within is set as: , According to the loss function, the feature mapping of all course multimodal data is completed into the same space, and finally the multimodal representation of the course is spliced ​​to obtain .

[0039] Step 4: Combine the multimodal representation of students, the multimodal representation of each course, and the trained course recommendation model to calculate the matching score between the course and the student, and recommend corresponding courses to the students based on the matching score; The specific steps include: Step 401: Construct a calculation formula for the student's matching score for the course: , in, For studentsFor courses The matching score of For Students For Courses In modal The matching score under For students in Learning features in modality, For Courses No. modal representation; Step 402: determine the hidden mode of the course, and calculate the matching score of the student to the hidden negative course by combining the student's multimodal representation and the hidden negative course; The existing recommendation objective function can only supervise the final student's fitness learning, and cannot optimize the learning of students in each specific modality. It is possible that the learning characteristics of students in a certain modality are not well learned, resulting in unreliable predicted matching scores of students in some modalities, thus affecting the courses finally recommended for students.

[0040] In this embodiment, in order to more accurately enhance the learning characteristics of students in the hidden mode, the hidden mode is first constructed: Given a training set , For Students The regular course of For Students of negative classes (randomly sampled from classes that the student has not interacted with); If the student is in For the positive course under the modal and negative example courses The difference in matching scores is small, making it difficult to distinguish The matching degree between the modal course and the user defines it as a hidden modal; For each training group , calculate the student For regular courses and negative example courses The matching score gap is defined as the hidden mode with the smallest score gap. The calculation formula is: , in, For triples The index of the hidden modal in and In the Students in mode For regular courses and negative example courses The matching score of For the Students in mode For regular courses and negative example courses The difference in matching scores; To enhance learning in latent patterns, for each training set Constructing a hidden negative example curriculum , expressed as: , According to the students' multimodal representation and Hidden Negative Class Multimodal Representation , get students For hidden negative examples Matching score : , The purpose of hidden modal regularization is to For regular courses The interest score is greater than that for hidden negative courses Interest score: , in, For Students For regular courses The matching score of .

[0041] Step 403: The constructed recommendation loss function constrains the recommendation result; The training goal is that the student's matching score for the positive example course is greater than the student's matching score for the negative example course and the hidden modal negative example course.

[0042] The loss function for course recommendation model training is defined as: , , in, is the overall loss function, is the recommendation loss, is the set of technical parameters, To adjust the balance parameter of importance, is the weight of parameter regularization to prevent overfitting, For Students Negative Example Course k The matching score of .

[0043] The present invention mines the correlation between students and courses from the multi-dimensional learning characteristics of students in the learning process of different courses, and well learns the preference characteristics, cognitive ability characteristics and learning status characteristics of students in the learning process of different courses, thereby enhancing the credibility of course prediction results. At the same time, the present invention considers the characteristics of students' hidden modes for non-interactive courses, mines some hidden modal features, and improves the accuracy of course matching.

[0044] In order to verify the effectiveness of the present invention, a comparative experimental analysis was conducted; The data of the present invention mainly includes: multimodal data of different courses and students' learning data in different courses. Multimodal data of different courses: course videos, course audios, course names and text information. Students' learning data in different courses include: homework data, test data and online learning behavior data of each course, and classroom scene image data in the classroom; This dataset started in March 2020, and collected 32,413 learners, 813 courses, 63,936 explicit feedback data, and 81,537 implicit feedback data. During the experiment, the present invention further divided the dataset into three disjoint sets, and randomly selected 80%, 10%, and 10% of the course data as the training set, validation set, and test set, respectively. The validation set was used for model optimization. Finally, the final model performance was verified on the test set.

[0045] This experiment compares the hit rate performance of different course recommendation algorithms in practical applications. The evaluation indicators include accuracy, recall rate and F1 score. The specific data are shown in Table 2. The specific comparison results are as follows Figure 3-Figure 6 As shown, Figure 3 This is the accuracy comparison result of different course recommendation algorithms; Figure 4 It is the recall comparison result; Figure 5 This is the F1 score comparison result; Figure 6 This is the average response time comparison result.

[0046] Table 2 Comparison of hit rates of different course recommendation algorithms

[0047] The experimental results show that the course recommendation method of the present invention can achieve better course recommendation effect, with an accuracy rate of 82.5%, indicating that the present invention can effectively improve the accuracy of course recommendation. Compared with the traditional recommendation algorithm, it is more effective, has a higher hit rate, and has the highest recall rate, which can avoid omissions and has more comprehensive coverage. At the same time, the response time is 210m, which is relatively short.

[0048] Embodiment 2 This embodiment provides a personalized course recommendation system based on multimodal data analysis, including: A data acquisition module, which is used to acquire students' learning data in different courses and multimodal data of different courses; A student feature extraction module, which is used to extract multimodal features of students based on their learning data; The course feature extraction module is used to extract the corresponding modal features from the multimodal data of each course, map different modal features to the same space, obtain the mapped modal features of each course, and concatenate the mapped features to obtain the multimodal representation of each course; The course recommendation module is used to combine the multimodal representation of students, the multimodal representation of each course and the trained course recommendation model to calculate the matching score between courses and students, and recommend corresponding courses to students based on the matching score.

[0049] It should be noted that the specific implementation method of the personalized course recommendation system based on multimodal data analysis in the embodiment of the present invention is similar to the specific implementation method of the personalized course recommendation method based on multimodal data analysis in the embodiment of the present invention. Please refer to the description of the method part for details. In order to reduce redundancy, it will not be repeated here.

[0050] Embodiment 3 This embodiment provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the steps in the personalized course recommendation method based on multimodal data analysis as described above are implemented.

[0051] Embodiment 4 This embodiment provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps in the personalized course recommendation method based on multimodal data analysis as described above are implemented.

[0052] Embodiment 5 This embodiment provides a program product, which is a computer program product, including a computer program. When the computer program is executed by a processor, the steps in the personalized course recommendation method based on multimodal data analysis as described above are implemented.

[0053] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A personalized course recommendation method based on multimodal data analysis, characterized in that: The steps include: Obtain students’ learning data in different courses and multimodal data of different courses; Extract students' multimodal features based on their learning data; Extract the corresponding modal features from the multimodal data of each course, map different modal features to the same space, obtain the mapped modal features of each course, and concatenate the mapped features to obtain the multimodal representation of each course; The matching score between the course and the student is calculated by combining the multimodal representation of the student, the multimodal representation of each course and the trained course recommendation model, and the corresponding courses are recommended to the students based on the matching score.

2. The personalized course recommendation method based on multimodal data analysis according to claim 1, characterized in that: The extracting of multimodal features of students based on their learning data includes: The learning ability vector is constructed based on learning data and the established multi-level learning ability evaluation index system; The text vectors are extracted based on the acquired text data of each course and the trained language model, and the text vectors are input into the bidirectional GRU network for training to obtain the hidden memory information in the text; The learning ability vector is used as a priori knowledge representation, combined with the hidden memory information carrying deep emotional characteristics for attention interaction, to obtain the student's learning interest feature vector; Locate the face in the image data, estimate the head posture in the face area, judge the student's attention state based on the head posture, and extract the learning state feature vector; The learning ability vector, learning interest feature vector and learning status feature vector are fused through a fully connected layer to obtain the final multimodal representation of the student.

3. The personalized course recommendation method based on multimodal data analysis according to claim 2, characterized in that: The multi-level learning ability evaluation index system includes primary indicators and secondary indicators, and each primary indicator includes multiple secondary indicators; Among them, the first-level indicators include knowledge mastery and problem solving , learning efficiency , Innovation capability ; Knowledge acquisition and problem solving Including knowledge point coverage , Depth of knowledge points and error rate , learning efficiency Including learning amount per unit time , the correlation between study time and grades , innovation capability Include scores or comments on open-ended tasks , the number of unique solutions proposed in the task , the number of times students actively searched for additional resources .

4. The personalized course recommendation method based on multimodal data analysis according to claim 1, characterized in that: The corresponding modal features are extracted from the multimodal data of each course, and different modal features are mapped to the same space to obtain the modal features of each course after mapping, including: Set the quantity modal size to N The sample pairs consist of: ,in, From modal U , From modal V , …, From modal L , the corresponding codes are expressed as , where each is taken as a positive example, and all other samples are It is considered a negative example; Each two modal data is treated as a group for migration mapping, and the corresponding loss function is: ， ， in, Representing modality V middle and The intra-modal similarity weight between is the temperature parameter, Representing modality V Middle k The encoding representation corresponding to each modal feature.

5. The personalized course recommendation method based on multimodal data analysis according to claim 1, characterized in that: The matching score between the course and the student is calculated by combining the multimodal representation of the student, the multimodal representation of each course and the trained course recommendation model, including: Construct a calculation formula for the student's match score for the course; Determine the student's matching scores for positive and negative courses based on the multimodal representation of each course; Determine the hidden mode of the course, combine the student's multimodal representation and the student's matching scores for the positive and negative courses, and calculate the student's matching score for the hidden negative course.

6. The personalized course recommendation method based on multimodal data analysis according to claim 1, characterized in that: The loss function of the course recommendation model is: ， ， ， in, is the overall loss function, is the recommendation loss, is the hidden mode regularization, is the set of technical parameters, To adjust the balance parameter of importance, is the weight of parameter regularization to prevent overfitting, For Students Negative Example Course k The matching score of For Students The regular course of For Students Negative example courses, is the training data set, For Students For regular courses The matching score of For Students Negative Example Course k The matching score of For Students For hidden negative examples The matching score of .

7. Personalized course recommendation system based on multimodal data analysis, characterized by: include: A data acquisition module, which is used to acquire students' learning data in different courses and multimodal data of different courses; A student feature extraction module, which is used to extract multimodal features of students based on their learning data; The course feature extraction module is used to extract the corresponding modal features from the multimodal data of each course, map different modal features to the same space, obtain the mapped modal features of each course, and concatenate the mapped features to obtain the multimodal representation of each course; The course recommendation module is used to combine the multimodal representation of students, the multimodal representation of each course and the trained course recommendation model to calculate the matching score between courses and students, and recommend corresponding courses to students based on the matching score.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps in the personalized course recommendation method based on multimodal data analysis as described in any one of claims 1 to 6 are implemented.

9. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps in the personalized course recommendation method based on multimodal data analysis as described in any one of claims 1 to 6 are implemented.

10. A program product, the program product being a computer program product, comprising a computer program, characterized in that: When the computer program is executed by a processor, the steps in the personalized course recommendation method based on multimodal data analysis as described in any one of claims 1 to 6 are implemented.

Images