Multi-modal teaching content recommendation system and establishing and using method thereof

Abstract

The invention relates to a method for establishing a multi-modal teaching content recommendation system. The method comprises the following steps: constructing a knowledge base and generating category labels; meanwhile, preparing a training set and a test set according to the text content of each slice, constructing a vocabulary, and performing text coding on the training set and the test set; the method comprises the following steps of: training an encoder framework of a Transform; establishing a user portrait generation function which generates a user portrait based on the input question and the historical dialogue; and integrating to obtain a multi-modal teaching content recommendation system, so that the multi-modal teaching content recommendation system obtains question-related slices and user-interested slices according to input questions and historical dialogues, and retrieves and outputs course resources from a knowledge base. According to the method provided by the invention, the character understanding and generation capability of the large model and the classification retrieval capability of the encoder architecture on the user semantics are organically combined, so that the retrieval precision of the user question information can be effectively improved, and an exclusive learning route is recommended for the user.

Description

Technical Field

[0001] The present invention relates to the fields of information processing technology and recommendation systems, and in particular to a multimodal teaching content recommendation system and a method for establishing and using the system. Background Art

[0002] In recent years, large language models have created revolutionary changes in many fields, especially in the field of natural language processing (NLP). These models are pre-trained on large-scale data sets, learn rich language features and world knowledge, and can understand and generate unprecedented complex texts. Their capabilities are not limited to basic text analysis, but also extend to complex semantic understanding and generation tasks, providing unprecedented opportunities for industries including educational technology, especially in the automated extraction of teaching content, knowledge structuring, and interactive learning assistance.

[0003] In addition, in the field of education, the traditional "one-size-fits-all" teaching method is gradually unable to meet the increasingly diverse learning needs. The modern educational concept emphasizes "teaching students in accordance with their aptitude and individuality", that is, providing personalized teaching content according to each student's ability, interest and learning pace. This teaching method requires educators to not only understand the specific needs of each student, but also to be able to provide corresponding teaching resources, which brings great challenges in practice. Therefore, there is an urgent need to realize personalized retrieval and recommendation of teaching content through technical means to improve teaching effectiveness and student satisfaction.

[0004] At present, by combining the big language model technology, building a private domain course resource knowledge base and workflow, branching and selecting user questions and outputting answers, it can effectively integrate the information of the course itself and enhance the accuracy of the answers to the course knowledge. However, relying solely on keywords or the ability of the big model itself to make judgments may cause the workflow to be unable to correctly connect to the sharded knowledge base, thereby relying on the ability or illusion of the big model itself to output answers to users, which does not meet the actual requirements of the course content. The preprocessing method of data balancing, summary generation and chi-square test based on the big model can effectively build a vocabulary library, assist in text encoding, and then use neural networks for classification to locate the retrieval content of the question, thereby effectively solving the problem of hallucinations caused by the big model.

[0005] Most existing recommendation systems use collaborative filtering or content-based recommendation methods. However, with the widespread use of user-generated content (UGC) and the development of large models, how to effectively use historical conversation information and generate personalized recommendations through deep learning algorithms has become an important issue. Traditional collaborative filtering may face sparsity problems, while personalized recommendations based on user conversation content can more accurately understand user needs and provide relevant content.

[0006] Therefore, it is necessary to provide a new multimodal teaching content recommendation system to recommend relevant knowledge to users based on the historical information of users' questions. Summary of the invention

[0007] The purpose of the present invention is to provide a multimodal teaching content recommendation system and its establishment method and use method, so as to effectively improve the retrieval accuracy of user question information through the combination of deep learning and recommendation technology.

[0008] In order to achieve the above object, the present invention provides a method for establishing a multimodal teaching content recommendation system, comprising:

[0009] S100: Prepare course resources;

[0010] S200: constructing a knowledge base including a plurality of slices according to the course resources and generating a category label for identifying the slices, wherein the knowledge base is used to output the course resources corresponding to the slices to the user according to the category label; at the same time, preparing a training set and a test set according to the text content of each slice, constructing a vocabulary, and performing text encoding on the training set and the test set;

[0011] S300: using the result of text encoding as an input sample, training the encoder architecture of the Transformer to obtain a trained neural network model, wherein the neural network model is used to identify the user's input question to obtain a category label of a slice related to the question;

[0012] S400: Establishing a function for generating a user portrait, wherein the function for generating a user portrait is configured to extract user preference information based on a user's input question and historical conversations, and generate a user portrait according to the user preference information, wherein the user portrait includes slices that the user is interested in;

[0013] S500: Integrate the knowledge base, neural network model and user portrait generation function into a multimodal teaching content recommendation system, so that the multimodal teaching content recommendation system obtains question-related slices and user-interested slices based on the user's input questions and historical conversations, and retrieves and outputs corresponding course resources from the knowledge base.

[0014] The step S100 includes:

[0015] Provide course materials and course videos of relevant teaching courses in pdf or markdown format as course resources;

[0016] Based on gpt-4-vision-preview and the Tongyi Tingwu model, the pictures in the course materials and course videos are used to generate picture text introductions and video text summaries.

[0017] The step S200 further comprises:

[0018] S210: Slice and organize the text materials of the textbook according to chapters and subsections to form a course text knowledge base that is easy to manage and retrieve; then establish an image knowledge base and a video knowledge base, wherein the relational data in the image knowledge base and the video knowledge base include the extracted image text introduction and video text summary as well as the image address and video address; then, the chapter and subsection are combined as the category label of the prediction model;

[0019] S230: Prepare a training set and a test set according to the text content of each slice, and build a vocabulary based on high-frequency words in the text content; wherein the training set includes the segmented text and corresponding category labels; the test set includes test questions and the category label corresponding to each test question; and perform text encoding on the segmented text and the test questions according to the vocabulary.

[0020] After step S210 and before step S230, the method further includes step S220: using a large language model to perform text balancing on the text content of each section of each chapter and generate a summary text; and / or after step S230, the method further includes step S240: padding the training samples and test samples obtained after text encoding to ensure that the lengths are consistent.

[0021] The step S300 specifically includes:

[0022] S320: configuring and initializing a neural network model of a Transformer encoder architecture, the neural network model comprising a word embedding, layer, a position encoding layer, a Transformer encoding layer, and an output layer; the word embedding layer is used to map the index of each word of the input sample to a 128-dimensional vector to obtain a first word embedding tensor; the sine-cosine position encoding layer captures the relationship between words based on the first word embedding tensor and the sine-cosine position encoding to obtain a third result tensor containing position information; the Transformer encoder layer uses a multi-head attention mechanism to obtain a fourth encoder layer output tensor; the category output layer converts the output of the Transformer encoder layer into a score of the sample for each category;

[0023] The third result tensor containing position information is obtained by adding the second position tensor and the first word embedding tensor corresponding to the training sample; for the position encoding tensor of the second position tensor before dimension transformation, the calculation formula for the position encoding of each dimension i is:

[0024] If i is even,

[0025] If i is an odd number,

[0026] Among them, pos is the position index, i is the dimension index, and d_model is the dimension of word embedding.

[0027] S330: The neural network model is trained using the training tensor in the result of text encoding, wherein the cross entropy loss function CELLoss is used as the loss function and the Adam optimizer is used for training, and a confidence threshold is set for each category. The neural network model outputs a prediction result only when the prediction probability of the category exceeds this confidence threshold.

[0028] Before step S320, the method further includes step S310: loading training samples and test samples from the results of text encoding to obtain training data sets and test data sets, and the loading is used to ensure the random order of training samples in each training cycle; and / or after step S330, the method further includes step S340: testing the neural network model using the test samples from the results of text encoding, and evaluating the model performance of the trained neural network model based on the test results.

[0029] The step S400 includes:

[0030] S410: Establishing a user preference information extraction module, wherein the user preference information extraction module is configured to obtain user preference information based on the user's input questions and historical conversations; the user preference information includes preference for text, picture or video, and whether the current question is related to the course content;

[0031] S420: Establish a function for generating a user portrait, wherein the function is configured to collect historical conversations and call the LLMChain chain to generate a user portrait, wherein the user portrait includes slices that the user is interested in.

[0032] The function for generating a user portrait is obtained in the following way:

[0033] S421: Select a recommended model;

[0034] S422: constructing a chat prompt template, wherein the chat prompt template includes prompt words for generating a user portrait;

[0035] S423: Define an LLMChain chain, which is configured to receive the user's input questions, historical conversations, and created chat prompt templates, and pass the chat prompt templates as input prompt words to the recommendation model, so that the recommendation model generates a corresponding user portrait based on this, and the user portrait includes chapters and sections that the user is interested in.

[0036] On the other hand, the present invention provides a multimodal teaching content recommendation system, which is established using the method for establishing a multimodal teaching content recommendation system described above.

[0037] In another aspect, the present invention provides a method for using a multimodal teaching content recommendation system, comprising:

[0038] A1: Providing a multimodal teaching content recommendation system according to claim 9;

[0039] A2: Use the trained neural network model to identify the relevant slices of the questions based on the user's input questions, and retrieve and output the course resources of the corresponding slices from the knowledge base;

[0040] A3: Use the user profile generation function to obtain the slices that the user is interested in based on the user's input questions and historical conversations, and retrieve and output the recommended links of the course resources of the corresponding slices from the knowledge base.

[0041] The method for establishing a multimodal teaching content recommendation system of the present invention utilizes a neural network based on a Transformer encoder architecture to establish a knowledge retrieval classification model for user questions, and provides personalized content recommendations through a content-based recommendation algorithm; thereby, the text understanding and generation capabilities of the large model are organically combined with the classification and retrieval capabilities of the Transformer-Encoder architecture for user semantics, which can effectively improve the retrieval accuracy of user question information, alleviate the large model illusion problem, and recommend exclusive learning routes for users, making the recommendation system efficient, accurate and user-friendly, and suitable for various scenarios such as e-books, online learning platforms, and information aggregation applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 4 is an architecture diagram of a multimodal teaching content recommendation system according to an embodiment of the present invention.

[0043] Figure 2 4 is a schematic diagram of steps S100 to S300 of the modal teaching content recommendation system according to an embodiment of the present invention. DETAILED DESCRIPTION

[0044] The preferred embodiments of the present invention are given below in conjunction with the accompanying drawings and described in detail.

[0045] like Figure 1 As shown, the workflow of the multimodal teaching content recommendation system established by the present invention mainly includes two parts:

[0046] First, a neural network model based on the Transformer Encoder architecture is used to predict relevant chapters and subsections based on the user's query input, and to retrieve and output the recommended links of the corresponding course resources from the knowledge base through a large language model;

[0047] Secondly, using content-based recommendation algorithms, we combine the user's query results and historical conversations to obtain the chapters and subsections that the user is interested in, and retrieve and output the recommended links of the corresponding course resources from the knowledge base through a large language model.

[0048] That is to say, the recommendation system obtained by the method for establishing the multimodal teaching content recommendation system of the present invention can not only improve the accuracy and relevance of content recommendations, but also enhance the user experience through intelligent content integration and output. First, the neural network part adopts the Transformer-Encoder model to effectively process and understand the natural language of user queries and extract high-quality feature vectors for subsequent content matching and recommendation decisions. Secondly, the content-based recommendation algorithm analyzes the content features and user interaction data in the knowledge base to identify and recommend content similar to the user query, so as to provide richer and more accurate recommendation options.

[0049] like Figure 2 As shown, according to one embodiment of the present invention, a multimodal teaching content recommendation system based on a private domain course resource knowledge base includes the following steps:

[0050] S100: Prepare course resources.

[0051] The step S100 includes:

[0052] Provide course materials and course videos of relevant teaching courses in pdf or markdown format as course resources;

[0053] Based on gpt-4-vision-preview and the Tongyi Tingwu model, the pictures in the course materials and course videos are used to generate picture text introductions and video text summaries.

[0054] In this embodiment, the course resources adopt the information management engineering textbooks and teaching videos of a certain university, so as to establish a multimodal teaching content recommendation system based on the information management engineering textbooks and teaching videos of a certain university, aiming to push relevant text descriptions and course videos based on the textbooks to students through students' independent questioning, realize interactive course retrieval, and realize polishing and modification based on the big model. Among them, the above-mentioned information management engineering textbooks and teaching videos are organized in the form of: there are 10 chapters in total, each chapter has 2-5 sections, and students can locate relevant knowledge points through chapters-sections. The pictures in the course textbook files are the contents nested in the chapters and sections of the textbooks, which are used to assist in understanding the text content, and the pictures have been stored in the files of the textbook courseware. The video is an explanation video of each section of the textbook, with teacher voice explanation.

[0055] There are several pictures embedded in each section of the teaching material courseware, so the picture is a graphical display of the theoretical content of the section, or a mind map of the content of the section, which can summarize or supplement the content of each section. The present invention uses the gpt-4 model developed by the OpenAI team for image processing, and the selected model version number is "gpt-4-vision-preview". The gpt-4-vision-preview model can understand pictures and generate natural language text through deep learning technology. According to the picture, the picture text introduction is obtained, specifically including: first, obtaining the API of the gpt-4 model on the OpenAI official website; then, based on the Python code, encoding the picture file into Base64 format; then calling the API of the gpt-4 model to obtain the picture text introduction intro_for_pics, wherein the message parameters of the set API are: "role": "system", "content": "Please summarize and describe the knowledge point content introduced in this picture", calling the API to complete the image processing, and obtaining the picture text introduction intro_for_pics.

[0056] For each section of the course teaching video, speech-to-text recognition is performed on Alibaba's Tongyi Tingwu website based on a dedicated large model, and the video text introduction video_to_txt is obtained through the video-to-text version. In the present invention, the "glm-4" large language model developed by the Zhipu Qingyan team is used for video text processing and correction. The selected model version number is "glm-4-air". Compared with the OpenAI large model, the glm-4-air model is more economical in use and focuses on text processing.

[0057] Therefore, according to the video, we get the video_to_txt and the video text abstract_for_video, and generate the image index knowledge base and the image knowledge base, which specifically include:

[0058] Get the API of GLM-4-air from Zhipu official website;

[0059] Import the video_to_txt based on the Python code based on Tongyi Tingwu extraction;

[0060] The GLM-4-air model is called to summarize, refine and conclude the extracted video text to generate the video text abstract_for_video to ensure that there are no typos and semantic inconsistencies after recognition. The message parameters of the API are set as follows: "role": "system", "content": "The following text content is a video-to-text version of a certain university course. Please summarize, refine and modify it to ensure that there are no sentences that are inconsistent with semantics and classroom reality."

[0061] Step S200: constructing a knowledge base including a plurality of slices according to the course resources and generating category labels for identifying the slices, wherein the knowledge base is used to output the course resources corresponding to the slices to the user according to the category labels; at the same time, preparing a training set and a test set according to the text content of each slice, constructing a vocabulary, and performing text encoding on the training set and the test set;

[0062] The knowledge base includes a course text knowledge base, a picture knowledge base and a video knowledge base.

[0063] The step S200 specifically includes:

[0064] Step S210: Slice and organize the text materials of the textbook according to chapters and sections to form a course text knowledge base that is easy to manage and retrieve; then establish a picture knowledge base and a video knowledge base, where the relational data in the picture knowledge base and the video knowledge base include the extracted picture text introduction and video text summary as well as the picture address and video address; then, combine the chapters and sections as category labels for the prediction model.

[0065] Step S210 specifically includes:

[0066] Step S211: convert the text data into a file format, and organize it into sections according to different chapters and subsections to form a course text knowledge base.

[0067] In the step S211, the PDF file is converted into the markdown format based on the PyMuPDF software, and the markdown documents corresponding to different sections under different chapters are first split according to the chapter content. That is: the corresponding section titles in the chapter are normalized by markdown using #, ##, etc. The corresponding documents are stored in the course text knowledge base base_for_textbook. Thus, the course text knowledge base includes a json overall overview file and a file library based on chapter and section arrangement. The storage format of the json overall overview file is: the chapters and sections to which the corresponding documents belong are recorded and stored, for example: Document 1 records the content of the first section of the first chapter, then the first record is: {"text_id":"text1","title":"Introduction","chapter":1,"section":1,"url":"https: / / example.com / 1"}.

[0068] Step S212: Process the pictures and videos; that is, generate a picture index knowledge base and a picture library based on cloud disk storage, as well as a video index knowledge base and a video library based on cloud disk storage according to the picture text introduction and video text summary. The picture library and video library based on cloud disk storage store the pictures and corresponding videos in the courseware; the picture index knowledge base and video index knowledge base record and store the picture text introduction and video text summary information and cloud disk storage link.

[0069] There are several pictures embedded in each section of the teaching material courseware, so the picture is a graphical display of the theoretical content of the section, or a mind map of the content of the section, which can summarize or supplement the content of each section. The present invention uses the gpt-4 model developed by the OpenAI team for image processing, and the selected model version number is "gpt-4-vision-preview". The gpt-4-vision-preview model can understand pictures and generate natural language text through deep learning technology. According to the picture text introduction, a picture index knowledge base and a picture knowledge base are formed, specifically including: replacing the original picture with the picture text introduction intro_for_pics, and establishing a picture index knowledge base according to the picture and the picture text introduction, the picture is stored in the folder directory of the picture index knowledge base to obtain the picture address url_for_pics, and the picture address url_for_pics and the picture text introduction intro_for_pics are imported into the picture knowledge base base_for_pics in the format of relational data.

[0070] In this embodiment, the picture knowledge base base_for_pics includes a json general overview file and a file library based on chapter and subsection arrangement. The storage format of the json general overview file is: record and store the chapter and subsection to which the corresponding picture belongs. For example: Figure 1 The first record records the content of the first section of the first chapter: {"pic_id":"pic1","title":"Introduction","chapter":1,"section":1,"url":"https: / / example.com / video1"}. The images are stored in the folder directory of the image index knowledge base, and the folders are arranged in the chapter-subsection format.

[0071] Therefore, according to the video_to_txt and video text abstract_for_video, the image index knowledge base and image knowledge base are generated, including:

[0072] Create a video index knowledge base (located on the resource cloud disk) based on the video and video text abstract_for_video, store the video in the folder directory of the video index knowledge base, and obtain the video address url_for_video of each video; put the video address url_for_video, video conversion text video_to_txt and video text abstract_for_video into the video description table video_excel, and import them into the local video knowledge base base_for_video.

[0073] In this embodiment, the video knowledge base base_for_video includes a json general overview file and a file library based on chapter and subsection arrangement; the json file storage format is: record and store the chapters and subsections to which the corresponding video belongs, for example: video 1 records the content of the first subsection of the first chapter, then the first record is: {"video_id":"video1","title":"Introduction","chapter":1,"section":1,"url":"https: / / example.com / video1"}. Videos are stored in folder directories, and folders are arranged according to the arrangement of chapters and subsections.

[0074] Step S213: Process the course text; that is, extract the text content under the first-level chapter name (corresponding to chapter chapter) and the second-level title name (corresponding to subsection session) according to the course text knowledge base, and merge the image text description intro_for_pics and the video to text video_to_txt and the video text abstract_for_video into the corresponding text content, so that each piece of text content uniquely points to a group (chapter chapter, subsection session).

[0075] The teaching material used in the training of the present invention has a total of 10 chapters, in which the first-level chapter name (i.e., the title of the chapter) based on the markdown format has been expressed with # before the chapter title in step S201. Each chapter title is composed of 2-5 subsections, and the second-level title name (i.e., the title of the subsection session) has been expressed with ## in the markdown format in step S201 above.

[0076] Therefore, in step S203, a regular expression is used based on Python code to extract the content under the title, wherein the regular expression for the first-level chapter name is: ^#[^#].*, and the regular expression for the second-level title name is: ^##[^#].*, and finally the text content under the first-level chapter name and the second-level title name is extracted; then, according to the image and video processing conditions, the image text description intro_for_pics and the video to text video_to_txt and the video text abstract_for_video are added to the subsection of the corresponding chapter as part of the text content of the subsection (i.e., the second-level title name); finally, each piece of text content uniquely points to a group (chapter, session).

[0077] Step S214: data labeling, that is, combining the first-level chapter name and the second-level title name as the category label of the prediction model to convert the dual-objective prediction model into a single-objective prediction model;

[0078] For example, the predicted category label for the text content in the first subsection of the first chapter is 1, and the predicted category label for the text content in the second subsection of the first chapter is 2, and so on, until all chapters and subsections are labeled with their categories.

[0079] In the present invention, the prediction model is a Transformer-Encoder model.

[0080] Step S220 (optional): Use a large language model to perform text balancing on the text content of each section in each chapter, and generate summary text to obtain integrated text content, which is used to help understand key information.

[0081] The step S220 specifically includes:

[0082] Step S221: Perform text balancing, that is, count the number of words in each section of each chapter (i.e., all categories), and calculate the average μ and standard deviation σ of the number of words in all sections based on the 3-σ principle. If the number of words in a section is less than μ-3σ, the text content in the section is enhanced.

[0083] Enhance the text content under this section, including: calling the API of the GLM-4-air model based on Python code, and setting the API message parameters as follows: {"role":"system","content":"Suppose you are a senior education expert who is compiling a textbook"}, {"role":"user","content":"The above text is the knowledge point content of Section x of Chapter x of this textbook. Since the number of words is too different from other sections, in order to ensure the effectiveness of subsequent modeling and prediction, please enhance the knowledge points involved in this section without excluding other irrelevant knowledge points, expand its text, and add your output to the end of the original section text"}.

[0084] Step S222: perform text enhancement, that is, summarize the text content under each category label based on Python calling the API of GLM-4-air to obtain the abstract text abstract.

[0085] The text content is summarized mainly in terms of knowledge points and methods, so as to generate a summary text abstract for the text content under each category label. The summary text abstract enhances the original text.

[0086] Step S223: text integration, that is, by simple splicing, the abstract text abstract is directly added to the end of the original text content content to obtain the integrated text content complete_file.

[0087] Among them, add between the original text content and the abstract text abstract <sep>In addition, since the present invention is aimed at studying professional knowledge textbooks, the text of this type of textbooks is highly descriptive and has no emotional tone, so the influence of punctuation marks on the semantic understanding of the text can be ignored. Therefore, a unified strategy is adopted to standardize all punctuation marks in the text - the punctuation marks (,,;.!?......——) are also standardized. <sep>Mark replacement. Store the integrated text as a txt file and record it as the integrated text content complete_file.

[0088] Step S230: Prepare a training set and a test set according to the text content of each slice, and build a vocabulary based on the high-frequency words in the text content; wherein the training set includes the segmented text and the corresponding category labels; the test set includes test questions and the category label corresponding to each test question; and perform text encoding on the segmented text and the test questions according to the vocabulary.

[0089] The word segmentation process is performed based on the jieba library. The word frequency is initially screened, and the chi-square test is performed on the tokens with a word frequency ≥ 3. The p-value of the chi-square statistic is calculated, and the threshold of the significance level of the p-value is set to 0.05. When p-<0.05, the token is retained in the vocabulary, and finally a vocabulary suitable for teaching content is constructed.

[0090] Step S231: segmenting the text content to obtain segmented text, preparing a training set according to the segmented text, and preparing a test set according to the text content;

[0091] The text content is segmented, specifically including: based on Python calling the API of the large model GLM-4-air to segment the integrated text content complete_file in each chapter and section.

[0092] The message parameters of the API are: {"role":"system","content":"Assume that you are a professor of relevant courses in colleges and universities."},{"role":"user","content":"Now you need to segment the following text. The segmentation method can be determined by keyword and concept recognition to determine whether to segment. If a paragraph is logically divided into different parts (such as explanations, examples, summaries, etc.), it can also be segmented. Make sure that each segment corresponds to the same knowledge point after segmentation. Please output the original text directly and add the symbol & between the two segments to be segmented."}.

[0093] The role of segmentation is to break down long textbook content (including summary text) into small segments that are easier to handle and understand, and each small segment corresponds to only a single knowledge point. Each segmented part is not necessarily a complete knowledge point. A knowledge point may span multiple segmented segments. Segmentation is only to make each text more structured and easy to handle. Among them, & is used to mark the segmentation position as a segmentation mark. Based on the split function of python, the segments on the left and right of the & symbol are separated, and the segmented text knowledge under different subsection sessions of each chapter chapter is obtained by segmentation. Among them, the segmented text knowledge under the same subsection session of the same chapter chapter corresponds to the same label label, and the segmented text is stored as the segmented text file split_knowledge.csv. This csv file has 2 columns, the first column is the segmented text knowledge, and the second column is the category label label corresponding to the segmented text knowledge, and the category label label has been obtained in step S214 above.

[0094] The training set is a csv file, which is the split_knowledge.csv file obtained above, that is, the text knowledge after segmentation and the corresponding category label label. According to the actual content of each section in the textbook, the text content of each section is sorted out.

[0095] Based on Python calling the GLM-4-air API, 100 relevant academic questions test_qusetion are generated for each section as the test set text. The message parameters of the API are set as follows: {"role":"system","content":"Assume that you are a student receiving higher education."},{"role":"user","content":"Now you are studying the following text. Please ask questions about different concepts, examples, and logical structures in the text from different angles, different contents, and different ways of thinking. The questions raised must meet academic standards."}. The test set is also stored as the test_qusetion.csv file. The file has two columns. The first column is the test question test_qusetion, and the second column is the category label label corresponding to the test question, which indicates which section (that is, which category) the test question is asked for.

[0096] Step S232: Preliminarily construct a vocabulary based on the high-frequency words in the text content, and then perform a chi-square test on the high-frequency words to filter out the high-frequency words that are independent of the category label label to obtain the final vocabulary.

[0097] Among them, a vocabulary is preliminarily constructed based on the high-frequency words in the text content, specifically including: using the pandas library to import the integrated text content complete_file obtained in step S223, using the jieba library to segment the integrated text content complete_file, counting the frequency of each word, obtaining the high-frequency words in the text content and preliminarily constructing the vocabulary vocab.

[0098] Among them, except <sep>In addition to the separator, the words that appear more than or equal to 3 times (ie: freq≥3) are defined as high-frequency words. The high-frequency words are added to the vocabulary vocab.

[0099] Perform a chi-square test on high-frequency words to filter out high-frequency words that are independent of the category label label and obtain the final vocabulary, which includes:

[0100] Step S2321: Calculate the expected frequency of high-frequency words under the assumption that the high-frequency words and the category label label are independent.

[0101] For high-frequency words ω and category labels label, n ω is the total frequency of the high-frequency word ω in the entire corpus, which refers to the total number of times the high-frequency word ω appears in the entire corpus. label is the total frequency of the high-frequency word ω in the category label label, which refers to the total number of times the high-frequency word ω appears under a specific category label. N is the total word frequency of the entire corpus, that is, the sum of the number of times each word appears in the entire corpus. Expected frequency E ω,label It is the number of times the high-frequency word ω is expected to appear, assuming that the variables of the category labels label are independent.

[0102] Among them, the expected frequency E of high-frequency words ω,label The calculation method is

[0103] Step S2322: For each high-frequency word ω, calculate its chi-square statistic in all category labels label.

[0104] The expression for the chi-square statistic is

[0105] Among them, O ω,label is the observed frequency of the high-frequency word ω in the category label label, is the number of times a specific high-frequency word ω actually appears in the category label label, and E ω,label is the expected frequency of the high frequency words.

[0106] Step S2322: Evaluate the significance of each high-frequency word ω and optimize the vocabulary accordingly.

[0107] Step S2322 specifically includes: calculating the p-value of the chi-square statistic, and setting the threshold of the significance level of the p-value to 0.05. As long as the p-value of the chi-square statistic of the high-frequency word and any category is less than the threshold, we reject the hypothesis (the word is independent of the category) and believe that the word is significantly associated with the specific category label label, and retain the word with a significant p-value in the vocabulary vocab. If a word is not significant in all category labels, that is, the p-value is greater than the threshold, the word is considered to be irrelevant to the category, fails the chi-square test, and is removed from the vocabulary vocab. Finally, a special unknown word marker is added to the vocabulary vocab. <unk>, used to represent words with low frequency (not stored in the vocabulary) in subsequent encoding. Set the vocabulary vocab to a dictionary form, where the key is the word and the key value is a sequence of positive integers starting from 2. <unk>The key value is 0. <sep>The key value is 1. <sep>It is an indicator of punctuation and the integration of original text and summary text.

[0108] Step S233: text encoding, that is, according to the vocabulary, text encoding is performed on the segmented text knowledge of the training set to obtain a training tensor train_sequences, and text encoding is performed on the test set to obtain a test tensor test_sequences.

[0109] Among them, the segmented text knowledge is the segmented text that has been segmented by GLM-4-air. According to the segmented text file split_knowledge.csv obtained in step S231 and the vocabulary obtained in step S232, an index-based representation method is adopted to map the vocabulary to an index, perform precise word segmentation on the segmented text, and then perform index mapping. When traversing each word in the sample, check whether the word is in the vocabulary. If not, convert it to 0; if it is in the vocabulary, it is represented as the key value of the corresponding word in the vocabulary. Similarly, each test question in the test set test_qusetion.csv file is encoded based on the vocabulary index to obtain a test sample.

[0110] Since the maximum length of the splitted_content of all training samples obtained is the maximum sample length max_len, after step S230, step S240 can also be included: padding the splitted_content of training samples whose sample length is less than the maximum sample length max_len, with a padding value of 0, so that the length of all training samples split_content is consistent through padding; similarly, padding the encoding results of test problems whose sample length is less than the maximum test sample length, with a padding value of 0, so that the length of all test texts and training text encodings is consistent through padding, thereby maintaining the unity and stability of the model.

[0111] Finally, each section (i.e. each slice) under each chapter in the course file was encoded and obtained the corresponding training sample split_content, and different training samples of different slices were combined into the training tensor train_sequences. Different encodings of different test samples were combined into the test tensor test_sequences.

[0112] Step S300: using the result of text encoding as an input sample, training the encoder architecture of the Transformer to obtain a trained neural network model, wherein the neural network model is used to identify the user's input question to obtain a category label of a slice related to the question;

[0113] The step S300 specifically includes:

[0114] Step S310 (optional): Load the training samples and test samples in the text encoding results. In step S200, the training samples (including the text under each section of each chapter) and the test samples (including each question in the test set) have been text encoded, and the training tensor train_sequences and the test tensor test_sequences have been generated. To facilitate batch loading and training, PyTorch's DataLoader is used to load and manage data. First, load the training samples. Convert the encoded training sample sequences into a PyTorch Dataset object, and batch load them through DataLoader, save them as training data set train_data_loader, and the tensor shape of the loaded training tensor is [batch_size, seq_len]. Batch_size represents the batch size, which is the number of samples processed by the model at the same time during a training process. seq_len represents the sequence length, which is the number of characters in each input sample. Set shuffle = True during training to ensure the random order of training samples in each training cycle. Secondly, load the test samples. Based on the obtained test set file test_qusetion.csv, the encoded test samples are also converted into PyTorch Dataset objects, and batch loaded through DataLoader and saved as test_data_loader. The tensor shape of the loaded test tensor is [batch_size, seq_len]. Since the test samples do not need to be randomized, the DataLoader configuration of the test set does not use shuffle.

[0115] Step S320: configure and initialize a neural network model of the encoder architecture of the Transformer model, wherein the neural network model includes a word embedding layer, a position encoding layer, a Transformer encoder layer, and a category output layer.

[0116] It should be noted that the encoder architecture of the Transformer model is existing, and the present invention only uses its encoder part but not the decoder part.

[0117] The word embedding layer is used to map the index of each word of the input sample to a 128-dimensional vector to obtain the first word embedding tensor. The sine-cosine position encoding layer captures the relationship between words based on the first word embedding tensor and sine-cosine position encoding to obtain the third result tensor containing position information. The Transformer encoder layer uses a multi-head attention mechanism to obtain the fourth encoder layer output tensor. The category output layer converts the output of the Transformer encoder layer into a score for each category of the sample.

[0118] For the word embedding layer, the input data format is tensor [batch_size, seq_len], and the output data format is tensor [batch_size, seq_len, embedding_dim]; for the sine-cosine positional encoding layer, the input data format is tensor [batch_size, seq_len, embedding_dim], and the output data format is still in the form of tensor [batch_size, seq_len, embedding_dim]; for the Transformer encoding layer, the input data format is the data from the word embedding layer and the positional encoding layer. The output of the layer is a tensor [batch_size, seq_len, embedding_dim], and the output is a tensor processed by the self-attention mechanism and the feedforward neural network, with a shape of [batch_size, seq_len, hidden_dim]; for the output layer, the input is the output tensor [batch_size, seq_len, hidden_dim] of the Transformer encoding layer, and the output is the probability distribution of each category. The output shape is a tensor [batch_size, num_classes], and each sample corresponds to the score or probability distribution of the category.

[0119] Among them, batch_size represents the batch size, which is the number of samples processed by the model at the same time during a training process; seq_len and sequence_length represent the sequence length, which is the number of characters in each input sample; embedding_dim represents the embedding dimension, which is the dimension of the vector represented by each input character. During the forward propagation process, the input is transposed to ensure that the input meets the requirements of nn.TransformerEncoder in PyTorch. nn.TransformerEncoder is the encoder part of the Transformer model, which is responsible for processing the input sequence and generating context-related representations.

[0120] The word embedding layer is used to map the index of each word of the input sample to a 128-dimensional vector. The input of the word embedding layer is the loaded training tensor train_dataloader and the loaded test tensor test_dataloader, where the format of each input sample is a tensor [batch_size, seq_len]. In the word embedding layer, each word (i.e., the index of the word) is mapped to the corresponding high-dimensional vector representation by looking up the word embedding matrix. Each word will be mapped to an embedding vector with a dimension of embedding_dim=128. After word embedding, the output tensor shape is [batch_size, seq_len, embedding_dim], that is, each word corresponds to a 128-dimensional vector representation, and finally forms the embedding representation tensor train_embedding_sequences (recorded as the first word embedding tensor) containing all training samples and the embedding representation tensor test_embedding_sequences containing all test samples.

[0121] The sine-cosine positional encoding layer captures the relationship between words based on sine-cosine positional encoding. Define the PositionalEncoding class to add position information to the input, where the input is the tensor output by the word embedding layer in the format of [batch_size, seq_len, embedding_dim].

[0122] First, create an all-zero tensor pe0 of size (max_len, d_model) to initialize the data structure. Pre-allocate memory by creating a base tensor to store the position encoding information of each position later. The all-zero tensor pe0 ensures that before the actual calculation of the position encoding, there is an all-zero tensor pe0 with a matching structure and size, so that operations can be performed directly on it, such as filling the calculated position encoding value. Among them, max_len is the maximum length of the input sequence, and the size is equal to the maximum length of the sample max_len; d_model is the dimension of word embedding, that is, embedding_dim, which is set to 128 here, which is consistent with the embedding_dim in the tensor output by the word embedding layer.

[0123] Next, generate a first position tensor position, which is a column of values ​​from 0 to max_len-1, representing the index of each position. The format of the first position tensor position is (max_len,).

[0124] Through the unsqueeze(1) function, the dimension of the first position tensor position is changed from (max_len,) to (max_len,1), that is, a new dimension is inserted into the first dimension of the first position tensor position to adapt it to the subsequent broadcast operation. In this way, the dimension of the first position tensor position can match the dimension d_model of the word embedding after expansion, so as to perform the correct addition operation with the position encoding later.

[0125] The first position tensor position is expanded to a larger dimension based on a broadcast operation to match the dimension d_model of the word embedding, resulting in a position encoding tensor pe stored at the all-zero tensor pe0.

[0126] The position encoding tensor pe consists of multiple position encoding vectors, each of which has a dimension of d_model, where d_model = 128. For each position in each dimension, the position encoding is calculated using two formulas. For positions with even dimensions, the position encoding is based on the formula Calculation, for odd-numbered dimensions, the position encoding is based on the formula Calculate, where pos is the position index, indicating the position of the word in the sequence, and is an integer sequence from 0 to max_len-1; i is the dimension index, indicating a dimension in the embedding vector, and the value range of i is an integer between 0 and 63;

[0127] Finally, through the unsqueeze(0) function, a new dimension is inserted at position 0 of the position encoding tensor pe, and through the transpose(0,1) function, the 0th dimension and the 1st dimension of the position encoding tensor pe are swapped, and finally the dimension of the position encoding tensor pe is transformed to (1, max_len, d_model), so that the position encoding tensor pe obtains the second position tensor through dimension transformation.

[0128] The second position tensor is added to the first word embedding tensor corresponding to the training sample to obtain a third result tensor containing position information in the format of [batch_size, seq_len, embedding_dim], which can assist the model in understanding semantic information.

[0129] The Transformer encoder layer uses a multi-head attention mechanism to obtain the output tensor of the fourth encoder layer.

[0130] The Transformer encoder layer is created using nn.TransformerEncoderLayer, and the Transformer encoder layer uses a multi-head attention mechanism and a feedforward neural network to extract contextual information of the input sequence. The multi-head attention mechanism enables the model to process multiple information points in the sequence in parallel, thereby more comprehensively capturing semantics and dependencies and enhancing the model's information integration capabilities.

[0131] The Transformer encoder layer consists of a self-attention mechanism and a feedforward network, and usually also includes layer normalization and residual connections.

[0132] The maximum length of each group of input samples of the Transformer encoder layer is max_len, and the shape of the generated attention mask attention_mask is [batch_size, seq_len], which is used to control the attention of the Transformer-Encoder model to certain positions when calculating self-attention, and can mask the padding part in the input data when calculating self-attention. In this embodiment, the specific configuration includes: setting hidd en dim=128, num head=4, dim feedforward=512, drop out=0.1, and stacking 2 layers of encoders. The hidden layer dimension (hidden_dim) represents the feature dimension of each head output and the internal feedforward network in the multi-head attention mechanism and feedforward network, which is set to 128 in the present invention. The number of attention heads (num_head) is the number of independent attention mechanisms operating simultaneously in the multi-head attention mechanism, and num_head is set to 4. The feedforward network dimension (dim_feedforward) is the dimension of the hidden layer in the feedforward neural network, which is set to 512. The dropout rate (drop_out) refers to randomly discarding a certain proportion of data units during training to prevent the model from overfitting. The setting value is 0.1. The Transformer encoder layer uses relu as the activation function. In addition, the Transformer encoder layer enhances the processing power of the model by stacking 2 layers of encoders (num_layers=2). The overall encoding process is completed by nn.TransformerEncoder, and the shape of the output tensor of the fourth encoder layer is [batch_size, seq_len, hidden_dim]. The output contains the complex representation of each position after multi-layer Transformer encoding, which can effectively capture various dependencies within the sequence.

[0133] The category output layer converts the output of the Transformer encoder layer into the score of the sample for each category. Through the indexing method src=src[:,-1,:], the output of the last time step of each sample sequence of the fourth encoder layer output tensor is used as the input of the linear layer, in order to capture and utilize the cumulative information of the entire sequence. The linear layer nn.Linear(hidden_dim,num_class) mainly performs a linear transformation, mapping each feature vector extracted from the encoder from the hidden_dim dimension to a new dimensional space (num_class dimension), where each dimension represents the score of a category, which is set according to the number of category labels predefined in the chapters and sections. The output of this layer is the fifth category output layer output tensor logits, in the format of [batch_size,num_class]. Each row of the fifth category output layer output tensor logits represents the score of a sample for each category.

[0134] Output the probability distribution of samples belonging to each category. The output of the category output layer is the fifth category output layer output tensor, where each row represents the score of a sample for each category. These scores are converted into predicted category probabilities through the softmax function, forming a tensor of shape [batch_size, num_class], and the probability distribution of each input sample belonging to each category is output.

[0135] The loss function and optimizer settings of the neural network model. The present invention uses the cross entropy loss function nn.CELLoss() as the loss function for training, and the formula is It is used to calculate the gap between the model prediction result and the actual target. Where N is the total number of samples, M is the total number of categories, and y ij is an indicator variable, which is 1 if sample i belongs to category j, and 0 otherwise. is the probability that the i-th sample is predicted to be category j. The optimizer selected the Adam optimizer optim.Adam(), and the initial value of the learning rate was set to 0.001. The Adam optimizer performs adaptive learning rate adjustment based on gradient descent, which can speed up the convergence speed and improve the stability of the model. The process of the Adma algorithm is as follows:

[0136] Get the new round of gradient value: g t =▽ θ f(θ t-1 )

[0137] Update the first moment vector: m t =β1·m t-1 +(1-β1)·g t

[0138] Update the second moment vector:

[0139] Compute the bias-corrected first moment vector:

[0140] Compute the bias-corrected second moment vector:

[0141] Update parameters: Until θ t convergence.

[0142] Among them: α is the learning rate, which is initially set to 0.001.

[0143] Step S330: Use the training tensor in the result of text encoding to train the neural network model, wherein the cross entropy loss function CELLoss is used as the loss function and the Adam optimizer is used for training, and a confidence threshold is set for each category. Only when the prediction probability of the category exceeds this confidence threshold, the neural network model outputs a prediction result.

[0144] The training process of the neural network model is carried out in multiple training rounds epochs. In each training round epoch, the training data set train_data_loader obtained according to the training tensor in step S310 is traversed. The training data set train_data_loader divides the training tensor train_sequences in the result of text encoding into multiple small batches, allowing the neural network model to process only a small part of the data in each training step instead of loading the entire training data set, effectively managing memory usage and improving computing efficiency. The data of each batch is input into the neural network model for forward propagation. The original output of the neural network model is the fifth category output layer output tensor logits, corresponding to the original prediction score of each category.

[0145] When using the loss function to evaluate the model loss, the output tensor logits of the fifth category output layer is based on the softamax function to obtain the log probability log_probs predicted by the model. This log probability is the log probability of each category predicted by the neural network model based on the input text, not the true log probability. When processing batch data containing padding, use mask mask to identify valid data points to ensure that only non-filled, valid logits are converted to log probability log_probs, and log_probs[mask] only calculates and stores the log probability of the corresponding valid data points. Based on the file split_knowledge.csv stored above, obtain the category label label of the training sample, that is, the target label targets of the model. Similarly, obtain targets[mask] of the valid data point through mask selection operation. Then use pytorch's nn.CrossEntropyLoss (i.e., cross entropy loss function CELLoss) to calculate the cross entropy loss, and calculate the loss value between the log probability log_probs[mask] predicted by the model and the target label targets[mask]. The mask here is used to ensure that only the loss of the valid input part (non-filled part) is calculated. Call optimizer.zero_grad() to clear the gradient calculated last time, perform back propagation through loss.backward(), and calculate the gradient. Finally, call optimizer.step() to update the model parameters. The loss of each batch is accumulated and recorded in the total loss total_loss. After each round, the total loss of the current training round is output to facilitate monitoring of the loss change trend during training. After the training is completed, the model will apply a confidence threshold of 0.5 to the probability of each predicted category in actual application. In this embodiment, the confidence threshold is 0.5.

[0146] The predicted category is the category label label of the training sample in the stored split_knowledge.csv file. Only when the predicted probability exceeds this confidence threshold, the neural network model outputs the relevant prediction results to ensure that the output results have high reliability. If there is no category greater than the confidence threshold, it is judged as "irrelevant question" and no relevant detection and answer support is provided to the user.

[0147] Step S340: Use the test samples in the text encoding result to test the neural network model, and evaluate the model performance of the trained neural network model based on the test results.

[0148] Traverse the test data set test_data_loader obtained according to the test tensor test_sequences in step S310. During the test process, the test data set test_data_loader loads samples one by one in the order of the data set, does not disrupt the continuity of the data, but loads the data in batches. During the test process, the neural network model uses torch.no_grad() to prohibit gradient calculation to prevent back propagation and improve computational efficiency, performs forward propagation on the input data, and generates the model output output. Get the predicted category of each sample, that is, the predicted result of the category label label involved in the test text question content, and compare it with the target true label target label of the test set.

[0149] In an exemplary embodiment, the present invention adopts the following computer hardware and software configuration: the central processing unit (CPU) is the 13th generation Intel (R) Core (TM) i9-13900HX, equipped with two graphics processing units (GPUs), including Intel (R) UHD Graphics and NVIDIA GeFo rce RTX 4050 Laptop GPU. The CUDA compiler version used by the system is CUDA12.4, specifically V12.4.131. The system runs on the Windows 11 operating system, the programming language is Python 3.8.19, the deep learning framework uses PyTorch 1.9.1, and supports the GPU acceleration function of CUDA11.6 for building and training neural network models.

[0150] The model performance of the trained neural network model is evaluated based on the test results, including: the neural network model will traverse all samples in the training data set train_data_loader in each training round epoch, and then the test set will not be used as a training sample, but only the test set will be used to evaluate the training function value of the trained neural network model.

[0151] The test results are as follows: from the 11th to the 13th epoch, the training loss of the neural network model tends to stabilize with the increase of epochs, and no longer gradually decreases, triggering the early stopping mechanism, which indicates that the neural network model has converged.

[0152] Evaluate the model performance of the trained neural network model based on the test results, and also include: Get the predicted category of the model based on output.argmax (dim = 1). This function determines the predicted category label label by finding the category index of the maximum value in each sample (i.e., each row of the output tensor). Dim = 1 means that the specified function finds the maximum value along the category dimension (i.e., the category score in each row). Compare the predicted category label label with the actual target category targetslabel, calculate the number of correct predictions and accumulate them. Finally, calculate the overall accuracy acc of the neural network model on the test set, and output the overall accuracy on the test set.

[0153] The examples show that analyzing the scope of user questions through text preprocessing and the Transformer-Encoder model is more efficient than directly searching in various knowledge bases with the computing power of a large language model, and the accuracy rate is improved by about 15%. Through this strategy, the scope of user questions can be effectively located and answers can be output.

[0154] The performance of the trained neural network model was evaluated based on the test results, which also included: Based on Python calling the API of GLM-4-air, 50 groups of irrelevant questions were generated in the financial field, technology field, etc., and the relevant questions were stored as not_related_questions.csv files, and encoded and imported to be processed as test data sets. The results showed that the neural network model predicted 48 groups of questions as irrelevant questions and did not provide output to the user, which means that the model's ability to capture the relevance of questions exceeded 90%.

[0155] Step S400: Establish a function for generating a user portrait, wherein the function for generating a user portrait is configured to extract user preference information based on the user's input questions and historical conversations, and generate a user portrait according to the user preference information, wherein the user portrait includes slices that the user is interested in.

[0156] Wherein, step S400 specifically includes:

[0157] S410: Establish a user preference information extraction module, wherein the user preference information extraction module is configured to analyze and obtain user preference information (such as interests, needs and preferences) based on user input questions and historical conversations.

[0158] The user preference information extraction module is configured to perform the following steps:

[0159] Step S411: determine whether there is a historical conversation, if so, execute step S412, otherwise, only obtain the category label of the question-related slice according to the user's input question;

[0160] In this embodiment, the question record of the current user in the conversation is obtained through the get_conversation_history(conversation_id) function, and a maximum of 5 question records are obtained. If there are more than 5 question records, it is determined that there is a historical conversation, and step S12 is executed to generate personalized recommendations for the user based on the content of the historical conversation.

[0161] Step S412: extracting user preference information from the acquired user's historical conversations based on the large language model GLM-4-air.

[0162] Among them, the input parameter settings when calling the GLM-4-air API are as follows: {"role":"system","content":"As a course consultant, your goal is to identify the user's interpretation of text materials or resource query needs. Please analyze the user's historical conversation records and extract user preference information. Specifically, you need to determine whether the user prefers to obtain course text materials or other resources (such as pictures, videos, etc.). At the same time, evaluate whether the user's current question is closely related to the course content. Please output your analysis results in the following format: 'User preference: textbook materials / picture materials / video materials; question relevance: relevant / irrelevant'."}, if the current user question is not related to the course, it will not be answered or recommended.

[0163] Therefore, the user preference information finally extracted includes the preference for text, picture or video, and whether the current question is related to the course content. Therefore, when making recommendations below, the relevant content (text, picture or video) will be output in combination with the user preference information.

[0164] Step S420: Establish a function for generating a user portrait, wherein the function is configured to collect historical conversations and call the LLMChain chain to generate a user portrait, wherein the user portrait includes slices that the user is interested in.

[0165] The user profile user_profile_ans is generated by the generate_user_profile_and_extract_info function, where the user profile user_profile_ans only includes [Chapter] and [Session]. The former is used to identify the course chapter that the user is interested in, and the latter is used to identify the user's interest in the section or unit within the course.

[0166] The generate user portrait function is an asynchronous method that is designed to generate a user portrait based on the user's chat history.

[0167] The input parameters of the function for generating user profiles include the user chat history message list chat_messages, the prompt word user_profile_prompt for generating user profiles, and the instance model of the large language model GLM-4-air. The instance model of the large language model GLM-4-air is the model loaded by calling the API of GLM-4-air. When instantiating the model, the temperature should also be set to adjust the randomness of the generated content.

[0168] The function to generate a user profile is obtained by creating a chat prompt template for generating a user profile, and defining an LLMChain chain for receiving user input questions, historical conversations, and the created chat prompt template and sending it to the recommendation model. Specifically, the function generate_user_profile_and_extract_info for generating a user profile is obtained by:

[0169] Step S421: selecting a recommendation model, wherein the recommendation model is used to generate recommendation content that meets user needs.

[0170] In this embodiment, the recommendation model uses the GLM-4-air large language model, and the default model generates content with a sampling temperature of temperature = 0.7, and there is no limit on the maximum token of the generated content. Based on the GLM-4-air large language model and temperature settings, the system calls the large language model to generate recommended content that meets user needs. By adjusting the temperature parameter, the system can control the randomness of the generated content to adapt to different user scenarios.

[0171] Step S422: construct a chat prompt template, wherein the chat prompt template includes a prompt word user_profile_prompt for generating a user profile.

[0172] Step S422 specifically includes: first, creating a chat prompt template through the function ChatPromptTemplate.from_mess ages(). The chat prompt template contains a user's dialogue input, that is, the prompt word user_profile_prompt for generating a user profile, which is the core prompt information for the model to generate a user profile. The chat prompt template constructs appropriate prompt words so that the user profile generation function can understand that its task is to generate a user profile based on the chat history and extract key information.

[0173] The chat prompt template includes a prompt word user_profile_prompt for generating a user portrait. When organizing the prompt word user_profile_prompt for generating a user portrait, it is necessary to first input the GLM-4-air large language model to input the user chat history message list history(List), and then write the prompt word user_profile_prompt of the user portrait after the message list to ensure that the model processes the history message list. The specific content of the prompt word user_profile_prompt for generating a user portrait is: Please record history[List] based on the historical conversation. Analyze the needs of the current user and generate a user portrait. The user portrait should be in the following format: [Chapter]: Extract the most matching chapter name from the course chapter. [Session]: Extract the relevant section name from the course knowledge base, ensure that the chapter name and the section name are processed separately, which can represent different levels of user interests and needs, and can provide users with more detailed and specific personalized services.

[0174] Step S423: define an LLMChain chain, which is configured to receive the user's input query, historical conversations and created chat prompt template, and pass the chat prompt template as the input prompt word to the recommendation model, so that the recommendation model generates a corresponding user portrait based on this.

[0175] Define the LLMChain class, encapsulate all necessary logic for interacting with the recommendation model, and specify that the recommended model to be called is the GLM-4-air large language model. Instantiate the LLMChain class, which is a chain implementation that uses a large language model to handle specific tasks.

[0176] The LLMChain class receives the created chat prompt template, and passes this chat prompt template as the input prompt word to the GLM-4-air model, and the GLM-4-air model generates the corresponding output based on this.

[0177] The LLMChain class also executes the main functions of the chain object through the invoke method of the user profile chain function user_profile_chain, allowing the chain to perform the configured tasks. That is, the user profile chain function use r_profile_chain receives input data from the function caller, including the user's input question query and the chat history message list List, uses the GLM-4-air model to analyze and process the problem, and generates a user profile user_profile_ans. The user profile user_profile_ans includes [Chapter] and [Session], that is, the chapters and subsections that the user is interested in (that is, the slices that the user is interested in). Among them, in order to ensure that the [Chapter] and [Session] generated by the prompt words are within the scope of the course title, the present invention will predefine it in the prompt words that call the GLM-4 large model: inform the large model of the correspondence between the course chapters and subsections, such as [1-1, 1-2, 2-1, 2-2], and ask the large model to ensure that the output chapters and subsections are within the scope.

[0178] Step S500: Integrate the knowledge base, the neural network model and the function for generating user portraits into a multimodal teaching content recommendation system, so that the multimodal teaching content recommendation system obtains question-related slices and slices of interest to the user based on the user's input questions and historical conversations, and retrieves and outputs corresponding course resources from the knowledge base. Specifically, the multimodal teaching content recommendation system uses the trained neural network model to identify the question-related slices obtained based on the user's input questions, and uses the function for generating user portraits to obtain slices of interest to the user based on the user's input questions and historical conversations.

[0179] Therefore, one of the functions of the multimodal teaching content recommendation system is to output basic search answers based on the user's input questions and the knowledge base. Specifically, the multimodal teaching content recommendation system uses a trained neural network model to identify the relevant slices (i.e., relevant chapters and subsections) of the questions according to the user's input questions. According to the relevant slices of the questions, the course materials under the corresponding slices are retrieved and output from the knowledge base, including text, pictures, and video materials. When preprocessing the textbook text, pictures, and course videos, the text, pictures, and course videos have been sorted into the knowledge base according to each subsection under each chapter, namely, the course text knowledge base base_for_textbook, the picture knowledge base base_for_pics, and the video knowledge base base_for_video. The slices (i.e., chapters and subsections) of the relevant questions are successfully located through the Transformer-Encoder model. Therefore, the text, picture, and video information of the corresponding slices can be directly obtained from the knowledge base based on the slices and GLM-4-air, that is, the text, picture, and video information of the entire subsection can be directly obtained. Among them, the prompts based on Python calling the GLM-4-air large model include: obtaining category labels based on user input questions and neural network models, and requiring the generation of text information, image links, and video links corresponding to the category labels.

[0180] Another function of the multimodal teaching content recommendation system is personalized recommendation. After the multimodal teaching content recommendation system extracts the user portrait through the user portrait generation function, it calls the knowledge base search service MilvusKBService through the personalized recommendation module to retrieve the documents most relevant to the user portrait. According to the user preference information (preferred text, pictures, or videos) and the user portrait, it searches for similarity of documents from the course text knowledge base base_for_textbook, the picture knowledge base base_for_pics, and the video knowledge base base_for_video, and returns the top 3 most relevant files from the knowledge base. This not only supplements the relevant knowledge points originally retrieved for users through the Transformer-Encoder architecture, but also customizes the recommended content of relevant knowledge for users based on the user portrait.

[0181] Therefore, step S500 specifically includes: defining an asynchronous function recommend_base_chat(), and using the asynchronous function recommend_base_chat() to integrate the knowledge base, the neural network model and the function of generating a user portrait, so that the asynchronous function recommend_base_chat() enables the multimodal teaching content recommendation system to use the trained neural network model to identify the question-related slices and the slices of interest to the user based on the user's input questions and historical conversations, and retrieve and output the corresponding course resources from the knowledge base. This design pattern enables the function to effectively process inputs, while using the GLM-4-air large language model to generate high-quality answers and recommendations without blocking the main execution thread, thereby improving user experience and system performance. In the present invention, the asynchronous function generates answers and recommendations for users based on user preference information (preferred text, pictures, or videos) in combination with the conversation context and the data in the knowledge base by calling the API of GLM-4-air.

[0182] Table 1: The form, processing and input method of multiple parameters of asynchronous functions

[0183] The asynchronous function recommend_base_chat() is a multi-parameter API interface that receives multiple input parameters, including the question query entered by the user, the user ID number user_id, the conversation ID number conversation_id, the conversation name conversation_name, the knowledge base name knowledge_base_name, and the user history conversation history. The form, processing process, and input method of the multiple parameters of the asynchronous function recommend_base_chat() are shown in Table 1 above.

[0184] For the large language model GLM-4-air, its output is the generated text of the large language model. This text is a combination of the category labels of the question-related slices obtained by identifying the user's input questions based on the Transformer-Encoder model above, and the course text, pictures and video information are obtained from the knowledge base. The integrated output is based on the relevant course questions asked by the user, providing users with output that is easy to understand and read. Set the parameter stream = bool of the asynchronous function recommend_base_chat() to trigger the asynchronous data processing process, and process the streaming data based on Python's asynchronous iterator async for. The purpose of this setting is to achieve streaming output. The system will gradually return the generated content fragments to the user, instead of waiting for all content to be generated and returned at once.

[0185] On the other hand, the present invention provides a multimodal teaching content recommendation system, which is established using the method for establishing a multimodal teaching content recommendation system described above.

[0186] like Figure 1 As shown, on the other hand, the present invention provides a method for using a multimodal teaching content recommendation system, which includes:

[0187] Step A1: providing a multimodal teaching content recommendation system;

[0188] Step A2: Using the trained neural network model to identify the relevant slices of the questions according to the user's input questions, and retrieving and outputting the course resources (including course text, pictures and video materials) of the corresponding slices from the knowledge base;

[0189] Step A3: Use the function to generate user portraits to implement a content-based recommendation algorithm, obtain the slices that the user is interested in based on the user's input questions and historical conversations, and retrieve and output the recommended links of the course resources of the corresponding slices from the knowledge base.

[0190] The above is only a preferred embodiment of the present invention, and is not intended to limit the scope of the present invention. The above embodiment of the present invention can also be modified in various ways. That is, all simple, equivalent changes and modifications made according to the claims and the description of the present invention fall within the scope of protection of the claims of the present invention. The contents not described in detail in the present invention are all conventional technical contents.< / sep> < / sep> < / unk> < / unk> < / sep> < / sep> < / sep>

Claims

1. A method for establishing a multimodal teaching content recommendation system, characterized in that: include: Step S100: preparing course resources; Step S200: constructing a knowledge base including a plurality of slices according to the course resources and generating category labels for identifying the slices; At the same time, prepare training and test sets according to each slice, build vocabulary, and perform text encoding on the training and test sets; Step S300: using the result of text encoding as an input sample, training the encoder architecture of the Transformer to obtain a trained neural network model, wherein the neural network model is used to identify the user's input question to obtain a category label of a slice related to the question; Step S400: establishing a function for generating a user portrait, wherein the function for generating a user portrait is configured to generate a user portrait based on an input question and a historical conversation, wherein the user portrait includes slices that the user is interested in; Step S500: Integrate the knowledge base, neural network model and user portrait generation function into a multimodal teaching content recommendation system, so that the multimodal teaching content recommendation system obtains question-related slices and user-interested slices based on input questions and historical conversations, and retrieves and outputs corresponding course resources from the knowledge base.

2. The method for establishing a multimodal teaching content recommendation system according to claim 1, characterized in that: The step S100 includes: Provide course materials and course videos of relevant teaching courses in pdf or markdown format as course resources; Based on gpt-4-vision-preview and the Tongyi Tingwu model, the pictures in the course materials and course videos are used to generate picture text introductions and video text summaries.

3. The method for establishing a multimodal teaching content recommendation system according to claim 1, characterized in that: The step S200 further comprises: Step S210: Slice and organize the text materials of the textbook according to chapters and subsections to form a course text knowledge base that is easy to manage and retrieve; then establish an image knowledge base and a video knowledge base, wherein the relational data in the image knowledge base and the video knowledge base include the extracted image text introduction and video text summary as well as the image address and video address; then, combine the chapters and subsections as category labels for the prediction model; Step S230: Prepare a training set and a test set according to the text content of each slice, and build a vocabulary based on the high-frequency words in the text content; wherein the training set includes the segmented text and the corresponding category labels; the test set includes test questions and the category label corresponding to each test question; and perform text encoding on the segmented text and the test questions according to the vocabulary.

4. The method for establishing a multimodal teaching content recommendation system according to claim 3, characterized in that: After step S210 and before step S230, the method further includes step S220: using a large language model to perform text balancing on the text content of each section of each chapter, and generating a summary text; and / or After step S230, the method further includes step S240: padding the training samples and test samples obtained after the text encoding to ensure that the lengths are consistent.

5. The method for establishing a multimodal teaching content recommendation system according to claim 1, characterized in that: The step S300 specifically includes: Step S320: configure and initialize the neural network model of the Transformer encoder architecture, the neural network model includes a word embedding, layer, a position encoding layer, a Transformer encoding layer and an output layer; the word embedding layer is used to map the index of each word of the input sample to a 128-dimensional vector to obtain a first word embedding tensor; the sine-cosine position encoding layer captures the relationship between words based on the first word embedding tensor and the sine-cosine position encoding to obtain a third result tensor containing position information; the Transformer encoder layer uses a multi-head attention mechanism to obtain a fourth encoder layer output tensor; the category output layer converts the output of the Transformer encoder layer into a score of the sample for each category; The third result tensor containing position information is obtained by adding the second position tensor and the first word embedding tensor corresponding to the training sample; for the position encoding tensor of the second position tensor before dimension transformation, the calculation formula for the position encoding of each dimension i is: If i is even, If i is an odd number, Among them, pos is the position index, i is the dimension index, and d_model is the dimension of word embedding; Step S330: Use the training tensor in the result of text encoding to train the neural network model, wherein the cross entropy loss function CELLoss is used as the loss function and the Adam optimizer is used for training, and a confidence threshold is set for each category. Only when the prediction probability of the category exceeds this confidence threshold, the neural network model outputs a prediction result.

6. The method for establishing a multimodal teaching content recommendation system according to claim 5, characterized in that: Before step S320, the method further includes step S310: loading training samples and test samples in the result of text encoding to obtain a training data set and a test data set, and loading the training samples to ensure a random order of the training samples in each training cycle; and / or After step S330, step S340 is also included: testing the neural network model using the test samples in the text encoding result, and evaluating the model performance of the trained neural network model according to the test results.

7. The method for establishing a multimodal teaching content recommendation system according to claim 1, characterized in that: The step S400 includes: Step S410: Establishing a user preference information extraction module, wherein the user preference information extraction module is configured to analyze and obtain user preference information based on the user's input questions and historical conversations; the user preference information includes preference for text, picture or video, and whether the current question is related to the course content; Step S420: Establish a function for generating a user portrait, wherein the function is configured to collect historical conversations and call the LLMChain chain to generate a user portrait, wherein the user portrait includes slices that the user is interested in.

8. The method for establishing a multimodal teaching content recommendation system according to claim 7, characterized in that: The function for generating a user portrait is obtained in the following way: Step S421: selecting a recommendation model; Step S422: constructing a chat prompt template, wherein the chat prompt template includes prompt words for generating a user portrait; Step S423: define an LLMChain chain, which is configured to receive the user's input questions, historical conversations and created chat prompt templates, and pass the chat prompt templates as input prompt words to the recommendation model, so that the recommendation model generates a corresponding user portrait based on this, and the user portrait includes chapters and sections that the user is interested in.

9. A multimodal teaching content recommendation system, characterized in that: It is established by using the method for establishing a multimodal teaching content recommendation system according to one of claims 1-8.

10. A method for using a multimodal teaching content recommendation system, characterized in that: include: Step A1: providing a multimodal teaching content recommendation system, which includes a trained neural network model and a function for generating a user profile; Step A2: Using the trained neural network model to identify the relevant slices of the questions input by the user, and retrieving and outputting the course resources of the corresponding slices from the knowledge base; Step A3: Use the function to generate user portraits to implement a content-based recommendation algorithm, obtain the slices that the user is interested in based on the user's input questions and historical conversations, and retrieve and output the recommended links of the course resources of the corresponding slices from the knowledge base.

Images