A method and system for generating summary of smart classroom lectures based on large language model

Abstract

The present invention relates to the field of speech recognition, and specifically discloses a smart classroom lecture summary generation system based on a large language model, including: a speech acquisition module for acquiring the voice information of a teacher's classroom lecture; a speech recognition module for recognizing the voice information as text information; a text error correction module for identifying the erroneous sentences in the text information according to a preset text error library and in combination with the context, and performing error correction and replacement on the erroneous part to form secondary text information; a topic cutting module for performing topic cutting on the sentences in the secondary text information according to a sentence relevance algorithm to form several subtopics; the sentence relevance algorithm includes the following contents: first extracting the words in the two sentences respectively, and then grouping the words in the two sentences one by one in turn to form a word pair. The technical solution of the present invention can be used to perform consistent summary recognition for long texts (about 10,000 words) in a specific field.

Description

Technical Field

[0001] The present invention relates to the field of speech recognition, and in particular to a method and system for generating smart classroom lecture summaries based on a large language model. Background Art

[0002] The digital transformation of universities is urgent. Educators are actively exploring how to leverage digital technologies and internet platforms to improve teaching and learning processes, enhancing educational quality and efficiency. It's widely believed that providing students with information support beyond the 45-minute class time can help them review and consolidate classroom knowledge. Compared to watching recorded lectures or self-studying textbooks, PowerPoint presentations, and other materials, this approach can save significant time and effort and significantly improve learning efficiency.

[0003] Large language models (LLMs) are deep learning models trained on massive amounts of text data. They not only generate natural language text but also deeply understand the meaning of text and handle various natural language tasks such as text summarization, question-answering, and translation. Currently, large language models can be used directly to generate summaries. In smart classroom scenarios, all that is required is to convert classroom lecture audio into text, input the text into the LLMs, and finally generate summary text using instructions.

[0004] However, directly using LLMs to generate summaries is not very effective. LLMs rely on the accuracy of text prompts. Undergraduate classes usually last 40-50 minutes, and each course lasts 2-4 classes per week. The text generated by the speech of each class is about 10,000 words, which is relatively long. The prompts directly used to generate summaries for LLMs are vague and numerous, resulting in low accuracy of the generated summaries. In addition, the summaries generated from the same text can be quite different and lack consistency.

[0005] In addition, because LLMs are general language models and lack expertise in specific fields, if the teacher teaches multiple topics or knowledge points in a class, LLMs will not be able to accurately identify the topic or knowledge point to which the text belongs. The generated summary will have low accuracy and blurred boundaries, resulting in information omissions and distortion, and ignoring some important information that students need in the text, such as certain key numbers, concept definitions, words, etc.

[0006] Therefore, there is an urgent need for a smart classroom lecture summary generation method and system based on a large language model that can perform consistent summary recognition for long texts (about 10,000 words) in a specific field. Summary of the Invention

[0007] The present invention provides a method and system for generating smart classroom lecture summaries based on a large language model, which can perform consistent summary recognition for long texts (about 10,000 words) in specific fields.

[0008] In order to solve the above technical problems, this application provides the following technical solutions: a smart classroom lecture summary generation system based on a large language model, comprising:

[0009] Voice acquisition module, used to obtain the voice information of teachers’ classroom lectures;

[0010] A speech recognition module, used to recognize speech information into text information;

[0011] The text error correction module is used to identify erroneous sentences in text information based on a preset text error library and combined with the context, and to correct and replace the erroneous parts to form secondary text information;

[0012] The topic cutting module is used to perform topic cutting on the sentences in the secondary text information according to the sentence relevance algorithm to form a number of subtopics; the sentence relevance algorithm includes the following contents: first, the words in the two sentences are extracted respectively, and then the words in the two sentences are grouped one by one in turn to form a word pair, and all the word pairs form a set A, and then the two words in each word pair are encoded to form two vectors, and the information content of each pair of words is calculated, and the cosine similarity is multiplied by the minimum information content of each pair of words, and the sum of all such word pairs is calculated to calculate the overall semantic relevance in the two sentences as semantic similarity. According to the semantic similarity being lower than a preset recognition threshold, the topic boundary is determined as block pairs Ri and Ri+1, and each block pair is the same topic;

[0013] The sub-abstract generation module is used to input each sub-topic into LLMs to generate the corresponding sub-abstract;

[0014] The summary generation module is used to organize the sub-summaries according to a preset recognition order to form the final classroom teaching summary.

[0015] The basic scheme and its beneficial effects are as follows: In this invention, the voice acquisition module captures the teacher's voice information during classroom lectures, then the voice recognition module converts the teacher's voice information into text information. The text error correction module then corrects and replaces errors in the text information. In other words, the system matches the text information with a pre-set text error library, and stores common text errors in the text error library in a centralized manner, facilitating subsequent correction.

[0016] Because secondary text is still long, direct summary recognition is bound to result in various errors. Therefore, the topic segmentation module segments the secondary text into subtopics suitable for existing LLMs. LLMs are then used to identify sub-summaries, which are then organized sequentially to form the final lecture summary.

[0017] In the above process, it is more important to ensure the accuracy of the secondary text information cutting to ensure that the sub-topics can fit into LLMs. In the present invention, a sentence association algorithm designed independently is used to extract the words in the two sentences respectively, and then the words in the two sentences are grouped one by one to form a word pair. All word pairs form a set A, and then the two words in each word pair are encoded to form two vectors, and the information content of each pair of words is calculated. The cosine similarity is multiplied by the minimum information content of each pair of words, and all such word pairs are summed to calculate the overall semantic association in the two sentences, that is, the semantic similarity. If the semantic similarity is lower than the recognition threshold, segmentation is performed; if not, clustering is performed into the same sub-topic. The recognition threshold of speech similarity is the evaluation standard for the accuracy of sub-topic cutting. In the present invention, teachers or students can preset or adjust it according to needs, which can better fit the specific field in which the teacher or student is located. Under the premise of a certain recognition threshold, the consistency of the classroom lecture summary can be guaranteed.

[0018] This invention can improve students' learning efficiency and comprehension. By summarizing and concluding classroom content, students can more quickly grasp key knowledge and better understand the logical structure and internal connections of course content, helping them better integrate and apply knowledge, thereby improving their academic performance and abilities. It can also help teachers better understand students' learning progress and adjust teaching methods and content in a timely manner to better meet students' learning needs.

[0019] The inventors conducted teaching experiments, distributing the generated lecture summaries to teachers and students after class and conducting learning effectiveness tests before the next class. The results showed that the classes that received the summary intervention had better learning outcomes than those that did not. Furthermore, the generated lecture summaries can help teachers reflect on the clarity of the teaching logic and the appropriateness of the lecture speed, thus providing teachers with classroom feedback.

[0020] In summary, the present invention uses the topic segmentation module to classify and segment long text information according to a preset recognition threshold, and then uses LLMs to perform summary recognition, which can ensure the consistency of summary recognition of long texts (about 10,000 words) in specific fields.

[0021] Furthermore, it also includes an error correction database update module for receiving error correction information input by the user and updating the text error library; the error correction information includes error correction conditions, error correction objects and error correction results; the text error correction module is also used to perform a full-text search on the text information according to the error correction objects to obtain several suspected objects, and then judge whether the suspected objects meet the error correction conditions. If so, the suspected objects are replaced with the error correction results; if not, the process ends.

[0022] In this way, the text error library can be updated according to the teacher's personal characteristics to ensure the accuracy of recognition.

[0023] Furthermore, the topic segmentation module is also used to receive directory data input by the user, and then form an initial recommended recognition threshold based on the key classification of the directory data; the directory data includes image data and text data, and if it is image data, OCR recognition is performed as text data.

[0024] This approach can help users quickly determine the appropriate recognition threshold.

[0025] Furthermore, the speech recognition module is also used to identify the loudness and audio in the speech information during the process of recognizing the speech information into text information. If the loudness and audio of a part of the speech information segment and its adjacent segments exceed a preset threshold, the speech information segment is marked as a key segment and highlighted in the final classroom teaching summary.

[0026] Using this method can help students find the key points quickly.

[0027] Furthermore, the speech recognition module is also used to identify the timbre in the speech information, and mark the speech segments in which the proportion of the same timbre in the speech information exceeds a preset threshold as teacher's speech, and mark the speech segments with a timbre different from the teacher's speech as interactive speech; if there is interactive speech in the speech information recognition process, it is identified whether there are preset keywords in the teacher's speech after the interactive speech. If so, the interactive speech recognition and the teacher's speech are continuously recognized as text information; if not, the recognition of the interactive speech is abandoned.

[0028] In this way, when teachers and students interact in class, they can use preset keywords to determine whether to include students' speeches in text information, which can ensure that no abnormal information appears in the text information and ensure the accuracy of the classroom lecture summary.

[0029] Furthermore, the summary generation module is further configured to receive a recognition order input by a user, wherein the recognition order is defaulted to a time order, and when a recognition order input by a user exists, the summary generation module is configured to sort the information according to the recognition order input by the user.

[0030] This approach makes it easier for users to adjust the order and improves user experience.

[0031] A method for generating summaries of smart classroom lectures based on a large language model, including the following:

[0032] S1. Voice acquisition step, obtaining voice information of the teacher's classroom lecture;

[0033] S2 speech recognition step, the voice information is recognized as text information;

[0034] S3 text error correction step, according to the preset text error library and combined with the context to identify the error statement in the text information, and the error part of the error correction replacement, forming a secondary text information;

[0035] S4. A topic cutting step, performing topic cutting on the sentences in the secondary text information according to a sentence relevance algorithm to form a number of subtopics; the sentence relevance algorithm includes the following contents: first extracting the words from the two sentences respectively, then grouping the words in the two sentences one by one in turn to form a word pair, and all the word pairs form a set A, then encoding the two words in each word pair to form two vectors, and calculating the information content of each pair of words, multiplying the cosine similarity by the minimum information content of each pair of words, and summing all such word pairs to calculate the overall semantic relevance in the two sentences as semantic similarity, and determining the topic boundary as block pairs Ri and Ri+1 based on the semantic similarity being lower than a preset recognition threshold, and each block pair is the same topic;

[0036] S5. Sub-abstract generation step: input each sub-topic into LLMs to generate the corresponding sub-abstract;

[0037] S6. Summary generation step: arranging the sub-summaries according to a pre-set recognition order to form the final classroom teaching summary. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 This is a module block diagram of Example 1 of a smart classroom lecture summary generation system based on a large language model. DETAILED DESCRIPTION

[0039] The following is further described in detail through specific implementation methods:

[0040] Example 1

[0041] A smart classroom lecture summary generation system based on a large language model (such as Figure 1 shown), including:

[0042] A voice acquisition module (in this embodiment, it can be a device such as a pickup or microphone that directly converts sound signals into electrical signals; in other embodiments, it can also be an information receiving module that directly receives recording files) is used to obtain the voice information of the teacher's classroom lectures;

[0043] A speech recognition module (in this embodiment, for cost considerations, the existing mature iFlytek speech recognition model is directly selected for implementation. Of course, there are actually open source recognition models that can be used directly), which is used to recognize speech information as text information;

[0044] The text error correction module is used to identify erroneous sentences in text information based on a preset text error library and combined with the context, and to correct and replace the erroneous parts to form secondary text information;

[0045] The topic cutting module is used to perform topic cutting on the sentences in the secondary text information according to the sentence relevance algorithm to form a number of subtopics; the sentence relevance algorithm includes the following contents: first, the words in the two sentences are extracted respectively, and then the words in the two sentences are grouped one by one in turn to form a word pair, and all the word pairs form a set A, and then the two words in each word pair are encoded to form two vectors, and the information content of each pair of words is calculated, and the cosine similarity is multiplied by the minimum information content of each pair of words, and the sum of all such word pairs is calculated to calculate the overall semantic relevance in the two sentences as semantic similarity. According to the semantic similarity being lower than a preset recognition threshold, the topic boundary is determined as block pairs Ri and Ri+1, and each block pair is the same topic;

[0046] The sub-abstract generation module is used to input each sub-topic into LLMs to generate the corresponding sub-abstract;

[0047] The summary generation module is used to sort the sub-summaries according to a preset recognition order (in this embodiment, the time order is selected) to form a final classroom teaching summary.

[0048] In specific use: First, a microphone is installed in the classroom, and then the teacher conducts the class as normal. The voice information is converted into text information through the speech recognition module. Then, the text error correction module corrects and replaces the errors in the text information. The so-called text database is essentially more of a replacement rule for common errors. Common text errors are stored in the text error library in the form of items for future use. For example, if the teacher's Mandarin is not standard, they may confuse "h" and "f". If there is a clear phrase that cannot be formed and an alternative phrase exists, the incorrect word is replaced. Finally, the error correction results in secondary text information.

[0049] The resulting secondary text information is then segmented based on sentence-by-sentence relationships using a sentence relevance algorithm to form subtopics. Specifically, the sentence relevance algorithm extracts the words from each sentence and groups them one by one, forming word pairs. All word pairs form a set A. The two words in each word pair are then encoded to form two vectors. The information content of each word pair is calculated, and the cosine similarity is multiplied by the minimum information content of each word pair. The sum of all such word pairs is then calculated to calculate the overall semantic relevance between the two sentences, i.e., the semantic similarity. If the semantic similarity is below the recognition threshold, segmentation is performed; otherwise, the sentences are clustered into the same subtopic.

[0050] Next, each subtopic is input into LLMs to generate a corresponding sub-abstract. Finally, the sub-abstracts are sorted according to a preset recognition order (a simple chronological order is selected in this embodiment) to form the final classroom teaching summary.

[0051] This embodiment also discloses a method for generating a summary of a smart classroom lecture based on a large language model, including the following contents:

[0052] S1. Voice acquisition step, obtaining voice information of the teacher's classroom lecture;

[0053] S2 speech recognition step, the voice information is recognized as text information;

[0054] S3 text error correction step, according to the preset text error library and combined with the context to identify the error statement in the text information, and the error part of the error correction replacement, forming a secondary text information;

[0055] S4. A topic cutting step, performing topic cutting on the sentences in the secondary text information according to a sentence relevance algorithm to form a number of subtopics; the sentence relevance algorithm includes the following contents: first extracting the words from the two sentences respectively, then grouping the words in the two sentences one by one in turn to form a word pair, and all the word pairs form a set A, then encoding the two words in each word pair to form two vectors, and calculating the information content of each pair of words, multiplying the cosine similarity by the minimum information content of each pair of words, and summing all such word pairs to calculate the overall semantic relevance in the two sentences as semantic similarity, and determining the topic boundary as block pairs Ri and Ri+1 based on the semantic similarity being lower than a preset recognition threshold, and each block pair is the same topic;

[0056] S5. Sub-abstract generation step: input each sub-topic into LLMs to generate the corresponding sub-abstract;

[0057] S6. Summary generation step: arranging the sub-summaries according to a pre-set recognition order to form the final classroom teaching summary.

[0058] Example 2

[0059] Compared with Example 1, the only difference is that the topic cutting module is also used to receive directory data input by the user, and then form an initial recommended recognition threshold based on the key point classification of the directory data; the directory data includes image data and text data, and if it is image data, OCR recognition is performed as text data.

[0060] In specific use, users can use the text data or image data of the catalog to identify the number of key points in the catalog. This method can be used as a reference for the recognition threshold.

[0061] Example 2

[0062] Compared with Example 1, the only difference is that the speech recognition module is also used to identify the loudness and audio in the speech information during the process of recognizing the speech information as text information. If the loudness and audio of a part of the speech information segment and its adjacent segments exceed a preset threshold, the speech information segment is marked as a key segment and highlighted in the final classroom teaching summary.

[0063] When used specifically: the voice file is not sent directly to iFlytek's speech recognition model. Instead, the open source speech recognition model needs to be optimized. Usually, the open source speech recognition model splits the voice information into the smallest units, generally called phonemes (similar to the way of distinguishing initials and finals), and then determines the recognized text information based on the combination of phonemes. In this embodiment, the intensity (i.e. sound intensity) recognition of a phoneme is essentially added. The specific logic is to set a standard sound intensity (the average sound intensity of the first 3 minutes can be used as the standard sound intensity, and after the peak exceeds 30% of the standard sound intensity, it is identified as a key segment). After it obviously exceeds the threshold, the text information near the phoneme is marked as a key segment.

[0064] Example 3

[0065] Compared with Example 2, the only difference is that the speech recognition module is also used to recognize the timbre in the speech information, and mark the speech segments in which the proportion of the same timbre in the speech information exceeds a preset threshold as teacher's speech, and mark the speech segments with a timbre different from the teacher's speech as interactive speech; if there is interactive speech in the speech information recognition process, it is identified whether there is a preset keyword (the key in this embodiment is "right" or "yes") in the teacher's speech after the interactive speech. If so, the interactive speech recognition and the teacher's speech are continuously recognized as text information; if not, the recognition of the interactive speech is abandoned.

[0066] In specific use, the essence is to process whether the student's voice information is recognized as text information when the teacher and students interact. In this embodiment, the distinction is made by timbre. The so-called timbre is the waveform characteristics of the sound waves emitted by each person. Similar to the Fourier transform, sound waves of different frequencies can be roughly distinguished. There are already technologies in the prior art that can recognize the voices of different speakers, which will not be described here. The biggest difference in this embodiment is that after the student's voice is recognized, the teacher's voice information is immediately identified to see whether there are words that affirm the previous student's speech. If so, the student's voice information is recognized; if not, it is abandoned. This method can avoid recognizing incorrect information and then correcting it, and prevent other students from learning incorrect knowledge due to taking it out of context, thereby ensuring the compactness and accuracy of the classroom lecture summary.

[0067] Example 4

[0068] Compared with Example 3, the only difference is that it also includes an error correction database update module for receiving error correction information input by the user and updating the text error library; the error correction information includes error correction conditions, error correction objects and error correction results; the text error correction module is also used to perform a full-text search on the text information according to the error correction objects to obtain several suspected objects, and then judge whether the suspected objects meet the error correction conditions respectively. If they meet the conditions, the suspected objects are replaced with the error correction results; if not, the process ends; the summary generation module is also used to receive the recognition order input by the user, and the recognition order defaults to the time order. When there is an recognition order input by the user, it is sorted according to the recognition order input by the user.

[0069] In specific use, the database update module is more to help teachers or students adjust the text error library to be more accurate and personalized.

[0070] The above are only embodiments of the present invention. The invention is not limited to the fields involved in this implementation case. Common knowledge such as the known specific structures and characteristics in the scheme is not described in detail here. Ordinary technicians in the relevant field are aware of all common technical knowledge in the technical field to which the invention belongs before the application date or priority date, can obtain all existing technologies in the field, and have the ability to apply conventional experimental means before that date. Ordinary technicians in the relevant field can improve and implement this scheme in combination with their own abilities under the inspiration given by this application. Some typical known structures or known methods should not become obstacles for ordinary technicians in the relevant field to implement this application. It should be pointed out that for those skilled in the art, without departing from the structure of the present invention, several variations and improvements can be made, which should also be regarded as the scope of protection of the present invention. These will not affect the effect of the implementation of the present invention and the practicality of the patent. The scope of protection required by this application shall be based on the content of its claims, and the specific implementation methods and other records in the specification can be used to interpret the content of the claims.

Claims

1. A smart classroom lecture summary generation system based on a large language model, characterized by: include: Voice acquisition module, used to obtain the voice information of teachers’ classroom lectures; A speech recognition module, used to recognize speech information into text information; The text error correction module is used to identify erroneous sentences in text information based on the preset text error library and the context, and to correct and replace the erroneous parts to form secondary text information; The topic cutting module is used to perform topic cutting on the sentences in the secondary text information according to the sentence relevance algorithm to form a number of subtopics; the sentence relevance algorithm includes the following contents: first, the words in the two sentences are extracted respectively, and then the words in the two sentences are grouped one by one in turn to form a word pair, and all the word pairs form a set A, and then the two words in each word pair are encoded to form two vectors, and the information content of each pair of words is calculated, and the cosine similarity is multiplied by the minimum information content of each pair of words, and the sum of all such word pairs is calculated to calculate the overall semantic relevance in the two sentences as semantic similarity. According to the semantic similarity being lower than a preset recognition threshold, the topic boundary is determined as block pairs Ri and Ri+1, and each block pair is the same topic; Sub-abstract generation module, used to input each sub-topic into LLMs to generate the corresponding sub-abstract; The summary generation module is used to organize the sub-summaries according to a preset recognition order to form the final classroom teaching summary.

2. The smart classroom lecture summary generation system based on a large language model according to claim 1 is characterized by: It also includes an error correction database update module for receiving error correction information input by the user and updating the text error library; the error correction information includes error correction conditions, error correction objects and error correction results; the text error correction module is also used to perform a full-text search on the text information according to the error correction objects to obtain several suspected objects, and then judge whether the suspected objects meet the error correction conditions. If so, the suspected objects are replaced with the error correction results; if not, the process ends.

3. The smart classroom lecture summary generation system based on a large language model according to claim 2 is characterized by: The topic cutting module is also used to receive directory data input by the user, and then form an initial recommended recognition threshold based on the key classification of the directory data; the directory data includes image data and text data. If it is image data, OCR recognition is performed as text data.

4. The smart classroom lecture summary generation system based on a large language model according to claim 3 is characterized by: The speech recognition module is also used to identify the loudness and audio in the speech information during the process of recognizing the speech information into text information. If the loudness and audio of a certain speech information segment and its adjacent segments exceed a preset threshold, the speech information segment is marked as a key segment and highlighted in the final classroom teaching summary.

5. The smart classroom lecture summary generation system based on a large language model according to claim 4 is characterized by: The speech recognition module is also used to identify the timbre in the speech information, and mark the speech segments in which the proportion of the same timbre in the speech information exceeds a preset threshold as teacher's speech, and mark the speech segments with a timbre different from the teacher's speech as interactive speech; if there is interactive speech in the speech information recognition process, it is identified whether there is a preset keyword in the teacher's speech after the interactive speech. If so, the interactive speech recognition and the teacher's speech are continuously recognized as text information; if not, the recognition of the interactive speech is abandoned.

6. The smart classroom lecture summary generation system based on a large language model according to claim 5 is characterized by: The summary generation module is further configured to receive a recognition order input by a user, wherein the recognition order is defaulted to a time order, and when a recognition order input by a user exists, the summary is sorted according to the recognition order input by the user.

7. A method for generating summaries of smart classroom lectures based on a large language model, comprising the following: S1. Voice acquisition step, obtaining voice information of the teacher's classroom lecture; S2 speech recognition step, the voice information is recognized as text information; S3 text error correction step, according to the preset text error library and combined with the context to identify the error statement in the text information, and the error part of the error correction replacement, forming a secondary text information; S4. A topic cutting step, performing topic cutting on the sentences in the secondary text information according to a sentence relevance algorithm to form a number of subtopics; the sentence relevance algorithm includes the following contents: first extracting the words from the two sentences respectively, then grouping the words in the two sentences one by one in turn to form a word pair, and all the word pairs form a set A, then encoding the two words in each word pair to form two vectors, and calculating the information content of each pair of words, multiplying the cosine similarity by the minimum information content of each pair of words, and summing all such word pairs to calculate the overall semantic relevance in the two sentences as semantic similarity, and determining the topic boundaries as block pairs Ri and Ri+1 based on the semantic similarity being lower than a preset recognition threshold, and each block pair is the same topic; S5. Sub-abstract generation step: input each sub-topic into LLMs to generate the corresponding sub-abstract; S6. Summary generation step: arranging the sub-summaries according to a pre-set recognition order to form the final classroom teaching summary.

Images