Video processing method and device based on large model, equipment and storage medium

Abstract

The invention provides a video processing method and device based on a large model, equipment and a storage medium, and relates to the field of artificial intelligence, in particular to the field of deep learning. The specific implementation scheme is as follows: extracting exercise information and knowledge information from a to-be-processed video; inputting the knowledge information into a preset lecture large model for processing, and generating a content text of the lecture; dividing the content text of the lecture to obtain a plurality of content blocks; and matching the exercise information with the content blocks, and inserting the exercise information into the matched content blocks to obtain a to-be-displayed file of the lecture.

Description

Technical Field

[0001] The present disclosure relates to deep learning in artificial intelligence, and more particularly to a video processing method, apparatus, device and storage medium based on a large model. Background Art

[0002] At present, the handouts of the videos that users watch during the learning process mainly rely on manual output. That is, the handout editor needs to edit the handouts according to the video content after watching the video content. Obviously, the handouts edited in this way are easily limited by personal ability, and there are problems with uneven handout quality and efficiency. Summary of the invention

[0003] The present invention provides a video processing method, device, equipment and storage medium based on a large model.

[0004] According to a first aspect of the present disclosure, a video processing method based on a large model is provided, comprising:

[0005] Identify the video to be processed to obtain exercise information and knowledge information; wherein the exercise information includes the text of the exercise in the courseware screenshot of the video to be processed; the knowledge information includes the knowledge text in the courseware screenshot of the video to be processed and the subtitle text corresponding to the audio of the video to be processed; the courseware screenshot is a screenshot of the video to be processed;

[0006] Inputting the knowledge information into a preset lecture note model to generate a content text of the lecture note, and dividing the content text of the lecture note to obtain at least one content block;

[0007] The exercises in the exercise information are inserted into at least one of the content blocks to obtain a to-be-displayed file of the lecture notes of the to-be-processed video.

[0008] According to a second aspect of the present disclosure, a large model-based video processing device is provided, comprising:

[0009] The extraction unit is used to identify the video to be processed and obtain exercise information and knowledge information; wherein the exercise information includes the text of the exercise in the courseware screenshot of the video to be processed; the knowledge information includes the knowledge text in the courseware screenshot of the video to be processed and the subtitle text corresponding to the audio of the video to be processed; the courseware screenshot is a screenshot of the video to be processed;

[0010] A first generating unit is used to input the knowledge information into a preset lecture note model to generate a content text of the lecture note, and divide the content text of the lecture note to obtain at least one content block;

[0011] The second generating unit is used to insert the exercises in the exercise information into at least one of the content blocks to obtain a to-be-displayed file of the lecture notes of the to-be-processed video.

[0012] According to a third aspect of the present disclosure, there is provided an electronic device, including:

[0013] At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method described in the first aspect.

[0014] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause the computer to execute the method described in the first aspect.

[0015] According to a fifth aspect of the present disclosure, a computer program product is provided, comprising: a computer program, wherein the computer program is stored in a readable storage medium, at least one processor of an electronic device can read the computer program from the readable storage medium, and the at least one processor executes the computer program so that the electronic device executes the method described in the first aspect.

[0016] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0018] Figure 1 is a schematic diagram according to a first embodiment of the present disclosure;

[0019] Figure 2 is a schematic diagram according to a second embodiment of the present disclosure;

[0020] Figure 3 is a schematic diagram according to a third embodiment of the present disclosure;

[0021] Figure 4 is a schematic diagram according to a fourth embodiment of the present disclosure;

[0022] Figure 5 is a schematic diagram according to a fifth embodiment of the present disclosure;

[0023] Figure 6 is a schematic diagram according to a sixth embodiment of the present disclosure;

[0024] Figure 7 A schematic block diagram of an electronic device for implementing an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0025] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0026] With the development of the Internet, more and more students tend to learn new knowledge or consolidate existing skills through video courses. At present, the generation of lecture documents for video courses mainly relies on manual methods. Teachers or teaching assistants are often required to manually watch the video content, extract key knowledge points, and take relevant screenshots from the video, and then combine the subtitle text and video content to write a detailed lecture document. However, this way of making lecture documents is easily limited by personal understanding and expression ability, and it is difficult to ensure the standardization and high quality of the documents, and it is easy to have uneven quality of lecture documents. In addition, when the video content is long and there are many knowledge points, it takes a lot of time and effort to manually generate lecture documents, which is inefficient.

[0027] In order to improve the defects of manual generation, some rule-based or template-based automatic handout generation methods have been proposed. However, these methods can only process simple text and image information, and it is difficult to accurately understand and parse the complex knowledge points and logical relationships in the video, and it is difficult to accurately extract and parse the key information in the video. Therefore, the generated handout documents are insufficient in accuracy and completeness. In addition, it is difficult for this technology to generate personalized handout documents based on students' learning progress and comprehension ability, and it cannot meet the needs of different students.

[0028] In order to overcome the above limitations, the present invention innovatively introduces advanced large model technology. Based on the powerful data processing capabilities and deep learning algorithms of large model technology, it can understand and analyze massive video content and related text information, automatically parse the knowledge points in the video course, identify key information, and deeply understand its internal logical relationships and hierarchical structures, thereby realizing the automatic generation of video handout documents. Specifically, the video processing method based on large models disclosed in the present invention utilizes the existing large language model capabilities, and realizes the generation of learning handouts through prompt words, as well as some pre-processing and post-processing rules.

[0029] Furthermore, the present disclosure not only generates a handout document containing detailed knowledge point analysis, but also cleverly integrates the corresponding video screenshots and combines the knowledge points with exercises. Among them, the screenshots can intuitively show how the knowledge points are presented in the video, helping students to better understand and remember the course content. Moreover, this combination of text and pictures also makes the handout document more vivid and interesting, which increases students' interest in learning and participation. Among them, the arrangement order of the exercises after the knowledge points can help users consolidate and check for omissions in the knowledge points after completing the study of the knowledge points.

[0030] From the perspective of practical application, the video lecture notes disclosed in this paper can not only effectively assist students in after-class review and consolidation, but also greatly reduce the workload of teachers in making lecture notes. Teachers can devote more time and energy to optimizing teaching content and innovating teaching methods, thereby improving overall teaching efficiency and quality.

[0031] The implementation of the present disclosure can be specifically completed through three functional modules, which can be a pre-processing module, a content generation module and a post-processing module.

[0032] The preprocessing module can process the image and audio of the video to be processed to obtain the courseware screenshots and subtitle texts therein. The courseware screenshots can be deduplicated courseware screenshots. Each courseware screenshot can correspond to a time period in which the courseware screenshot is displayed in the video to be processed. The preprocessing module can also correspond the courseware screenshots and the subtitle texts, and use the large model to achieve the fusion of the knowledge text and the subtitle text identified in the courseware screenshots to obtain knowledge information.

[0033] Among them, the content generation module can use the big model to classify the text information of the courseware screenshot into knowledge and questions, and obtain the exercise information and knowledge information therein. The knowledge information is fused with the subtitle text to obtain the final knowledge information. In addition, the big model can also realize the extraction of exercises in the exercise information. In addition, the big model can also realize the generation of the content text of the handout based on the knowledge information, and divide the content text into multiple content blocks. Furthermore, based on the time sequence, the exercise information can be fused with the content blocks in the handout content to obtain the final handout.

[0034] Among them, the post-processing module can use the rules to modify the content of the handouts generated in the previous step to ensure that the handout document can meet the requirements of the markdown format, and call pandoc and xelatex to render and generate word and pdf files, and finally output them as handouts that can be sent to users.

[0035] The handout generation method disclosed in the present invention can automatically generate handout documents for the video to be processed efficiently and with high quality, which completely subverts the traditional cumbersome process that relies on manual operation by teachers. In addition, the present invention is based on advanced large model technology, which can accurately parse the knowledge points in the video course, not only ensuring the standardization and high quality of the document, but also achieving a deep understanding of the knowledge points and a clear presentation of the logical relationship. In addition, the handouts generated by the present invention realize the integration of video screenshots and the interspersion of questions, making the handout documents more intuitive and vivid, greatly improving the students' learning experience and effect. The handout documents disclosed in the present invention not only effectively assist students in their after-school review, but also greatly reduce the workload of teachers, providing users with more excellent educational products and services.

[0036] The present invention provides a large-model-based video processing method, apparatus, device, and storage medium, which are applied to deep learning in the field of artificial intelligence to improve the quality and generation efficiency of lecture notes.

[0037] It should be noted that the head model in this embodiment is not a head model for a specific user and cannot reflect the personal information of a specific user. It should be noted that the two-dimensional face image in this embodiment comes from a public data set.

[0038] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0039] In order to enable readers to more deeply understand the implementation principle of the present disclosure, the following Figure 2-Figure 4 right Figure 1 The illustrated embodiment is further refined.

[0040] Figure 1 is a schematic diagram according to the first embodiment of the present disclosure, such as Figure 1 As shown, the present disclosure provides a video processing method based on a large model, the method comprising:

[0041] 101. Identify the video to be processed and obtain exercise information and knowledge information. The exercise information includes the text of the exercise in the courseware screenshot of the video to be processed. The knowledge information includes the knowledge text in the courseware screenshot of the video to be processed and the subtitle text corresponding to the audio of the video to be processed. The courseware screenshot is a screenshot of the video to be processed.

[0042] In this embodiment, first, a video to be processed can be obtained. Optionally, the video to be processed can be used to assist users in learning knowledge. Optionally, the playback content of the video to be processed can include the courseware to be played. Optionally, the video can also include a portrait of a teacher. Optionally, the audio of the video to be processed is the teacher explaining the courseware. Optionally, the courseware can be a document such as ppt, pdf, etc.

[0043] After determining the video to be processed, the exercise information and knowledge information therein can be identified from the video to be processed. The exercise information can be the exercise text obtained by identifying the screenshot of the courseware played by the video to be processed. The knowledge information can include the knowledge text obtained by identifying the screenshot of the courseware played by the video to be processed, and the subtitle text obtained by processing the audio of the video to be processed. Optionally, the knowledge information can be obtained by fusing the knowledge text and the subtitle text.

[0044] Optionally, after determining the video to be processed, the video to be processed can be screenshotted according to a preset frame rate, and adjacent similar screenshots can be merged to obtain a final courseware screenshot. The text in the courseware screenshot can be identified to obtain the exercise text and knowledge text therein. Among them, the exercise text is the exercise that appears in the courseware screenshot, and the knowledge text is the text in the courseware screenshot other than the exercise text. Optionally, the classification of the exercise text and the knowledge text can be obtained by classifying the text after identifying the text in the courseware screenshot using a preset classification model.

[0045] Optionally, a video database may also be included. The video database may contain multiple videos. The acquisition of the video to be processed may be achieved by inputting identification information such as the video ID of the video to be processed.

[0046] 102. Input the knowledge information into a preset lecture note model to generate a content text of the lecture note, and divide the content text of the lecture note to obtain at least one content block.

[0047] In this embodiment, after obtaining the knowledge information in the video to be processed according to step 101, the knowledge information can be input into a preset lecture model for processing to generate the content text of the lecture. Optionally, the processing process can include extracting titles from the knowledge information, dividing the extracted titles into levels, etc.

[0048] After obtaining the lecture content, the lecture content can also be divided to obtain multiple content blocks. Each content block can include a portion of continuous content text. Optionally, the division process of the content block can be implemented by a large model. Optionally, one content block can correspond to one knowledge point.

[0049] Optionally, in order to preserve the validity of the courseware screenshots, the large model can process the knowledge information corresponding to each courseware screenshot based on the courseware screenshots to obtain the content text of the handouts corresponding to each courseware screenshot. According to the playback order of all courseware screenshots in the video to be processed, the content text of the handouts of these courseware screenshots can be combined to obtain the content text of the handouts corresponding to the entire video to be processed. Optionally, each content block can include the content text corresponding to at least one courseware screenshot.

[0050] 103. Insert the exercises in the exercise information into at least one content block to obtain a to-be-displayed file of the lecture notes of the to-be-processed video.

[0051] In this embodiment, the exercises can be extracted from the exercise information obtained in step 101. Then, the exercises can be inserted into the content block obtained in step 102. The content block with the exercises inserted can constitute the to-be-displayed file of the lecture of the to-be-processed video. Optionally, the to-be-displayed file is a lecture document that can be output to the user for reading.

[0052] Optionally, the exercise information can be inserted into the matching content block, so as to assist the user in learning the content block and consolidating the content block through the exercises in the exercise information. Optionally, the exercise can be inserted at the end of the content block, so as to realize the handout structure of knowledge point-exercise, better assist the user in learning knowledge and consolidating knowledge through exercises.

[0053] Optionally, the matching between the exercise and the content block can be achieved based on the correlation between the exercise and the content block. Alternatively, the matching between the exercise information and the content block can also be achieved based on the time corresponding to the content block in the video to be processed and the time corresponding to the exercise in the video to be processed. Optionally, the time corresponding to the content block in the video to be processed can be determined based on the courseware screenshot included in the content block. Optionally, the time corresponding to the exercise in the video to be processed can be determined based on the courseware screenshot where the exercise is located.

[0054] In this embodiment, exercise information and knowledge information are identified from the video to be processed; the knowledge information is input into a preset lecture note model for processing to generate a content text of the lecture note; the content text of the lecture note is divided to obtain at least one content block; the exercises in the exercise information are inserted into at least one content block to obtain a final lecture note with a display file, thereby automatically generating a lecture note for the video to be processed and improving the effect of the lecture note instructions; and the present disclosure achieves more accurate placement of the exercises in the exercise information by additionally inserting the exercises in the exercise information into the lecture note content, which helps users to consolidate the knowledge points after learning the knowledge points of the content block through the exercises in the exercise information, improve learning effects, and improve the effectiveness and practicality of the lecture notes.

[0055] Based on the above embodiment, the specific process of extracting exercise information and knowledge information from the video to be processed may include:

[0056] 1011. Use a text recognition model to recognize the courseware screenshots in the video to be processed to obtain exercise information and knowledge text.

[0057] In this embodiment, after obtaining the video to be processed, the courseware screenshots therein can be obtained from the video to be processed. Optionally, the courseware screenshots can be obtained by taking screenshots of the video to be processed according to a preset frequency and removing duplicates from the screenshots. Optionally, each courseware screenshot can correspond to a time information. The time information includes the appearance time and end time of the courseware screenshot in the video to be processed.

[0058] Optionally, deduplication of the screenshot can be achieved by the similarity of the text in the screenshot. Specifically, after identifying the text information in two adjacent screenshots, the similarity of the text information of the two screenshots can be calculated. If the similarity is less than the similarity threshold, it is determined that the two screenshots are not similar, and it is determined that the two screenshots belong to two courseware screenshots. If the similarity is greater than or equal to the similarity threshold, it is determined that the two screenshots are similar, and the two screenshots belong to the same courseware screenshot. Optionally, after determining multiple consecutive and similar screenshots, a screenshot can be selected from them as a courseware screenshot. Optionally, the first of the multiple consecutive and similar screenshots, or the last one, can be used as the courseware screenshot.

[0059] After obtaining the screenshot of the courseware, the text information in the screenshot of the courseware can be obtained by text recognition. Optionally, the text recognition can be implemented by a preset text recognition model. Optionally, after obtaining the text information, the text information can be classified to obtain the exercise text and knowledge text therein. The exercise text is the exercise information.

[0060] Optionally, the courseware screenshot may have third time information. Optionally, the third time information may be used to indicate the time when the courseware screenshot appears in the video to be processed. Alternatively, the third time information may be used to indicate the time when the courseware screenshot appears and the time when it ends in the video to be processed.

[0061] 1012. Use the audio recognition model to recognize the audio of the video to be processed and obtain subtitle text corresponding to the audio.

[0062] In this embodiment, after obtaining the video to be processed, the audio of the video to be processed can also be obtained. Furthermore, the audio can be recognized using an audio recognition model to obtain the subtitle text of the audio. Optionally, the smallest unit in the subtitle text can be a sentence.

[0063] Optionally, the subtitle text may have fourth time information. Optionally, each sentence in the subtitle text may correspond to a fourth time information. Optionally, the fourth time information is the timestamp of the sentence. Optionally, the fourth time information may be used to indicate the time when the sentence in the subtitle text appears in the audio. Alternatively, the fourth time information may be used to know the time when the sentence in the subtitle text appears and the time when it ends in the audio.

[0064] 1013. The knowledge text and the subtitle text are integrated to obtain knowledge information.

[0065] In this embodiment, after obtaining the knowledge text of the courseware screenshot and the subtitle text of the audio according to step 1011, the courseware screenshot and the subtitle text can be aligned according to the third time information of the courseware screenshot and the fourth time information of the subtitle text. Optionally, after completing the alignment, the subtitle text corresponding to the courseware screenshot can be determined. Optionally, the subtitle text corresponding to the courseware screenshot can be the content of the teacher's narration during the period when the courseware screenshot is played in the video to be processed. Optionally, the subtitle text corresponding to the courseware screenshot can be the teacher's explanation content relative to the courseware screenshot. Therefore, based on the knowledge text in the courseware screenshot and the subtitle text corresponding to the courseware screenshot, the knowledge information corresponding to the courseware screenshot can be fused. Optionally, the knowledge information of all courseware screenshots can constitute the knowledge information of the video to be processed. Optionally, each piece of knowledge information can correspond to a courseware screenshot. Optionally, the time information corresponding to the knowledge information can be the third time information of the courseware screenshot.

[0066] Optionally, the fusion can be implemented using a preset fusion model. Optionally, the fusion of the knowledge text and the subtitle text can be to regenerate some text based on the knowledge text and the subtitle text, and the text is the knowledge information.

[0067] In this embodiment, by identifying the exercise information and knowledge information in the video to be processed, the automatic extraction of the exercise information and knowledge information is achieved, the extraction accuracy of the exercise information and knowledge information is improved, and a basis is provided for subsequent processing.

[0068] In one example, the specific process of fusing the knowledge text and the subtitle text to obtain the knowledge information may include:

[0069] 10121. Determine, based on the third time information of the knowledge text, a subtitle text corresponding to the fourth time information that matches the third time information.

[0070] In this embodiment, the third time information corresponding to the courseware screenshot can be determined based on the courseware screenshot where the knowledge text is located. Optionally, the third time information may include a start time and an end time. The start time is the time when the courseware screenshot begins to appear in the video to be processed, and the end time is the time when the courseware screenshot appears at that place in the video to be processed and switches to the next courseware screenshot.

[0071] Since the fourth time information of the subtitle text is the timestamp corresponding to the sentence in the subtitle text. For a courseware screenshot, the teacher usually needs to say a lot of sentences to explain the content in the courseware screenshot. Therefore, the subtitle text corresponding to the fourth time information that matches the third time information can be determined by judging whether the fourth time information is within the range of the third time information. That is, if the fourth time information indicates that the start time of the sentence in the subtitle text is within the time period indicated by the third time information, it can be determined that the fourth time information matches the third time information. That is, the sentence of the subtitle text matches the knowledge text.

[0072] Optionally, the subtitle text finally determined to correspond to the knowledge text in a courseware screenshot may be a plurality of consecutive sentences in the subtitle text.

[0073] 10122. The knowledge text and its matching subtitle text are integrated to obtain knowledge information.

[0074] In this embodiment, after the knowledge text and the subtitle text corresponding to the knowledge text are determined, the knowledge text and the subtitle text are fused. Optionally, the fusion of the knowledge text and the subtitle text can be achieved by inputting the knowledge text and the subtitle text into a preset fusion model. Optionally, knowledge information can be obtained after fusion. The knowledge information can include multiple continuous and smooth sentences. Each courseware screenshot can correspond to a piece of knowledge information. The knowledge information can correspond to third time information.

[0075] In this embodiment, by integrating the knowledge text and the subtitle text, the text on the courseware and the voice content of the teacher's explanation are integrated, which improves the effectiveness and completeness of the integrated knowledge information and provides a basis for subsequent processing.

[0076] Figure 2 is a schematic diagram according to a second embodiment of the present disclosure, Figure 1 Based on the embodiment shown, Figure 2 As shown, the present disclosure provides a video processing method based on a large model, wherein the specific process of inserting the exercises in the exercise information into the content block to obtain the to-be-displayed file of the lecture notes may include:

[0077] S201, identifying each exercise in the exercise information to obtain at least one exercise; and matching a content block corresponding to the obtained exercise from at least one content block.

[0078] In this embodiment, the exercise information may include the exercise parts in all the courseware screenshots of the video to be processed. Optionally, multiple exercises in the exercise information are not extracted separately. Therefore, after obtaining the exercise information, the exercise information can be further processed to extract the exercises from the exercise information.

[0079] Furthermore, each exercise can be matched with a content block to determine the content block that matches the exercise. Alternatively, the content block that matches the exercise can be determined based on the content similarity between the exercise and the content block. Alternatively, the content block corresponding to the exercise can be determined based on the courseware screenshot where the exercise is located and the courseware screenshot corresponding to the content block.

[0080] S202: insert the obtained exercises into the content block corresponding to the obtained exercises to obtain a file to be displayed.

[0081] In this embodiment, after determining the correspondence between the exercises and the content blocks, the exercises can be inserted into the content blocks. Optionally, the exercises can be inserted at the end of the content blocks. Optionally, a content block can correspond to a knowledge point. Optionally, the inserted exercises can form a sequence of knowledge points-exercises, so that after completing the learning of the knowledge points, the user can consolidate the knowledge points through exercises, thereby improving the user's learning efficiency.

[0082] In one example, an exercise has first time information. The first time information indicates the time when the courseware screenshot where the exercise is located appears on the video to be processed. Optionally, the first time information may include a start time and an end time. The start time is the time when the courseware screenshot appears. The end time is the time when the courseware screenshot switches to the next courseware screenshot after the first appearance. The content block has second time information. Optionally, each content block can correspond to at least one courseware screenshot. The second time information of the content block can be determined based on the third time information of the courseware screenshot corresponding to the content block.

[0083] Optionally, the exercise and the content block may be matched based on the first time information of the exercise and the second time information of the content block. Optionally, if the first time information of the exercise and the second time information of the content block have an intersection, it is determined that the exercise matches the content block.

[0084] In this embodiment, by matching the exercises and the content block, the insertion of exercises is achieved, the layout of knowledge points-exercises is ensured, and the user experience of the user when using the handout is improved.

[0085] In one example, a specific method of extracting the exercise from the exercise information may include:

[0086] First, according to the exercise sequence number in the exercise information, the exercises in the exercise information are split and processed to obtain at least one exercise.

[0087] In this embodiment, since an exercise number is usually added to the exercise in the design of the exercise, so that the exercise number can be used to correspond when the exercise is explained or the answer is provided, the exercise text in the exercise information can be split according to the exercise number to obtain at least one exercise contained therein.

[0088] Second, perform format recognition processing on the exercises in the exercise information to obtain at least one exercise.

[0089] In this embodiment, the exercises may generally include multiple choice questions, blank questions, and answer questions. For different knowledge, the question types are generally fixed. Therefore, based on the question type, the format of the exercise can be determined. Thus, based on the format, the exercise can be segmented to obtain at least one exercise. For example, for multiple choice questions, the stem and options can be extracted to obtain an exercise.

[0090] In this embodiment, the above two methods are used to extract the exercises from the exercise information, thereby improving the efficiency of obtaining the exercises.

[0091] Based on the above embodiment, the process of determining the matching relationship between exercises and content blocks may further include the following steps:

[0092] 2011. Obtain a timestamp indicated by the subtitle text corresponding to the content block, and determine time information corresponding to the content block according to the timestamp.

[0093] In this embodiment, the knowledge information for generating the content block can be determined based on the content block. Furthermore, the subtitle text contained therein can be obtained from the knowledge information. The subtitle text can include a timestamp. According to the timestamps of all subtitle texts corresponding to a content block, the time information corresponding to the content block can be generated. Since the segmentation of a content block has no direct correspondence with the courseware screenshot, there may be a situation where multiple content blocks appear in a courseware screenshot. Therefore, using the timestamp to determine the time information of the content block can be more accurate.

[0094] 2012. Determine the time information corresponding to the exercise information according to the courseware screenshot where the exercise information is located and the time information corresponding to the courseware screenshot. The courseware screenshot is a screenshot of the video to be processed. The time information of the courseware screenshot is determined according to the time when the courseware screenshot appears in the video to be processed.

[0095] In this embodiment, since the exercise information is directly extracted from the courseware screenshot, the corresponding courseware screenshot can be determined based on the exercise information. Each courseware screenshot can correspond to a time information. The time information indicates the time when the courseware screenshot appears in the video to be processed. Furthermore, the time information of the courseware screenshot can be used as the time information corresponding to the exercise information.

[0096] 2013. Determine the matching relationship between the exercise information and the content block according to the time information of the content block and the time information of the exercise information.

[0097] In this embodiment, the matching relationship between the content block and the exercise information can be determined by matching the time information of the content block and the time information of the exercise information. Optionally, the time overlap or close relationship between the exercise information and the content block can be identified by comparing and analyzing the time information, thereby determining that the content block and exercise information with time overlap or close relationship are matched.

[0098] Optionally, when the time information of the content block and the time information of the exercise information intersect, it can be determined that the content block matches the exercise information. Specifically, based on the time information of each content block, it is determined whether there is exercise information that overlaps with the time information of the content block. If the exercise information exists, it is determined that the content block and the exercise information intersect. Furthermore, it can be determined that the content block and the exercise information that have an intersection have a matching relationship. Optionally, a content block can match one or more exercise information. Optionally, if the time information of the content block overlaps with the time information of the exercise information, it means that the exercise information is explained at the same time when the content block is explained in the video to be processed.

[0099] Optionally, if there is exercise information that has no intersection with any content block, a content block with the closest time information to the exercise information may be determined as the content block matching the exercise information.

[0100] In this embodiment, the time information corresponding to a content block can be generated based on the timestamp of all subtitle texts corresponding to the content block; the time information of the courseware screenshot can be used as the time information corresponding to the exercise information; the matching relationship between the content block and the exercise information can be determined by matching the time information of the content block and the time information of the exercise information, thereby realizing automatic association between the content block and the exercise information, allowing learners to quickly locate relevant questions and content blocks when watching videos, thereby enhancing learning efficiency and effectiveness, and improving the relevance and navigability of the content blocks and exercise information in the video to be processed.

[0101] Figure 3 is a schematic diagram according to a third embodiment of the present disclosure, Figure 1 and Figure 2 Based on the embodiment shown, Figure 3 As shown, the present disclosure provides a video processing method based on a large model. In the method, the knowledge information is processed to generate the lecture content, and the specific process of dividing the lecture content into multiple content blocks may include:

[0102] 301. Input the knowledge information into a preset lecture model according to the playback order of the videos to be processed, and extract multiple knowledge topics, wherein the knowledge topic represents the title extracted from part of the knowledge information.

[0103] In this embodiment, each courseware screenshot can correspond to the knowledge information obtained by fusion. Therefore, the knowledge information of each courseware screenshot can be input into the preset lecture note model according to the playback order of the video to be processed, and multiple knowledge topics output by the preset lecture note model can be obtained. Specifically, the preset lecture note model can be a large model. Optionally, the preset lecture note model can segment the knowledge information so that the knowledge content corresponding to different knowledge topics is segmented. At the same time, for the segmented knowledge content, the preset lecture note model can extract the knowledge topic corresponding to each piece of knowledge content. Optionally, the knowledge topic can be the title corresponding to the piece of knowledge content.

[0104] 302. Input the knowledge topics into the preset lecture model according to the playback order of the videos to be processed to generate a knowledge outline, wherein the knowledge outline represents the hierarchical relationship of the knowledge topics.

[0105] In this embodiment, after obtaining the knowledge themes through step 301, these knowledge themes can be input into the preset lecture note model again according to the playback order of the video to be processed. The preset lecture note model can generate a knowledge outline according to the knowledge themes. The knowledge outline can include the hierarchical relationship of each knowledge theme. The hierarchical relationship can include the hierarchy corresponding to each knowledge theme, and the corresponding relationship between multiple knowledge themes.

[0106] 303. Input the knowledge information corresponding to each knowledge topic into the preset lecture note model to generate the explanation content of each knowledge topic.

[0107] In this embodiment, after determining the knowledge topic, the knowledge information corresponding to each knowledge topic can also be input into a preset lecture note model. The preset lecture note model will automatically generate the explanation content of each knowledge topic based on the input knowledge information. The explanation content is intended to clearly and systematically present the core concepts and details of the knowledge topic, ensuring that learners can understand each topic more quickly.

[0108] 304. According to the knowledge theme, combine the explanation content with the knowledge outline to obtain the content of the handout.

[0109] In this embodiment, after obtaining the knowledge outline and the explanation content corresponding to each knowledge theme according to the above steps, the explanation content corresponding to the knowledge theme can be inserted into the position corresponding to the knowledge theme in the knowledge outline. The process of inserting the explanation content is the process of perfecting the lecture content according to the knowledge outline, and finally the complete lecture content can be obtained, so that the lecture content finally generated can provide a comprehensive and organized learning text to help learners systematically understand and master each knowledge theme.

[0110] 305. According to the knowledge outline of the lecture content, multiple high-level knowledge topics at the highest level and their lecture contents are input into a preset segmentation model to obtain the integrated content of each high-level knowledge topic.

[0111] In this embodiment, multiple knowledge topics at the highest level of the knowledge outline can be obtained, and the knowledge topics can be named as advanced knowledge topics. Furthermore, the lecture content corresponding to each advanced knowledge topic can be obtained from the lecture content obtained in step 304. Optionally, the lecture content corresponding to the advanced knowledge topic can include the explanation content of the advanced knowledge topic, the subordinate knowledge topics of the advanced knowledge topic and their explanation content. Furthermore, each advanced knowledge topic and its corresponding lecture content can be input into a preset segmentation model. The preset segmentation model can fuse the lecture content of each advanced knowledge topic to obtain the fused content of each advanced knowledge topic.

[0112] 306. Input the integrated content of the advanced knowledge topic into the preset segmentation model according to the order of the knowledge outline to obtain multiple content blocks.

[0113] In this embodiment, each high-level knowledge topic and its fused content can be sorted according to the order of the knowledge outline. Thereafter, the fused content of all high-level knowledge topics can be input into a preset segmentation model. The preset segmentation model can merge the input fused content to obtain multiple content blocks. Each content block can include at least one high-level knowledge topic. The preset segmentation model can segment the consistency of the input information and take at least one high-level knowledge topic with high content relevance as a content block. Specifically, the preset segmentation model can calculate the relevance of the lecture content of adjacent high-level knowledge topics. If the calculated relevance is greater than the relevance threshold, it can be determined that the two high-level knowledge topics belong to the same content block. Otherwise, it can be determined that the two high-level knowledge topics belong to two content blocks respectively.

[0114] In this embodiment, by using a preset lecture model, the knowledge information is automatically generated into lecture content, thereby improving the efficiency of lecture generation. In addition, in the process of generating the lecture content, by optimizing the knowledge topics and knowledge lectures, the efficiency of the lecture content is further improved, and the quality of the lecture content is improved. In addition, by using a preset segmentation model, the lecture content is segmented into multiple content blocks, providing a good foundation for the subsequent addition of exercise information.

[0115] In one example, after obtaining the multiple knowledge topics, the knowledge topics can also be optimized by a preset knowledge topic optimization strategy. Optionally, when the optimization is implemented by a topic optimization model, the output of the optimized knowledge topic can be obtained by directly inputting the knowledge topic into the topic optimization model. The optimized knowledge topic can be a knowledge topic obtained by splitting and merging the original knowledge topic. Alternatively, when the optimization is implemented by a prediction rule, therefore, based on the above step 301, it can also include:

[0116] 3011. If the knowledge topic does not match the preset video topic of the video to be processed, the knowledge topic is deleted.

[0117] In this embodiment, a preset topic optimization model can be used to judge the input knowledge topic to determine whether it matches the preset video topic of the video to be processed. That is, whether the knowledge topic is included in the scope of the video topic. If the knowledge topic matches the video topic, there is no need to process the knowledge topic. If the knowledge topic does not match the video topic, the topic can be deleted.

[0118] 3012. If there are two knowledge topics and their corresponding knowledge information whose content matching degree is greater than a first threshold, merge the knowledge topics and the knowledge information corresponding to the knowledge topics.

[0119] In this embodiment, the knowledge information of two adjacent knowledge topics can be matched. Optionally, the matching degree of the knowledge information of the two adjacent knowledge topics can be calculated. Optionally, the matching degree can be calculated by a preset text matching algorithm. If the matching degree is greater than a first threshold, it means that the two adjacent knowledge topics have the same content, and the knowledge topics and the knowledge information corresponding to the knowledge topics can be merged.

[0120] Optionally, the knowledge information of any two non-adjacent knowledge topics may be matched. If the matching degree of the knowledge information of the two knowledge topics is greater than a first threshold, the knowledge information of the two knowledge topics may be deduplicated to obtain deduplicated knowledge information. Based on the deduplicated knowledge information, the knowledge topic may be regenerated.

[0121] In this embodiment, the accuracy and effectiveness of the knowledge subject are improved through the two further verifications of the knowledge subject.

[0122] In one example, after obtaining the knowledge outline, the knowledge outline can also be optimized by a preset knowledge outline optimization strategy. Optionally, when the optimization is implemented by an outline optimization model, the output of the optimized knowledge outline can be obtained by directly inputting the knowledge outline into the outline optimization model. The optimized knowledge outline can be a knowledge outline obtained by adjusting the hierarchy of knowledge topics in the original knowledge outline. Alternatively, when the optimization is implemented by a prediction rule, on the basis of the above step 302, it can also include:

[0123] 3021. According to the hierarchical relationship of knowledge topics in the knowledge outline, the hierarchical parameters between the explanation content of the upper knowledge topic and the explanation content of the lower knowledge topic are calculated.

[0124] In this embodiment, the hierarchical relationship of each knowledge theme can be determined from the knowledge outline. Furthermore, the upper knowledge theme and the corresponding lower knowledge theme can be obtained from the knowledge outline. Among them, the upper knowledge theme and the lower knowledge theme are knowledge themes with a hierarchical relationship. The upper knowledge theme and the lower knowledge theme can be non-adjacent knowledge themes. For example, when the level of the upper knowledge theme is 1, the level of the lower knowledge theme can be 1.1, 1.2, 1.3, etc. Among them, after 1.1, knowledge themes of levels such as 1.1.1 and 1.1.2 may exist between 1.1 and 1.2. After determining the upper knowledge theme and the lower knowledge theme, the hierarchical parameter between the two explanation contents can be calculated based on the explanation content of the upper knowledge theme and the explanation content of the lower knowledge theme. Optionally, the explanation content can be a summary and descriptive content generated based on the knowledge content of the knowledge theme that is more convenient for users to understand. Optionally, the hierarchical parameter can include relevance indicators, priority indicators, etc. The relevance index may be used to indicate the relevance of the knowledge content of the upper knowledge topic and the lower knowledge topic. The priority index may be used to indicate the probability that the upper knowledge topic is at the upper level of the lower knowledge topic.

[0125] 3022. If the hierarchical parameter is less than the second threshold, redetermine the hierarchical relationship between the upper knowledge topic and the lower knowledge topic.

[0126] In this embodiment, after the hierarchical parameter is calculated according to step 3021, the hierarchical parameter can be compared with the second threshold value. If the hierarchical parameter is greater than or equal to the second threshold value, it means that the upper knowledge subject and the lower knowledge subject meet expectations. If the hierarchical parameter is less than the second threshold value, it means that there is a problem in the setting of the upper knowledge subject and the lower knowledge subject. Therefore, when the hierarchical parameter is less than the second threshold value, the hierarchical relationship between the upper knowledge subject and the lower knowledge subject can be re-determined.

[0127] Optionally, when the level parameter includes multiple numerical values, it may correspond to multiple second thresholds.

[0128] Optionally, the specific process of re-determining the hierarchical relationship between the upper knowledge subject and the lower knowledge subject may include:

[0129] First, determine whether the first correlation between the upper knowledge topic and the lower knowledge topic is less than the second threshold. If the first correlation is less than the second threshold, it means that the upper knowledge topic does not contain the lower knowledge topic. At this time, the second correlation between the lower knowledge topic and the upper knowledge topic of the upper knowledge topic can be further calculated. If the second correlation is greater than or equal to the second threshold, the level of the lower knowledge topic is modified so that the lower knowledge topic has the same level as the upper knowledge topic. Otherwise, if the second correlation is less than the second threshold, the correlation between the lower knowledge topic and the knowledge topic of the upper layer is calculated. By analogy, if the third correlation obtained after the correlation calculation between the lower knowledge topic and the knowledge topic of the uppermost layer is less than the second threshold, the level of the lower knowledge topic is modified to the same level as the knowledge topic of the uppermost layer.

[0130] 3023. Update the knowledge outline based on the redefined hierarchical relationship.

[0131] In this embodiment, after determining that the level of the knowledge subject has changed, the knowledge outline can be updated according to the changed knowledge subject to obtain an updated knowledge outline.

[0132] In this embodiment, the effectiveness of the lecture notes is improved by optimizing the knowledge outline.

[0133] In one example, the specific process of segmenting multiple high-level knowledge topics to obtain multiple content blocks may include:

[0134] 3061. According to the knowledge outline, calculate the content relevance of the integrated content of two adjacent high-level knowledge topics.

[0135] In this embodiment, the order of high-level knowledge topics can be determined according to the knowledge outline, and two adjacent high-level knowledge topics can be obtained therefrom. Then, appropriate analysis tools or algorithms can be used to calculate the content relevance of the fusion content of the two adjacent high-level knowledge topics. For example, the subject keywords, concept overlap, information complementarity, etc. of the two fusion contents can be analyzed to obtain the final content relevance.

[0136] 3062. If the content relevance of two adjacent high-level knowledge topics is greater than or equal to a third threshold, it is determined that the two knowledge topics belong to the same content block.

[0137] In this embodiment, after the content relevance is calculated according to step 3061, the content relevance can be compared with the third threshold. If the content relevance is greater than or equal to the third threshold, it means that the two adjacent high-level knowledge topics have a high relevance and belong to the same content block.

[0138] 3063. If the content relevance of two adjacent high-level knowledge topics is less than a third threshold, it is determined that the two knowledge topics belong to two content blocks.

[0139] In this embodiment, if the content relevance is determined to be less than the third threshold value in step 3062, it can be determined that the relevance of the two adjacent high-level knowledge topics is low, and it can be determined that the two adjacent high-level knowledge topics belong to two content blocks. That is, among the two adjacent high-level knowledge topics, the high-level knowledge topic arranged in front according to the knowledge outline is the last high-level knowledge topic belonging to the previous content block. And the high-level knowledge topic arranged in the back according to the knowledge outline is the first high-level knowledge topic belonging to the next content block.

[0140] Figure 4 is a schematic diagram according to a fourth embodiment of the present disclosure, Figures 1 to 3 Based on the embodiment shown, Figure 4 As shown, the present disclosure provides a specific implementation of a video processing method based on a large model, and the specific implementation process can be divided into three stages, which can correspond to three functional modules respectively. The three functional modules can respectively include a preprocessing stage, a content generation stage, and a post-processing stage. The three modules corresponding to the three stages can respectively be a preprocessing module, a content generation module, and a post-processing module.

[0141] 401. Execute a preprocessing module in the preprocessing stage. The preprocessing module mainly processes the input data required for generating handouts, including the subtitle text of the video to be processed, the courseware screenshots, and the text information of the courseware screenshots, wherein the content of the input large model is the subtitle text and the text information of the courseware screenshots.

[0142] 4011. Input is a video ID, and the video to be processed is queried through the video ID. Then, based on the video to be processed, the subtitle text and the courseware screenshot are obtained. The subtitle text may also include a timestamp. The courseware screenshot may also include a timestamp.

[0143] 4012. De-duplicate and merge adjacent courseware screenshots whose content similarity meets a certain threshold. That is, only one of multiple similar courseware screenshots is retained. Based on the multiple similar courseware screenshots, the time information of the courseware spinous process can be generated according to its minimum timestamp and maximum timestamp. Since the courseware screenshots are extracted from the video to be processed, there may be duplicate courseware screenshots. The sliding window algorithm can be used for intelligent screening to identify and merge adjacent courseware with highly similar text information, so as to effectively remove redundancy and ensure the refinement and accuracy of information.

[0144] 4013. According to the time information of the courseware screenshot, all subtitle texts within this time range are circled to achieve alignment between the courseware screenshot and the subtitle text. Each page of the courseware screenshot covers a time period in the video. The subtitle text in this time period is the subtitle text corresponding to the courseware.

[0145] 4014. Use the big model to fuse the text information of the courseware screenshot and the corresponding subtitle text. After obtaining the correspondence between the courseware screenshot and the subtitle text, the text information of the courseware screenshot can be extracted and input into the big model together with the subtitle text for information fusion to obtain complete classroom knowledge information. Because the content in the courseware screenshot is generally prepared in advance by the teacher and contains relatively complete written information. When the teacher explains the content involved in the courseware screenshot on this page, it may include additional understanding and key points. Therefore, only by combining the two can we have relatively complete classroom knowledge information.

[0146] 402. Execute a content generation module in the content generation phase.

[0147] 4021. Use the big model to classify the text information of the courseware screenshots to obtain exercise information and content information. The content information can be integrated with the subtitle text to obtain knowledge information. According to the content structure of the video to be processed and the key needs of students, the content can be simply divided into two categories, one is exercise information and the other is knowledge information. In order to ensure the quality and completeness of these two pieces of content, they need to be processed differently.

[0148] 4022. Use the big model to process the knowledge information and generate the lecture content. Input the text classified as knowledge information into the big model to generate hierarchical knowledge content and obtain the lecture content.

[0149] 4023. Use the big model to extract questions from the exercise information. Input the text information of the courseware screenshot classified as questions into the big model to extract the question content.

[0150] 4024. Sort the knowledge information and exercise information according to the timestamp to ensure that the order is consistent with the teacher's explanation order, restore the original explanation logic of the video to be processed, and generate the final handouts.

[0151] 403. In the post-processing stage, a post-processing module is executed. The post-processing module mainly uses the lecture content obtained by the content generation module to perform legalization processing and render and generate lecture files.

[0152] 4031. Correct the format of the handout using preset rules. The format is markdown. The rules specifically include: adding line breaks before and after the markdown image logo; verifying whether the image name generated by the model actually exists; if multiple identical image logos are generated consecutively, only the last one is retained; adding line breaks before and after the secondary title; using $ to include latex formulas, etc. Since the present disclosure mainly uses a large model for content generation and adopts the markdown format, the content generated by some large models may not meet the requirements of markdown format rendering. Therefore, in this step, according to the image rendering rules, formula rendering rules, etc., the content is legalized so that it can correctly render and generate the file in the next step.

[0153] 4032. Use pandoc to render markdown files into word and pdf formats. The present disclosure mainly uses pandoc to generate files in pdf and word formats, where the generation of pdf files includes two steps. First, the markdown is converted into a tex format file, and then the tex file is rendered into a pdf file using xelatex. The advantage of this is that different tex templates can be customized according to needs, and finally various styles of handout files are generated.

[0154] In summary, through the pre-processing module, we can achieve the generation of high-quality courseware screenshots and text content, and integrate them with subtitle text to achieve a comprehensive description of classroom content, and obtain a more complete effect of the video content to be processed, which provides a basis for subsequent processing. Furthermore, through the content generation module and the post-processing module, the accurate and efficient generation of handouts is achieved, and the comprehensiveness and vividness of the handouts are ensured, which improves the efficiency and auxiliary effect of the handouts.

[0155] Figure 5 is a schematic diagram according to a fifth embodiment of the present disclosure, Figure 5 As shown, the present disclosure provides a video processing device 500 based on a large model, the device comprising:

[0156] The extraction unit 501 is used to identify the video to be processed and obtain exercise information and knowledge information; wherein the exercise information includes the text of the exercises in the courseware screenshot of the video to be processed; the knowledge information includes the knowledge text in the courseware screenshot of the video to be processed and the subtitle text corresponding to the audio of the video to be processed; the courseware screenshot is a screenshot of the video to be processed.

[0157] The first generating unit 502 is used for inputting the knowledge information into a preset lecture note model, generating a content text of the lecture note, and dividing the content text of the lecture note to obtain at least one content block.

[0158] The second generating unit 503 is used to insert the exercises in the exercise information into at least one of the content blocks to obtain a to-be-displayed file of the lecture notes of the to-be-processed video.

[0159] The device of the embodiment can execute the technical solution in the above method. Its specific implementation process and technical principle are the same and will not be repeated here.

[0160] Figure 6 is a schematic diagram according to a sixth embodiment of the present disclosure, Figure 6 As shown, the present disclosure provides a video processing device 600 based on a large model, the device comprising:

[0161] The extraction unit 601 is used to identify the video to be processed and obtain exercise information and knowledge information; wherein the exercise information includes the text of the exercises in the courseware screenshot of the video to be processed; the knowledge information includes the knowledge text in the courseware screenshot of the video to be processed and the subtitle text corresponding to the audio of the video to be processed; the courseware screenshot is a screenshot of the video to be processed.

[0162] The first generating unit 602 is used to input the knowledge information into a preset lecture note model, generate a content text of the lecture note, and divide the content text of the lecture note to obtain at least one content block.

[0163] The second generating unit 603 is used to insert the exercises in the exercise information into at least one of the content blocks to obtain a to-be-displayed file of the lecture notes of the to-be-processed video.

[0164] Optionally, the second generating unit 603 includes:

[0165] The identification module 6031 is used to identify each exercise in the exercise information to obtain at least one exercise; and to match a content block corresponding to the obtained exercise from the at least one content block;

[0166] The inserting module 6032 is used to insert the obtained exercises into the content block corresponding to the obtained exercises to obtain the file to be displayed.

[0167] Optionally, the exercise has first time information, which indicates the time when the screenshot of the courseware in which the exercise is located appears on the video to be processed; the content block has second time information;

[0168] The identification module 6031 includes:

[0169] The matching submodule 60311 is used to determine the content block that matches the obtained exercise according to the first time information of the obtained exercise and the second time information of the content block.

[0170] Optionally, the identification module 6031 includes:

[0171] The splitting submodule is used to split the exercises in the exercise information according to the exercise sequence numbers in the exercise information to obtain the at least one exercise; or to perform format recognition processing on the exercises in the exercise information to obtain the at least one exercise.

[0172] Optionally, the first generating unit 602 includes:

[0173] The topic generation module 6021 is used to input the knowledge information into a preset lecture model according to the playback order of the video to be processed, and extract multiple knowledge topics; wherein the knowledge topic represents the title extracted from part of the knowledge information.

[0174] The outline generation module 6023 is used to input the knowledge topics into the preset lecture model according to the playback order of the videos to be processed to generate a knowledge outline; wherein the knowledge outline represents the hierarchical relationship of the knowledge topics.

[0175] The explanation generation module 6025 is used to input the knowledge information corresponding to each of the knowledge topics into the preset lecture note model to generate the explanation content of each of the knowledge topics.

[0176] The combination module 6026 is used to combine the explanation content with the knowledge outline according to the knowledge theme to obtain the content text of the lecture notes.

[0177] Optionally, the first generating unit 602 further includes a theme optimization module 6022 for

[0178] The first optimization submodule 60221 is used to delete the knowledge topic if the knowledge topic does not match the preset video topic of the video to be processed.

[0179] The second optimization submodule 60222 is used to merge the knowledge topics and the knowledge information corresponding to the knowledge topics if there are two adjacent knowledge topics and their corresponding knowledge information whose content matching degree is greater than a first threshold.

[0180] Optionally, the first generating unit 602 further includes an outline optimization module 6024. The outline optimization module 6024 includes:

[0181] The calculation submodule 60241 is used to calculate the hierarchical parameters between the upper knowledge topic in the hierarchical relationship and the lower knowledge topic corresponding to the upper knowledge topic according to the hierarchical relationship of the knowledge topics in the knowledge outline.

[0182] The judgment submodule 60242 is used to redetermine the hierarchical relationship between the upper knowledge topic and the lower knowledge topic if the hierarchical parameter is less than a second threshold.

[0183] The updating submodule 60243 is used to update the knowledge outline according to the re-determined hierarchical relationship.

[0184] Optionally, the first generating unit 602 includes:

[0185] The extraction module 6027 is used to divide the content text of the lecture notes into multiple high-level knowledge topic blocks according to the multiple high-level knowledge topics with the highest levels in the knowledge outline.

[0186] The segmentation module 6028 is used to combine the advanced knowledge subject blocks to obtain a plurality of the content blocks; wherein each of the content blocks includes at least one advanced knowledge subject block.

[0187] Optionally, the segmentation module 6028 includes:

[0188] The calculation submodule 60281 is used to calculate the content relevance between two adjacent high-level knowledge topic blocks.

[0189] The first judgment submodule 60282 is configured to combine two adjacent high-level knowledge subject blocks into one content block if the content relevance between the two adjacent high-level knowledge subject blocks is greater than or equal to a third threshold.

[0190] Optionally, the extraction unit 601 includes:

[0191] The first recognition module 6011 is used to use a text recognition model to recognize the courseware screenshots in the video to be processed to obtain exercise information and knowledge text.

[0192] The second recognition module 6012 is used to use an audio recognition model to recognize the audio of the video to be processed and obtain subtitle text corresponding to the audio.

[0193] The fusion module 6013 is used to fuse the knowledge text and the subtitle text to obtain the knowledge information.

[0194] Optionally, the knowledge text has third time information, and the third time information indicates the time when the screenshot of the courseware containing the knowledge text appears on the video to be processed; the subtitle text has fourth time information, and the fourth time information indicates the time point when the subtitle text appears in the speech;

[0195] Fusion module 6013, including:

[0196] The matching submodule 60121 is used to determine, based on the third time information of the knowledge text, the subtitle text corresponding to the fourth time information that matches the third time information.

[0197] The fusion submodule 60122 is used to fuse the knowledge text and its matching subtitle text to obtain the knowledge information.

[0198] The device of the embodiment can execute the technical solution in the above method. Its specific implementation process and technical principles are the same and will not be repeated here.

[0199] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0200] According to an embodiment of the present disclosure, the present disclosure also provides a computer program product, which includes: a computer program, the computer program is stored in a readable storage medium, at least one processor of an electronic device can read the computer program from the readable storage medium, and at least one processor executes the computer program so that the electronic device executes the solution provided by any of the above embodiments.

[0201] Figure 7 A schematic block diagram of an electronic device 700 for implementing an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0202] like Figure 7 As shown, the device 700 includes a computing unit 701, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 702 or a computer program loaded from a storage unit 708 into a random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the device 700 can also be stored. The computing unit 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0203] A number of components in the device 700 are connected to the I / O interface 705, including: an input unit 706, such as a keyboard, a mouse, etc.; an output unit 707, such as various types of displays, speakers, etc.; a storage unit 708, such as a disk, an optical disk, etc.; and a communication unit 709, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 709 allows the device 700 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0204] The computing unit 701 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 701 performs the various methods and processes described above, such as a video processing method based on a large model. For example, in some embodiments, a video processing method based on a large model may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 708. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 700 via ROM 702 and / or communication unit 709. When the computer program is loaded into RAM 703 and executed by the computing unit 701, one or more steps of the video processing method based on the large model described above may be performed. Alternatively, in other embodiments, the computing unit 701 may be configured to perform a video processing method based on a large model in any other appropriate manner (e.g., by means of firmware).

[0205] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0206] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0207] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0208] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0209] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0210] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services ("Virtual Private Server", or "VPS" for short). The server may also be a server of a distributed system, or a server combined with a blockchain.

[0211] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0212] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A video processing method based on a large model, comprising: Identify the video to be processed to obtain exercise information and knowledge information; wherein the exercise information includes the text of the exercise in the courseware screenshot of the video to be processed; the knowledge information includes the knowledge text in the courseware screenshot of the video to be processed and the subtitle text corresponding to the audio of the video to be processed; the courseware screenshot is a screenshot of the video to be processed; Inputting the knowledge information into a preset lecture note model to generate a content text of the lecture note, and dividing the content text of the lecture note to obtain at least one content block; The exercises in the exercise information are inserted into at least one of the content blocks to obtain a to-be-displayed file of the lecture notes of the to-be-processed video.

2. The method according to claim 1, wherein: The step of inserting the exercises in the exercise information into at least one of the content blocks to obtain a to-be-displayed file of the lecture notes of the to-be-processed video includes: Identify each exercise in the exercise information to obtain at least one exercise; and match a content block corresponding to the obtained exercise from the at least one content block; The obtained exercises are inserted into the content block corresponding to the obtained exercises to obtain the file to be displayed.

3. The method according to claim 2, wherein: The exercise has first time information, and the first time information indicates the time when the screenshot of the courseware containing the exercise appears on the video to be processed; The content block has second time information; The step of matching a content block corresponding to the obtained exercise from the at least one content block comprises: The content block matching the obtained exercise is determined according to the first time information of the obtained exercise and the second time information of the content block.

4. The method according to claim 2 or 3, wherein: The step of identifying each exercise in the exercise information to obtain at least one exercise comprises: splitting the exercises in the exercise information according to the exercise sequence numbers in the exercise information to obtain the at least one exercise; Alternatively, format recognition processing is performed on the exercises in the exercise information to obtain the at least one exercise.

5. The method according to any one of claims 1 to 4, wherein: The step of inputting the knowledge information into a preset lecture note model to generate a content text of the lecture note includes: Input the knowledge information into a preset lecture model according to the playback order of the videos to be processed, and extract multiple knowledge topics; wherein the knowledge topics represent titles extracted from part of the knowledge information; Inputting the knowledge topics into the preset lecture model according to the playback order of the videos to be processed to generate a knowledge outline; wherein the knowledge outline represents the hierarchical relationship of the knowledge topics; Inputting the knowledge information corresponding to each of the knowledge topics into the preset lecture notes model to generate the explanation content of each of the knowledge topics; According to the knowledge theme, the explanation content is combined with the knowledge outline to obtain the content text of the lecture notes.

6. The method according to claim 5, wherein: After inputting the knowledge information into a preset lecture model according to the playback order of the videos to be processed and extracting a plurality of knowledge topics, the method further includes: If the knowledge topic does not match the preset video topic of the video to be processed, deleting the knowledge topic; If there are two adjacent knowledge topics and their corresponding knowledge information whose content matching degree is greater than a first threshold, the knowledge topics and the knowledge information corresponding to the knowledge topics are merged.

7. The method according to claim 6, wherein: After inputting the knowledge topics into the preset lecture model according to the playback order of the videos to be processed to generate the knowledge outline, the method further includes: According to the hierarchical relationship of the knowledge topics in the knowledge outline, calculating the hierarchical parameters between the upper knowledge topic in the hierarchical relationship and the lower knowledge topic corresponding to the upper knowledge topic; If the level parameter is less than a second threshold, re-determining the level relationship between the upper level knowledge topic and the lower level knowledge topic; The knowledge outline is updated according to the re-determined hierarchical relationship.

8. The method according to any one of claims 5 to 7, wherein: The step of dividing the content text of the lecture notes to obtain at least one content block includes: According to the plurality of advanced knowledge themes of the highest level in the knowledge outline, the content text of the lecture notes is divided into a plurality of advanced knowledge theme blocks; The high-level knowledge subject blocks are combined to obtain a plurality of the content blocks; wherein each of the content blocks includes at least one high-level knowledge subject block.

9. The method according to claim 8, wherein: The high-level knowledge subject blocks are combined to obtain a plurality of content blocks, including: Calculating the content relevance of two adjacent high-level knowledge topic blocks; If the content relevance of two adjacent high-level knowledge topic blocks is greater than or equal to a third threshold, the two knowledge topic blocks are combined into one content block.

10. The method according to any one of claims 1 to 9, wherein: The process of identifying the video to be processed and obtaining the exercise information and knowledge information includes: Using a text recognition model, identifying the courseware screenshots from the video to be processed to obtain exercise information and knowledge text; Using an audio recognition model, recognizing the audio of the video to be processed to obtain subtitle text corresponding to the audio; The knowledge text and the subtitle text are merged to obtain the knowledge information.

11. The method according to claim 10, wherein: The knowledge text has third time information, and the third time information indicates the time when the screenshot of the courseware in which the knowledge text is located appears on the video to be processed; The subtitle text has fourth time information, and the fourth time information indicates a time point at which the subtitle text appears in the speech; The knowledge text and the subtitle text are merged to obtain the knowledge information, including: Determining, according to the third time information of the knowledge text, a subtitle text corresponding to the fourth time information matching the third time information; The knowledge text and its matching subtitle text are fused to obtain the knowledge information.

12. A video processing device based on a large model, comprising: The extraction unit is used to identify the video to be processed and obtain exercise information and knowledge information; wherein the exercise information includes the text of the exercise in the courseware screenshot of the video to be processed; the knowledge information includes the knowledge text in the courseware screenshot of the video to be processed and the subtitle text corresponding to the audio of the video to be processed; the courseware screenshot is a screenshot of the video to be processed; A first generating unit is used to input the knowledge information into a preset lecture note model to generate a content text of the lecture note, and divide the content text of the lecture note to obtain at least one content block; The second generating unit is used to insert the exercises in the exercise information into at least one of the content blocks to obtain a to-be-displayed file of the lecture notes of the to-be-processed video.

13. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 11.

14. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-11.

15. A computer program product, comprising a computer program, which implements the steps of the method according to any one of claims 1 to 11 when executed by a processor.

Images