Courseware page classification-based video marking method and device, equipment and medium

Abstract

The embodiment of the invention discloses a video marking method and device based on courseware page classification, electronic equipment and a storage medium. According to common specific teaching links in the teaching activity development process, each courseware page of the teaching courseware corresponding to the teaching activity corresponds to one teaching link, specifically, the courseware pages in the teaching courseware are predicted correspondingly through a pre-trained teaching link recognition model, and the courseware pages in the teaching courseware are obtained. In addition, a data source and a change event which cause the change of main display content in the classroom video are recorded, the occurrence moment of the change event and the teaching link of the courseware page displayed after the occurrence are confirmed, and the teaching link is used as the outline information of the display time period of the corresponding courseware page. Under the condition that the classroom video content does not need to be identified, accurate structure division and outline information identification of the classroom video based on the teaching content are realized according to the identification of the conventional links in the teaching activity.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of educational multimedia technology, and in particular to a video tagging method, apparatus, device and medium based on courseware page classification. Background Art

[0002] With the continuous development and popularization of information technology, in the field of teaching, computer-assisted teaching, distance learning, autonomous online learning, electronic classrooms and other forms of education based on information technology have made great progress. Classroom video recording has become the norm in the modern teaching process. Recorded classroom videos can be used for students' distance learning, after-class learning, teachers' after-class review and teaching reflection, and play a more obvious role in achieving educational equity and improving teaching standards.

[0003] Classroom videos are usually long videos of about 40 minutes. Students who did not participate in the classroom recording usually need to watch the video carefully to roughly understand the content of the class. When students who participated in the classroom recording need to go back and forth to find the location of some knowledge points that they did not understand or were interested in after class, they also need to use the progress bar to constantly search and locate; and when teachers who participated in the classroom recording want to reflect on their teaching after class, they need to use the video progress bar to constantly jump back and forth to retrieve and locate specific teaching clips. In order to achieve rapid positioning, classroom videos are usually manually added or the outline of the classroom video is added after the video content is recognized by the algorithm. Manual addition is inefficient. In order to add the outline of the classroom video efficiently and accurately, the outline of the classroom video is mostly added after the video content is recognized by the algorithm.

[0004] When the inventor studied the existing solutions for automatically dividing video segments in non-educational scenarios and automatically identifying the video outlines of the corresponding segments, he found that the existing solutions for automatically identifying video outlines usually require a certain amount of video annotation data to train the model, and the model is used to identify the video subtitle content and the picture to complete the division. Since the boundaries of video segments are usually fuzzy, the existing solutions for automatically identifying video outlines cannot accurately extract and identify the video outlines. Summary of the invention

[0005] The present invention provides a video marking method, device, equipment and medium based on courseware page classification to solve the technical problem that the existing solution for automatically marking video outline cannot accurately extract and mark the video outline.

[0006] In a first aspect, an embodiment of the present application provides a video marking method based on courseware page classification, the video marking method based on courseware page classification comprising:

[0007] Get the teaching courseware played during the classroom video recording and the page switching events of the teaching courseware;

[0008] Based on the pre-trained teaching link recognition model, the teaching link corresponding to each courseware page of the teaching courseware is identified to obtain the teaching link information corresponding to each courseware page;

[0009] According to the page switching event, confirm the displayed target courseware page and the display time period corresponding to each target courseware page;

[0010] Mark the video outline information of the corresponding display period in the classroom video according to the teaching link information corresponding to the target courseware page.

[0011] In a second aspect, an embodiment of the present application further provides a video marking device based on courseware page classification, the video marking device based on courseware page classification comprising:

[0012] A classroom data acquisition unit, used to acquire the teaching courseware played during the classroom video recording process and the page switching events of the teaching courseware;

[0013] The courseware page classification unit is used to identify the teaching link corresponding to each courseware page of the teaching courseware based on the pre-trained teaching link recognition model, and obtain the teaching link information corresponding to each courseware page;

[0014] A target page confirmation unit, used to confirm the displayed target courseware page and the display period corresponding to each target courseware page according to the page switching event;

[0015] The outline information marking unit is used to mark the video outline information of the corresponding display period in the classroom video according to the teaching link information corresponding to the target courseware page.

[0016] In a third aspect, an embodiment of the present application further provides an electronic device, the electronic device comprising:

[0017] one or more processors;

[0018] a memory for storing one or more computer programs;

[0019] When one or more computer programs are executed by one or more processors, the electronic device implements the video marking method based on courseware page classification as in the first aspect.

[0020] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the video marking method based on courseware page classification as in the first aspect.

[0021] The above-mentioned video marking method, device, equipment and medium based on courseware page classification, in which the teaching courseware played during the classroom video recording and the page switching events of the teaching courseware are obtained; based on the pre-trained teaching link recognition model, the teaching link corresponding to each courseware page of the teaching courseware is identified to obtain the teaching link information corresponding to each courseware page; according to the page switching event, the displayed target courseware page and the display time period corresponding to each target courseware page are confirmed; according to the teaching link information corresponding to the target courseware page, the video outline information of the corresponding display time period in the classroom video is marked. According to the common specific teaching links in the process of teaching activities, each courseware page of the teaching courseware of the corresponding teaching activity is corresponded to a teaching link. The specific correspondence is obtained by predicting the courseware pages in the teaching courseware through the pre-trained teaching link recognition model. In addition, the data source and change event that cause the main display content in the classroom video to change are recorded, and the occurrence time of the change event and the teaching link of the courseware page displayed after the occurrence are confirmed. The teaching link is used as the outline information of the display period of the corresponding courseware page. Without the need to identify the content of the classroom video, according to the identification of the regular links in the teaching activities, the accurate structural division and outline information identification of the classroom video based on the teaching content are realized. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 A method flow chart of a video tagging method based on courseware page classification provided in an embodiment of the present application.

[0023] Figure 2 This is a diagram of the screen structure of the classroom video.

[0024] Figure 3 A schematic diagram of the generation process of data processing related models in the video tagging method based on courseware page classification provided in an embodiment of the present application.

[0025] Figure 4 A schematic diagram of the relationship between page switching events and the timeline in a video marking method based on courseware page classification provided in an embodiment of the present application.

[0026] Figure 5 A schematic diagram showing a video outline generated by a video tagging method based on courseware page classification provided in an embodiment of the present application.

[0027] Figure 6 A schematic diagram of the structure of a video marking device based on courseware page classification provided in an embodiment of the present application.

[0028] Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0029] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only the parts related to the present invention, rather than all structures, are shown in the accompanying drawings.

[0030] It should be noted that due to space limitations, this application description does not list all optional implementation methods. After reading this application description, those skilled in the art should be able to understand that as long as the technical features do not contradict each other, any combination of technical features can constitute an optional implementation method.

[0031] The following is a detailed description of the various embodiments of the present invention.

[0032] With the development of information technology, the forms of computer-assisted teaching are constantly enriched and the effects are constantly improved. For example, an interactive tablet is configured in the teaching scene, and the electronic display and electronic recording of teaching information are realized based on the interactive tablet. The interactive tablet mentioned in this solution can be an integrated device that controls the content displayed on the display tablet and realizes human-computer interaction through touch technology. It integrates one or more functions such as a projector, an electronic whiteboard, a screen, a speaker, a television, and a video conferencing terminal.

[0033] Generally speaking, the interactive tablet includes at least one screen. For example, the interactive tablet is equipped with a screen with a touch function, and the screen can be a capacitive screen, a resistive screen, or an electromagnetic screen. In this embodiment, further, the user can implement touch operation by touching the screen with a finger or a stylus, and accordingly, the interactive tablet detects the touch position and responds according to the touch position to implement the touch function. Typically, the interactive tablet is installed with at least one operating system, wherein the operating system includes but is not limited to the Android system, the Linux system, and the Windows system.

[0034] In the teaching scenario, the interactive tablet is an interactive device mainly used to support the electronic display and electronic recording of teaching information, and the software therein is mainly used to realize the display and recording function requirements in the teaching scenario. For example, the interactive tablet can install at least one application with a document presentation function, wherein the application can be an application that comes with the operating system; at the same time, an application downloaded from a third-party device or server is also installed. Optionally, in addition to the basic function of displaying the edited content, the program with the document presentation function also has other editing functions in the presentation process, such as inserting tables, inserting pictures, inserting graphics, drawing tables, drawing graphics, etc., to realize the real-time display of user information input and input content during the information presentation process, thereby improving the information presentation and interaction effect during the teaching-related document presentation process. For example, the interactive tablet can install at least one application with a whiteboard function, and teachers can realize real-time writing, recording, modification and saving in the teaching scenario based on this application.

[0035] When the interactive tablet is used alone, it is mainly suitable for teachers to electronically display teaching information to students in the same teaching scene during offline teaching, such as teaching courseware, simulation animations, simulation software, etc.; and to electronically record the blackboard writing, notes, annotations, etc. generated by teachers during teaching activities.

[0036] On the basis of the interactive tablet in the teaching scene, a camera can be set up in the teaching scene to record the classroom video during the teaching process. The camera lens faces the side where the interactive tablet is located. When implementing specific teaching activities, the interactive tablet is displayed on the screen according to the teacher's operation. From the relative position of the relevant elements in the teaching process, the camera's shooting targets are mainly the teacher and the interactive tablet, and the wall where the interactive tablet is located is the teacher's lecture background. When the classroom video recorded by the camera is used later, the classroom video is only obtained by shooting the camera toward a shooting area. The classroom video is not structured when it is shot and generated. The direct use of the classroom video requires the playback progress to be adjusted to locate the possible target teaching moment. For example, students who do not participate in the classroom recording usually need to watch the video carefully to roughly understand the content of the classroom explanation; when students who participate in the classroom recording need to trace back some knowledge points that they did not understand or were interested in after class, they also need to constantly search for the location through the progress bar; and when the teacher who participated in the classroom recording wants to do teaching reflection after class, he needs to constantly jump back and forth through the video progress bar to retrieve and locate the specific teaching clip. In order to improve the efficiency of subsequent browsing of classroom videos, we can consider using the information summary of each segmented content in the segmented content structure as the outline information of the classroom video based on the segmented content structure of the substantial content recorded in the classroom video, so as to obtain the video outline of the classroom video. Based on the video outline, we can quickly confirm the general content of each part of the classroom video, and realize fast jump viewing of a certain part of the content based on the video outline. The video outline identified by the classroom video implemented by the automatic content recognition solution in the general video processing scenario usually cannot accurately identify the segmented content structure of the classroom video with reference to the teaching content.

[0037] In view of the above technical problems, the embodiment of the present application proposes a video marking method based on the classification of courseware pages. According to the common specific teaching links in the process of carrying out teaching activities, each courseware page of the teaching courseware corresponding to the teaching activity is corresponded to a teaching link. The specific correspondence is obtained by predicting the courseware pages in the teaching courseware through a pre-trained teaching link recognition model. In addition, the data source and change event that cause the main display content in the classroom video to change are recorded, and the time of occurrence of the change event and the teaching link of the courseware page displayed after the occurrence are confirmed. The teaching link is used as the outline information of the display period of the corresponding courseware page. Without the need to identify the content of the classroom video, according to the identification of the conventional links in the teaching activities, the accurate structural division and outline information identification of the classroom video based on the teaching content are realized. The various embodiments of the present invention are described in detail below.

[0038] Figure 1The method flow chart of a video tagging method based on courseware page classification provided in the embodiment of the present application is applied to electronic devices that process relevant pictures of the teaching scene, and is implemented by electronic devices, such as a recording and broadcasting host, a cloud server that stores and manages teaching data, etc. Figure 1 As shown, the video marking method based on courseware page classification includes steps S110 to S140:

[0039] Step S110: Acquire the teaching courseware played during the classroom video recording and the page switching event of the teaching courseware.

[0040] Taking the implementation based on the recording and broadcasting host as an example, on the basis of the interactive tablet in the teaching scene, a camera and a recording and broadcasting host can be set up in the teaching scene, and the camera and the interactive tablet are connected to the recording and broadcasting host. The lens of the camera faces the side where the interactive tablet is located. From the relative position of the relevant elements in the teaching process, the shooting target of the camera when recording the classroom video includes the teacher and the interactive tablet in one embodiment. The interactive tablet is used to display the teaching courseware, that is, the teaching courseware is displayed in the recording area of ​​the classroom video. The picture captured by the camera is transmitted to the recording and broadcasting host in real time through the corresponding connection. While the interactive tablet displays on the screen according to the teacher's operation, the page switching event in operation is transmitted to the recording and broadcasting host in real time through the corresponding connection. The page switching event specifically refers to the event of operating the editing software of the teaching courseware to enter the slide show mode, the event of switching the courseware page in the slide show mode from one page to another, or the event of ending the slide show mode. The courseware page is displayed in full screen in the slide show mode.

[0041] In the process of teachers organizing teaching activities, there may be many operations that occur on the interactive tablet, such as annotation operations, blackboard operations, etc., where the file structure of the teaching courseware is usually based on pages as the basic unit. During the display of the teaching courseware, one page of content is displayed each time. The switching of the specific display content can be automatic switching, or the teacher can trigger the switching through laser pens, touch operations, mouse operations, keyboard shortcuts, etc. In the embodiment of the present application, all events that cause the pages of the teaching courseware to switch are defined as page switching events. Based on the above screen transmission and page switching events, the original data corresponding to the change in the interactive tablet display screen, that is, the page switching event, is received synchronously (or at a very small interval, which is equivalent to synchronization) from the interactive tablet in the screen received by the recording and broadcasting host from the camera. At this time, it can be regarded as realizing the recording of page switching events and classroom videos based on the same timeline.

[0042] like Figure 2As shown, when the teaching scene is a classroom 10, the classroom video is obtained by shooting the camera toward the teacher's lecture background. From the perspective of the structure of the screen content of the classroom video, it includes the foreground with the teacher 30 and the interactive tablet 20 as the main body, and the explanation background with the wall as the main body. The content in the classroom video is the video screen. Although the changes in the displayed content in the interactive tablet 20 and the movement of the teacher 30 may cause changes in the content details of the video screen, because the state of the camera remains basically unchanged, the layout of the classroom 10 also remains basically unchanged, and the structure of the screen content in the classroom video also remains basically unchanged. If the acquisition of the classroom video is completed by the recording and broadcasting host, the recording and broadcasting host can transmit data with the data processing related hardware for realizing the demonstration of teaching courseware through various existing connection methods (wired or wireless). For example, if the hardware device for realizing the data processing of the courseware demonstration is an interactive tablet, the recording and broadcasting host obtains the teaching courseware from the interactive tablet; if the hardware device for realizing the data processing of the courseware demonstration is a laptop connected to the interactive tablet, the recording and broadcasting host obtains the teaching courseware from the laptop. In addition, the hardware device for realizing the courseware demonstration may also be a projector, and the recording and broadcasting host obtains the teaching courseware from the computer host currently connected to the projector.

[0043] It should be understood that the recording and broadcasting host is only an optional implementation method. It can also be based on a cloud server. If the classroom video is processed in real time, the page switching event is also generated and received in real time, which is equivalent to recording the page switching event and the classroom video based on the same time axis through real-time processing. If the video outline is generated after the classroom video is recorded, considering that the starting time of the classroom video recording is not necessarily synchronized with the starting time of the teacher's operation of the teaching courseware, in order to ensure that the page switching event is aligned with the changes in the interactive flat-panel display content in the classroom video, when recording the classroom video and recording the page switching event, in addition to generating a classroom video with the recording length as the time axis, it is also necessary to record the system time when the classroom video starts recording. Correspondingly, each page switching event also needs to record the system time, so as to realize the teaching link information in the target courseware page corresponding to the page switching event as the video outline information of the classroom video, and the final video outline information can accurately correspond to the changes in the actual picture in the classroom video because of the system time. When specifically realizing the acquisition and recording of basic data and information, each page switching event at least needs to record the occurrence time of the page switching event and the page information of the courseware page displayed after the switch. For the entire processing process of the embodiment of the present application, the outline information of the classroom video is generated with reference to the courseware page displayed after each page switching event. The page switching event and the courseware page displayed after the switching are directly obtained from the data source. Therefore, the location for displaying the teaching courseware does not necessarily need to be within the recording area of ​​the classroom video. Figure 2What is shown is only an exemplary shooting picture. If the camera performs target tracking, the display of the teaching courseware may no longer be captured. The page switching event of the teaching courseware can always be received, and this solution can be implemented accordingly.

[0044] In addition, the classroom video and teaching courseware can be obtained separately. For example, before or after obtaining the classroom video from the camera, the user can upload the teaching courseware through another electronic device, and the recording and broadcasting host obtains the teaching courseware accordingly. That is, the embodiment of the present application does not limit the acquisition of the classroom video and the teaching courseware to a certain time sequence or synchronization relationship. It is only necessary to have the classroom video and the corresponding teaching courseware before the specific data processing process of the video outline generation method in the embodiment of the present application is implemented.

[0045] If the acquisition of classroom video is completed by the recording and broadcasting host, the recording and broadcasting host can transmit data with the camera through an HDMI (High Definition Multimedia Interface) cable or other multimedia protocol connection cables, that is, the classroom video can be sent from the camera to the recording and broadcasting host through any of the various multimedia protocol connection cables.

[0046] Step S120: Based on the pre-trained teaching link recognition model, the teaching link corresponding to each courseware page of the teaching courseware is identified to obtain the teaching link information corresponding to each courseware page.

[0047] In the teaching activities carried out in the classroom, from a macro perspective, teachers and students are in the same space to transmit information, but the teaching activities themselves have clear communication goals, more specifically, they are communication activities with clear teaching goals. In order to achieve the teaching goals, teachers usually design the teaching process and strive to achieve the established teaching goals through rigorous and meticulous teaching links. Teaching activities in different subjects and grades have their own subject characteristics. Take primary school Chinese as an example: Common teaching links in primary school Chinese teaching include introduction of new lessons, learning goals, word learning, character introduction, background information, discussion, practice, class summary, consolidation of new knowledge, knowledge expansion, assignment, etc.

[0048] In order to ensure the continuity of teaching activities, teachers will not verbally indicate which teaching link will be carried out next, but will connect different teaching links through detailed design (such as short stories, transition words, etc.). Although these contents will be recorded in the classroom video, most of the existing automatic content recognition solutions can only recognize surface information such as pictures and voices, and cannot extract and structure the information recorded in the classroom video. The randomness of the specific information recorded in the classroom video at the time of generation has brought great difficulties to information extraction and structured division.

[0049] In the embodiment of the present application, in order to ensure the accurate division of the structure of the classroom video, considering that teaching activities are often carried out based on teaching courseware, the recording target of the classroom video also includes the display of teaching courseware, so the structural division of the classroom video is realized by focusing on the structural division of the teaching courseware. First of all, the teaching courseware is designed based on the teaching ideas of carrying out teaching activities, that is, the courseware pages in the teaching courseware will inevitably correspond to the designed teaching links; secondly, the development of teaching activities based on the teaching courseware will inevitably be centered on the display of the teaching courseware. The teacher may adjust the teaching details at any time during the teaching process, but the structure of the teaching courseware remains unchanged in other changes. According to the above analysis of specific application scenarios, the embodiment of the present application uses the correspondence between the courseware pages and the teaching links in the teaching courseware as a reference for generating the outline information of the classroom video.

[0050] When editing teaching courseware, the specific content in the courseware page is specifically designed for the teaching links. In order to reduce the output of invalid information, the courseware page itself will not specifically record which teaching link it corresponds to. In an embodiment of the present application, the courseware page of the teaching courseware is specifically identified with the corresponding teaching link through a pre-trained teaching link recognition model to obtain the teaching link information corresponding to each courseware page.

[0051] The teaching link recognition model can be trained in the following way: obtain the first label information added to each courseware page in multiple first sample courseware according to the predefined teaching link, and take each first sample courseware and the corresponding first label information as a first sample data to obtain a first sample courseware data set; train the preset language model based on the first sample courseware data set to obtain a teaching link recognition model with the teaching courseware as input and the probability distribution of different label information corresponding to each courseware page in the teaching courseware as the basis for output; wherein the teaching link information is confirmed based on the basic output.

[0052] For example, for the teaching courseware of primary school Chinese, the teaching links of primary school Chinese predefined in the previous text are used to add the teaching link label, that is, the first label information, to each courseware page. For multiple teaching courseware, after manually adding the first label information to each page of the courseware page, the first sample courseware data set D = (d1, d2, ..., d n ), where d i =(a i ,b i ), where a i For the content of a certain teaching courseware page, b i The content of the courseware page for teaching courseware i All corresponding first label information.

[0053] Using the first sample courseware dataset D, a model for identifying the teaching links corresponding to the courseware page is trained based on the preset language model, i.e., the teaching link identification model. The model input of the teaching link identification model is the teaching courseware, specifically all the courseware pages of the teaching courseware, and the basic output is the probability distribution of the teaching links corresponding to the courseware page, i.e., the probability distribution of different label information corresponding to each courseware page in the teaching courseware. Based on the basic output, the label information with the highest corresponding probability can be used as the teaching link information corresponding to the courseware page.

[0054] Taking Bert as a preset language model as an example, assuming that the number of teaching links (i.e., the number of first label information) is N, all the courseware pages of the teaching courseware can be input into the Bert model, and the embedding corresponding to the CLS of the last layer of Bert is taken as the courseware page encoding, and then a fully connected layer (softmax as the activation function) is connected externally. The number of neurons in the output layer is N, corresponding to the probability distribution of N first label information. The first label information with the largest probability in the N probability distribution is taken as the final output, that is, the teaching link information corresponding to the courseware page.

[0055] Assume that "bert(x)" means to encode the text x using bert, "[CLS]" means to take the embedding corresponding to the CLS of the last layer of bert, f(x) means the CLS encoding result of the text x after bert, h(x) means the teaching link prediction model of the courseware page, out(x) means the teaching link prediction result of the text x on the courseware page, W and b respectively represent the weight parameter and bias parameter of the fully connected layer. When the input courseware content is x i The above process can be expressed as:

[0056] f(x i )=bert(x i )[CLS]

[0057] h(x i )=softmax(Wf(x i )+b)

[0058] out(x i )=argmax(h(x i ))

[0059] After a certain number of training rounds using the first sample courseware data set D, a teaching link recognition model can be obtained that predicts the teaching link corresponding to a courseware page.

[0060] Of course, when specifically generating the first courseware sample data set, a single courseware page and the first label information corresponding to the courseware page may also be used as a data element. The overall training method is generally the same and will not be repeated here.

[0061] Step S130: confirming the displayed target courseware pages and the display time period corresponding to each target courseware page according to the page switching event.

[0062] The picture in the classroom video is the video picture. An important picture content of the video picture is the display picture of the teaching courseware. Corresponding to the relationship between the picture content and the display picture of the video picture, the change process of the picture content in the video picture is strongly related to the operation of the teaching courseware in the specific teaching process, that is, the annotation operation, blackboard operation, page switching operation, etc. of the teaching courseware will cause the picture content in the recorded classroom video to change.

[0063] For teaching courseware, one courseware page is displayed each time, and each courseware page usually has a content core. The annotation and blackboard writing operations performed in the process of displaying a courseware page are generally centered on the content core of the courseware page. Therefore, although the annotation and blackboard writing operations may cause changes in the picture content in the classroom video, it is still considered that the change occurs within the display stage of a courseware page. Based on this, when generating the video outline of the classroom video, the display time period of each courseware page is used as a reference for dividing the segmented content structure of the video outline, and the information summary of the courseware page is used as the outline information of each segmented content structure.

[0064] The display change of the courseware page is determined by the page switching operation, such as page turning (including forward page turning and backward page turning) and page jump. It should be understood that starting to display the courseware page and ending to display the courseware page are also a kind of page switching operation. In the embodiment of the present application, based on the specific information of the corresponding record of each page switching event, it is possible to obtain which courseware pages enter the display state caused by the page switching operation of the teacher during the teaching process, and the display time period corresponding to each courseware page. In the embodiment of the present application, considering that not all courseware pages of a teaching courseware may necessarily enter the display state, or not all courseware pages are necessarily effectively displayed, the courseware page that is only effectively displayed is defined as the target courseware page, and the display time period corresponding to the target courseware page and each target courseware page is obtained according to the statistics of the page switching event. Effective display refers to the display whose display duration reaches the preset threshold value.

[0065] For example, when a certain teaching courseware is used in the teaching process, the page switching event of the teacher in the teaching process is monitored, and the specific moment when the page switching event occurs each time and the page information displayed after the page switching event are recorded. Of course, the recording method of the page switching event can also be a process of comprehensive information processing, such as recording the display period of the target courseware page that has recently ended display according to the page change and the specific moment, that is, recording the display start time and display end time of the target courseware page that has recently ended display. If the interval between two adjacent switching operations is very short (for example, 1 second or 2 seconds), it can be considered that the previous switching operation is only to realize the transition of the display of the target courseware page (for example, quickly turning from page 1 to page 3, page 2 is transition), at this time, this series of page switching operations can be corrected as a page switching event, and the courseware page whose final display duration reaches the preset threshold value is used as the target courseware page, and the display period of the target courseware page obtained by this processing actually displays multiple courseware pages, but only one courseware page is normally displayed, and the remaining courseware pages are transitional displays, without actual display significance, and there is no need to generate outline information accordingly.

[0066] The specific display time period confirmation method can be designed according to the time information recording method in the page switching event. For example, the moment when the classroom video starts recording is directly used as the revelation moment of the first segmented content structure, and the courseware page before the first page switching event is used as the first target courseware page. Then the first target courseware page may be the first courseware page used in the teaching activity (before starting to record the classroom video, the teacher has operated the teaching courseware to start the display of the courseware page, and the courseware page is displayed in full screen from the beginning of the classroom video). The first target courseware page may also be empty (the slide show mode is entered after the classroom video is recorded, and there is no courseware page displayed in full screen at the beginning of the classroom video). Or directly divide the display time period based on the existence of a clear target courseware page as the standard, that is, only when there is a clear target courseware page display, it is necessary to confirm the corresponding display time period, and only then is there a necessity to confirm the outline information in the corresponding segmented content structure in the classroom video. When the page switching event and the classroom video are recorded based on the same timeline, the accurate division of the display time period can be achieved at any time when the slide show mode is entered.

[0067] like Figure 4 As shown, in a certain teaching activity, the recording of the classroom video starts at time t0. At time t1, the teacher performs the slide show operation, and a page switching event occurs to enter the slide show mode. The first page of the 15 pages of the teaching courseware is displayed, and then the second page, the third page, and the fourth page are switched to the actual page at t2, t3, and t4 respectively. In this process, a classroom video starting from time t0 and page switching events corresponding to times t1, t2, t3, and t4 are generated.

[0068] Another exemplary specific implementation method is to assume that the slide show mode is entered before the recording of the classroom video begins. At 20 seconds after the recording of the classroom video begins, the teacher switches from page 1 to page 2. After 2 minutes of speaking, the teacher switches to page 5, and ends the course recording 10 minutes later. The recording format of each page switching event is (page start time (in seconds), page end time (in seconds), courseware page number), and the above page switching event can be recorded as: [(0,20,1), (20,140,2), (140,740,5)]; each page switching event is recorded as (page switching time (in seconds), page number before switching, page number after switching), and the above page switching event can be recorded as: [(20,1,2), (140,2,5), (740,5,0)]. For the former recording method, the display period can be directly confirmed from the first two digits of the recording format of the page switching event, and the target courseware page corresponding to the display period can be confirmed from the third digit. For the latter recording method, secondary processing is required to confirm the display period and the corresponding target courseware page. The specific processing result is based on the time of the page switching event and the target courseware pages before and after the switching, and the same confirmation result as the previous recording method can be obtained.

[0069] It should be understood that there is no strict restriction on the execution order of the teaching link identification and confirmation of the display time period of the courseware page of the teaching courseware. Steps S120 and S130 only indicate the identification of the data processing content, and do not indicate the restriction on the execution order. In the specific implementation process, real-time processing can be performed according to the data reception situation, or parallel processing can be performed after the reception is completed, or any one of them can be processed first after the reception is completed. From the final processing result, it can be realized that there is a confirmation result of the display time period when executing step S140, and the teaching link information of the courseware page corresponding to each display time period can be achieved.

[0070] Step S140: Mark the video outline information of the corresponding display period in the classroom video according to the teaching link information corresponding to the target courseware page.

[0071] In an optional implementation, the first label information with the highest probability among the N probability distributions can be directly used as the final output, and then directly marked as the corresponding teaching link information to the corresponding display period of the courseware page to complete the marking of the video partner information.

[0072] In another specific implementation process, since the change of the teaching link information of the courseware page is correlated to the change of the courseware page sequence, for example, the first few pages of the teaching courseware may often belong to the link of introducing a new course, and the last page of the teaching courseware often belongs to the link of assigning homework. In order to further improve the accuracy of the teaching link recognition results of the courseware page, in the embodiment of the present application, when marking the video outline information, the teaching courseware and the corresponding teaching link information are first corrected based on the pre-trained link sequence error correction model; then the corrected teaching link information corresponding to the target courseware page is marked as the video outline information of the corresponding display period in the classroom video. That is, a link sequence error correction model is proposed, and the predicted teaching link information is corrected before marking the video outline information.

[0073] Among them, the link sequence error correction model is trained in the following way: obtaining the second label information added to each courseware page in multiple second sample courseware according to the predefined teaching link, and the third label information of each second sample courseware, each second sample courseware and the corresponding second label information and third label information as a second sample data, to obtain a second sample courseware data set, the third label information is the label information with the highest probability among the label information obtained by inputting the teaching link recognition model for each courseware page; based on the second sample courseware data set, the preset neural network model is trained to obtain a link sequence error correction model with the teaching courseware and the third label information of the teaching courseware as input, and the correction result of the third label information corresponding to each courseware page in the teaching courseware as output.

[0074] The sample data used for the training phase sequence error correction model can be constructed based on the first sample courseware data set that has been annotated before the training teaching phase recognition model. Assume that a courseware X with n pages has been manually annotated, X = (x1, x2, x3, ..., x n ), x i is the i-th page of the courseware, and its corresponding manual annotation result is Y, Y = (y1, y2, y3, ..., y n ), y i is the first label information of the i-th page of the courseware. Using the trained teaching link recognition model, the predicted label P is predicted for the courseware X, where P = (p1, p2, p3, ..., p n ), p i is the third label information of the i-th page of courseware content, that is, the label information with the highest probability among the label information obtained by the courseware page input teaching link recognition model. At this time, X, P, and Y constitute a training data of the courseware page teaching link error correction model, that is, the second sample courseware data set.

[0075] When training the link sequence error correction model based on the second sample courseware data set, since the content length of the entire teaching courseware is much longer than the length of a single page of the courseware, and also far exceeds the maximum supported length of the preset language model (usually 512, 1024), it brings huge challenges to the learning of the link sequence error correction model. In order to reduce the difficulty of the model in understanding super-long texts, reduce the model's dependence on video memory, and more easily capture the relationship between the courseware page and the teaching link, as well as the relationship between the link sequence and the teaching courseware content sequence, the encoding part f(x) of the teaching link recognition model can be reused to encode the text of the input teaching courseware X, and obtain EX, EX = (ex1, ex2, ..., ex n ), where ex i =f(x i ), indicating the courseware page x i The encoding of the predicted teaching link label P is recorded as EP, that is, each label p i , mapped into a k-dimensional vector (k defaults to 32) through a fully connected layer and recorded as ep i , as the encoding of the label. Concatenate each courseware page code exi and the corresponding label code epi as the final code EH of the courseware page, and then pass through a bidirectional LSTM (Long Short-Term Memory) recorded as E-LSTM, and then pass through a fully connected layer (assuming the weight is M, the bias is Z, and softmax is used as the activation function). The number of neurons in the output layer is N, corresponding to the probability distribution of N teaching link information. Take the teaching link information with the highest probability in the probability distribution of N teaching link information as the final corrected teaching link information output of the corresponding courseware page. Assume that the final corrected output teaching link information sequence of courseware X is C, C = (c1, c2, c3, ..., c n ), where c i Indicates courseware content x i and predicted teaching session information p i The teaching link information after model correction; BiLSTM(x) means inputting vector x into the bidirectional LSTM network, then the above process can be expressed as:

[0076] EX=(ex1,ex2,...,ex n )

[0077] EP=(ep1,ep2,...,ep n )

[0078] EH=(concat(ex1,ep1),concat(ex2,ep2),...,concat(ex n ,ep n ))

[0079] LSTM=BiLSTM(EH)

[0080] C=argmax(softmax(M(E-LSTM)+Z))

[0081] After a certain number of rounds of training according to the above network using the second sample courseware data set, a link sequence error correction model can be obtained.

[0082] The above comprehensive construction of training sample sets and the training process can refer to 3. For a teaching courseware consisting of 15 pages of courseware pages, the teaching link recognition model may not be able to fully predict the accurate teaching section change information after identifying the courseware page as a unit. For example, the initial courseware page is predicted to be a class summary. At this time, the training sample set can be further constructed to correct the link sequence and obtain accurate teaching link information.

[0083] When marking the video outline information, adjacent display periods with the same teaching link information can be merged. For example, if multiple consecutive display periods are "word learning", the multiple display periods can be merged into one display period and marked as "word learning".

[0084] After obtaining accurate teaching link information in step S110-step S130, it is equivalent to obtaining the video outline information of the display period corresponding to each target courseware page used in the classroom video recorded in a class. Assume that the target courseware page used by the teacher for teaching is X = (x1, x2, x5), x i is the i-th page of the courseware. Through the prediction and error correction method of the teaching link information described above, the teaching link information sequence of the teaching courseware used on each page is obtained, that is, the video outline information sequence C, C = (c1, c2, c5), where c i It is the video outline information of the display period corresponding to the i-th courseware page in the classroom video.

[0085] like Figure 5 As shown, the video outline information 303 and the classroom video 302 are displayed in the same video playback window 301. The video outline information 302 can display the teaching link information corresponding to each target courseware page as the outline information of the corresponding content segment structure, and correspondingly present the time start and end information of the display period (including the start time and the end time). When a trigger operation for a certain outline information is received in the video outline information 303, the classroom video 302 jumps to the start time of the corresponding display period.

[0086] The video marking method based on courseware page classification obtains the teaching courseware played during the classroom video recording process and the page switching event of the teaching courseware; based on the pre-trained teaching link recognition model, the corresponding teaching link of each courseware page of the teaching courseware is identified to obtain the teaching link information corresponding to each courseware page; according to the page switching event, the displayed target courseware page and the display period corresponding to each target courseware page are confirmed; according to the teaching link information corresponding to the target courseware page, the video outline information of the corresponding display period in the classroom video is marked. According to the common specific teaching links in the teaching activities, each courseware page of the teaching courseware corresponding to the teaching activity is corresponded to a teaching link, and the specific correspondence is obtained by predicting the courseware page in the teaching courseware through the pre-trained teaching link recognition model. In addition, the data source and change event that cause the main display content in the classroom video to change are recorded, and the occurrence time of the change event and the teaching link of the courseware page displayed after the occurrence are confirmed. The teaching link is used as the outline information of the display period of the corresponding courseware page. Without the need to identify the content of the classroom video, the classroom video is accurately divided into structures based on the teaching content and the outline information is marked according to the identification of the conventional links in the teaching activities.

[0087] Figure 6 The structure diagram of a video marking device based on courseware page classification provided in an embodiment of the present application is shown in FIG. Figure 6 As shown, the video marking device based on courseware page classification includes a classroom data acquisition unit 210, a courseware page classification unit 220, a target page confirmation unit 230 and an outline information marking unit 240.

[0088] Among them, the classroom data acquisition unit 210 is used to obtain the teaching courseware played during the classroom video recording and the page switching events of the teaching courseware; the courseware page classification unit 220 is used to identify the teaching link corresponding to each courseware page of the teaching courseware based on the pre-trained teaching link identification model, and obtain the teaching link information corresponding to each courseware page; the target page confirmation unit 230 is used to confirm the displayed target courseware page and the display time period corresponding to each target courseware page according to the page switching event; the outline information marking unit 240 is used to mark the video outline information of the corresponding display time period in the classroom video according to the teaching link information corresponding to the target courseware page.

[0089] Based on the above embodiment, the teaching link recognition model is trained in the following way:

[0090] Acquire first label information added to each courseware page in a plurality of first sample courseware according to a predefined teaching link, each first sample courseware and the corresponding first label information as a first sample data, and obtain a first sample courseware data set;

[0091] The preset language model is trained based on the first sample courseware data set to obtain a teaching link recognition model that takes the teaching courseware as input and outputs a probability distribution of each courseware page in the teaching courseware corresponding to different label information;

[0092] Among them, the teaching session information is confirmed based on the basic output.

[0093] Based on the above embodiment, the outline information marking unit 240 includes:

[0094] An information correction module is used to correct the teaching courseware and the corresponding teaching link information based on the pre-trained link sequence error correction model;

[0095] The information marking module is used to mark the corrected teaching link information corresponding to the target courseware page as the video outline information of the corresponding display period in the classroom video.

[0096] Based on the above embodiment, the link sequence error correction model is trained in the following way:

[0097] Acquire the second label information added to each courseware page in the plurality of second sample courseware according to the predefined teaching link, and the third label information of each second sample courseware, each second sample courseware and the corresponding second label information and third label information are taken as a second sample data, and obtain the second sample courseware data set, the third label information is the label information with the highest probability among the label information obtained by inputting the teaching link recognition model for each courseware page;

[0098] The preset neural network model is trained based on the second sample courseware data set to obtain a link sequence error correction model that takes the teaching courseware and the third label information of the teaching courseware as input and takes the correction result of the third label information corresponding to each courseware page in the teaching courseware as output.

[0099] On the basis of the above-mentioned embodiment, the link sequence error correction model and the teaching link identification model reuse the coding part of the teaching courseware.

[0100] Based on the above embodiment, the outline information marking unit 240 includes:

[0101] The display period merging module is used to merge adjacent display periods with the same teaching link information.

[0102] Based on the above embodiment, the page switching event and the classroom video are recorded based on the same timeline.

[0103] The video marking device based on courseware page classification provided in the embodiment of the present application is included in an electronic device and can be used to execute the corresponding video marking method based on courseware page classification provided in the above embodiment, and has corresponding functions and beneficial effects.

[0104] It is worth noting that in the above-mentioned embodiment of the video marking device based on courseware page classification, the various units and modules included are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention.

[0105] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 7 As shown, the electronic device includes a processor 310 and a memory 320, and may also include an input device 330, an output device 340 and a communication device 350; the number of the processor 310 in the electronic device may be one or more, Figure 7 A processor 310 is taken as an example; the processor 310, the memory 320, the input device 330, the output device 340 and the communication device 350 in the electronic device can be connected via a bus or other means. Figure 7 The example of connecting through bus is taken in the following.

[0106] The memory 320, as a computer-readable storage medium, can be used to store software programs, computer executable programs and modules, such as program instructions / modules corresponding to the video marking method based on courseware page classification in the embodiment of the present application. The processor 310 executes various functional applications and data processing of the electronic device by running the software programs, instructions and modules stored in the memory 320, that is, implements the above-mentioned video marking method based on courseware page classification.

[0107] The memory 320 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 320 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some instances, the memory 320 may further include a memory remotely arranged relative to the processor 310, and these remote memories may be connected to the electronic device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0108] The input device 330 may be used to receive input digital or character information and generate key signal input related to user settings and function control of the electronic device. The output device 340 may include a display device such as a display screen.

[0109] The electronic device mentioned above includes a video marking device based on courseware page classification, which can be used to execute any video marking method based on courseware page classification and has corresponding functions and beneficial effects.

[0110] An embodiment of the present application also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, it is used to perform relevant operations in the video marking method based on courseware page classification provided in any embodiment of the present application, and has corresponding functions and beneficial effects.

[0111] Those skilled in the art should understand that the embodiments of the present application may be provided as methods, systems, or computer program products.

[0112] Therefore, the present application may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the functions in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing device to work in a specific way, so that the instructions stored in the computer-readable memory produce a product including an instruction device, which implements the functions specified in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide for implementing the process in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0113] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory. The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0114] Computer readable media include permanent and non-permanent, removable and non-removable media that can be used to store information by any method or technology. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0115] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0116] Note that the above are only preferred embodiments of the present invention and the technical principles used. Those skilled in the art will understand that the present invention is not limited to the specific embodiments herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present invention, and the scope of the present invention is determined by the scope of the appended claims.

Claims

1. A video tagging method based on courseware page classification, characterized in that: include: Obtain the teaching courseware played during the classroom video recording and the page switching event of the teaching courseware; Based on the pre-trained teaching link recognition model, the teaching link corresponding to each courseware page of the teaching courseware is identified to obtain the teaching link information corresponding to each courseware page; Confirming the displayed target courseware page and the display time period corresponding to each target courseware page according to the page switching event; The video outline information of the corresponding display time period in the classroom video is marked according to the teaching session information corresponding to the target courseware page.

2. The video tagging method based on courseware page classification according to claim 1 is characterized in that: The teaching link recognition model is trained in the following way: Acquire first label information added to each courseware page in a plurality of first sample courseware according to a predefined teaching link, each first sample courseware and the corresponding first label information as a first sample data, and obtain a first sample courseware data set; Based on the first sample courseware data set, a preset language model is trained to obtain a teaching link recognition model that takes the teaching courseware as input and outputs a probability distribution of different label information corresponding to each courseware page in the teaching courseware; Wherein, the teaching session information is confirmed based on the basic output.

3. The video marking method based on courseware page classification according to claim 1 or 2, characterized in that: The step of marking the video outline information of the corresponding display period in the classroom video according to the teaching link information corresponding to the target courseware page includes: Based on the pre-trained link sequence error correction model, the teaching courseware and the corresponding teaching link information are corrected; The corrected teaching session information corresponding to the target courseware page is marked as the video outline information of the corresponding display period in the classroom video.

4. The video marking method based on courseware page classification according to claim 3 is characterized in that: The link sequence error correction model is trained in the following way: Acquire the second label information added to each courseware page in the plurality of second sample courseware according to the predefined teaching link, and the third label information of each of the second sample courseware, and use each second sample courseware and the corresponding second label information and third label information as a second sample data to obtain a second sample courseware data set, wherein the third label information is the label information with the highest probability among the label information obtained by inputting the teaching link recognition model for each courseware page; The preset neural network model is trained based on the second sample courseware data set to obtain a link sequence error correction model that takes the teaching courseware and the third label information of the teaching courseware as input and takes the correction result of the third label information corresponding to each courseware page in the teaching courseware as output.

5. The video marking method based on courseware page classification according to claim 4 is characterized in that: The link sequence error correction model and the teaching link identification model reuse the coding part of the teaching courseware.

6. The video marking method based on courseware page classification according to claim 1 or 2, characterized in that: The step of marking the video outline information of the corresponding display period in the classroom video according to the teaching link information corresponding to the target courseware page includes: Merge adjacent display periods with the same teaching link information.

7. The video marking method based on courseware page classification according to claim 1 or 2, characterized in that: The page switching event is recorded based on the same timeline as the classroom video.

8. A video marking device based on courseware page classification, characterized in that: include: A classroom data acquisition unit, used to acquire the teaching courseware played during the classroom video recording process and the page switching event of the teaching courseware; A courseware page classification unit is used to identify the teaching link corresponding to each courseware page of the teaching courseware based on a pre-trained teaching link recognition model, and obtain the teaching link information corresponding to each courseware page; A target page confirmation unit, used to confirm the displayed target courseware page and the display time period corresponding to each target courseware page according to the page switching event; The outline information marking unit is used to mark the video outline information of the corresponding display time period in the classroom video according to the teaching link information corresponding to the target courseware page.

9. An electronic device, characterized in that: include: one or more processors; a memory for storing one or more computer programs; When the one or more computer programs are executed by the one or more processors, the electronic device implements the video marking method based on courseware page classification as described in any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the video marking method based on courseware page classification as described in any one of claims 1 to 7 is implemented.

Images