Note generation method and device, readable storage medium and computer program product

Abstract

The invention discloses a note generation method and device, a readable storage medium and a computer program product, and the method comprises the steps: obtaining a plurality of text regions of a handout region in a target video and text information corresponding to the voice information of the target video; obtaining a logic relationship between the character areas and character contents of the character areas; determining initial note content according to the logic relation, the text content and the text information; and collecting user behavior data in the playing process of the target video, adjusting the initial note content according to the user behavior data, and generating user note content. According to the scheme, the initial note content is generated on the basis of the text content of the text area corresponding to the handout area and the text information corresponding to the voice information, then the initial note content is adjusted on the basis of the user behavior data, the user note content is generated, notes meeting user requirements can be obtained, and the user experience is improved. And flexible recording of teaching notes of video software is realized.

Description

Technical Field

[0001] The present application belongs to the field of video technology, and in particular, relates to a note generation method, device, readable storage medium and computer program product. Background Art

[0002] Teaching video software has a large number of teaching videos for users to watch and learn, but the courseware in the video is not provided, which is inconvenient for later review and consolidation; even if PPT courseware is provided, its content is numerous and does not form the key points. When looking up knowledge points, you either need to watch the video again or read the entire PPT courseware, which is time-consuming and laborious and not conducive to review.

[0003] Currently, it is possible to generate notes to facilitate users to check knowledge points, but the generated notes are batch-produced in the background and cannot be applied to individual users. Summary of the invention

[0004] The embodiments of the present application provide a note generation method, device, readable storage medium and computer program product to achieve flexible recording of teaching notes for video software and enhance user experience.

[0005] In order to solve the above technical problems, the present application embodiment provides a note generation method, including:

[0006] Acquire multiple text areas in the handout area of ​​the target video and text information corresponding to the voice information of the target video;

[0007] Acquire the logical relationship between the text areas and the text contents of the text areas;

[0008] Determining initial note content according to the logical relationship, the text content, and the text information;

[0009] Collect user behavior data during the target video playback process, adjust the initial note content according to the user behavior data, and generate user note content.

[0010] Optionally, the step of obtaining multiple text regions in a lecture area in a target video includes:

[0011] Performing object detection of the lecture notes on each video frame to obtain the location area of ​​the lecture note content in the video frame, wherein the object detection includes: boundary detection and / or text detection;

[0012] Determine the handout areas of different pages according to the first rule, and obtain at least one page of handout areas;

[0013] Divide each page of the handout area into content areas to obtain multiple text areas corresponding to each page of the handout area;

[0014] The first rule includes at least one of the following:

[0015] If the similarity of the multiple target areas in the position areas in the multiple video frames is greater than or equal to the first threshold, determining that the position areas in the multiple video frames correspond to the same page of handout areas;

[0016] If there are multiple target areas in the position area in at least two video frames whose similarities are less than or equal to the first threshold, it is determined that the at least two video frames correspond to different lecture areas.

[0017] Optionally, performing object detection on each video frame to obtain a location area of ​​the lecture content in the video frame includes:

[0018] Perform target object detection on each of the multiple consecutive video frames;

[0019] If there is a target object, obtaining the target object and position information of the target object in each video frame;

[0020] If it is determined according to the position information that the similarity of the target object in a plurality of consecutive video frames is greater than or equal to a second threshold, deleting the area where the target object is located in the plurality of video frames;

[0021] The target detection of the handout is performed on each video frame where the target object is deleted, and the position area of ​​the handout content in the video frame is obtained.

[0022] Optionally, performing object detection on each video frame to obtain a location area of ​​the lecture content in the video frame includes:

[0023] If the lecture content in the video frame is not in full screen state, the video frame is segmented to obtain multiple segmented areas;

[0024] The segmented area with the largest area is taken as the effective area of ​​the handout;

[0025] The object detection of the handout is performed on the effective area in each video frame to obtain the position area of ​​the handout content in the video frame.

[0026] Optionally, the division of each page of the handout area into content areas to obtain a plurality of text areas corresponding to each page of the handout area includes:

[0027] Perform line detection on the content area of ​​each page of handouts;

[0028] If lines are detected, each page of the handout area is divided into regions according to the detected lines, and multiple text regions corresponding to each page of the handout area are obtained respectively.

[0029] Optionally, the division of each page of the handout area into content areas to obtain a plurality of text areas corresponding to each page of the handout area includes:

[0030] Obtaining a target spacing of text corresponding to each page of the handout area, wherein the target spacing includes row spacing and / or column spacing;

[0031] The content area of ​​each page of the lecture area is divided according to a preset value, and a plurality of divided text areas are obtained, wherein the preset value is obtained based on a target spacing of text corresponding to the lecture area.

[0032] Optionally, the acquiring the logical relationship between the text areas includes the following:

[0033] Obtaining content layout of the divided multiple text areas, and determining the logical relationship between the text areas according to the correspondence between the content layout and the logical relationship;

[0034] Keyword recognition is performed on the voice information of the video frame corresponding to each page of the handout area, and the logical relationship between the text areas is determined based on the recognized keywords.

[0035] Optionally, determining the initial note content according to the logical relationship, the text content and the text information includes:

[0036] Create indexes between handout areas;

[0037] Determine the directory framework based on the indexes between the handout areas and the corresponding logical relationships between the text areas;

[0038] Extracting content based on the text information of the audio information corresponding to the handout area on each page and the text content of the text area, and adding the extracted content to the position indicated by the directory frame to generate initial note content;

[0039] Among them, the note content corresponding to the lecture area in the initial note content is positively correlated with the time length of the lecture area, and the time length of the lecture area is determined according to the start time and end time corresponding to the lecture area obtained according to the video frame rate and the video frame number.

[0040] Optionally, the extracting of content according to the text information of the audio information corresponding to each page of the handout area and the text content of the text area, and adding the extracted content to the position indicated by the directory frame to generate the initial note content includes:

[0041] Acquire target information from the text information of the audio information corresponding to the handout area of ​​each page and the text content, wherein the target information is information that exists in both the text information of the audio information corresponding to the handout area and the text content;

[0042] The target information is added to the location indicated by the directory frame to generate initial note content.

[0043] Optionally, collecting user behavior data during the target video playback process, adjusting the initial note content according to the user behavior data, and generating user note content includes:

[0044] Adjusting the initial note content according to the second rule to generate user note content;

[0045] The second rule includes at least one of the following:

[0046] In the case where the user behavior data includes a mouse position operated by the user, the collected mouse position is associated with a text area of ​​a handout area, a first handout content of the mouse position is obtained, and the content corresponding to the handout area associated with the mouse position in the initial note content is adjusted using the first handout content;

[0047] In the case where the user behavior data includes a fast-forward operation, collecting the second lecture content corresponding to the lecture area corresponding to the fast-forward operation, and deleting the second lecture content in the initial note content;

[0048] In the case where the user behavior data includes a pause operation, the third lecture content corresponding to the mouse stop position of the pause operation is collected. If the mouse stop time corresponding to the pause operation is less than or equal to a third threshold, the third lecture content is used to adjust the initial note content.

[0049] Optionally, the method further includes:

[0050] Based on user operations, obtain the content in the shared notes;

[0051] The content in the shared note is added to the user note content to generate final user note content.

[0052] The embodiment of the present application also provides a video extraction device, including:

[0053] A first acquisition module is used to acquire multiple text areas in the handout area of ​​the target video and text information corresponding to the voice information of the target video;

[0054] A second acquisition module is used to acquire the logical relationship between the text areas and the text contents of the text areas;

[0055] A determination module, used to determine the initial note content according to the logical relationship, the text content and the text information;

[0056] The generation module is used to collect user behavior data during the playback of the target video, adjust the initial note content according to the user behavior data, and generate user note content.

[0057] An embodiment of the present application also provides a note generating device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned note generating method when executing the program.

[0058] An embodiment of the present application also provides a readable storage medium on which a computer program is stored, and when the program is executed by a processor, the steps in the above-mentioned note generation method are implemented.

[0059] An embodiment of the present application also provides a computer program product, including computer instructions, which implement the steps of the above-mentioned note generation method when executed by a processor.

[0060] The beneficial effects of this application are:

[0061] The above scheme generates initial note content based on the text content of the text area corresponding to the lecture area and the text information corresponding to the voice information, and then adjusts the initial note content based on the user behavior data to generate user note content. Notes that meet user needs can be obtained, realizing flexible recording of teaching notes in video software and improving user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Figure 1 It is a flowchart of the note generation method of an embodiment of the present application;

[0063] Figure 2 It is a schematic diagram of line spacing;

[0064] Figure 3 This is one of the content layout diagrams of the handout area;

[0065] Figure 4 This is the second diagram of the content layout of the handout area;

[0066] Figure 5 It is a schematic diagram of a specific implementation process of an embodiment of the present application;

[0067] Figure 6 It is a schematic diagram of the video interface operation;

[0068] Figure 7 It is a schematic diagram of the implementation logic of the embodiment of the present application;

[0069] Figure 8 is a module schematic diagram of a note generating device according to an embodiment of the present application;

[0070] Fig. 9 A structural diagram showing a note generating device according to an embodiment of the present application. DETAILED DESCRIPTION

[0071] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0072] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0073] The note generation method, device, readable storage medium and computer program product provided in the embodiments of the present application are described in detail below with reference to the accompanying drawings through specific embodiments and their application scenarios.

[0074] like Figure 1 As shown, at least one embodiment of the present application provides a note generation method, comprising:

[0075] Step 101, obtaining multiple text areas in the lecture area of ​​the target video and text information corresponding to the voice information of the target video;

[0076] It should be noted that the target video refers to any teaching video that can assist users in learning knowledge. The handout area can be understood as the content area displayed in the target video; for example, if the target video is an explanation of several pages of documents, the handout area refers to the documents, that is, one page of the document is one page of handouts.

[0077] Step 102, obtaining the logical relationship between the text areas and the text contents of the text areas;

[0078] It should be noted that the logical relationship may include but is not limited to: parallel relationship, sequential relationship, progressive relationship, etc.

[0079] Step 103, determining the initial note content according to the logical relationship, the text content and the text information;

[0080] Step 104, collecting user behavior data during the target video playback process, adjusting the initial note content according to the user behavior data, and generating user note content;

[0081] It should be noted that the user behavior data includes but is not limited to at least one of the following:

[0082] The mouse position, pause operation, fast forward operation, and rewind operation of the user's operation.

[0083] It should be noted that the embodiment of the present application generates initial note content based on the text content of the text area corresponding to the lecture area and the text information corresponding to the voice information, and then adjusts the initial note content based on the user behavior data to generate user note content, so as to obtain notes that meet user needs, realize flexible recording of teaching notes of video software, and enhance user experience.

[0084] Optionally, in one implementation, the specific implementation of obtaining multiple text areas in the lecture area in the target video includes steps a1, a2 and a3, specifically:

[0085] Step a1, performing object detection of the lecture notes on each video frame to obtain the position area of ​​the lecture notes content in the video frame, wherein the object detection includes: boundary detection and / or text detection;

[0086] It should be noted that steps a1 and a2 are for extracting the handout area of ​​each page in the target video.

[0087] It should be noted that before processing the video frame, it is necessary to clean up the information in the video frame to ensure that the acquired handout area is valid for the user. Optionally, in one implementation, the specific implementation of step a1 includes:

[0088] Perform target object detection on each of the multiple consecutive video frames; if the target object exists, obtain the target object and the position information of the target object in each video frame; based on the position information, if it is determined that the similarity of the target objects in the multiple consecutive video frames is greater than or equal to a second threshold, delete the area where the target object is located in the multiple video frames; perform handout target detection on each video frame where the target object is deleted, and obtain the position area of ​​the handout content in the video frame.

[0089] Optionally, if the similarity of the target objects in the consecutive multiple video frames is greater than or equal to the second threshold, it can be understood that the target objects in the consecutive multiple video frames are the same or similar.

[0090] It should be noted that this case is to remove a specific object in the video frame. For example, the target object may be a face, that is, the image of the teacher teaching the class needs to be removed. Optionally, the target object may be pre-set or may be set by the user based on the target video.

[0091] For example, in the embodiment of the present application, it is necessary to remove the occlusion of the teacher's body. Specifically, face detection is first performed on the continuous video frames. If a face exists, the face content and the position information of the selected face in the video frame are recorded. When the data of the continuous video frames are the same or similar, the face is determined to be the teacher, and the area is deducted when the video frames are analyzed later.

[0092] It should be noted that, in one case, the handout in the video frame may not be in full screen state, but only occupy a part of the area in the video frame. In this case, it is necessary to clear the invalid area and obtain the valid area occupied by the handout, so that the subsequent processing of the handout is based on the valid handout; optionally, in one implementation, the specific implementation of step a1 includes:

[0093] If the handout content in the video frame is not in full screen state, the video frame is segmented to obtain multiple segmented areas; the segmented area with the largest area is used as the effective area of ​​the handout; the handout target is detected in the effective area in each video frame to obtain the position area of ​​the handout content in the video frame.

[0094] It should be noted that, generally speaking, the presentation of the lecture content is the most important part of the video frame, and the space it occupies should also be the largest. In this implementation method, the area with the largest area after segmentation is taken out as the effective area of ​​the lecture, and the location area of ​​the lecture content is subsequently obtained based on the effective area.

[0095] For example, taking the target video explaining a PPT document as an example, when the PPT document is not in full screen state, three lines are used for trial segmentation: left, bottom and right (there is a default dividing line for the handouts in full screen and non-full screen states). If a dividing line is found, the PPT is considered to be in non-full screen state, and the larger area after the video frame is divided is taken as the valid handout area.

[0096] Step a2, determining the lecture areas of different pages according to the first rule, and obtaining at least one page of lecture areas;

[0097] Optionally, in this step, at least one target area in the position area in different video frames needs to be obtained respectively;

[0098] Optionally, the target area may be a specific area in a pre-set location area, such as a central area in the location area, an area at a specific distance from an edge of the location area, etc.; optionally, the target area may also be set based on a user's selection.

[0099] Optionally, in the embodiment of the present application, before selecting the target area, it is necessary to first normalize the size of the position areas in different video frames to ensure the validity of the target area selected by the position areas in different video frames.

[0100] Then, at least one target area within the position area in different video frames is compared;

[0101] Optionally, the comparison of target areas in the embodiments of the present application may be understood as a comparison of similarities of target areas in different video frames.

[0102] In the comparison process, the handout areas of different pages can be determined according to the first rule, and at least one handout area of ​​one page can be obtained;

[0103] Optionally, the first rule includes at least one of the following:

[0104] B11, if the similarity of multiple target areas in the position areas in the multiple video frames is greater than or equal to the first threshold, then determining that the position areas in the multiple video frames correspond to the same page of handout areas;

[0105] Optionally, when the similarity of multiple target areas in the position area in multiple video frames is greater than or equal to a first threshold, it can be understood that the multiple target areas in the position area in the multiple video frames are close or identical. In this case, it means that the compared multiple target areas are the same lecture area located in different video frames.

[0106] B12, if the similarity of multiple target areas in the position area in at least two video frames is less than or equal to the first threshold, determine that at least two video frames correspond to different handout areas;

[0107] Optionally, when the similarity of multiple target areas in the position area in multiple video frames is less than a first threshold, it can be understood that the multiple target areas in the position area in the multiple video frames are not similar or identical. In this case, it means that the multiple target areas for comparison are different lecture areas located in different video frames.

[0108] Step a3, dividing each page of the handout area into content areas, and obtaining a plurality of text areas corresponding to each page of the handout area;

[0109] It should be noted that after obtaining the lecture area of ​​each page, in order to extract more accurate text content of the lecture area, it is necessary to divide the lecture area of ​​each page into regions, and then extract text content based on the divided regions.

[0110] Optionally, in one implementation, the specific implementation of dividing each page of the handout area into content areas to obtain multiple text areas corresponding to each page of the handout area includes:

[0111] Step a301, performing line detection on the content area of ​​each page of the handout area;

[0112] It should be noted that the line can be understood as a line corresponding to a template used when making the lecture area or a line of a control (eg, a square box, a triangle box, etc.) added to the lecture area.

[0113] Step a302: if lines are detected, each page of the handout area is divided into regions according to the detected lines, and a plurality of text regions corresponding to each page of the handout area are obtained respectively;

[0114] In this case, it can be understood that the corresponding template or control is set in the handout area. This step is to divide the handout area based on the lines of the template or control to obtain the corresponding text area.

[0115] Optionally, in another implementation, the specific implementation of dividing each page of the handout area into content areas to obtain multiple text areas corresponding to each page of the handout area includes:

[0116] Step a311, obtaining the target spacing of the text corresponding to each page of the handout area, wherein the target spacing includes row spacing and / or column spacing;

[0117] It should be noted that this method uses the text spacing in the handout area to divide the area. Because some text is filled in by row and some is filled in by column, this step needs to obtain the corresponding spacing based on the way the text is filled in.

[0118] For example, try to horizontally select the text in the handout area, calculate the distance difference between multiple adjacent boxes in the vertical direction (depicted in pixels), and get Hds, as follows Figure 2As shown, Hds is the distance difference between adjacent boxes in the vertical direction. If Hds is empty, try to select the text vertically, calculate the distance difference between multiple adjacent boxes in the horizontal direction, and get Vds, which is the distance difference between adjacent boxes in the horizontal direction. The width values ​​of multiple boxes are formed into a list Dws. Because Dws is relatively discrete, it is necessary to integrate it first and then count the mode. Therefore, it is necessary to compress the three consecutive values ​​in the list Dws to the middle value. For example, 3.2 and 2.9 are different values, but they can both be represented by 3. Assuming that 3.2 appears 2 times and 2.9 appears 1 time, when it is represented by 3, it appears 2 times + 1 time = 3 times. After compression and integration, take the mode, and the value of the mode is meaningful. Example: 3, 4, 5, 6, 7, 8, 9, because 3, 4, 5 are continuous, 6, 7, 8 are continuous, so after merging 3, 4, 5, and merging 6, 7, 8, their number of occurrences is also merged. After merging, the updated mode of the list Dws can be obtained, and the updated mode is set as the word height WordH. The non-empty Hds and Vds are merged with a granularity of one-third WordH, in order to control the granularity of the discretized values. For example, three values ​​are finally obtained, namely Hds-1, Hds-2, and Hds-3. In fact, Hds does not have only three values ​​(the logic is the same as Dws compression to take the middle value); then the mode is taken to obtain the line spacing of the characters in the handout P; similarly, the method of obtaining the column spacing is similar to the method of obtaining the line spacing, which will not be repeated here.

[0119] It should be noted that before obtaining the target spacing of the text, the background color of the handout area needs to be removed. The specific implementation is: first remove the text from the handout area, obtain the background color of the handout area, and save the background color to the background color list.

[0120] Step a312, dividing the content area of ​​each page of the handout area according to a preset value, and obtaining a plurality of divided text areas, wherein the preset value is obtained based on a target spacing of text corresponding to the handout area;

[0121] Optionally, the preset value may be a preset multiple of the target spacing, for example, the preset multiple is a value greater than 1.

[0122] It should be noted that after the target spacing is obtained, the text area can be divided based on the target spacing. For example, the handout area is divided using the line width as a preset value, and each divided area is a text area.

[0123] Optionally, when dividing, the background color in the background color list is taken as a line, and the line width is a preset value to divide the handout area to obtain a plurality of text areas after division.

[0124] It should be noted that the above-mentioned two implementation methods of dividing the content area of ​​each page of the lecture area to obtain multiple text areas corresponding to each page of the lecture area can be implemented separately or used in combination. For example, the content area of ​​each page of the lecture area is first detected for lines. If lines are detected, the area of ​​each page of the lecture area is divided according to the detected lines, and the multiple text areas corresponding to each page of the lecture area are obtained respectively; if no lines are detected, the target spacing of the text corresponding to each page of the lecture area is obtained, and the target spacing includes row spacing and / or column spacing. The content area of ​​each page of the lecture area is divided according to preset values, and the multiple text areas after division are obtained.

[0125] For example, perform line detection on the handout area, segment closed lines or controls, and obtain a list of segmented areas, that is, a list of text areas. If line detection fails, the list of segmented areas is empty, and the handout area is still a whole without any area division. In this case, it is necessary to obtain the row spacing and / or column spacing, and divide the handout area based on the obtained row spacing and / or column spacing to obtain multiple text areas that are finally divided.

[0126] It should be noted that in the embodiment of the present application, optical character recognition (OCR) technology is used to extract text from the lecture area. If the lecture area is not divided into text areas first, the text finally obtained may be disordered (because OCR is scanned by line, the same line may belong to different areas. If the area division is not performed, the recognized text will be inaccurate). In the embodiment of the present application, the area division is first performed, and then the OCR technology is used to obtain the text content of each text area, so as to ensure that the full amount of text obtained in the lecture area is accurate, thereby improving the accuracy of text content extraction.

[0127] Optionally, in one implementation, the specific implementation of obtaining the logical relationship between the text regions includes the following:

[0128] C11, obtaining content layout of the multiple divided text areas, and determining the logical relationship between the text areas according to the correspondence between the content layout and the logical relationship;

[0129] It should be noted that since some lecture areas are made according to templates (i.e. content layout), different templates will correspond to specific logical relationships. In this case, the logical relationship between different text areas is determined based on the correspondence between the template and the logical relationship.

[0130] For example, there can be a four-paragraph content layout with a regular / irregular layout. For example, Figure 3The content layout of the middle lecture area 31 is regular Tianzi distribution; vertical distribution (regular / stepped / bevel deformation), Figure 4 The content layout of the middle lecture area 41 is regular vertical distribution; horizontal distribution (regular / irregular), and annular distribution (base / center dispersion). Figure 3 and Figure 4 The corresponding content layout may correspond to a parallel relationship; the corresponding content layout of the circular distribution may correspond to a progressive relationship.

[0131] C12, performing keyword recognition on the voice information of the video frame corresponding to each page of the handout area, and determining the logical relationship between the text areas according to the recognized keywords;

[0132] It should be noted that in the process of explaining the handouts, when explaining different contents in the voice of the lecturer, some relational words may be used to connect the front and back. In this case, the keywords in the voice information are used to determine the logical relationship. For example, the logical relationship between the text areas divided by the area is processed in parallel by default, and the change of the logical relationship is triggered in the form of hot words; the voice when the frame is displayed is subjected to keyword recognition, and the so-called keywords are the hot word list, such as then, secondly, these keywords. When the lecturer says these words, it is considered to be a parallel relationship; when the first step, the second step, these keywords appear, it is considered to be a sequential relationship; when the Xth year, established in, etc., these keywords appear, it is considered to be a progressive relationship; when no logical hot words are matched, it is processed as a parallel relationship.

[0133] Optionally, in one implementation, the specific implementation of determining the initial note content according to the logical relationship, the text content and the text information includes steps b1 to b3, specifically:

[0134] Step b1, establishing indexes between lecture areas;

[0135] It should be noted that the index is used for subsequent association between handouts. For example, the page number of the handout area can be used as an aid to establish an index for the handout area, that is, the page number of the handout area is the corresponding index; for example, when the page number of the handout area cannot be extracted, the index can be determined based on the order in which the handout area appears in the video.

[0136] Step b2, determining the directory framework according to the indexes between the handout areas and the corresponding logical relationships between the text areas;

[0137] It should be noted that the directory framework can be understood as the directory of the notes, through which the content of a specific lecture area can be located.

[0138] Step b3, extracting content according to the text information of the audio information corresponding to the handout area of ​​each page and the text content of the text area, adding the extracted content to the position indicated by the directory frame to generate initial note content;

[0139] Among them, the note content corresponding to the lecture area in the initial note content is positively correlated with the time length of the lecture area, and the time length of the lecture area is determined according to the start time and end time corresponding to the lecture area obtained according to the video frame rate and the video frame number.

[0140] Optionally, in one implementation, the specific implementation of extracting content according to the text information of the audio information corresponding to each page of the handout area and the text content of the text area, adding the extracted content to the position indicated by the directory frame, and generating the initial note content includes:

[0141] Acquire target information from the text information of the audio information corresponding to the handout area of ​​each page and the text content, wherein the target information is information that exists in both the text information of the audio information corresponding to the handout area and the text content;

[0142] The target information is added to the location indicated by the directory frame to generate initial note content.

[0143] It should be noted that after obtaining the directory framework, the text content in the notes is filled in. In the embodiment of the present application, the text information of the audio information and the text content corresponding to the text area are used to extract the content, and then the extracted content is added to the position indicated by the directory framework to realize the construction of the initial note content.

[0144] Optionally, in one implementation, the collecting of user behavior data during the target video playback process, adjusting the initial note content according to the user behavior data, and generating the user note content specifically includes:

[0145] Adjusting the initial note content according to the second rule to generate user note content;

[0146] The second rule includes at least one of the following:

[0147] D11, when the user behavior data includes the mouse position of the user operation, the collected mouse position is associated with the text area of ​​the lecture area, the first lecture content of the mouse position is obtained, and the content corresponding to the lecture area associated with the mouse position in the initial note content is adjusted by using the first lecture content;

[0148] Optionally, one adjustment method is to directly add the first lecture content at the mouse position to the corresponding position in the initial note content; optionally, another adjustment method is to compare the first lecture content with the content at the corresponding position in the initial note content, and add the content of the first lecture content that does not exist in the initial note content to the initial note content.

[0149] D12. When the user behavior data includes a fast-forward operation, collect the second lecture content corresponding to the lecture area corresponding to the fast-forward operation, and delete the second lecture content in the initial note content;

[0150] It should be noted that fast forwarding indicates that the user does not need to make additional records for this part of the content. At this time, the second lecture content that already exists in the initial note content can be deleted.

[0151] D13, when the user behavior data includes a pause operation, collecting the third lecture content corresponding to the mouse stop position of the pause operation, and if the mouse stop time corresponding to the pause operation is less than or equal to a third threshold, adjusting the initial note content using the third lecture content;

[0152] It should be noted that this method is to judge the pause operation. If the mouse dwell time corresponding to the pause operation is less than or equal to the third threshold, it indicates that the user is viewing the video content. At this time, it can be understood that this part of the content is the focus, and the third lecture content at the mouse dwell position can be directly added to the corresponding position in the initial note content; the third lecture content can also be compared with the content at the corresponding position in the initial note content, and the content of the third lecture content that does not exist in the initial note content is added to the initial note content. If the mouse dwell time corresponding to the pause event is greater than the third threshold, it indicates that the user may have paused the viewing of the video, and the initial note content is not adjusted at this time.

[0153] Optionally, in one implementation, the method further includes:

[0154] Based on user operations, obtain the content in the shared notes;

[0155] The content in the shared note is added to the user note content to generate final user note content.

[0156] Optionally, in this case, it can be understood that after obtaining the user notes, the user can view the user note content. At the same time, after the user obtains the content in the shared notes, if it is determined that the content in the shared notes also needs to be recorded, then the content to be recorded in the shared notes is selected, and this part of the content will be automatically added to the corresponding position in the user note content.

[0157] Optionally, after obtaining the end-user note content, the end-user note content may be uploaded to a cloud server with the identifier of the target video as an index to achieve sharing of the user notes.

[0158] Optionally, in one implementation, after obtaining the user note content, the user note content may be uploaded to a cloud server with the identifier of the target video as an index to achieve sharing of the user notes.

[0159] In summary, the embodiment of the present application proposes a method for automatically generating teaching notes for a teaching video while watching the video for learning. It starts with collecting the user's viewing behavior through user authorization, and generates initial note content by combining the video screen content and the lecturer's voice data at the current progress. Then, the user behavior data and shared notes are superimposed to generate user-exclusive teaching notes for convenient subsequent review.

[0160] When users enter the teaching video viewing interface to study, they can choose to turn on the function of generating exclusive notes (off by default). When turned on, the user will be prompted to authorize the function to collect the user's viewing behavior; the system analyzes the content of the video screen according to the progress of the teaching video, and processes the voice data of the lecturer at that progress to generate the initial note content; collects user behavior data during video playback (including but not limited to: pause operation, mouse position, mouse dwell time, fast-forward operation (such as progress bar drag event), etc.) to generate user-exclusive focus points and obtain user note content (which can be understood as advanced notes); when there are available shared notes for the video, finally integrate the shared notes on the basis of the advanced notes to generate the final user note content (i.e., user-exclusive teaching notes).

[0161] Taking video teaching of PPT documents as an example, the specific application of the embodiments of the present application is illustrated as follows.

[0162] like Figure 5 As shown, the specific implementation process may include:

[0163] Step S51: The user enters the teaching video viewing interface, chooses to generate exclusive notes, and authorizes the system to track and collect the user's viewing behavior;

[0164] Specifically, the user enters the teaching video viewing interface and selects a certain episode of the video to watch. Figure 6As shown, on the left is the video interface 61 being played, and in the video list on the right, there is a button 62 for generating notes in the episode being played, for the user to click to enable the automatic generation of teaching notes function. When the user clicks the button for generating notes, the system will pop up an authorization window to prompt the user to authorize. When the user clicks OK to complete the authorization, the automatic generation of teaching notes function begins to work. At the same time, a note status bar 63 is displayed in the status bar below the video interface 61. Initially, when the function of generating notes is not enabled, this note status bar 63 is not displayed.

[0165] Step S52: The user watches the video, and the system analyzes the video and audio information to generate initial note content;

[0166] The specific steps include:

[0167] The first step is to obtain the frame data of the teaching video in real time in the background. For the automatic generation of teaching notes system integrated in the video platform, this data is known; for the generation note system used independently outside the platform, it is obtained by regularly taking screenshots and uploading video frames. Similar to the screen recording program, the process of obtaining video frame data is not explained in detail.

[0168] Step 2: Identify the handouts and obtain at least one page of handout area;

[0169] It should be noted that by determining the handout area, the start time and end time of the handout area on each page of the teaching video can also be extracted, and the importance of the content can be characterized by the length of time spent.

[0170] In order to accurately obtain the content of the lecture, necessary image processing is performed on the video frames, that is, specific content cleaning, that is, cleaning the target objects and cleaning the non-valid areas to obtain only the valid areas.

[0171] After the cleaning is completed, the boundary detection and / or text detection of the handouts is performed on each video frame to obtain the position area of ​​the handout content in the video frame; at least one target area in the position area in different video frames is compared; and the handout areas of different pages are determined according to the first rule mentioned above to obtain at least one page of handout areas.

[0172] Step 3: Obtain logical relationships;

[0173] The method mainly includes: dividing each page of the lecture area into content areas, obtaining multiple text areas corresponding to each page of the lecture area, and then obtaining the logical relationship between the multiple text areas.

[0174] Step 4: Obtain text contents corresponding to the multiple text areas divided in step 3;

[0175] Step 5: Establish indexes between lecture areas, and determine the directory framework based on the indexes between lecture areas and the logical relationships between multiple text areas;

[0176] Step 6: Extract content based on the text information of the audio information corresponding to the handout area of ​​each page and the text content of the text area, add the extracted content to the position indicated by the directory frame, and generate initial note content.

[0177] Step S53: adjusting the initial note content according to the collected user behavior data to generate user note content;

[0178] The main process includes the following:

[0179] The first step is to obtain the real-time position of the mouse and upload it to the background server, and store the continuous duration of a point.

[0180] Step 2: Associate the mouse position information with the position of the PPT handout. Special processing is done for the correspondence in non-full screen situations to obtain the focus of the user's perspective. The default rule is that the mouse position expresses the area where the user's eyes are looking.

[0181] Step 3: Get the lecture content at the mouse position and correct the initial note content.

[0182] Step 4. When the user performs playback operations such as pause, fast forward, and rewind, the corresponding operation callback is triggered. When the operation is paused, the mouse position information is extracted. When the mouse position information remains unchanged for a long time (this time is a predetermined threshold), the system exits, that is, it does not pay attention to this pause event. Otherwise, the pause event is judged as a meaningful subjective event. Recall the video frame processing service, for example, extract the handout content corresponding to the mouse position of the pause event, and add it to the initial note content; optionally, the length of the extracted handout content is related to the duration of the pause event. The longer the time, the more text content is extracted.

[0183] The fast forward operation is to delete the relevant content in the initial note content generated in step S52 (if it has been generated).

[0184] The back operation is handled the same as the pause operation, but there is no need to extract the mouse position information for classification judgment.

[0185] Step S54: Generate end-user note content based on the shared note;

[0186] This step is an interactive, shared and scalable process.

[0187] The so-called shared note library is a collection of existing teaching notes. The automatically generated teaching notes that belong to user A can be shared by A (if A agrees to share) and added to the shared note library, which is stored in the cloud service with the video ID and name as index. When subsequent users choose to generate notes, they can refer to the existing notes to form an automatic iteration, upgrade and improvement of the notes.

[0188] Combined with the above description, the implementation logic of the embodiment of the present application is as follows Figure 7 As shown, it should be noted that retrieving data is optional, and large model capabilities need to be introduced to support the expansion of note content and enrich references.

[0189] Compared with the existing solutions, the embodiments of the present application have the following advantages:

[0190] 1. The embodiment of the present application can automatically generate teaching notes to facilitate later review and learning; during the viewing process, users have different operating behaviors (users' subjective focus is different). By recording such behaviors, the generated notes can be tailored to each user. For example, in a computer network course, some people are very familiar with IP addresses, while others are unfamiliar with them. Their time spent and operating behaviors will be different. For example, users who are familiar with them will choose to fast forward, while users who are unfamiliar with them will consider pausing. Therefore, the details of the IP addresses in the notes will be different.

[0191] 2. The embodiment of the present application can be independent of the video platform and integrated as a plug-in, and can be used independently outside the video platform.

[0192] 3. The embodiment of the present application first divides the content of the lecture notes into regions, and then performs OCR recognition based on the divided regions, thereby solving the problem of text confusion.

[0193] 4. The embodiment of the present application does not simply take screenshots of key frames of the video as a resource for lecture notes. In the embodiment of the present application, unnecessary content of the video frame is removed (such as the removal of the lecturer and the removal of non-valid areas in non-full-screen state), which can ensure that the notes obtained are more accurate.

[0194] like Figure 8 As shown, at least one embodiment of the present application further provides a note generating device 800, comprising:

[0195] The first acquisition module 801 is used to acquire multiple text areas in the lecture area of ​​the target video and text information corresponding to the voice information of the target video;

[0196] A second acquisition module 802 is used to acquire the logical relationship between the text regions and the text contents of the text regions;

[0197] A determination module 803, for determining the initial note content according to the logical relationship, the text content and the text information;

[0198] The generation module 804 is used to collect user behavior data during the playback of the target video, adjust the initial note content according to the user behavior data, and generate user note content.

[0199] Optionally, the first acquisition module 801 includes:

[0200] A first acquisition unit is used to perform object detection of the lecture notes on each video frame to obtain a location area of ​​the lecture notes content in the video frame, wherein the object detection includes: boundary detection and / or text detection;

[0201] A second acquisition unit is used to determine the areas of the handouts on different pages according to the first rule, and acquire the area of ​​at least one page of the handouts;

[0202] The third acquisition unit is used to divide the content area of ​​each page of the lecture area to obtain multiple text areas corresponding to each page of the lecture area;

[0203] The first rule includes at least one of the following:

[0204] If the similarity of the multiple target areas in the position areas in the multiple video frames is greater than or equal to the first threshold, determining that the position areas in the multiple video frames correspond to the same page of handout areas;

[0205] If there are multiple target areas in the position area in at least two video frames whose similarities are less than or equal to the first threshold, it is determined that the at least two video frames correspond to different lecture areas.

[0206] Optionally, the first acquiring unit is used to:

[0207] Perform target object detection on each of the multiple consecutive video frames;

[0208] If there is a target object, obtaining the target object and position information of the target object in each video frame;

[0209] If it is determined according to the position information that the similarity of the target object in a plurality of consecutive video frames is greater than or equal to a second threshold, deleting the area where the target object is located in the plurality of video frames;

[0210] The target detection of the handout is performed on each video frame where the target object is deleted, and the position area of ​​the handout content in the video frame is obtained.

[0211] Optionally, the first acquiring unit is used to:

[0212] If the lecture content in the video frame is not in full screen state, the video frame is segmented to obtain multiple segmented areas;

[0213] The segmented area with the largest area is taken as the effective area of ​​the handout;

[0214] The object detection of the handout is performed on the effective area in each video frame to obtain the position area of ​​the handout content in the video frame.

[0215] Optionally, the third acquiring unit is used to:

[0216] Perform line detection on the content area of ​​each page of handouts;

[0217] If lines are detected, each page of the handout area is divided into regions according to the detected lines, and multiple text regions corresponding to each page of the handout area are obtained respectively.

[0218] Optionally, the third acquiring unit is used to:

[0219] Obtaining a target spacing of text corresponding to each page of the handout area, wherein the target spacing includes row spacing and / or column spacing;

[0220] The content area of ​​each page of the lecture area is divided according to a preset value, and a plurality of divided text areas are obtained, wherein the preset value is obtained based on a target spacing of text corresponding to the lecture area.

[0221] Optionally, the second acquisition module 802 includes one of the following:

[0222] A fourth acquisition unit is used to acquire content layout of the divided multiple text areas, and determine the logical relationship between the text areas according to the correspondence between the content layout and the logical relationship;

[0223] The first determination unit is used to perform keyword recognition on the voice information of the video frame corresponding to each page of the handout area, and determine the logical relationship between the text areas according to the recognized keywords.

[0224] Optionally, the determining module 803 includes:

[0225] Establishing units for indexing between handout areas;

[0226] A second determining unit is used to determine the directory framework according to the indexes between the handout areas and the logical relationship corresponding to the text areas;

[0227] A generating unit, configured to extract content according to the text information of the audio information corresponding to the handout area of ​​each page and the text content of the text area, and add the extracted content to the position indicated by the directory frame to generate initial note content;

[0228] Among them, the note content corresponding to the lecture area in the initial note content is positively correlated with the time length of the lecture area, and the time length of the lecture area is determined according to the start time and end time corresponding to the lecture area obtained according to the video frame rate and the video frame number.

[0229] Optionally, the generating unit is used to:

[0230] Acquire target information from the text information of the audio information corresponding to the handout area of ​​each page and the text content, wherein the target information is information that exists in both the text information of the audio information corresponding to the handout area and the text content;

[0231] The target information is added to the location indicated by the directory frame to generate initial note content.

[0232] Optionally, the generating module 804 is used to:

[0233] Adjusting the initial note content according to the second rule to generate user note content;

[0234] The second rule includes at least one of the following:

[0235] In the case where the user behavior data includes a mouse position operated by the user, the collected mouse position is associated with a text area of ​​a handout area, a first handout content of the mouse position is obtained, and the content corresponding to the handout area associated with the mouse position in the initial note content is adjusted using the first handout content;

[0236] In the case where the user behavior data includes a fast-forward operation, collecting the second lecture content corresponding to the lecture area corresponding to the fast-forward operation, and deleting the second lecture content in the initial note content;

[0237] In the case where the user behavior data includes a pause operation, the third lecture content corresponding to the mouse stop position of the pause operation is collected. If the mouse stop time corresponding to the pause operation is less than or equal to a third threshold, the third lecture content is used to adjust the initial note content.

[0238] Optionally, the device further comprises:

[0239] The third acquisition module is used to acquire the content in the shared note based on the user operation;

[0240] The content in the shared note is added to the user note content to generate final user note content.

[0241] It should be noted that if the device provided by at least one embodiment of the present application is a device capable of executing the above-mentioned note generation method, then all embodiments of the above-mentioned note generation method are applicable to the device and can achieve the same or similar beneficial effects.

[0242] At least one embodiment of the present application also provides a note generating device, including a transceiver and a processor;

[0243] The processor is used to: obtain multiple text areas in the handout area of ​​the target video and text information corresponding to the voice information of the target video;

[0244] Acquire the logical relationship between the text areas and the text contents of the text areas;

[0245] Determining initial note content according to the logical relationship, the text content, and the text information;

[0246] Collect user behavior data during the target video playback process, adjust the initial note content according to the user behavior data, and generate user note content.

[0247] Optionally, the processor is configured to:

[0248] Performing object detection of the lecture notes on each video frame to obtain the location area of ​​the lecture note content in the video frame, wherein the object detection includes: boundary detection and / or text detection;

[0249] Determine the handout areas of different pages according to the first rule, and obtain at least one page of handout areas;

[0250] Divide each page of the handout area into content areas to obtain multiple text areas corresponding to each page of the handout area;

[0251] The first rule includes at least one of the following:

[0252] If the similarity of the multiple target areas in the position areas in the multiple video frames is greater than or equal to the first threshold, determining that the position areas in the multiple video frames correspond to the same page of handout areas;

[0253] If there are multiple target areas in the position area in at least two video frames whose similarities are less than or equal to the first threshold, it is determined that the at least two video frames correspond to different lecture areas.

[0254] Optionally, the processor is configured to:

[0255] Perform target object detection on each of the multiple consecutive video frames;

[0256] If there is a target object, obtaining the target object and position information of the target object in each video frame;

[0257] If it is determined according to the position information that the similarity of the target object in a plurality of consecutive video frames is greater than or equal to a second threshold, deleting the area where the target object is located in the plurality of video frames;

[0258] The target detection of the handout is performed on each video frame where the target object is deleted, and the position area of ​​the handout content in the video frame is obtained.

[0259] Optionally, the processor is configured to:

[0260] If the lecture content in the video frame is not in full screen state, the video frame is segmented to obtain multiple segmented areas;

[0261] The segmented area with the largest area is taken as the effective area of ​​the handout;

[0262] The object detection of the handout is performed on the effective area in each video frame to obtain the position area of ​​the handout content in the video frame.

[0263] Optionally, the processor is configured to:

[0264] Perform line detection on the content area of ​​each page of handouts;

[0265] If lines are detected, each page of the handout area is divided into regions according to the detected lines, and multiple text regions corresponding to each page of the handout area are obtained respectively.

[0266] Optionally, the processor is configured to:

[0267] Obtaining a target spacing of text corresponding to each page of the handout area, wherein the target spacing includes row spacing and / or column spacing;

[0268] The content area of ​​each page of the lecture area is divided according to a preset value, and a plurality of divided text areas are obtained, wherein the preset value is obtained based on a target spacing of text corresponding to the lecture area.

[0269] Optionally, the processor is configured to implement one of the following:

[0270] Obtaining content layout of the divided multiple text areas, and determining the logical relationship between the text areas according to the correspondence between the content layout and the logical relationship;

[0271] Keyword recognition is performed on the voice information of the video frame corresponding to each page of the handout area, and the logical relationship between the text areas is determined based on the recognized keywords.

[0272] Optionally, the processor is configured to:

[0273] Create indexes between handout areas;

[0274] Determine the directory framework based on the indexes between the handout areas and the corresponding logical relationships between the text areas;

[0275] Extracting content based on the text information of the audio information corresponding to the handout area on each page and the text content of the text area, and adding the extracted content to the position indicated by the directory frame to generate initial note content;

[0276] Among them, the note content corresponding to the lecture area in the initial note content is positively correlated with the time length of the lecture area, and the time length of the lecture area is determined according to the start time and end time corresponding to the lecture area obtained according to the video frame rate and the video frame number.

[0277] Optionally, the processor is configured to:

[0278] Acquire target information from the text information of the audio information corresponding to the handout area of ​​each page and the text content, wherein the target information is information that exists in both the text information of the audio information corresponding to the handout area and the text content;

[0279] The target information is added to the location indicated by the directory frame to generate initial note content.

[0280] Optionally, the processor is configured to:

[0281] Adjusting the initial note content according to the second rule to generate user note content;

[0282] The second rule includes at least one of the following:

[0283] In the case where the user behavior data includes a mouse position operated by the user, the collected mouse position is associated with a text area of ​​a handout area, a first handout content of the mouse position is obtained, and the content corresponding to the handout area associated with the mouse position in the initial note content is adjusted using the first handout content;

[0284] In the case where the user behavior data includes a fast-forward operation, collecting the second lecture content corresponding to the lecture area corresponding to the fast-forward operation, and deleting the second lecture content in the initial note content;

[0285] In the case where the user behavior data includes a pause operation, the third lecture content corresponding to the mouse stop position of the pause operation is collected. If the mouse stop time corresponding to the pause operation is less than or equal to a third threshold, the third lecture content is used to adjust the initial note content.

[0286] Optionally, the processor is further configured to:

[0287] Based on user operations, obtain the content in the shared notes;

[0288] The content in the shared note is added to the user note content to generate final user note content.

[0289] like Fig. 9 As shown, the embodiment of the present invention further provides a video extraction device, including a processor 900, a transceiver 910, a memory 920, and a program stored in the memory 920 and executable on the processor 900; wherein the transceiver 910 is connected to the processor 900 and the memory 920 through a bus interface, wherein the processor 900 is used to read the program in the memory and execute the following process:

[0290] Acquire multiple text areas in the handout area of ​​the target video and text information corresponding to the voice information of the target video;

[0291] Acquire the logical relationship between the text areas and the text contents of the text areas;

[0292] Determining initial note content according to the logical relationship, the text content, and the text information;

[0293] Collect user behavior data during the target video playback process, adjust the initial note content according to the user behavior data, and generate user note content.

[0294] The transceiver 910 is configured to receive and send data under the control of the processor 900 .

[0295] Among them, Fig. 9 In the embodiment, the bus architecture may include any number of interconnected buses and bridges, specifically linking together various circuits of one or more processors represented by processor 900 and memory represented by memory 920. The bus architecture may also link together various other circuits such as peripherals, voltage regulators, and power management circuits, which are well known in the art and are therefore not further described herein. The bus interface provides an interface. The transceiver 910 may be a plurality of components, namely, a transmitter and a receiver, providing a unit for communicating with various other devices on a transmission medium, including transmission media such as wireless channels, wired channels, optical cables, and the like.

[0296] The processor 900 is responsible for managing the bus architecture and general processing, and the memory 920 can store data used by the processor 900 when performing operations.

[0297] Optionally, the processor 900 may be a CPU (central processing unit), an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array) or a CPLD (Complex Programmable Logic Device), and the processor may also adopt a multi-core architecture.

[0298] The processor calls the computer program stored in the memory to execute any of the methods provided in the embodiments of the present application according to the obtained executable instructions. The processor and the memory can also be arranged physically separately.

[0299] Furthermore, when the processor 900 executes the program, the following steps are implemented:

[0300] Performing object detection of the lecture notes on each video frame to obtain the location area of ​​the lecture note content in the video frame, wherein the object detection includes: boundary detection and / or text detection;

[0301] Determine the handout areas of different pages according to the first rule, and obtain at least one page of handout areas;

[0302] Divide each page of the handout area into content areas to obtain multiple text areas corresponding to each page of the handout area;

[0303] The first rule includes at least one of the following:

[0304] If the similarity of the multiple target areas in the position areas in the multiple video frames is greater than or equal to the first threshold, determining that the position areas in the multiple video frames correspond to the same page of handout areas;

[0305] If there are multiple target areas in the position area in at least two video frames whose similarities are less than or equal to the first threshold, it is determined that the at least two video frames correspond to different lecture areas.

[0306] Furthermore, when the processor 900 executes the program, the following steps are implemented:

[0307] Perform target object detection on each of the multiple consecutive video frames;

[0308] If there is a target object, obtaining the target object and position information of the target object in each video frame;

[0309] If it is determined according to the position information that the similarity of the target object in a plurality of consecutive video frames is greater than or equal to a second threshold, deleting the area where the target object is located in the plurality of video frames;

[0310] The target detection of the handout is performed on each video frame where the target object is deleted, and the position area of ​​the handout content in the video frame is obtained.

[0311] Furthermore, when the processor 900 executes the program, the following steps are implemented:

[0312] If the lecture content in the video frame is not in full screen state, the video frame is segmented to obtain multiple segmented areas;

[0313] The segmented area with the largest area is taken as the effective area of ​​the handout;

[0314] The object detection of the handout is performed on the effective area in each video frame to obtain the position area of ​​the handout content in the video frame.

[0315] Furthermore, when the processor 900 executes the program, the following steps are implemented:

[0316] Perform line detection on the content area of ​​each page of handouts;

[0317] If lines are detected, each page of the handout area is divided into regions according to the detected lines, and multiple text regions corresponding to each page of the handout area are obtained respectively.

[0318] Furthermore, when the processor 900 executes the program, the following steps are implemented:

[0319] Obtaining a target spacing of text corresponding to each page of the handout area, wherein the target spacing includes row spacing and / or column spacing;

[0320] The content area of ​​each page of the lecture area is divided according to a preset value, and a plurality of divided text areas are obtained, wherein the preset value is obtained based on a target spacing of text corresponding to the lecture area.

[0321] Further, when the processor 900 executes the program, one of the following steps is implemented:

[0322] Obtaining content layout of the divided multiple text areas, and determining the logical relationship between the text areas according to the correspondence between the content layout and the logical relationship;

[0323] Keyword recognition is performed on the voice information of the video frame corresponding to each page of the handout area, and the logical relationship between the text areas is determined based on the recognized keywords.

[0324] Furthermore, when the processor 900 executes the program, the following steps are implemented:

[0325] Create indexes between handout areas;

[0326] Determine the directory framework based on the indexes between the handout areas and the corresponding logical relationships between the text areas;

[0327] Extracting content based on the text information of the audio information corresponding to the handout area on each page and the text content of the text area, and adding the extracted content to the position indicated by the directory frame to generate initial note content;

[0328] Among them, the note content corresponding to the lecture area in the initial note content is positively correlated with the time length of the lecture area, and the time length of the lecture area is determined according to the start time and end time corresponding to the lecture area obtained according to the video frame rate and the video frame number.

[0329] Furthermore, when the processor 900 executes the program, the following steps are implemented:

[0330] Acquire target information from the text information of the audio information corresponding to the handout area of ​​each page and the text content, wherein the target information is information that exists in both the text information of the audio information corresponding to the handout area and the text content;

[0331] The target information is added to the location indicated by the directory frame to generate initial note content.

[0332] Furthermore, when the processor 900 executes the program, the following steps are implemented:

[0333] Adjusting the initial note content according to the second rule to generate user note content;

[0334] The second rule includes at least one of the following:

[0335] In the case where the user behavior data includes a mouse position operated by the user, the collected mouse position is associated with a text area of ​​a handout area, a first handout content of the mouse position is obtained, and the content corresponding to the handout area associated with the mouse position in the initial note content is adjusted using the first handout content;

[0336] In the case where the user behavior data includes a fast-forward operation, collecting the second lecture content corresponding to the lecture area corresponding to the fast-forward operation, and deleting the second lecture content in the initial note content;

[0337] In the case where the user behavior data includes a pause operation, the third lecture content corresponding to the mouse stop position of the pause operation is collected. If the mouse stop time corresponding to the pause operation is less than or equal to a third threshold, the third lecture content is used to adjust the initial note content.

[0338] Furthermore, when the processor 900 executes the program, the following steps are implemented:

[0339] Based on user operations, obtain the content in the shared notes;

[0340] The content in the shared note is added to the user note content to generate final user note content.

[0341] At least one embodiment of the present application also provides a note generation device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, each process in the note generation method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be described here.

[0342] At least one embodiment of the present application further provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, each process in the above-mentioned note generation method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it is not repeated here. The computer-readable storage medium is, for example, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0343] At least one embodiment of the present application also provides a computer program product, including computer instructions, which, when executed by a processor, implement the various processes in the note generation method embodiment as described above and can achieve the same technical effect. To avoid repetition, they will not be described here.

[0344] It should be noted that, in this article, the term "comprises", "includes" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "including one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the method and device in the embodiment of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0345] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0346] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. A note generation method, characterized in that: include: Acquire multiple text areas in the handout area of ​​the target video and text information corresponding to the voice information of the target video; Acquire the logical relationship between the text areas and the text contents of the text areas; Determining initial note content according to the logical relationship, the text content, and the text information; Collect user behavior data during the target video playback process, adjust the initial note content according to the user behavior data, and generate user note content.

2. The method according to claim 1, characterized in that The method of obtaining multiple text regions in the lecture area of ​​the target video includes: Performing object detection of the lecture notes on each video frame to obtain the location area of ​​the lecture note content in the video frame, wherein the object detection includes: boundary detection and / or text detection; Determine the handout areas of different pages according to the first rule, and obtain at least one page of handout areas; Divide each page of the handout area into content areas to obtain multiple text areas corresponding to each page of the handout area; The first rule includes at least one of the following: If the similarity of the multiple target areas in the position areas in the multiple video frames is greater than or equal to the first threshold, determining that the position areas in the multiple video frames correspond to the same page of handout areas; If there are multiple target areas in the position area in at least two video frames whose similarities are less than or equal to the first threshold, it is determined that the at least two video frames correspond to different lecture areas.

3. The method according to claim 2, characterized in that The step of performing object detection on each video frame to obtain the position area of ​​the lecture content in the video frame includes: Perform target object detection on each of the multiple consecutive video frames; If there is a target object, obtaining the target object and position information of the target object in each video frame; If it is determined according to the position information that the similarity of the target object in a plurality of consecutive video frames is greater than or equal to a second threshold, deleting the area where the target object is located in the plurality of video frames; The target detection of the handout is performed on each video frame where the target object is deleted, and the position area of ​​the handout content in the video frame is obtained.

4. The method according to claim 2, characterized in that: The step of performing object detection on each video frame to obtain the position area of ​​the lecture content in the video frame includes: If the lecture content in the video frame is not in full screen state, the video frame is segmented to obtain multiple segmented areas; The segmented area with the largest area is taken as the effective area of ​​the handout; The object detection of the handout is performed on the effective area in each video frame to obtain the position area of ​​the handout content in the video frame.

5. The method according to claim 2, characterized in that: The division of each page of the handout area into content areas obtains a plurality of text areas corresponding to each page of the handout area, including: Perform line detection on the content area of ​​each page of handouts; If lines are detected, each page of the handout area is divided into regions according to the detected lines, and multiple text regions corresponding to each page of the handout area are obtained respectively.

6. The method according to claim 2, characterized in that The division of each page of the handout area into content areas obtains a plurality of text areas corresponding to each page of the handout area, including: Obtaining a target spacing of text corresponding to each page of the handout area, wherein the target spacing includes row spacing and / or column spacing; The content area of ​​each page of the lecture area is divided according to a preset value, and a plurality of divided text areas are obtained, wherein the preset value is obtained based on a target spacing of text corresponding to the lecture area.

7. The method according to claim 1, characterized in that The obtaining of the logical relationship between the text regions includes the following: Obtaining content layout of the divided multiple text areas, and determining the logical relationship between the text areas according to the correspondence between the content layout and the logical relationship; Keyword recognition is performed on the voice information of the video frame corresponding to each page of the handout area, and the logical relationship between the text areas is determined based on the recognized keywords.

8. The method according to claim 1, characterized in that The determining of the initial note content according to the logical relationship, the text content and the text information includes: Create indexes between handout areas; Determine the directory framework based on the indexes between the handout areas and the corresponding logical relationships between the text areas; Extracting content based on the text information of the audio information corresponding to the handout area on each page and the text content of the text area, and adding the extracted content to the position indicated by the directory frame to generate initial note content; Among them, the note content corresponding to the lecture area in the initial note content is positively correlated with the time length of the lecture area, and the time length of the lecture area is determined according to the start time and end time corresponding to the lecture area obtained according to the video frame rate and the video frame number.

9. The method according to claim 8, characterized in that The method of extracting content according to the text information of the audio information corresponding to each page of the handout area and the text content of the text area, and adding the extracted content to the position indicated by the directory frame to generate the initial note content includes: Acquire target information from the text information of the audio information corresponding to the handout area of ​​each page and the text content, wherein the target information is information that exists in both the text information of the audio information corresponding to the handout area and the text content; The target information is added to the location indicated by the directory frame to generate initial note content.

10. The method according to claim 1, characterized in that The collecting of user behavior data during the target video playback process, adjusting the initial note content according to the user behavior data, and generating user note content includes: Adjusting the initial note content according to the second rule to generate user note content; The second rule includes at least one of the following: In the case where the user behavior data includes a mouse position operated by the user, the collected mouse position is associated with a text area of ​​a handout area, a first handout content of the mouse position is obtained, and the content corresponding to the handout area associated with the mouse position in the initial note content is adjusted using the first handout content; In the case where the user behavior data includes a fast-forward operation, collecting the second lecture content corresponding to the lecture area corresponding to the fast-forward operation, and deleting the second lecture content in the initial note content; In the case where the user behavior data includes a pause operation, the third lecture content corresponding to the mouse stop position of the pause operation is collected. If the mouse stop time corresponding to the pause operation is less than or equal to a third threshold, the third lecture content is used to adjust the initial note content.

11. A note generating device, characterized in that: It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and when the processor executes the program, the steps of the note generating method as described in any one of claims 1 to 10 are implemented.

12. A readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps in the note generation method as described in any one of claims 1-10 are implemented.

13. A computer program product, characterized in that The method comprises computer instructions, which, when executed by a processor, implement the steps of the note generating method according to any one of claims 1 to 10.

Images