A method and system for processing structured content of narrative media data

Abstract

The present application relates to the field of data processing, and in particular to a method, system, device, and medium for processing structured content of narrative media data, the method comprising obtaining narrative media data; extracting voice information and visual information from the narrative media data; identifying and extracting text data from the voice information, and determining keywords of the text data; identifying and extracting visual entities from the visual information, wherein the visual entities include at least one of text, formulas, and graphics; performing topic segmentation on the text data according to the keywords to obtain topic text; generating a summary of the topic text; and associating the generated summary with the visual entity. The present application extracts voice and video multimodal information from narrative media data and associates them, which can improve the comprehensiveness of the structured content of the narrative media data.

Description

Technical Field

[0001] The present application relates to the field of data processing, and in particular to a method for processing structured content of narrative media data. Background Art

[0002] Against the backdrop of the rapid development of Internet technology, scenarios such as remote meetings, online live broadcasts, interactive training, and online education are becoming increasingly common in daily life, and narrative media data such as narrative texts, audio and video are also generated in large quantities.

[0003] Traditional structured processing only targets textual unimodal information. However, narrative media data such as narrative text, audio and video contain multimodal information such as video and audio. Therefore, if traditional structured content processing methods are used to process narrative media, it is difficult to ensure the comprehensiveness of the generated structured content, resulting in a significant reduction in the user's information acquisition efficiency.

[0004] Therefore, how to improve the comprehensiveness of the structured content of narrative media data is an urgent problem that needs to be solved today. Summary of the Invention

[0005] The present application provides a method for processing structured content of narrative media data, which can improve the comprehensiveness of the structured content of narrative media data.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions:

[0007] In a first aspect, the present invention provides a method for processing structured content of narrative media data, the method comprising:

[0008] Acquiring narrative media data;

[0009] extracting voice information and visual information from the narrative media data;

[0010] Identifying and extracting text data from the voice information, and determining keywords and intervals within the text data;

[0011] Identifying and extracting visual entities in the visual information, wherein the visual entities include at least one of text, formula, and graphic;

[0012] Perform topic segmentation on the text data according to the intervals where the keywords are located to obtain topic texts;

[0013] Generate a summary of the topic text;

[0014] Associate the generated summary with the visual entity.

[0015] In a preferred example of the present application, it can be further configured that the determining of the keywords of the text data and the intervals where the keywords are located includes:

[0016] Segmenting the text data into separate segments according to sentences;

[0017] Divide the text data into multiple units with the same number of sentences;

[0018] For each word in the text data, calculating the frequency of occurrence of the word in each unit and the average frequency of occurrence of the word in the entire text;

[0019] Selecting a corresponding unit of each word whose frequency is greater than the average frequency as the interval corresponding to the word;

[0020] When the frequency of each word in the unselected unit is greater than a preset threshold, adding the unselected unit to the interval;

[0021] Shorten the interval by removing sentences where the word does not appear at both ends of the interval, and determine the frequency of each word in the shortened interval;

[0022] Selecting the interval with the largest frequency of each word from one or more intervals as the interval where the word is located, and obtaining a set of intervals where each word is located;

[0023] Sort each word in the text data from high to low according to the number of occurrences, preset the number of keywords to be determined, and select words that rank higher than or equal to the preset number in the sorting as keywords for the text data;

[0024] In the interval set, the interval in which the keyword is located is determined.

[0025] In a preferred example of the present application, it can be further configured that the identifying and extracting visual entities in the visual information includes:

[0026] Extract key frame images from visual information;

[0027] Determine the position of the slide in the key frame image and segment the slide image;

[0028] Determine whether the slide page content in the adjacent slide image has changed;

[0029] retaining the changed slide image;

[0030] Extract visual entities from the retained images.

[0031] In a preferred example of the present application, it can be further configured that the text data is segmented into topics according to the intervals where the keywords are located, and the obtained topic texts include:

[0032] Determine whether the keyword intervals overlap. If not, merge the sentences in the keyword interval into one paragraph, and separate the sentences that are not in the interval into separate paragraphs.

[0033] If so, re-divide the interval into non-overlapping intervals and then merge the sentences in the interval where the keyword is located into one paragraph, and make the sentences that are not in the interval into separate paragraphs;

[0034] Determine whether the text length of the paragraph is less than the topic text length threshold, and if so, combine the paragraphs that are less than the topic text length threshold to obtain a new paragraph;

[0035] Calculate the similarity between adjacent paragraphs;

[0036] Based on the similarity, calculate the score of each paragraph gap as the topic segmentation point;

[0037] The gaps with high scores are selected as boundaries, and the texts are combined according to the obtained boundaries to obtain the topic text.

[0038] In a preferred example of the present application, it can be further configured that the summary includes a generative summary and an extractive summary, and the generating of the summary of the topic text and associating the generated summary with the visual entity includes:

[0039] Generate multiple topics from the topic text through generative summarization;

[0040] Generate multiple subtopics under each topic through extractive summarization;

[0041] Match each subtopic to a visual entity.

[0042] In a preferred example of the present application, it can be further configured that, after associating the generated abstract with the visual entity, the method further includes:

[0043] A mind map is generated based on the generated multiple topics, multiple subtopics under each topic, and the visual entities matching each subtopic.

[0044] In a preferred example of the present application, it can be further configured that the method further includes:

[0045] Displaying the acquired narrative media data in a first area of a human-computer interaction interface, wherein the first area displays the topic and amount of information of the narrative media data at a certain point in time upon receiving a first trigger instruction;

[0046] Displaying high-frequency words corresponding to the topic in a second area of the human-computer interaction interface, wherein the second area modifies at least one of the text topic, the number of high-frequency words, the color of the words, and the font when receiving a second trigger instruction;

[0047] Arranging the generated summary in the third area of the human-computer interaction interface in the time sequence of the narrative media data according to the summary sentences for display;

[0048] After associating the generated keywords, a network diagram is generated and displayed in a fourth area of the human-computer interaction interface, wherein the fourth area modifies the style of the network diagram when receiving a fourth trigger instruction;

[0049] The generated mind map is displayed in a fifth area of the human-computer interaction interface, and the fifth area modifies the mind map when receiving a fifth trigger instruction.

[0050] In a second aspect, the present application provides a system for processing structured content of narrative media data, the system comprising:

[0051] an acquisition unit, configured to acquire narrative media data and extract voice information and visual information from the narrative media data;

[0052] a recognition and extraction unit, configured to recognize and extract text data from the speech, determine keywords and intervals within the text data, and recognize and extract visual entities from the visual information, wherein the visual entities include at least one of text, formulas, and graphics;

[0053] a topic segmentation unit, configured to segment the text data into topics according to intervals where the keywords are located, to obtain topic texts;

[0054] A summary generating unit, configured to generate a summary of the topic text;

[0055] The association unit is used to associate the generated summary with the visual entity.

[0056] In a third aspect, the present application provides an electronic device comprising: a memory for storing executable instructions; and a processor for implementing any of the above-mentioned methods for processing structured content of narrative media data when executing the executable instructions stored in the memory.

[0057] In a fourth aspect, the present application provides a computer-readable storage medium having a program stored thereon, wherein when the program is executed by a processor, the method for processing structured content of narrative media data as described in any one of the above items is implemented.

[0058] In summary, compared with the prior art, the technical solutions provided by the embodiments of the present application have at least the following beneficial effects:

[0059] The present application provides a method for processing structured content of narrative media data. The method comprises the following steps: acquiring narrative media data; extracting speech and visual information from the narrative media data; identifying and extracting text data from the speech and determining keywords from the text data; identifying and extracting visual entities from the visual information, wherein the visual entities include at least one of text, formulas, and graphics; segmenting the text into topics based on the keywords; generating summaries of the text obtained from the topic segmentation; and associating the text summaries with the visual entities. The method can extract multimodal information, including speech and video, from the narrative media data and further associate the extracted multimodal information, thereby improving the comprehensiveness of the structured content of the narrative media data. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 A flowchart of a method for processing structured content of narrative media data provided in one embodiment of the present application.

[0061] Figure 2 An embodiment of the present application provides a flowchart for determining keywords and keyword intervals for narrative media data structured content.

[0062] Figure 3 This is a partial result diagram of the text data provided in one embodiment of the present application after being segmented by sentence.

[0063] Figure 4 A text segment diagram in a long text provided for one embodiment of the present application.

[0064] Figure 5 This is an example diagram of text topic segmentation results provided by an embodiment of the present application.

[0065] Figure 6 A structural diagram of a narrative media data structured content processing system provided in one embodiment of the present application. DETAILED DESCRIPTION

[0066] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0067] In one embodiment of the present application, a method for processing structured content of narrative media data is provided. Figure 1As shown, the main steps are described as follows:

[0068] Step S100, obtaining narrative media data;

[0069] Step S200, extracting voice information and visual information from the narrative media data;

[0070] Step S300, identifying and extracting text data from the voice information, and determining keywords and intervals within the text data;

[0071] Step S400, identifying and extracting visual entities of slides in the visual information, wherein the visual entities include at least one of text, formula, and graphic;

[0072] Step S500, dividing the text data into topics according to the intervals where the keywords are located to obtain topic texts;

[0073] Step S600, generating a summary of the topic text;

[0074] Step S700: Associating the generated summary with the visual entity.

[0075] Narrative media data refers to videos from scenarios such as teleconferencing, live online broadcasts, interactive training, and online education. The input narrative media data contains both audio and visual information. Audio information refers to the speech in the videos, and visual information refers to the visual information in the videos. Automatic speech recognition (ASR) can be used to extract text from audio information. Optical character recognition (OCR) can be used to extract visual entities from visual information.

[0076] In a specific implementation, narrative media data is obtained, voice information and visual information in the narrative media data are extracted, and text data in the voice information is identified and extracted. After the text data is extracted, the keywords in the text data and the intervals corresponding to the keywords need to be determined, and the text data is topic-segmented based on the intervals where the keywords are located to obtain topic texts, and a summary is generated using the topic texts. The determination and extraction of text data keywords and the intervals where the keywords are located, the topic segmentation of text data, and the text summary technology can all be processed using methods in the existing technology. For example, the tf-idf algorithm is commonly used to determine and extract keywords from text data. That is, if a word appears frequently in one article and rarely appears in other articles, it is considered that this word has better category discrimination ability, that is, it has a greater probability of being a keyword of the article; the topic segmentation of text data is to cluster multiple sentences in a text according to the theme; the text summary technology can use generative summary or extractive summary. For generative summary, labeled data is often used to train deep neural networks such as sequence models RNN to generate summaries. The results obtained are not required to be content that has appeared in the text; for extractive summary, the tf-idf algorithm or clustering algorithm can be used, where the clustering algorithm uses the central sentence in the obtained cluster as part of the extractive summary result. After the topic text is summarized, the generated summary is associated with the visual entity. For example, the generated summary and the visual entity can be associated by associating the text summary and the visual entity within a preset range of appearance time; or, associating the text summary and the visual entity with a semantic similarity greater than a preset threshold. Specifically, since both the extracted summary and visual entity contain timestamp information, by comparing their timestamp values, we can determine whether the two modalities' appearance times in the video overlap or are close. The closer the appearance times, the higher the likelihood of a match between the text summary and the visual entity. Simultaneously, textual information is extracted from the visual entity, and the semantic similarity between the textual information and the summary is calculated to assist in matching the text summary with the visual entity. This matching process can integrate multiple aspects of information, including semantics, time, and language understanding.

[0077] In this embodiment, voice and visual multimodal information is extracted from narrative media data, and the extracted multimodal information is further associated, thereby improving the comprehensiveness of the structured content of the narrative media data and greatly increasing the user's information acquisition rate.

[0078] In some embodiments, see Figure 2 As shown, determining the keywords of the text data and the intervals corresponding to the keywords in step S300 includes:

[0079] Step S301, dividing the text data into separate segments according to sentences;

[0080] Step S302, dividing the text data into a single segment into a plurality of units having the same number of sentences;

[0081] Step S303, for each word in the text data, calculating the frequency of occurrence of the word in each unit and the average frequency of occurrence of the word in the entire text;

[0082] Step S304, selecting the corresponding unit of each word whose frequency is greater than the average frequency as the interval corresponding to the word;

[0083] Step S305: when the frequency of each word in the unselected unit is greater than a preset threshold, the unselected unit is added to the interval;

[0084] Step S306: shorten the interval by removing sentences where the word does not appear at both ends of the interval, and determining the frequency of each word in the shortened interval;

[0085] Step S307: selecting the interval with the highest frequency of each word from one or more intervals as the interval where the word is located, and obtaining a set of intervals where each word is located;

[0086] Step S308: sorting each word in the text data from high to low according to the number of occurrences, presetting the number of keywords to be determined, and selecting words that rank higher than or equal to the preset number in the sorting as keywords for the text data;

[0087] Step S309: Determine the interval where the keyword is located in the interval set.

[0088] In specific implementation, since it is difficult to simply divide the text into paragraphs based on the context, and the converted sentences are relatively short, the text data can be segmented by sentence using the Whisper algorithm of Open AI, and each sentence has time sequence information. Figure 3 The following are some examples of the results of using Open AI's Whisper algorithm to segment the text data into separate sentences. In the generated results, each sentence has no punctuation marks. In subsequent processing, each sentence is treated as a separate paragraph, and each sentence has time sequence information.

[0089] After dividing the text data into multiple units with the same number of sentences, for each word in the text data, calculate the frequency of the word in each unit, that is, the result of the number of occurrences divided by the unit length, and select the unit with a frequency greater than the average frequency of the word in the entire text as the interval corresponding to the word. If the selected corresponding intervals are adjacent, such adjacent units will be merged. When the frequency of each word in the unselected unit is greater than the preset threshold, the unselected unit will be added to the interval. Figure 4 The following is a text fragment from a long text. The average word frequency of each word in the entire text fragment is calculated to obtain the frequency data:

[0090] {'Mini Games': 0.48, 'Business': 0.04, 'Angle': 0.04, 'Development': 0.04, 'Support': 0.04, 'Satisfaction': 0.04, 'Expectation': 0.08, 'Gap': 0.04, 'More': 0.04, 'Cash': 0.04, 'Return': 0.04, 'Current Situation': 0.04, 'High Quality': 0.04, 'Original': 0.04, 'Special': 0.04, 'Most': 0.04, 'Reference': 0.04, 'Imitation': 0.04, 'Round': 0.08, 'Introduction': 0.04, 'User': 0.04, 'Traffic': 0.04, 'Driving Force': 0.08, 'Company': 0.08, 'Profit': 0.08, 'channel':0.04,'WeChat':0.04,'do':0.04,'do well':0.08,'product':0.04,'service':0.04,'natural':0.04,'come':0.04,'by-product':0.04,'creativity':0.24,'platform':0.12,'reflection':0.08,'value':0.04,'understanding':0.12,'outside world':0.08,'too':0.04,'miniaturization':0.04,'game':0.08,'apply':0.04,'program':0.04,'shell':0.04,'place':0.04,'carrier':0.08,'carrying':0.04,'various':0.04};

[0091] Set the length of each unit to 5 sentences, and divide the text fragment into units with the endpoints of the units (left open and right closed intervals): [0, 5, 10, 15, 20, 25];

[0092] Calculate the frequency of each word in each unit, taking the word "mini game" as an example, including:

[0093] The word frequencies of “mini-game” in the five units are:

[0094] {[0,5]:0.2,[5,10]:0.6,[10,15]:0.4,[15,20]:0.4,[20,25]:0.8},

[0095] The average frequency of the word "mini-game" is 0.48. Since the average word frequencies of units [5,10] and [20,25] are greater than 0.48, they are selected as the corresponding intervals for this word;

[0096] If there are adjacent cases in the corresponding intervals of the word, such adjacent intervals are merged. Since the corresponding intervals [5,10] and [20,25] of the word "mini-game" are not adjacent, no adjacent interval merging operation is required for this example;

[0097] Set the threshold to the average frequency of all words in the text. The average frequency value obtained in the example of the text segment is 0.064. Since the word frequencies of the word "mini-game" on the unselected units are all greater than the average frequency of all words, the units [0,5], [10,15], and [15,20] are added to the interval corresponding to this word, resulting in the interval [0,25];

[0098] Then, remove the sentences at both ends of the interval corresponding to each word where no keyword appears to shorten the interval. Finally, determine the frequency of each word in the shortened interval. The frequency is the result of the number of occurrences divided by the interval length. Select the interval with the maximum frequency among the one or more intervals that may be obtained as the interval where the word is located, and obtain the set of intervals where each word is located. Sort each word in the text segment from high to low according to the number of occurrences, preset the number of keywords to be determined, and select the words ranked higher than or equal to the preset number in the sorting as the keywords of this text data. In the interval set, determine the intervals where the keywords are located. Specifically, taking the above text segment as an example, it includes:

[0099] Taking the word "mini-game" as an example, since in the interval corresponding to this word, the last sentence does not contain the word "mini-game", while the first sentence and the penultimate sentence contain it, the last sentence of the interval [0,25] is removed, resulting in the interval [0,24] corresponding to this word;

[0100] Calculate the frequency of "mini-game" in this interval, and the result is 0.5. Since the word "mini-game" only obtains one corresponding interval [0,24] and there is no need to compare the frequencies between multiple corresponding intervals, the interval where the word "mini-game" is located is determined to be [0,24], and the corresponding maximum frequency is 0.5;

[0101] In this example, the text segment interval length threshold is set to at least 3. The above operation is performed on each word in the text segment for which the average word frequency is calculated. The set of all intervals with a length greater than 3 and their corresponding words in this example is:

[0102] {'Mini Games':{'segment':[0,24],'max_freq':0.5},'Power':{'segment':[9,14],'max_freq':0.4},'Creativity':{'segment':[14,25],'max_freq':0.54},'Platform':{'segment':[14,24],'max_freq':0.3},'Embody':{'segment':[15,23],'max_freq':0.25},'Understanding':{'segment':[17,22],'max_freq':0.6}};

[0103] In this example, we select 3 keywords to be determined for the text segment. We sort each word in the text by the number of occurrences from highest to lowest, and select the top 3 keywords: mini-games, creativity, and understanding. In this example, we select the intervals corresponding to these 3 keywords from the set of all intervals greater than 3 and their corresponding words: [0, 24] [14, 25], and [17, 22]. Finally, we obtain the keywords, the number of occurrences, and the intervals in which the keywords are located for this text segment:

[0104] Mini Games 12.0[0,24]

[0105] Creativity 6.0[14,25]

[0106] Understanding 3.0[17,22]

[0107] In this embodiment, the frequency and position of words in the text are comprehensively analyzed, especially based on the characteristic that narrative media such as teaching and lectures are more concentrated on the same topic, the frequency of words in the interval segment is calculated. Compared with the existing technology algorithm, the influence of the same words appearing in adjacent sentences on keyword determination is increased, thereby improving the accuracy of keyword determination.

[0108] In some embodiments, in step S400, identifying and extracting visual entities from the visual information includes:

[0109] Step S401, extracting key frame images from visual information;

[0110] Step S402, determining the position of the slide in the key frame image and segmenting the slide image;

[0111] Step S403, determining whether the content of the slide page in the adjacent slide image has changed;

[0112] Step S404, retaining the changed slide image;

[0113] Step S405: extracting visual entities from the retained image.

[0114] In specific implementation, after extracting key frame images from the video (such as extracting at a frequency of 25 frames / second), the position of the slide in the picture is determined by tracking the slide area in adjacent frames and combining key features such as the rectangular border that the slide often has.

[0115] In determining the position of the slide on the screen, the process of determining the quadrilateral frame is as follows: for a given key frame image, first convert it into a grayscale image, then use different thresholds to convert the grayscale image into a series of binary images, and for all the obtained binary images, use the contour extraction algorithm of Open CV to extract all contours in the image, and filter the contours according to the following rules:

[0116] 1) The area enclosed by the contour line is not less than one tenth of the key frame area;

[0117] 2) The number of contour line vertices is not less than 4;

[0118] 3) The area of the rectangular bounding box of the contour line is not more than twice the area of the area enclosed by the contour line;

[0119] 4) The aspect ratio of the rectangular bounding box of the contour line needs to be between [0.5, 2];

[0120] From all the contours that meet all the above conditions, the area enclosed by the contour line with the largest area is selected as the detected slide area.

[0121] After determining the area of the slide containing the keyframe image, the video frame image is segmented using methods such as image segmentation. By determining whether the texture, color, edge features, and overall layout of the slide have undergone significant changes, frames that have increased or decreased the slide page content compared to the previous frame are retained, thereby discarding duplicate frame images. Then, optical character recognition (OCR) technology is used to extract visual entities such as text, formulas, and graphics from the slide. These entity components can also be further combined or split based on entity type and the time of entity occurrence, while also recording the time of entity occurrence.

[0122] In this embodiment, starting from the characteristics of narrative media data, key frames are first screened out to reduce the amount of subsequent image processing, and then the area where the slides (blackboard writings) are located in the key frames is determined. After the slide images are segmented, the repeated slide images are screened out and the changed slide images are retained, thereby reducing the repeated extraction of visual entities, ensuring that the extracted content meets the user's information acquisition needs, and improving the efficiency of information acquisition.

[0123] In some embodiments, the step S500 of segmenting the text data into topics according to the intervals where the keywords are located to obtain topic texts includes:

[0124] Step S501, determine whether the keyword intervals overlap, if not, execute step S502, if yes, re-divide the intervals into non-overlapping intervals, and execute step S502;

[0125] Step S502: merge the sentences in the interval where the keyword is located into one paragraph, and separate the sentences that are not in the interval into separate paragraphs;

[0126] Step S503, determining whether the text length of the paragraph is less than a topic text length threshold;

[0127] Step S504: If yes, then combine the paragraphs that are shorter than the topic text length threshold to obtain a new paragraph;

[0128] Step S505, calculating the similarity between adjacent paragraphs;

[0129] Step S506, based on the similarity, calculate the score of each paragraph gap as a topic segmentation point;

[0130] Step S507: select gaps with high scores as boundaries, and combine the texts according to the obtained boundaries to obtain topic texts.

[0131] In practice, we first determine whether the keyword intervals overlap. If so, we further divide the intervals into smaller intervals. For example, if we have a list of intervals [[0,10],[2,6],[5,12],[20,30],[20,22]], we can further divide the list into [[0,2],[2,5],[5,6],[6,10],[10,12],[20,22],[22,30]].

[0132] After confirming that the keyword intervals do not overlap, the sentences in the keyword interval are merged into one paragraph, and the sentences that are not in the interval are separated into separate paragraphs for further topic segmentation. Set the minimum topic text length as the topic text length threshold, combine the paragraphs that are less than the text length threshold, and obtain new paragraph divisions; then, calculate the similarity between adjacent paragraphs, and based on the similarity, calculate the score of each paragraph gap as the topic segmentation point. The greater the similarity between the paragraphs on both sides of the paragraph gap, the lower the score corresponding to the paragraph gap, and select the gap with a high score as the boundary; finally, combine the text according to the obtained boundary to achieve topic segmentation. Specifically, an example of a topic text after topic segmentation is as follows: Figure 5 As shown in the figure, each blank line in this text is a paragraph gap. Assuming that the text length is greater than the length threshold, based on the similarity between paragraphs, the similarity between the first and second paragraphs in the figure is lower than the similarity between the second and third paragraphs. That is, the score of the second paragraph gap as a topic segmentation point is lower than the score of the first paragraph gap as a topic segmentation point. Therefore, the first paragraph gap will be preferred as the topic segmentation point.

[0133] In this embodiment, each short sentence in the text is segmented into a separate paragraph, and topic segmentation is performed based on the extracted keywords and the paragraph merging of sentences according to the keywords. This will have a better effect than directly segmenting the text consisting of separate paragraphs of sentences, and to a certain extent ensure that the adjacent paragraphs segmented into a topic have relatively close or identical themes. Compared with existing algorithms, clustering and other technologies for segmenting long texts into topics, this embodiment obtains topic text through topic segmentation during the topic determination process, ensuring that sentences with the same topic appear continuously in the text, and improving the accuracy of segmenting text data into topics based on keywords.

[0134] In some embodiments, in step S600, the summary includes a generative summary and an extractive summary, and generating a summary of the topic text and associating the generated summary with a visual entity includes:

[0135] Generate multiple topics from the topic text through generative summarization;

[0136] Generate multiple subtopics under each topic through extractive summarization;

[0137] Match each subtopic to a visual entity.

[0138] Specifically, generative summary generation is divided into the following steps:

[0139] 1. Text preprocessing. Since there are no punctuation marks at the end of sentences after topic segmentation, sentences that will be divided into paragraphs are added with periods at the end and then connected in chronological order. The resulting topic contains one or more natural paragraphs, and a summary is generated for each topic separately.

[0140] 2. Model Construction: Build deep learning models, such as RNN models, Transformer models, and Pointer-generator models. Use a large amount of labeled data to train a generative summary generation model. Learn the mapping relationship between text and summary from the input text content and the corresponding summary content. Some content in the summary may not appear in the original text.

[0141] 3. Model Evaluation: Use test data to evaluate the model. Evaluation metrics typically include the accuracy, fluency, and information content of the summary content. The summary should contain as many extracted keywords as possible.

[0142] 4. Summary generation: Use the constructed model to generate a summary.

[0143] Extractive summarization is divided into the following steps:

[0144] First, text preprocessing involves sentence segmentation, tokenization, and stop word removal. Each sentence between two topic split points is processed, meaning it corresponds to the same topic, without adding punctuation. For each sentence in a topic, we use the jieba Python Chinese token segmentation component to tokenize the text, and remove stop words from the tokenization results based on a selected stop word list.

[0145] 2. Calculate word weights. Use the TF-IDF algorithm to calculate word weights:

[0146]

[0147] TF (Term Frequency TF) represents word frequency, and IDF (Inverse Document Frequency) represents the inverse document frequency index. The more times a word appears in a sentence and the fewer times it appears in other sentences in the text, the greater the weight of the word.

[0148] 3. Calculate the similarity between sentences. The similarity between two sentences is expressed by calculating the cosine similarity of the tf-idf weight:

[0149]

[0150] Where q and d are two sentences in the paragraph, t and qi are words obtained by segmenting the sentence corresponding to q, and di is a word obtained by segmenting the sentence corresponding to d. TF-IDF(t,q) is the TF-IDF algorithm mentioned above.

[0151] Clustering: Randomly initialize k sentences as centers. Based on similarity, assign sentences to the closest centers to form k clusters. Recalculate the centers for each category and regroup them. Repeat this process until the cluster centers stop changing or the maximum number of iterations is reached. The k centers obtained from the last iteration are the extractive summary of the text.

[0152] In this embodiment, summaries are generated at different granularities, multiple topics are generated from the topic text through generative summarization, multiple subtopics under each topic are generated through extractive summarization, each subtopic is matched with a visual entity, and natural language processing technology is used to extract explicit / implicit relationships in the text to improve the accuracy of the generated structured content.

[0153] In some embodiments, after associating the generated abstract with the visual entity, the method further includes:

[0154] Step S800: Generate a mind map based on the generated multiple topics, the multiple subtopics under each topic, and the visual entities matching each subtopic.

[0155] Specifically, after topic segmentation, the generative summaries of each group of texts are used as multiple topics of the mind map, and the multi-sentence extractive summaries are used as multiple subtopics under the topic, and a matching visual entity is attached to each subtopic. Among them, the visual entities in the generated mind map have three forms: icons, small pictures, and full-screen displays. Users can choose according to their own information acquisition needs. In addition to the content in the mind map meeting certain readability and logic, the layout must also meet certain rationality and aesthetics. For example, the topics can be laid out in an order from top to bottom or from left to right that conforms to human reading habits. After using natural language processing technology to perform operations such as extracting explicit / implicit relationships and generating summaries on text information, the text information is matched and aligned with the visual information, and the analysis results are presented to the user, which greatly improves the user's information acquisition rate.

[0156] In this embodiment, the matched text summaries and visual entities are laid out in the form of a mind map. Each topic corresponds to a theme in the mind map, and each extracted summary serves as a sub-theme under the corresponding theme, thereby improving the aesthetics of the structured content display of narrative text data and the user's browsing efficiency.

[0157] In some embodiments, the method further comprises:

[0158] Displaying the acquired narrative media data in a first area of a human-computer interaction interface, wherein the first area displays the topic and amount of information of the narrative media data at a certain point in time upon receiving a first trigger instruction;

[0159] Displaying high-frequency words corresponding to the topic in a second area of the human-computer interaction interface, wherein the second area modifies at least one of the text topic, the number of high-frequency words, the color of the words, and the font when receiving a second trigger instruction;

[0160] Arranging the generated summary in the third area of the human-computer interaction interface in the time sequence of the narrative media data according to the summary sentences for display;

[0161] After associating the generated keywords, a network diagram is generated and displayed in a fourth area of the human-computer interaction interface, wherein the fourth area modifies the style of the network diagram when receiving a fourth trigger instruction;

[0162] The generated mind map is displayed in a fifth area of the human-computer interaction interface, and the fifth area modifies the mind map when receiving a fifth trigger instruction.

[0163] Specifically, the first area of the human-computer interaction interface features a video player that displays captured narrative media data. In addition to normal operations such as play, pause, and dragging the progress bar, users can hover their mouse over the progress bar to view the topic and information content of the video at that point in time. The higher the curve on the video progress bar at that point in time, the greater the information content and the more important the video content. Information content is measured by the richness of visual features and the density of topic-related elements in the video. The second area of the human-computer interaction interface displays a visualization of high-frequency words for the corresponding topic. By receiving user-selected text topics, the number of high-frequency words, word color, and font, users can customize the visualization of high-frequency words for the corresponding topic based on their needs and preferences. The third area of the human-computer interaction interface displays a timeline. Each item in the timeline consists of an extractive summary and the start time of the summary sentence in the video, arranged in chronological order. Summaries extracted from the same topic are placed in the same block. The fourth area of the human-computer interaction interface displays a network diagram. Nodes represent keywords. The more times a keyword appears in a segment, the larger the node becomes. The time below the node corresponds to the start of the segment containing the keyword. An edge represents the intersection of the segments corresponding to two nodes, indicating that the two keywords are related in the text. The greater the overlap, the thicker the edge. A network diagram allows you to specify a keyword and highlight the edges connected to it. A network diagram visualization can also be replaced with a topic river visualization, using the midpoint of the interval created in "Text Topic Segmentation" as the horizontal axis and the number of times the keyword appears in the interval as the value at the midpoint of the interval. The fifth area of the human-computer interaction interface is used to display the mind map. In the generated mind map, each topic corresponds to a generative summary of the topic text, and the subtopics are the corresponding extractive summaries. The extracted visual entities are displayed as icons and thumbnails, as shown in the figure above. Clicking an icon switches to a thumbnail display, and further clicking on the thumbnail displays a larger image. Furthermore, users can modify the generated mind map according to their preferences and needs, such as adding or deleting nodes, modifying node content, and changing node style, enabling human-computer interaction.

[0164] In this embodiment, the acquired narrative text data and its structured content can be displayed on a human-computer interaction interface, allowing users to quickly and accurately find the information they need and efficiently browse and locate the content. At the same time, efficient information processing tools are designed and developed for specific application scenarios to meet users' diverse information acquisition needs and significantly improve their information acquisition efficiency and browsing experience.

[0165] Most current products on the market are limited to analyzing textual content like notes and PDFs, with limited research on narrative audio and video content like real-time online meetings and distance learning, resulting in limited application scenarios. This application is no longer limited to analyzing inherent information; it can perform speech recognition on live audio and video, and present the analysis results of the current content in real time, enriching the application scenarios.

[0166] This application also provides a structured content processing system for narrative media data, see Figure 6 As shown, the system includes:

[0167] An acquisition unit 101 is configured to acquire narrative media data and extract voice information and visual information from the narrative media data;

[0168] a recognition and extraction unit 102 for recognizing and extracting text data from the speech, determining keywords and intervals within the text data, and recognizing and extracting visual entities from the visual information, wherein the visual entities include at least one of text, formulas, and graphics;

[0169] A topic segmentation unit 103 is configured to segment the text data into topics according to the intervals where the keywords are located, to obtain topic texts;

[0170] A summary generating unit 104, configured to generate a summary of the topic text;

[0171] The associating unit 105 is configured to associate the generated summary with the visual entity.

[0172] The functional implementation of each module in the above-mentioned narrative media data structured content processing system corresponds to the steps in the above-mentioned narrative media data structured content processing system method embodiment, and their functions and implementation processes are not repeated here one by one.

[0173] The present application also provides an electronic device comprising a processor, a memory, a network interface, and a database connected via a system bus. The processor of the electronic device is configured to provide computing and control capabilities. The memory of the electronic device comprises a non-volatile storage medium and an internal memory. The non-volatile storage medium contains an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The network interface of the electronic device is configured to communicate with an external terminal via a network connection. When executed by the processor, the computer program implements any of the aforementioned methods for processing structured content of narrative media data.

[0174] This application also provides a computer-readable storage medium having a program stored thereon. The computer-readable storage medium refers to a data storage medium and may include, but is not limited to, a floppy disk, an optical disk, a hard disk, a flash memory, a USB flash drive, and / or a memory stick. The computer may be a general-purpose computer, a dedicated computer, a computer network, or other programmable device. The operating process, operating details, and technical effects of the computer-readable storage medium provided in this embodiment can be found in the above-mentioned embodiment of a method for processing structured content of narrative media data, and are not further described here.

[0175] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0176] The technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification. The above-mentioned embodiments only express several implementation methods of the present application. The description is relatively specific and detailed, but it cannot be understood as a limitation on the scope of the invention patent. It should be pointed out that for ordinary technicians in this field, without departing from the concept of the present application, several variations and improvements can be made, which all fall within the scope of protection of the present application. Therefore, the scope of protection of the patent of this application shall be based on the attached claims.

Claims

1. A method for processing structured content of narrative media data, characterized in that: The method comprises: Acquiring narrative media data; extracting voice information and visual information from the narrative media data; Identifying and extracting text data from the voice information, and dividing the text data into separate segments according to sentences; Divide the text data into multiple units with the same number of sentences; For each word in the text data, calculating the frequency of occurrence of the word in each unit and the average frequency of occurrence of the word in the entire text; Selecting a corresponding unit of each word whose frequency is greater than the average frequency as the interval corresponding to the word; When the frequency of each word in the unselected unit is greater than a preset threshold, adding the unselected unit to the interval; Shorten the interval by removing sentences where the word does not appear at both ends of the interval, and determine the frequency of each word in the shortened interval; Selecting the interval with the largest frequency of each word from one or more intervals as the interval where the word is located, and obtaining a set of intervals where each word is located; Sort each word in the text data from high to low according to the number of occurrences, preset the number of keywords to be determined, and select words that rank higher than or equal to the preset number in the sorting as keywords for the text data; In the interval set, determining the interval where the keyword is located; Identifying and extracting visual entities in the visual information, wherein the visual entities include at least one of text, formula, and graphic; Determine whether the keyword intervals overlap. If not, merge the sentences in the keyword interval into one paragraph, and make the sentences not in the interval into separate paragraphs. If so, re-divide the intervals into non-overlapping intervals and then merge the sentences in the keyword interval into one paragraph, and make the sentences not in the interval into separate paragraphs. Determine whether the text length of the paragraph is less than the topic text length threshold, and if so, combine the paragraphs that are less than the topic text length threshold to obtain a new paragraph; Calculate the similarity between adjacent paragraphs; Based on the similarity, calculate the score of each paragraph gap as the topic segmentation point; Select the gaps with high scores as boundaries, combine the texts according to the obtained boundaries, and obtain the topic texts; Generate a summary of the topic text; Associate the generated summary with the visual entity.

2. The method for processing structured content of narrative media data according to claim 1, characterized in that: The identifying and extracting visual entities from the visual information comprises: Extract key frame images from visual information; Determine the position of the slide in the key frame image and segment the slide image; Determine whether the slide page content in the adjacent slide image has changed; retaining the changed slide image; Extract visual entities from the retained images.

3. The method for processing structured content of narrative media data according to claim 1, characterized in that: The summary includes a generative summary and an extractive summary. The generating of the summary of the topic text and associating the generated summary with the visual entity includes: Generate multiple topics from the topic text through generative summarization; Generate multiple subtopics under each topic through extractive summarization; Match each subtopic to a visual entity.

4. The method for processing structured content of narrative media data according to claim 3, wherein: After associating the generated summary with the visual entity, the method further includes: A mind map is generated based on the generated multiple topics, multiple subtopics under each topic, and the visual entities matching each subtopic.

5. The method for processing structured content of narrative media data according to claim 4, characterized in that: The method further comprises: Displaying the acquired narrative media data in a first area of a human-computer interaction interface, wherein the first area displays the topic and amount of information of the narrative media data at a certain point in time upon receiving a first trigger instruction; Displaying high-frequency words corresponding to the topic in a second area of the human-computer interaction interface, wherein the second area modifies at least one of the text topic, the number of high-frequency words, the color of the words, and the font when receiving a second trigger instruction; Arranging the generated summary in the third area of the human-computer interaction interface in the time sequence of the narrative media data according to the summary sentences for display; After associating the generated keywords, a network diagram is generated and displayed in a fourth area of the human-computer interaction interface, wherein the fourth area modifies the style of the network diagram when receiving a fourth trigger instruction; The generated mind map is displayed in a fifth area of the human-computer interaction interface, and the fifth area modifies the mind map when receiving a fifth trigger instruction.

6. A system for processing structured content of narrative media data, characterized in that: The system comprises: an acquisition unit, configured to acquire narrative media data and extract voice information and visual information from the narrative media data; a recognition and extraction unit, configured to recognize and extract text data from the speech information, segment the text data into separate sentences, divide the separate segmented text data into multiple units with the same number of sentences, calculate the frequency of each word in each unit and the average frequency of the word in the entire text for each word in the text data, select the corresponding unit in which the frequency of each word is greater than the average frequency as the interval corresponding to the word, add the unselected unit to the interval when the frequency of each word in the unselected unit is greater than a preset threshold, remove sentences in which the word does not appear at both ends of the interval to shorten the interval, and determine the frequency of each word in the shortened interval, select the interval with the highest frequency of each word from one or more intervals as the interval in which the word is located, and obtain a set of intervals in which each word is located, sort each word in the text data from high to low according to the number of occurrences, preset the number of keywords to be determined, and select words ranked higher than or equal to the preset number in the sorting as keywords for the text data, and determine the interval in the interval set in which the keyword is located, and recognize and extract visual entities from the visual information, wherein the visual entity includes at least one of text, formula, and graphic. The topic segmentation unit is used to determine whether the intervals of keywords overlap. If not, the sentences in the interval where the keywords are located are merged into one paragraph, and the sentences not in the interval are separated into separate paragraphs. If so, the interval is re-divided into non-overlapping intervals and the sentences in the interval where the keywords are located are merged into one paragraph, and the sentences not in the interval are separated into separate paragraphs; determine whether the text length of the paragraph is less than the topic text length threshold. If so, the paragraphs less than the topic text length threshold are combined to obtain new paragraphs; calculate the similarity between adjacent paragraphs; based on the similarity, calculate the score of each paragraph gap as the topic segmentation point; select the gap with the high score as the boundary, and combine the text according to the obtained boundary to obtain the topic text; A summary generating unit, configured to generate a summary of the topic text; The association unit is used to associate the generated summary with the visual entity.

7. An electronic device, characterized in that: The electronic device comprises: a memory for storing executable instructions; and a processor for implementing the narrative media data structured content processing method according to any one of claims 1 to 5 when executing the executable instructions stored in the memory.

8. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a program, wherein when the program is executed by a processor, the method for processing structured content of narrative media data according to any one of claims 1 to 5 is implemented.

Images