Video plot description generation method and device, electronic equipment and readable medium

Abstract

The invention relates to a video plot description generation method and device, electronic equipment and a readable medium, and the method comprises the steps: recognizing character interaction information in a video file, carrying out the hierarchical division of the video file, and obtaining each shot segment and a scene segment, and the scene segment comprises at least one shot segment; generating a first plot description of each shot fragment according to the character interaction information; generating a second plot description of the scene segment according to the first plot description of each shot segment in the scene segment and the character interaction information; a third plot description of a paragraph segment is generated according to the second plot description of each scene segment and the character interaction information, and the paragraph segment is composed of at least one scene segment; and integrating the third plot descriptions to generate an overall plot description of the video file. The problem that it is difficult to directly use a language model to generate plot description of a long video is solved.

Description

Technical Field

[0001] The present application relates to the technical field of video processing, and in particular, to a method, apparatus, electronic device, and readable medium for generating a video plot description. Background Art

[0002] With the rapid development of information and communication technologies, videos have gradually become the most popular information exchange carrier on the Internet. Video description generation has been a research hotspot in the field of artificial intelligence in recent years and has received much attention in the computer vision community and the natural language processing community. Video description generation aims to automatically generate a corresponding sentence for a given video to summarize the events occurring in the video. It requires accurate identification of the key entities and events appearing in the video and description of them in a language manner that conforms to human habits.

[0003] Currently, it is common to use video-language multimodal large models to generate video descriptions. However, the training data of this model is a large number of short videos, so the videos that can be processed are all short videos, and it is difficult to generate a plot description of a long video more accurately. Because when generating a plot description of a long video, it is also necessary to maintain the coherence of the narrative and the rationality of logical reasoning. That is to say, it is difficult to directly use a language model to generate a plot description of a long video.

[0004] For the above problems, no effective solution has been proposed yet. Summary of the Invention

[0005] The present application provides a method, apparatus, electronic device, and readable medium for generating a video plot description to solve the technical problem of "difficulty in directly using a language model to generate a plot description of a long video".

[0006] According to one aspect of the embodiments of the present application, the present application provides a method for generating a video plot description, including: identifying the character interaction information in the video file, and hierarchically partitioning the video file to obtain each shot segment and scene segment, where the scene segment includes at least one shot segment; generating a first plot description for each shot segment according to the character interaction information; generating a second plot description for the scene segment according to the first plot descriptions of the shot segments in the scene segment and the character interaction information; generating a third plot description for the paragraph segment according to the second plot descriptions of the scene segments and the character interaction information, where the paragraph segment is composed of at least one scene segment; integrating each third plot description to generate an overall plot description of the video file.

[0007] Optionally, identify the human interaction information in the video file, including: obtaining a character mapping table of the video file, where the character mapping table includes the mapping relationship between the character name and the character's face; performing face recognition on the video file to obtain the face information of the video file; performing speech recognition on the video file, and obtaining the line information of the video file according to the face information and the character mapping table, where the line information includes the line content and the character information of the line speaker; determining the character mapping table, the face information, and the line information as the human interaction information, where the human interaction information carries a timestamp.

[0008] Optionally, perform hierarchical division on the video file to obtain each shot segment and scene segment, including: extracting multiple video frames in the video file; performing shot boundary detection on each video frame to obtain multiple shot segments; obtaining the attention change between each consecutive shot segment, and dividing the multiple shot segments into different scene segments according to the attention change.

[0009] Optionally, generate a first plot description for each shot segment according to the human interaction information, including: performing plot recognition on the shot segment through a first language model to output an initial plot description; extracting each video frame in the shot segment, where the video frame carries a timestamp; aligning the human interaction information with each video frame in the shot segment according to the timestamp to obtain the shot information corresponding to the shot segment; inputting the initial plot description and the shot information into a second language model to generate a first plot description of the shot segment through the second language model.

[0010] Optionally, generate a second plot description of the scene segment according to the first plot description of each shot segment in the scene segment and the human interaction information, including: extracting the line information from the human interaction information; determining the first plot description and the line information as the context information; inputting the context information under the same scene segment into the second language model to generate a second plot description of the scene segment through the second language model.

[0011] Optionally, generate a third plot description of the paragraph segment according to the second plot description of each scene segment and the human interaction information, including: dividing the multiple scene segments into each paragraph segment according to the second plot description of each scene segment; inputting the second plot description of the scene segments under the same paragraph segment into the second language model to generate a third plot description of the paragraph segment through the second language model.

[0012] Optionally, multiple scene segments are divided into respective paragraph segments according to the second plot descriptions of the respective scene segments, including: identifying event keywords in each of the second plot descriptions to determine the key events of each scene segment; determining the correlation degree of the key events of two adjacent scene segments; if the correlation degree of the key events of two adjacent scene segments is greater than or equal to a preset threshold, determining the two adjacent scene segments as the same paragraph segment, and if the correlation degree of the key events of two adjacent scene segments is less than the preset threshold, dividing the two adjacent scene segments into different paragraph segments.

[0013] According to another aspect of the embodiments of the present application, the present application provides a video plot description generation device, including: an identification module, configured to identify the character interaction information in a video file and perform hierarchical division on the video file to obtain respective shot segments and scene segments, where the scene segment includes at least one shot segment; a first generation module, configured to generate a first plot description of each shot segment according to the character interaction information; a second generation module, configured to generate a second plot description of the scene segment according to the first plot descriptions of the respective shot segments in the scene segment and the character interaction information; a third generation module, configured to generate a third plot description of the paragraph segment according to the second plot descriptions of the respective scene segments and the character interaction information, where the paragraph segment is composed of at least one scene segment; and an integration module, configured to integrate the respective third plot descriptions to generate an overall plot description of the video file.

[0014] According to another aspect of the embodiments of the present application, the present application provides an electronic device, including a memory, a processor, a communication interface, and a communication bus. A computer program that can run on the processor is stored in the memory. The memory and the processor communicate through the communication bus and the communication interface. When the processor executes the computer program, the steps of the above method are implemented.

[0015] According to another aspect of the embodiments of the present application, the present application further provides a computer-readable medium having non-volatile program code executable by a processor, and the program code causes the processor to execute the above method.

[0016] The above technical solutions provided by the embodiments of the present application have the following advantages compared with the related technologies:

[0017] The present application provides a method for generating a video plot description, including: identifying the character interaction information in a video file, hierarchically partitioning the video file to obtain each shot segment and scene segment, where the scene segment includes at least one shot segment; generating a first plot description for each shot segment according to the character interaction information; generating a second plot description for the scene segment according to the first plot descriptions of the shot segments in the scene segment and the character interaction information; generating a third plot description for the paragraph segment according to the second plot descriptions of the scene segments and the character interaction information, where the paragraph segment is composed of at least one scene segment; integrating the third plot descriptions to generate an overall plot description of the video file. By partitioning a long video file into segments of different levels, then generating plot descriptions from the smallest to the largest segment granularity, and finally obtaining the overall plot description of the long video, the problem of difficultly directly using a language model to generate the plot description of a long video is solved. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The accompanying drawings herein are incorporated into the specification and form a part of the specification, showing embodiments consistent with the present application, and are used together with the specification to explain the principles of the present application.

[0019] To more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the following will briefly introduce the accompanying drawings required for use in the description of the embodiments or related technologies. Obviously, for those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0020] Figure 1 FIG. [FIG NUMBER] is a flowchart of an optional method for generating a video plot description according to an embodiment of the present application;

[0021] Figure 2 FIG. [FIG NUMBER] is a block diagram of an optional device for generating a video plot description according to an embodiment of the present application;

[0022] Figure 3 FIG. [FIG NUMBER] is a schematic structural diagram of an optional electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0023] To make the objectives, technical solutions, and advantages of the embodiments of the present application clearer, the following will clearly and completely describe the technical solutions in the embodiments of the present application with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are some, but not all, of the embodiments of the present application. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present application without creative efforts fall within the scope of protection of the present application.

[0024] In the following description, suffixes such as "module", "component", or "unit" used to represent elements are only for the convenience of the description of this application, and they have no specific meaning in themselves. Therefore, "module" and "component" can be used interchangeably.

[0025] With the rapid development of information and communication technologies, videos have gradually become the most popular information exchange carrier on the Internet. Video description generation has been a research hotspot in the field of artificial intelligence in recent years and has received much attention in the computer vision community and the natural language processing community. Video description generation aims to automatically generate a corresponding text sentence for a given video to summarize the events occurring in the video. It requires accurate identification of the key entities and events appearing in the video and description of them in a language manner that conforms to human habits.

[0026] Currently, it is relatively common to use video-language multimodal large models to generate video descriptions. However, the training data of this model are a large number of short videos, so the videos that can be processed are all short videos, and it is difficult to generate relatively accurate plot descriptions for long videos. Because when generating plot descriptions for long videos, it is also necessary to maintain the coherence of the narrative and the rationality of logical reasoning. That is to say, it is difficult to directly use language models to generate plot descriptions for long videos.

[0027] To solve the problems mentioned in the background art, according to one aspect of the embodiments of this application, a method for generating video plot descriptions is provided, as Figure 1 shown, including:

[0028] Step 101, identifying the human interaction information in the video file and hierarchically partitioning the video file to obtain each shot segment and scene segment, where the scene segment includes at least one shot segment;

[0029] Step 103, generating a first plot description for each shot segment according to the human interaction information;

[0030] Step 105, generating a second plot description for the scene segment according to the first plot descriptions of the shot segments in the scene segment and the human interaction information;

[0031] Step 107, generating a third plot description for the paragraph segment according to the second plot descriptions of each scene segment and the human interaction information, where the paragraph segment is composed of at least one scene segment;

[0032] Step 109, integrating each third plot description to generate an overall plot description of the video file.

[0033] In existing technical means, when generating a plot description, it is difficult to directly use a vision-language model or a language model to generate a plot description for a long video at the hour level. Currently, the optimal full-modal or vision-language model can only support descriptions at the granularity of dozens of seconds. Based on this, the present application provides a method for generating a video plot description, which is particularly applicable to the generation of a plot description for a long video.

[0034] Since it is difficult to directly generate a plot description for a long video, the present application hierarchically divides the video file and then generates a plot description. The long video is divided into video segments at different levels, including shot segments and scene segments, and each scene segment is further divided into different paragraph segments during subsequent processing. Among them, a shot segment includes at least one video frame, a scene segment includes at least one shot segment, and a paragraph segment includes at least one scene segment.

[0035] Identifying the character interaction information in the video file is to obtain the character interaction information through methods such as face detection and text recognition. The character interaction information includes a character role mapping table, face information, and line information. Among them, the line information includes the line content and the role information of the line speaker.

[0036] First, generate the first plot description of the shot segment through the character interaction information, then generate the second plot description of the scene segment through the character interaction information and the first plot description, then generate the third plot description of the paragraph segment according to the character interaction information and the second plot description, and finally integrate each third plot description to generate the overall plot description of the long video. By generating plot descriptions at different granularities, the plot description of a large granularity (which can be understood as a long video) can be obtained by combining the plot description of a small granularity (which can be understood as a short video) and the development logic between plot descriptions of different granularities. In this way, it is not necessary to change the training data of the language model, nor to use long videos as training data to retrain.

[0037] The present application proposes a hierarchical architecture and a video plot description generation framework at multiple different granularities. Based on a multi-modal vision-language large model and a text-language large model, it can process video inputs with very large length differences (such as from 1 second to 3 hours) and generate video plot descriptions for video segments at different granularities and different levels.

[0038] As an alternative embodiment, identifying the person interaction information in a video file includes: obtaining a person role mapping table of the video file, where the person role mapping table includes the mapping relationship between the person role name and the role face; performing face recognition on the video file to obtain the face information of the video file; performing speech recognition on the video file, and obtaining the line information of the video file according to the face information and the person role mapping table, where the line information includes the line content and the role information of the line speaker; determining the person role mapping table, the face information, and the line information as the person interaction information, where the person interaction information carries a timestamp.

[0039] Specifically, obtaining the person role mapping table of the video file includes: collecting the cast list of the video file, including: the person role name and the actor; obtaining the person face corresponding to the actor as the role face from big data or other face databases; establishing a mapping relationship between the person role name and the role face of the same actor and storing it; generating a person role mapping table according to the mapping relationship between the person role name and the role face of each actor.

[0040] Pictures representing the facial features of each role can be collected or generated in advance, and these pictures can be used to construct a face database.

[0041] The person role mapping table can be a database table, where the key is the role name and the value is the corresponding face feature.

[0042] Specifically, performing face recognition on the video file to obtain the face information of the video file includes: extracting each video frame in the video file and performing data preprocessing; using a face detection algorithm to detect faces in the preprocessed frames; performing feature extraction on the detected faces to obtain face feature vectors; matching the extracted face feature vectors with the face features in the person role mapping table to determine the person role name corresponding to each detected face.

[0043] Face detection by the face detection algorithm can be performed through Haar features, HOG (Histogram of Oriented Gradients) + SVM (Support Vector Machine), a deep learning model, etc. The present application does not limit the selection of the face detection algorithm.

[0044] Specifically, performing speech recognition on the video file includes: extracting the audio track from the video file; using a speech recognition algorithm to convert the audio into text; associating the recognized lines with their timestamps in the video.

[0045] Speech recognition is performed through a speech recognition algorithm, which can be through a hidden Markov model or a deep learning model, etc. This application does not limit the selection of the speech recognition algorithm.

[0046] Obtain the line information of the video file according to the face information and the character role mapping table, including: combining the face recognition result and the speech recognition result to determine the character role of the speaker of each line; organizing the line content, the line speaker role information and the time stamp into line information.

[0047] Obtain the human interaction information in the video file, which can provide a data basis for subsequent plot description.

[0048] As an optional embodiment, the video file is hierarchically divided to obtain each shot segment and scene segment, including: extracting multiple video frames in the video file; performing shot boundary detection on each video frame to obtain multiple shot segments; obtaining the attention change between each consecutive shot segment, and dividing the multiple shot segments into different scene segments according to the attention change.

[0049] Specifically, extracting multiple video frames in the video file includes: reading the video file and extracting video frames at a certain frame rate (e.g., extracting one frame or more per second), and the video frames can be extracted through libraries such as FFmpeg and OpenCV.

[0050] Specifically, performing shot boundary detection on each video frame to obtain multiple shot segments includes: performing shot boundary detection on each pair of adjacent frames to determine whether a shot transition occurs; marking all detected shot boundaries; dividing the video into multiple shot segments according to the marked boundaries.

[0051] Shot boundary detection can be based on pixel difference, histogram or image features to determine whether a shot transition occurs. For example, comparing the pixel difference between adjacent video frames, when the difference exceeds a certain threshold, it is determined that a shot transition has occurred; or calculating the color histogram of adjacent frames and comparing the histogram difference to judge the shot transition; or using more complex image features (such as edges, textures, etc.) to detect the shot transition.

[0052] Scene division is based on shot division, and further divides scenes according to the content coherence between shots and the change of the audience's attention.

[0053] Specifically, obtaining the attention change between each consecutive shot segment can evaluate the attention change by calculating the visual feature difference, text content difference or audio feature difference between shot segments, or can also evaluate the attention change by combining the visual feature difference, text content difference or audio feature difference.

[0054] The change in attention can be evaluated based on visual features, including: using similarity metrics of image features (such as color, texture, and shape, etc.) to evaluate the coherence between shot segments, thereby determining the change in attention.

[0055] The change in attention can be evaluated based on text content, including: extracting the text content in the video (the line information can be extracted from the character interaction information); using the similarity of the text content to evaluate the coherence, thereby determining the change in attention.

[0056] The change in attention can be evaluated based on audio features, including: using the similarity of audio features (such as volume, pitch, and timbre, etc.) to evaluate the coherence, thereby determining the change in attention.

[0057] Specifically, multiple shot segments are divided into different scene segments according to the change in attention, including: calculating the change value of attention between adjacent shot segments; when the change value of attention exceeds the threshold, it is determined that a scene switch has occurred, and when the change value of attention does not exceed the threshold, it is determined that no scene switch has occurred; two shot segments without a scene switch are divided into the same scene segment, and two shot segments with a scene switch are divided into different scene segments.

[0058] Through the hierarchical division of the video file, each shot segment and scene segment can be obtained, which helps to generate video plot descriptions at different granularities subsequently.

[0059] As an optional embodiment, the first plot description of each shot segment is generated according to the character interaction information, including: performing plot recognition on the shot segment through the first language model to output the initial plot description; extracting each video frame in the shot segment, where the video frame carries a timestamp; aligning the character interaction information with each video frame in the shot segment according to the timestamp to obtain the shot information corresponding to the shot segment; inputting the initial plot description and the shot information into the second language model to generate the first plot description of the shot segment through the second language model.

[0060] The first language model is a vision-language large model, which can generate an initial plot description of a short video (i.e., a shot segment). The generation of the initial plot description is a low-level visual description, mainly describing around dimensions such as objects, scenes, action behaviors, character attributes, and expressions in the visual picture, and generating primary description texts.

[0061] Specifically, aligning the character interaction information with each video frame in the shot segment according to the timestamp to obtain the shot information corresponding to the shot segment includes: extracting face information, line content, and line speaker from the character interaction information; aligning the face information, line content, and line speaker to each video frame in the shot segment according to the timestamp to obtain the shot information.

[0062] In addition, in addition to aligning character interaction information with each video frame in the shot clip on the time axis, character interaction information can also be aligned with each video frame in the shot clip on the spatial axis. By analyzing the spatial position of the face in each video frame (such as the coordinates on the screen), the visual alignment of the face and the character is ensured.

[0063] The second language model provided in this application can be an LLM (Large Language Model). By training massive amounts of text data, this model masters the grammar, semantics, and contextual information of the language, so that it can accurately process and generate text language that users can understand.

[0064] The second language model provided in this application may also be other language models, such as GPT-4 (Generative Pre-training Transformer 4).

[0065] For example, the initial plot description and shot information are used as input data and input into the second language model. The second language model will generate a plot description that conforms to the video and text images and has a certain logic based on the entire story description, detailed description, character roles, and line speakers (for example, you can imagine the script description format, scene, character roles, and character dialogue content).

[0066] By combining preliminary plot recognition from the first language model, extraction and alignment of video frames, and further optimization using the second language model, a first-level plot description for each shot is generated. This not only considers the visual content of the shot but also incorporates character interactions aligned with video frame timestamps, providing a more detailed and accurate plot description.

[0067] As an optional embodiment, a second plot description of a scene segment is generated based on the first plot description of each shot segment in the scene segment and the character interaction information, including: extracting dialogue information from the character interaction information; determining the first plot description and dialogue information as context information; and inputting the context information under the same scene segment into a second language model to generate a second plot description of the scene segment through the second language model.

[0068] Specifically, the first plot description and the dialogue information are determined as context information, including: summarizing and integrating the first plot descriptions of all shot segments in the scene segment; inserting the extracted dialogue information into the first plot description according to the time sequence and scene logic to obtain the context information.

[0069] Use the first plot descriptions and line information of each shot segment in the same scene segment as input data, and input it into the second language model to obtain the second plot description generated by the model.

[0070] Effectively integrate the first plot descriptions of each shot segment and the line information in the character interaction information to generate a more detailed and coherent second plot description of the scene segment, which not only improves the accuracy and readability of the plot description, but also enhances the coherence and attractiveness of the story.

[0071] As an optional embodiment, generate the third plot description of the paragraph segment according to the second plot description of each scene segment and the character interaction information, including: dividing multiple scene segments into each paragraph segment according to the second plot description of each scene segment; inputting the second plot descriptions of the scene segments in the same paragraph segment into the second language model to generate the third plot description of the paragraph segment through the second language model.

[0072] This application also needs to divide multiple scene segments into each paragraph segment, and the division basis is the second plot description of the scene segment.

[0073] Use the second plot descriptions of each scene segment in the same paragraph segment as input data, input it into the second language model, and combine the idea of the thought tree to obtain the third plot description generated by the model.

[0074] Since the video frame carries a timestamp, the start time of each paragraph segment can be known. According to the start time corresponding to each paragraph segment, connect each paragraph segment in an orderly manner to generate an orderly and complete plot description of the long video, so as to obtain the overall plot description of the entire video file.

[0075] Integrate the second plot descriptions of multiple scene segments into a coherent third plot description of the paragraph segment, so as to construct a complete and organized story narrative.

[0076] As an optional embodiment, divide multiple scene segments into each paragraph segment according to the second plot description of each scene segment, including: identifying the event keywords in each second plot description to determine the key events of each scene segment; determining the correlation degree of the key events of two adjacent scene segments; if the correlation degree of the key events of two adjacent scene segments is greater than or equal to the preset threshold, determine the two adjacent scene segments as the same paragraph segment, and if the correlation degree of the key events of two adjacent scene segments is less than the preset threshold, divide the two adjacent scene segments into different paragraph segments.

[0077] For the division of scene-level granularity, it is mainly considered from a visual perspective whether there are changes in this scene. However, from the perspective of plot understanding, if different scene segments describe the same event or plot, then these scene segments need to be merged and defined as "paragraph segments".

[0078] Identify and extract event keywords from each second plot description. These event keywords usually represent the core activities or turning points in the scene. For example, the keywords may include "encounter", "conflict", "resolution", "farewell", etc.

[0079] The degree of association of key events can be whether two key events are in the same location, involve the same characters, or have a direct chronological relationship, etc.

[0080] For two adjacent scene segments, calculate the degree of association between their key events according to the defined criteria, and a numerical score (e.g., 1 - 10 points) can be used to represent the degree of association.

[0081] Judge whether two scene segments belong to the same paragraph segment by comparing the degree of association of key events with a preset threshold. If a clear distinction between paragraphs is desired, the preset threshold can be set higher; if a certain coherence between paragraphs is desired, the preset threshold can be set lower.

[0082] If the degree of association of two adjacent scene segments is greater than or equal to the preset threshold, then these two scene segments are determined to be the same paragraph segment. If the degree of association of two adjacent scene segments is less than the preset threshold, then these two scene segments are divided into different paragraph segments. Repeat the above steps until all scene segments are assigned to the corresponding paragraph segments.

[0083] Exemplarily, using the characters appearing in each scene and the plot description of this scene, first sparsify the scene-level plot description (into one sentence or several important keywords) using a text large model. Using these sparse descriptions, multiple different scene granularities, but with certain similarity and coherence in plot descriptions, are merged into a large category, called "paragraph".

[0084] The plot description at the segment granularity has a low-level visual description effect, can better capture fine-grained character behaviors, actions, and expressions, and can combine multi-modal fine-grained information such as visual images, character expressions, attributes, environments, actions, and line contents. Moreover, this application also uses the multi-modal idea to generate plot descriptions at the scene granularity and paragraph granularity, not just simply using the text language large model.

[0085] By dividing key events with close associations into the same paragraph segment, readers or viewers can more easily follow the development context of the plot and understand the causal relationships and logical connections between various events.

[0086] This application provides a method for generating a video plot description, including: identifying the character interaction information in the video file, and performing hierarchical division on the video file to obtain each shot segment and scene segment, where the scene segment includes at least one shot segment; generating a first plot description for each shot segment according to the character interaction information; generating a second plot description for the scene segment according to the first plot descriptions of each shot segment in the scene segment and the character interaction information; generating a third plot description for the paragraph segment according to the second plot descriptions of each scene segment and the character interaction information, where the paragraph segment is composed of at least one scene segment; integrating each third plot description to generate an overall plot description of the video file. By dividing the long video file into segments at different levels, then generating the plot description in ascending order of segment granularity, and finally obtaining the overall plot description of the long video, the problem of being difficult to directly use a language model to generate the plot description of the long video is solved.

[0087] According to another aspect of the embodiments of the present application, the present application provides a device for generating a video plot description, as Figure 2 shown, including:

[0088] An identification module 202, configured to identify the character interaction information in the video file, and perform hierarchical division on the video file to obtain each shot segment and scene segment, where the scene segment includes at least one shot segment;

[0089] A first generation module 204, configured to generate a first plot description for each shot segment according to the character interaction information;

[0090] A second generation module 206, configured to generate a second plot description for the scene segment according to the first plot descriptions of each shot segment in the scene segment and the character interaction information;

[0091] A third generation module 208, configured to generate a third plot description for the paragraph segment according to the second plot descriptions of each scene segment and the character interaction information, where the paragraph segment is composed of at least one scene segment;

[0092] An integration module 210, configured to integrate each third plot description to generate an overall plot description of the video file.

[0093] It should be noted that the recognition module 202 in this embodiment can be used to execute step 101 in the embodiment of the present application, the first generation module 204 in this embodiment can be used to execute step 103 in the embodiment of the present application, the second generation module 206 in this embodiment can be used to execute step 105 in the embodiment of the present application, the third generation module 208 in this embodiment can be used to execute step 107 in the embodiment of the present application, and the integration module 210 in this embodiment can be used to execute step 109 in the embodiment of the present application.

[0094] Optionally, the recognition module 202 is further configured to obtain a character mapping table of the video file, where the character mapping table includes the mapping relationship between the character name and the character face; perform face recognition on the video file to obtain the face information of the video file; perform speech recognition on the video file, and obtain the line information of the video file according to the face information and the character mapping table, where the line information includes the line content and the role information of the line speaker; determine the character mapping table, the face information, and the line information as character interaction information, where the character interaction information carries a timestamp.

[0095] Optionally, the recognition module 202 is further configured to extract multiple video frames from the video file; perform shot boundary detection on each video frame to obtain multiple shot segments; obtain the attention change between each consecutive shot segment, and divide the multiple shot segments into different scene segments according to the attention change.

[0096] Optionally, the first generation module 204 is further configured to perform plot recognition on the shot segment through a first language model and output an initial plot description; extract each video frame in the shot segment, where the video frame carries a timestamp; align the character interaction information with each video frame in the shot segment according to the timestamp to obtain shot information corresponding to the shot segment; input the initial plot description and the shot information into a second language model to generate a first plot description of the shot segment through the second language model.

[0097] Optionally, the second generation module 206 is further configured to extract line information from the character interaction information; determine the first plot description and the line information as context information; input the context information in the same scene segment into the second language model to generate a second plot description of the scene segment through the second language model.

[0098] Optionally, the third generation module 208 includes:

[0099] a sub-module for dividing multiple scene segments into respective paragraph segments according to the second plot description of each scene segment;

[0100] A generating sub-module, configured to input a second plot description of a scene segment under the same paragraph segment into a second language model, so as to generate a third plot description of the paragraph segment through the second language model.

[0101] Optionally, the sub-module for dividing is further configured to identify event keywords in each second plot description to determine the key events of each scene segment; determine the correlation degree of the key events of two adjacent scene segments; if the correlation degree of the key events of two adjacent scene segments is greater than or equal to a preset threshold, determine the two adjacent scene segments as the same paragraph segment, and if the correlation degree of the key events of two adjacent scene segments is less than the preset threshold, divide the two adjacent scene segments into different paragraph segments.

[0102] It should be noted here that the examples and application scenarios implemented by the above modules and corresponding steps are the same, but are not limited to the content disclosed in the above embodiments.

[0103] According to another aspect of the embodiments of the present application, the present application provides an electronic device, as Figure 3 shown, including a memory 301, a processor 303, a communication interface 305 and a communication bus 307. A computer program that can run on the processor 303 is stored in the memory 301. The memory 301 and the processor 303 communicate through the communication interface 305 and the communication bus 307. When the processor 303 executes the computer program, the steps of the above method are implemented.

[0104] The memory and the processor in the above electronic device communicate through a communication bus and a communication interface. The communication bus may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc.

[0105] The memory may include a Random Access Memory (RAM), and may also include a non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located far from the aforementioned processor.

[0106] The above-mentioned processor may be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it may also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0107] According to another aspect of the embodiments of the present application, there is also provided a computer-readable medium having non-volatile program code executable by a processor.

[0108] Optionally, the specific examples in this embodiment may refer to the examples described in the above embodiments, and will not be repeated here.

[0109] When the embodiments of the present application are specifically implemented, reference may be made to the above various embodiments, and they have corresponding technical effects.

[0110] It can be understood that these embodiments described herein can be implemented using hardware, software, firmware, middleware, microcode, or a combination thereof. For hardware implementation, the processing unit can be implemented in one or more Application Specific Integrated Circuits (ASICs), Digital Signal Processors (DSPs), Digital Signal Processing Devices (DSPDs), Programmable Logic Devices (PLDs), Field-Programmable Gate Arrays (FPGAs), general-purpose processors, controllers, microcontrollers, microprocessors, other electronic units for performing the functions described in the present application, or a combination thereof.

[0111] For software implementation, the technologies described herein can be implemented by units that execute the functions described herein. The software code can be stored in a memory and executed by a processor. The memory can be implemented inside or outside the processor.

[0112] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in connection with the embodiments disclosed herein can be implemented in electronic hardware, or in a combination of computer software and electronic hardware. Whether these functions are executed in hardware or software depends on the specific application and design constraints of the technical solution. Skilled artisans may use different methods for each specific application to implement the described functions, but such implementation should not be considered to exceed the scope of this application.

[0113] Those skilled in the art can clearly understand that for the convenience and brevity of description, the specific working processes of the systems, devices, and units described above can refer to the corresponding processes in the foregoing method embodiments and will not be repeated here.

[0114] In the embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the modules is only a logical function division, and there may be other division methods in actual implementation. For example, multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed couplings or direct couplings or communication connections to each other can be through some interfaces, and the indirect couplings or communication connections of devices or units can be in electrical, mechanical, or other forms.

[0115] The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0116] In addition, the functional units in the various embodiments of this application can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit.

[0117] When the above-mentioned functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on such an understanding, the technical solutions of the embodiments of the present application, in essence, or the parts that contribute to the prior art, or parts of the technical solutions, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the embodiments of the present application. The foregoing storage medium includes: various media such as USB flash drives, mobile hard disks, ROM, RAM, magnetic disks, or optical discs that can store program codes. It should be noted that in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements, but also includes other elements not expressly listed, or also includes elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "including a..." does not exclude the existence of additional identical elements in the process, method, article or device including the said element.

[0118] The above are only specific embodiments of the present application, enabling those skilled in the art to understand or implement the present application. Various modifications to these embodiments will be obvious to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to these embodiments shown herein, but rather to the broadest scope consistent with the principles and novel features claimed herein.

Claims

1. A method for generating a video plot description, characterized in that, Including: Identifying the human interaction information in the video file, hierarchically partitioning the video file to obtain each shot segment and scene segment, where the scene segment includes at least one of the shot segments; Generating a first plot description for each of the shot segments according to the human interaction information; Generating a second plot description for the scene segment according to the first plot descriptions of the shot segments in the scene segment and the human interaction information; Generating a third plot description for the paragraph segment according to the second plot descriptions of the scene segments and the human interaction information, where the paragraph segment is composed of at least one of the scene segments; Integrating the third plot descriptions to generate an overall plot description of the video file.

2. The method according to claim 1, wherein The identifying the human interaction information in the video file includes: Obtaining a human character mapping table of the video file, where the human character mapping table includes the mapping relationship between the human character name and the character face; Performing face recognition on the video file to obtain the face information of the video file; Performing speech recognition on the video file and obtaining the line information of the video file according to the face information and the human character mapping table, where the line information includes the line content and the role information of the line speaker; Determining the human character mapping table, the face information, and the line information as the human interaction information, where the human interaction information carries a timestamp.

3. The method according to claim 1, wherein The hierarchically partitioning the video file to obtain each shot segment and scene segment includes: Extracting a plurality of video frames from the video file; Performing shot boundary detection on each of the video frames to obtain a plurality of the shot segments; Obtaining the attention change between each consecutive shot segment and partitioning the plurality of shot segments into different scene segments according to the attention change.

4. The method according to claim 1, wherein The generating a first plot description for each of the shot segments according to the human interaction information includes: Performing plot recognition on the shot segment through a first language model and outputting an initial plot description; Extracting each video frame in the shot segment, where the video frame carries a timestamp; Aligning the human interaction information with each of the video frames in the shot segment according to the timestamp to obtain shot information corresponding to the shot segment; Inputting the initial plot description and the shot information into a second language model to generate the first plot description of the shot segment through the second language model.

5. The method according to claim 2, wherein The generating a second plot description for the scene segment according to the first plot descriptions of the shot segments in the scene segment and the human interaction information includes: Extracting the line information from the human interaction information; Determining the first plot description and the line information as context information; Inputting the context information under the same scene segment into a second language model to generate the second plot description of the scene segment through the second language model.

6. The method according to claim 2, wherein Generating a third plot description of a paragraph segment based on the second plot descriptions of the respective scene segments and the character interaction information includes: Dividing a plurality of the scene segments into respective paragraph segments according to the second plot descriptions of the respective scene segments; Inputting the second plot descriptions of the scene segments under the same paragraph segment into a second language model to generate the third plot description of the paragraph segment through the second language model.

7. The method according to claim 6, wherein The dividing a plurality of the scene segments into respective paragraph segments according to the second plot descriptions of the respective scene segments includes: Identifying event keywords in each of the second plot descriptions to determine the key events of each of the scene segments; Determining the correlation degree of the key events of two adjacent scene segments; If the correlation degree of the key events of two adjacent scene segments is greater than or equal to a preset threshold, determining the two adjacent scene segments as the same paragraph segment, and if the correlation degree of the key events of two adjacent scene segments is less than the preset threshold, dividing the two adjacent scene segments into different paragraph segments.

8. A generating device for video plot description, characterized in that including: An identification module, configured to identify character interaction information in a video file, and perform hierarchical division on the video file to obtain respective shot segments and scene segments, where the scene segment includes at least one of the shot segments; A first generation module, configured to generate a first plot description of each of the shot segments according to the character interaction information; A second generation module, configured to generate a second plot description of the scene segment according to the first plot descriptions of the respective shot segments in the scene segment and the character interaction information; A third generation module, configured to generate a third plot description of a paragraph segment according to the second plot descriptions of the respective scene segments and the character interaction information, where the paragraph segment is composed of at least one of the scene segments; An integration module, configured to integrate each of the third plot descriptions to generate an overall plot description of the video file.

9. An electronic device, comprising a memory, a processor, a communication interface and a communication bus, wherein a computer program capable of running on the processor is stored in the memory, and the memory and the processor communicate through the communication bus and the communication interface, characterized in that, When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 above are implemented.

10. A computer-readable medium having non-volatile program code executable by a processor, characterized in that, The program code causes the processor to execute the method according to any one of claims 1 to 7.

Images