Video generation method and device, electronic equipment, storage medium and program
By analyzing long videos and adjusting the plot segments, short videos are automatically generated, solving the problem of low efficiency in short video production and achieving automated production and rhythm optimization.
Patent Information
- Application Number
- CN202411154056.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-21
- Publication Date
- 2024-11-29
AI Technical Summary
Short videos are inefficient to produce, relying mainly on manual production by users, making automated production difficult.
By analyzing long videos, using a cross-modal model to determine the plot description range, and adjusting it, several relatively independent video units are generated, and finally, short videos containing key plot points are automatically generated.
It enables automated production of short videos, shortens production time, improves production efficiency, and optimizes video pacing, making the generated short videos more suitable for distribution and delivery.
Smart Images

Figure CN119052601A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of multimedia processing technology, and in particular to a video generation method, a video generation device, an electronic device, a computer-readable storage medium and a computer program. Background Art
[0002] Long videos generally refer to videos that are longer than half an hour, mainly film and television dramas, and are usually produced by professional companies; short videos are short videos, generally videos that are less than 5 minutes long and are circulated on new media on the Internet.
[0003] Short videos are a way of disseminating Internet content, but most of them are produced manually by users, and the efficiency of video production is low. Summary of the invention
[0004] The purpose of the embodiments of the present invention is to provide a video generation method, a video generation device, an electronic device, a computer-readable storage medium and a computer program to realize the automatic production of short videos of plot mixes. The specific technical solution is as follows:
[0005] In a first aspect of the present invention, a video generation method is provided, the method comprising:
[0006] Acquire a first video, and analyze the first video to obtain video information;
[0007] Determine a plot description interval from the first video, adjust the plot description interval according to the video information, and obtain an interval positioning result; the interval positioning result is used to indicate a number of video units with relatively independent plots;
[0008] A second video is generated according to the interval positioning result.
[0009] In a second aspect of the present invention, a video generation device is provided, the device comprising:
[0010] A video information acquisition module, used to acquire a first video and analyze the first video to obtain video information;
[0011] A plot description interval determination module, used to determine a plot description interval from the first video;
[0012] A plot description interval adjustment module, used to adjust the plot description interval according to the video information to obtain an interval positioning result; the interval positioning result is used to indicate a number of relatively independent video units of the plot;
[0013] A video generation module is used to generate a second video according to the interval positioning result.
[0014] In another aspect of the implementation of the present invention, there is also provided an electronic device, including a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus; the memory is used to store computer programs; and the processor is used to implement any of the video generation methods when executing the programs stored in the memory.
[0015] In another aspect of the present invention, a computer-readable storage medium is provided, wherein instructions are stored in the computer-readable storage medium, and when the computer-readable storage medium is run on a computer, the computer executes any of the above-mentioned video generation methods.
[0016] In yet another aspect of the present invention, a computer program product comprising instructions is provided, which, when executed on a computer, enables the computer to execute any of the above-mentioned video generation methods.
[0017] The video generation method, device, electronic device and storage medium provided by the embodiments of the present invention adjust the plot description interval in the first video based on the video information obtained by analyzing the video content of the first video to obtain an interval positioning result. The interval positioning result can be used to indicate a number of video units with relatively independent plots. That is, the first video can be divided into a number of video units with relatively independent plots, and then a second video is generated based on the obtained video units, thereby realizing the automatic production of short videos with mixed plots, thereby shortening the production time of short videos and improving the production efficiency of short videos. BRIEF DESCRIPTION OF THE DRAWINGS
[0018] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art are briefly introduced below.
[0019] Figure 1 A flowchart of a video generation method according to an embodiment of the present invention;
[0020] Figure 2 is a flowchart of another video generation method embodiment in an embodiment of the present invention;
[0021] Figure 3 A schematic diagram of a video generation process provided by an embodiment of the present invention;
[0022] Figure 4 is a structural block diagram of an embodiment of a video generating device in an embodiment of the present invention;
[0023] Figure 5 The figure is a schematic diagram of the structure of an electronic device in an embodiment of the present invention. DETAILED DESCRIPTION
[0024] The technical solutions in the embodiments of the present invention will be described below in conjunction with the accompanying drawings in the embodiments of the present invention.
[0025] Short video is a way of disseminating Internet content. The embodiment of the present invention can use AI technology to analyze and process long video content, automatically generate short videos containing key plots, and realize the automatic production of short videos of mixed plots. Specifically, the video information obtained by analyzing the video content of the first video can be used to adjust the plot description interval in the first video to obtain the interval positioning result. The interval positioning result can be used to indicate several relatively independent video units of the plot, that is, the first video can be divided into several relatively independent video units of the plot, and then the second video is generated based on the obtained video units, so as to realize the automatic production of short videos of mixed plots. Compared with manual video editing, it greatly reduces the cost of manual participation, shortens the production time of short videos, improves the production efficiency of short videos, and then increases the video output. Furthermore, the video duration of the video unit obtained after the split can be compressed to achieve the concentration of the video, further optimize the mixed video effect, so that the video rhythm of the mixed short video produced is faster and more suitable for distribution and delivery. In addition, the mixed short video produced can be applied to user growth business, and the mixed video produced for film and television dramas is also conducive to the distribution of short videos, attracting users to watch long videos in the station.
[0026] Reference Figure 1 , shows a flowchart of a video generation method embodiment of the present invention, which may specifically include the following steps:
[0027] Step 101, obtaining a first video, and analyzing the first video to obtain video information;
[0028] The first video may be video data provided as video material when a new video is produced. It may be an existing video in a video station, and may be long video or short video data. In an embodiment of the present invention, in order to enrich the plot richness of the mixed and produced video, a long video, such as several movies or several episodes of a TV series, may usually be used as the first video for plot splitting.
[0029] In order to facilitate the subsequent plot segmentation of the first video, we can first perform video analysis on the video content of the first video to obtain the video information of the first video, and based on the obtained video information, we can learn the relevant basic video information of the video, such as dialogue information, transition information, etc., which is conducive to the subsequent plot segmentation processing.
[0030] Step 102, determining a plot description interval from the first video, adjusting the plot description interval according to the video information, and obtaining an interval positioning result;
[0031] The first video obtained may include video clips corresponding to multiple plots, and different plot clips correspond to different plot description contents. The plot description interval may mainly be a video playback interval for a certain plot description content. In actual applications, the video playback interval may be determined by the video start time and video end time corresponding to the plot description content.
[0032] Exemplarily, a pre-trained cross-modal model can be obtained, and the plot description interval can be determined from the first video according to the cross-modal model, wherein the cross-modal model is mainly a model obtained by explicit modeling based on the correlation between image and text, and can be mainly used for cross-modal retrieval or cross-mode query. Cross-modal retrieval / cross-modal query is a new retrieval method for returning related retrieval results of other different modalities for a query term of one modality. For example, when the image and text spaces have a natural correspondence, the two spaces can be reversibly mapped. In an embodiment of the present invention, the cross-modal query involved can be a query of two different modalities between the text of the input plot description content and the image frame.
[0033] In one embodiment of the present invention, the plot description interval determined from the first video can be used as the initial result of plot segment positioning. At this time, the initial result can be adjusted accordingly to obtain the final result of plot segment positioning, thereby achieving accurate segmentation of the plot fragments in the first video, so as to facilitate subsequent plot remixing during video production and realize automatic production of plot remixed short videos.
[0034] The adjustment of the plot description interval can be expressed as the adjustment of the video start time and the video end time, and the interval positioning result obtained after the adjustment can refer to the adjusted video start time and the adjusted video end time. Since the object of the adjustment is the video playback interval for a single plot description content, the interval positioning result can be used as the final result of plot segment positioning, which is used to indicate several relatively independent video units of the plot.
[0035] Step 103: Generate a second video according to the interval positioning result.
[0036] The several relatively independent video units of the plot indicated by the interval positioning result may be various key plot segments. At this time, a second video containing the key plot can be automatically generated based on the acquired key plot segments, thereby realizing the automatic production of short videos of the plot mix.
[0037] In an embodiment of the present invention, the plot description interval in the first video is adjusted by analyzing the video information obtained from the video content of the first video to obtain an interval positioning result. The interval positioning result can be used to indicate a number of video units with relatively independent plots. That is, the first video can be divided into a number of video units with relatively independent plots, and then a second video is generated based on the obtained video units, thereby realizing the automatic production of short videos with mixed plots, thereby shortening the production time of short videos and improving the production efficiency of short videos.
[0038] Reference Figure 2 , shows a flowchart of another video generation method embodiment in an embodiment of the present invention, which may specifically include the following steps:
[0039] Step 201, analyzing the video content of the first video to obtain the line information and transition information of the first video;
[0040] In an embodiment of the present invention, in order to enrich the plot richness of the mixed and produced video, a long video, such as several movies or several episodes of TV series data, can usually be used as the first video for plot splitting.
[0041] In one embodiment of the present invention, the plot of the video can be split based on the relevant basic video information of the first video. Specifically, the video content of the first video can be analyzed to obtain the dialogue information and transition information of the first video. The dialogue information can refer to the single-sentence dialogue information in the video, which can be mainly used to indicate the dialogue start time and the dialogue end time of the single-sentence dialogue; the transition information can refer to the scene transition information in the video, which can be mainly used to indicate the transition moment.
[0042] like Figure 3 As shown, the video analysis of the video content of the first video may include dialogue analysis and transition analysis. In the process of obtaining video information, the dialogue information can be obtained by performing dialogue analysis using OCR recognition technology, which can be specifically manifested as performing text recognition on the characters contained in the dialogue in the video content of the first video to locate the dialogue information of the first video; and the transition information can be obtained by using lens detection technology, which can be specifically manifested as performing lens detection on the points where the lens screen switches in the first video to identify the transition information of the first video.
[0043] Step 202, obtaining a pre-trained cross-modal model, and determining a plot description interval from the first video according to the cross-modal model;
[0044] In practical applications, the video information relied upon in plot splitting can be mainly used as a basis for adjustment processing, and the object for adjustment can be a plot description interval determined from the first video. The plot description interval can be a video playback interval for a certain plot description content.
[0045] For the determination of the plot description interval, Figure 3 As shown, this can be achieved by utilizing cross-modal retrieval. Specifically, a pre-trained cross-modal model can be obtained, and the plot description interval can be determined from the first video according to the cross-modal model.
[0046] Among them, the cross-modal model can be expressed as video-text cross-modal retrieval, which can be mainly a model obtained by explicit modeling based on the correlation between images and texts. When the image and text spaces have a natural correspondence, the two spaces can be reversibly mapped to achieve cross-modal retrieval or cross-modal query. Specifically, the effect of cross-modal retrieval can be achieved by minimizing the difference between the video modality representation and the text modality representation of the same video and determining the similarity of representations in different modalities in the feature space.
[0047] In an embodiment of the present invention, the cross-modal query involved may be a query of two different modalities between the input plot description content text and the image frame. In a specific implementation, after obtaining a pre-trained cross-modal model, the plot description content can be obtained from the video content of the first video, specifically the description text for the plot description content, so that the plot description content is input into the cross-modal model, so as to output the video start time and video end time corresponding to the plot description content, and then determine the interval from the video start time to the video end time as the plot description interval.
[0048] It should be noted that the cross-modal model used may be CLIP (Contrastive Language-Image Pretraining, a large cross-modal model that maps images and texts into a shared embedding space so that the model can understand images and texts at the same time), UNIMO (Unified Multimodal Pretraining for Language Understanding, a large cross-modal model that focuses on integrating images and texts to improve the performance of natural language understanding and generation), M6: A Language-Model-Driven Multimodal Pretraining Framework (a multimodal pre-training framework), etc., and the embodiments of the present invention are not limited to this.
[0049] Exemplarily, for the plot description interval obtained by the cross-modal model output, the part-of-speech classification can be performed on each word in the text sequence based on the text sequence of the input plot description text, and a corresponding text sequence feature vector can be output. At this time, the text sequence feature vector can be similar (correlated) with the video sequence feature vector for multiple video sequences in another vector space, so as to determine the target frame corresponding to the plot description content based on the feature similarity measure calculated in the vector space. The target frame can include a target frame corresponding to the start time position of the video and a target frame corresponding to the end time position of the video, so as to determine the video sequence of pictures corresponding to the two target frames as the plot description interval. This embodiment of the present invention is not limited to this.
[0050] Step 203, adjusting the plot description interval according to the line information to obtain an interval positioning result;
[0051] The plot description interval determined in the above steps is a potential or predicted interval, which may be equivalent to the initial result of plot segment positioning. The embodiment of the present invention may adjust the initial result accordingly to obtain the final result of plot segment positioning, that is, the interval positioning result, such as Figure 3 As shown, the plot description interval can be adjusted based on video analysis, such as video information obtained from dialogue analysis and transition analysis.
[0052] In one embodiment of the present invention, the video information may include dialogue information, which may be used to indicate the start time and end time of a single line of dialogue. At this time, the plot description interval located by the aforementioned cross-modal model may be adjusted and corrected in combination with the start of the lines to generate an interval location result containing a complete plot semantic paragraph.
[0053] From the perspective of lines, whether the plot semantics of the interval is complete can be judged based on whether the interval contains the complete line duration; and whether the line duration is complete can be judged based on whether the single line content is truncated. That is, if there is a truncated single line content in the paragraph content of the plot description interval, it can be determined that the paragraph content does not guarantee the integrity of the line duration, and then it is determined that the interval is not an interval that can contain a complete plot semantics paragraph. At this time, the plot description interval can be adjusted based on the line information to obtain the interval positioning result.
[0054] The plot description interval is mainly determined by the video start time and video end time corresponding to the plot description content. The adjustment of the plot description interval can be expressed as the adjustment of the video start time and the video end time.
[0055] When there is truncated single-sentence dialogue content in the paragraph content of the plot description interval, the video start time and / or end time of the plot description interval may be adjusted using the start time and / or end time of the single-sentence dialogue.
[0056] As an example, if the start time of a single line of dialogue is before the start time of the video, and the end time of the single line of dialogue is within the plot description interval, it can be determined that there is truncated single line content in the paragraph content of the plot description interval. At this time, the line start time can be used to adjust the plot description interval. Specifically, the line start time can be used as the video start time of the plot description interval to complete the adjustment and correction of the plot description interval.
[0057] As another example, if the start time of a single line of dialogue is within the plot description interval, and the end time of the single line of dialogue is after the video end time, it can be determined that there is truncated single line of dialogue content in the paragraph content of the plot description interval. At this time, the line end time can be used to adjust the plot description interval. Specifically, the line end time can be used as the video end time of the plot description interval to complete the adjustment and correction of the plot description interval.
[0058] Step 204, adjusting the plot description interval according to the transition information to obtain an interval positioning result;
[0059] In one embodiment of the present invention, the video information may include transition information, which can be used to indicate the transition moment. At this time, the plot description interval located by the aforementioned cross-modal model can be adjusted and corrected in combination with the transition moment to generate an interval positioning result containing a complete plot semantic paragraph.
[0060] From the perspective of picture integrity, whether the plot semantics section of the interval is complete can be judged based on whether the interval contains a complete shot duration; and whether the shot duration is complete can be judged based on whether the single shot content is truncated. That is, if there is a truncated single shot content in the paragraph content of the plot description interval, it can be determined that the paragraph content does not guarantee the integrity of the shot duration, and then it can be determined that the interval is not an interval that can contain a complete plot semantics section. At this time, the plot description interval can be adjusted based on the transition information to obtain the interval positioning result.
[0061] The plot description interval is mainly determined by the video start time and video end time corresponding to the plot description content. The adjustment of the plot description interval can be expressed as the adjustment of the video start time and the video end time.
[0062] Specifically, a scene is composed of a series of continuous shots, a single shot or multiple shots can be determined as the same / different scenes, and the transition moment can include the transition moments of multiple shots. At this time, the shot interval of the current shot can be determined based on the transition moment of the previous shot and the transition moment of the next shot. Specifically, the transition moment of the previous shot is used as the shot start time of the current shot, and the transition moment of the next shot is used as the shot end time of the current shot, thereby determining the shot interval.
[0063] Optionally, when there is truncated single shot content in the paragraph content of the plot description interval, the shot start time and / or shot end time of the single shot may be used to adjust the video start time and video end time of the plot description interval.
[0064] Optionally, when the shot start time and / or shot end time of a single shot is used to adjust the plot description interval, as an example, if the shot start time is before the video start time and the shot end time is within the plot description interval, it can be considered that the shot interval of the current shot exceeds the plot description time, and it can be determined that there is truncated single shot content in the paragraph content of the plot description interval. At this time, the shot start time can be used to adjust the plot description interval, which can be specifically manifested as taking the shot start time as the video start time of the plot description interval to complete the adjustment and correction of the plot description interval.
[0065] As another example, if the shot start time is in the plot description interval and the shot end time is after the video end time, it can be considered that the shot interval of the current shot exceeds the plot description time, and it can be determined that there is a truncated single shot content in the paragraph content of the plot description interval. At this time, the shot end time can be used to adjust the plot description interval. Specifically, the shot end time can be used as the video end time of the plot description interval to complete the adjustment and correction of the plot description interval.
[0066] Optionally, the interval positioning result can be obtained based on any one of the following video information or a combination of the following video information. The video information may include dialogue information, transition information, etc. In order to ensure that the adjusted paragraph interval can meet the requirements of including the complete dialogue duration and the complete shot duration, the plot description interval can usually be adjusted in combination with the dialogue information and transition information to obtain the interval positioning result.
[0067] It should be noted that there is no chronological order for the execution of step 203 and step 204, and they are mainly based on conditions, such as the above-mentioned "the start time of the single-sentence dialogue is before the start time of the video, and the end time of the single-sentence dialogue is within the plot description interval", "the start time of the single-sentence dialogue is within the plot description interval, and the end time of the single-sentence dialogue is after the end time of the video", "the start time of the shot is before the start time of the video, and the end time of the shot is within the plot description interval", "the start time of the shot is within the plot description interval, and the end time of the shot is after the end time of the video", etc., which trigger the adjustment of the plot description interval, so that the adjusted paragraph interval can meet the requirements of including the complete duration of the dialogue without truncating the content of the single-sentence dialogue, and including the complete duration of the shot without truncating the content of the single shot, thereby ensuring that the interval positioning result obtained after the adjustment is an interval containing a complete plot semantic paragraph.
[0068] Optionally, when step 203 and step 204 are executed simultaneously, that is, when the plot description interval is adjusted according to the dialogue information and the transition information, since the adjustment of the plot description interval is specifically manifested as the adjustment of its video start time and video end time, the overlapping paragraph interval can be obtained in the process of adjusting based on the dialogue information and based on the transition information, and the overlapping paragraph interval can be used as the adjusted paragraph interval, so that the adjusted paragraph interval can simultaneously meet the requirements of including the complete dialogue duration and the complete shot duration.
[0069] Exemplarily, the acquisition of overlapping paragraph intervals can be performed by acquiring the video start time 1 and the video end time 1 of the paragraph interval after the plot description interval is adjusted in the manner of step 203, and simultaneously acquiring the video start time 2 and the video end time 2 of the paragraph interval after the plot description interval is adjusted in the manner of step 204. At this time, two time intervals are generated based on the video time of the adjusted paragraph interval, for example, [video start time 1, video end time 1] and [video start time 2, video end time 2], and the overlapping interval of [video start time 1, video end time 1] and [video start time 2, video end time 2] is used as the start time and end time of the overlapping paragraph interval, thereby obtaining the overlapping paragraph interval, and using the overlapping paragraph interval as the final adjusted paragraph interval. This is not limited in the embodiments of the present invention.
[0070] Step 205: Generate a second video according to the interval positioning result.
[0071] The interval positioning result contains a complete plot semantic segment, which can be used to indicate several relatively independent video units of the plot. At this time, based on the key plot segments obtained, a second video containing the key plot can be automatically generated to realize the automatic production of short videos of plot mixes.
[0072] In a preferred embodiment of the present invention, Figure 3 As shown, the video duration of the video units obtained after the splitting can be compressed to achieve video concentration, so that the video rhythm of the produced mixed short video is faster and more suitable for distribution.
[0073] The compression strategy for video concentration can be expressed as deleting the slow-motion shots in the plot segments and retaining the main dialogue content segments. That is, after compressing the video length of the video unit to obtain the dialogue segments of the video unit, the dialogue segments can be spliced in time sequence to obtain the second video of the plot mix.
[0074] Among them, the video segment of each dialogue segment can contain a pair of start / end point information, and an independent video semantic segment interval may contain several shots, each shot contains a pair of start / end point information. At this time, the minimum start time video corresponding to all the shots contained can be used as the start time of the video semantic segment interval (i.e., video unit), and the maximum end time of all the shots contained can be used as the end time of the video semantic segment interval (i.e., video unit), and the video can be spliced based on the start / end point information of each video segment.
[0075] In practical applications, the compression process can mainly delete the relevant video content with less information in the plot paragraphs of the video unit. Among them, the less information can be expressed as the number of times each line appears in the entire video. Lines with a large number of appearances (for example, greater than or equal to 5 times) can be determined as repeated lines. Since the information brought by the repeated lines is repeated, the extracted content is more but the amount of semantic information contained is less; the less information can also be expressed as the number of characters contained in a single line. When the number of characters is small (for example, less than or equal to 5), it can be considered that the line contains less semantic information.
[0076] In a specific implementation, since the duration of the lines in the slow shots is relatively low, the amount of information it brings is relatively small. In order to avoid redundant information other than the lines, the slow shots in the plot section of the video unit can be obtained, and the first target shot in the slow shots can be deleted. The first target shot can refer to a shot containing non-lines content, that is, the shot containing non-lines content in the slow shots is deleted, and only the lines content is retained; on the basis of retaining the lines content, there may be repeated lines content, in which case the amount of information brought is relatively small. At this time, the second target shot can be deleted in the video unit where the shot containing non-lines content in the slow shots has been deleted, and the dialogue segment of the video unit finally used for time sequence splicing is obtained. Among them, the second target shot can refer to a shot with repeated lines content, that is, the shot with repeated lines content can be deleted.
[0077] In the embodiment of the present invention, by analyzing the video information obtained from the long video content, the plot description interval in the video is adjusted, the video is divided into a number of relatively independent video units, and then a second video is generated based on the obtained video units, so as to realize the automatic production of short videos of the plot mix and cut, and improve the production efficiency of short videos. Furthermore, the video duration of the video units obtained after the splitting can also be compressed to realize the condensation of the video, so that the video rhythm of the produced mixed and cut short video is faster and more suitable for distribution and delivery.
[0078] It should be noted that, for the sake of simplicity, the method embodiments are described as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.
[0079] Reference Figure 4 , shows a structural block diagram of an embodiment of a video generation device in an embodiment of the present invention, which may specifically include the following modules:
[0080] The video information acquisition module 401 is used to acquire a first video and analyze the first video to obtain video information;
[0081] A plot description interval determination module 402, configured to determine a plot description interval from the first video;
[0082] The plot description interval adjustment module 403 is used to adjust the plot description interval according to the video information to obtain an interval positioning result; the interval positioning result is used to indicate a number of relatively independent video units of the plot;
[0083] The video generation module 404 is used to generate a second video according to the interval positioning result.
[0084] In one embodiment of the present invention, the video information acquisition module 401 may include the following submodules:
[0085] The video information acquisition submodule is used to perform text recognition on the characters contained in the lines in the video content of the first video to obtain the line information of the first video; and to perform shot detection on the points where the shots in the first video switch to obtain the transition information of the first video.
[0086] In one embodiment of the present invention, the plot description interval determination module 402 may include the following submodules:
[0087] The plot description interval determination submodule is used to obtain a pre-trained cross-modal model and obtain plot description content from the video content of the first video; input the plot description content into the cross-modal model, and output a video start time and a video end time corresponding to the plot description content; and determine the interval from the video start time to the video end time as the plot description interval.
[0088] In an embodiment of the present invention, the video information includes dialogue information; the plot description interval adjustment module 403 may include the following submodules:
[0089] A first plot description interval adjustment submodule is used to adjust the plot description interval based on the line information to obtain an interval positioning result when there is a truncated single sentence line content in the paragraph content of the plot description interval;
[0090] The plot description interval is determined based on the video start time and the video end time of the corresponding plot, and the line information is used to indicate the line start time and the line end time of a single line;
[0091] The first plot description interval adjustment submodule may include the following units:
[0092] The first plot description interval adjustment unit is used to determine that there is truncated single-sentence content in the paragraph content of the plot description interval when the speech start time of the single-sentence is before the video start time and the speech end time of the single-sentence is in the plot description interval, and use the speech start time as the video start time of the plot description interval; and / or, when the speech start time of the single-sentence is in the plot description interval and the speech end time of the single-sentence is after the video end time, determine that there is truncated single-sentence content in the paragraph content of the plot description interval, and use the speech end time as the video end time of the plot description interval.
[0093] In an embodiment of the present invention, the video information includes transition information; the plot description interval adjustment module 403 may include the following submodules:
[0094] A second plot description interval adjustment submodule is used to adjust the plot description interval based on the transition information to obtain an interval positioning result when there is a truncated single shot content in the paragraph content of the plot description interval;
[0095] The plot description interval is determined based on the video start time and video end time of the corresponding plot, and the transition information is used to indicate the transition moments of multiple shots;
[0096] The second plot description interval adjustment submodule may include the following units:
[0097] The second plot description interval adjustment unit is used to determine the shot interval of the current shot based on the transition moment of the previous shot and the transition moment of the next shot; the shot interval is determined based on the shot start time and the shot end time; if the shot start time is before the video start time, and the shot end time is located in the plot description interval, it is determined that there is a truncated single shot content in the paragraph content of the plot description interval, and the shot start time is used as the video start time of the plot description interval; and / or, if the shot start time is located in the plot description interval, and the shot end time is after the video end time, it is determined that there is a truncated single shot content in the paragraph content of the plot description interval, and the shot end time is used as the video end time of the plot description interval.
[0098] In one embodiment of the present invention, the video generation module 404 may include the following submodules:
[0099] The video generation submodule is used to compress the video duration of the video unit to obtain the dialogue segment of the video unit; and splice the dialogue segment in time sequence to obtain the second video.
[0100] In one embodiment of the present invention, the video generation submodule may include the following units:
[0101] A dialogue segment acquisition unit is used to acquire slow-motion shots in the plot segment of the video unit, delete a first target shot in the slow-motion shots; in the video unit in which the first target shot in the slow-motion shots is deleted, a second target shot is deleted to obtain a dialogue segment of the video unit; wherein the first target shot is used to indicate a shot containing non-dialogue content, and the second target shot is used to indicate a shot containing repeated dialogue content.
[0102] In an embodiment of the present invention, a video generating device provided by the embodiment of the present invention obtains video information by analyzing the video content of a first video, and adjusts the plot description interval in the first video to obtain an interval positioning result. The interval positioning result can be used to indicate a number of video units with relatively independent plots. That is, the first video can be divided into a number of video units with relatively independent plots, and then a second video is generated based on the obtained video units, thereby realizing automatic production of short videos of mixed plots, thereby shortening the production time of short videos and improving the production efficiency of short videos.
[0103] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.
[0104] The embodiment of the present invention further provides an electronic device, such as Figure 5 As shown, it includes a processor 501, a communication interface 502, a memory 503 and a communication bus 504, wherein the processor 501, the communication interface 502, and the memory 503 communicate with each other through the communication bus 504.
[0105] Memory 503, used for storing computer programs;
[0106] The processor 501 is used to execute the program stored in the memory 503, and implements the following steps:
[0107] Acquire a first video, and analyze the first video to obtain video information;
[0108] Determine a plot description interval from the first video, adjust the plot description interval according to the video information, and obtain an interval positioning result; the interval positioning result is used to indicate a number of relatively independent video units of the plot;
[0109] A second video is generated according to the interval positioning result.
[0110] In an embodiment of the present invention, the electronic device provided by the embodiment of the present invention obtains video information by analyzing the video content of the first video, and adjusts the plot description interval in the first video to obtain an interval positioning result. The interval positioning result can be used to indicate a number of video units with relatively independent plots. That is, the first video can be divided into a number of video units with relatively independent plots, and then a second video is generated based on the obtained video units, thereby realizing automatic production of short videos with mixed plots, thereby shortening the production time of short videos and improving the production efficiency of short videos.
[0111] The communication bus mentioned in the above terminal can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.
[0112] The communication interface is used for communication between the above terminal and other devices.
[0113] The memory may include a random access memory (RAM) or a non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.
[0114] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.
[0115] In another embodiment of the present invention, a computer-readable storage medium is provided. The computer-readable storage medium stores instructions. When the computer-readable storage medium is executed on a computer, the computer executes the video generation method described in any one of the above embodiments.
[0116] In another embodiment of the present invention, a computer program product including instructions is provided. When the computer program product is run on a computer, the computer executes the video generation method described in any one of the above embodiments.
[0117] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive Solid State Disk (SSD)), etc.
[0118] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.
[0119] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.
[0120] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.
Claims
1. A video generation method, characterized in that: The method comprises: Acquire a first video, and analyze the first video to obtain video information; Determine a plot description interval from the first video, adjust the plot description interval according to the video information, and obtain an interval positioning result; the interval positioning result is used to indicate a number of video units with relatively independent plots; A second video is generated according to the interval positioning result.
2. The method according to claim 1, characterized in that The step of analyzing the first video to obtain video information includes: Performing text recognition on characters contained in lines in the video content of the first video to obtain line information of the first video; And, performing lens detection on the points where the lens images in the first video are switched to obtain transition information of the first video.
3. The method according to claim 1, characterized in that The determining of a plot description interval from the first video includes: Obtaining a pre-trained cross-modal model, and obtaining plot description content from the video content of the first video; Inputting the plot description content into the cross-modal model, and outputting a video start time and a video end time corresponding to the plot description content; Determine the interval from the start time of the video to the end time of the video as the plot description interval.
4. The method according to claim 1, characterized in that: The video information includes dialogue information; and adjusting the plot description interval according to the video information to obtain an interval positioning result includes: If there is truncated single-sentence dialogue content in the paragraph content of the plot description interval, the plot description interval is adjusted based on the dialogue information to obtain an interval positioning result; The plot description interval is determined based on the video start time and the video end time of the corresponding plot, and the line information is used to indicate the line start time and the line end time of a single line; If there is truncated single-sentence dialogue content in the paragraph content of the plot description interval, adjusting the plot description interval based on the dialogue information includes: If the speech start time of the single-sentence dialogue is before the video start time, and the speech end time of the single-sentence dialogue is within the plot description interval, it is determined that there is truncated single-sentence dialogue content in the paragraph content of the plot description interval, and the speech start time is used as the video start time of the plot description interval; And / or, if the start time of the single-sentence dialogue is within the plot description interval, and the end time of the single-sentence dialogue is after the video end time, it is determined that there is truncated single-sentence dialogue content in the paragraph content of the plot description interval, and the end time of the dialogue is used as the video end time of the plot description interval.
5. The method according to claim 1, characterized in that The video information includes transition information; and the plot description interval is adjusted according to the video information to obtain an interval positioning result, including: If there is a truncated single shot content in the paragraph content of the plot description interval, the plot description interval is adjusted based on the transition information to obtain an interval positioning result; The plot description interval is determined based on the video start time and video end time of the corresponding plot, and the transition information is used to indicate the transition moments of multiple shots; If there is a truncated single shot content in the paragraph content of the plot description interval, adjusting the plot description interval based on the transition information includes: Determine the shot interval of the current shot based on the transition moment of the previous shot and the transition moment of the next shot; the shot interval is determined based on the shot start time and the shot end time; If the shot start time is before the video start time, and the shot end time is within the plot description interval, it is determined that there is a truncated single shot content in the paragraph content of the plot description interval, and the shot start time is used as the video start time of the plot description interval; And / or, if the shot start time is within the plot description interval and the shot end time is after the video end time, it is determined that there is truncated single shot content in the paragraph content of the plot description interval, and the shot end time is used as the video end time of the plot description interval.
6. The method according to claim 1, characterized in that The generating a second video according to the interval positioning result includes: Compressing the video duration of the video unit to obtain a dialogue segment of the video unit; The dialogue segments are spliced in time sequence to obtain a second video.
7. The method according to claim 6, characterized in that The compressing the video duration of the video unit to obtain the dialogue segment of the video unit includes: Acquire slow-motion shots in the plot section of the video unit, and delete a first target shot in the slow-motion shots; In the video unit in which the first target shot in the slow shot is deleted, the second target shot is deleted to obtain a dialogue segment of the video unit; The first target shot is used to indicate a shot containing non-dialogue content, and the second target shot is used to indicate a shot containing repeated dialogue content.
8. A video generating device, characterized in that: The device comprises: A video information acquisition module, used to acquire a first video and analyze the first video to obtain video information; A plot description interval determination module, used to determine a plot description interval from the first video; A plot description interval adjustment module, used to adjust the plot description interval according to the video information to obtain an interval positioning result; the interval positioning result is used to indicate a number of relatively independent video units of the plot; A video generation module is used to generate a second video according to the interval positioning result.
9. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; Memory, used to store computer programs; The processor is used to implement the video generation method described in any one of claims 1-7 when executing the program stored in the memory.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the video generating method as described in any one of claims 1 to 7 is implemented.
11. A computer program, characterized in that When the computer program is executed by a processor, the video generating method according to any one of claims 1 to 7 is implemented.
Citation Information
Patent Citations
Method and device for generating plot explanation short video and electronic equipment
CN114222196A
Pre-notice automatic generation method and product based on movie and television drama plot content
CN117221670A
Video generation method and device, computer equipment and storage medium
CN117278699A
Video generation techniques
US20240095989A1
Automatic generation of compilation videos
WO2015127385A1
