Description video generation method and related device

Abstract

The invention provides a commentary video generation method and device. According to the specific implementation scheme, the method comprises the steps of obtaining plot fragments of an original video according to a video copywriting of the original video and time information of the video copywriting; obtaining explanation sub-fragments suitable for explanation from the plot fragments according to the video copywriting corresponding to each plot fragment; generating an explanation copywriting and a corresponding reference physical shot according to the explanation sub-fragment; obtaining original phonon fragments corresponding to the explanation copywriting from the plot fragments; generating an explanation video of the plot clip according to the explanation copywriting, the reference physical shot and the original phonon clip; and based on the explanation videos of all the plot segments, combining to obtain an explanation video of the original video. According to the invention, the watching experience of the user speaking the video is improved.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, specifically to the field of artificial intelligence content generation technology, and in particular to a method and device for generating an explanation video, an electronic device, a computer-readable storage medium, and a computer program product. Background Art

[0002] In the field of artificial intelligence content generation, the existing technology uses artificially produced commentary videos, or simply uses text to generate commentary voices, to generate commentary videos for long videos or TV series. Among them, artificially producing plot commentary is time-consuming and labor-intensive. The method of simply using text to generate commentary voices is to generate commentary texts based on methods such as optical character recognition subtitles or automatic voice recognition video sounds, and then use text-to-speech technology to generate commentary voices based on the commentary texts. Finally, the commentary voice is matched with the original video; when the two durations are different, the original video is processed by tail cropping, acceleration, etc. The methods of the existing technology have many defects: they can only be carried out in a straightforward manner according to subtitles or video sounds, and cannot generate adaptive commentary according to changes in the content of the video plot; the matching of commentary voice and original video screen is prone to problems such as mismatching of sound and picture, which seriously affects the viewing experience. Furthermore, it is difficult for the existing technology to accurately align the commentary comments with the corresponding plot like professional film critics, and it is also difficult to take into account the logic and continuity of the plot while explaining. In general, the commentary videos generated by the existing technology make it difficult for users to quickly and efficiently obtain the video plot, which may cause users to spend more time and energy in understanding the video content, and the user's experience of watching the commentary video is greatly reduced. Summary of the invention

[0003] The present disclosure provides a method and apparatus for generating an explanation video, an electronic device, a computer-readable storage medium, and a computer program product.

[0004] According to a first aspect, a method for generating a commentary video is provided, the method comprising:

[0005] A plot segment acquisition step, acquiring the plot segment of the original video according to the video text and time information of the original video;

[0006] A step of obtaining a narration sub-segment, obtaining a narration sub-segment suitable for narration from each plot segment according to the video text corresponding to each plot segment;

[0007] A step of generating a commentary copy, generating a commentary copy and a corresponding reference physical shot according to the commentary sub-segment;

[0008] an original sound sub-segment obtaining step, obtaining an original sound sub-segment corresponding to the narration copy from the plot segment;

[0009] An explanation video generation step is to generate an explanation video of the plot segment according to the explanation text, the reference physical lens, and the original sound sub-segment;

[0010] The step of combining the narration videos is to combine the narration videos of all the plot segments to obtain the narration video of the original video.

[0011] According to a second aspect, a device for generating a commentary video is provided, the device comprising:

[0012] A plot segment acquisition unit, which acquires the plot segment of the original video according to the video text and time information of the original video;

[0013] A narration sub-segment acquisition unit, which acquires a narration sub-segment suitable for narration from each plot segment according to the video text corresponding to each plot segment;

[0014] A commentary copy generating unit, which generates a commentary copy and a corresponding reference physical shot according to the commentary sub-segment;

[0015] An original sound sub-segment acquisition unit, which acquires an original sound sub-segment corresponding to the narration text from the plot segment;

[0016] An explanation video generating unit generates an explanation video of the plot segment according to the explanation text, the reference physical lens, and the original sound sub-segment;

[0017] The narration video combining unit combines the narration videos of all the plot segments to obtain the narration video of the original video.

[0018] According to a third aspect, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method described in any implementation manner of the first aspect.

[0019] According to a fourth aspect, a non-transitory computer-readable storage medium storing computer instructions is provided, where the computer instructions are used to cause a computer to execute the method described in any implementation of the first aspect.

[0020] According to a fifth aspect, a computer program product is provided, comprising a computer program, which implements the method described in any implementation manner of the first aspect when executed by a processor.

[0021] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0023] Figure 1 is a schematic diagram of a commentary video generation scenario provided by an embodiment of the present disclosure;

[0024] Figure 2 is a flow chart of a method for generating an explanation video provided by an embodiment of the present disclosure;

[0025] Figure 3 is a structural diagram of a commentary video generating device provided by an embodiment of the present disclosure;

[0026] Figure 4 The block diagram is a block diagram of an electronic device for implementing the explanation video generating method according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0027] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0028] The present disclosure first explains the terms involved. Such explanations are exemplary and cover some feasible implementation forms. Those skilled in the art should be aware that such explanations are only examples and do not exhaust all possible situations. Under the premise of following the scope and purpose of the present disclosure, those skilled in the art may implement various changes and modifications to it, and such operations do not depart from the scope of protection of the present disclosure.

[0029] The original video is the object processed by this disclosure, covering various types, such as long videos, film and television series, webcast video clips, homemade short video works, professional film and television production materials and other types of videos. It should be noted that although it is called "original video", it does not mean that the video has not been processed at all. It is just that in the method involved in this disclosure, it is the initial processing object and the starting point and basis for a series of subsequent processing processes.

[0030] The explanation video is the result of the processing of this disclosure, which includes various explanation elements for the original video, such as voice narration, text subtitles, special effects logos, animation demonstrations, etc. It should be clear that although it is called "explanatory video", it does not mean that the video content is all explanation, but it can selectively retain part of the original video content according to actual needs, so that the original video and the explanation content can be organically integrated to achieve specific presentation effects and functional purposes, and this integration and processing method are within the scope and technical solution framework defined in this disclosure.

[0031] Plot fragments are the key intermediate data of the present disclosure, and are also the core feature that distinguishes the method of generating commentary voice from simple copywriting in the prior art. The so-called plot fragments are determined by comprehensively considering the multi-dimensional complexity of the plot of the original video. The basis for the division of plot fragments includes but is not limited to the characteristics of each fragment being suitable for commentary or original sound, the attributes of belonging to the main plot or branch plot, the development rhythm of the plot, the emotional tone of the plot, the expression intensity of the plot, the importance of the plot in the overall story structure and other factors. The division of plot fragments can be based on the technology of text semantic extraction and understanding, and divided according to the above-mentioned basis for the division of plot fragments. The division of plot fragments can also be divided based on a pre-trained large language model. The generation of commentary copywriting and commentary videos based on plot fragments can effectively cope with the complexity of the original video content itself, realize the in-depth analysis and interpretation of the plot like professional film critics, and accurately extract the internal logic behind the plot and the coherence and integrity of the plot. When explaining a plot segment, based on the video copy of the plot segment, a suitable explanation sub-segment is extracted from it. The explanation sub-segment can contain a summary of the previous situation and lead to the plot topic; the explanation copy generated based on the explanation sub-segment can quickly let users understand the key plot of the video. Extracting an original sound sub-segment suitable for original sound playback from the plot segment can echo the previous summary and plot topic, so that users can not only quickly understand the video plot, but also get an intuitive experience when watching the original sound segment. The explanation copy generated based on the explanation sub-segment can also trace back to the physical shots of the original video. These physical shots carry time information. When further generating the explanation audio, each sentence of the explanation copy is aligned with the physical shot of the original video, so that the explanation audio can be combined with the picture of the original video in audio and video synchronization. By acquiring the plot segments, the plot segments can be accurately divided according to the video text and its time information of the original video, which lays a good foundation for the subsequent targeted generation of explanations, effectively avoiding the problem that the existing technology can only be straightforward and cannot adapt to the changes in the plot, thereby helping users to understand the video plot more efficiently and reducing the time and energy required for understanding. Secondly, the acquisition of the explanation sub-segment selects the part suitable for explanation from the plot segment, so that the explanation is highlighted, which can better take into account the logic and continuity of the plot, overcome the shortcomings of the existing technology in this regard, and allow users to quickly grasp the core content of the video. Furthermore, the explanation copy and reference physical lens containing the plot topic are generated, and the original sound sub-segment acquisition step obtains the corresponding original sound answer, and then integrates and generates the plot segment explanation video, and finally combines the explanation videos of each plot segment to obtain the original video explanation video. This series of processes can achieve accurate matching of the explanation and the picture, effectively solving the defect of the mismatch of sound and picture in the existing technology that affects the viewing experience, and greatly improves the user's experience of watching the explanation video.

[0032] In the field of artificial intelligence content generation, especially in the field of commentary video generation, the existing technology has significant technical problems. The methods for generating commentary videos in the existing technology are mostly to generate text based on video subtitles or video sounds, and then generate commentary voice based on the text. The methods of the existing technology can only be carried out in a straightforward manner according to the subtitles or video sounds, and cannot generate adaptive commentary according to changes in the content of the video plot; the matching of the commentary voice with the original video screen is prone to problems such as mismatch between sound and picture, which seriously affects the viewing experience. Furthermore, it is difficult for the existing technology to accurately align the commentary comments with the corresponding plot like professional film critics, and it is also difficult to take into account the logic and continuity of the plot while explaining. In general, the commentary videos generated by the existing technology make it difficult for users to quickly and efficiently obtain the video plot, resulting in users spending more time and energy to understand the video content, and the user's experience of watching the commentary video is greatly reduced.

[0033] In view of the above technical problems, the present disclosure proposes a method for generating a commentary video and a related device.

[0034] It is understandable that the method can be integrated into various computing devices, including but not limited to terminal devices with text processing capabilities (such as smart phones, tablets, personal computers, etc.) and servers (such as application servers, Web servers, whether single servers or cluster-deployed server systems). The method does not rely on a specific hardware platform or software architecture. During the execution of the method, both independently running terminal devices and terminal-server architectures that work collaboratively through a network can effectively utilize the method disclosed in the present invention. When the terminal device is executed independently, the method can be independent of the external network. In scenarios where higher processing performance or wider data resource support is required, the method of the present invention also supports communication between the terminal device and the server, utilizing the powerful computing power and rich data resources of the server to jointly complete the method. The method can adapt to different operating systems and platform environments, including mobile operating systems such as iOS, Android, and Hongmeng, desktop operating systems such as Windows and macOS, and server operating systems such as Linux and Unix.

[0035] It can be understood that the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved in the technical solution of the present disclosure are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0036] Embodiment 1

[0037] For ease of understanding, Figure 1 A content generation scenario according to an embodiment of the present disclosure is shown.

[0038] like Figure 1As shown, in the step of letter "A", the original video is divided into two plot segments.

[0039] Next, in step "B", for the first plot segment, the explanation sub-segment numbered 1 and the explanation sub-segment numbered 2 suitable for explanation are obtained. For the explanation sub-segment numbered 1 and the explanation sub-segment numbered 2, a summary of the previous story and an explanation copy for introducing the plot topic are generated respectively. Afterwards, based on the proposed plot topic, the original sound sub-segment numbered 3 is further obtained from the first plot segment, and the original sound sub-segment can just provide an original sound answer to the corresponding plot topic.

[0040] In the step for letter "B", for the second plot segment, the commentary segment numbered 4 was selected. The commentary copy generated by the commentary segment numbered 4 does not contain the previous plot summary content, but it also contains another plot topic. Around this plot topic, the original sound sub-segment numbered 5 was obtained from the second plot segment, and this original sound sub-segment can well answer the plot topic in the commentary copy corresponding to the commentary sub-segment numbered 4.

[0041] Then we proceed to the step represented by the letter "C". For the first plot, we first splice and integrate the physical shots corresponding to the commentary texts of the commentary sub-segments numbered 1 and 2, and then add the original sound sub-segment numbered 3. In this way, we have formed the first commentary video for the first plot, which contains both commentary content and original sound.

[0042] Similarly, in step "C", for the second plot, we first look for a physical shot that matches the commentary text of the commentary sub-segment numbered 4, and then splice this physical shot with the original sound sub-segment numbered 5, thereby generating a second commentary video for explaining the second plot.

[0043] Finally, in step numbered “D”, the first explanation video and the second explanation video generated for the two plots are merged together to obtain the explanation video corresponding to the entire original video.

[0044] The multiple embodiments of the present disclosure are not isolated individuals from each other, but rather present mutually complementary characteristics. The schemes involved in each embodiment are intended to be exemplified from a specific aspect or dimension for the overall scheme. The contents of each embodiment in the present disclosure can be integrated with other embodiments to constitute one of the multiple feasible schemes of the present disclosure. Such a design enables the present disclosure to form an organic whole, which together provides comprehensive and systematic technical support for achieving the purpose of the invention, so as to meet the needs of different application scenarios, and ensure the integrity and effectiveness of the technical scheme of the present disclosure in multiple dimensions and levels.

[0045] Embodiment 2

[0046] Figure 2 The flowchart 100 of the method for generating a narration video according to an embodiment of the present disclosure is shown. The narration video generating method comprises the following steps:

[0047] The plot segment acquisition step 101 is to acquire the plot segment of the original video according to the video text and time information of the original video;

[0048] By integrating the video text and its time information of the original video, it helps to subsequently interpret the video plot in detail, so that the processing system can carry out targeted operations such as narration sub-segment selection and text generation based on these plot fragments, thereby improving the accuracy and efficiency of the video commentary and allowing the video commentary to better fit the narrative rhythm and story structure of the video itself.

[0049] A step 102 of obtaining a narration sub-segment, obtaining a narration sub-segment suitable for narration from each plot segment according to the video text corresponding to each plot segment;

[0050] By obtaining the appropriate sub-segments for explanation based on the video copy corresponding to the plot segment, we can accurately locate the part of the video that is most needed and most suitable for adding explanations to assist the audience in understanding. This effectively avoids redundancy and mismatch in explanations, and makes the explanation content closely focus on the main line and key plots.

[0051] A commentary copy generating step 103, generating a commentary copy and a corresponding reference physical shot according to the commentary sub-segment;

[0052] Generate commentary copy containing plot topics and corresponding reference physical lenses based on commentary sub-segments to improve the matching degree between commentary copy and video images. The introduction of plot topics gives the commentary copy a clear main line and core points, which can guide the audience to quickly understand the key plot of the video. The reference physical lenses provide intuitive picture references for the commentary copy, ensuring the accuracy and vividness of the copy creation. This helps to create high-quality commentary content, so that the audience can better understand the picture and commentary information synchronously when watching the video.

[0053] The original sound sub-segment acquisition step 104 is to acquire the original sound sub-segment corresponding to the narration text from the plot segment;

[0054] The original sub-segments corresponding to the plot topics are obtained from the plot clips for original answers, which effectively preserves the original flavor and authenticity of the video. The original sub-segments are inserted at appropriate times during the commentary process. This organic combination of original sound and commentary enriches the auditory experience level of the video and avoids the monotony of the commentary. At the same time, the original sound is used to supplement the plot topics, further improving the accuracy and completeness of the audience's understanding of the video plot.

[0055] An explanation video generation step 105 is to generate an explanation video of the plot segment according to the explanation text, the reference physical lens, and the original sound sub-segment;

[0056] The narration video of the plot segment is generated based on the narration copy, reference physical shots and original sound sub-segments, achieving a high degree of integration of multiple elements. This integration method ensures the integrity and richness of the information conveyed by the narration video, allowing the audience to clearly understand the core points and picture content of the video plot through the narration copy and reference physical shots, and feel the original atmosphere of the video with the help of the original sound sub-segments.

[0057] The narration video combination step 106 is to combine the narration videos of all the plot segments to obtain the narration video of the original video.

[0058] The commentary video of the original video is obtained by combining the commentary videos of all the plot segments, which ensures the coherence and systematicness of the commentary of the entire original video. Through reasonable combination and splicing, the commentary videos of each plot segment can be seamlessly connected to form a complete video commentary work.

[0059] Embodiment 3

[0060] This embodiment explains in detail how to obtain the video text.

[0061] Before executing the plot segment acquisition step 101 of the present disclosure, the video text must first be acquired.

[0062] The video copy is obtained in advance and executed before the plot segment acquisition step 101. The acquisition of the video copy can be stored after the acquisition in advance to establish an association with the original video. In this way, when executing the disclosed method, the pre-acquired video copy can be directly read. It is also possible to execute the method of obtaining the video copy each time when executing the disclosed method.

[0063] In this embodiment, the video text is obtained based on the subtitle information and / or voice information of the original video, and the time information of the video text is obtained according to the time information of the subtitle information and / or voice information.

[0064] The following is an example of how to obtain video text. Assume that the original video has a subtitle file. First, the text content in the subtitle file can be read through a special subtitle parsing program, and these subtitle texts become the source of the video text. For example, a video text in the subtitle file records its start time as 0:10:10 (indicates that it appears 10 seconds after the video starts playing) and its end time is 0:10:15. It can be considered that the time information of the video text is 0:10:10 to 0:10:15.

[0065] In addition, subtitles of the original video can also be obtained based on image recognition technology.

[0066] At the same time, voice recognition technology can also be used to extract text corresponding to voice information in the video. For example, when a video plays to a certain episode, the voice recognition software converts this voice into text. These texts are also the source of video copy. The time information of the voice is determined according to its time position in the video. Assume that this voice starts at 0:20:05 and ends at 0:20:12. Then it can be considered that the time information of the video copy is 0:20:05 to 0:20:12.

[0067] After the video text is obtained, it can be used to execute other steps of the present disclosure.

[0068] The plot segment acquisition step 101 is to acquire the plot segment of the original video according to the video text and time information of the original video;

[0069] Integrate the video text and its time information obtained from subtitles and / or voice. Divide the plot segments according to factors such as the semantics of the video text, the development rhythm of the plot, and the continuity of time. For example, taking the plot corresponding to the subtitles and voice mentioned above as an example, the text content from 0:10:00 to 0:20:12 is regarded as a plot segment because they all revolve around the same plot.

[0070] A step 102 of obtaining a narration sub-segment, obtaining a narration sub-segment suitable for narration from each plot segment according to the video text corresponding to each plot segment;

[0071] A commentary copy generating step 103, generating a commentary copy and a corresponding reference physical shot according to the commentary sub-segment;

[0072] The video copy is used to generate the commentary copy in this step. It is found that the video copy of a sub-segment of the plot segment has a high amount of information, which is the key content that the audience may need further explanation to understand. Therefore, the video segment containing these key contents and the related video segments before and after are determined as the commentary sub-segment suitable for explanation.

[0073] The original sound sub-segment acquisition step 104 is to acquire the original sound sub-segment corresponding to the narration text from the plot segment;

[0074] An explanation video generation step 105 is to generate an explanation video of the plot segment according to the explanation text, the reference physical lens, and the original sound sub-segment;

[0075] The narration video combination step 106 is to combine the narration videos of all the plot segments to obtain the narration video of the original video.

[0076] This embodiment integrates subtitles and voice information to obtain video text and time information, accurately divides plot segments, and avoids plot comprehension deviations. It extracts sub-segments suitable for interpretation and creates interpretation text, focusing on key content to meet the understanding needs of different audiences. It introduces original sub-segments, combines interpretation text with reference physical shots, enhances audience immersion and sense of reality, and allows audiences to fully integrate into the video context. Its process is flexible and scalable, suitable for all types of videos, and each step is interrelated and relatively independent, which is convenient for optimization and customized development. It is efficient and innovative, and has strongly promoted the development of the video processing field.

[0077] Embodiment 4

[0078] This embodiment explains in detail another method of obtaining video text.

[0079] Before executing the plot segment acquisition step 101 of the present disclosure, the video text must first be acquired.

[0080] In this embodiment, the physical lens of the original video is obtained; the lens text corresponding to the physical lens and its time information are obtained based on the subtitle information and / or voice information of the physical lens; and all the lens texts are combined to obtain the video text.

[0081] First, obtain the physical lens of the original video. For each physical lens, parse its subtitle information and / or voice information, for example, read the text content corresponding to this lens period in the subtitle file, and use voice recognition technology to process the voice information in the lens and convert it into text. Taking a physical lens as an example, the front and back division points of the lens are 0:05:00 to 0:05:10 respectively. During this period of time, a subtitle text is presented, which belongs to the video copy of the physical lens, and each sentence of the video copy and its time information can also be obtained according to the methods in other embodiments. The lens copy corresponding to all physical lenses is combined in sequence to obtain a complete video copy.

[0082] Then, the plot segments are divided according to the video text and its time information, and then the explanation sub-segments are obtained according to the established steps, the explanation text and reference physical shots are generated, and the original sound sub-segments are obtained, and finally the explanation video of the original video is generated and combined.

[0083] After the video text is obtained, it can be used to execute other steps of the present disclosure.

[0084] The plot segment acquisition step 101 is to acquire the plot segment of the original video according to the video text and time information of the original video;

[0085] A step 102 of obtaining a narration sub-segment, obtaining a narration sub-segment suitable for narration from each plot segment according to the video text corresponding to each plot segment;

[0086] A commentary copy generating step 103, generating a commentary copy and a corresponding reference physical shot according to the commentary sub-segment;

[0087] The original sound sub-segment acquisition step 104 is to acquire the original sound sub-segment corresponding to the narration text from the plot segment;

[0088] An explanation video generation step 105 is to generate an explanation video of the plot segment according to the explanation text, the reference physical lens, and the original sound sub-segment;

[0089] The narration video combination step 106 is to combine the narration videos of all the plot segments to obtain the narration video of the original video.

[0090] This embodiment obtains video text by parsing the subtitles and voice information of physical shots, so that the text acquisition is more in line with the actual content of the video screen. Accurately dividing physical shots helps to accurately divide plot segments and helps to deeply analyze the video structure. After obtaining physical shots, generating commentary text containing plot topics can directly correspond to physical shots, enhancing the pertinence and guidance of the commentary.

[0091] Embodiment 5

[0092] This embodiment explains in detail how to obtain the plot segments.

[0093] When processing an original video, the first step is to enter the scene segment acquisition step 101.

[0094] In this embodiment, based on the pre-trained first language model, the plot segments of the original video are obtained from the video text of the original video; based on the time information of the video text covered by the plot segments, the boundary points of the plot segments are obtained.

[0095] The acquisition of plot segments is based on the video text of the original video and its corresponding time information. Here, we use the pre-trained first language model to deeply analyze the video text. The video text is input into the large language model. It will use its powerful semantic understanding and logical analysis capabilities to identify key plot turning points, theme transition points, etc. in the text, so as to accurately determine the plot segments of the original video. At the same time, based on the time information of the video text covered by these plot segments, such as the corresponding start time and end time of a certain plot in the video text, the boundary points of the plot segments are accurately obtained.

[0096] Then, according to the established steps, the commentary sub-segment is obtained, the commentary text and reference physical shots are generated, the original sound sub-segment is obtained, and finally the commentary video of the original video is generated and combined.

[0097] A step 102 of obtaining a narration sub-segment, obtaining a narration sub-segment suitable for narration from each plot segment according to the video text corresponding to each plot segment;

[0098] A commentary copy generating step 103, generating a commentary copy and a corresponding reference physical shot according to the commentary sub-segment;

[0099] The original sound sub-segment acquisition step 104 is to acquire the original sound sub-segment corresponding to the narration text from the plot segment;

[0100] An explanation video generation step 105 is to generate an explanation video of the plot segment according to the explanation text, the reference physical lens, and the original sound sub-segment;

[0101] The narration video combination step 106 is to combine the narration videos of all the plot segments to obtain the narration video of the original video.

[0102] Obtaining plot fragments with the help of the pre-trained first language model has many significant technical effects. First, the powerful semantic understanding ability of the large language model can deeply analyze the video copy, accurately locate key plot turning points and theme transition points, effectively overcome the shortcomings of traditional methods in complex semantic analysis, and make the division of plot fragments more accurate and reasonable. Secondly, based on this precise division, it is more efficient to determine the commentary sub-segments closely related to each plot fragment, and improve the pertinence of subsequent commentary creation. Furthermore, the generated commentary copy is more coherent and logical due to the accuracy of the determination of the plot fragments. With the corresponding reference physical lenses and original sound sub-segments, the perfect fusion of multiple elements can be achieved when the commentary video is generated. This not only allows the commentary video to better restore the core content and emotional context of the original video, but also meets the audience's demand for high-quality video interpretation.

[0103] Embodiment 6

[0104] This embodiment explains in detail the correction of boundary points of plot segments.

[0105] After executing the plot segment acquisition step 101 of the present disclosure, the boundary points of the plot segments may be corrected.

[0106] Acquire a physical lens of the original video; and modify the boundary points of the plot segment based on the physical lens.

[0107] In this embodiment, a physical shot of the original video is obtained; in response to a segmentation point of the physical shot being within a first preset threshold range of a boundary point of the plot segment, the shot segmentation point is used as a boundary point of the plot segment; the first preset threshold range is pre-set according to the type of the original video.

[0108] Before the above steps, first obtain the physical lens of the original video. For each physical lens, by parsing its subtitle information and / or voice information, for example, reading the text content corresponding to this lens period in the subtitle file, and using voice recognition technology to process the voice information in the lens and convert it into text. Taking a physical lens as an example, the front and back division points of the lens are 0:09:55 to 0:10:07 respectively. During this period of time, a subtitle text is presented, which belongs to the video copy of the physical lens, and each sentence of the video copy and its time information can also be obtained according to the method in other embodiments. The lens copy corresponding to all physical lenses is combined in sequence to obtain a complete video copy.

[0109] The physical shot includes a shot division point; in response to the shot division point being within a first preset threshold range of a boundary point of the plot segment, the shot division point is used as the boundary point of the plot segment; the first preset threshold range is pre-set according to the type of the original video.

[0110] A plot segment revolves around the same plot. The plot segment has boundary points. For example, taking the plot corresponding to the subtitles and voice mentioned above as an example, the text content between 0:10:00 and 0:20:12 is regarded as a plot segment. Near the starting boundary point 0:10:00 of the plot segment, there is a physical shot, and its front and back segmentation points are 0:09:55 to 0:10:07 respectively. The distance between the segmentation point of the physical shot and the boundary point of the plot segment is 5 seconds and 7 seconds respectively. Assuming that the predefined first preset threshold is 10 seconds, the plot boundary point can be corrected according to the segmentation point 0:09:55 of the physical shot. The corrected plot boundary points are 0:09:55 to 0:20:12.

[0111] When a physical shot is between two plot segments, the first thing to consider is the relevance of the video copy content of the physical shot to the plots around the two plot segments. For example, if the split point time of a physical shot is 0:30:00-0:30:15, the two adjacent plot segments A (0:19:50-0:30:05) and B (0:30:10-0:51:00) are located before and after it. By performing semantic analysis on the text converted from the subtitle information and voice information in the shot, if the copy content is more closely connected with the plot around plot segment B in terms of semantic logic, such as mentioning the foreshadowing information of the key characters or events in the subsequent development of plot segment B, then this physical shot should be attributed to plot segment B. On the contrary, if the copy has a more direct continuity with the core plot of plot segment A in terms of character relationships, event trends, etc., it belongs to plot segment A.

[0112] At the same time, the matching degree between the visual elements of the shot and the plot segments can also be comprehensively considered. For example, if the visual features such as the scene layout and character costumes in the physical shot are more consistent with the typical scene and character image corresponding to plot segment B, even if it is close to plot segment A in time, it tends to be classified into plot segment B. In this way, through the comprehensive judgment of the semantics of the video copy and the visual elements of the shot, the ownership of the physical shot between the two plot segments can be determined more accurately, thereby further optimizing the division of the plot segments and providing a more reliable basis for subsequent operations such as the generation of commentary videos based on plot segments.

[0113] By correcting the boundary points of the plot segments based on the physical lens of the original video, the accuracy of determining the boundaries of the plot segments can be improved. According to the relative position relationship between the lens segmentation point and the boundary points of the plot segments, correction is performed when the lens segmentation point is within the first preset threshold range of the boundary points of the plot segments, so that the plot segment division is more in line with the actual content of the video. When the physical lens is between two plot segments, the relevance of the video copy content of the lens and the plot surrounding the two plot segments, as well as the matching degree between the visual elements of the lens screen and the plot segments, can more accurately determine the ownership of the physical lens, optimize the division of the plot segments, and provide a more reliable basis for operations such as the generation of commentary videos based on plot segments, which helps to improve the quality and efficiency of related video processing work, so that the division of plot segments is more in line with the internal logic and semantic structure of the video, so as to better serve subsequent video analysis, editing and other tasks.

[0114] Embodiment 7

[0115] This embodiment limits the acquisition of explanation sub-segments suitable for explanation from plot segments.

[0116] After executing the plot segment acquisition step 101 of the present disclosure, the narration sub-segment acquisition step 102 is executed to identify sub-segments suitable for narration from the plot segment, which are called narration sub-segments.

[0117] In this embodiment, based on the pre-trained second largest language model, a narration sub-segment suitable for narration is obtained from the plot segment.

[0118] Before performing this step, you need to obtain a pre-trained second language model, which has strong capabilities in natural language understanding and text analysis and can perform in-depth processing of video-related text information.

[0119] For a given original video, we obtain its plot segments according to the method mentioned above. For example, there is an original video of a movie, and after the plot segments are divided, multiple plot segments are obtained. One plot segment covers the content from 0:20:00 to 0:35:00 of the movie timeline. This plot segment contains a series of character dialogues, action scenes, and plot development.

[0120] The video text of this episode (such as the complete text obtained by subtitle extraction and speech-to-text conversion) is input into the pre-trained second largest language model. The model analyzes the text in the episode based on the language patterns, semantic understanding, and understanding of the features of the commentary text learned during its training. It evaluates the importance, appeal, coherence, and adaptability of each part of the text to the common commentary style.

[0121] For example, in this episode, there is a section from 0:25:00 to 0:28:00, which contains a wonderful dialogue between the protagonist and the villain and a key plot twist. Through semantic analysis of the text, the model recognizes that this section contains rich emotional expressions, character portrayals, and information that plays an important role in promoting the subsequent plot development. Moreover, from the perspective of language fluency and vividness, this section of content is more suitable for commentary, which can effectively attract the audience's attention and help them better understand the video plot.

[0122] Therefore, the model determines the video portion corresponding to the text content from 0:25:00 to 0:28:00 as a sub-segment suitable for explanation. In this way, in the subsequent process of generating an explanation video, it is possible to focus on this sub-segment to create a detailed and attractive explanation, thereby improving the quality and viewing experience of the entire explanation video, allowing the audience to more accurately grasp the key plots and exciting content in the video, and enhancing the audience's understanding and experience of the video.

[0123] After the explanation sub-segment acquisition step 102, the subsequent processing steps are continued.

[0124] A commentary copy generating step 103, generating a commentary copy and a corresponding reference physical shot according to the commentary sub-segment;

[0125] The original sound sub-segment acquisition step 104 is to acquire the original sound sub-segment corresponding to the narration text from the plot segment;

[0126] An explanation video generation step 105 is to generate an explanation video of the plot segment according to the explanation text, the reference physical lens, and the original sound sub-segment;

[0127] The narration video combination step 106 is to combine the narration videos of all the plot segments to obtain the narration video of the original video.

[0128] Using the pre-trained second largest language model to obtain the sub-segments of the explanation can accurately locate the most suitable part of the plot segment for explanation. With its powerful natural language understanding and text analysis capabilities, the model comprehensively considers factors such as the importance, attractiveness, coherence and compatibility with the explanation style of the text, making the selected sub-segments of the explanation more valuable and laying the foundation for subsequent high-quality explanation creation. The generation of explanation copy and explanation video based on plot segments can effectively cope with the complexity of the original video content itself, achieve in-depth analysis and explanation of the plot like professional film critics, and accurately extract the internal logic behind the plot and the coherence and integrity of the plot.

[0129] Embodiment 8

[0130] This embodiment limits the acquisition of explanation sub-segments suitable for explanation from plot segments.

[0131] After executing the plot segment acquisition step 101 of the present disclosure, the narration sub-segment acquisition step 102 is executed to identify sub-segments suitable for narration from the plot segment, which are called narration sub-segments.

[0132] In this embodiment, based on the text features of the video text covered by the plot segment, a commentary sub-segment suitable for commentary is obtained from the plot segment; the text features include at least one of the following: the amount of dialogue information in the text, the main plot attributes of the text, the plot development rhythm of the text, the emotional tone of the text, the plot expression intensity of the text, and the importance of the text.

[0133] The division of plot segments can be based on the technology of text semantic extraction and understanding, and according to the basis for the division of plot segments. The basis for the division of plot segments includes but is not limited to the amount of dialogue information in the video copy, the attributes of the main plot or the side plot, the development rhythm of the plot, the emotional tone of the plot, the intensity of the plot, the importance of the plot in the overall story structure, and other factors.

[0134] When explaining a plot segment, a suitable explanation sub-segment is extracted from the video text of the plot segment. The explanation sub-segment may contain a summary of the previous story and may also introduce plot topics. The explanation text generated based on the explanation sub-segment can quickly enable users to understand the key plot of the video.

[0135] For example, when considering the characteristics of each segment suitable for explanation or original sound, if a video copy contains lines with a lot of information, such as detailed dialogues, inner monologues of characters, etc., it means that there are complex plot twists that need to be explained. This kind of information-rich content that needs further explanation to assist the audience's understanding is often more suitable as the source of the explanation sub-segment. If the copy is mostly concise and powerful dialogue, with strong emotional impact and does not require too much additional explanation, it may be more suitable for original sound presentation.

[0136] From the perspective of whether it belongs to the main plot or the side plot, the main plot is usually the core thread that drives the development of the entire story. Among them, key plot advancement nodes, key actions of the main characters, and important turning points in the development of the story are often the areas of focus for the selection of sub-segments for explanation. Because these parts play a vital role in the audience's understanding of the entire story structure. The unique highlights of the side plots and the parts that have potential connections or contrasts with the main plot may also be selected as sub-segments for explanation to enrich the layering and diversity of the story.

[0137] Regarding the development rhythm of the plot, in the fast-paced plot segments where events occur one after another, those text paragraphs that can serve as a link between the previous and the next, summarize key events, or reveal the core conflict will be given priority as the commentary sub-segments. For example, in a fierce battle plot, the initial description of the strategic layout of the two sides of the battle and the part where the battle results have a significant impact on the subsequent plot can be used as the commentary sub-segment. In the plot with a slower rhythm, such as the emotional exchanges between characters and the rendering of the environmental atmosphere, the introductory copy that can deeply reflect the delicate expression of emotional changes or unique environmental elements may become the commentary sub-segment.

[0138] In terms of the emotional tone of the plot, whether it is exciting, sad, or light-hearted and humorous, the parts with the strongest emotional expression and the most resonant emotions of the audience, such as the grief-stricken inner monologue of the character in a sad plot, or the inspiring call to action in an exciting plot, are very likely to be identified as commentary sub-segments, so that the audience can be better guided to feel the emotional power in the video during the commentary.

[0139] Regarding the intensity of plot expression, if the plot corresponding to the video copy has a very impactful scene, such as a grand natural landscape display, a spectacular special effects scene presentation, or a unique artistic style visual expression, then the copy part that describes, explains, and emphasizes these visual elements can be used as a sub-segment of the explanation. In this way, in the explanation video, the audience can better understand and feel it in a coordinated manner in terms of hearing and vision.

[0140] Finally, considering the importance of the plot in the overall story structure, the key texts in the plot segments that have a decisive influence on the direction of the story, reveal the core theme of the story, or reflect the key links of the character growth of the main characters are undoubtedly the focus of the sub-segments for commentary. By explaining these parts, the audience can clearly grasp the core connotation and development context of the whole story, thereby improving their understanding and appreciation of the entire video content.

[0141] After the explanation sub-segment acquisition step 102, the subsequent processing steps are continued, which will not be described in detail in this embodiment.

[0142] The basis for dividing the above explanation sub-segments can also be used to train the pre-trained second language model in the aforementioned embodiment.

[0143] Based on the text semantic extraction and understanding technology, the plot segment division is carried out, and multiple aspects are integrated to make the division of plot segments more scientific and reasonable. Considering the characteristics of suitable commentary or original sound can better balance the parts presented by the commentary and the original sound, and improve the overall audio-visual effect; distinguishing the main and branch plot attributes is conducive to the audience to clearly grasp the main and secondary story context; combining the division of the rhythm of the plot development can make the commentary fit the rhythm changes and enhance the fluency of the audience's understanding; referring to the emotional tone division, it is easy to highlight the emotional expression and arouse the emotional resonance of the audience; according to the division of the intensity of expression, it is helpful to better show the unique highlights of the video in the commentary; considering the importance of the overall story structure, it can ensure that the key plots are explained in detail, so that the audience can deeply understand the core connotation of the story, and comprehensively improve the audience's understanding and appreciation of the video content.

[0144] Embodiment 9

[0145] This embodiment describes in detail how to obtain the explanation text.

[0146] After executing the plot segment acquisition step 101 and the narration sub-segment acquisition step 102 of the present disclosure, the narration text generation step 103 is executed.

[0147] In this embodiment, based on the pre-trained third language model, a commentary copy is generated according to the video copy of the commentary sub-clip, and the source video copy corresponding to the commentary copy is given; according to the time information of the source video copy, the physical shot corresponding to the commentary copy is matched and obtained as a reference physical shot.

[0148] Before performing this step, it is necessary to obtain a pre-trained third language model, which has strong capabilities in natural language understanding and text analysis and can deeply process the video copy of the commentary sub-segment suitable for commentary to obtain the commentary copy.

[0149] The pre-trained third language model has excellent semantic understanding and copywriting generation capabilities. For example, for a specific commentary sub-segment, it comes from a video showing the Battle of Red Cliffs, and its video copy is a copy of the Battle of Red Cliffs.

[0150] The video text is input into the third pre-trained language model. The model first performs a deep semantic analysis of the text to accurately grasp the key information. Based on this understanding, the model begins to generate commentary: "In 208 AD, after Cao Cao unified the north, he wanted to march south to unify the world. The Yangtze River was full of storms. Facing Cao's 300,000 soldiers, the Sun-Liu coalition had only 50,000 soldiers. So how did the Sun-Liu coalition defeat Cao's army?"

[0151] At the same time, the model clearly gives the source video copy corresponding to each commentary copy. For example, for the first sentence of the commentary copy "In 208 AD, after Cao Cao unified the north, he wanted to march south to unify the world, and the Yangtze River was full of storms", the corresponding video copy is a video subtitle from 0:10:10 to 0:10:50 on the video timeline: "After Cao Cao defeated Yuan Shao in the Battle of Guandu, his power rose rapidly. After Liu Biao of Jingzhou died of illness, Cao Cao took the opportunity to march south, and Liu Cong of Jingzhou surrendered without a fight. Cao Cao received a large number of soldiers, money, food and warships from Jingzhou. In 208 AD, Cao's army advanced along the Yangtze River, pointing directly to Jiangdong, and the curtain of the Battle of Chibi was slowly opened." Since the commentary copy generated by the large model is a summary and abstraction of the video copy of the original video, the commentary copy and the corresponding video copy are not necessarily the same.

[0152] Furthermore, according to the video copy from 0:10:10 to 0:10:50, the physical lens in this period can be matched, which presents the pre-war scene on both sides of the Yangtze River. This matched physical lens is used as a reference physical lens, which will be combined with the generated commentary copy. When the commentary video is generated later, the commentary and the video screen can complement each other, vividly showing the audience the tense and suspenseful historical background on the eve of the Battle of Red Cliffs.

[0153] In this embodiment, the explanatory text includes a summary of the previous situation.

[0154] The above commentary, "In 208 AD, after unifying the north, Cao Cao wanted to march south to unify the world, and the Yangtze River brought a stormy gathering" can be considered a summary of the previous events. The following commentary, "Facing Cao's 300,000 strong soldiers, the Sun-Liu coalition had only 50,000 soldiers, so how did the Sun-Liu coalition defeat Cao's army?" can be considered a plot question.

[0155] The model gives the following commentary, "Facing Cao's 300,000 soldiers, the Sun-Liu coalition only had 50,000 soldiers. So how did the Sun-Liu coalition defeat Cao's army?", and also gives the corresponding source video text. The video text is a subtitle from 0:11:30 to 0:12:25 on the video timeline: Cao Cao personally led an army of 300,000, claiming to be a million-strong army, and marched south to Jiangdong. The Sun-Liu coalition had a much larger force than them, only 50,000, and Sun Quan's subordinates had different opinions, some advocating peace, some advocating war. However, at this critical moment, they will eventually unite to resist the powerful enemy and win the Battle of Chibi.

[0156] The video text from 0:11:30 to 0:12:25 is a statement, which introduces the huge difference in the strength of Cao's army and the coalition forces of Sun and Liu, but in the end the coalition forces of Sun and Liu will win. The commentary generated based on the video text is a question. At this time, the commentary raises questions about the plot. Based on the ability of the third language model, interrogative content can be generated from declarative corpus.

[0157] Furthermore, according to the video text from 0:11:30 to 0:12:25, two physical shots can be matched during this period. The first physical shot presents the shot of Cao’s military camp, and the second physical shot presents the shot of the Sun-Liu coalition military camp.

[0158] These two physical shots cover the panoramic view of the two sides of the Battle of Red Cliffs busy preparing for battle in the riverside camp, the spectacular mid-shot of the warships connected and stretched out, and the scene of the coalition forces of Wu and Liu Bei on the south bank. The matched physical shots are used as reference physical shots, which will be combined with the generated commentary copy, so that the commentary and video images can complement each other when the commentary video is generated later.

[0159] It is worth noting that when generating the commentary copy, the preliminary summary and topic questions in the commentary copy are both a type of plot topic, and the commentary copy can also contain other types of plot topics. The commentary copy can contain one or more plot topics at the same time.

[0160] After the explanation sub-segment acquisition step 103, the subsequent processing steps are continued, which will not be described in detail in this embodiment.

[0161] With the help of the pre-trained third language model, the video copy of the commentary sub-segment can be efficiently analyzed and processed in depth, thereby generating a highly generalized and logical commentary copy. The model accurately extracts key information, so that the generated commentary copy can not only accurately convey the core content of the video, but also attract the audience with concise, clear and contagious language. By clearly giving the source video copy corresponding to the commentary copy, the traceability and accuracy of the source of the commentary content are guaranteed. When generating a commentary video, the audience can hear the commentary while seeing the video screen that is highly consistent with it.

[0162] Embodiment 10

[0163] This embodiment explains in detail how to obtain the original sound sub-segment.

[0164] After executing steps 101 and 102 of the present disclosure, executing step 103 of generating explanation text;

[0165] In this embodiment, in the commentary copy generating step 103, the commentary copy includes a plot topic; the original sound sub-segment obtaining step 104 includes: obtaining an original sound sub-segment corresponding to the plot topic from the plot segment.

[0166] The commentary should form a coherent relationship with the original sub-segment that follows. One way to form a coherent relationship is for the commentary to raise a plot topic, and the original sub-segment that follows to answer the plot topic. For example, the previously generated commentary, "Facing Cao's 300,000 soldiers, the Sun-Liu coalition only had 50,000 soldiers, so how did the Sun-Liu coalition defeat Cao's army?" raises a plot topic in the form of a question.

[0167] After executing step 103 of the present disclosure, the original sound sub-segment acquisition step 104 is executed.

[0168] In this embodiment, based on the pre-trained fourth language model, a sub-video text is obtained from the plot segment according to the commentary text; the sub-video text is matched with the commentary text; and based on the time information of the sub-video text, the original sound sub-segment corresponding to the sub-video text is determined.

[0169] Before performing this step, you need to obtain a pre-trained fourth language model, which has strong capabilities in natural language understanding and text analysis, and can deeply process the video text of the plot segment to obtain the original sub-segment that matches the commentary text.

[0170] According to the pre-trained fourth language model, the sub-video text corresponding to the plot topic is obtained from the video text of the plot segment. For example, the sub-video text from 0:30:05 to 0:30:50 is "The Sun-Liu coalition used a counter-espionage strategy. Huang Gai led dozens of warships filled with flammable materials and pretended to surrender to Cao's army. When they approached Cao's water stronghold, the southeast wind happened to rise, so they set fire and rushed towards Cao's army. Cao's warships caught fire one after another, and Cao's army was in chaos. The Sun-Liu coalition took advantage of the situation and attacked, thus winning the Battle of Chibi." This sub-video text is highly matched with the plot topic in the commentary text "Facing Cao's 300,000 soldiers, the Sun-Liu coalition had only 50,000 soldiers, so how did the Sun-Liu coalition defeat Cao's army?" Therefore, this sub-video text can answer the plot topic in the aforementioned commentary text.

[0171] Further, according to the sub-video copy of 0:30:05-0:30:50, the original sound sub-segment 0:30:05-0:30:50 is obtained from the plot segment. The video portion corresponding to this time period is determined to be the original sound sub-segment. When generating the commentary video, this original sound sub-segment can be inserted into the corresponding commentary copy part, vividly showing the key plot of the Sun-Liu coalition's victory over Cao's army in the form of original sound and original video, allowing the audience to more intuitively feel the excitement and intensity of the Battle of Chibi, and enhancing the appeal and credibility of the commentary video.

[0172] After the explanation sub-segment acquisition step 104, the subsequent processing steps are continued, which will not be described in detail in this embodiment.

[0173] With the powerful natural language understanding and text analysis capabilities of the pre-trained fourth language model, it is possible to accurately mine sub-video texts that match the plot topics of the commentary text from the video text of the plot clips. This precise matching ensures that the selected content is highly consistent with the core of the commentary, avoids interference from irrelevant information, and makes the content logic of the entire commentary video more rigorous and coherent.

[0174] Embodiment 11

[0175] This embodiment explains in detail how to obtain the explanation video.

[0176] After executing steps 101 , 102 , 103 , and 104 of the present disclosure, an explanation video generation step 105 is executed.

[0177] In this embodiment, the commentary voice generated by the commentary copy and the reference physical lens are merged according to the timeline order and semantic logic; the original sound sub-segment is connected immediately after the merged commentary voice and the reference physical lens to obtain a commentary video.

[0178] According to the above embodiment, it is assumed that the generated explanation copy is as follows: "In 208 AD, after unifying the north, Cao Cao wanted to march south to unify the world, and the Yangtze River was full of storms. Facing Cao's 300,000 soldiers, the Sun-Liu coalition had only 50,000 soldiers. So how did the Sun-Liu coalition defeat Cao's army?"

[0179] The corresponding time axis of the reference physical lens is 0:10:10-0:10:50 (corresponding to the first sentence of the commentary) and 0:11:30-0:12:25 (corresponding to the second sentence of the commentary).

[0180] The above commentary copy is first used to generate commentary voice. The commentary voice is accurately merged with the above two physical shots according to the timeline.

[0181] According to the order of description in the commentary, the corresponding physical shots are arranged on the timeline in sequence, and the commentary voice is also embedded in the corresponding time track to ensure that when a certain plot is mentioned, the corresponding shot images can be displayed synchronously to achieve close coordination between the two.

[0182] If the commentary voice describes a certain plot for a long time, but the corresponding physical shot is short, you can use slow motion, freeze-frame and close-up to show key elements, or loop the shot to extend the visual display time of the physical shot so that it can better match the commentary voice. For example, when the commentary voice describes in detail the spectacular scene of Cao Cao's warships marching in formation on the Yangtze River, the physical shot that originally showed the warships in formation is short, so you can slow down the picture speed of the warships, or when the overall picture of the warships is frozen, give close-ups of key elements such as the warships' flags and soldiers, so as to extend the display time of the physical shot and ensure that the picture has reasonable and consistent content presentation during the duration of the commentary voice.

[0183] On the contrary, when the commentary voice finishes a certain plot quickly, but the corresponding physical shot has more content and a longer duration, the most representative part that best reflects the core of the plot can be automatically obtained from the physical shot and retained. Remove redundant pictures to make the physical shot duration adapt to the commentary voice duration. For example, for the physical shot of the Sun Liu coalition forces preparing for battle on the south bank, its original duration may be long, but the commentary voice only briefly mentions this situation, then some pictures of soldiers' preparations can be automatically edited out, leaving pictures such as generals commanding and dispatching, key defense facilities layout, etc. that best reflect the core content of preparations, so that the physical shot duration matches the commentary voice duration to ensure the compactness and smoothness of the entire video rhythm.

[0184] After the explanation sub-segment acquisition step 105, the subsequent processing steps are continued, which will not be described in detail in this embodiment.

[0185] By combining the commentary voice and reference physical shots according to the timeline sequence and semantic logic, the precise synchronization of sound and picture is achieved. When watching the video, the audience can hear the commentary's description of the plot while seeing the corresponding picture, which greatly improves the accuracy and intuitiveness of information transmission.

[0186] It provides an effective solution for the situation where the length of the commentary voice and the physical lens are inconsistent. When the commentary voice is long and the physical lens is short, the use of slow motion, freeze-frame close-up or loop playback can not only make the picture better match the commentary, but also strengthen the audience's visual perception of key plot elements. Conversely, when the commentary voice is short and the physical lens is long, automatic editing to retain the core picture ensures the compact and smooth rhythm of the video, removes redundant information, and highlights the key plot.

[0187] Embodiment 12

[0188] This embodiment explains the combination of explanation videos in detail.

[0189] After executing steps 101 , 102 , 103 , 104 , and 105 of the present disclosure, a commentary video combining step 106 is executed.

[0190] In this embodiment, the commentary videos of the plot segments are sorted and organized according to the overall plot context of the original video, and the commentary videos of adjacent plot segments are connected and transitioned to obtain the commentary video of the original video.

[0191] After the narration videos for the multiple plot segments of the original video are generated, the narration videos for the plot segments are sorted and organized according to the overall plot context of the original video.

[0192] For example, the first plot segment commentary video focuses on the background of the turbulent conflicts in the late Eastern Han Dynasty, including the chaos caused by the Yellow Turban Rebellion and the rise of local warlords; the second plot segment commentary video revolves around the rise of Cao Cao, telling how Cao Cao defeated Yuan Shao in the Battle of Guandu and gradually unified the north; the third plot segment focuses on the Battle of Chibi, that is, the confrontation and battle process between the Sun-Liu coalition forces and Cao's army mentioned earlier.

[0193] In this embodiment, the process may further include performing a connection and transition process of the video images and the video sounds of the commentary video for the adjacent plot segments.

[0194] The connection and transition between the commentary videos of adjacent plot segments is processed. In the connection and transition between the commentary videos of adjacent plot segments, a combination of multiple video processing and / or commentary word association strategies is adopted to ensure that the commentary video of the entire original video is smooth and coherent.

[0195] In terms of video processing methods, it includes fade-in, fade-out, screen switching and other methods.

[0196] The first is the fade-in and fade-out effect. When two adjacent plot segment commentary videos are connected, for example, when transitioning from plot segment A to plot segment B, a fade-out effect is set at the end of plot segment A. The transparency of the picture can be gradually reduced, from the normal 100% transparency to 0% linearly within 2-3 seconds, so that the picture gradually disappears. At the same time, a fade-in effect is used at the beginning of plot segment B, and its transparency gradually increases from 0% to 100% within the same time. For example, when transitioning from the plot segment of Cao Cao's unification of the north to the plot segment of the Battle of Red Cliffs, the picture of Cao Cao's victory celebration in the northern battlefield fades out, and the hazy panoramic picture of Red Cliffs on the Yangtze River fades in, avoiding the abrupt feeling caused by the sudden switching of the picture.

[0197] The second is the picture gradient and blur processing. In addition to fade in and fade out, the picture gradient effect can also be used. When the plot segment A transitions to the plot segment B, the ending picture of the plot segment A is blurred, so that it gradually loses details, and the color saturation of the picture is gradually reduced. The starting picture of the plot segment B gradually becomes clear from the blurred state, and the color saturation gradually increases.

[0198] In addition, the key elements can be continued and transformed. For example, when transitioning from the plot segment of Guan Yu's single-sword meeting to the subsequent battle between the Eastern Wu and Shu Han over Jingzhou, the key element of the Green Dragon Crescent Blade in the scene of Guan Yu's single-sword meeting can be gradually blurred and transformed into the outline of Jingzhou City during the transition, and then the picture of Jingzhou City gradually becomes clear and complete, and the explanation of the new plot segment begins.

[0199] During the entire transition process, we not only focused on the natural switching of the images, but also processed the audio. When switching, the audio volume of the previous plot segment commentary video gradually decreases, while the audio of the next plot segment commentary video slowly increases, and the tone and timbre are also kept consistent as much as possible to avoid sudden or inharmonious sounds.

[0200] In this embodiment, the process may further include performing a connection and transition process of the adjacent plot segments and the previous situation summary in the commentary copy in association with the commentary words.

[0201] In terms of the commentary association strategy: adopt the previous situation summary association. At the beginning of the commentary of plot segment B, set a short and concise previous situation summary to mention the key content related to plot segment A. For example, at the beginning of the commentary of the Red Cliff plot segment, you can say: "In the previous plot, Cao Cao has unified the north and has a strong army. His army went south to Jingzhou. Jingzhou Liu Cong surrendered without a fight. Cao Cao took advantage of the situation and went east. At this time, although the Sun-Liu coalition forces had a huge gap in strength, they were already ready on the south bank of the Yangtze River. So how will this unequal contest unfold?" Through such a previous situation summary, the audience's memory of the previous plot is quickly aroused, paving the way for the development of the new plot, so that the audience can better understand the coherence of the plot.

[0202] Use suspense and foreshadowing. Set up suspense or foreshadowing at the end of the commentary of plot segment A, and then answer or continue it at the beginning of the commentary of plot segment B. For example, at the end of the plot segment about the battle of Tongguan between Cao Cao and Ma Chao, the commentary can be: "Although Cao Cao encountered many dangers in the battle of Tongguan, he finally defeated Ma Chao by cleverly using the strategy of alienation. However, the victory of this war planted a seed of instability in the Xiliang region. How will this seed germinate in the future and affect Cao Cao's strategic layout?" At the beginning of the commentary of the next plot segment (such as the subsequent events of Cao Cao's pacification of Xiliang), you can say: "As foreshadowing planted before, after Cao Cao won the battle of Tongguan, the remaining forces in the Xiliang region were not completely pacified. They colluded secretly and planned new actions..." This kind of suspense and foreshadowing can firmly grasp the curiosity of the audience and make them more interested in continuing to watch the subsequent plots.

[0203] It can also be linked by themes. From plot segment A to plot segment B, the commentary is linked around a common theme. For example, when transitioning from the plot segment about Cao Cao's way of employing people to the plot segment about Cao Cao's military strategy, the commentary can be connected like this: "Cao Cao's way of employing people has brought together many talents under his command, and these talents have played a vital role in the implementation of Cao Cao's military strategy. Previously we saw how Cao Cao recruited talents, and now we will see how Cao Cao relied on these talents to display his military wisdom on the battlefield." Through the association of themes, the audience can clearly understand the relationship between different plot segments within the overall theme framework, and enhance the macro grasp of the entire video content.

[0204] Through the comprehensive application of the above-mentioned video processing methods and commentary association strategies, it is possible to effectively achieve seamless transitions between adjacent plot segment commentary videos, and create high-quality, smooth and coherent original video commentary videos, allowing the audience to immerse themselves in the complete story narrative and better understand and feel the information and emotions conveyed by the video.

[0205] After such sorting, organization, connection and transition processing, we finally obtained a complete commentary video of the original video.

[0206] In a related embodiment, there are multiple possible combinations of the first largest language model, the second largest language model, the third largest language model, and the fourth largest language model.

[0207] First, they can be the same instance of the same large language model, which means that in the entire implementation process, only a single running entity of a specific large language model is used to undertake all related tasks and functions, and all interactions and processing are completed by this one instance.

[0208] Secondly, they may be multiple instances of the same large language model. For example, in order to meet different task load requirements or improve processing efficiency, multiple parallel running instances are created for the same large language model. The first large language model, the second large language model, the third large language model, and the fourth large language model correspond to these different instances. They are the same in basic model architecture and functions, but may differ in specific operating status, resource allocation, etc., and each independently processes different task flows or data blocks.

[0209] Furthermore, they can be instances of multiple different large language models. For example, the first large language model is an instance of model A, which has unique advantages in natural language understanding; the second large language model is an instance of model B, which is good at text generation; the third large language model is an instance of model C, which performs well in knowledge graph construction; the fourth large language model is an instance of model D, which has good performance in multi-language processing. These different large language model instances cooperate with each other or are applied to different links and scenarios in the embodiments to achieve comprehensive goals.

[0210] Another possibility is that some of them are instances of the same large language model, and some are instances of other large language models. For example, the first large language model and the second large language model are different instances of the same large language model, while the third large language model is an instance of another different large language model, and the fourth large language model is an instance of another large language model. They play corresponding roles in the overall architecture of the embodiment according to their respective characteristics and functions, and jointly build a complex and diverse language processing system.

[0211] Embodiment 13

[0212] like Figure 3 As shown, the explanation video generating device 200 provided in this embodiment includes:

[0213] The plot segment acquisition unit 201 acquires the plot segment of the original video according to the video text and time information of the original video;

[0214] A narration sub-segment acquisition unit 202 acquires a narration sub-segment suitable for narration from each plot segment according to the video text corresponding to each plot segment;

[0215] A commentary copy generating unit 203 generates a commentary copy and a corresponding reference physical shot according to the commentary sub-segment;

[0216] The original sound sub-segment obtaining unit 204 obtains the original sound sub-segment corresponding to the narration text from the plot segment;

[0217] A narration video generating unit 205 generates a narration video of the plot segment according to the narration text, the reference physical lens, and the original sound sub-segment;

[0218] The narration video combining unit 206 combines the narration videos of all the plot segments to obtain the narration video of the original video.

[0219] In this embodiment, in the commentary video generating device 200, the specific processing of each unit and the technical effects brought about by it can refer to the relevant descriptions of each step in the aforementioned embodiment, and will not be repeated here.

[0220] Embodiment 14

[0221] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0222] Figure 4 A schematic block diagram of an example electronic device 800 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0223] like Figure 4As shown, the device 800 includes a computing unit 801, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 802 or a computer program loaded from a storage unit 808 into a random access memory (RAM) 803. In the RAM 803, various programs and data required for the operation of the device 800 can also be stored. The computing unit 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0224] A number of components in the device 800 are connected to the I / O interface 805, including: an input unit 806, such as a keyboard, a mouse, etc.; an output unit 807, such as various types of displays, speakers, etc.; a storage unit 808, such as a disk, an optical disk, etc.; and a communication unit 809, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 809 allows the device 800 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0225] The computing unit 801 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 801 performs the various methods and processes described above, such as the narration video generation method. For example, in some embodiments, the narration video generation method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 808. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 800 via ROM 802 and / or communication unit 809. When the computer program is loaded into RAM 803 and executed by the computing unit 801, one or more steps of the narration video generation method described above may be performed. Alternatively, in other embodiments, the computing unit 801 may be configured to perform the narration video generation method in any other appropriate manner (e.g., by means of firmware).

[0226] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0227] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable information processing device, so that the program code, when executed by the processor or controller, implements the functions / operations specified in the flow chart and / or block diagram. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0228] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0229] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0230] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an information server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communications network). Examples of communications networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0231] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0232] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0233] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A method for generating a commentary video, characterized in that: The method comprises: A plot segment acquisition step, acquiring the plot segment of the original video according to the video text and time information of the original video; A step of obtaining a narration sub-segment, obtaining a narration sub-segment suitable for narration from each plot segment according to the video text corresponding to each plot segment; A step of generating a commentary copy, generating a commentary copy and a corresponding reference physical shot according to the commentary sub-segment; an original sound sub-segment obtaining step, obtaining an original sound sub-segment corresponding to the narration copy from the plot segment; An explanation video generation step is to generate an explanation video of the plot segment according to the explanation text, the reference physical lens, and the original sound sub-segment; The step of combining the narration videos is to combine the narration videos of all the plot segments to obtain the narration video of the original video.

2. The method according to claim 1, characterized in that Before the plot segment acquisition step, the method further includes: The video text is obtained based on the subtitle information and / or voice information of the original video, and the time information of the video text is obtained according to the time information of the subtitle information and / or voice information.

3. The method according to claim 1, characterized in that Before the plot segment acquisition step, the method further includes: The physical lens of the original video is obtained; based on the subtitle information and / or voice information of the physical lens, the lens copy corresponding to the physical lens and its time information are obtained; and all the lens copies are combined to obtain the video copy.

4. The method according to claim 1, characterized in that The plot segment acquisition step includes: Based on the pre-trained first language model, a plot segment of the original video is obtained from the video text of the original video; and based on time information of the video text covered by the plot segment, a boundary point of the plot segment is obtained.

5. The method according to claim 1, characterized in that After the plot segment acquisition step, the method further includes: A physical shot of the original video is obtained; in response to a segmentation point of the physical shot being within a first preset threshold range of a boundary point of the plot segment, the shot segmentation point is used as a boundary point of the plot segment; the first preset threshold range is pre-set according to the type of the original video.

6. The method according to claim 1, characterized in that The step of obtaining the commentary sub-segment comprises: Based on the pre-trained second largest language model, a narration sub-segment suitable for narration is obtained from the plot segment.

7. The method according to claim 1, characterized in that The step of obtaining the commentary sub-segment comprises: Based on the text features of the video text covered by the plot segment, a narration sub-segment suitable for narration is obtained from the plot segment; the text features include at least one of the following: the amount of dialogue information of the text, the main plot attributes of the text, and the plot development rhythm of the text.

8. The method according to claim 7, characterized in that The step of generating the explanation text comprises: Based on the pre-trained third language model, a commentary copy is generated according to the video copy of the commentary sub-clip, and a source video copy corresponding to the commentary copy is given; according to the time information of the source video copy, the physical shot corresponding to the commentary copy is matched and obtained as a reference physical shot.

9. The method according to claim 1, characterized in that: In the step of generating the narration copy, the narration copy includes a plot topic; The step of acquiring the original sound sub-segment includes: acquiring the original sound sub-segment corresponding to the plot topic from the plot segment.

10. The method according to claim 1, characterized in that In the step of generating the explanation text, the explanation text includes a summary of the previous situation.

11. The method according to claim 1, characterized in that: The step of obtaining the original sound sub-video includes: Based on the pre-trained fourth language model, a sub-video text is obtained from the plot segment according to the commentary text; the sub-video text is matched with the commentary text; and according to time information of the sub-video text, an original sound sub-segment corresponding to the sub-video text is determined.

12. The method according to claim 1, characterized in that The step of generating the explanation video comprises: The commentary voice generated by the commentary copy and the reference physical lens are merged according to the timeline order and semantic logic; the original sound sub-segment is connected immediately after the merged commentary voice and the reference physical lens to obtain a commentary video.

13. The method according to claim 1, characterized in that The step of combining the explanation videos comprises: The commentary videos of the plot segments are sorted and organized according to the overall plot context of the original video, and the commentary videos of adjacent plot segments are connected and transitioned to obtain the commentary video of the original video.

14. The method according to claim 13, characterized in that In the step of combining the commentary videos, the connection and transition processing includes: For the adjacent plot segments, the commentary video is processed with the connection and transition of the video images and the video sounds.

15. The method according to claim 13, characterized in that The step of combining the commentary videos and the step of transitioning the video together include: For the adjacent plot segments, the previous story summary in the commentary copy is processed with the connection and transition of the commentary.

16. A device for generating a commentary video, the device comprising: A plot segment acquisition unit, which acquires the plot segment of the original video according to the video text and time information of the original video; A narration sub-segment acquisition unit, which acquires a narration sub-segment suitable for narration from each plot segment according to the video text corresponding to each plot segment; A commentary copy generating unit, which generates a commentary copy and a corresponding reference physical shot according to the commentary sub-segment; An original sound sub-segment acquisition unit, which acquires an original sound sub-segment corresponding to the narration text from the plot segment; An explanation video generating unit generates an explanation video of the plot segment according to the explanation text, the reference physical lens, and the original sound sub-segment; The narration video combining unit combines the narration videos of all the plot segments to obtain the narration video of the original video.

17. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 15.

18. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 15.

19. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements the method according to any one of claims 1 to 15.

Images