Generation method and device of commentary video, equipment and storage medium

Abstract

The invention provides a speaking video generation method and device, equipment and a storage medium, and relates to the technical field of data processing, in particular to the technical field of video processing. The specific implementation scheme is as follows: performing plot splitting processing on a to-be-processed video to obtain at least one plot fragment; the plot segments are identified, the identification result of the plot segments is obtained, and the identification result of the plot segments represents that the plot segments are highlight plot segments or non-highlight plot segments; identifying the non-highlight plot segment to obtain the explanation voice of the non-highlight plot segment, and replacing the original audio in the non-highlight plot segment with the explanation voice to obtain the processed non-highlight plot segment; and generating a commentary video of the to-be-processed video according to the highlight plot segment and the processed non-highlight plot segment. The effect of improving the generation efficiency of the explanation video can be achieved, and meanwhile the video effect of the generated explanation video is improved.

Description

Technical Field

[0001] The present disclosure relates to the field of video processing technology in the field of data processing technology, and in particular to a method, device, equipment and storage medium for generating a commentary video. Background Art

[0002] Watching various videos has become an important way of life and entertainment for people, for example, watching TV series or watching short plays. In order to enable users to understand the content of the video as quickly as possible, it is necessary to provide users with video explanation information.

[0003] Furthermore, there is an urgent need for a solution that can generate a video explanation video. Summary of the invention

[0004] The present disclosure provides a method, apparatus, device and storage medium for generating a commentary video.

[0005] According to a first aspect of the present disclosure, a method for generating an explanation video is provided, comprising:

[0006] Performing plot splitting processing on the video to be processed to obtain at least one plot segment; and identifying the plot segment to obtain an identification result of the plot segment, wherein the identification result of the plot segment indicates whether the plot segment is a highlight plot segment or a non-highlight plot segment;

[0007] Identifying the non-highlighted plot segment to obtain a narration voice of the non-highlighted plot segment, and replacing the original audio in the non-highlighted plot segment with the narration voice to obtain a processed non-highlighted plot segment;

[0008] A commentary video of the video to be processed is generated according to the highlighted plot segment and the processed non-highlight plot segment.

[0009] According to a second aspect of the present disclosure, a device for generating a commentary video is provided, comprising:

[0010] The recognition module is used to perform plot segmentation processing on the video to be processed to obtain at least one plot segment; and to recognize the plot segment to obtain a recognition result of the plot segment, wherein the recognition result of the plot segment indicates whether the plot segment is a highlight plot segment or a non-highlight plot segment;

[0011] A processing module, used for identifying the non-highlighted plot segment, obtaining the commentary voice of the non-highlighted plot segment, and replacing the original audio in the non-highlighted plot segment with the commentary voice to obtain the processed non-highlighted plot segment;

[0012] A generating module is used to generate a commentary video of the video to be processed according to the highlighted plot segment and the processed non-highlight plot segment.

[0013] According to a third aspect of the present disclosure, a computer program product is provided, comprising: a computer program, wherein the computer program is stored in a readable storage medium, at least one processor of an electronic device can read the computer program from the readable storage medium, and the at least one processor executes the computer program so that the electronic device executes the method described in the first aspect.

[0014] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the method described in the first aspect.

[0015] According to a fifth aspect of the present disclosure, a computer program product is provided, comprising: a computer program, wherein the computer program is stored in a readable storage medium, at least one processor of an electronic device can read the computer program from the readable storage medium, and the at least one processor executes the computer program so that the electronic device executes the method described in the first aspect.

[0016] The technology disclosed in the present invention solves the problem of low generation efficiency of commentary videos and improves the video effect of the generated commentary videos.

[0017] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0019] Figure 1 is a schematic diagram according to a first embodiment of the present disclosure;

[0020] Figure 2 is a schematic diagram according to a second embodiment of the present disclosure;

[0021] Figure 3 is a schematic diagram according to a third embodiment of the present disclosure;

[0022] Figure 4 is a schematic diagram according to a fourth embodiment of the present disclosure;

[0023] Figure 5 is a block diagram of a device for generating a narration video for implementing an embodiment of the present disclosure;

[0024] Figure 6 The block diagram is a block diagram of an electronic device for implementing the method for generating an explanation video according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0025] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0026] The present disclosure provides a method, device, equipment and storage medium for generating an explanation video, which are applied to the field of video processing technology in the field of data processing technology to improve the generation efficiency of the explanation video and improve the video effect of the generated explanation video.

[0027] Figure 1 Schematic diagram of the first embodiment of the present disclosure. Figure 1 As shown, the method for generating a commentary video provided by the first embodiment of the present disclosure includes:

[0028] Step S101: performing plot segmentation processing on the video to be processed to obtain at least one plot segment; and identifying the plot segment to obtain a recognition result of the plot segment.

[0029] Specifically, the video to be processed can be subjected to plot splitting processing to obtain at least one plot segment. The video to be processed is a video to generate a commentary video, and the present disclosure does not limit the video to be processed. Any video that needs to generate a commentary video can be used as the video to be processed provided by the present disclosure. Optionally, the video to be processed can be a short play video, or a drama video or other video. The plot segment is a video segment belonging to the same video plot in the video to be processed. Specifically, if the video to be processed includes only one video plot, a plot segment can be obtained after the video to be processed is subjected to plot splitting processing. If the video to be processed includes multiple video plots, multiple plot segments can be obtained after the video to be processed is subjected to plot splitting processing, and the video plot corresponds to the plot segment one by one.

[0030] Among them, the present disclosure does not limit the process of performing plot splitting processing on the video to be processed to obtain at least one plot fragment. Any process that can perform plot splitting processing on the video to be processed to obtain at least one plot fragment can be used as the process provided by the present disclosure to perform plot splitting processing on the video to be processed to obtain at least one plot fragment. Optionally, the human voice in the video to be processed can be first identified to obtain at least one character information of the video to be processed; then, based on the character information, the video to be processed can be plot split to obtain at least one plot fragment.

[0031] Specifically, after the video to be processed is subjected to plot splitting processing and at least one plot segment is obtained, the plot segment can be identified to obtain a recognition result of the plot segment. The recognition result of the plot segment characterizes whether the plot segment is a highlight plot segment or a non-highlight plot segment. The highlight plot segment refers to a plot segment including a highlight picture, and the non-highlight plot segment refers to a plot segment not including a highlight picture. The present disclosure does not limit the process of identifying the plot segment and obtaining the recognition result of the plot segment. Any process that can identify the plot segment and obtain the recognition result of the plot segment can be used as the process of identifying the plot segment and obtaining the recognition result of the plot segment provided by the present disclosure. Optionally, the plot segment may be firstly subjected to shot segmentation processing to obtain at least one shot segment; then the shot segment may be identified to obtain an identification result of the shot segment; wherein the identification result of the shot segment characterizes whether the shot segment is a highlight shot segment or a non-highlight shot segment; then, if at least one shot segment corresponding to the plot segment includes one or more highlight shot segments, the plot segment is determined to be a highlight plot segment; otherwise, the plot segment is determined to be a non-highlight plot segment.

[0032] Step S102: Identify the non-highlighted plot segments to obtain the commentary voice of the non-highlighted plot segments, and replace the original audio in the non-highlighted plot segments with the commentary voice to obtain the processed non-highlighted plot segments.

[0033] Specifically, after obtaining the recognition result of the plot segment according to the steps in step S101, the non-highlight plot segment can be identified to obtain the commentary voice of the non-highlight plot segment. Among them, the commentary voice refers to the voice that explains the video. The present disclosure does not limit the commentary voice. Any voice that can explain the video can be used as the commentary voice provided by the present disclosure. Among them, the present disclosure does not limit the process of identifying the non-highlight plot segment and obtaining the commentary voice of the non-highlight plot segment. Any process that can identify the non-highlight plot segment and obtain the commentary voice of the non-highlight plot segment can be used as the process of identifying the non-highlight plot segment and obtaining the commentary voice of the non-highlight plot segment provided by the present disclosure. Optionally, at least one role information of the non-highlight plot segment can be determined first; then, according to the role information, the non-highlight plot segment is identified to obtain the commentary voice of the non-highlight plot segment.

[0034] Specifically, after identifying the non-highlight plot segment and obtaining the narration voice of the non-highlight plot segment, the original audio in the non-highlight plot segment can be replaced with the narration voice to obtain the processed non-highlight plot segment. The present disclosure does not limit the process of replacing the original audio in the non-highlight plot segment with the narration voice to obtain the processed non-highlight plot segment. Any process that can replace the original audio in the non-highlight plot segment with the narration voice to obtain the processed non-highlight plot segment can be used as the process of replacing the original audio in the non-highlight plot segment with the narration voice to obtain the processed non-highlight plot segment provided by the present disclosure. Optionally, the duration of the original audio in the non-highlight plot segment and the duration of the narration voice of the non-highlight plot segment can be determined first, and then the replacement process is determined based on the relationship between the duration of the original audio in the non-highlight plot segment and the duration of the narration voice of the non-highlight plot segment, and the processed non-highlight plot segment is further obtained according to the determined replacement process.

[0035] Step S103: Generate a commentary video of the video to be processed according to the highlighted plot segments and the processed non-highlight plot segments.

[0036] Specifically, based on the highlighted plot segments identified in step S101 and the processed non-highlight plot segments obtained in step S102, an explanation video of the video to be processed can be generated. The present disclosure does not limit the process of generating the explanation video of the video to be processed based on the highlighted plot segments and the processed non-highlight plot segments. Any process that can generate the explanation video of the video to be processed based on the highlighted plot segments and the processed non-highlight plot segments can be used as the process provided by the present disclosure for generating the explanation video of the video to be processed based on the highlighted plot segments and the processed non-highlight plot segments.

[0037] Optionally, you can first determine the sequence of each highlighted plot segment and each processed non-highlight plot segment based on the start and end times of the highlighted plot segment and the start and end times of the processed non-highlight plot segment, and then merge each highlighted plot segment and each processed non-highlight plot segment based on their sequence to generate a merged video, i.e., a commentary video of the video to be processed.

[0038] In the disclosed embodiment, a video to be processed is subjected to plot splitting processing to obtain at least one plot segment; the plot segments are identified to obtain identification results of the plot segments, non-highlight plot segments are identified to obtain commentary voices of the non-highlight plot segments, and the original audio in the non-highlight plot segments is replaced with the commentary voices to obtain processed non-highlight plot segments; an commentary video of the video to be processed is generated based on the highlight plot segments and the processed non-highlight plot segments, wherein the commentary video of the video to be processed is automatically generated through the processes of plot splitting processing and generating commentary voices, thereby improving the generation efficiency of the commentary video, wherein, after the plot splitting processing, if the plot segment is identified as a highlight plot segment, it is retained in the generated commentary video, thereby improving the video effect of the generated commentary video.

[0039] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0040] In order to enable readers to more deeply understand the implementation principle of the present disclosure, the following Figure 2-Figure 6 right Figure 1 The illustrated embodiment is further refined.

[0041] Figure 2 is a schematic diagram according to the second embodiment of the present disclosure. Figure 2As shown, the process of performing plot splitting processing on the video to be processed to obtain at least one plot segment provided by the second embodiment of the present disclosure includes:

[0042] Step S201: Identify the human voice in the video to be processed and obtain at least one character information of the video to be processed.

[0043] Specifically, the human voice in the video to be processed can be identified to obtain at least one character information of the video to be processed. The human voice refers to the sound of a person speaking. The present disclosure does not limit the method of identifying the human voice in the video to be processed. Any method that can identify the human voice in the video to be processed can be used as the method of identifying the human voice in the video to be processed provided by the present disclosure. Optionally, the human voice in the video to be processed can be identified by speech recognition technology. The speech recognition technology (Automatic Speech Recognition, referred to as ASR), also known as automatic speech recognition, aims to convert the vocabulary content in human speech into computer-readable input, such as keys, binary codes or character sequences.

[0044] Among them, the present disclosure does not limit the process of identifying the human voice in the video to be processed and obtaining at least one character information of the video to be processed. Optionally, the human voice in the video to be processed can be first identified to obtain at least one triple data, and then the audio data corresponding to the triple data is extracted from the video to be processed according to the start time of the human voice and the end time of the human voice in the triple data. Then, according to the audio data corresponding to the triple data, the character information corresponding to the triple data is determined. Among them, in the process of identifying the human voice in the video to be processed and obtaining at least one character information of the video to be processed, according to the triple data of the video to be processed and the audio data corresponding to the triple data, the character information corresponding to the triple data can be accurately determined, further improving the accuracy of the plot splitting processing of the video to be processed and obtaining at least one plot segment, thereby improving the accuracy of the generation of the commentary video.

[0045] Optionally, in the process of identifying the human voice in the video to be processed and obtaining at least one role information of the video to be processed described above, the human voice in the video to be processed can be first identified to obtain at least one triple data. Wherein, the triple data is the information data of the human voice, for example: the triple data can be the information data of the human voice corresponding to a sentence, wherein the present disclosure does not limit the triple data, and optionally, the triple data can include the starting time of the human voice, the ending time of the human voice, and the text data of the human voice. Wherein, the triple data can be expressed as (t_start, t_end, text), wherein t_start is the starting time of the human voice, t_end is the ending time of the human voice, and text is the text data of the human voice, for example: if the triple data is the information data of the human voice corresponding to a sentence, then the starting time of the human voice is the starting time of the human voice corresponding to the sentence, the ending time of the human voice is the ending time of the human voice corresponding to the sentence, and the text data of the human voice is the text data of the human voice corresponding to the sentence.

[0046] Among them, the present disclosure does not limit the process of identifying the human voice in the video to be processed and obtaining at least one triplet data. Optionally, the human voice in the video to be processed can be first identified to obtain the initial start time of the human voice in the video to be processed, the initial end time of the human voice, and the text data of the human voice. And the subtitles in the video to be processed are identified to obtain at least one subtitle information, and the subtitle information has a subtitle start time and a subtitle end time. Then, according to the subtitle start time and subtitle end time of the subtitle information, the initial start time and the initial end time of the human voice are adjusted to obtain the adjusted initial start time of the human voice and the adjusted initial end time of the human voice. Then, according to the adjusted initial start time of the human voice, the adjusted initial end time of the human voice, and the text data of the human voice, the triplet data is generated. Among them, in the process of identifying the human voice in the video to be processed and obtaining at least one triplet data, the initial start time and the initial end time of the human voice in the video to be processed are adjusted according to the subtitle start time and the subtitle end time of the subtitle of the video to be processed, which can improve the accuracy of the generated triplet data, further improve the accuracy of the plot splitting processing of the video to be processed, and obtain at least one plot fragment, thereby improving the accuracy of the generation of the commentary video.

[0047] Specifically, the initial start time of the human voice and the initial end time of the human voice refer to the moments obtained after identifying the human voice in the video to be processed, and the text data of the human voice refers to the text obtained after identifying the human voice in the video to be processed, wherein the method of identifying the human voice in the video to be processed in the present disclosure can refer to the above description, and will not be repeated here. Specifically, the subtitle information is the information of the subtitles of the video to be processed, for example: the subtitle information can be the information of the subtitles corresponding to a sentence, wherein the present disclosure does not limit the subtitle information, optionally, the subtitle information may include the subtitle start time and the subtitle end time, and the subtitle start time and the subtitle end time refer to the moments obtained after identifying the subtitles in the video to be processed. wherein the present disclosure does not limit the method of identifying the subtitles in the video to be processed, any method that can identify the subtitles in the video to be processed can be used as the method of identifying the subtitles in the video to be processed provided by the present disclosure, optionally, the subtitles in the video to be processed can be identified by optical character recognition technology. Among them, Optical Character Recognition (OCR) refers to the process in which an electronic device (such as a scanner or digital camera) checks characters printed on paper, determines their shapes by detecting dark and light patterns, and then translates the shapes into computer text using character recognition methods; that is, for printed characters, an optical method is used to convert the text in a paper document into a black and white dot matrix image file, and recognition software is used to convert the text in the image into a text format for further editing and processing by word processing software.

[0048] Specifically, there is a one-to-one correspondence between the subtitle information and the triplet data. If the subtitle information of the video to be processed and the triplet data have a high degree of matching, the start time of the subtitles and the initial start time of the human voice will be the same, and the end time of the subtitles and the initial end time of the human voice will be the same. However, in most cases, the match between the subtitle information and the triplet data of the video to be processed is not high. Therefore, the start time of the subtitles and the initial start time of the human voice will be different, and / or the end time of the subtitles and the initial end time of the human voice will be different. If the subtitle start time is earlier than the initial start time of the human voice, and / or the subtitle end time is later than the initial end time of the human voice, the subtitle overflow problem will occur. Therefore, it is necessary to adjust the initial start time and the initial end time of the human voice according to the subtitle start time and the subtitle end time of the subtitle information to obtain the adjusted initial start time of the human voice and the adjusted initial end time of the human voice. Specifically, if the subtitle start time is earlier than the initial start time of the human voice, the initial start time of the human voice is adjusted to the subtitle start time. If the subtitle end time is later than the initial end time of the human voice, the initial end time of the human voice is adjusted to the subtitle end time. If the subtitle start time is not earlier than the initial start time of the human voice, and / or the subtitle end time is not later than the initial end time of the human voice, the initial start time and the initial end time of the human voice are not adjusted according to the subtitle start time and the subtitle end time of the subtitle information.

[0049] Specifically, the initial starting time of the adjusted human voice can be determined as the starting time of the human voice in the triplet data, and the initial ending time of the adjusted human voice can be determined as the ending time of the human voice in the triplet data, so that the newly determined starting time of the human voice and the newly determined ending time of the human voice, together with the text data of the human voice, constitute the triplet data.

[0050] Optionally, in the process of identifying the human voice in the video to be processed and obtaining at least one character information of the video to be processed as described above, after identifying the human voice in the video to be processed and obtaining at least one triplet data, the audio data corresponding to the triplet data can be extracted from the video to be processed according to the starting time of the human voice and the ending time of the human voice in the triplet data. Among them, the description of the starting time of the human voice and the ending time of the human voice can refer to the above description, which will not be repeated here. Among them, the present disclosure does not limit the process of extracting the audio data corresponding to the triplet data from the video to be processed according to the starting time of the human voice and the ending time of the human voice in the triplet data. Optionally, the original audio of the video to be processed can be extracted first, and then the audio data corresponding to the triplet data can be extracted from the original audio according to the starting time of the human voice and the ending time of the human voice in the triplet data.

[0051] Optionally, in the process of identifying the human voice in the video to be processed and obtaining at least one role information of the video to be processed as described above, after extracting the audio data corresponding to the triple data from the video to be processed according to the start time and the end time of the human voice in the triple data, the role information corresponding to the triple data can be determined according to the audio data corresponding to the triple data. Specifically, the role information represents the information of the video role corresponding to the triple data, such as the heroine, the hero, the passerby, and the like.

[0052] Among them, the present disclosure does not limit the process of determining the role information corresponding to the triple data according to the audio data corresponding to the triple data. Optionally, the audio features of the audio data corresponding to the triple data can be extracted first. Then, based on the extracted audio features, the audio category corresponding to the triple data is determined. Then, based on the triple data and the audio category corresponding to the triple data, the role information corresponding to the triple data is determined. Among them, in the process of determining the role information corresponding to the triple data according to the audio data corresponding to the triple data, the audio category corresponding to the triple data is first determined according to the audio features of the audio data, and then the role information corresponding to the triple data is determined according to the audio category corresponding to the triple data, which improves the accuracy of the determined role information, further improves the accuracy of the plot splitting processing of the video to be processed, and obtains at least one plot segment, thereby improving the accuracy of the generation of the commentary video.

[0053] The present disclosure does not limit the process of extracting audio features of audio data corresponding to triple data. Optionally, the audio features of audio data corresponding to triple data can be extracted based on an audio classification algorithm. The present disclosure does not limit the process of determining the audio category corresponding to triple data based on the extracted audio features. Optionally, the audio data can be clustered based on a hierarchical clustering algorithm and the extracted audio features to obtain the audio category corresponding to the triple data. The audio category corresponding to the triple data can be represented as speak. If the audio category corresponding to the triple data is three categories, they can be represented as speak01, speak02, and speak03 in sequence. The present disclosure does not limit the process of determining the role information corresponding to the triple data based on the triple data and the audio category corresponding to the triple data. Optionally, the triple data can be merged according to the audio category corresponding to the triple data to obtain the merged triple data. Then, according to the triple data after the merge processing and the audio category corresponding to the triple data, the character information corresponding to the triple data is determined, wherein, in the process of determining the character information corresponding to the triple data according to the triple data and the audio category corresponding to the triple data, the process of merging the triple data is added, the accuracy of determining the character information corresponding to the triple data is improved, and the accuracy of performing plot splitting processing on the video to be processed and obtaining at least one plot segment is further improved, thereby improving the accuracy of generating the commentary video. Specifically, in the process of merging the triple data according to the audio category corresponding to the triple data to obtain the merged triple data, according to the start time and the end time of the human voice included in the triple data, if it is determined that the audio category corresponding to the adjacent triple data is the same category, the adjacent triple data is merged, otherwise, the merging is not performed. For example: if it is determined that two adjacent triplet data are (t_start1, t_end1, text1) and (t_start2, t_end2, text2), then the result obtained after merging the two adjacent triplet data is (t_start1, t_end2, text 1+text2).

[0054] Step S202: performing plot segmentation processing on the video to be processed according to the role information to obtain at least one plot segment.

[0055] Specifically, after identifying the human voice in the video to be processed and obtaining at least one role information of the video to be processed, the video to be processed can be subjected to plot splitting processing according to the role information to obtain at least one plot fragment. Wherein, the description of the role information can refer to the description in step S201, which will not be repeated here. Wherein, the present disclosure does not limit the process of performing plot splitting processing on the video to be processed according to the role information to obtain at least one plot fragment. Optionally, the video to be processed can be subjected to plot splitting processing according to the triple data and the role information corresponding to the triple data to obtain at least one plot fragment. Wherein, in the process of performing plot splitting processing on the video to be processed according to the role information to obtain at least one plot fragment, the corresponding triple data can be combined to improve the accuracy of the plot splitting processing, thereby improving the accuracy of the generation of the commentary video. Wherein, the present disclosure does not limit the process of performing plot splitting processing on the video to be processed according to the triple data and the role information corresponding to the triple data. Optionally, the triple data and the role information corresponding to the triple data can be input into the large language model to obtain at least one plot fragment.

[0056] In the disclosed embodiment, in the process of performing plot splitting processing on the video to be processed to obtain at least one plot segment, at least one character information of the video to be processed is obtained by identifying the human voice in the video to be processed, and based on the character information, the video to be processed is plot splitting processing is performed to obtain at least one plot segment, wherein, performing plot splitting processing on the video to be processed based on the character information can improve the rationality and accuracy of the obtained plot segments, and further improve the rationality and accuracy of the commentary video.

[0057] Figure 3 is a schematic diagram according to the third embodiment of the present disclosure. Figure 3 As shown, the process of identifying the plot segment and obtaining the identification result of the plot segment provided by the third embodiment of the present disclosure includes:

[0058] Step S301: Perform shot segmentation processing on the plot segment to obtain at least one shot segment.

[0059] Specifically, the plot segment may be subjected to shot segmentation processing to obtain at least one shot segment, wherein the shot segment is a video segment obtained after the shot segmentation processing, and a shot segment includes at least one video frame.

[0060] The present disclosure does not limit the process of performing lens segmentation processing on the plot segment to obtain at least one lens segment. Optionally, the plot segment can be input into a physical lens segmentation model to output at least one lens segment. The present disclosure does not limit the physical lens segmentation model. Any model that can perform lens segmentation processing on the plot segment can be used as the physical lens segmentation model provided by the present disclosure. For example, it can be a model such as a TransNetV model based on 3D convolution, a TransNetV2 model, etc. Optionally, the video to be processed can be first input into the physical lens segmentation model described above to obtain at least one lens segment, and then one or more lens segments in the at least one lens segment corresponding to the plot segment can be determined. Optionally, the corresponding one or more lens segments can be determined based on the time information of the plot segment. For example, the corresponding one or more lens segments can be determined based on the starting time of the human voice of the first triple corresponding to the plot segment and the ending time of the human voice of the last triple corresponding to the plot segment.

[0061] Step S302: Identify the shot fragment to obtain a recognition result of the shot fragment.

[0062] Specifically, after the plot segment is processed by shot segmentation to obtain at least one shot segment, the shot segment can be identified to obtain the identification result of the shot segment. The identification result of the shot segment characterizes whether the shot segment is a highlight shot segment or a non-highlight shot segment. A highlight shot segment refers to a shot segment including a highlight picture, and a non-highlight shot segment refers to a shot segment not including a highlight picture. The present disclosure does not limit the process of identifying the shot segment and obtaining the identification result of the shot segment. Optionally, each video frame in the shot segment can be identified based on a highlight recognition algorithm to determine whether the picture in the video frame is a highlight picture. If at least one video frame corresponding to the shot segment includes one or more video frames in which the picture is a highlight picture, the shot segment is determined to be a highlight shot segment. Otherwise, the shot segment is determined to be a non-highlight shot segment.

[0063] Step S303: If at least one shot segment corresponding to the plot segment includes one or more highlight shot segments, the plot segment is determined to be a highlight plot segment; otherwise, the plot segment is determined to be a non-highlight plot segment.

[0064] Specifically, according to the identification in step S302, if it is determined that at least one shot segment corresponding to the plot segment includes one or more highlight shot segments, the plot segment is determined to be a highlight plot segment; otherwise, the plot segment is determined to be a non-highlight plot segment.

[0065] In the disclosed embodiment, in the process of identifying a plot segment and obtaining a recognition result of the plot segment, the plot segment is subjected to shot segmentation processing to obtain at least one shot segment, and the shot segment is identified to obtain a recognition result of the shot segment. If at least one shot segment corresponding to the plot segment includes one or more highlight shot segments, the plot segment is determined to be a highlight plot segment; otherwise, the plot segment is determined to be a non-highlight plot segment. By identifying the shot segments, the accuracy of identifying the plot segment can be improved, and the accuracy of generating the commentary video can be further improved.

[0066] Figure 4 is a schematic diagram according to the fourth embodiment of the present disclosure. Figure 4 As shown, the process of identifying non-highlight plot segments and obtaining the narration voice of the non-highlight plot segments provided by the fourth embodiment of the present disclosure includes:

[0067] Step S401: Determine at least one character information of a non-highlight plot segment.

[0068] Specifically, at least one character information of a non-highlight plot segment can be determined. For the description of the character information, reference can be made to the description in step S201, which will not be repeated here. The present disclosure does not limit the process of determining at least one character information of a non-highlight plot segment, and any process that can determine at least one character information of a non-highlight plot segment can be used as the process of determining at least one character information of a non-highlight plot segment provided by the present disclosure. Optionally, at least one triple data of a non-highlight plot segment can be determined; and at least one character information of the non-highlight plot segment is determined based on at least one triple data of the non-highlight plot segment. The triple data is the data obtained after identifying the human voice in the video to be processed. The triple data includes the starting time of the human voice, the ending time of the human voice, and the text data of the human voice. For the description of the triple data and the character information corresponding to the triple data, reference can be made to the description in step S201, which will not be repeated here. Among them, in the process of determining at least one character information of a non-highlight plot segment, at least one triplet data of the non-highlight plot segment is first determined, and then the corresponding character information is determined based on the triplet data. This can more accurately determine at least one character information of the non-highlight plot segment and further improve the accuracy of commentary video generation.

[0069] Among them, the present disclosure does not limit the process of determining at least one triple data of the non-highlight plot segment. Optionally, at least one triple data of the video to be processed can be obtained first. The specific process can refer to the description in step S201 and will not be repeated here; then, according to the start time of the non-highlight plot segment and the end time of the non-highlight plot segment, determine the corresponding at least one triple data.

[0070] Among them, the present disclosure does not limit the process of determining at least one character information of a non-highlighted plot segment based on at least one triple data of the non-highlighted plot segment. Optionally, if the process of determining at least one triple data of the non-highlighted plot segment is as shown above, then while obtaining at least one triple data of the video to be processed, the character information corresponding to each triple data can also be obtained. Therefore, after determining at least one triple data of the non-highlighted plot segment, the character information corresponding to each triple data in at least one triple data of the non-highlighted plot segment can be determined. Among them, the character information corresponding to each triple data in the determined at least one triple data of the non-highlighted plot segment is the at least one character information of the non-highlighted plot segment.

[0071] Step S402: Identify the non-highlighted plot segments according to the character information to obtain the narration voice of the non-highlighted plot segments.

[0072] Specifically, according to at least one character information of the non-highlighted plot segment determined in step S401, the non-highlighted plot segment can be identified to obtain the commentary voice of the non-highlighted plot segment. Among them, the present disclosure does not limit the process of identifying the non-highlighted plot segment according to the character information and obtaining the commentary voice of the non-highlighted plot segment. Optionally, the non-highlighted plot segment can be first identified according to the character information to obtain the commentary copy of the non-highlighted plot segment. Then, based on the commentary copy of the non-highlighted plot segment, the commentary voice of the non-highlighted plot segment is generated. Among them, in the process of identifying the non-highlighted plot segment according to the character information and obtaining the commentary voice of the non-highlighted plot segment, generating the commentary copy first can improve the accuracy of the generated commentary voice and further improve the accuracy of the commentary video generation.

[0073] The commentary copy refers to the copy that explains the non-highlighted plot segments, wherein the present disclosure does not limit the process of identifying the non-highlighted plot segments according to the character information and obtaining the commentary copy of the non-highlighted plot segments. Optionally, at least one character information of the non-highlighted plot segments can be input into the large language model to output the commentary copy of the non-highlighted plot segments. Optionally, if the process of determining at least one triple data of the non-highlighted plot segments is as shown above, the process of identifying the non-highlighted plot segments according to the character information and obtaining the commentary copy of the non-highlighted plot segments can be inputting at least one triple data of the non-highlighted plot segments and the character information corresponding to each triple data into the large language model to output the commentary copy of the non-highlighted plot segments.

[0074] For the description of the commentary voice, reference can be made to the description in step S102, which will not be repeated here. The present disclosure does not limit the process of generating the commentary voice of the non-highlight plot segment based on the commentary text of the non-highlight plot segment. Optionally, the initial commentary voice of the non-highlight plot segment and the commentary soundtrack of the non-highlight plot segment can be generated based on the commentary text of the non-highlight plot segment. Then, the commentary soundtrack of the non-highlight plot segment and the initial commentary voice of the non-highlight plot segment are combined and processed to obtain the commentary voice of the non-highlight plot segment. In the process of generating the commentary voice of the non-highlight plot segment based on the commentary text of the non-highlight plot segment, on the basis of the generated initial commentary voice, the commentary soundtrack is combined to improve the voice effect of the generated commentary voice, and further improve the video effect of the generated commentary video.

[0075] Among them, the present disclosure does not limit the process of generating the initial commentary voice of the non-highlight plot segment according to the commentary copy of the non-highlight plot segment. Optionally, the commentary copy of the non-highlight plot segment can be converted into the initial commentary voice of the non-highlight plot segment based on the text-to-speech technology (Text To Speec, referred to as TTS). Among them, the text-to-speech technology is an outstanding work that uses linguistics and psychology at the same time. With the support of the built-in chip, the text is intelligently converted into a natural speech stream through the design of a neural network. TTS technology converts text files in real time, and the conversion time can be calculated in seconds. Under the action of its unique intelligent voice controller, the voice rhythm of the text output is smooth, so that the listener feels natural when listening to the information, without the indifference and jerky feeling of machine voice output. TTS speech synthesis technology will soon cover the first and second level Chinese characters of the national standard, have an English interface, automatically recognize Chinese and English, and support mixed reading of Chinese and English. All voices use real-person Mandarin as standard pronunciation, achieving fast speech synthesis of 120-150 Chinese characters per minute and a reading speed of 3-4 Chinese characters per second, so that users can hear clear and pleasant sound quality and coherent and fluent intonation. A small number of MP3 players have TTS function.

[0076] Among them, the present invention combines the commentary soundtrack of the non-highlight plot segment with the initial commentary voice of the non-highlight plot segment to obtain the commentary voice of the non-highlight plot segment without limiting the process. Optionally, the duration of the commentary soundtrack of the non-highlight plot segment and the duration of the initial commentary voice of the non-highlight plot segment can be determined first. If the duration of the commentary soundtrack of the non-highlight plot segment is equal to the duration of the initial commentary voice of the non-highlight plot segment, the commentary soundtrack of the non-highlight plot segment and the initial commentary voice of the non-highlight plot segment can be directly combined. If the duration of the commentary soundtrack of the non-highlight plot segment is not equal to the duration of the initial commentary voice of the non-highlight plot segment, the commentary soundtrack of the non-highlight plot segment is adjusted according to the duration of the initial commentary voice of the non-highlight plot segment, so that the duration of the commentary soundtrack of the non-highlight plot segment after the adjustment is equal to the duration of the initial commentary voice of the non-highlight plot segment, and then the commentary soundtrack of the adjusted non-highlight plot segment and the initial commentary voice of the non-highlight plot segment are combined. Among them, the present disclosure does not limit the process of adjusting the commentary soundtrack of the non-highlight plot segment, and optionally, it can be one or more of the processing methods such as cropping, compression, lengthening, and copying.

[0077] In the disclosed embodiment, in the process of identifying non-highlighted plot segments and obtaining commentary voices for the non-highlighted plot segments, at least one character information of the non-highlighted plot segments is determined, and based on the character information, the non-highlighted plot segments are identified to obtain commentary voices for the non-highlighted plot segments, wherein the character information is first determined and then the commentary voice is determined based on the character information, which can improve the accuracy of the determined commentary voice and further improve the accuracy of commentary video generation.

[0078] In a possible embodiment, the process of replacing the original audio in the non-highlight plot segment with the commentary voice to obtain the processed non-highlight plot segment may include: first determining the duration of the original audio in the non-highlight plot segment and the duration of the commentary voice of the non-highlight plot segment. If the duration of the original audio in the non-highlight plot segment is equal to the duration of the commentary voice of the non-highlight plot segment, the original audio in the non-highlight plot segment is replaced with the commentary voice to obtain the processed non-highlight plot segment. If the duration of the original audio in the non-highlight plot segment is not equal to the duration of the commentary voice of the non-highlight plot segment, the non-highlight plot segment is adjusted so that the duration of the adjusted non-highlight plot segment is equal to the duration of the commentary voice of the non-highlight plot segment. And the adjusted original audio in the adjusted non-highlight plot segment is replaced with the commentary voice to obtain the processed non-highlight plot segment. The present disclosure does not limit the process of adjusting the non-highlight plot segments. Optionally, the non-highlight plot segments can be adjusted to accelerate the processing, or to decelerate the processing. Specifically, if the duration of the original audio in the non-highlight plot segment is greater than the duration of the narration voice of the non-highlight plot segment, the non-highlight plot segment is adjusted to decelerate. If the duration of the original audio in the non-highlight plot segment is less than the duration of the narration voice of the non-highlight plot segment, the non-highlight plot segment is adjusted to accelerate the processing.

[0079] In the disclosed embodiment, in the process of replacing the original audio in the non-highlight plot segment with the commentary voice to obtain the processed non-highlight plot segment, the non-highlight plot segment is adjusted according to the duration of the original audio in the non-highlight plot segment and the duration of the commentary voice of the non-highlight plot segment, which can improve the matching degree between the commentary voice and the non-highlight plot segment, further improve the video effect of the processed non-highlight plot segment, thereby improving the video effect of the generated commentary video.

[0080] In a possible embodiment, the method for generating the commentary video further includes: replacing the original subtitles in the processed non-highlight plot segments with commentary texts. For the description of the commentary texts, reference can be made to the description in step S402, which will not be repeated here.

[0081] In the disclosed embodiment, the original subtitles in the processed non-highlight plot segments are replaced with commentary texts, which can improve the video effects of the processed non-highlight plot segments, thereby improving the video effects of the generated commentary video.

[0082] Figure 5 is a block diagram of a device for generating a narration video for implementing an embodiment of the present disclosure, such as Figure 5 As shown, the apparatus 500 for generating a commentary video provided in the fifth embodiment of the present disclosure includes: a recognition module 501 , a processing module 502 , and a generation module 503 .

[0083] The recognition module 501 is used to perform plot segmentation processing on the video to be processed to obtain at least one plot segment; and to recognize the plot segment to obtain a recognition result of the plot segment, wherein the recognition result of the plot segment indicates whether the plot segment is a highlight plot segment or a non-highlight plot segment;

[0084] The processing module 502 is used to identify the non-highlight plot segments, obtain the commentary voice of the non-highlight plot segments, and replace the original audio in the non-highlight plot segments with the commentary voice to obtain the processed non-highlight plot segments;

[0085] The generating module 503 is used to generate a commentary video of the video to be processed according to the highlighted plot segments and the processed non-highlight plot segments.

[0086] In a possible embodiment, the recognition module 501 includes a splitting unit 504 (not shown in the figure), which is used to identify the human voice in the video to be processed and obtain at least one character information of the video to be processed; according to the character information, the video to be processed is subjected to plot splitting processing to obtain at least one plot segment.

[0087] In a possible embodiment, the splitting unit 504 (not shown in the figure) is specifically used to identify the human voice in the video to be processed and obtain at least one triplet data; wherein the triplet data includes the starting time of the human voice, the ending time of the human voice, and the text data of the human voice; according to the starting time of the human voice and the ending time of the human voice in the triplet data, the audio data corresponding to the triplet data is extracted from the video to be processed; according to the audio data corresponding to the triplet data, the role information corresponding to the triplet data is determined.

[0088] In a possible embodiment, the splitting unit 504 (not shown in the figure) is further specifically used to identify the human voice in the video to be processed, and obtain the initial starting time of the human voice in the video to be processed, the initial ending time of the human voice, and the text data of the human voice; and identify the subtitles in the video to be processed, and obtain at least one subtitle information, and the subtitle information has a subtitle starting time and a subtitle ending time; according to the subtitle starting time and the subtitle ending time of the subtitle information, adjust the initial starting time and the initial ending time of the human voice to obtain the adjusted initial starting time of the human voice and the adjusted initial ending time of the human voice; generate triple data according to the adjusted initial starting time of the human voice, the adjusted initial ending time of the human voice, and the text data of the human voice.

[0089] In a possible embodiment, the splitting unit 504 (not shown in the figure) is further specifically used to extract audio features of the audio data corresponding to the triplet data; determine the audio category corresponding to the triplet data based on the extracted audio features; and determine the role information corresponding to the triplet data based on the triplet data and the audio category corresponding to the triplet data.

[0090] In a possible embodiment, the splitting unit 504 (not shown in the figure) is further specifically used to merge the triplet data according to the audio category corresponding to the triplet data to obtain the merged triplet data; based on the merged triplet data and the audio category corresponding to the triplet data, determine the role information corresponding to the triplet data.

[0091] In a possible embodiment, the splitting unit 504 (not shown in the figure) is further specifically used to perform plot splitting processing on the video to be processed according to the triple data and the role information corresponding to the triple data to obtain at least one plot segment.

[0092] In a possible embodiment, the identification module 501 further includes an identification unit 505 (not shown in the figure), which is used to perform shot segmentation processing on the plot segment to obtain at least one shot segment; identify the shot segment to obtain an identification result of the shot segment; wherein the identification result of the shot segment indicates whether the shot segment is a highlight shot segment or a non-highlight shot segment; if at least one shot segment corresponding to the plot segment includes one or more highlight shot segments, the plot segment is determined to be a highlight plot segment; otherwise, the plot segment is determined to be a non-highlight plot segment.

[0093] In a possible embodiment, the processing module 502 includes a first processing unit 506 (not shown in the figure), which is used to determine at least one character information of a non-highlighted plot segment; and a second processing unit 507 (not shown in the figure), which is used to identify the non-highlighted plot segment based on the character information and obtain a commentary voice of the non-highlighted plot segment.

[0094] In a possible embodiment, the first processing unit 506 (not shown in the figure) is specifically used to determine at least one triplet data of a non-highlight plot segment; and based on the at least one triplet data of the non-highlight plot segment, determine at least one character information of the non-highlight plot segment; wherein the triplet data is the data obtained after identifying the human voice in the video to be processed; the triplet data includes the starting time of the human voice, the ending time of the human voice, and the text data of the human voice.

[0095] In a possible embodiment, the second processing unit 507 (not shown in the figure) is specifically used to identify non-highlight plot segments based on character information to obtain commentary text for the non-highlight plot segments; and generate commentary voice for the non-highlight plot segments based on the commentary text for the non-highlight plot segments.

[0096] In a possible embodiment, the second processing unit 507 (not shown in the figure) is further specifically used to generate an initial commentary voice for the non-highlighted plot segment and a commentary soundtrack for the non-highlighted plot segment based on the commentary text of the non-highlighted plot segment; and combine the commentary soundtrack for the non-highlighted plot segment and the initial commentary voice for the non-highlighted plot segment to obtain the commentary voice for the non-highlighted plot segment.

[0097] In a possible embodiment, the processing module 502 also includes a third processing unit 508 (not shown in the figure), which is used to determine the duration of the original audio in the non-highlight plot segment and the duration of the commentary voice of the non-highlight plot segment; if the duration of the original audio in the non-highlight plot segment is equal to the duration of the commentary voice of the non-highlight plot segment, the original audio in the non-highlight plot segment is replaced with the commentary voice to obtain a processed non-highlight plot segment; if the duration of the original audio in the non-highlight plot segment is not equal to the duration of the commentary voice of the non-highlight plot segment, the non-highlight plot segment is adjusted so that the duration of the adjusted non-highlight plot segment is equal to the duration of the commentary voice of the non-highlight plot segment; and the adjusted original audio in the adjusted non-highlight plot segment is replaced with the commentary voice to obtain a processed non-highlight plot segment.

[0098] In a possible embodiment, the device further includes: a replacement module 509 (not shown in the figure), which is used to replace the original subtitles in the processed non-highlight plot segments with commentary texts.

[0099] Figure 5 The provided device for generating the commentary video can execute the steps involved in the terminal in the above-mentioned corresponding method embodiments. Its implementation principle and technical effects are similar and will not be repeated here.

[0100] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a non-transitory computer-readable storage medium storing computer instructions, and a computer program product.

[0101] According to an embodiment of the present disclosure, the present disclosure also provides a computer program product, which includes: a computer program, the computer program is stored in a non-transitory computer-readable storage medium, at least one processor of an electronic device can read the computer program from the non-transitory computer-readable storage medium, and at least one processor executes the computer program so that the electronic device executes the solution provided by any of the above embodiments.

[0102] Figure 6 6 is a block diagram of an electronic device 600 for implementing a method for generating a narration video of an embodiment of the present disclosure. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0103] like Figure 6 As shown, the device 600 includes a computing unit 601, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 602 or a computer program loaded from a storage unit 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the device 600 can also be stored. The computing unit 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0104] A number of components in the device 600 are connected to the I / O interface 605, including: an input unit 606, such as a keyboard, a mouse, etc.; an output unit 607, such as various types of displays, speakers, etc.; a storage unit 608, such as a disk, an optical disk, etc.; and a communication unit 609, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 609 allows the device 600 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0105] The computing unit 601 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 601 performs the various methods and processes described above, such as the generation method of the narration video. For example, in some embodiments, the generation method of the narration video may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 608. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 600 via ROM 602 and / or communication unit 609. When the computer program is loaded into RAM 603 and executed by the computing unit 601, one or more steps of the generation method of the narration video described above may be performed. Alternatively, in other embodiments, the computing unit 601 may be configured to perform the generation method of the narration video in any other appropriate manner (e.g., by means of firmware).

[0106] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0107] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0108] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0109] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0110] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0111] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services ("Virtual Private Server", or "VPS" for short). The server may also be a server of a distributed system, or a server combined with a blockchain.

[0112] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0113] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A method for generating an explanation video, comprising: Performing plot splitting processing on the video to be processed to obtain at least one plot segment; and identifying the plot segment to obtain a recognition result of the plot segment, wherein the recognition result of the plot segment indicates whether the plot segment is a highlight plot segment or a non-highlight plot segment; Identifying the non-highlighted plot segment to obtain a narration voice of the non-highlighted plot segment, and replacing the original audio in the non-highlighted plot segment with the narration voice to obtain a processed non-highlighted plot segment; A commentary video of the video to be processed is generated according to the highlighted plot segment and the processed non-highlight plot segment.

2. The method according to claim 1, wherein: Perform plot splitting on the video to be processed to obtain at least one plot segment, including: Identify a human voice in the video to be processed, and obtain at least one character information of the video to be processed; According to the role information, the video to be processed is subjected to plot segmentation processing to obtain at least one plot segment.

3. The method according to claim 2, wherein: Identifying a human voice in the video to be processed and obtaining at least one character information of the video to be processed includes: Identify the human voice in the video to be processed to obtain at least one triplet of data; wherein the triplet of data includes the start time of the human voice, the end time of the human voice, and text data of the human voice; Extracting audio data corresponding to the triplet data from the video to be processed according to the start time and the end time of the human voice in the triplet data; According to the audio data corresponding to the triplet data, character information corresponding to the triplet data is determined.

4. The method according to claim 3, wherein: Identify the human voice in the video to be processed to obtain at least one triplet of data, including: Identify the human voice in the video to be processed, obtain the initial start time of the human voice in the video to be processed, the initial end time of the human voice and the text data of the human voice; and identify the subtitles in the video to be processed to obtain at least one subtitle information, wherein the subtitle information has a subtitle start time and a subtitle end time; According to the subtitle start time and the subtitle end time of the subtitle information, adjusting the initial start time and the initial end time of the human voice to obtain the adjusted initial start time and the adjusted initial end time of the human voice; The triplet data is generated according to the adjusted initial start time of the human voice, the adjusted initial end time of the human voice, and the text data of the human voice.

5. The method according to claim 3 or 4, wherein: Determining, according to the audio data corresponding to the triplet data, the role information corresponding to the triplet data, comprises: Extracting audio features of the audio data corresponding to the triplet data; Determining the audio category corresponding to the triplet data according to the extracted audio features; According to the triplet data and the audio category corresponding to the triplet data, role information corresponding to the triplet data is determined.

6. The method according to claim 5, wherein: Determining, according to the triplet data and the audio category corresponding to the triplet data, role information corresponding to the triplet data includes: According to the audio categories corresponding to the triplet data, the triplet data are merged to obtain merged triplet data; According to the merged triple data and the audio category corresponding to the triple data, the role information corresponding to the triple data is determined.

7. The method according to any one of claims 3 to 6, wherein: According to the role information, the video to be processed is subjected to plot splitting processing to obtain at least one plot segment, including: According to the triple data and the role information corresponding to the triple data, the video to be processed is subjected to plot splitting processing to obtain the at least one plot segment.

8. The method according to claim 1, wherein: Identifying the plot segment to obtain an identification result of the plot segment includes: Performing a shot segmentation process on the plot segment to obtain at least one shot segment; Identify the shot segment to obtain an identification result of the shot segment; wherein the identification result of the shot segment indicates whether the shot segment is a highlight shot segment or a non-highlight shot segment; If the at least one shot segment corresponding to the plot segment includes one or more highlight shot segments, the plot segment is determined to be a highlight plot segment; otherwise, the plot segment is determined to be a non-highlight plot segment.

9. The method according to claim 1, wherein: Identifying the non-highlighted plot segment to obtain the narration voice of the non-highlighted plot segment includes: Determining at least one character information of the non-highlight plot segment; The non-highlighted plot segment is identified according to the character information to obtain a narration voice of the non-highlighted plot segment.

10. The method according to claim 9, wherein: Determining at least one character information of the non-highlight plot segment includes: Determine at least one triplet of the non-highlighted plot segment; and determine at least one character information of the non-highlighted plot segment based on the at least one triplet of the non-highlighted plot segment; The triplet data is data obtained after identifying the human voice in the video to be processed; the triplet data includes the starting time of the human voice, the ending time of the human voice, and the text data of the human voice.

11. The method according to claim 9 or 10, wherein: According to the character information, the non-highlight plot segment is identified to obtain the commentary voice of the non-highlight plot segment, including: According to the character information, the non-highlight plot segment is identified to obtain a commentary copy of the non-highlight plot segment; According to the commentary copy of the non-highlighted plot segment, a commentary voice of the non-highlighted plot segment is generated.

12. The method according to claim 11, wherein: Generating a narration voice of the non-highlighted plot segment according to the narration copy of the non-highlighted plot segment includes: Generating an initial narration voice of the non-highlighted plot segment and a narration soundtrack of the non-highlighted plot segment according to the narration copy of the non-highlighted plot segment; The commentary soundtrack of the non-highlighted plot segment and the initial commentary voice of the non-highlighted plot segment are combined and processed to obtain the commentary voice of the non-highlighted plot segment.

13. The method according to any one of claims 9 to 12, wherein: Replacing the original audio in the non-highlighted plot segment with the commentary voice to obtain a processed non-highlighted plot segment, including: Determine the duration of the original audio in the non-highlighted plot segment and the duration of the commentary voice of the non-highlighted plot segment; If the duration of the original audio in the non-highlighted plot segment is equal to the duration of the commentary voice of the non-highlighted plot segment, the original audio in the non-highlighted plot segment is replaced with the commentary voice to obtain a processed non-highlighted plot segment; If the duration of the original audio in the non-highlighted plot segment is not equal to the duration of the commentary voice of the non-highlighted plot segment, the non-highlighted plot segment is adjusted so that the duration of the adjusted non-highlighted plot segment is equal to the duration of the commentary voice of the non-highlighted plot segment; and the adjusted original audio in the adjusted non-highlighted plot segment is replaced with the commentary voice to obtain a processed non-highlighted plot segment.

14. The method according to claim 13, wherein: The method further comprises: Replace the original subtitles in the processed non-highlight plot segments with the commentary copy.

15. A device for generating a commentary video, comprising: An identification module is used to perform plot segmentation processing on the video to be processed to obtain at least one plot segment; and identifying the plot segment to obtain a recognition result of the plot segment, wherein the recognition result of the plot segment indicates whether the plot segment is a highlight plot segment or a non-highlight plot segment; A processing module, used for identifying the non-highlighted plot segment, obtaining the commentary voice of the non-highlighted plot segment, and replacing the original audio in the non-highlighted plot segment with the commentary voice to obtain the processed non-highlighted plot segment; A generating module is used to generate a commentary video of the video to be processed according to the highlighted plot segment and the processed non-highlight plot segment.

16. The device according to claim 15, wherein: The recognition module includes a splitting unit, which is used to recognize the human voice in the video to be processed and obtain at least one role information of the video to be processed; according to the role information, the video to be processed is subjected to plot splitting processing to obtain at least one plot segment.

17. The device according to claim 15, wherein: The splitting unit is specifically used to identify the human voice in the video to be processed and obtain at least one triplet data; wherein the triplet data includes the starting time of the human voice, the ending time of the human voice, and the text data of the human voice; according to the starting time of the human voice and the ending time of the human voice in the triplet data, the audio data corresponding to the triplet data is extracted from the video to be processed; according to the audio data corresponding to the triplet data, the role information corresponding to the triplet data is determined.

18. The device according to claim 15, wherein: The splitting unit is further specifically used to identify the human voice in the video to be processed, obtain the initial starting time of the human voice, the initial ending time of the human voice and the text data of the human voice in the video to be processed; and identify the subtitles in the video to be processed, obtain at least one subtitle information, and the subtitle information has a subtitle starting time and a subtitle ending time; according to the subtitle starting time and the subtitle ending time of the subtitle information, adjust the initial starting time of the human voice and the initial ending time of the human voice to obtain the adjusted initial starting time of the human voice and the adjusted initial ending time of the human voice; generate the triplet data according to the adjusted initial starting time of the human voice, the adjusted initial ending time of the human voice, and the text data of the human voice.

19. The device according to claim 17 or 18, wherein: The splitting unit is further specifically used to extract audio features of the audio data corresponding to the triplet data; determine the audio category corresponding to the triplet data based on the extracted audio features; and determine the role information corresponding to the triplet data based on the triplet data and the audio category corresponding to the triplet data.

20. The device according to claim 19, wherein The splitting unit is further specifically used to merge the triplet data according to the audio category corresponding to the triplet data to obtain the merged triplet data; and determine the role information corresponding to the triplet data according to the merged triplet data and the audio category corresponding to the triplet data.

21. The device according to any one of claims 18 to 21, wherein: The splitting unit is further specifically used to perform plot splitting processing on the video to be processed according to the triple data and the role information corresponding to the triple data to obtain the at least one plot segment.

22. The device according to claim 15, wherein: The recognition module also includes a recognition unit, which is used to perform shot segmentation processing on the plot fragment to obtain at least one shot fragment; recognize the shot fragment to obtain a recognition result of the shot fragment; wherein the recognition result of the shot fragment characterizes whether the shot fragment is a highlight shot fragment or a non-highlight shot fragment; if the at least one shot fragment corresponding to the plot fragment includes one or more highlight shot fragments, the plot fragment is determined to be a highlight plot fragment; otherwise, the plot fragment is determined to be a non-highlight plot fragment.

23. The device according to claim 15, wherein: The processing module includes a first processing unit, which is used to determine at least one character information of the non-highlighted plot segment; and a second processing unit, which is used to identify the non-highlighted plot segment according to the character information to obtain the commentary voice of the non-highlighted plot segment.

24. The device according to claim 23, wherein: The first processing unit is specifically used to determine at least one triplet of the non-highlighted plot segment; and determine at least one character information of the non-highlighted plot segment based on the at least one triplet of the non-highlighted plot segment; wherein the triplet data is the data obtained after identifying the human voice in the video to be processed; the triplet data includes the starting time of the human voice, the ending time of the human voice, and the text data of the human voice.

25. The device according to claim 23 or 24, wherein: The second processing unit is specifically used to identify the non-highlight plot segment according to the character information to obtain the commentary text of the non-highlight plot segment; and generate the commentary voice of the non-highlight plot segment according to the commentary text of the non-highlight plot segment.

26. The device according to claim 25, wherein The second processing unit is further specifically used to generate an initial commentary voice of the non-highlighted plot segment and a commentary soundtrack of the non-highlighted plot segment based on the commentary text of the non-highlighted plot segment; and to combine the commentary soundtrack of the non-highlighted plot segment and the initial commentary voice of the non-highlighted plot segment to obtain the commentary voice of the non-highlighted plot segment.

27. The device according to any one of claims 23 to 26, wherein: The processing module further includes a third processing unit for determining the duration of the original audio in the non-highlighted plot segment and the duration of the commentary voice of the non-highlighted plot segment; If the duration of the original audio in the non-highlighted plot segment is equal to the duration of the commentary voice of the non-highlighted plot segment, the original audio in the non-highlighted plot segment is replaced with the commentary voice to obtain a processed non-highlighted plot segment; if the duration of the original audio in the non-highlighted plot segment is not equal to the duration of the commentary voice of the non-highlighted plot segment, the non-highlighted plot segment is adjusted so that the duration of the adjusted non-highlighted plot segment is equal to the duration of the commentary voice of the non-highlighted plot segment; and the adjusted original audio in the adjusted non-highlighted plot segment is replaced with the commentary voice to obtain a processed non-highlighted plot segment.

28. The device according to claim 27, wherein The device also includes: a replacement module, which is used to replace the processed original subtitles in the non-highlight plot segment with the commentary copy.

29. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 14.

30. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-14.

31. A computer program product, comprising a computer program, which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 14.

Images