Video generation method, device, equipment, computer readable medium and program product

Abstract

The embodiment of the invention discloses a video generation method, device and equipment, a computer readable medium and a program product. A specific embodiment of the method comprises the steps of obtaining a commentary text and a candidate video set corresponding to a target article; performing text preprocessing on the commentary text to obtain a text fragment sequence; for each text segment, executing a determination step: according to the text segment, performing video segmentation on each candidate video to generate a segmented video sequence, and obtaining a segmented video sequence set; determining video content information corresponding to each segmented video; screening out a target segmented video of which the semantic matching information meets a target matching condition from the segmented video sequence set; and generating a commentary video according to the target segmented video sequence and the text fragment sequence. The implementation mode is related to artificial intelligence, and the high-quality explanation video related to the target article in the explanation text can be accurately and efficiently generated.

Description

Technical Field

[0001] Embodiments of this disclosure relate to the field of artificial intelligence, and more specifically to video generation methods, apparatus, devices, computer-readable media, and program products. Background Technology

[0002] Currently, in e-commerce scenarios, high-quality explanatory videos for target products can effectively showcase their advantages and attract more user attention. The typical method for generating explanatory videos for target products is to utilize deep learning models to directly generate videos tailored to the specific product.

[0003] However, the inventors discovered that the following technical problems often arise when using the above method:

[0004] Deep learning models typically generate narration videos based directly on relevant item information. However, in e-commerce, it's often necessary to generate narration videos based on explanatory text. Therefore, conventional deep learning models used for video generation are generally not applicable to generating narration videos for e-commerce scenarios.

[0005] The information disclosed in this background section is only intended to enhance the understanding of the background of the inventive concept, and therefore may contain information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0006] The summary portion of this disclosure is intended to provide a brief overview of the concepts, which will be described in detail in the detailed description portion. This summary portion is not intended to identify key or essential features of the claimed technical solutions, nor is it intended to limit the scope of the claimed technical solutions.

[0007] Some embodiments of this disclosure provide video generation methods, apparatuses, devices, computer-readable media, and program products to address the technical problems mentioned in the background section above.

[0008] In a first aspect, some embodiments of this disclosure provide a video generation method, including: acquiring explanatory text corresponding to a target item and a set of candidate videos; performing text preprocessing on the explanatory text to obtain a text segment sequence; for each text segment in the text segment sequence, performing a determination step: based on the text segment, segmenting each candidate video in the candidate video set to generate a segmented video sequence, obtaining a set of segmented video sequences; determining the video content information corresponding to each segmented video in the set of segmented video sequences; selecting target segmented videos from the set of segmented video sequences whose semantic matching information satisfies the target matching condition, wherein the semantic matching information is the degree of content matching between the video content information and the text segment; and generating an explanatory video corresponding to the target item based on the obtained target segmented video sequence and the text segment sequence.

[0009] Optionally, the above-mentioned text preprocessing of the explanatory text to obtain a text segment sequence includes dividing the explanatory text into segments to obtain an initial text segment sequence; converting each initial text segment in the initial text segment sequence into a text segment in the target language to generate a language conversion segment, thus obtaining a language conversion segment sequence; and visually modifying and expanding each speech conversion segment in the language conversion segment sequence to generate a text segment, thus obtaining the above-mentioned text segment sequence.

[0010] Optionally, the above-mentioned method of segmenting each candidate video in the candidate video set according to the above-mentioned text fragment to generate a segmented video sequence includes: determining the estimated time required to elaborate on the content of the above-mentioned text fragment; segmenting the candidate video according to the estimated time required to generate a segmented video sequence, wherein the duration of each segmented video is the same as the estimated time required, and there is a predetermined proportion of identical video fragments between every two adjacent segmented videos.

[0011] Optionally, determining the video content information corresponding to each segmented video in the segmented video sequence set includes: generating video description content corresponding to the segmented video using a pre-trained multimodal large model; generating item description content related to the target item in the segmented video using the multimodal large model; and determining the video description content and the item description content as the video content information.

[0012] Optionally, the video content information is text-based content information; and the semantic matching information is generated through the following steps: extracting the first text feature semantic information corresponding to the video content information and the second text feature semantic information corresponding to the text segment; determining the feature semantic similarity between the first text feature semantic information and the second text feature semantic information as the semantic matching information.

[0013] Optionally, generating the narration video corresponding to the target item based on the obtained target segmented video sequence and the text fragment sequence includes: smoothing the video junctions between every two adjacent target segmented videos in the target segmented video sequence to obtain a smoothed segmented video sequence; sequentially splicing the segmented videos in the smoothed segmented video sequence to obtain a spliced ​​video; and generating the narration video based on the spliced ​​video and the text fragment sequence.

[0014] Secondly, some embodiments of this disclosure provide a video generation apparatus, including: an acquisition unit configured to acquire explanatory text corresponding to a target item and a candidate video set; a preprocessing unit configured to perform text preprocessing on the explanatory text to obtain a text segment sequence; an execution unit configured to perform a determination step for each text segment in the text segment sequence: based on the text segment, segmenting each candidate video in the candidate video set to generate a segmented video sequence, obtaining a segmented video sequence set; determining video content information corresponding to each segmented video in the segmented video sequence set; filtering target segmented videos whose semantic matching information satisfies the target matching condition from the segmented video sequence set, wherein the semantic matching information is the content matching degree between the video content information and the text segment; and a video generation unit configured to generate an explanatory video corresponding to the target item based on the obtained target segmented video sequence and the text segment sequence.

[0015] Optionally, the preprocessing unit can be configured to: divide the above-mentioned explanatory text into segments to obtain an initial text segment sequence; convert each initial text segment in the above-mentioned initial text segment sequence into a text segment in the target language to generate a language conversion segment, thereby obtaining a language conversion segment sequence; and visually enhance each speech conversion segment in the above-mentioned language conversion segment sequence to generate a text segment, thereby obtaining the above-mentioned text segment sequence.

[0016] Optionally, the execution unit can be configured to: determine the estimated time required to elaborate on the content of the above-mentioned text fragments; and, based on the estimated time required, segment the above-mentioned candidate videos to generate a segmented video sequence, wherein the duration of each segmented video is the same as the estimated time required, and there is a predetermined proportion of identical video fragments between every two adjacent segmented videos.

[0017] Optionally, the execution unit can be configured to: generate video description content corresponding to the segmented video using a pre-trained multimodal large model; generate item description content related to the target item in the segmented video using the multimodal large model; and determine the video description content and the item description content as the video content information.

[0018] Optionally, the video generation unit can be configured to: smooth the video junctions corresponding to every two adjacent target segmented videos in the above target segmented video sequence to obtain a smoothed segmented video sequence; sequentially splice the segmented videos in the above smoothed segmented video sequence to obtain a spliced ​​video; and generate the above narration video based on the above spliced ​​video and the above text segment sequence.

[0019] Thirdly, some embodiments of this disclosure provide an electronic device, including: one or more processors; and a storage device having one or more programs stored thereon, such that when the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any implementation of the first aspect.

[0020] Fourthly, some embodiments of this disclosure provide a computer-readable medium having a computer program stored thereon, wherein the program, when executed by a processor, implements the method as described in any implementation of the first aspect.

[0021] Fifthly, some embodiments of this disclosure provide a computer program product, including a computer program that, when executed by a processor, implements the method described in any of the implementations of the first aspect above.

[0022] The above embodiments of this disclosure have the following beneficial effects: The video generation methods of some embodiments of this disclosure can accurately and efficiently generate high-quality explanatory videos related to the target item based on the explanatory text. Specifically, the reason why related explanatory videos cannot be generated with high quality and efficiency is that deep learning models usually directly generate explanatory videos based on the relevant item information of the target item. However, in the e-commerce field, it is often necessary to generate explanatory videos based on the explanatory text. Therefore, conventional deep learning models used for video generation are often not applicable to the generation of explanatory videos in e-commerce scenarios. Based on this, the video generation method of some embodiments of this disclosure first obtains the explanatory text and candidate video set corresponding to the target item. Here, the obtained explanatory text serves as the core text content in the subsequent explanatory video and is used for the generation of the explanatory video. The obtained video set serves as the video data basis for the subsequent explanatory video and is used for the generation of the explanatory video. Then, the above explanatory text is preprocessed to obtain a text segment sequence, which divides the explanatory text according to the content of each stage, facilitating subsequent matching of stage-specific videos, allowing the text content of each stage to be semantically matched with the corresponding video. Next, for each text segment in the above text segment sequence, the following determination steps are performed: First, based on the text segment, each candidate video in the above candidate video set is segmented to generate a segmented video sequence set. Here, video segmentation facilitates matching suitable segmented videos to the text segments, resulting in more accurate narration videos. Second, the video content information corresponding to each segmented video in the above segmented video sequence set is determined to facilitate identifying the segmented video that best matches the semantic content of the text segment. Third, target segmented videos whose semantic matching information meets the target matching conditions are selected from the above segmented video sequence set, obtaining the most suitable segmented video corresponding to the above text segment. Here, the semantic matching information refers to the degree of content matching between the video content information and the above text segment. Finally, based on the obtained target segmented video sequence and the above text segment sequence, the narration video corresponding to the above target item can be accurately generated. In summary, by dividing the narration text into stages and determining the most suitable segmented video for each text segment, the segmented video sequence corresponding to the text segment sequence can be accurately obtained. Therefore, an accurate narration video matching the aforementioned narration text can be generated based on the segmented video sequence. Attached Figure Description

[0023] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and elements are not necessarily drawn to scale.

[0024] Figure 1 This is a schematic diagram illustrating an application scenario of a video generation method according to some embodiments of the present disclosure;

[0025] Figure 2 This is a flowchart of some embodiments of the video generation method according to the present disclosure;

[0026] Figure 3 This is a schematic diagram of a sequence of text segments in some embodiments of the video generation method according to the present disclosure;

[0027] Figure 4 This is a flowchart of some other embodiments of the video generation method according to the present disclosure;

[0028] Figure 5 These are schematic diagrams illustrating the structure of some embodiments of the video generation apparatus according to this disclosure;

[0029] Figure 6 This is a schematic diagram of the structure of an electronic device suitable for implementing some embodiments of the present disclosure. Detailed Implementation

[0030] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0031] It should also be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings. Unless otherwise specified, the embodiments and features described in this disclosure can be combined with each other.

[0032] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0033] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0034] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0035] Before performing any of the operations related to the collection, storage, and use of item information (such as explanatory text) involved in this disclosure, the relevant organizations or individuals shall fulfill their obligations, including conducting an impact assessment on the security of item information, informing the subject of the item information, and obtaining prior authorization and consent from the subject of the item information.

[0036] This disclosure will now be described in detail with reference to the accompanying drawings and embodiments.

[0037] Figure 1 This is a schematic diagram illustrating an application scenario of a video generation method according to some embodiments of the present disclosure.

[0038] exist Figure 1 In this application scenario, firstly, the electronic device 101 can acquire the explanatory text 103 and candidate video set 104 corresponding to the target item 102. In this application scenario, the target item 102 can be a "treadmill". The explanatory text 103 can be something like, "Have you ever tried working while looking at the sea and exercising at the same time? I recently got my eye on a treadmill from brand A, and it's a perfect match for my height-adjustable desk and pruning chair. The treadmill can be unfolded and folded, and can be stored under the bed when not in use." Then, the electronic device 101 can perform text preprocessing on the above explanatory text 102 to obtain a text fragment sequence 105. In this application scenario, the text fragment sequence 105 can include: text fragment 1051, text fragment 1052, and text fragment 1053. Text fragment 1051 can be something like, "Have you ever tried working while looking at the sea and exercising at the same time?" Text fragment 1052 can be something like, "I recently got my eye on a treadmill from brand A, and it's a perfect match for my height-adjustable desk and pruning chair." Text fragment 1053 could be "The treadmill can be unfolded and folded, and can be tucked under the bed when not in use." Next, for each text fragment in the above text fragment sequence 105, the electronic device 101 can perform the following steps: First, based on the above text fragment, segment each candidate video in the above candidate video set 104 to generate a segmented video sequence set. Second, determine the video content information corresponding to each segmented video in the above segmented video sequence set. Third, filter out target segmented videos from the above segmented video sequence set whose semantic matching information satisfies the target matching condition. Here, the semantic matching information refers to the degree of content matching between the video content information and the above text fragment. In this application scenario, the target matching condition can be the highest semantic matching information. Finally, the electronic device 101 can generate an explanatory video 110 corresponding to the target item based on the obtained target segmented video sequence and the above text fragment sequence. In this application scenario, the target segmented video sequence may include: target segmented video 107 corresponding to text segment 1051, target segmented video 108 corresponding to text segment 1052, and target segmented video 109 corresponding to text segment 1053.

[0039] It should be noted that the aforementioned electronic device 101 can be either hardware or software. When the electronic device is hardware, it can be implemented as a distributed cluster consisting of multiple servers or terminal devices, or as a single server or a single terminal device. When the electronic device is software, it can be installed in the hardware devices listed above. It can be implemented as, for example, multiple software programs or software modules used to provide distributed services, or as a single software program or software module. No specific limitations are made here.

[0040] It should be understood that Figure 1 The number of electronic devices shown is merely illustrative. Any number of electronic devices can be used depending on the implementation requirements.

[0041] Continue to refer to Figure 2 The diagram illustrates a flow 200 of some embodiments of a video generation method according to the present disclosure. The video generation method includes the following steps:

[0042] Step 201: Obtain the explanatory text and candidate video set corresponding to the target item.

[0043] In some embodiments, the execution entity of the above video generation method (e.g. Figure 1 The electronic device 101 shown can acquire the explanatory text and candidate video set corresponding to the target item via a wired or wireless connection. The target item can be the item for which an explanatory video is to be generated. The explanatory video can be a video describing the various characteristics of the target item. In practice, for e-commerce scenarios, the target item can be the product for which an explanatory video is to be generated. For example, the target item can be a treadmill. The explanatory video can be a video describing the advantages, uses, and other information of the target product. The explanatory text can be descriptive text describing the various characteristics of the target item; it is a professional, contextualized, and emotionally resonant textual description designed for the product. The explanatory text can be the explanatory text corresponding to the subsequent explanatory video. The candidate video set can be various video clips related to the target item, to be selected for generating the subsequent explanatory video. That is, video segments from the candidate video set can be extracted to generate the subsequent explanatory video.

[0044] It should be noted that each candidate video in the candidate video set has a corresponding target item tag. The target item tag can be a tag corresponding to the target item. The target item tag can represent the identity information corresponding to the target item.

[0045] In practice, candidate video sets can be generated through the following steps:

[0046] The first step is to obtain the initial video set, where the initial videos are those for which the corresponding item tags have not yet been determined.

[0047] The second step involves using a pre-trained item recognition model to determine the item label corresponding to each initial video in the initial video set. The item recognition model can be a neural network model for recognizing item information. In practice, the item recognition model can be a YOLO model.

[0048] The third step is to select a subset of initial videos from the initial video set whose corresponding item tags are the target item tags, and use this subset as the candidate video set.

[0049] Step 202: Perform text preprocessing on the above explanatory text to obtain a sequence of text fragments.

[0050] In some embodiments, the aforementioned execution entity may perform text preprocessing on the aforementioned explanatory text to obtain a text fragment sequence. The sequence order of the text fragments may be the order in which they are explained. Text preprocessing may include: text fragment segmentation, symbol removal, and case adjustment. A text fragment is a portion of the text content within the explanatory text. Each text fragment represents a content fragment from a different stage of the explanatory text. The core content corresponding to each stage's content fragment differs. For example, for a target item called a "treadmill," the text fragment sequence may include: {explaining information about the treadmill manufacturer, explaining information about the treadmill's price, explaining information about the treadmill's functions}.

[0051] See Figure 3 The diagram illustrates a sequence of text fragments. The text fragment sequence is: {"00:00-00:02": "Have you ever tried working while looking at the sea?", "00:02-00:03": "While exercising", "00:03-00:05": "Recently I've been eyeing this treadmill recommended by Luo Yonghao", "00:05-00:07": "It's a perfect match for my height-adjustable desk", "00:07-00:09": "Unfolding and folding only takes two steps", "00:09-00:11": "No need to..."} "When needed, it can be stuffed under the bed." "00:11-00:12": "When needed, it can be dragged out." "00:12-00:13": "Go to any room." "00:13-00:16": "Today, let him act as the fitness coach for working people." "00:16-00:17": "Raise the table." "00:17-00:18": "Standing while working." "00:18-00:19": "Is this environment suitable for overtime work?"

[0052] In some optional implementations of certain embodiments, the execution entity may perform text preprocessing on the explanatory text to obtain a sequence of text fragments, including the following steps:

[0053] The first step is to divide the above explanatory text into segments to obtain an initial text segment sequence. Each initial text segment is a content fragment from a different stage of the explanatory text.

[0054] As an example, firstly, the aforementioned execution entity can input the explanatory text into a pre-trained text key content extraction model to obtain the key content and scope of each key content in the explanatory text. Each key content corresponds to a different stage. Then, based on the scope of each key content, the explanatory text is divided into an initial text segment sequence.

[0055] The second step involves converting each initial text segment in the initial text segment sequence into a text segment in the target language to generate a language-converted segment sequence. The target language can be one that facilitates semantic extraction by a subsequent large language model. For example, the target language could be English.

[0056] The third step involves visually expanding each speech conversion segment in the above language conversion segment sequence to generate text segments, resulting in the aforementioned text segment sequence. Visual expansion can involve adding visually relevant content to the segments corresponding to the speech conversion segments. That is, transforming relatively abstract speech conversion segments into visually descriptive segments. For example, for the speech conversion segment "The weather is getting colder," the corresponding visually expanded text segment would be "Frost flowers hang on the branches, the ground is covered with white frost, pedestrians wrap themselves tightly in their clothes, and their breath turns into white mist."

[0057] As an example, firstly, the aforementioned execution entity can generate generation prompts representing the visual enhancement and expansion of the speech-to-text segment. For example, the generation prompts could be "this short sentence is converted into content related to specific objects and scenes." Then, the generation prompts and the speech-to-text segment are input into a large language model (e.g., ChanGPT) to generate the speech-to-text segment.

[0058] Step 203: For each text segment in the above text segment sequence, perform the determination step:

[0059] Step 2031: Based on the above text fragment, perform video segmentation on each candidate video in the above candidate video set to generate a segmented video sequence, thereby obtaining a segmented video sequence set.

[0060] In some embodiments, the execution entity may segment each candidate video in the candidate video set based on the aforementioned text fragment to generate a segmented video sequence, thereby obtaining a set of segmented video sequences. A segmented video may be a sub-video segmented from a candidate video.

[0061] As an example, firstly, the aforementioned execution entity can generate representations to segment candidate videos based on text fragments, obtaining segmentation prompts whose content similarity to the corresponding text fragments is higher than the target similarity. Then, the text fragments, candidate videos, and segmentation prompts are input into a large language model to obtain a segmented video sequence.

[0062] In some optional implementations of certain embodiments, the execution entity may perform video segmentation on each candidate video in the candidate video set based on the aforementioned text fragment to generate a segmented video sequence, including the following steps:

[0063] The first step is to determine the estimated time required to elaborate on the above text fragment. This estimated time can be the actual time needed to explain the text fragment. The content elaboration can be a presentation of the content.

[0064] As an example, first, the number of characters corresponding to the above text fragment is determined. Then, based on the mapping table between the consumption time and the number of characters, the estimated consumption time is determined. The mapping representation between consumption time and the number of characters characterizes the mapping relationship between the consumption time and the number of characters. Generally, the higher the number of characters, the higher the consumption time.

[0065] The second step involves segmenting the candidate videos based on the estimated duration, generating a segmented video sequence. Each segmented video has the same duration as the estimated duration. Furthermore, each adjacent segmented video contains a predetermined proportion of identical video segments. This predetermined proportion can be a pre-determined percentage of identical video segments, such as 50%. Specifically, for segmented video 1 and segmented video 2, which are adjacent videos, the latter half of segmented video 1 is identical to the first half of segmented video 2.

[0066] Step 2032: Determine the video content information corresponding to each segmented video in the above segmented video sequence set.

[0067] In some embodiments, the execution entity may determine the video content information corresponding to each segmented video in the segmented video sequence set. The video content information may be the semantic content of the video expressed in the segmented video. The video content information may be in various forms; for example, it may be in vector form.

[0068] As an example, the aforementioned execution entity can input each segmented video into a video content extraction model to obtain video content information. The video content extraction model can be a model that extracts the semantic content of the video. For example, the video content extraction model could be a large language model.

[0069] Step 2033: Select target segmented videos from the above-mentioned set of segmented video sequences whose semantic matching information meets the target matching conditions.

[0070] In some embodiments, the execution entity can filter target segmented videos from the segmented video sequence set whose semantic matching information satisfies the target matching condition. Here, the semantic matching information refers to the degree of content matching between the video content information and the text segment. The semantic matching information can be in numerical form. A higher semantic matching information indicates a closer semantic similarity between the text segment and the video content information. The target matching condition can be the target segmented video with the highest corresponding semantic matching information. For example, the semantic matching information can be used to determine the similarity of semantic content between the video content information and the text segment using a large language model.

[0071] As an example, firstly, the aforementioned execution entity can determine the content similarity between each segmented video and a text fragment, serving as semantic matching information. Then, it filters out the target segmented video with the highest semantic matching information from the aforementioned set of segmented video sequences.

[0072] In some optional implementations of certain embodiments, the aforementioned video content information is in text form.

[0073] Optionally, the above semantic matching information is generated through the following steps:

[0074] First, the executing entity can extract the first textual feature semantic information corresponding to the video content information and the second textual feature semantic information corresponding to the text segment. The first textual feature semantic information can be in vector form, representing the semantic content corresponding to the video content information. The second textual feature semantic information can also be in vector form, representing the semantic content corresponding to the text segment.

[0075] The second step involves the executing entity determining the semantic similarity between the first and second textual semantic features, using this as the semantic matching information. Here, semantic similarity refers to the similarity of the semantic content. For example, semantic similarity could be cosine similarity.

[0076] Step 204: Based on the obtained target segmented video sequence and the above-mentioned text fragment sequence, generate the narration video corresponding to the above-mentioned target item.

[0077] In some embodiments, the execution entity may generate an explanatory video corresponding to the target item based on the obtained target segmented video sequence and the above-mentioned text fragment sequence.

[0078] As an example, the aforementioned execution entity can directly splice together the individual target segmented videos in the target segmented video sequence to obtain the narration video.

[0079] As another example, the aforementioned execution entity can input the target segmented video sequence and the aforementioned text segment sequence into a video generation model to obtain an narration video. The video generation model can be a model that adjusts the content smoothness of each target segmented video in the target segmented video sequence. For example, the video generation model can be an encoding and decoding model.

[0080] In some optional implementations of certain embodiments, the execution entity can generate an explanatory video corresponding to the target item based on the obtained target segmented video sequence and the above-mentioned text fragment sequence, including the following steps:

[0081] The first step is to smooth the video junctions between every two adjacent target segmented video segments in the above target segmented video sequence to obtain a smoothed segmented video sequence.

[0082] As an example, the aforementioned execution entity can utilize a dissolving transition method to smooth the video junctions between two adjacent target segmented video locations, resulting in a smoothed segmented video sequence. The dissolving transition method can involve the preceding segment gradually disappearing while the following segment gradually appears. Typically, the transition time does not exceed 2 seconds.

[0083] The second step is to stitch the segmented videos from the smoothed video sequence together sequentially to obtain the stitched video.

[0084] The third step is to generate the above-mentioned narration video based on the above-mentioned spliced ​​video and the above-mentioned text fragment sequence.

[0085] As an example, the aforementioned entity can input spliced ​​video and text segment sequences into a video generation model to generate narration videos.

[0086] As yet another example, the aforementioned implementing entity can directly identify the spliced ​​video as the narration video.

[0087] The above embodiments of this disclosure have the following beneficial effects: The video generation methods of some embodiments of this disclosure can accurately and efficiently generate high-quality explanatory videos related to the target item based on the explanatory text. Specifically, the reason why related explanatory videos cannot be generated with high quality and efficiency is that deep learning models usually directly generate explanatory videos based on the relevant item information of the target item. However, in the e-commerce field, it is often necessary to generate explanatory videos based on the explanatory text. Therefore, conventional deep learning models used for video generation are often not applicable to the generation of explanatory videos in e-commerce scenarios. Based on this, the video generation method of some embodiments of this disclosure first obtains the explanatory text and candidate video set corresponding to the target item. Here, the obtained explanatory text serves as the core text content in the subsequent explanatory video and is used for the generation of the explanatory video. The obtained video set serves as the video data basis for the subsequent explanatory video and is used for the generation of the explanatory video. Then, the above explanatory text is preprocessed to obtain a text segment sequence, which divides the explanatory text according to the content of each stage, facilitating subsequent matching of stage-specific videos, allowing the text content of each stage to be semantically matched with the corresponding video. Next, for each text segment in the above text segment sequence, the following determination steps are performed: First, based on the text segment, each candidate video in the above candidate video set is segmented to generate a segmented video sequence set. Here, video segmentation facilitates matching suitable segmented videos to the text segments, resulting in more accurate narration videos. Second, the video content information corresponding to each segmented video in the above segmented video sequence set is determined to facilitate identifying the segmented video that best matches the semantic content of the text segment. Third, target segmented videos whose semantic matching information meets the target matching conditions are selected from the above segmented video sequence set, obtaining the most suitable segmented video corresponding to the above text segment. Here, the semantic matching information refers to the degree of content matching between the video content information and the above text segment. Finally, based on the obtained target segmented video sequence and the above text segment sequence, the narration video corresponding to the above target item can be accurately generated. In summary, by dividing the narration text into stages and determining the most suitable segmented video for each text segment, the segmented video sequence corresponding to the text segment sequence can be accurately obtained. Therefore, an accurate narration video matching the aforementioned narration text can be generated based on the segmented video sequence.

[0088] Further reference Figure 4 The diagram illustrates a flow 400 of another embodiment of the video generation method according to the present disclosure. This video generation method includes the following steps:

[0089] Step 401: Obtain the explanatory text and candidate video set corresponding to the target item.

[0090] Step 402: Perform text preprocessing on the above explanatory text to obtain a sequence of text fragments.

[0091] Step 403: For each text segment in the above text segment sequence, perform the determination step:

[0092] Step 4031: Based on the above text fragment, perform video segmentation on each candidate video in the above candidate video set to generate a segmented video sequence, thereby obtaining a segmented video sequence set.

[0093] Step 4032: Use a pre-trained multimodal large model to generate video description content corresponding to the above-mentioned segmented video.

[0094] In some embodiments, the executing entity (e.g. Figure 1 The electronic device 101 shown can utilize a pre-trained multimodal large model to generate video descriptions corresponding to the segmented videos. The video descriptions can be explanatory content of the segmented videos. For example, the video description could be "the segmented videos explain the functions of the treadmill." Multimodal information can be information from various modalities. For example, each modality can include: video modality, audio modality, and image modality. The multimodal large model can be a large language model whose input and / or output is multimodal information.

[0095] As an example, firstly, content generation prompts are generated to represent the video description content corresponding to the segmented video. Then, the generated content and the segmented video are input into the multimodal large model to obtain the video description content.

[0096] Step 4033: Using the above-mentioned multimodal large model, generate item descriptions related to the target item in the above-mentioned segmented video.

[0097] In some embodiments, the executing entity can utilize the multimodal large model to generate item description content related to the target item in the segmented video. The multimodal large model can be a large language model that supports processing multimodal information. The item description content can be content in the segmented video describing information related to the target item. In practice, for e-commerce scenarios, the target item is the target product. The corresponding item description content can include, but is not limited to, one of the following: the target product details page, or an advertisement.

[0098] As an example, firstly, extraction prompts are generated to represent the extraction of item description information corresponding to the target item from the segmented video. Then, the extraction prompts and the segmented video are input into the multimodal large model to obtain the item description content.

[0099] Step 4034: The above video description content and the above item description content are determined as the above video content information.

[0100] In some embodiments, the aforementioned executing entity may determine the aforementioned video description content and the aforementioned item description content as the aforementioned video content information.

[0101] Step 4035: Select target segmented videos from the above-mentioned set of segmented video sequences whose semantic matching information meets the target matching conditions.

[0102] Step 404: Combine the obtained target segmented video sequences to obtain the narration video corresponding to the above-mentioned target items.

[0103] In some embodiments, the specific implementation of steps 401-402, 4031, 4035, and 404, and their resulting technical effects, can be found in [reference needed]. Figure 2 Steps 201-202, 2031, 2033 and 204 in the corresponding embodiments will not be repeated here.

[0104] from Figure 3 It can be seen from this that, with Figure 2 Compared to the description of some corresponding embodiments, Figure 3 In some corresponding embodiments, the video generation method process 300 utilizes a multimodal large model to fully extract the item description content and video description content related to the target item from the segmented video, so as to obtain diverse feature content, enabling more accurate matching between text fragments and segmented videos in the subsequent implementation.

[0105] Further reference Figure 5 As an implementation of the methods shown in the above figures, this disclosure provides some embodiments of a video generation apparatus, which are similar to... Figure 2 Corresponding to the method embodiments shown, this video generation apparatus can be specifically applied to various electronic devices.

[0106] like Figure 5As shown, a video generation apparatus 500 includes: an acquisition unit 501, a preprocessing unit 502, an execution unit 503, and a video generation unit 504. The acquisition unit 501 is configured to acquire explanatory text corresponding to a target item and a set of candidate videos; the preprocessing unit 502 is configured to perform text preprocessing on the explanatory text to obtain a text segment sequence; the execution unit 503 is configured to perform a determination step for each text segment in the text segment sequence: based on the text segment, segment each candidate video in the candidate video set to generate a segmented video sequence set; determine the video content information corresponding to each segmented video in the segmented video sequence set; and select target segmented videos from the segmented video sequence set whose semantic matching information satisfies the target matching condition, wherein the semantic matching information is the degree of content matching between the video content information and the text segment; the video generation unit 504 is configured to generate an explanatory video corresponding to the target item based on the obtained target segmented video sequence and the text segment sequence.

[0107] In some optional implementations of certain embodiments, the preprocessing unit 502 may be further configured to: divide the explanatory text into segments to obtain an initial text segment sequence; convert each initial text segment in the initial text segment sequence into a text segment in the target language to generate a language conversion segment, thereby obtaining a language conversion segment sequence; and visually enhance each speech conversion segment in the language conversion segment sequence to generate a text segment, thereby obtaining the text segment sequence.

[0108] In some optional implementations of some embodiments, the execution unit 503 may be further configured to: determine the estimated time required to elaborate on the content of the text segment; and segment the candidate video according to the estimated time required to generate a segmented video sequence, wherein the duration of each segmented video is the same as the estimated time required, and there is a predetermined proportion of identical video segments between each pair of adjacent segmented videos.

[0109] In some optional implementations of certain embodiments, the execution unit 503 may be further configured to: generate video description content corresponding to the segmented video using a pre-trained multimodal large model; generate item description content related to the target item in the segmented video using the multimodal large model; and determine the video description content and the item description content as the video content information.

[0110] In some optional implementations of some embodiments, the video generation unit 504 may be further configured to: smooth the video junctions corresponding to every two adjacent target segmented videos in the target segmented video sequence to obtain a smoothed segmented video sequence; sequentially splice the segmented videos in the smoothed segmented video sequence to obtain a spliced ​​video; and generate the narration video based on the spliced ​​video and the text segment sequence.

[0111] It is understandable that the units described in the video generation device 500 are related to the reference. Figure 2 The steps in the described method correspond accordingly. Therefore, the operations, features, and beneficial effects described above for the method also apply to the video generation apparatus 500 and the units contained therein, and will not be repeated here.

[0112] The following is for reference. Figure 6 It illustrates electronic devices suitable for implementing some embodiments of this disclosure (e.g., Figure 1 A schematic diagram of the structure of electronic device 101)600 in the middle. Figure 6 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments of this disclosure.

[0113] like Figure 6 As shown, the electronic device 600 may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory 602 or a program loaded from a storage device 608 into a random access memory 603. The random access memory 603 also stores various programs and data required for the operation of the electronic device 600. The processing unit 601, the read-only memory 602, and the random access memory 603 are interconnected via a bus 604. An input / output interface 605 is also connected to the bus 604.

[0114] Typically, the following devices can be connected to the input / output interface 605: input devices 606 including, for example, a touchscreen, touchpad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, etc.; output devices 607 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 608 including, for example, magnetic tape, hard disk, etc.; and communication devices 609. Communication device 609 allows electronic device 600 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 6 An electronic device 600 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively. Figure 6 Each box shown can represent a device or multiple devices as needed.

[0115] In particular, according to some embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, some embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 609, or installed from a storage device 608, or installed from a read-only memory 602. When the computer program is executed by the processing device 601, it performs the functions defined above in the methods of some embodiments of this disclosure.

[0116] It should be noted that, in some embodiments of this disclosure, the computer-readable medium described above may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In some embodiments of this disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In some embodiments of this disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0117] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol) and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future-developed networks.

[0118] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device. The aforementioned computer-readable medium carries one or more programs that, when executed by the electronic device, cause the electronic device to: acquire explanatory text and a set of candidate videos corresponding to the target item; perform text preprocessing on the explanatory text to obtain a sequence of text fragments; for each text fragment in the text fragment sequence, perform a determination step: based on the text fragment, segment each candidate video in the candidate video set to generate a segmented video sequence, obtaining a set of segmented video sequences; determine the video content information corresponding to each segmented video in the segmented video sequence set; filter out target segmented videos from the segmented video sequence set whose semantic matching information satisfies the target matching condition, wherein the semantic matching information is the degree of content matching between the video content information and the text fragment; and generate an explanatory video corresponding to the target item based on the obtained target segmented video sequence and the aforementioned text fragment sequence.

[0119] Computer program code for performing operations of some embodiments of this disclosure can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0120] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0121] The units described in some embodiments of this disclosure can be implemented in software or hardware. The described units can also be housed in a processor; for example, a processor may be described as including an acquisition unit, a preprocessing unit, an execution unit, and a video generation unit. The names of these units do not necessarily limit the specific unit; for example, an acquisition unit may also be described as "a unit that acquires the explanatory text and candidate video set corresponding to the target item."

[0122] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0123] Some embodiments of this disclosure also provide a computer program product, including a computer program that, when executed by a processor, implements any of the video generation methods described above.

[0124] The above description is merely a selection of preferred embodiments of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in the embodiments of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described inventive concept. For example, technical solutions formed by substituting the above-described features with (but not limited to) technical features with similar functions disclosed in the embodiments of this disclosure.

Claims

1. A video generation method, comprising: Retrieve the explanatory text and candidate video set corresponding to the target item; The explanatory text is preprocessed to obtain a sequence of text fragments; For each text segment in the text segment sequence, perform the determination step: Based on the text fragment, each candidate video in the candidate video set is segmented to generate a segmented video sequence, resulting in a segmented video sequence set; Determine the video content information corresponding to each segmented video in the segmented video sequence set; Target segmented videos whose semantic matching information satisfies the target matching conditions are selected from the segmented video sequence set, wherein the semantic matching information is the degree of content matching between the video content information and the text segment; Based on the obtained target segmented video sequence and the text fragment sequence, an explanatory video corresponding to the target item is generated.

2. The method according to claim 1, wherein, The text preprocessing of the explanatory text to obtain a sequence of text segments includes: The explanatory text is divided into segments to obtain an initial text segment sequence; Each initial text fragment in the initial text fragment sequence is converted into a text fragment in the target language to generate language-converted fragments, resulting in a language-converted fragment sequence; Each speech conversion segment in the language conversion segment sequence is visually enhanced and expanded to generate a text segment, thus obtaining the text segment sequence.

3. The method according to claim 1, wherein, The step of segmenting each candidate video in the candidate video set according to the text fragment to generate a segmented video sequence includes: Determine the estimated time required to elaborate on the content of the text fragment; Based on the estimated consumption time, the candidate video is segmented to generate a segmented video sequence, wherein the duration of each segmented video is the same as the estimated consumption time, and there is a predetermined proportion of identical video segments between every two adjacent segmented videos.

4. The method according to claim 1, wherein, The step of determining the video content information corresponding to each segmented video in the segmented video sequence set includes: Using a pre-trained multimodal large model, video description content corresponding to the segmented video is generated; Using the multimodal large model, generate item descriptions related to the target item in the segmented video; The video description and the item description are identified as the video content information.

5. The method according to claim 1, wherein, The video content information is in text format; and The semantic matching information is generated through the following steps: Extract the first text feature semantic information corresponding to the video content information and the second text feature semantic information corresponding to the text segment; The semantic similarity between the first text feature semantic information and the second text feature semantic information is determined as the semantic matching information.

6. The method according to claim 1, wherein, The step of generating an explanatory video corresponding to the target item based on the obtained target segmented video sequence and the text fragment sequence includes: Smoothing is performed on the video junctions corresponding to every two adjacent target segmented videos in the target segmented video sequence to obtain a smoothed segmented video sequence. The segmented videos in the smoothed segmented video sequence are spliced ​​together sequentially to obtain a spliced ​​video; The narration video is generated based on the spliced ​​video and the text segment sequence.

7. A video generation apparatus, comprising: The acquisition unit is configured to acquire the explanatory text and candidate video set corresponding to the target item; The preprocessing unit is configured to perform text preprocessing on the explanatory text to obtain a sequence of text fragments; The execution unit is configured to perform the following determination steps for each text segment in the text segment sequence: segmenting each candidate video in the candidate video set according to the text segment to generate a segmented video sequence set; determining the video content information corresponding to each segmented video in the segmented video sequence set; and selecting target segmented videos from the segmented video sequence set whose semantic matching information satisfies the target matching condition, wherein the semantic matching information is the degree of content matching between the video content information and the text segment. The video generation unit is configured to generate an explanatory video corresponding to the target item based on the obtained target segmented video sequence and the text fragment sequence.

8. An electronic device, comprising: One or more processors; Storage device, on which one or more programs are stored, When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-6.

9. A computer-readable medium having a computer program stored thereon, wherein, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-6.

10. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-6.

Images