Video generation method, system and device based on large language model and medium

Abstract

The invention discloses a video generation method, system and device based on a large language model and a medium, and the method comprises the steps: obtaining product information inputted by a user, the product information comprising the name, description and selling point of a product; preprocessing the product information, and carrying out semantic information split processing on the preprocessed product information through a large language model to obtain split description information corresponding to the product information; the method comprises the following steps: acquiring an original video, and performing video clip segmentation on the original video to generate a plurality of video clips and picture description information corresponding to the video clips; semantic matching processing is carried out on the sub-mirror description information and the picture description information, and a video clip with the highest similarity with each piece of sub-mirror description information is obtained through matching; and splicing the video clips according to the sequence of the sub-mirror description information to generate a complete video. Compared with the prior art, the product video can be automatically and efficiently generated by integrating natural language processing, video content analysis and an intelligent matching algorithm.

Description

Technical Field

[0001] The present invention relates to the technical field of video processing, and particularly to a video generation method, system, device, and medium based on a large language model. Background Art

[0002] With the popularity of e-commerce and social media, short videos have become a core tool for brand marketing. However, traditional video production relies on manual script design, storyboard planning, and post-editing, which not only takes a long time and is costly but also requires producers to have high professional skills. In recent years, automated video generation tools have gradually emerged, attempting to improve efficiency through templatization or basic editing functions, but these tools are still insufficient in handling user personalized needs, understanding complex video content, and achieving precise matching between scripts and images. Therefore, there is an urgent need for a new video generation method. Summary of the Invention

[0003] The technical problem to be solved by the present invention is to provide a video generation method, system, device, and medium based on a large language model, which constructs an efficient automated process from user input to finished video by integrating advanced natural language processing, video content analysis, and intelligent matching algorithms, thereby solving the limitations of traditional methods and existing tools.

[0004] To solve the above technical problems, the present application provides the following technical solutions:

[0005] In a first aspect, the present application provides a video generation method based on a large language model, including:

[0006] Obtain product information input by a user, where the product information includes: the name, description, and selling points of the product;

[0007] Preprocess the product information, and perform semantic information storyboarding on the preprocessed product information through a preset large language model to obtain storyboard description information corresponding to the product information;

[0008] Obtain an original video, and perform video segment segmentation on the original video to generate a plurality of video segments and picture description information corresponding to the video segments;

[0009] Perform semantic matching processing on the storyboard description information and the picture description information, and match the video segment with the highest similarity to each storyboard description information;

[0010] Stitch the corresponding video segments according to the order of the storyboard description information to generate a complete video.

[0011] Further, the preprocessing includes: semantic enhancement processing, and the following calculation formula is used:

[0012] Vin = V base + ∑w i ·V keyi

[0013] where w i is the keyword weight; V keyi is the keyword embedding vector; V base is the original input vector.

[0014] Furthermore, the large language model includes: a preset storyboard template, and the preset storyboard template includes: a product display template, a function demonstration template, and a promotion guidance template.

[0015] Furthermore, the loss function of the preset storyboard template has the following specific calculation formula:

[0016] L template = λ·∑(S gen - S temp ) 2

[0017] where S gen is the semantic vector of the generated storyboard, S temp is the template target vector, and λ is the weight coefficient.

[0018] Furthermore, the segmentation of the original video includes: segmenting the video using a dynamic threshold segmentation algorithm, and the calculation formula is as follows:

[0019] T = α·mean(ΔF) + β·std(ΔF)

[0020] where ΔF is the inter-frame feature change, and α and β are hyperparameters.

[0021] Furthermore, the semantic matching process includes: weighted semantic matching processing and duration matching processing.

[0022] Furthermore, the video generation method further includes: adding subtitles and / or audio information to the complete video according to custom requirements.

[0023] In a second aspect, the present application also provides a video generation system based on a large language model, including:

[0024] An acquisition module, configured to acquire product information input by a user, where the product information includes: the name, description, and selling points of the product;

[0025] A storyboard processing module, configured to preprocess the product information and perform semantic information storyboard processing on the preprocessed product information through a preset large language model to obtain storyboard description information corresponding to the product information;

[0026] A video segmentation module, configured to obtain an original video and segment the original video into multiple video segments and generate corresponding scene description information for the video segments;

[0027] A semantic matching module, configured to perform semantic matching processing on the storyboard description information and the scene description information, and match the video segment with the highest similarity to each storyboard description information;

[0028] A video generation module, configured to splice the corresponding video segments according to the order of the storyboard description information to generate a complete video.

[0029] Further, the video generation system further includes: a requirement processing module, configured to add subtitles and / or audio information to the complete video according to custom requirements.

[0030] In a third aspect, the present application further provides a computer electronic device, including a memory and a processor, the memory stores a computer program, and when the processor executes the computer program, the steps of the video generation method described in any one of the above are implemented.

[0031] In a fourth aspect, the present application further provides a computer-readable storage medium, the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the video generation method described in any one of the above are implemented.

[0032] A video generation method, system, device and medium based on a large language model provided by the present application have the beneficial effects that:

[0033] First, the present application generates a storyboard description corresponding to each storyboard through the storyboarding process of the product information by the large language model. Secondly, the self-developed multi-modal large model is used to automatically segment the video uploaded by the user and generate the scene semantics description of each segment. Then, the storyboard description and the scene semantics description are semantically matched to obtain the video segment with the highest similarity to each storyboard description. Finally, according to the order of the storyboards, the video segments are automatically spliced to generate a complete video. Compared with the related technologies, the present application can automatically and efficiently generate product videos by integrating advanced natural language processing, video content analysis and intelligent matching algorithms. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the following drawings are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0035] Figure 1 is a schematic flowchart of a video generation method based on a large language model according to an embodiment of the present application;

[0036] Figure 2 is a schematic structural diagram of a video generation system based on a large language model according to an embodiment of the present application;

[0037] Figure 3 is a schematic structural diagram of a computer electronic device according to an embodiment of the present application. Specific embodiments

[0038] Next, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application.

[0039] It should be noted that when an element is referred to as being "fixed to" another element, it can be directly on the other element or there can also be a central element. When an element is considered to be "connected" to another element, it can be directly connected to the other element or there may be a central element at the same time. On the contrary, when an element is referred to as being "directly on" another element, there is no intermediate element. The terms "vertical", "horizontal", "left", "right" and similar expressions used herein are only for the purpose of illustration.

[0040] In the present application, unless otherwise clearly defined and limited, the terms "installed", "connected", "connected", "fixed" and other terms should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or integrated; it can be a mechanical connection or an electrical connection; it can be directly connected or indirectly connected through an intermediate medium, and it can be the communication inside two elements or the interaction relationship between two elements. For those of ordinary skill in the art, the specific meanings of the above terms in the present application can be understood according to specific circumstances.

[0041] In addition, the terms "first" and "second" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the quantity of the indicated technical features. Thus, the features defined with "first" and "second" may explicitly or implicitly include one or more of such features. In the description of the present application, the meaning of "a plurality" is two or more, unless otherwise clearly specifically defined.

[0042] The terms used in one or more embodiments of the present application are for the purpose of describing specific embodiments only and are not intended to limit one or more embodiments of the present application. The singular forms "a", "the", and "said" used in one or more embodiments of the present application are also intended to include the plural forms unless the context clearly dictates otherwise.

[0043] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terms used in the specification of the template herein are for the purpose of describing specific embodiments only and are not intended to limit this application. The term "and / or" used herein includes any and all combinations of one or more of the related listed items.

[0044] It should be understood that although the terms first, second, etc. may be used in one or more embodiments of the present application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of the present application, the first may also be referred to as the second, and similarly, the second may also be referred to as the first. Depending on the context, the word "if" as used herein may be interpreted as "when" or "while".

[0045] Currently, existing video generation methods include:

[0046] 1. Template-based video generation tools: Allow users to select preset templates and fill in content to generate standardized videos.

[0047] 2. Video segmentation and editing software: Provide functions for cutting, splicing video clips, and adding basic effects, and are commonly used for post-processing.

[0048] 3. Semantic analysis tools: Perform semantic extraction based on a single modality (such as text or image), and are usually used for content classification or annotation.

[0049] The above methods have the following disadvantages:

[0050] 1. Strong template dependence: Existing tools mostly rely on fixed templates and cannot dynamically generate personalized storyboards according to user input, which limits the creative flexibility.

[0051] 2. Insufficient content understanding: Video segmentation is mostly based on fixed time intervals or simple visual features, lacking in-depth understanding of the semantics of the pictures, resulting in inaccurate description of the clips.

[0052] 3. Low matching efficiency: Lack of a systematic semantic matching mechanism, and the generated video content often deviates from the script requirements, and users need to adjust manually.

[0053] The following uses specific embodiments to elaborate in detail on the technical solution of this application and how the technical solution of this application solves the above technical problems. The following several specific embodiments can be combined with each other, and the same or similar concepts or processes will not be repeated in some embodiments. The embodiments of this application will be described below with reference to the accompanying drawings.

[0054] Please refer to Figure 1 , a video generation method based on a large language model provided by an embodiment of this application, which is applied to a video generation system, at least includes the following steps:

[0055] S10. Obtain the product information input by the user, where the product information includes: the name, description, and selling points of the product.

[0056] Specifically, the user can input the product information by keyboard or voice. For example, when a promotional video of a product needs to be generated, the user can, according to the voice direction, tell the video generation system the name of the product, the introduction of the product, and the corresponding selling points of the product, etc.

[0057] S20. Preprocess the product information, and perform semantic information storyboarding processing on the preprocessed product information through a preset large language model to obtain the storyboard description information corresponding to the product information.

[0058] Specifically, when the product information is received, first, a preprocessing operation needs to be performed on the product information. Since the product information is a piece of text information, preprocessing the text information can facilitate subsequent processing.

[0059] In one embodiment, the preprocessing includes: semantic enhancement processing, using the following calculation formula:

[0060]

[0061] where w i is the keyword weight; is the keyword embedding vector; V base is the original input vector.

[0062] It can be understood that semantic enhancement processing can highlight the key information of the product. By performing semantic enhancement processing by weighting the keywords in the product information, it can ensure that the generated content highlights the key information.

[0063] Secondly, after preprocessing the product information, it is necessary to perform semantic information storyboarding processing on the preprocessed product information through a large language model to obtain the storyboard description information corresponding to the product information.

[0064] It should be noted that in the large language model of this example, a storyboard template constraint mechanism is built in, that is, several basic storyboard templates are preset as follows:

[0065] (1) Product display template: used to highlight the appearance or features of the product, including two parts: "close-up shot" and "static display", such as "close-up of the watch appearance";

[0066] (2) Function demonstration template: used to show the product usage scenario, including two parts: "action shot" and "result presentation", such as "the user wears the watch to swim";

[0067] (3) Promotion guidance template: used to emphasize the price or purchase motivation, including two parts: "text annotation" and "dynamic guidance", such as "display of limited-time offers".

[0068] It should be noted that the selection of the template is automatically triggered according to the semantic classification of the input keywords. For example, "long battery life" triggers the "function demonstration type", and the loss function of the storyboard template is used to adjust the model output to make it close to the template structure while retaining semantic flexibility. This storyboard template mechanism ensures that the storyboard content is both logical and in line with the marketing rhythm, avoiding generating long or disordered descriptions.

[0069] In a certain embodiment, the loss function of the storyboard template has the following specific calculation formula:

[0070] L template =λ·∑(S gen -S temp ) 2

[0071] Among them, S gen is the semantic vector for generating the storyboard, S temp is the template target vector, and λ is the weight coefficient.

[0072] As an example, when the user inputs "smart watch, long battery life" → preprocessing generates "the watch is for daily wear" → the output storyboard description is "close-up of the watch appearance".

[0073] It can be understood that in this embodiment, by adding semantic enhancement preprocessing (extracting keywords and complementing the context) and the storyboard template constraint mechanism on the basis of the large language model, the generated content is more in line with the marketing scenario, the logical consistency is improved, and the generation time is shortened to the second level.

[0074] S30. Obtain the original video, and perform video segment segmentation on the original video to generate a plurality of video segments and the corresponding picture description information of the video segments.

[0075] Specifically, obtain the original video data of the product, and use the self-developed multi-modal large model to automatically segment the original video data of the product uploaded by the user, and generate the picture description information of each video segment.

[0076] It should be noted that the self-developed multi-modal large model in this embodiment includes: a temporal attention mechanism (Temporal Attention), which fuses the context information of the frame sequence and uses a dynamic threshold segmentation algorithm for video segmentation, improving the segmentation accuracy by about 15% and the picture description matching degree by 25%. For example, the input video is segmented into "close-up of the watch rotating" (0-3 seconds), and the picture description is automatically generated by the multi-modal large model.

[0077] In a certain embodiment of the present application, the formula of the dynamic threshold segmentation algorithm is as follows:

[0078] T = α·mean(ΔF) + β·std(ΔF)

[0079] Where ΔF is the inter-frame feature change, and α and β are hyperparameters.

[0080] S40. Perform semantic matching processing on the storyboard description information and the picture description information, and match to obtain the video segment with the highest similarity to each storyboard description information.

[0081] Specifically, perform semantic matching processing on the storyboard description information and the picture description information of the video segment, screen out the picture description information with the highest similarity to the storyboard description information, and the video segment corresponding to the picture description information is used as the video segment of the storyboard description information.

[0082] It can be understood that in this embodiment, the semantic similarity between the storyboard description information and the picture description information can be calculated through the embedding layer (for example, using cosine similarity for similarity calculation) to automatically select the optimally matched video segment.

[0083] In a certain embodiment, the semantic matching processing includes: weighted semantic matching processing and duration matching processing.

[0084] Specifically, in this embodiment, on the basis of the existing embedding technology (such as CLIP), a weighted semantic matching mechanism is introduced, a higher weight is given to keywords (such as "waterproof"), and the adjusted similarity is calculated

[0085] (V d 、V p are description vectors, W is the weight), and combined with the duration matching degree Optimize the Top-1 selection, and the matching accuracy is improved by about 10%. For example, the storyboard "waterproof function" is matched to the "swimming scene" (similarity 0.92).

[0086] S50. Concatenate the corresponding video clips in the order of the storyboard description information to generate a complete video.

[0087] Specifically, when all the storyboard description information is matched to video clips, concatenate the corresponding video clips in the order of the storyboard description information to generate a complete video.

[0088] In one embodiment of the present application, the video generation method further includes: adding subtitles and / or audio information to the complete video according to custom requirements.

[0089] Specifically, after generating the complete video information, subtitle information and / or audio information, etc. can also be added to the video according to the actual needs of the user.

[0090] A video generation method based on a large language model provided by the present application has the following beneficial effects:

[0091] First, the present application generates a storyboard description corresponding to each storyboard through the storyboard processing of product information by a large language model. Secondly, a self-developed multi-modal large model is used to automatically segment the video uploaded by the user and generate a semantic description of the picture of each segment. Then, the storyboard description and the semantic description of the picture are semantically matched to obtain the video segment with the highest similarity to each storyboard description. Finally, the video segments are automatically concatenated in the order of the storyboards to generate a complete video. Compared with the related art, the present application can automatically and efficiently generate product videos by integrating advanced natural language processing, video content analysis, and intelligent matching algorithms.

[0092] Please refer to Figure 2 , the present application also provides a video generation system 200 based on a large language model, including:

[0093] An acquisition module 201, configured to acquire product information input by a user, where the product information includes: the name, description, and selling points of the product;

[0094] A storyboard processing module 202, configured to preprocess the product information and perform semantic information storyboard processing on the preprocessed product information through a preset large language model to obtain storyboard description information corresponding to the product information;

[0095] A video segmentation module 203, configured to acquire an original video and segment the original video into multiple video segments and generate picture description information corresponding to the video segments;

[0096] A semantic matching module 204, configured to perform semantic matching processing on the storyboard description information and the picture description information to match and obtain the video segment with the highest similarity to each storyboard description information;

[0097] The video generation module 205 is configured to splice corresponding video segments according to the sequence of the storyboard description information to generate a complete video.

[0098] In an embodiment of the present application, the video generation system 200 further includes: a requirement processing module 206, configured to add subtitles and / or audio information to the complete video according to custom requirements.

[0099] Please refer to Figure 3 , an embodiment of the present application further provides a computer electronic device 300, including a memory 303 and a processor 302, where the memory 303 stores a computer program, and when the processor executes the computer program, the steps of the video generation method described in any one of the above are implemented.

[0100] Specifically, the electronic device 300 includes: a transceiver 301, a bus interface, and a processor 302. The processor 302 is configured to obtain product information input by a user, where the product information includes: the name, description, and selling points of the product; preprocess the product information, and perform semantic information storyboarding processing on the preprocessed product information through a preset large language model to obtain storyboard description information corresponding to the product information; obtain an original video, and perform video segment segmentation on the original video to generate a plurality of video segments and picture description information corresponding to the video segments; perform semantic matching processing on the storyboard description information and the picture description information to match the video segment with the highest similarity to each storyboard description information; splice the corresponding video segments according to the sequence of the storyboard description information to generate a complete video.

[0101] In an embodiment of the present application, the electronic device 300 further includes: a memory 303. In Figure 3 , the bus architecture may include any number of interconnected buses and bridges, specifically, various circuits represented by one or more processors represented by the processor 302 and a memory represented by the memory 303 are linked together. The bus architecture can also link various other circuits such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art, and therefore, they will not be further described herein. The bus interface provides an interface. The transceiver 301 may be a plurality of components, that is, including a transmitter and a receiver, and provides a unit for communicating with various other devices on a transmission medium. The processor 302 is responsible for managing the bus architecture and general processing, and the memory 303 may store data used by the processor 302 when performing operations.

[0102] An embodiment of the present application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the video generation method described in any one of the above.

[0103] In this embodiment, the computer-readable storage medium may be a non-volatile storage medium or a volatile storage medium. For example, the computer storage medium may include, but is not limited to, various media that can store program codes such as USB flash drives, mobile hard disks, read-only memories (ROMs), random access memories (RAMs), magnetic disks, or optical discs.

[0104] In all the examples shown and described here, any specific value should be construed as merely exemplary, not as a limitation. Therefore, other examples of the exemplary embodiments may have different values.

[0105] It should be noted that like reference numerals and letters denote like items in the following figures. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0106] In several embodiments provided by the present application, it should be understood that the disclosed apparatus and method may also be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the figures show the possible architectures, functions, and operations of apparatus, methods, and computer program products according to multiple embodiments of the present application. In this regard, each block in the flowchart or block diagram may represent a module, a program segment, or a part of code, and the module, program segment, or part of code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in alternative implementations, the functions marked in the blocks may occur in a different order than marked in the figures. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, as well as the combinations of blocks in the block diagram and / or flowchart, may be implemented by a dedicated hardware-based system for performing the specified functions or actions, or may be implemented by a combination of dedicated hardware and computer instructions.

[0107] In addition, in each embodiment of the present application, the various functional modules or units may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.

[0108] When the above-mentioned functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to enable a terminal device (which can be a smart phone, a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of this application.

[0109] As mentioned above, the above are only specific implementation manners of this application, but the protection scope of this application is not limited thereto. Any person skilled in the art within the technical scope disclosed by this application can easily think of changes or substitutions, which should all be covered within the protection scope of this application.

Claims

1. A video generation method based on a large language model, characterized in that It includes: Obtain the product information input by the user, where the product information includes: the name, description, and selling points of the product; Preprocess the product information, and perform semantic information storyboarding processing on the preprocessed product information through a preset large language model to obtain the storyboard description information corresponding to the product information; Obtain the original video, and perform video segment segmentation on the original video to generate multiple video segments and the corresponding scene description information of the video segments; Perform semantic matching processing on the storyboard description information and the scene description information, and match the video segment with the highest similarity to each storyboard description information; Splice the corresponding video segments according to the order of the storyboard description information to generate a complete video.

2. The video generation method according to claim 1, wherein The preprocessing includes: semantic enhancement processing, using the following calculation formula: Among them, w i is the keyword weight; is the keyword embedding vector; V base is the original input vector.

3. The video generation method according to claim 1, wherein The large language model includes: a preset storyboard template, and the preset storyboard template includes: a product display type template, a function demonstration type template, and a promotion guidance type template.

4. The video generation method according to claim 3, wherein The loss function of the preset storyboard template is specifically calculated as follows: L template = λ·∑(S gen - S temp ) 2 Among them, S gen is the semantic vector for generating storyboards, S temp is the template target vector, and λ is the weight coefficient.

5. The video generation method according to claim 1, wherein The video segment segmentation of the original video includes: performing video segment segmentation using a dynamic threshold segmentation algorithm, and the calculation formula is as follows: T = α·mean(ΔF)+β·std(ΔF) where ΔF is the inter-frame feature change; α and β are hyperparameters.

6. The video generation method according to claim 1, wherein The semantic matching processing includes: weighted semantic matching processing and duration matching processing.

7. The video generation method according to claim 1, wherein The video generation method further includes: adding subtitles and / or audio information to the complete video according to custom requirements.

8. A video generation system based on a large language model, characterized in that, It includes: An acquisition module, configured to acquire the product information input by the user, where the product information includes: the name, description, and selling points of the product; A storyboarding processing module, configured to preprocess the product information, and perform semantic information storyboarding processing on the preprocessed product information through a preset large language model to obtain the storyboard description information corresponding to the product information; A video segmentation module, configured to acquire the original video, and perform video segment segmentation on the original video to generate multiple video segments and the corresponding scene description information of the video segments; A semantic matching module, configured to perform semantic matching processing on the storyboard description information and the scene description information, and match the video segment with the highest similarity to each storyboard description information; A video generation module, configured to splice the corresponding video segments according to the order of the storyboard description information to generate a complete video.

9. A computer electronic device, characterized in that, It includes a memory and a processor, the memory stores a computer program, and when the processor executes the computer program, the steps of the video generation method according to any one of claims 1-7 are implemented.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the video generation method according to any one of claims 1-7 are implemented.

Images