A video generation method, system, device and medium based on a large language model

Abstract

The application discloses a video generation method, system, device and medium based on a large language model. The method obtains product information input by a user, the product information including the name, description and selling point of the product. The product information is preprocessed, and the preprocessed product information is subjected to semantic information split-screen processing through a large language model to obtain split-screen description information corresponding to the product information. An original video is obtained, and the original video is subjected to video segment splitting to generate a plurality of video segments and picture description information corresponding to the video segments. The split-screen description information and the picture description information are subjected to semantic matching processing, and the video segments with the highest similarity to each split-screen description information are matched. The video segments are spliced according to the sequence of the split-screen description information to generate a complete video. Compared with the related art, the application can automatically and efficiently generate a product video by integrating natural language processing, video content analysis and intelligent matching algorithms.

Description

TECHNICAL FIELD

[0001] The application relates to the technical field of video processing, and in particular to a video generation method, system and device based on a large language model and a medium. BACKGROUND

[0002] With the popularity of e-commerce and social media, short videos have become a core tool for brand marketing. However, traditional video production relies on manual script design, shot planning and post-editing, which not only takes a long time and costs a lot, but also requires the producer to have high professional skills. In recent years, automated video generation tools have gradually emerged, trying to improve efficiency through templating or basic editing functions, but these tools still have limitations in handling user personalized needs, understanding complex video content and achieving accurate matching between scripts and pictures. Therefore, a new video generation method is urgently needed. SUMMARY

[0003] The technical problem to be solved by the application is to provide a video generation method, system, device and medium based on a large language model, which integrates advanced natural language processing, video content analysis and intelligent matching algorithms to build an efficient and automated process from user input to finished video, thereby solving the limitations of traditional methods and existing tools.

[0004] To solve the above technical problems, the application provides the following technical solutions:

[0005] In a first aspect, the application provides a video generation method based on a large language model, comprising:

[0006] Obtaining product information input by a user, the product information including the name, description and selling points of a product;

[0007] Preprocessing the product information and performing semantic information shot processing on the preprocessed product information through a preset large language model to obtain shot description information corresponding to the product information;

[0008] Obtaining an original video and performing video segment segmentation on the original video to generate a plurality of video segments and picture description information corresponding to the video segments;

[0009] Performing semantic matching processing on the shot description information and the picture description information to obtain a video segment with the highest similarity to each shot description information;

[0010] Splicing the corresponding video segments according to the sequence of the shot description information to generate a complete video.

[0011] Further, the preprocessing includes semantic enhancement processing, which uses the following calculation formula:

[0012] Vin = V base +∑w i ·V keyi

[0013] wherein, w i is a keyword weight; V keyi is a keyword embedding vector; V base is an original input vector.

[0014] Further, the large language model comprises: a preset shot template, the preset shot template comprising: a product display type template, a function demonstration type template and a promotion guide type template.

[0015] Further, the loss function of the preset shot template is specifically calculated as follows:

[0016] L template = λ·∑(S gen -S temp ) 2

[0017] wherein, S gen is a semantic vector of generated shots, S temp is a template target vector, and λ is a weight coefficient.

[0018] Further, the video segment segmentation of the original video comprises: video segment segmentation using a dynamic threshold segmentation algorithm, and the calculation formula is as follows:

[0019] T = a·mean(ΔF) + β·std(ΔF)

[0020] wherein, ΔF is an inter-frame feature change, and a and β are hyperparameters.

[0021] Further, the semantic matching processing comprises: weighted semantic matching processing and duration matching processing.

[0022] Further, the video generation method further comprises: adding subtitles and / or audio information to the complete video according to a self-defined requirement.

[0023] In a second aspect, the present application also provides a video generation system based on a large language model, comprising:

[0024] An acquisition module is configured to acquire product information input by a user, wherein the product information comprises: a name, a description and a selling point of a product;

[0025] A shot processing module is configured to pre-process the product information, and perform semantic information shot processing on the pre-processed product information through a preset large language model to obtain shot description information corresponding to the product information.

[0026] The video segmentation module is used to acquire the original video, segment the original video into video segments, and generate multiple video segments and corresponding screen description information for each video segment.

[0027] The semantic matching module is used to perform semantic matching processing on the storyboard description information and the scene description information to obtain the video segment with the highest similarity to each storyboard description information.

[0028] The video generation module is used to splice the corresponding video segments according to the chronological order of the storyboard description information to generate a complete video.

[0029] Furthermore, the video generation system also includes a demand processing module, used to add subtitles and / or audio information to the complete video according to customized requirements.

[0030] Thirdly, this application also provides a computer electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the video generation method described in any one of the above.

[0031] Fourthly, this application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the video generation method described in any one of the above descriptions.

[0032] This application provides a video generation method, system, device, and medium based on a large language model, the advantages of which are:

[0033] First, this application uses a large language model to process product information scenes, generating scene descriptions for each scene. Second, it employs a self-developed multimodal large model to automatically segment user-uploaded videos and generate semantic descriptions for each segment. Next, it performs semantic matching between the scene descriptions and the semantic descriptions to obtain the video segments with the highest similarity to each scene description. Finally, it automatically splices the video segments according to the scene order to generate a complete video. Compared to related technologies, this application, by integrating advanced natural language processing, video content analysis, and intelligent matching algorithms, can automatically and efficiently generate product videos. Attached Figure Description

[0034] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0035] Figure 1 This is a flowchart illustrating a video generation method based on a large language model according to an embodiment of this application;

[0036] Figure 2 This is a schematic diagram of the structure of a video generation system based on a large language model according to an embodiment of this application;

[0037] Figure 3 This is a schematic diagram of the structure of a computer electronic device according to an embodiment of this application. Detailed Implementation

[0038] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0039] It should be noted that when an element is said to be "fixed" to another element, it can be directly on the other element or there may be an intervening element. When an element is said to be "connected" to another element, it can be directly connected to the other element or there may be an intervening element. Conversely, when an element is said to be "directly" on another element, there is no intervening element. The terms "vertical," "horizontal," "left," "right," and similar expressions used in this document are for illustrative purposes only.

[0040] In this application, unless otherwise expressly specified and limited, the terms "installation," "connection," "linking," and "fixing," etc., should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components. Those skilled in the art can understand the specific meaning of the above terms in this application according to the specific circumstances.

[0041] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this application, "multiple" means two or more, unless otherwise explicitly specified.

[0042] The terminology used in one or more embodiments of this application is for the purpose of describing particular embodiments only and is not intended to limit the scope of one or more embodiments of this application. The singular forms “a,” “the,” and “the” used in one or more embodiments of this application are also intended to include the plural forms unless the context clearly indicates otherwise.

[0043] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein in the template description is for the purpose of describing particular embodiments only and is not intended to be limiting of this application. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0044] It should be understood that although the terms first, second, etc., may be used to describe various information in one or more embodiments of this application, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first may also be referred to as second without departing from the scope of one or more embodiments of this application, and similarly, second may also be referred to as first. Depending on the context, the word "if" as used herein may be interpreted as "when" or "when".

[0045] Currently, existing video generation methods include:

[0046] 1. Template-based video generation tool: Allows users to select preset templates and fill in content to generate standardized videos.

[0047] 2. Video segmentation and editing software: Provides functions for cutting and splicing video clips and adding basic effects, and is often used for post-processing.

[0048] 3. Semantic analysis tools: These tools extract semantic meaning based on a single modality (such as text or image) and are typically used for content classification or annotation.

[0049] The above method has the following drawbacks:

[0050] 1. Strong template dependence: Existing tools mostly rely on fixed templates and cannot dynamically generate personalized storyboards based on user input, which limits creative flexibility.

[0051] 2. Insufficient content understanding: Video segmentation is often based on fixed time intervals or simple visual features, lacking a deep understanding of the semantics of the images, resulting in inaccurate segment descriptions.

[0052] 3. Low matching efficiency: Due to the lack of a systematic semantic matching mechanism, the generated video content often deviates from the script requirements, requiring users to make manual adjustments.

[0053] The technical solution of this application and how it solves the above-mentioned technical problems will be described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes will not be repeated in some embodiments. The embodiments of this application will be described below with reference to the accompanying drawings.

[0054] Please refer to Figure 1 This application provides a video generation method based on a large language model, applied to a video generation system, which includes at least the following steps:

[0055] S10. Obtain product information input by the user, including: product name, description, and selling points.

[0056] Specifically, users can input product information via keyboard or voice. For example, when generating a promotional video for a product, users can speak the product name, product description, and product selling points to the video generation system.

[0057] S20. The product information is preprocessed, and the preprocessed product information is processed through a preset large language model to obtain the segmentation description information corresponding to the product information.

[0058] Specifically, upon receiving product information, the first step is to preprocess the information. Since the product information is text, preprocessing the text information facilitates subsequent processing.

[0059] In one embodiment, the preprocessing includes semantic enhancement processing, using the following calculation formula:

[0060]

[0061] Among them, w i Keyword weight; V is the keyword embedding vector; base This is the original input vector.

[0062] It is understandable that semantic enhancement can highlight key product information. By weighting keywords in product information for semantic enhancement, it can be ensured that the generated content highlights key information.

[0063] Secondly, after preprocessing the product information, the preprocessed product information needs to be processed through semantic information segmentation using a large language model to obtain the segmentation description information corresponding to the product information.

[0064] It should be noted that the large language model in this example has a built-in storyboard template constraint mechanism, that is, several basic storyboard templates are pre-defined, as follows:

[0065] (1) Product display template: used to highlight the appearance or features of a product, including two parts: "close-up" and "static display", such as "close-up of watch appearance";

[0066] (2) Functional demonstration template: used to showcase product usage scenarios, including two parts: "action footage" and "result presentation", such as "user wearing a watch swimming";

[0067] (3) Promotional guidance template: used to emphasize price or purchase motivation, including two parts: "text label" and "dynamic guidance", such as "limited time offer display".

[0068] It's important to note that template selection is automatically triggered based on the semantic classification of input keywords. For example, "long battery life" triggers "feature demonstration type." The model output is adjusted using the loss function of the storyboard template to closely match the template structure while retaining semantic flexibility. This storyboard template mechanism ensures that the storyboard content is both logical and in line with the marketing rhythm, avoiding the generation of lengthy or disordered descriptions.

[0069] In one embodiment, the loss function of the storyboard template is specifically calculated using the following formula:

[0070] L template =λ·∑(S gen -S temp ) 2

[0071] Among them, S gen To generate the semantic vector of the storyboard, S temp Let λ be the template target vector, and λ be the weight coefficient.

[0072] As an example, when a user inputs "smartwatch, long battery life" → preprocessing generates "watch for daily wear" → outputs a storyboard description "close-up of watch appearance".

[0073] It is understandable that in this embodiment, by adding semantic enhancement preprocessing (extracting keywords and completing the context) and a storyboard template constraint mechanism on the basis of the large language model, the generated content is more in line with the marketing scenario logic and the consistency is improved, thereby reducing the generation time to the second level.

[0074] S30. Obtain the original video, and perform video segmentation on the original video to generate multiple video segments and corresponding screen description information for each video segment.

[0075] Specifically, the system acquires the product's original video data and uses a self-developed multimodal large model to automatically segment the user-uploaded product's original video data, generating image description information for each video segment.

[0076] It should be noted that the self-developed multimodal large model in this embodiment includes: a temporal attention mechanism, which integrates frame sequence context information and uses a dynamic threshold segmentation algorithm for video segmentation, improving segmentation accuracy by approximately 15% and scene description matching by 25%. For example, if the input video is segmented as a "close-up of a watch rotating" (0-3 seconds), the scene description is automatically generated by the multimodal large model.

[0077] In one embodiment of this application, the dynamic threshold segmentation algorithm is calculated using the following formula:

[0078] T=α·mean(ΔF)+β·std(ΔF)

[0079] Where ΔF represents the inter-frame feature variation, and α and β are hyperparameters.

[0080] S40. Perform semantic matching processing on the storyboard description information and the scene description information to obtain the video segment with the highest similarity to each storyboard description information.

[0081] Specifically, the storyboard description information and the image description information of the video clip are semantically matched, and the image description information with the highest similarity to the storyboard description information is selected. The video clip corresponding to the image description information is then used as the video clip of the storyboard description information.

[0082] It is understood that, in this embodiment, the optimal matching video segment can be automatically selected by calculating the semantic similarity between the storyboard description information and the scene description information through the embedding layer (e.g., using cosine similarity for similarity calculation).

[0083] In one embodiment, the semantic matching process includes: weighted semantic matching process and duration matching process.

[0084] Specifically, in this embodiment, based on existing embedding technologies (such as CLIP), a weighted semantic matching mechanism is introduced to assign higher weights to keywords (such as "waterproof") and calculate the adjusted similarity.

[0085] (V d V p The vector is used to describe the weights (W is the weight), and the duration matching degree is also considered. Optimizing the Top-1 selection improved matching accuracy by approximately 10%. For example, the scene "waterproof function" was matched with "swimming scene" (similarity 0.92).

[0086] S50. Based on the chronological order of the storyboard description information, splice the corresponding video segments to generate a complete video.

[0087] Specifically, once all the storyboard description information has been matched with video segments, the corresponding video segments are spliced ​​together according to the order of the storyboard description information to generate a complete video.

[0088] In one embodiment of this application, the video generation method further includes adding subtitles and / or audio information to the complete video according to customized requirements.

[0089] Specifically, after generating complete video information, users can add subtitles and / or audio information to the video according to their actual needs.

[0090] This application provides a video generation method based on a large language model, the advantages of which are:

[0091] First, this application uses a large language model to process product information scenes, generating scene descriptions for each scene. Second, it employs a self-developed multimodal large model to automatically segment user-uploaded videos and generate semantic descriptions for each segment. Next, it performs semantic matching between the scene descriptions and the semantic descriptions to obtain the video segments with the highest similarity to each scene description. Finally, it automatically splices the video segments according to the scene order to generate a complete video. Compared to related technologies, this application, by integrating advanced natural language processing, video content analysis, and intelligent matching algorithms, can automatically and efficiently generate product videos.

[0092] Please see Figure 2 This application also provides a video generation system 200 based on a large language model, comprising:

[0093] The acquisition module 201 is used to acquire product information input by the user, wherein the product information includes: the product name, description and selling points;

[0094] The storyboard processing module 202 is used to preprocess the product information and perform semantic information storyboard processing on the preprocessed product information through a preset large language model to obtain the storyboard description information corresponding to the product information.

[0095] The video segmentation module 203 is used to acquire the original video, segment the original video into video segments, and generate multiple video segments and corresponding screen description information for the video segments;

[0096] Semantic matching module 204 is used to perform semantic matching processing on the storyboard description information and the scene description information to obtain the video segment with the highest similarity to each storyboard description information;

[0097] The video generation module 205 is used to splice the corresponding video segments according to the chronological order of the storyboard description information to generate a complete video.

[0098] In one embodiment of this application, the video generation system 200 further includes a demand processing module 206, used to add subtitles and / or audio information to the complete video according to custom requirements.

[0099] Please see Figure 3 This application also provides a computer electronic device 300, including a memory 303 and a processor 302. The memory 303 stores a computer program, and the processor executes the computer program to implement the steps of the video generation method described above.

[0100] Specifically, the electronic device 300 includes: a transceiver 301, a bus interface, and a processor 302. The processor 302 is used to acquire product information input by the user, including the product name, description, and selling points; preprocess the product information, and perform semantic information segmentation processing on the preprocessed product information through a preset large language model to obtain segmentation description information corresponding to the product information; acquire the original video, and segment the original video into video segments to generate multiple video segments and corresponding scene description information; perform semantic matching processing on the segmentation description information and the scene description information to obtain the video segment with the highest similarity to each segmentation description information; and splice the corresponding video segments according to the order of the segmentation description information to generate a complete video.

[0101] In this embodiment of the application, the electronic device 300 further includes a memory 303. Figure 3 In this context, the bus architecture can include any number of interconnected buses and bridges, specifically linking various circuits together, represented by one or more processors (processor 302) and memory (memory 303). The bus architecture can also link various other circuits such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and therefore will not be described further herein. The bus interface provides an interface. The transceiver 301 can be multiple elements, including transmitters and receivers, providing a unit for communicating with various other devices over a transmission medium. The processor 302 is responsible for managing the bus architecture and general processing, and the memory 303 can store data used by the processor 302 during operation.

[0102] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the video generation method described in any one of the above-described embodiments.

[0103] In this embodiment, the computer-readable storage medium can be a non-volatile storage medium or a volatile storage medium. For example, the computer storage medium may include, but is not limited to, various media capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0104] In all examples shown and described herein, any specific values ​​should be interpreted as merely exemplary and not as limitations; therefore, other examples of exemplary embodiments may have different values.

[0105] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0106] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can also be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings show the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that, in alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, and combinations of blocks in the block diagram and / or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0107] In addition, the functional modules or units in the various embodiments of this application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0108] If the aforementioned functions are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a terminal device (which may be a smartphone, personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application.

[0109] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. A video generation method based on a large language model, characterized in that, include: Obtain product information input by the user, including: product name, description, and selling points; The product information is preprocessed, and the preprocessed product information is then subjected to semantic information segmentation processing through a preset large language model to obtain segmentation description information corresponding to the product information. The large language model includes: preset segmentation templates, which include: product display templates, function demonstration templates, and promotional guidance templates. The preprocessing includes: semantic enhancement processing, using the following calculation formula: ； in, Keyword weight; Embed vectors for keywords; The original input vector; The loss function of the preset storyboard template is calculated using the following formula: ； in, To generate semantic vectors for storyboards, The template target vector, These are the weighting coefficients; The original video is acquired, and video segmentation is performed on the original video to generate multiple video segments and corresponding scene description information. The video segmentation of the original video includes: using a dynamic threshold segmentation algorithm for video segmentation, with the calculation formula as follows: ； in, For inter-frame feature changes; , For hyperparameters; The storyboard description information and the scene description information are semantically matched to obtain the video segment with the highest similarity to each storyboard description information. By splicing the corresponding video clips according to the chronological order of the storyboard descriptions, a complete video is generated.

2. The video generation method according to claim 1, characterized in that, The semantic matching process includes: weighted semantic matching process and duration matching process.

3. The video generation method according to claim 1, characterized in that, The video generation method further includes adding subtitles and / or audio information to the complete video according to customized requirements.

4. A video generation system based on a large language model, characterized in that, include: The acquisition module is used to acquire product information input by the user, wherein the product information includes: the product name, description, and selling points; The storyboard processing module is used to preprocess the product information and perform semantic storyboard processing on the preprocessed product information through a preset large language model to obtain the storyboard description information corresponding to the product information. The large language model includes: preset storyboard templates, which include: product display templates, function demonstration templates, and promotional guidance templates. The preprocessing includes: semantic enhancement processing, using the following calculation formula: ； in, Keyword weight; Embed vectors for keywords; The original input vector; The loss function of the preset storyboard template is calculated using the following formula: ； in, To generate semantic vectors for storyboards, The template target vector, These are the weighting coefficients; The video segmentation module is used to acquire the original video and segment the original video into video segments, generating multiple video segments and corresponding scene description information for each video segment. The segmentation of the original video into video segments includes: using a dynamic threshold segmentation algorithm for video segmentation, with the calculation formula as follows: ； in, For inter-frame feature changes; , For hyperparameters; The semantic matching module is used to perform semantic matching processing on the storyboard description information and the scene description information to obtain the video segment with the highest similarity to each storyboard description information. The video generation module is used to splice the corresponding video segments according to the chronological order of the storyboard description information to generate a complete video.

5. A computer electronic device, characterized in that, The device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps of the video generation method according to any one of claims 1-3.

6. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the video generation method according to any one of claims 1-3.

Images