Multi-mode-based video advertisement generation method, system, equipment and medium

Abstract

The invention discloses a multi-mode-based video advertisement generation method, system and device and a medium, and the method comprises the steps: obtaining a video material of a product, and carrying out the preprocessing of the video material; performing video content analysis and video clip scoring on the preprocessed video material data to obtain a corresponding video content analysis result and a video clip score; generating a mixed cutting video corresponding to the product according to the video clip score; according to the analysis result of the video content, the name of the product and the selling point, generating an advertisement copywriting corresponding to the product; forming audio and subtitle data according to the content of the advertisement copywriting; and integrating the mixed video, the audio and the subtitles to generate a video advertisement of the product. According to the method, technologies such as video content analysis, propaganda copywriting automatic generation and an intelligent matching algorithm are fused, and an efficient automatic process from user input to finished video is constructed, so that the problems that traditional video advertisement generation needs a large amount of manual intervention and the generation efficiency is relatively low are solved.

Description

Technical Field

[0001] The present invention belongs to the field of video processing technology, and in particular relates to a method, system, device and medium for generating multimodal video advertisements. Background Art

[0002] With the rapid development of digital marketing, video advertising has become a core vehicle for brand communication. However, the traditional video advertising production process, which typically includes creative planning, scriptwriting, footage shooting, post-production editing, and multi-platform adaptation, presents significant technical bottlenecks.

[0003] Currently, the majority of video ad creation tools on the market rely on fixed templates, standalone speech synthesis software, and digital human avatar generation platforms. Users must use these tools separately for video editing, copywriting, speech synthesis, and subtitle creation, resulting in a fragmented workflow that relies heavily on manual intervention.

[0004] Therefore, a new method for generating video advertisements is urgently needed. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to provide a method, system, device and medium for generating multimodal video advertisements, so as to solve the problem in related technologies that the generation of video advertisements requires a lot of manual intervention and has low generation efficiency.

[0006] In order to solve the above technical problems, this application provides the following technical solutions:

[0007] In a first aspect, the present application provides a method for generating a multimodal video advertisement, comprising:

[0008] Obtaining video material of the product and preprocessing the video material;

[0009] Perform video content analysis and video segment scoring on the pre-processed video material data to obtain the corresponding video content analysis results and video segment scores;

[0010] Generating a mashup video corresponding to the product according to the video clip score;

[0011] Generate advertising copy corresponding to the product based on the analysis results of the video content, the name and selling points of the product;

[0012] Generating audio and subtitle data according to the content of the advertisement copy;

[0013] The mixed video, the audio and the subtitle data are integrated to generate a video advertisement for the product.

[0014] Furthermore, the method further comprises:

[0015] Generate digital avatars and background music;

[0016] The digital virtual character and the background music are embedded into the video advertisement of the product.

[0017] Furthermore, the pre-processed video material data is subjected to video content analysis to obtain the corresponding video content analysis result, using the following calculation formula:

[0018]

[0019] fused_feature=concat(attention(Q img ,K text ,V text ),attention(Q text ,K img ,V img ))

[0020] Among them, Q is the query vector; K is the key vector; V is the value vector; d k is the dimension of the key vector.

[0021] Furthermore, generating an advertisement copy corresponding to the product based on the analysis result of the video content, the name of the product, and the selling point of the product includes:

[0022] Input the analysis results of the video content, the name of the product, and its selling points into the big model to generate advertising copy;

[0023] The advertising copy is optimized according to a weighted voting algorithm mechanism to obtain an optimized advertising copy.

[0024] Furthermore, the optimal advertising copy is selected based on the weighted voting mechanism, using the following calculation formula:

[0025]

[0026] score=w1·score length +w2·score match +w3·score sales

[0027] Among them, w1, w2, w3 are the weights of each agent; score length ,score match ,score sales Score each dimension; script is the advertising copy.

[0028] Furthermore, the step of generating audio and subtitle data according to the content of the advertisement copy includes:

[0029] Using TTS technology to convert the content of the advertising copy into audio data;

[0030] Using a subtitle generation tool on the audio data to generate subtitle data;

[0031] The audio data and subtitle data are synchronized by using a timestamp matching algorithm.

[0032] Furthermore, the synchronization of the audio data and the subtitles using a timestamp matching algorithm includes:

[0033] MFCC(i)=DCT(log(Mel(S i )))

[0034] D(i,j)=d(x i ,y j )+min{D(i-1,j),D(i,j-1),D(i-1,j-1)}

[0035] Among them, S i is the short-time Fourier transform of the audio signal; Mel is the Mel filter bank; DCT is the discrete cosine transform; d(x i ,y j ) is the Euclidean distance between features.

[0036] In a second aspect, the present application further provides a system for generating multimodal video advertisements, comprising:

[0037] An acquisition module is used to acquire video material of a product and pre-process the video material;

[0038] The video analysis module is used to perform video content analysis and video segment scoring on the pre-processed video material data to obtain the corresponding video content analysis results and video segment scores;

[0039] A video editing module, configured to generate a mashup video corresponding to the product according to the video clip scores;

[0040] A copywriting generation module, configured to generate an advertising copy corresponding to the product based on the analysis results of the video content, the name of the product, and its selling points;

[0041] An audio and subtitle generation module, configured to generate audio and subtitle data based on the content of the advertisement copy;

[0042] The video advertisement generating module is used to integrate the mixed video, the audio and the subtitle data to generate a video advertisement for the product.

[0043] In a third aspect, the present application further provides a computer electronic device comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of any one of the above-mentioned methods for generating video advertisements when executing the computer program.

[0044] In a fourth aspect, the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of any one of the above-mentioned methods for generating video advertisements are implemented.

[0045] The present application provides a method, system, device, and medium for generating multimodal video advertisements, which have the following beneficial effects:

[0046] 1. Highly integrated multimodal processing capabilities: By integrating multiple large-scale model tools (such as GPT-4o, GeminiAPI, MoviePy, FFmpeg, TTS, and the local Whisper ASR model), one-stop automated processing of video content analysis, text generation, speech synthesis, and subtitle generation is achieved, significantly reducing manual intervention.

[0047] 2. Precise timestamp matching algorithm: Detailed scoring and segmentation of videos, combined with timestamp information, enables precise matching of voice, subtitles, and images, completely resolving the traditional audio, video, and subtitle synchronization issues.

[0048] 3. Full-link automated workflow design: From material pre-processing to intelligent editing, copywriting generation, and audio and video synthesis, full-process automation significantly improves the efficiency and personalization level of advertising video production, meeting the needs of corporate digital transformation.

[0049] 4. Multimodal data fusion: Video analysis integrates text and audio data extracted by OCR and ASR, and combines them with visual features to improve product recognition and classification accuracy.

[0050] 5. Multi-Agent Copywriting Collaboration: Use agents to optimize copywriting from multiple angles to ensure that the output is both accurate and marketing-appealing. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0052] Figure 1 This is a flow chart of a method for generating a multimodal video advertisement according to an embodiment of the present application;

[0053] Figure 2 This is a structural diagram of a multimodal video advertisement generation system according to an embodiment of the present application;

[0054] Figure 3 It is a structural diagram of a computer electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0055] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0056] It should be noted that when an element is referred to as being "fixed to" another element, it may be directly on the other element or there may be an intermediate element. When an element is considered to be "connected to" another element, it may be directly connected to the other element or there may be an intermediate element. Conversely, when an element is referred to as being "directly on" another element, there is no intermediate element. The terms "vertical," "horizontal," "left," "right," and similar expressions used herein are for illustrative purposes only.

[0057] In this application, unless otherwise expressly specified or limited, terms such as "mounted," "connected," "connect," and "fixed" should be understood in a broad sense. For example, they may refer to fixed connections, detachable connections, or integration; mechanical connections or electrical connections; direct connections or indirect connections through an intermediate medium; and internal connections between two components or interactions between two components. Those skilled in the art will understand the specific meanings of these terms in this application based on specific circumstances.

[0058] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of the technical features being referred to. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. Throughout the description of this application, "plurality" means two or more, unless otherwise specifically defined.

[0059] The terms used in one or more embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of the present application. The singular forms "a", "the", and "the" used in one or more embodiments of the present application are also intended to include plural forms unless the context clearly indicates otherwise.

[0060] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application belongs. The terms used in the template description herein are for the purpose of describing specific embodiments only and are not intended to limit this application. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0061] It should be understood that although the terms first, second, etc. may be used to describe various information in one or more embodiments of the present application, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from each other. For example, without departing from the scope of one or more embodiments of the present application, the first may also be referred to as the second, and similarly, the second may also be referred to as the first. Depending on the context, the word "if" as used herein may be interpreted as "when..." or "when...".

[0062] Currently, existing video ad generation methods generally rely on a library of pre-set ad templates, allowing users to create ad videos by filling in text, images, and other materials. For example, Adobe Premiere Pro's template market allows users to quickly replace materials and generate ads.

[0063] The above method has the following disadvantages:

[0064] 1. Lack of flexibility: Template-based generation tools are difficult to meet highly personalized and diversified marketing needs.

[0065] 2. Audio and video synchronization issues: Voice commentary and video content often need to be manually adjusted, making it impossible to achieve precise synchronization.

[0066] 3. Complex operation: There is a lack of connection between multiple tools, the process is cumbersome, and it poses a high technical threshold for non-professional users.

[0067] 4. Low degree of automation: The overall process still requires a lot of manual intervention, which affects production efficiency and content consistency.

[0068] The following specific embodiments describe in detail the technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems. The following specific embodiments can be combined with each other, and the same or similar concepts or processes in certain embodiments will not be repeated. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0069] Please refer to Figure 1 The embodiment of the present application provides a method for generating a multimodal video advertisement, which includes at least the following steps:

[0070] S10: Obtain video material of the product and pre-process the video material.

[0071] Specifically, in this example, it is necessary to first obtain the video material provided by the user. The video material may include multimodal data such as filmed product videos, various advertising videos, and corresponding landing page screenshots.

[0072] Secondly, after obtaining the video material, it needs to be pre-processed. The pre-processing operation may include:

[0073] Format Conversion: Different video materials may have different formats, such as MP4, AVI, MOV, MKV, etc. To ensure that the material can be used properly in the subsequent video ad generation method, it needs to be converted to a unified format that is well supported by the software. For example, most video editing software has good compatibility with the MP4 format and can convert other formats to MP4.

[0074] Resolution Adjustment: Adjust the resolution of video assets based on the delivery platform and display requirements. For mobile ads, 720P or 1080P is generally sufficient. For outdoor large screens, 4K or even higher resolutions may be required. For assets with excessively high resolutions, the resolution should be appropriately reduced to reduce data volume and processing pressure.

[0075] Color Correction: Video footage from different sources may have color variations. Color correction can unify tones for a more aesthetically pleasing and harmonious image. This includes adjusting parameters like brightness, contrast, saturation, and color temperature to create vibrant, richer colors and layers to capture the viewer's attention. For example, for dark-toned footage, increasing brightness and contrast can enhance the visual impact.

[0076] S20: Perform video content analysis and video segment scoring on the pre-processed video material data to obtain corresponding video content analysis results and video segment scores.

[0077] Specifically, after obtaining the pre-processed video material, it is necessary to perform two processing steps on the video in the video material, namely video content analysis and video segment scoring. It should be noted that video content analysis and video segment scoring can be processed in parallel or serially.

[0078] In this embodiment, in addition to traditional image analysis, we also use optical character recognition (OCR) technology to extract text information from video frames to improve the accuracy of product recognition and classification.

[0079] It can be understood that in this embodiment, multimodal information fusion (image + text) makes the video content analysis more comprehensive, can more accurately reflect product features, and provide richer context for subsequent copywriting generation.

[0080] It should be noted that in this embodiment, first, a convolutional neural network (such as ResNet-50) is used to extract image visual features from video frames. Then, OCR is used to recognize the text content in the video. Finally, the image features and text information are weighted and fused using a multimodal attention mechanism to generate a comprehensive feature vector. The calculation formula for the weighted fusion of image and text features using the multimodal attention mechanism is as follows:

[0081]

[0082] fused_feature=concat(attention(Q img ,K text ,V text ),attention(Q text ,K img ,V img ))

[0083] Among them, Q is the query vector; K is the key vector; V is the value vector; d k is the dimension of the key vector.

[0084] It is understandable that, in order to obtain video advertisements that match the product, in this example, the video clips in the video material are also scored according to a preset scoring rule, wherein the predicted scoring rule may include:

[0085] Content relevance: The video clip content should be closely related to the ad's theme. For example, if the ad is promoting sportswear, then a video clip that includes sports scenes and clothing displays will be highly relevant and will be given a higher score, while a clip unrelated to sports will receive a lower score.

[0086] Completeness: The clip must be relatively complete, with a clear beginning, development, transition, and conclusion, or a clear presentation logic. For example, a product demonstration video clip that fully captures the entire process from opening the product to its completion will receive a higher score. A video clip that only partially captures the usage process will be incomplete and its score will be affected.

[0087] Image quality and clarity: Clear, detailed images are preferred. High-definition video clips without blur or noticeable noise effectively showcase the content and receive high scores. Conversely, blurry, low-resolution, or noisy clips detract from the visual experience and receive lower scores.

[0088] Color expression principle: Video clips with vibrant colors and harmonious tones are more appealing. High scores will be awarded if product colors in advertising videos are accurately reproduced and the color scheme aligns with the brand image and advertising atmosphere. Dull, off-color, or cluttered color schemes will affect the visual effect and result in a lower score.

[0089] Creativity and Uniqueness: Video clips with novel ideas and unique presentation are more engaging. For example, clips that use unique camera angles, innovative storytelling, or distinctive visual effects to present the ad content will receive higher creativity scores. On the other hand, clips that are conventional and lack novelty will receive lower scores.

[0090] The user can select a single scoring rule or combine multiple scoring rules to score, thereby obtaining a video score corresponding to each video clip.

[0091] In this implementation, the Gemini API model can be used for video content analysis and scoring. The Gemini API is a tool developed by Google DeepMind for multimodal applications that provides developers with access to multimodal models.

[0092] S30: Generate a mashup video corresponding to the product according to the video clip scores.

[0093] Specifically, after obtaining the scores of the video clips, you can select videos with high scores for remixing (for example, select the top 5 video clips with the highest scores for remixing) to obtain the remixed video corresponding to the product.

[0094] In this example, you can use the MoviePy tool library to automatically edit and mix videos to generate creative mixed-cut videos.

[0095] S40: Generate an advertisement copy corresponding to the product based on the analysis result of the video content, the name of the product, and the selling point.

[0096] In one embodiment of the present application, the above step S40 includes:

[0097] S41. The analysis results of the video content, the name of the product, and the selling points are input into a large model to generate advertising copy.

[0098] Specifically, in this embodiment, the large model can be an intelligent large model such as GPT or deepseek. For example, advertising copy is generated by multiple Agent systems in GPT-4o.

[0099] S42. Optimize the advertisement copy according to a weighted voting algorithm mechanism to obtain an optimized advertisement copy.

[0100] It can be understood that in order to make the advertising copy more relevant to the product, in this embodiment, the advertising copy is also optimized, specifically using the following formula:

[0101]

[0102] score=w1·score length +w2·score match +w3·score sales

[0103] Among them, w1, w2, w3 are the weights of each agent; score length ,score match ,score sales Score each dimension; script is the advertising copy.

[0104] S50: Generating audio and subtitle data according to the content of the advertisement copy.

[0105] In one embodiment of the present application, the above step S50 includes:

[0106] S51. Using TTS technology, convert the content of the advertisement copy into audio data.

[0107] S52: Using a subtitle generation tool to generate subtitle data for the audio data.

[0108] Specifically, the subtitle generation tool can be the Whisper ASR model. Whisper ASR (Automatic Speech Recognition) is an open source speech recognition model launched by OpenAI in September 2022, which aims to provide high-precision multilingual speech-to-text functions.

[0109] S53 , using a timestamp matching algorithm to synchronize the audio data and the subtitle data.

[0110] Specifically, in order to synchronize the generated audio data and character data, in this embodiment, a timestamp matching algorithm is used for synchronization, as follows:

[0111] 1. Feature extraction:

[0112] 1.1 Extract Mel-frequency cepstral coefficients (MFCC) from video audio and TTS-generated speech to form a feature sequence: X1, X2,\dots, X n .

[0113] MFCC(i)=DCT(log(Mel(S i )))

[0114] Among them, S i is the short-time Fourier transform of the audio signal; Mel is the Mel filter bank; DCT is the discrete cosine transform.

[0115] 1.2. Estimate the duration of each text unit (such as sentence or word) for the subtitle content generated by the copywriting to form a time series: Y1, Y2, \dots, Y m

[0116] 2. Construct a distance matrix:

[0117] Define the distance function d(X i ,Y j ) (such as Euclidean distance or cosine distance), measure the similarity between audio features and estimated time, and calculate the distance matrix: D(i,j)=d(X i ,Y j ).

[0118] 3. Dynamic Time Warping (DTW):

[0119] 3.1 Calculate the cumulative distance matrix:

[0120] D(i,j)=d(x i ,y j )+min{D(i-1,j),D(i,j-1),D(i-1,j-1)}

[0121] Among them, d(x i ,y j ) is the Euclidean distance between features.

[0122] 3.2. Backtrack the optimal path P, which corresponds to the optimal alignment relationship between the video time point and the subtitle time point.

[0123] 4. Timestamp generation:

[0124] According to the path P, the precise start and end time of each subtitle unit is determined to achieve accurate synchronization of audio, video and subtitles.

[0125] S60: Integrate the mixed video, the audio data, and the subtitle data to generate a video advertisement for the product.

[0126] Specifically, the above-mentioned mixed video, audio data and subtitle data are integrated to obtain a video advertisement of the product.

[0127] In an embodiment, the integration can be performed in the following ways:

[0128] Audio and video synthesis: Use MoviePy to automatically synthesize mixed video and speech.

[0129] Post-processing: Compress and optimize the finished video through the FFmpeg model to ensure good playback quality on various platforms.

[0130] In one embodiment of the present application, the method further includes:

[0131] S70: Generate a digital virtual character and background music.

[0132] S80: Embed the digital virtual character and the background music into the video advertisement of the product.

[0133] Specifically, in order to make the product video advertisement more vivid and vivid, in this implementation, digital virtual characters (i.e., digital human images, used to explain the product) and background music can also be added to the video advertisement. For example, an AI-driven avatar generation tool can be used to create a digital human image, which can be integrated into the video through color keying or alpha blending technology to present a professional commentary effect.

[0134] The present application provides a method for generating multimodal video advertisements, which has the following beneficial effects:

[0135] 1. Highly integrated multimodal processing capabilities: By integrating multiple large-scale model tools (such as GPT-4o, GeminiAPI, MoviePy, FFmpeg, TTS, and the local Whisper ASR model), one-stop automated processing of video content analysis, text generation, speech synthesis, and subtitle generation is achieved, significantly reducing manual intervention.

[0136] 2. Precise timestamp matching algorithm: Detailed scoring and segmentation of videos, combined with timestamp information, enables precise matching of voice, subtitles, and images, completely resolving the traditional audio, video, and subtitle synchronization issues.

[0137] 3. Full-link automated workflow design: From material pre-processing to intelligent editing, copywriting generation, and audio and video synthesis, full-process automation significantly improves the efficiency and personalization level of advertising video production, meeting the needs of corporate digital transformation.

[0138] 4. Multimodal data fusion: Video analysis integrates text and audio data extracted by OCR and ASR, and combines them with visual features to improve product recognition and classification accuracy.

[0139] 5. Multi-Agent Copywriting Collaboration: Use agents to optimize copywriting from multiple angles to ensure that the output is both accurate and marketing-appealing.

[0140] See also Figure 2 The embodiment of the present application further provides a multimodal video advertisement generation system 200, comprising:

[0141] An acquisition module 201 is used to acquire video material of a product and pre-process the video material;

[0142] The video analysis module 202 is used to perform video content analysis and video segment scoring on the pre-processed video material data to obtain corresponding video content analysis results and video segment scores;

[0143] The video editing module 203 is used to generate a mashup video corresponding to the product according to the video segment scores;

[0144] A copywriting generation module 204 is configured to generate an advertising copy corresponding to the product based on the analysis results of the video content, the name of the product, and its selling points;

[0145] An audio and subtitle generation module 205 is configured to generate audio and subtitle data based on the content of the advertisement copy;

[0146] The video advertisement generating module 205 is used to integrate the mixed video, the audio and the subtitle data to generate a video advertisement for the product.

[0147] See also Figure 3 The embodiment of the present application further provides a computer electronic device 300, comprising a memory 303 and a processor 302, wherein the memory 303 stores a computer program, and when the processor executes the computer program, the steps of the method for generating a video advertisement described in any one of the above are implemented.

[0148] Specifically, the electronic device 300 includes: a transceiver 301, a bus interface and a processor 302, the processor 302 is used to obtain video materials of a product and pre-process the video materials; perform video content analysis and video clip scoring on the pre-processed video material data to obtain corresponding video content analysis results and video clip scores; generate a mashup video corresponding to the product based on the video clip scores; generate advertising copy corresponding to the product based on the analysis results of the video content, the name and selling points of the product; form audio and subtitle data based on the content of the advertising copy; integrate the mashup video, the audio and subtitle data to generate a video advertisement for the product.

[0149] In the embodiment of the present application, the electronic device 300 further includes: a memory 303. Figure 3 In the embodiment, the bus architecture may include any number of interconnected buses and bridges, specifically linking various circuits such as one or more processors represented by processor 302 and memory represented by memory 303. The bus architecture may also link various other circuits such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and therefore will not be described further herein. The bus interface provides an interface. The transceiver 301 may be a plurality of components, i.e., a transmitter and a receiver, providing a unit for communicating with various other devices over a transmission medium. The processor 302 is responsible for managing the bus architecture and general processing, and the memory 303 may store data used by the processor 302 when performing operations.

[0150] An embodiment of the present application further provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the steps of any one of the above-mentioned methods for generating video advertisements are implemented.

[0151] In this embodiment, the computer-readable storage medium may be a non-volatile storage medium or a volatile storage medium. For example, the computer storage medium may include, but is not limited to, a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, among other media capable of storing program code.

[0152] In all examples shown and described herein, any specific values should be interpreted as merely exemplary and not limiting, and thus other examples of the exemplary embodiments may have different values.

[0153] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, it does not need to be further defined or explained in subsequent drawings.

[0154] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and structure diagrams in the accompanying drawings show the possible architectures, functions and operations of the devices, methods and computer program products according to the multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of the code, and the module, program segment or a part of the code contains one or more executable instructions for implementing the specified logical functions. It should also be noted that in an alternative implementation, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the structure diagram and / or flowchart, and the combination of boxes in the structure diagram and / or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or action, or can be implemented using a combination of dedicated hardware and computer instructions.

[0155] In addition, the functional modules or units in the various embodiments of the present application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0156] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a terminal device (which can be a smart phone, personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application.

[0157] The above is only a specific implementation method of the present application, but the scope of protection of the present application is not limited thereto. Any technician familiar with this technical field can easily think of changes or replacements within the technical scope disclosed in this application, which should be covered by the scope of protection of the present application.

Claims

1. A method for generating a multimodal video advertisement, characterized in that: include: Obtaining video material of the product and preprocessing the video material; Perform video content analysis and video segment scoring on the pre-processed video material data to obtain the corresponding video content analysis results and video segment scores; Generating a mashup video corresponding to the product according to the video clip score; Generate advertising copy corresponding to the product based on the analysis results of the video content, the name and selling points of the product; Generating audio and subtitle data according to the content of the advertisement copy; The mixed video, the audio and the subtitle data are integrated to generate a video advertisement for the product.

2. The method for generating a video advertisement according to claim 1, wherein: The method further comprises: Generate digital avatars and background music; The digital virtual character and the background music are embedded into the video advertisement of the product.

3. The method for generating a video advertisement according to claim 1, wherein: The video content analysis is performed on the pre-processed video material data to obtain the analysis result of the corresponding video content, using the following calculation formula: fused_fearture=concat(attention(Q img ,K text ,V text ),attention(Q text ,K img ,V img )) Among them, Q is the query vector; K is the key vector; V is the value vector; d k is the dimension of the key vector.

4. The method for generating a video advertisement according to claim 1, wherein: Generating an advertisement copy corresponding to the product based on the analysis result of the video content, the name of the product, and the selling point of the product includes: Input the analysis results of the video content, the name of the product, and its selling points into the big model to generate advertising copy; The advertising copy is optimized according to a weighted voting algorithm mechanism to obtain an optimized advertising copy.

5. The method for generating a video advertisement according to claim 4, wherein: The advertising copy is optimized according to the weighted voting algorithm mechanism to obtain the optimized advertising copy, using the following calculation formula: score=w1·score length +w2·score match +w3·score sales Among them, w1, w2, w3 are the weights of each agent; score length ,score match ,score sales Score each dimension; script is the advertising copy.

6. The method for generating a video advertisement according to claim 1, wherein: The step of generating audio and subtitle data according to the content of the advertisement copy includes: Using TTS technology to convert the content of the advertising copy into audio data; Using a subtitle generation tool on the audio data to generate subtitle data; The audio data and subtitle data are synchronized by using a timestamp matching algorithm.

7. The method for generating a video advertisement according to claim 5, wherein: The step of using a timestamp matching algorithm for the audio data and the subtitles to synchronize the audio data and the subtitles includes: MFCC(i)=DCT(log(Mel(S i ))) D(i,j)=d(x i ,y j )+min{D(i-1,j),D(i,j-1),D(i-1,j-1)} Among them, S i is the short-time Fourier transform of the audio signal; Mel is the Mel filter bank; DCT is the discrete cosine transform; d(x i ,y j ) is the Euclidean distance between features.

8. A system for generating multimodal video advertisements, characterized in that: include: An acquisition module is used to acquire video material of a product and pre-process the video material; The video analysis module is used to perform video content analysis and video segment scoring on the pre-processed video material data to obtain the corresponding video content analysis results and video segment scores; A video editing module, configured to generate a mashup video corresponding to the product according to the video clip scores; A copywriting generation module, configured to generate an advertising copy corresponding to the product based on the analysis results of the video content, the name of the product, and its selling points; An audio and subtitle generation module, configured to generate audio and subtitle data based on the content of the advertisement copy; The video advertisement generating module is used to integrate the mixed video, the audio and the subtitle data to generate a video advertisement for the product.

9. A computer electronic device, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the method for generating a video advertisement according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method for generating a video advertisement according to any one of claims 1 to 7 are implemented.

Images