Marketing commodity short video generation method and device based on AI, medium and product

Abstract

The invention discloses an AI-based marketing commodity short video generation method and device, a medium and a product, and relates to the field of short video generation. The method comprises the following steps: constructing a short video industry knowledge base; obtaining question information input by a user; searching answers from a short video industry knowledge base according to the question information; obtaining a video selected from the answers by the user as a reference short video, and obtaining commodity information uploaded by the user; the commodity information comprises commodity selling points and commodity materials; retrieving a reference copywriting from the short video industry knowledge base according to the commodity selling point, taking the reference copywriting as knowledge supplementary information, generating a script copywriting according to the reference short video and the knowledge supplementary information, and generating picture information according to the script copywriting and the commodity material; and editing the picture information to obtain a short video of the commodity information. The output efficiency of the running amount materials can be improved.

Description

Technical Field

[0001] The present invention relates to the field of short video generation, and in particular to an AI-based short video generation method, device, medium and product for marketing commodities. Background Art

[0002] With the rise of digital media, short videos have become an important channel for brand marketing and personal promotion. Especially driven by social media platforms such as Douyin, Xiaohongshu, Tiktok, etc., the potential and influence of short video marketing are becoming increasingly significant.

[0003] Short video marketing can be generally divided into two major stages: 1) the production of marketing ideas; 2) the creative delivery strategy. The current media advertising platform is based on the iteration of OCPX (Optimized Cost per Action) bidding technology, which has made the delivery strategy tend to be automated and managed. Therefore, the production of marketing ideas has a decisive impact on the ROI (Return on Investment) of marketing delivery.

[0004] However, the production of marketing creative materials faces the problem of difficulty in producing large quantities of materials, which costs brand customers a lot of money. The main problems are: (1) The process is complicated and the production efficiency is low: First, the content strategy direction needs to be determined, and then the director produces a specific execution brief and script. The brief is a creative brief to guide the shooting and editing personnel to shoot video materials and edit content, and then match the screen according to the script, add special effects processing, etc. (2) The demand is large and the inspiration is exhausted: The life cycle of platform short videos is about 1 week. In order to meet the needs of a large amount of creative materials, when hundreds or thousands of creative ideas need to be produced, the director will run out of ideas and words. (3) The quality of the produced materials depends on personal experience, without systematic data support and assistance. Summary of the invention

[0005] The purpose of the present invention is to provide an AI-based marketing commodity short video generation method, device, medium and product, which can improve the output efficiency of running volume materials.

[0006] To achieve the above object, the present invention provides the following solutions:

[0007] A method for generating a short video of a marketing product based on AI, the method comprising:

[0008] Constructing a short video industry knowledge base; wherein the short videos in the short video industry knowledge base are structured data with labels;

[0009] Get the question information entered by the user;

[0010] According to the question information, searching for answers from the short video industry knowledge base; the answers include a plurality of corresponding short videos and content information of each of the short videos;

[0011] Obtaining a video selected by the user from the answers as a reference short video, and obtaining product information uploaded by the user; the product information includes product selling points and product materials;

[0012] Retrieving reference text from the short video industry knowledge base according to the product selling point, and using the reference text as knowledge supplementary information, generating a script text according to the reference short video and the knowledge supplementary information, and generating screen information according to the script text and the product material;

[0013] The screen information is edited to obtain a short video of the product information.

[0014] Optionally, the specific process of building a short video industry knowledge base includes:

[0015] Collect a preset number of short video samples;

[0016] Preprocessing each of the short video samples to obtain a plurality of preprocessed short video samples; the preprocessing operations include storyboard frame extraction, text error correction, and separation of BGM background sound;

[0017] Performing video analysis on each of the preprocessed short video samples to obtain structured data with labels; wherein the video analysis includes overall analysis, storyline analysis, and text and picture label analysis.

[0018] Optionally, preprocessing each of the short video samples to obtain a plurality of preprocessed short video samples specifically includes:

[0019] Applying the PySceneDetect algorithm to each of the short video samples to divide the video into independent shot segments according to shot transitions, and extracting representative frames of the shot segments to obtain representative frame shots;

[0020] Using ffmpeg to separate the speech of each short video sample, and using the ASR algorithm to convert each speech into an initial text;

[0021] Performing OCR recognition on each of the short video samples, and correcting the initial text according to the recognition result to obtain a corrected text;

[0022] The speech sound and BGM of each short video sample are separated to obtain the separated BGM background sound, and the representative frame shot, the error-corrected text and the separated BGM background sound are used as preprocessed short video samples.

[0023] Optionally, each of the pre-processed short video samples is subjected to video labeling to obtain structured data with labels, specifically including:

[0024] The trained Saga multimodal model is applied to perform video analysis on each of the preprocessed short video samples to obtain structured data with labels.

[0025] Optionally, based on the question information, RAG technology is applied to search for multiple corresponding short videos from the short video industry knowledge base and determine the answer to the question information.

[0026] Optionally, according to the question information, RAG technology is applied to search for answers from the short video industry knowledge base, specifically including:

[0027] Each of the corresponding short videos is divided into small blocks of preset length by chunking;

[0028] Apply the BGE model to convert the small blocks into embedding vector form and store them in the knowledge vector database of the ES vector engine;

[0029] Convert the question information into an embedding vector form, and identify the corresponding meta information and index type from the knowledge vector database to obtain the TopN search results;

[0030] Applying the reranker model to the TopN search results to obtain TopK search results;

[0031] Apply LLM to the TopK search results to obtain the answer.

[0032] Optionally, according to the product selling point, a reference copy is retrieved from the short video industry knowledge base, and the reference copy is used as knowledge supplementary information, and a script copy is generated according to the reference short video and the knowledge supplementary information, and screen information is generated according to the script copy and the product material, specifically including:

[0033] Retrieving reference copy from the short video industry knowledge base according to the product selling point;

[0034] Performing style analysis on the reference short video to obtain a style analysis result;

[0035] Generate a script copy according to the reference copy and the style analysis result;

[0036] Preprocessing the commodity material to obtain preprocessed material;

[0037] Retrieving and sorting corresponding materials from the preprocessed materials according to the screen of the reference short video to obtain sorted materials;

[0038] The screen information is generated according to the script text and the sorted materials.

[0039] A computer device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement any of the above-mentioned AI-based short video generation methods for marketing products.

[0040] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the above-mentioned AI-based methods for generating short videos of marketing products.

[0041] A computer program product includes a computer program, which, when executed by a processor, implements any of the above-mentioned AI-based short video generation methods for marketing products.

[0042] According to the specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0043] The present invention discloses a method, device, medium and product for generating short videos of marketing products based on AI. Through automated data collection tools, the collection efficiency and accuracy of short video data are greatly improved, so that video creators can quickly identify and obtain popular videos in the industry, reduce the time investment of creators in finding and analyzing popular videos, and enable them to focus more energy and resources on creativity and content production, thereby improving the density of popular marketing short video data. Through the data analysis and industry trend insight functions of the present invention, in-depth and comprehensive industry analysis reports can be provided to marketers to help them understand market dynamics and consumer behavior. Through customized dialogue robots to collect user needs, industry trend analysis reports that meet specific needs can be generated, providing strong data support for marketing decisions. These insights not only help marketers to formulate more accurate market strategies, but also improve the ROI (return on investment) of marketing activities, achieve optimal allocation of marketing resources, and provide assistance for marketers' industry trend insights and auxiliary decision-making. The use of AI technology to deeply analyze and structure the video content ensures the high accuracy and comprehensiveness of the disassembly process, and provides a solid foundation for video reproduction. AI (Artificial Intelligence)-assisted video reproduction technology not only improves the quality and realism of the reproduced video, but also speeds up the reproduction process, enabling video creators to quickly respond to market trends and launch marketing content that meets user preferences in a timely manner. Therefore, the present invention improves the accuracy and quality of video disassembly and reproduction through AI technology. In summary, the present invention will greatly promote the innovation and development of short video marketing tools, provide video creators and marketers with a more efficient and intelligent working platform, thereby enhancing the competitiveness and market response speed of the entire industry. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0045] Figure 1 A schematic diagram of a process flow of a method for generating a short video of a marketing product based on AI provided in Embodiment 1 of the present invention;

[0046] Figure 2 A schematic diagram of the structure of a short video generation system that applies an AI-based marketing product short video generation method provided in Example 1 of the present invention;

[0047] Figure 3 A schematic diagram of the structure of a video understanding module provided in Example 1 of the present invention;

[0048] Figure 4 A schematic diagram of the Saga multimodal model structure provided in Example 1 of the present invention;

[0049] Figure 5 A schematic diagram of the content strategy module structure provided in Example 1 of the present invention;

[0050] Figure 6 A schematic diagram of the structure of a hot-selling replica module provided in Example 1 of the present invention;

[0051] Figure 7 This is a diagram of the internal structure of a computer device. DETAILED DESCRIPTION

[0052] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0053] The purpose of the present invention is to provide an AI-based marketing commodity short video generation method, device, medium and product, aiming to improve the output efficiency of running volume materials.

[0054] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0055] Example 1

[0056] The AI-based marketing product short video generation method provided by the present invention is as follows: Figure 2 As shown in the figure, the overall process can be roughly divided into the following steps: (1) The collected industry hot-selling videos are converted from unstructured data to structured data with rich tags through the video understanding module, and then stored in the industry knowledge base. (2) When using it, users can first ask a series of questions to the content strategy module to understand the hot content direction of the industry; the system will retrieve relevant videos in the knowledge base, use LLM to summarize, and return answers and video links. (3) The user selects a video as a reference hot-selling product, and uploads product selling points and materials to enter the replication stage. In the replication stage, the system will retrieve excellent copywriting expressions in the knowledge base based on the product selling points as a knowledge supplement to LLM, and imitate the copywriting in the style of the reference hot-selling product; then automatically match the material pictures related to the copywriting. (4) Finally, the user exports the brief / cutting project for post-production fine editing.

[0057] Specifically, Figures 1 to 6As shown, the AI-based marketing product short video generation method in this embodiment includes:

[0058] Step S1: construct a short video industry knowledge base; wherein the short videos in the short video industry knowledge base are structured data with labels.

[0059] S1 specifically includes:

[0060] Step S11: Collect a preset number of short video samples.

[0061] Step S12: pre-processing each of the short video samples to obtain a plurality of pre-processed short video samples; the pre-processing operations include storyboard frame extraction, text error correction and separation of BGM background sound.

[0062] S12 specifically includes:

[0063] Step S121: applying the PySceneDetect (automated video scene analysis and cutting tool) algorithm to each of the short video samples to divide the video into independent shot segments according to shot transitions, and extracting representative frames of the shot segments to obtain representative frame shots.

[0064] Step S122: using ffmpeg (Fast forward moving picture expert group) to separate the speech of each short video sample, and using the ASR algorithm to convert each speech into an initial text.

[0065] Step S123: performing OCR (Optical Character Recognition) recognition on each of the short video samples, and correcting the initial text according to the recognition result to obtain a corrected text.

[0066] Step S124: Separate the speech sound and BGM (Background music) of each short video sample to obtain separated BGM background sound, and use the representative frame shot, the corrected text and the separated BGM background sound as preprocessed short video samples.

[0067] Step S13: Perform video analysis on each of the pre-processed short video samples to obtain structured data with labels; wherein the video analysis includes overall analysis, storyline analysis, and text and picture label analysis.

[0068] The trained Saga multimodal model is applied to perform video analysis on each of the pre-processed short video samples to obtain structured data with labels. Saga is a series of multi-modal large language models (multi-modal large language models) trained by Ultron Lightyear.

[0069] In practical applications, such as Figure 3 and Figure 4 As shown, the industry hit videos collected by the present invention are converted from unstructured data to structured data with rich tags through the video understanding module, and stored in the industry knowledge base. Among them, the data processing flow of the video understanding module is divided into two stages, the first stage is video preprocessing, and the second stage is video labeling.

[0070] like Figure 3 As shown in the figure, the video preprocessing process includes:

[0071] 1) Use the PySceneDetect algorithm to divide the video into independent shot segments according to shot transitions, and extract representative frames from the segments to prepare for image labeling.

[0072] 2) Use ffmpeg to separate the speech, and use the ASR (Automatic Speech Recognition) algorithm to convert the spoken content into text, in preparation for the spoken text tagging. However, because short video marketing contains a lot of professional vocabulary, coupled with interference such as BGM background sound, the general ASR algorithm has a low word preparation rate. In order to improve the accuracy of text recognition, the system will also extract frames of the picture for OCR recognition, and then correct the ASR output.

[0073] 3) Separate the spoken voice and BGM to prepare for timbre recognition and BGM style tagging.

[0074] The video labeling process includes:

[0075] The system uses the Saga multimodal model, such as Figure 4 As shown, a comprehensive multi-perspective understanding is conducted from the overall -> storyboard segments -> images & text. The overall part is analyzed from the dimensions of video structure, audience, psychology, etc. The storyboards will have storyline semantic labels, content summary of each storyboard segment, creative techniques, etc. The images under the storyboards will describe the content of the images, the characters / scenes / props that appear, the shooting techniques of the images, etc. Each spoken copy will extract knowledge through NER (Named Entity Recognition), such as: the pain points described, product ingredients, usage effects, etc. Figure 4In the , CLS is called Classification in English, which means classification algorithm in Chinese. Generally, CLS Tagging is a term, which means to classify and tag sequences using deep learning classification algorithms. Caption Tags is an algorithm that extracts tags from picture descriptions (captions) using deep learning technology. VGGish is an audio classification model developed by Google.

[0076] Saga draws on the idea of ​​BLIP2 (Bootstrapping Language-Image Pre-training with Frozen Image Encoders 2.0, an innovative visual language pre-training model) and obtains a visual language multimodal model in the marketing field through continuous training with marketing corpus. A lightweight Querying Transformer named Q-Former (Querying Transformer, a lightweight Transformer structure) is introduced as a bridge to connect the frozen image encoder and the Large Language Model (LLM). The design purpose of Q-Former is to achieve effective semantic alignment between visual and language models while solving the problem of high training costs.

[0077] The Q-Former structure consists of two Transformer submodules: the image transformer and the text transformer. These two submodules share the same self-attention layer, but the image transformer has an additional cross-attention layer to interact with the frozen image encoder. The image transformer is responsible for extracting the most relevant visual features of the text, while the text transformer acts as both a text encoder and decoder to process and generate text data.

[0078] The training strategy of Q-Former is divided into two stages:

[0079] 1) Visual-language representation learning: Q-Former learns how to extract the most relevant visual representation from the frozen image encoder. The system converts some industry semantic labels and entities of interest in the marketing field into image-text contrast learning (ITC) and image-text matching (ITM) tasks, and converts the picture language into three tasks: image-text generation (ITG), to achieve effective alignment of visual features and text features.

[0080] 2) Generative learning from vision to language: In the second stage, the output of Q-Former is connected to the frozen LLM for generative learning, and pre-trained with the language model loss so that the visual representation learned by Q-Former can be explained by LLM.

[0081] Step S2: Obtain question information input by the user.

[0082] Step S3: according to the question information, searching for answers from the short video industry knowledge base; the answers include multiple corresponding short videos and content information of each short video.

[0083] Specifically, according to the question information, RAG (Retrieval-Augmented Generation) technology is applied to search for multiple corresponding short videos and content information of each short video from the short video industry knowledge base.

[0084] Furthermore, S3 specifically includes:

[0085] Step S31: Each of the corresponding short videos is divided into small blocks of a preset length by chunk, wherein the chunk function can evenly divide the input tensor into a specific number of tensor blocks along a specified dimension, and return a tuple whose elements are tensor blocks.

[0086] Step S32: Apply the BGE (BAAI General Embedding, an open source general vector model developed by Beijing Zhiyuan Artificial Intelligence Research Institute) model to convert the small block into an embedding vector form and store it in the knowledge vector database of the ES (ElasticSearch, a distributed RESTful search and analysis engine) vector engine.

[0087] Step S33: Convert the question information into an embedding vector form, and identify the corresponding meta information and index type from the knowledge vector database to obtain the TopN search results. The meta information contains the original information of various content features of the video (such as video size, source, etc.).

[0088] Step S34: applying the reranker (reranking the candidate document set initially retrieved) model to the TopN search results to obtain TopK (selecting K results with the highest confidence scores for evaluation) search results.

[0089] Step S35: Apply LLM (Large Language Model) to the TopK search results to obtain the answer.

[0090] In actual application, when using it, users can first ask a series of questions to the content strategy module to understand the popular content direction of the industry; the system will retrieve relevant videos in the knowledge base, use LLM to summarize them, and return answers and video links.

[0091] Specifically, Figure 5 As shown, the present invention generally uses RAG technology to go through three major stages from accepting user questions to returning the generated Answer: (1) indexing & retrieval stage; (2) enhancement stage; (3) generation stage. The answer contains links to related videos and a summary of the content. According to the content of the Answer, a related video can be selected as a reference video for subsequent hot-selling reproduction. The summarized content (for example, the people, scenes, pain points, etc. contained therein) can also be used as the input for script imitation, so that the generated copy can better hit the pain points of the people in the scene and improve conversion.

[0092] Among them, (1) the indexing & retrieval stage specifically includes:

[0093] In the indexing stage, the structured data obtained in the video understanding stage is first divided into small chunks of 512 in length through chunks; then, the chunks are converted into embedding vectors through the BGE model and stored in the ES vector engine. In the retrieval stage, the user query is also converted into an embedding vector first, and then the TopN results are returned through retrieval.

[0094] Among them, chunk is more critical, in order to avoid incomplete speech caused by cutting off sentences due to the fixed length of 512. The present invention reorganizes the structured data according to the whole, storyboard-picture, storyboard-text, and stores the industry, brand, category, etc. as meta fields.

[0095] During the retrieval stage, the user query (the retrieval information entered by the user) will first be understood, the meta information and index type will be identified, and the recall will be performed through a hybrid retrieval method (ANN (Artificial Neural Network) vector similarity and BM25 (Best Matching 25, an algorithm used for information retrieval and text mining) keyword matching).

[0096] (2) The enhancement phase specifically includes:

[0097] Although the coverage of relevant doc (Document) recall can be improved through hybrid retrieval methods, there is a problem of incomparable scores between different retrieval methods, and the results must be integrated and standardized.

[0098] Reranker model: for each<query, doc> Perform unified scoring to make the scores standardized and comparable. In addition, by splicing the query and doc together and inputting them into the transformer model, reranker can more fully interact with each other when calculating attention (a mechanism for information interaction in deep learning models), and the scoring will be more accurate.

[0099] MRR algorithm: a duplicate elimination algorithm used to increase the diversity of results. It optimizes document re-ranking and text summary generation by reducing the redundancy of results while maintaining query relevance. The core idea is to select the item that is least similar to the current candidate list in each iteration to increase the diversity of the list. The MRR (Mean Reciprocal Rank, an algorithm that evaluates the performance of a search algorithm) algorithm is a submodule of the reranker. Reranker generally recalculates the scores of the docs under the query and then sorts them. MRR takes duplication control into consideration when sorting. Suppose there are K docs, and the re-scored ranking is (doc1, doc2, doc3, ..., docK), but the content of doc2 is very relevant to doc1. Based on the MRR adjustment, it will be adjusted to the i-th position (no longer ranked second).

[0100] (3) The generation phase specifically includes:

[0101] Different types of user queries will generate answers in different ways. 1) Answer in text form, using the results of the retrieval and enhancement phases, combined with the user's original query, as context input to LLM, and using LLM's summary ability to generate text output; 2) Answer in chart form, when the user's question is about the distribution of tags and the comparison of competitors, it is necessary to first use LLM's reasoning ability, call appropriate statistical tools and generate parameters, and display them with other visualization components (such as echarts, etc.). ECharts is an open source visualization chart library based on JavaScript.

[0102] Step S4: obtaining a video selected by the user from the multiple corresponding short videos as a reference short video, and obtaining product information uploaded by the user; the product information includes product selling points and product materials.

[0103] Step S5: Retrieve reference text from the short video industry knowledge base according to the product selling point, use the reference text as knowledge supplement information, generate script text according to the reference short video and the knowledge supplement information, and generate picture information according to the script text and the product material.

[0104] S5 specifically includes:

[0105] Step S51: Retrieve reference copy from the short video industry knowledge base according to the product selling point.

[0106] Step S52: performing style analysis on the reference short video to obtain a style analysis result.

[0107] Step S53: Generate a script copy based on the reference copy and the style analysis result.

[0108] Step S54: pre-processing the commodity material to obtain pre-processed material.

[0109] Step S55: Retrieve and sort corresponding materials from the preprocessed materials according to the frames of the reference short video to obtain sorted materials.

[0110] Step S56: Generate screen information according to the script text and the sorted materials.

[0111] In actual application, users select a video as a reference for a hit product, upload product selling points and materials, and enter the replication stage. In the replication stage, the system will search for excellent copywriting expressions in the knowledge base based on the product selling points as a knowledge supplement for LLM, and imitate the copywriting in the style of the reference hit product; then automatically match the material images related to the copywriting.

[0112] Specifically, Figure 6 As shown in the figure, the hot replica is mainly a tool to improve the efficiency when making hot replica videos, mainly including:

[0113] (1) One-click replication of reference videos and one-click generation of imitation scripts and matching of video materials.

[0114] 1) Copywriting: First, we analyze the style of the reference video, such as the overall storyline structure, the highlight sentences, rhetoric, and wording used; then, for the input product selling points, we search the knowledge base to obtain richer expressions of the selling points; then, we use LLM, combined with the style analysis of the reference copy and the selling points and expressions of the target product, to plan and execute the copywriting; finally, through reflection, we check the consistency of style and facts, and whether it violates the Advertising Law, etc.; then we make corrections where necessary.

[0115] 2) Material matching: After acquiring the video material, the present invention will automatically understand the material content and segment it, and then intelligently match the best material segment based on the imitation oral copy, saving the user the time of watching, remembering and searching for the material. In the process of material matching, the target product material will be pre-processed. The pre-processing process is as follows: Figure 3As shown, material preprocessing is also video preprocessing.

[0116] According to the popular video, that is, the screen and subtitles of the target video, relevant content is retrieved from the material index of the material preprocessing result, and the retrieved results are sorted to obtain the sorting result.

[0117] This resulted in the imitation text and corresponding pictures.

[0118] (2) After the replication is completed, you can export the Jianying project file, package the imitation script and the matched video materials into it, and open it directly in Jianying to facilitate subsequent fine editing.

[0119] Step S6: Edit the screen information to obtain a short video of the product information.

[0120] In actual applications, users export brief / cutting projects for later fine-tuning.

[0121] In the existing technology, video understanding can be achieved through Google Gemini (Google's artificial intelligence model) and OpenAI GPT-4V (an innovative feature of OpenAI's advanced model GPT-4), but there are two problems: 1) data security; 2) not all tags can be implemented by writing prompts (text instructions used in large language models to prompt the output of model results); and strategic insights can be built through the open source RAG (Retrieval-Augmented Generation) project service, but because there is no optimization for specialized video structured data and query retrieval intent, the accuracy of the answers may not be enough; in addition, the reproduction of popular products can be built through products such as COZE (an intelligent entity building platform launched by ByteDance) to build copywriting tools, but the scale of the selling point knowledge base and automatic matching of materials cannot be achieved.

[0122] The video understanding of the present invention trains the Saga multimodal model to understand the video from the overall video -> storyboard -> picture & text, from coarse to fine granularity; and analyzes and understands at different levels of abstraction from entity -> semantic label -> psychological label -> creative label; strategic insight, based on the RAG technical thinking, optimizes the index construction, query understanding, retrieval enhancement, LLM Tools call and other links for the purpose of strategic insight, thereby improving the accuracy of the answer; the hot replica realizes automatic copywriting and material matching through AI, realizes the rough editing of the hot replica, and greatly improves the efficiency. The RAG+SFT (Supervised Fine-Tun-ing) model is used to improve the consistency of the imitation, and optimizes the factual consistency, advertising law and other issues through post-processing; and the Saga model is used to realize multimodal + cross-modal material matching.

[0123] Therefore, the AI-based marketing product short video generation method provided by the present invention has the following technical effects:

[0124] 1. Improve the automation and efficiency of short video marketing.

[0125] 2. Improve the accuracy and quality of video disassembly and reproduction through AI technology.

[0126] 3. Provide marketers with insights into industry trends and assist in decision making.

[0127] Example 2

[0128] A computer device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the AI-based marketing product short video generation method in Example 1.

[0129] Example 3

[0130] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the AI-based marketing product short video generation method in Example 1.

[0131] Example 4

[0132] A computer program product includes a computer program, which, when executed by a processor, implements the AI-based marketing product short video generation method in Example 1.

[0133] Example 5

[0134] A computer device, which may be a database, may have an internal structure as shown in Figure 7As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store pending transactions. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, the AI-based marketing product short video generation method in Example 1 is implemented.

[0135] It should be noted that the object information (including but not limited to object device information, object personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in the present invention are all information and data authorized by the object or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.

[0136] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided by the present invention can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided by the present invention may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided by the present invention may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0137] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0138] The principles and implementation methods of the present invention are described in this article using specific examples. The description of the above embodiments is only used to help understand the method and core idea of ​​the present invention. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A method for generating short videos of marketing products based on AI, characterized in that: The method comprises: Constructing a short video industry knowledge base; wherein the short videos in the short video industry knowledge base are structured data with labels; Get the question information entered by the user; According to the question information, searching for answers from the short video industry knowledge base; the answers include a plurality of corresponding short videos and content information of each of the short videos; Obtaining a video selected by the user from the answers as a reference short video, and obtaining product information uploaded by the user; the product information includes product selling points and product materials; Retrieving reference text from the short video industry knowledge base according to the product selling point, and using the reference text as knowledge supplementary information, generating a script text according to the reference short video and the knowledge supplementary information, and generating screen information according to the script text and the product material; The screen information is edited to obtain a short video of the product information.

2. The AI-based marketing product short video generation method according to claim 1, characterized in that: The specific process of building a short video industry knowledge base includes: Collect a preset number of short video samples; Preprocessing each of the short video samples to obtain a plurality of preprocessed short video samples; the preprocessing operations include storyboard frame extraction, text error correction, and BGM background sound separation; Performing video analysis on each of the preprocessed short video samples to obtain structured data with labels; wherein the video analysis includes overall analysis, storyline analysis, and text and picture label analysis.

3. The AI-based marketing product short video generation method according to claim 2 is characterized in that: Preprocessing each of the short video samples to obtain a plurality of preprocessed short video samples specifically includes: Applying the PySceneDetect algorithm to each of the short video samples to divide the video into independent shot segments according to shot transitions, and extracting representative frames of the shot segments to obtain representative frame shots; Using ffmpeg to separate the speech of each short video sample, and using the ASR algorithm to convert each speech into an initial text; Performing OCR recognition on each of the short video samples, and correcting the initial text according to the recognition result to obtain a corrected text; The speech sound and BGM of each short video sample are separated to obtain the separated BGM background sound, and the representative frame shot, the error-corrected text and the separated BGM background sound are used as preprocessed short video samples.

4. The AI-based marketing product short video generation method according to claim 2, characterized in that: Perform video labeling on each of the pre-processed short video samples to obtain structured data with labels, specifically including: The trained Saga multimodal model is applied to perform video analysis on each of the preprocessed short video samples to obtain structured data with labels.

5. The AI-based marketing product short video generation method according to claim 1, characterized in that: According to the question information, RAG technology is applied to search for the answer from the short video industry knowledge base.

6. The AI-based marketing product short video generation method according to claim 5, characterized in that: According to the question information, the RAG technology is used to find the answer from the short video industry knowledge base, specifically including: Each of the corresponding short videos is divided into small blocks of preset length by chunking; Apply the BGE model to convert the small blocks into embedding vector form and store them in the knowledge vector database of the ES vector engine; Convert the question information into an embedding vector form, and identify the corresponding meta information and index type from the knowledge vector database to obtain the TopN search results; Applying the reranker model to the TopN search results to obtain TopK search results; Apply LLM to the TopK search results to obtain the answer.

7. The AI-based marketing product short video generation method according to claim 1, characterized in that: Retrieving reference text from the short video industry knowledge base according to the product selling point, and using the reference text as knowledge supplementary information, generating a script text according to the reference short video and the knowledge supplementary information, and generating screen information according to the script text and the product material, specifically including: Retrieving reference copy from the short video industry knowledge base according to the product selling point; Performing style analysis on the reference short video to obtain a style analysis result; Generate a script copy according to the reference copy and the style analysis result; Preprocessing the commodity material to obtain preprocessed material; Retrieving and sorting corresponding materials from the preprocessed materials according to the screen of the reference short video to obtain sorted materials; The screen information is generated according to the script text and the sorted materials.

8. A computer device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the AI-based short video generation method for marketing products as described in any one of claims 1-7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, it implements the AI-based short video generation method for marketing products as described in any one of claims 1-7.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, it implements the AI-based short video generation method for marketing products as described in any one of claims 1-7.

Images