Method and device for generating video from commodity page based on AI, equipment and medium

Abstract

The invention provides a method, a device and equipment for generating a video from a commodity page based on AI, and a medium. The method comprises the following steps: acquiring commodity information from a set commodity page of an e-commerce platform; generating a picture description text and a video description text; inputting the oral broadcast prompt words, the text data in the commodity information, the picture description text and the video description text into a multi-modal model to generate an oral broadcast text; inputting the time length of the total video, the scene number and the oral broadcast text into a multi-modal model, performing scene segmentation on the oral broadcast text to obtain at least one scene text, and generating a corresponding audio file for each scene text; and the video file is input into the video compiler to generate the required video, so that a user can conveniently and quickly generate the required small video, and the cost of the user is reduced.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a method, device, equipment and medium for generating videos from product pages based on AI. Background Art

[0002] In recent years, with the rapid development of computer vision and the introduction of generative adversarial networks, the research on image generation has received more and more attention, which has a very positive significance in the accumulation of materials and automatic generation of data sets. Compared with images, videos are more vivid and more difficult to generate, so the exploration of video generation is more meaningful.

[0003] At present, there is no systematic process for generating short videos from graphic information. The operation process is very cumbersome and lacks intelligent processing methods. Currently, short video platforms are very popular and have high traffic. Online sellers hope to shoot their products into beautiful videos and then automatically publish them on short video platforms to increase product exposure and sales. Therefore, there is an urgent need for a video creation tool that is easy to use and suitable for non-professionals. Summary of the invention

[0004] The technical problem to be solved by the present invention is to provide a method, device, equipment and medium for generating videos from product pages based on AI, so as to facilitate users to quickly generate the required short videos and reduce users' costs.

[0005] In a first aspect, the present invention provides a method for generating a video from a product page based on AI, comprising the following steps:

[0006] Step 1: Get product information from the product page set on the e-commerce platform;

[0007] Step 2: convert the pictures in the product information into picture description texts through a multimodal model, where the picture description texts correspond to the pictures one by one;

[0008] Step 3: disassemble the product video in the product information, perform scene switching detection on the product video, obtain at least one scene and a corresponding time code, cut the product video according to the time code, and obtain at least one segmented video; extract one frame from each segmented video by frame extraction to obtain a video graph, and input it into the multimodal model to convert it into a video description text, which corresponds one-to-one to the segmented video;

[0009] Step 4: Set the spoken prompt as required, input the spoken prompt, the text data in the product information, the picture description text and the video description text into the multimodal model to generate the spoken text; input the total video duration, the number of scenes and the spoken text into the multimodal model, segment the spoken text into scenes, obtain at least one scene text, and generate a corresponding audio file for each scene text;

[0010] Step 5: Match scene text, audio, scene, and segmented video through a multimodal model, and finally generate a video file in a set format: input the video file into a video compiler to generate the required video.

[0011] In a second aspect, the present invention provides a device for generating a video from a product page based on AI, comprising:

[0012] The information acquisition module obtains product information from the product page set on the e-commerce platform;

[0013] A picture description module, which converts the pictures in the product information into picture description texts through a multimodal model, and the picture description texts correspond to the pictures one by one;

[0014] The video description word module disassembles the product video in the product information, performs scene switching detection on the product video to obtain at least one scene and a corresponding time code, and cuts the product video according to the time code to obtain at least one segmented video; extracts one frame from each segmented video by frame extraction to obtain a video graph, and inputs it into the multimodal model to convert it into a video description text, which corresponds one-to-one to the segmented video;

[0015] The audio generation module sets the spoken prompts according to the requirements, inputs the spoken prompts, the text data in the product information, the picture description text and the video description text into the multimodal model, and generates the spoken text; inputs the total video duration, the number of scenes and the spoken text into the multimodal model, divides the spoken text into scenes, obtains at least one scene text, and generates a corresponding audio file for each scene text;

[0016] The video generation module matches scene text, audio, scene and segmented video through a multimodal model, and finally generates a video file in a set format: the video file is input into the video compiler to generate the required video.

[0017] In a third aspect, the present invention provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method described in the first aspect when executing the program.

[0018] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in the first aspect.

[0019] One or more technical solutions provided by the present invention have at least the following technical effects or advantages:

[0020] The present invention combines with a multimodal model to quickly generate a corresponding short video from the required product page, and then publishes the short video to a short video platform, so that merchants can promote their products. In addition, the present invention marks the text according to the settings and specific scenarios, and then enables the voice model to generate voices of various speaking speeds, volumes, and emotions, greatly improving the voice quality, so that it can be used for promotion and is infectious. Users can use it very conveniently, and its generation efficiency is fast, which greatly reduces the user's time cost.

[0021] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented according to the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The present invention will be further described below in conjunction with embodiments with reference to the accompanying drawings.

[0023] Figure 1 is a flow chart of the method in Embodiment 1 of the present invention;

[0024] Figure 2 It is a schematic diagram of the structure of the device in the second embodiment of the present invention. DETAILED DESCRIPTION

[0025] The technical solution in the embodiment of the present application has the following general idea:

[0026] 1. Collect information and materials

[0027] Obtain product page information through crawler technology;

[0028] This embodiment uses Amazon as an example. The key information contained in Amazon's product page (Listing page) is as follows:

[0029] Product title, product price, product description, including text and pictures; product highlights (About this item); product main picture and main video; pictures of other products related to the product;

[0030] After collection, it is necessary to cache to increase the generation speed;

[0031] 2. Understand the material

[0032] Technical issues:

[0033] The pictures on the product pages are usually large, high-definition pictures, and videos take up a lot of space;

[0034] Frequent use of AI models (multimodal) results in high costs (charged based on token length);

[0035] Most models do not yet understand videos, or even images, and a general solution is needed;

[0036] Image Processing

[0037] First, reduce the image size through image compression algorithm, and then use AI model (multimodal) to understand the image content and convert it into descriptive text to solve the above problems.

[0038] Generate description text from pictures;

[0039] Video Processing

[0040] The videos on the product page may be relatively long (duration). Short videos are generally less than 1 minute, so the videos need to be disassembled to meet the duration requirement and solve the volume problem.

[0041] The processing steps are as follows:

[0042] Use PySceneDetect for video scene switch detection. PySceneDetect is an open source tool for video scene switch detection. It mainly identifies scene switches by analyzing changes between video frames.

[0043] The output result after detection by the tool is a time code list, such as:

[0044] 00:00:00-00:01:00 Scene 1

[0045] 00:01:00-00:05:00 Scene 2

[0046] You also need to use the FFmpeg tool to cut according to the time point, so that you can get several 1-5 second video clips. Some clips are only 1 second long. Such clips cannot express the content and are not suitable as materials, so you need to clean up the 1 second video;

[0047] The remaining video clips are extracted by the AI ​​model (multimodal) to understand; the video generates description text;

[0048] 3. Generate video speech

[0049] Use AI models (such as GPT) to generate the entire spoken content; input:

[0050] Collected product data: title, unit price, description, highlights;

[0051] Description of product materials;

[0052] The style and samples that need to be expressed can be extracted from popular short videos on short video platforms using ASR technology;

[0053] Tips:

[0054] Here is the product info:

[0055] 1.Title:{product_title}

[0056] 2.About this item{about_item_str}

[0057] 3.Product Description{product_description_str}

[0058] 4.Prices:{product_price}

[0059] Here are product pictures / videos list,the list include assetidentifier,asset type and asset description:{assets_str}

[0060] Now,please product a script for the video.below is the requirement:

[0061] 1.You should use the theme:{theme_prompt}

[0062] 2.Don't use hashtag in return script full content

[0063] 3.Don't mention the price in final

[0064] Normally, the script will be split into some paragraphs, and each paragraph will use some assets. So it is very important for the script to match the asset.

[0065] If the asset is not enough, make sure to add more information to the script to fit the asset. So just return me JSON data without any other tips or notes.

[0066] So just return me JSON data without any other tips or notes.

[0067] Here is the JSON structure:

[0068] {

[0069] "theme":"{theme}",

[0070] "content":{Full Script content text},

[0071] "title":{Describe the script style}

[0072] }

[0073] The "title" is to describe the script style. DON'T MENTION THE SCRIPT CONTENT. Limit to 30 words.

[0074] The English translation of the above is:

[0075] The following is the product information:

[0076] 1. Title: {product_Title}

[0077] 2. About this project {About_item_str}

[0078] 3. Product description {Product_Description_str}

[0079] 4. Price: {product_price}

[0080] Below is a list of product images / videos, including asset identifier, asset type, and asset description: {assets_str}

[0081] Now, please create a script for your video. Here are the requirements:

[0082] 1. You should use this theme: {theme_prompt}.

[0083] 2. Do not use hashtags (#) in the returned script's full content.

[0084] 3. Don’t mention the price in the closing paragraph.

[0085] Typically, a script is split into sections, each of which uses a number of assets, so it is important that the script matches the assets.

[0086] If the resource is not enough, make sure to add more information to the resource that is suitable for the script. So you only return the json data without any other hints or comments.

[0087] Below is the json structure:

[0088] {

[0089] "theme": "{theme}",

[0090] "content": {full script content text},

[0091] "title": {description of the script style}

[0092] }

[0093] The "Title" should describe the style of the script. Do not mention the content of the script. Limit it to 30 words.

[0094] Output: spoken text of the entire video;

[0095] 4. Generate scene broadcast

[0096] Use AI models (such as GPT) to generate spoken content for each scene;

[0097] Input: duration and number of scenes of the entire video;

[0098] Tips:

[0099] According to the return content (Full Script content text), split it into {sence_count} paragraphs (also called scenes). Any paragraph needs to follow these rules:

[0100] 1. The caption is picked from the content, and it should be unique, not allowing duplicates. Don't use wrapped words.

[0101] 2. The caption should be at least two sentences, including complete meaning and purpose.

[0102] 3. Attention! The caption should only include the spoken words, not including any other words.

[0103] 4. Describe the scene and its purpose.

[0104] 5. The scene_id starts from 30000 and increases by 1.

[0105] 6. Consider the number of spoken words in the caption, not too long and not too short. The return needs to be structured as JSON data, like:

[0106]

[0107] Among the scenes (also called paragraphs) property array object, there should be {sence_count} in length.

[0108] The English translation of the above is:

[0109] The returned content (full script content text) is split into {sence_count} paragraphs (also called scenes). Any paragraph needs to follow the following roles:

[0110] 1. The title is selected from the content and should be unique. No duplication is allowed and no line breaks are used.

[0111] 2. The title should be some sentence of at least 2 sentences which contains the complete meaning and purpose.

[0112] 3. Note! The title should only include the spoken words and not any other words

[0113] 4. Describe the scenario and its purpose

[0114] 5.scene_id starts from 30000 and increases by 1

[0115] 6. Pay attention to the number of colloquial words in the title, not too long or too short. return needs to construct json data, for example:

[0116] {

[0117] "scenes": [{

[0118] "caption": string,

[0119] "desc": string,

[0120] "scene_id": number

[0121] }]}

[0122] In a scene (also known as paragraph) property array object, should be {sence_count} length.

[0123] Output:

[0124] The spoken word for each scene;

[0125] 5. Match the scene’s materials

[0126] Based on the previously generated material description and the spoken scene, AI models (such as GPT) are used to match the materials and finally generate a video file in JSON format. The video file is used to know the video editor to generate the video:

[0127] {

[0128] "backgroundMusic":"https: / / xxx / xxx.mp3", / / background music

[0129] "needAvatar":true, / / Do you need a digital person?

[0130] "scenes":[ / / Scenes

[0131] {

[0132] "scripts":[ / / subtitles

[0133] {

[0134] "text":"Introducing the ultimate tote bag for every occasion!"

[0135] }

[0136] ],

[0137] "assets":[ / / materials

[0138] {

[0139] "url":"https: / / xxx / xxx / xxx / xxx.jpg",

[0140] "type":"image",

[0141] "width":1245,

[0142] "height":1500,

[0143] "length":null

[0144] }

[0145] ],

[0146] "avatar":{

[0147] "video":{ / / Digital human video

[0148] "url":"https: / / xxx / xxx / xxx.webm",

[0149] "position":"left-bottom", / / Position: top left, top right, bottom left, bottom right, full screen

[0150] "width":1350,

[0151] "height":1080,

[0152] "length":3745

[0153] },

[0154] "audio":{ / / Digital human audio (voiceover mp3)

[0155] "url":"https: / / xxx / xxx / xxx.mp3",

[0156] "length":3768

[0157] }}}]};

[0158] Input the JSON of the video into the Remotion video compiler to generate a video file.

[0159] The specific method of generating audio above is as follows:

[0160] For example: When the text content is: Hey TikTok fam!You won't believe the amazing product I just discovered!It's a total game-changer for anyone who loves online shopping!

[0161] I was absolutely blown away when I found out about this deal!Imagine getting top-quality items at such an incredible price—it's almost too good to be true!

[0162] This product not only offers exceptional value but also stands out for its durability and style. Perfect for enhancing your daily life and making everything just a bit more fabulous. Trust me, you don't want to miss out on this one!

[0163] Translation: Hey, TikTok fam! You won't believe the amazing product I just discovered! It's a total game-changer for anyone who loves online shopping!

[0164] When I found this deal, I was totally blown away! Imagine getting top quality merchandise at such an incredible price – it’s almost too good to be true!

[0165] Not only does this product offer exceptional value, it also stands out for its durability and style. Perfect for elevating your everyday life and making everything a little more exciting. Trust me, you definitely don't want to miss this one!

[0166] The above are the original texts of 3 paragraphs. The original texts need to be converted into spoken texts for playback. The above three paragraphs can correspond to three different scenes or the same scene.

[0167] 1. Generate a spoken script

[0168] The input scene image is understood through the multimodal model; the material in the scene, that is, a frame of the picture or video, is uploaded to the multimodal model, and prompts are configured to allow the model to generate a spoken script;

[0169] The key code for calling the multimodal model (GPT-4O model):

[0170]

[0171]

[0172] Where <% content%> is:

[0173] Hey TikTok fam! You won't believe the amazing product Ijustdiscovered! It's a total game-changer for anyone who loves online shopping!

[0174] Suppose you are an internet celebrity with a large number of fans, and now you want to introduce a product to everyone. Please design how to give the above speech. You hope that the whole process has different emotions, tones, speaking speeds and pauses, but you must ensure that the overall rhythm is coherent, without any discontinuous feeling.

[0175] Notice:

[0176] 1. You must pause after each paragraph and mark it with: <% stop: 1%>;

[0177] 2. In order to emphasize the tone, a pause is also required after certain words. You can put a mark after the word, and the pause time can be different. The mark is: <% stop: x%>, where x is in seconds and supports a range of 0.01-1.00;

[0178] 3. The emotion of each paragraph will change. Put a mark before each paragraph to indicate which emotion should be used in this paragraph: <% emotion: y%>, y can be replaced by happy, sad, angry, fearful, disgusted, surprised, neutral;

[0179] 4. The speaking speed of each sentence can be changed. Put a mark before each sentence, marked as <%speed:z%>, where z is the speaking speed, which can be between 0.5-1.3;

[0180] 5. The volume of each sentence can be changed. It is indicated before each sentence as <% vol:v %>, where v is the volume and can be between 0.5-2.

[0181] The output of the multimodal model is:

[0182] <%emotion:happy%><%speed:1.0%><%vol:1.5%>Hey TikTok fam! <%stop:0.5%><%vol:1%>You won't believe<%stop:0.2%>the amazing product<%stop:0.3%><%vol:2%>I just discovered! <%stop:0.5%><%vol:1%>It's a totalgame-changer<%stop:0.2%>for anyone<%stop:0.1%>who loves online shopping! <%stop:1%>;

[0183] 2. Segment text

[0184] The output of the multimodal model includes changes in emotions, pauses, speaking speed, and volume, which personalizes the single sentence dimension and is more appropriate when combined with the understanding of the material:

[0185] The T2A speech model is used. The T2A speech model itself supports pauses, so it can be segmented according to <%emotion%>, <%speed%>, and <%vol%>, and then the T2A speech model API is called respectively;

[0186] The result of segmentation: [

[0188] {

[0189] “emotion”:“happy”,

[0190] “speed”:1.0,

[0191] “vol”:1.5,

[0192] “text”:“Hey TikTok fam!<#0.5#>”

[0193] },

[0194] {

[0195] “emotion”:”happy”,

[0196] “speed”:1.0,

[0197] “vol”:1.0,

[0198] “text”:“You won'tbelieve<#0.2#>the amazing product<#0.3#>”

[0199] },

[0200] {

[0201] “emotion”:”happy”,

[0202] “speed”:1.0,

[0203] “vol”:2.0,

[0204] “text”:“Ijust discovered!<#0.5#>“

[0205] },

[0206] {

[0207] “emotion”:”happy”,

[0208] “speed”:1.0,

[0209] “vol”:1.0,

[0210] “text”:”It's a total game-changer<#0.2#>for anyone<#0.1#>who lovesonline shopping!<#1#>”

[0211] } ];

[0213] 3. The sentences of other scenes are processed in sequence according to the above method, and finally spliced ​​with ffmpeg:

[0214] Suppose you have three audio files: audio1.mp3, audio2.mp3, and audio3.mp3. You can concatenate them together using the following command:

[0215] ffmpeg-i audio1.mp3-i audio2.mp3-i audio3.mp3-filter_complex\"[0:a][1:a][2:a]concat=n=3:v=0:a=1[outa]"\

[0216] -map "[outa]" output.mp3

[0217] Command analysis

[0218] -i audio1.mp3-i audio2.mp3-i audio3.mp3: specifies the input audio file.

[0219] concat=n=3:v=0:a=1: connect three audio streams together, n=3 means there are three audio streams, v=0 means there is no video stream, and a=1 means there is one audio stream output.

[0220] -map "[outa]" output.mp3: Map the final audio stream to the output file output.mp3.

[0221] [0:a][1:a][2:a] are identifiers used to reference the input audio streams. Specifically: 0, 1, 2: These numbers represent the index of the input file. FFmpeg starts numbering the input files at 0, so:

[0222] 0 represents the first input file ( audio1.mp3 in this example).

[0223] 1 represents the second input file (audio2.mp3).

[0224] 2 represents the third input file (audio3.mp3).

[0225] :a: indicates that the audio stream is being referenced. FFmpeg supports multiple stream types, including audio (a) and video (v). Among them, :a refers to the audio stream.

[0226] According to the above method, generate corresponding audio for each paragraph, and then splice them. After that, splice the audio of all paragraphs to complete it.

[0227] Embodiment 1

[0228] like Figure 1 As shown, this embodiment provides a method for generating a video from a product page based on AI, including the following steps:

[0229] Step 1: Get product information from the product page set on the e-commerce platform;

[0230] Step 2: convert the pictures in the product information into picture description texts through a multimodal model, where the picture description texts correspond to the pictures one by one;

[0231] Step 3: disassemble the product video in the product information, perform scene switching detection on the product video, obtain at least one scene and a corresponding time code, cut the product video according to the time code, and obtain at least one segmented video; extract one frame from each segmented video by frame extraction to obtain a video graph, and input it into the multimodal model to convert it into a video description text, which corresponds one-to-one to the segmented video;

[0232] Step 4: Set the spoken prompt as required, input the spoken prompt, the text data in the product information, the picture description text and the video description text into the multimodal model to generate the spoken text; input the total video duration, the number of scenes and the spoken text into the multimodal model, segment the spoken text into scenes, obtain at least one scene text, and generate a corresponding audio file for each scene text;

[0233] Step 5: Use the multimodal model to match scene text, audio, scene, and segmented video, and finally generate a video file in a set format: input the video file into the video compiler to generate the required video. The video file is in JSON format and is used to know that the video editor generates the video.

[0234] In this embodiment, preferably, the step 1 is specifically: obtaining product information from the product page set on the e-commerce platform; the product information includes: product title, product unit price, product description, product main picture, product video and related product pictures, and the product information is used to generate the required video.

[0235] In this embodiment, preferably, the step 3 is specifically as follows: disassembling the product video in the product information, performing scene switching detection on the product video using PySceneDetect to obtain at least one scene and a corresponding time code, cutting the product video according to the time code through FFmpeg, and cleaning the video clips with a duration of only 1 second to obtain at least one segmented video; extracting one frame from each segmented video by frame extraction to obtain a video graph, and inputting it into a multimodal model to convert it into a video description text, which corresponds one-to-one to the segmented video.

[0236] In this embodiment, preferably, the step 4 is specifically as follows: setting a spoken prompt as required, inputting the spoken prompt, the text data in the product information, the picture description text and the video description text into the multimodal model to generate a spoken text; inputting the total video duration, the number of scenes and the spoken text into the multimodal model, segmenting the spoken text into scenes, and obtaining at least one scene text;

[0237] The multimodal model matches each scene text with a corresponding scene picture or video graph;

[0238] Input the oral data and the set audio prompt words into the multimodal model; the oral data includes the scene text and the corresponding scene picture; the audio prompt words include: set output requirements and output format; the output format includes:

[0239] Setting a first pause mark after each paragraph, wherein the first pause mark includes a first pause time, and the first pause mark is used to set the pause time after the paragraph;

[0240] Setting a second pause mark after the word, wherein the second pause mark includes a second pause time, and the second pause mark is used to set the pause time after the word;

[0241] Set an emotion mark before each paragraph to set the emotion used in the paragraph;

[0242] A speech rate indicator is set before each sentence to set the speech rate used for the sentence;

[0243] A volume mark is set before each sentence to set the volume used for the sentence;

[0244] The multimodal model outputs an identification text of a setting output format, wherein the identification text includes a setting identification;

[0245] The identification text is segmented according to the emotion identification, the speech rate identification, the volume identification and the second pause identification by using regular matching to obtain segmented text, and the segmented text is used to generate corresponding audio through the T2A speech model; and audio corresponding to the scene text is generated in sequence;

[0246] If there is only one audio file, the audio file generation is completed; if not, all audio files are spliced ​​in a set order through FFmpeg to complete the audio file generation.

[0247] Based on the same inventive concept, the present application also provides a device corresponding to the method in Example 1, see Example 2 for details.

[0248] Embodiment 2

[0249] like Figure 2 As shown, in this embodiment, a device for generating a video from a product page based on AI is provided, including:

[0250] The information acquisition module obtains product information from the product page set on the e-commerce platform;

[0251] A picture description module, which converts the pictures in the product information into picture description texts through a multimodal model, and the picture description texts correspond to the pictures one by one;

[0252] The video description word module disassembles the product video in the product information, performs scene switching detection on the product video to obtain at least one scene and a corresponding time code, and cuts the product video according to the time code to obtain at least one segmented video; extracts one frame from each segmented video by frame extraction to obtain a video graph, and inputs it into the multimodal model to convert it into a video description text, which corresponds one-to-one to the segmented video;

[0253] The audio generation module sets the spoken prompts according to the requirements, inputs the spoken prompts, the text data in the product information, the picture description text and the video description text into the multimodal model, and generates the spoken text; inputs the total video duration, the number of scenes and the spoken text into the multimodal model, divides the spoken text into scenes, obtains at least one scene text, and generates a corresponding audio file for each scene text;

[0254] The video generation module matches scene text, audio, scene and segmented video through a multimodal model, and finally generates a video file in a set format: the video file is input into the video compiler to generate the required video. The video file is in JSON format and is used to know that the video editor generates the video.

[0255] In this embodiment, preferably, the information acquisition module specifically obtains product information from the product page set on the e-commerce platform; the product information includes: product title, product unit price, product description, product main picture, product video and related product pictures, and the product information is used to generate the required video.

[0256] In this embodiment, preferably, the video description word module is specifically as follows: disassembling the product video in the product information, using PySceneDetect to perform scene switching detection on the product video to obtain at least one scene and a corresponding time code, cutting the product video according to the time code through FFmpeg, and cleaning the video clips with a length of only 1 second to obtain at least one segmented video; extracting one frame from each segmented video by frame extraction to obtain a video graph, and inputting it into a multimodal model to convert it into a video description text, and the video description text corresponds one-to-one to the segmented video.

[0257] In this embodiment, preferably, the audio generation module specifically includes: setting a spoken prompt according to demand, inputting the spoken prompt, text data in the product information, picture description text and video description text into the multimodal model, and generating a spoken text; inputting the total video duration, the number of scenes and the spoken text into the multimodal model, and segmenting the spoken text into scenes to obtain at least one scene text;

[0258] The multimodal model matches each scene text with a corresponding scene picture or video graph;

[0259] Input the oral data and the set audio prompt words into the multimodal model; the oral data includes the scene text and the corresponding scene picture; the audio prompt words include: set output requirements and output format; the output format includes:

[0260] Setting a first pause mark after each paragraph, wherein the first pause mark includes a first pause time, and the first pause mark is used to set the pause time after the paragraph;

[0261] Setting a second pause mark after the word, wherein the second pause mark includes a second pause time, and the second pause mark is used to set the pause time after the word;

[0262] Set an emotion mark before each paragraph to set the emotion used in the paragraph;

[0263] A speech rate indicator is set before each sentence to set the speech rate used for the sentence;

[0264] A volume mark is set before each sentence to set the volume used for the sentence;

[0265] The multimodal model outputs an identification text of a setting output format, wherein the identification text includes a setting identification;

[0266] The identification text is segmented according to the emotion identification, the speech rate identification, the volume identification and the second pause identification by using regular matching to obtain segmented text, and the segmented text is used to generate corresponding audio through the T2A speech model; and audio corresponding to the scene text is generated in sequence;

[0267] If there is only one audio file, the audio file generation is completed; if not, all audio files are spliced ​​in a set order through FFmpeg to complete the audio file generation.

[0268] Since the device introduced in the second embodiment of the present invention is a device used to implement the method of the first embodiment of the present invention, those skilled in the art can understand the specific structure and deformation of the device based on the method introduced in the first embodiment of the present invention, so it is not described here in detail. All devices used in the method of the first embodiment of the present invention belong to the scope of protection of the present invention.

[0269] Based on the same inventive concept, the present application provides an electronic device embodiment corresponding to the first embodiment, see the third embodiment for details.

[0270] Embodiment 3

[0271] This embodiment provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, any implementation method in the first embodiment can be implemented.

[0272] Since the electronic device introduced in this embodiment is a device used to implement the method in the first embodiment of the present application, based on the method introduced in the first embodiment of the present application, a person skilled in the art can understand the specific implementation of the electronic device of the present embodiment and its various variations, so how the electronic device implements the method in the embodiment of the present application is not described in detail here. As long as a person skilled in the art implements the device used by the method in the embodiment of the present application, it belongs to the scope of protection of the present application.

[0273] Based on the same inventive concept, the present application provides a storage medium corresponding to the first embodiment, see the fourth embodiment for details.

[0274] Embodiment 4

[0275] This embodiment provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, any implementation method in the first embodiment can be implemented.

[0276] The technical solution provided in the embodiments of the present application has at least the following technical effects or advantages:

[0277] This embodiment integrates multimodal model technology, which can quickly transform product pages into eye-catching short video content and publish it to major short video platforms with one click, providing great convenience for merchants to promote their products. In addition, the solution can also intelligently mark text according to specific scenarios, and then guide the voice model to generate voices with different speaking speeds, volumes and emotions, significantly improving the quality and appeal of voice, making it a powerful tool for promotion.

[0278] Users can easily use this solution and enjoy a fast and efficient content generation experience, which greatly saves time and energy. This innovative approach not only simplifies the content creation process, but also helps merchants stand out in the fiercely competitive market by enhancing the attractiveness of the content.

[0279] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0280] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0281] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0282] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0283] Although the specific implementation modes of the present invention are described above, those skilled in the art should understand that the specific implementation modes described are only illustrative and are not intended to limit the scope of the present invention. Equivalent modifications and changes made by those skilled in the art in accordance with the spirit of the present invention should be included in the scope of protection of the claims of the present invention.

Claims

1. A method for generating videos from product pages based on AI, characterized in that: The steps include: Step 1: Get product information from the product page set on the e-commerce platform; Step 2: convert the pictures in the product information into picture description texts through a multimodal model, where the picture description texts correspond to the pictures one by one; Step 3: disassemble the product video in the product information, perform scene switching detection on the product video, obtain at least one scene and a corresponding time code, cut the product video according to the time code, and obtain at least one segmented video; extract one frame from each segmented video by frame extraction to obtain a video graph, and input it into the multimodal model to convert it into a video description text, which corresponds one-to-one to the segmented video; Step 4: Set the spoken prompt as required, input the spoken prompt, the text data in the product information, the picture description text and the video description text into the multimodal model to generate the spoken text; input the total video duration, the number of scenes and the spoken text into the multimodal model, segment the spoken text into scenes, obtain at least one scene text, and generate a corresponding audio file for each scene text; Step 5: Match scene text, audio, scene, and segmented video through a multimodal model, and finally generate a video file in a set format: input the video file into a video compiler to generate the required video.

2. The method for generating a video from a product page based on AI according to claim 1, characterized in that: The step 1 is specifically as follows: obtaining product information from the product page set on the e-commerce platform; the product information includes: product title, product unit price, product description, product main picture, product video and related product pictures, and the product information is used to generate the required video.

3. The method for generating a video from a product page based on AI according to claim 1, characterized in that: The step 3 is specifically as follows: disassembling the product video in the product information, using PySceneDetect to perform scene switching detection on the product video to obtain at least one scene and a corresponding time code, cutting the product video according to the time code through FFmpeg, and cleaning the video clips with a duration of only 1 second to obtain at least one segmented video; extracting one frame from each segmented video by frame extraction to obtain a video graph, and inputting it into a multimodal model to convert it into a video description text, which corresponds one-to-one to the segmented video.

4. The method for generating a video from a product page based on AI according to claim 1, characterized in that: The step 4 specifically includes: setting a spoken prompt according to the demand, inputting the spoken prompt, the text data in the product information, the picture description text and the video description text into the multimodal model to generate a spoken text; inputting the total video duration, the number of scenes and the spoken text into the multimodal model, segmenting the spoken text into scenes, and obtaining at least one scene text; The multimodal model matches each scene text with a corresponding scene picture or video graph; Input the spoken data and the set audio prompt words into the multimodal model; The oral broadcast data includes scene text and corresponding scene pictures; The audio prompt includes: setting output requirements and output format; the output format includes: Setting a first pause mark after each paragraph, wherein the first pause mark includes a first pause time, and the first pause mark is used to set the pause time after the paragraph; Setting a second pause mark after the word, wherein the second pause mark includes a second pause time, and the second pause mark is used to set the pause time after the word; Set an emotion mark before each paragraph to set the emotion used in the paragraph; A speech rate indicator is set before each sentence to set the speech rate used for the sentence; A volume mark is set before each sentence to set the volume used for the sentence; The multimodal model outputs an identification text of a setting output format, wherein the identification text includes a setting identification; The identification text is segmented according to the emotion identification, the speech rate identification, the volume identification and the second pause identification by using regular matching to obtain segmented text, and the segmented text is used to generate corresponding audio through the T2A speech model; and audio corresponding to the scene text is generated in sequence; If there is only one audio file, the audio file generation is completed; if not, all audio files are spliced ​​in a set order through FFmpeg to complete the audio file generation.

5. A device for generating videos from product pages based on AI, characterized in that: include: The information acquisition module obtains product information from the product page set on the e-commerce platform; A picture description module, which converts the pictures in the product information into picture description texts through a multimodal model, and the picture description texts correspond to the pictures one by one; The video description word module disassembles the product video in the product information, performs scene switching detection on the product video to obtain at least one scene and a corresponding time code, and cuts the product video according to the time code to obtain at least one segmented video; extracts one frame from each segmented video by frame extraction to obtain a video graph, and inputs it into the multimodal model to convert it into a video description text, which corresponds one-to-one to the segmented video; The audio generation module sets the spoken prompts according to the requirements, inputs the spoken prompts, the text data in the product information, the picture description text and the video description text into the multimodal model, and generates the spoken text; inputs the total video duration, the number of scenes and the spoken text into the multimodal model, divides the spoken text into scenes, obtains at least one scene text, and generates a corresponding audio file for each scene text; The video generation module matches scene text, audio, scene and segmented video through a multimodal model, and finally generates a video file in a set format: the video file is input into the video compiler to generate the required video.

6. The device for generating videos from product pages based on AI according to claim 5, characterized in that: The information acquisition module specifically obtains product information from the product page set on the e-commerce platform; the product information includes: product title, product unit price, product description, product main picture, product video and related product pictures, and the product information is used to generate the required video.

7. The device for generating videos from product pages based on AI according to claim 5, characterized in that: The video description word module specifically comprises: disassembling the product video in the product information, using PySceneDetect to perform scene switching detection on the product video to obtain at least one scene and a corresponding time code, cutting the product video according to the time code through FFmpeg, and cleaning the video clips with a duration of only 1 second to obtain at least one segmented video; extracting one frame from each segmented video by frame extraction to obtain a video graph, and inputting it into a multimodal model to convert it into a video description text, which corresponds one-to-one to the segmented video.

8. The device for generating videos from product pages based on AI according to claim 5, characterized in that: The audio generation module specifically includes: setting a spoken prompt according to demand, inputting the spoken prompt, text data in the product information, picture description text and video description text into the multimodal model to generate a spoken text; inputting the total video duration, the number of scenes and the spoken text into the multimodal model, segmenting the spoken text into scenes, and obtaining at least one scene text; The multimodal model matches each scene text with a corresponding scene picture or video graph; Input the spoken data and the set audio prompt words into the multimodal model; The oral broadcast data includes scene text and corresponding scene pictures; The audio prompt includes: setting output requirements and output format; the output format includes: Setting a first pause mark after each paragraph, wherein the first pause mark includes a first pause time, and the first pause mark is used to set the pause time after the paragraph; Setting a second pause mark after the word, wherein the second pause mark includes a second pause time, and the second pause mark is used to set the pause time after the word; Set an emotion mark before each paragraph to set the emotion used in the paragraph; A speech rate indicator is set before each sentence to set the speech rate used for the sentence; A volume mark is set before each sentence to set the volume used for the sentence; The multimodal model outputs an identification text of a setting output format, wherein the identification text includes a setting identification; The identification text is segmented according to the emotion identification, the speech rate identification, the volume identification and the second pause identification by using regular matching to obtain segmented text, and the segmented text is used to generate corresponding audio through the T2A speech model; and audio corresponding to the scene text is generated in sequence; If there is only one audio file, the audio file generation is completed; if not, all audio files are spliced ​​in a set order through FFmpeg to complete the audio file generation.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the method according to any one of claims 1 to 4 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 4 is implemented.

Images