Video end-to-end generation method and device, equipment and medium

Abstract

The invention relates to the field of media playing control, and discloses a video end-to-end generation method and device, equipment and a medium, and the method comprises the steps: obtaining a commodity image and an effect description text which is used for indicating the playing display effect of a to-be-generated target video; inputting the commodity image and the effect description text into an end-to-end generation model, and generating a target video which takes the commodity image as basic content and is presented according to a playing display effect indicated by the effect description text by the model; and in response to a user confirmation event, publishing the target video to a commodity promotion platform for access and playing. According to the method, the target video meeting the commodity image and the playing display effect description is quickly generated through the end-to-end generation model, so that the generation cost and time are effectively reduced, the matching degree of the video, the commodity and the description is improved, meanwhile, the release process of the target video is simplified, and the marketing effect and the market competitiveness are enhanced.

Description

Technical Field

[0001] The present application relates to video generation technology, and in particular to an end-to-end video generation method, its device, equipment, and medium. Background Art

[0002] In the current era of digital marketing, videos have become one of the important means of product promotion. Traditional video production usually relies on professional video shooting teams and post-production personnel, which is not only costly but also has a long production cycle, making it difficult to quickly respond to market changes and personalized needs. In recent years, with the development of artificial intelligence technology, especially the progress of generative artificial intelligence (AIGC) technology, the technology of generating videos based on images and texts has gradually emerged, providing new possibilities for the production of videos for product promotion.

[0003] However, there are still many technical problems in the application of existing video generation technologies to video production. On the one hand, traditional image-to-video generation technologies often require a large amount of training data and complex model training processes, which greatly increases the training cost and time cost of the model and makes it difficult to meet the demand for quickly generating videos. On the other hand, when existing technologies generate videos, it is difficult to accurately generate video content that meets the requirements according to product images and specific playback display effect descriptions. For example, the generated videos may not accurately reflect the appearance, functions, or usage scenarios of the products, resulting in a large deviation between the generated videos and the expected effects and being unable to effectively enhance the attractiveness and sales conversion rate of the products.

[0004] In addition, existing technologies usually lack the ability to flexibly adjust the style and rhythm of videos during the video generation process. Since videos need to be customized for display effects according to the characteristics of different products, target audiences, and marketing scenarios, a technology that can quickly and accurately generate videos based on product images and playback display effect descriptions and can flexibly adjust the style and rhythm of videos is particularly important.

[0005] In summary, the problems existing in the prior art mainly include: first, the training cost is high and the time is long, making it difficult to quickly generate videos; second, the matching degree between the generated video content and the product image and playback display effect description is low, unable to meet the needs of precision marketing; third, there is a lack of the ability to flexibly adjust the style and rhythm of videos, making it difficult to adapt to diverse promotion scenarios. These problems limit the wide application of generative artificial intelligence technology in the field of video production and also affect the production efficiency and marketing effect of videos. Summary of the Invention

[0006] The purpose of the present application is to solve the above problems and provide an end-to-end video generation method, its corresponding device, equipment, and a non-volatile readable storage medium.

[0007] According to one aspect of the present application, a video end-to-end generation method is provided, including the following steps:

[0008] Obtain a product image and an effect description text corresponding to the playback display effect of the target video to be generated;

[0009] Input the product image and the effect description text into an end-to-end generation model, and the model generates a target video based on the product image and presented according to the playback display effect indicated by the effect description text;

[0010] In response to a user confirmation event, publish the target video to a product promotion platform for access and playback.

[0011] According to another aspect of the present application, a video end-to-end generation device is provided, including:

[0012] A data acquisition module configured to obtain a product image and an effect description text corresponding to the playback display effect of the target video to be generated;

[0013] A video generation module configured to input the product image and the effect description text into an end-to-end generation model, and the model generates a target video based on the product image and presented according to the playback display effect indicated by the effect description text;

[0014] A video publishing module configured to, in response to a user confirmation event, publish the target video to a product promotion platform for access and playback.

[0015] According to another aspect of the present application, a video end-to-end generation device includes a central processing unit and a memory. The central processing unit is configured to call and run a computer program stored in the memory to execute the steps of the video end-to-end generation method described in the present application.

[0016] According to another aspect of the present application, a non-volatile readable storage medium stores a computer program implemented according to the video end-to-end generation method in the form of computer-readable instructions. When the computer program is called and run by a computer, it executes the steps included in the method.

[0017] The technical solution of this application demonstrates remarkable innovation and practicality in the field of e-commerce video generation. Compared with traditional technologies, by inputting product images and effect description texts into an end-to-end generation model, this application can quickly generate target videos that precisely match the product images and play display effect descriptions, effectively solving the problems of high training costs, long time, and low matching degree between the generated videos and descriptions in the existing technologies. This solution not only significantly reduces the time and resource consumption for generating target videos, but also improves the accuracy and attractiveness of video content, enhancing the sales conversion rate. At the same time, this application further optimizes the publishing process of target videos, enabling merchants to conveniently publish the generated videos to product promotion platforms, expanding the dissemination scope of the videos, enhancing the market competitiveness of products, providing a one-stop video production and promotion solution for merchants, and greatly promoting the improvement of the commercial value and user experience of e-commerce platforms. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 is a network architecture suitable for applying the end-to-end video generation method of this application exemplarily;

[0019] Figure 2 is a flowchart showing an embodiment of the end-to-end video generation method of this application;

[0020] Figure 3 is a schematic block diagram of the principle of the end-to-end video generation device of this application;

[0021] Figure 4 is a schematic structural diagram of an end-to-end video generation device adopted by this application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0022] Before introducing the specific embodiments of the technical solution of this application in detail, the network architecture and application scenarios suitable for supporting the implementation of the technical solution of this application are first disclosed.

[0023] As Figure 1 shown, the e-commerce platform network architecture involved in this application includes a server 80, a terminal device 90 of a merchant user, and a terminal device 92 of a consumer user. The server 80, as the core of the e-commerce platform, plays an important role in providing the e-commerce independent site operated by the merchant user to the terminal device, and is also responsible for processing various data and requests to ensure the stable operation and efficient service of the platform. The independent site of the merchant user serves as its product promotion platform.

[0024] In this application, the server 80 or the terminal device 90 of the merchant user can run a computer program product implemented according to the video end-to-end generation method of this application, constituting the video end-to-end generation device of this application. According to the technical solution of this application, this program product can process the target video generation request from the terminal device of the merchant user. The merchant user submits the product image and the text describing the playback display effect of the target video to be generated, that is, the effect description text, and this information is transmitted to the server 80 or the generation device running locally. Subsequently, the device quickly generates a target video that matches the product image and the effect description text in an end-to-end manner according to the technical solution of this application.

[0025] The target video of this application can be a product display video, a promotional video, an advertising video, or other related dynamic video content. After the end-to-end generation process, the image dynamic effect presented when the target video is played is based on the image content of the product image submitted by the user and is obtained by enriching the dynamic change information. The target video can not only be stored in the terminal device 90 of the merchant user, but also be uploaded to the server 80 and then published to the server where the merchant's independent site is located. The consumer user can play the target video by accessing the website page or promotional email page where the target video is located through the terminal device 92 and watch the playback display effect described in the effect description text.

[0026] For example, when a merchant is preparing to promote a new style of dress, they can use the technical solution of this application to submit the product image of the dress and the effect description text, such as "The model walks gracefully in the sun, and the skirt flutters in the wind". Based on these inputs, the end-to-end generation model of this application can quickly generate a video in which the model is wearing this dress, walking gracefully in a sunny scene, and the skirt gently fluttering with the breeze, perfectly showing the style and wearing effect of the product. The generated video can be stored in the merchant's terminal device or uploaded to the server and published on the product page of the merchant's independent site. When consumers browse the product page and click to play the target video, they can intuitively see the dynamic display of this dress, as if the model is presenting the product in front of them, greatly enhancing the shopping experience.

[0027] For another example, a merchant plans to promote a Christmas-themed home decoration during Christmas. The merchant submits an image of the decoration and a text description of the effect, such as "The decoration is placed beside the Christmas tree, with snowflakes falling and Christmas music playing around." The generation model of this application will generate a video based on this information. In the video, the home decoration is placed beside a beautifully decorated Christmas tree, with snowflakes slowly falling around, and warm Christmas music playing in the background, creating a strong festive atmosphere. After this video is posted on the merchant's website, when consumers visit the promotion page, they can feel the usage effect and atmosphere of the product in a specific festival scene through the video, thus stimulating the desire to purchase and enhancing the promotion effect.

[0028] As can be seen from the above examples, the target video generation process of this application not only provides a merchant with an efficient and convenient way to generate and publish a target video, but also brings a more intuitive and vivid product display experience to consumers.

[0029] In addition, some basic concepts involved in this application are also worthy of further explanation. For example, "end-to-end generation" refers to the entire process from inputting the product image and the text description of the effect to outputting the target video, without manual intervention, and is completely automatically completed by a computer program. This method not only improves the generation efficiency, but also ensures the quality and consistency of the generated video. The "text description of the effect" is the written description by the merchant user of the playback display effect of the target video, which provides specific instructions for the generation model, enabling the generated video to accurately reflect the merchant's needs and creativity.

[0030] Through the deployment of this network architecture and application scenario, the technical solution of this application can not only meet the merchant's need to quickly generate and publish a target video, but also enhance the shopping experience of consumers on the e-commerce platform, thereby promoting the dual improvement of the commercial value and user experience of the e-commerce platform.

[0031] Please refer to Figure 2 , according to a video end-to-end generation method provided by this application, it can be implemented as a computer program product installed and running on a computer device such as a server or a terminal device. In some embodiments of this method, the following steps are included:

[0032] Step S3100, obtain a product image and a text description of the effect corresponding to the playback display effect of the target video to be generated;

[0033] The product image can be completed by the merchant user operating on the terminal device 90. The merchant user can select the image file of the product from the local storage device or download the product image from the server 80 or other storage locations through the network. These images can be high-definition photos of the product, product promotion pictures, or other images that can clearly display the appearance and features of the product. For example, for a new style dress, the merchant can upload the front, side, and back photos of the model wearing the dress to more comprehensively display the details of the product in the generated target video.

[0034] The acquisition of the effect description text can also be completed by the merchant user operating on the terminal device 90. The merchant user needs to input a description text to indicate the playback display effect of the target video to be generated. This text can be described in natural language. For example, "The model walks gracefully in the sun, and the skirt flutters in the wind", or "The ornaments are placed beside the Christmas tree, with snowflakes falling and Christmas music playing around". The effect description text provides specific instructions for the generation model, enabling the generated video to accurately reflect the merchant's needs and creativity.

[0035] In practical applications, there can be multiple specific embodiments for the input method of the effect description text. For example, the merchant user can directly input the description text through a text editing box or select and modify the system preset template. In addition, natural language processing tools can also be provided. For example, based on the text input by the user, preset text optimization prompt words are added and input into the large language model to obtain the effect description text, helping the merchant user optimize the expression of the description text to ensure that it can be accurately understood by the generation model.

[0036] Step S3200: Input the product image and the effect description text into the end-to-end generation model, and the model generates a target video based on the product image and presented according to the playback display effect indicated by the effect description text;

[0037] The end-to-end generation model applied in this application is a deep learning-based model that can automatically convert the input product images and effect description texts into corresponding target videos. The model consists of multiple sub-modules, including an image encoder, a text encoder, a diffusion model, and a decoder. The image encoder and the text encoder respectively form the image encoding branch and the text encoding branch. Among them, the image encoder is responsible for extracting the visual features of the product images and converting the image content into a hidden layer feature representation that the model can understand; the text encoder processes the effect description text, extracts the semantic information therein to obtain the corresponding hidden layer feature representation, and provides specific playback display effect guidance for the video generation process. After receiving the hidden layer features of the image and the text, the diffusion model gradually constructs a hidden layer representation of the video content that meets the requirements through a series of denoising operations. Finally, the decoder decodes these hidden layer representations into specific video frames to form a complete target video.

[0038] In one embodiment, taking the example of generating a target video of a new style dress, the merchant user uploads photos of a model wearing the dress from one or more angles as product images and inputs the effect description text "The model walks gracefully in the sun, and the skirt flutters in the wind". After receiving these inputs, the end-to-end generation model, the image encoder extracts the visual features such as the style, color, and texture of the dress, and the text encoder understands the video scenes and dynamic effects required in the description text. Based on this information, the diffusion model generates a hidden layer representation of the video content where the model walks gracefully against a sunny background and the skirt of the dress gently sways in the wind. The decoder then decodes these hidden layer representations into specific video frames, and finally generates a target video that vividly demonstrates the wearing effect and dynamic beauty of the new style dress.

[0039] In another embodiment, for the case where a merchant plans to promote a Christmas-themed home decoration during Christmas, the merchant submits an image of the decoration and the effect description text "The decoration is placed beside the Christmas tree, and there are snowflakes falling and Christmas music playing around". Through the collaborative work of its internal sub-modules, the end-to-end generation model generates a video in which the home decoration is placed beside a beautifully decorated Christmas tree, snowflakes are slowly falling around, and warm Christmas music is playing in the background, creating a strong festive atmosphere and perfectly meeting the merchant's promotion needs.

[0040] In some embodiments, the end-to-end generation model can also generate target videos of various styles and scenarios according to different inputs and requirements. For example, for a sports equipment, a merchant can upload an image of an athlete using the equipment and input a description text such as "the athlete is doing high-intensity training outdoors". The model will generate a video demonstrating the high performance and applicability of the sports equipment. For an electronic product, a merchant can upload a product appearance picture and a description text such as "the product is used in a modern office environment". The model will then generate a video reflecting the technological sense and practicality of the product. These specific embodiments demonstrate the flexibility and powerful generation ability of the end-to-end generation model when processing different products and effect description texts, and can meet the diverse target video production needs of merchants.

[0041] Step S3300, in response to a user confirmation event, publish the target video to a product promotion platform for access and playback.

[0042] After the merchant user completes the generation of the target video on the terminal device 90, a confirmation event can be triggered through the user interface, such as clicking the "Publish" button. In response to this user confirmation event, a series of operations can be performed to ensure that the target video can be successfully published to the product promotion platform. Specifically, first, a video playback page corresponding to the target video is created on the product promotion platform. This page can be a dedicated product details page in the merchant's independent site or a special page for a promotional activity. The design of the video playback page usually includes elements such as a video player, a product information display area, and possibly a purchase link, etc., to provide a complete shopping experience. In some other embodiments, the product promotion platform of the present application can also be other types of advertising platforms, such as e-commerce advertising platforms, email promotion advertising platforms, etc.

[0043] Subsequently, the video file of the target video is uploaded to the server of the product promotion platform. This process can be achieved through various technical means, such as using FTP (File Transfer Protocol), HTTP / HTTPS protocol, or a dedicated API interface. During the upload process, the video file can be compressed or format-converted to ensure that the video can be smoothly played on different devices and network environments. In addition, in order to improve the video loading speed and playback quality, CDN (Content Delivery Network) technology can also be adopted to cache the video file on servers in multiple geographical locations, thereby achieving fast distribution.

[0044] After the video file is successfully uploaded to the server, an access link can be set on the video playback page. This link can directly point to the storage location of the video file or provide video content through a video playback service. This access link can be configured to play automatically or manually. For example, when a consumer user accesses the product promotion platform through a terminal device 92 and clicks on this access link, they can watch the target video on the video playback page. The video playback page can provide various playback options, such as full-screen playback, volume control, playback progress bar, etc., to enhance the user experience.

[0045] In one embodiment, after a merchant user generates a target video showing a new style dress, they click the "Publish" button. Immediately, a video playback area is created on the product details page of the merchant's independent site, and the video file is uploaded to the server. After the upload is complete, a video playback link is generated on the page. When a consumer user browses this product details page and clicks the playback link or relies on a pre-configured automatic playback mechanism, they can watch the target video of the model walking gracefully in the new style dress on the page, and intuitively feel the style and wearing effect of the product.

[0046] In another embodiment, a merchant user generates a target video for a Christmas-themed home decoration and hopes to use it in the Christmas promotion event. After the merchant user completes the video generation and confirms the release, a video playback page is created on the promotion event special page, and the video file is uploaded to the server. An access link has been set for this video in advance and embedded in the special page. When a consumer user accesses the promotion event page, they can watch the video by clicking the link or relying on the automatic playback mechanism, and see the warm scene of the home decoration placed beside the Christmas tree with snowflakes falling around and Christmas music playing in the background, thus stimulating the desire to purchase.

[0047] From the above embodiments, it can be seen that compared with the traditional technology, the technical solution of this application can achieve various beneficial effects, including but not limited to:

[0048] First of all, by inputting the product image and the text description of the effect into the end-to-end generation model, this application can quickly generate a target video based on the product image and meeting the description of the playback and display effect. This technical solution effectively solves the problems of high training cost and long time in the prior art, significantly reduces the time and resource consumption for generating the target video, enables merchants to produce target videos more efficiently, and quickly respond to market changes and personalized needs.

[0049] Secondly, the technical solution of this application can accurately generate a target video based on the product image and the effect description text, solving the problem of low matching degree between the generated video content and the product image and the description of the playback display effect in the prior art. Through the end-to-end generation model, the video content can not only accurately reflect the appearance, function and usage scenario of the product, but also achieve a specific playback display effect according to the effect description text, thereby enhancing the attractiveness and sales conversion rate of the target video and better meeting the needs of precision marketing.

[0050] In addition, this application also provides a function to publish the generated target video to the product promotion platform, further enhancing the practicality and convenience of the target video. By creating a video playback page on the product promotion platform and setting an access link, users can conveniently access and play the target video, thereby expanding the dissemination range of the target video, increasing the exposure rate of the product and the market competitiveness. The implementation of this function not only optimizes the publishing process of the target video, but also provides a one-stop video production and promotion solution for merchants, further enhancing the technical advantages of this application.

[0051] Based on any embodiment of the method of this application, before obtaining the product image and the effect description text corresponding to the playback display effect indicating the video to be generated, it includes:

[0052] Step S2100, obtaining video samples in the training dataset and the effect description text corresponding to the playback display effect describing each video sample;

[0053] The training dataset obtained in this embodiment contains a large number of video samples, which can be product display videos, promotional videos, advertising videos or other related dynamic video content provided by merchants. Each video sample is accompanied by corresponding effect description text, which uses natural language to describe the playback display effect of the video, such as "the model walks gracefully in the sun, and the skirt flutters in the wind" or "the decoration is placed beside the Christmas tree, with snowflakes falling and Christmas music around". These texts provide clear generation targets for the model, enabling the model to learn how to generate corresponding video content according to the text description.

[0054] In one embodiment, the training dataset can be obtained from the database of the e-commerce platform. The e-commerce platform usually stores a large number of product videos and related description information, which can be directly used for training. For example, when merchants upload product videos, they will simultaneously input a text describing the video content, which can be used as the effect description text. In addition, relevant video and text data can also be collected from the Internet through web crawler technology to enrich the content of the training dataset.

[0055] In another embodiment, the training dataset can be constructed by manual annotation based on the database of the e-commerce platform. For example, professional content creators or annotators can be invited to provide detailed descriptions of the existing video samples. These description texts can include not only the scenes, actions, and atmospheres of the videos, but also information such as the styles, rhythms, and colors of the videos. The training dataset constructed in this way can provide richer semantic information, which helps the model learn more complex video generation rules.

[0056] During the process of obtaining the training dataset, it is also necessary to preprocess the data. For example, operations such as frame rate adjustment, resolution unification, and format conversion are performed on the video samples to ensure the consistency and compatibility of the data. For the effect description texts, text cleaning and standardization can be carried out to remove useless symbols and stop words to ensure the quality and readability of the texts.

[0057] Step S2200: Call the video sample and the corresponding effect description text to train the end-to-end generation model to obtain the inference video corresponding to the video sample;

[0058] Input the video sample and the effect description text in the training dataset into the image encoding branch and the text encoding branch of the end-to-end generation model respectively. The inference ability of the model can be used to generate a video, which is called the inference video here.

[0059] The image encoding branch is responsible for extracting the visual features of each frame in the video sample, including the appearance of the product, the layout of the scene, and other visual elements. The text encoding branch processes the effect description text and extracts the semantic information, which provides the specific goals and style guidance for video generation for the model.

[0060] In one embodiment, assume that the training dataset contains a video sample showing a new style dress and the corresponding effect description text "The model walks gracefully in the sun, and the skirt flutters in the wind". The image encoder will extract the visual features such as the style, color, and texture of the model wearing the dress in the video, while the text encoder will understand the required video scene and dynamic effects in the description text. The outputs of these two branches - the hidden features of the image and the hidden features of the text - will be merged and input into the diffusion model.

[0061] The diffusion model is the core part of the end-to-end generation model. It gradually constructs the hidden representation of the video content that meets the requirements through a series of denoising operations. During the denoising process, the model tries to remove the noise in the input features while retaining the information related to the effect description text. This process can be regarded as a process of gradual refinement. Starting from the initial noise features, the model gradually generates a hidden representation that is closer and closer to the target video.

[0062] In another embodiment, the training dataset may include a video sample showing Christmas-themed home decorations and the corresponding effect description text "The decorations are placed beside the Christmas tree, with snowflakes falling and Christmas music playing around". Similarly, the image encoder extracts the appearance features of the decorations in the video, and the text encoder understands the scene and atmosphere requirements in the description text. The diffusion model generates a hidden layer representation of the video content that conforms to the description based on this information.

[0063] Step S2300: Supervise the training process of each video sample using a preset loss function, and train the end-to-end generation model until it reaches a converged state.

[0064] During the training process, the model continuously adjusts its own parameters to minimize the difference between the generated video and the video sample. This process can be supervised by a preset loss function such as the mean squared error loss function. The loss function quantifies the similarity between the generated video and the video sample, and the model continuously optimizes its parameters according to the feedback of the loss function until it reaches a converged state.

[0065] During the training process, the end-to-end generation model can be continuously iteratively trained. Each iteration takes a sample pair consisting of a video sample and its corresponding effect description text from the training dataset as input to the model, and continues to iteratively train according to the above mechanism until the model reaches a converged state, and then the model can be put into use.

[0066] This application makes use of local resources in the e-commerce platform. By obtaining the video samples in the training dataset and their corresponding effect description texts, it provides rich learning materials for the end-to-end generation model, enabling the model to accurately generate high-quality video content according to the text description. During the training process, the image encoding branch and the text encoding branch respectively extract visual features and semantic information, and perform denoising and refinement through the diffusion model to gradually generate a hidden layer representation that matches the target video. The preset loss function supervises the training process to ensure that the model continuously optimizes its parameters until convergence, thereby achieving efficient and accurate video generation capabilities. This training method not only improves the model's adaptability to different products and scenarios, but also significantly enhances the quality and diversity of the generated videos, providing an efficient and personalized target video generation solution for merchants, and greatly enhancing the flexibility and attractiveness of e-commerce marketing.

[0067] Based on any embodiment of the method of this application, call the video sample and the corresponding effect description text to train the end-to-end generation model, and obtain the inference video corresponding to the video sample, including:

[0068] Step S2210: Input the video samples in the training dataset and the effect description text into the image encoding branch and the text encoding branch of the end-to-end generation model respectively to extract their respective hidden layer features;

[0069] The role of the image encoding branch is to extract the visual features of each frame from the video samples. These features include the appearance of the product, the layout of the scene, colors, textures, and other visual elements. For example, for a video sample showing a new dress, the image encoder will extract information such as the style, color, texture of the dress, and the pose of the model. These information are transformed into hidden layer features, providing a visual basis for subsequent video generation.

[0070] The text encoding branch is responsible for processing the effect description text and extracting the semantic information therein. These information provide the specific goals and style guidance for video generation for the model. For example, for the effect description text "The model walks gracefully in the sun, and the skirt flutters in the wind", the text encoder will extract semantic information such as the scene to be shown in the video (in the sun), the action (walking gracefully), and the dynamic effect (skirt fluttering). These information are also transformed into hidden layer features, providing guidance for video generation together with the image features.

[0071] Step S2220: Construct noise data corresponding to the hidden layer features for each image frame in the video sample, and use some image frames in the video sample to replace the noise frames corresponding in time sequence in the noise data to obtain controlled noise;

[0072] When implementing this step, it is first necessary to analyze each frame image in the video sample to determine its time sequence position in the video sequence. For example, for a video containing a model showing a new dress, each frame in the video sample contains information such as the pose of the model, the appearance of the dress, and the background environment. The image encoding branch has extracted the visual features of these frames in step S2210 and transformed them into hidden layer feature representations.

[0073] Next, to construct the noise data, the model will generate random noise related to these hidden layer features. The noise data is constructed corresponding to each image frame in the video sample, so it contains multiple noise frames. These noise data can be regarded as a kind of random perturbation to the original video frames, and its purpose is to provide a broader learning space for the model so that the model can learn how to recover the video content that meets the requirements from the noise. However, pure noise data may not provide enough semantic information to guide video generation, so the concept of controlled noise is introduced in this embodiment.

[0074] The construction of the controlled noise is achieved by replacing some of the image frames in the video sample with the noise frames corresponding in time sequence in the noise data. Such replacement operations can be performed at multiple time sequence positions. For example, the first frame, the middle frame, and the last frame of the video or any one of them can be selected for replacement. In this way, the model can refer to the actual image frame information during the generation process, so as to better retain the key visual features and dynamic effects of the commodity.

[0075] In one embodiment, assuming that the video sample contains 16 image frames, the model can replace the image frames of the 1st frame, the 8th frame, and the 16th frame with the corresponding noise frames in the noise data respectively. In this way, the controlled noise contains the key frame information of the video, and the model can use this information to generate more accurate and coherent video content during the subsequent denoising process.

[0076] In another embodiment, the model can selectively replace certain specific frames according to the specified information in the effect description text. For example, if the effect description text emphasizes the importance of a certain specific action in the video, the model can replace the frames related to this action with the corresponding frames in the noise data, so as to more accurately display this action in the generated video. Or, if the effect description text contains a specification for a specific image frame, the specific image frame is replaced with the noise frame corresponding in time sequence in the noise data. This embodiment can further enhance the ability of the end-to-end generation model to generate a video that meets the user's needs according to the user-customized effect description text, opens a larger customization space for the user through the effect description text, and can better improve the user experience.

[0077] Step S2230: Input the controlled noise and the hidden layer features output by the text encoding branch into the diffusion model in the end-to-end generation model for denoising to obtain denoised features;

[0078] The diffusion model is a generative model based on deep learning, and its core lies in gradually removing noise to restore the target content. In this step, the controlled noise and the text hidden layer features are merged and then input into the diffusion model. The controlled noise contains the key frame information extracted from the video sample, and this information provides specific visual guidance for the model, while the text hidden layer features provide semantic-level guidance, enabling the generated video content to accurately reflect the requirements of the effect description text.

[0079] In one embodiment, it is assumed that the training dataset contains a video sample showing a new dress, and the corresponding effect description text "The model walks gracefully in the sun, and the skirt flutters in the wind". In steps S2210 and S2220, the image encoding branch extracts visual features such as the style, color, and texture of the dress, and the text encoding branch extracts semantic information such as the scene, actions, and dynamic effects to be shown in the video. Controlled noise is constructed by replacing some of the image frames in the video sample with noise frames, and these noise frames contain key frame information. When this information is input into the diffusion model, the model uses its denoising ability to gradually remove the noise while retaining the information related to the product image and the effect description text, and finally generates a hidden layer representation of the video content that meets the requirements.

[0080] In another embodiment, the training dataset may contain a video sample showing Christmas-themed home decorations, and the corresponding effect description text "The decorations are placed beside the Christmas tree, and there are snowflakes falling and Christmas music around". Similarly, the image encoding branch extracts the appearance features of the decorations, and the text encoding branch extracts the scene and atmosphere requirements. Controlled noise is constructed by replacing specific frames, and during the denoising process, the diffusion model generates a hidden layer representation of the video content that meets the description based on this information.

[0081] The denoising process of the diffusion model is a gradually refined process. Starting from the initial noise features, the model removes the noise through multiple iterations, and at the same time, using the key frame information in the controlled noise and the semantic information of the text hidden features, gradually constructs a hidden layer representation that is closer and closer to the target video as the denoising features. This process not only ensures the visual coherence of the generated video but also guarantees a high degree of consistency between the video content and the effect description text.

[0082] Step S2240: Use the decoder in the end-to-end generation model to decode the denoising features to obtain an inference video presenting the playback display effect indicated by the effect description text.

[0083] Using the decoder in the end-to-end generation model to decode the denoising features can generate a video that meets the playback display effect indicated by the effect description text. By converting the hidden layer features generated by the model, that is, the denoising features, into specific video content, it ensures that the visual effect and semantic content of the generated video are highly consistent with the merchant's requirements.

[0084] The role of the decoder in the end-to-end generation model is to gradually transform the latent features after denoising by the diffusion model into specific video frames in order to obtain the entire inference video. These latent features have retained semantic and visual information related to the product image and the effect description text during the denoising process. The task of the decoder is to restore this information into intuitive video content. The decoder can adopt technologies such as generative adversarial networks (GANs) or variational autoencoders (3DVAE) to generate high-quality video frames by learning the distribution of the data.

[0085] In one embodiment, assume that after being processed by the diffusion model, a set of latent features showing a new style dress is obtained. These features contain information such as the model's pose, the style, color, texture of the dress, and the background environment. The decoder will gradually decode these latent features into specific video frames to generate a video where the model walks gracefully, the skirt of the dress fluttering in the wind, and the scene is set outdoors on a sunny day. The finally generated video not only shows the appearance of the product but also enhances the attractiveness of the product through dynamic effects.

[0086] In another embodiment, if the training dataset contains a video sample showing Christmas-themed home decorations and the corresponding effect description text "The decorations are placed beside the Christmas tree, with snowflakes falling and Christmas music playing around", the decoder will generate a video based on the denoised latent features. In the video, the home decorations are placed beside a beautifully decorated Christmas tree, snowflakes are slowly falling, and warm Christmas music is playing in the background, creating a strong festive atmosphere.

[0087] When generating the video, the decoder will consider the coherence and smoothness of the video. It not only needs to ensure that each frame meets the requirements of the effect description text but also needs to guarantee a natural and smooth transition between frames. To this end, the decoder may adopt temporal consistency techniques to ensure that the actions and scene changes in the video conform to physical laws and visual habits.

[0088] Through the above embodiments, the present application has achieved significant technical advantages in the end-to-end generation model of the present application by replacing noise frames with image frames during the training process. First, this operation enables the model to refer to actual image frame information during the generation process, thereby better retaining the key visual features and dynamic effects of the product. By introducing controlled noise, the model can not only learn how to recover the required video content from the noise, but also utilize the key frame information during the generation process to ensure the accuracy and coherence of the generated video. This method of combining actual image frames with noise data provides the model with richer learning materials, enabling it to generate more realistic and merchant-demand-compliant video content. In addition, this technology also enhances the adaptability of the model to different products and scenarios, improves the quality and diversity of the generated videos, provides an efficient and personalized target video generation solution for merchants, and greatly enhances the flexibility and attractiveness of e-commerce marketing.

[0089] Based on any embodiment of the method of the present application, replacing the noise frames corresponding in time sequence in the noise data with some image frames in the video sample includes:

[0090] Step S2241: Detect each image frame of the video sample to determine at least one key frame, where the key frame is the first, middle, and / or last image frame in the video sample;

[0091] In this step, each image frame of the video sample is detected to determine the key frames therein. A key frame refers to an image frame with important visual information or representativeness in the video sample, usually including the first, middle, and / or last image frame of the video. These key frames play a crucial role in the video generation process because they provide reference points for the key visual features and dynamic effects of the product.

[0092] In specific implementation, the per-image-frame detection can be achieved by various technical means. In one embodiment, computer vision techniques such as edge detection, feature point extraction, or optical flow analysis can be used to identify the frames with significant visual changes in the video as key frames. These techniques can help the model identify important moments in the video, such as the movement changes of the model, scene transitions, or the key display angles of the product. In another embodiment, based on time series analysis, the video can be divided into multiple time periods, and a representative frame is selected from each time period as the key frame. This method can ensure that the key frames are evenly distributed in time, thereby providing more comprehensive visual information for the generated video.

[0093] For example, when processing a video sample showing a new dress, the model may detect frames where the model starts walking, turns around, or shows details of the dress as key frames. These key frames not only contain the appearance features of the dress but also show the model's movements and scene changes. By replacing these key frames at the corresponding positions in the noise data, the model can generate a more realistic target video that meets the merchant's requirements.

[0094] Step S2242: Determine the timing of the key frame in the video sample, and determine the corresponding position in the noise data according to this timing;

[0095] After determining the key frames, the model records the timing positions of these key frames in the video sample. These timing positions will be used as corresponding positions in the subsequent steps to replace the corresponding noise frames in the noise data with key frames. In this way, the model can utilize the actual image frame information during the generation process, thereby better retaining the key visual features and dynamic effects of the product and ensuring the accuracy and coherence of the generated video.

[0096] Step S2243: Replace the noise frame corresponding to the corresponding position in the noise data with the corresponding key frame.

[0097] After steps S2241 and S2242, the model has determined the key frames in the video sample and their timing positions in the video sequence. These key frames contain important visual information of the product, such as the appearance of the product, action display, and scene changes, etc. In step S2243, the model replaces the corresponding noise frames in the noise data with key frames according to these timing positions. This replacement operation enables the generation model to refer to the actual image frame information during the subsequent denoising and decoding processes, thereby generating more realistic video content that meets the merchant's requirements.

[0098] In one embodiment, assume that the video sample is a video showing a new dress. The model determines key frames such as when the model starts walking, turns around to show details of the dress, etc. through frame-by-frame detection. These key frames are recorded at specific timing positions in the video sample. In step S2243, the model replaces the noise frames corresponding to these timing positions in the noise data with key frames. In this way, the generation model can utilize the information of these key frames during subsequent processing to ensure that the movements of the model and the display effect of the dress in the generated video are highly consistent with the original video sample.

[0099] In another embodiment, if the video sample is a video showing Christmas-themed home decorations, the model may detect key frames such as decorations placed beside a Christmas tree and snowflakes starting to fall. These key frames are also replaced at the corresponding positions in the noise data. In this way, the generation model can better understand and reproduce the scenes and atmosphere in the video, and generate a target video that meets the requirements of the effect description text.

[0100] By performing the steps in the above embodiments, the technical solution of the present application has achieved significant technical advantages, including but not limited to:

[0101] First, by replacing specific noise frames in the noise data with key frames in the video sample, the model can directly utilize the actual image frame information during the generation process, thereby significantly improving the accuracy and coherence of the generated video. This operation not only retains the key visual features and dynamic effects of the product, but also provides a clear visual reference point for the generation model, enabling the generated video to more accurately reflect the appearance, action display, and scene changes of the product.

[0102] Second, this replacement operation enhances the model's adaptability to complex scenes and dynamic changes. The introduction of key frames provides the model with rich semantic and visual information, making the generated video not only highly consistent with the product image visually, but also accurately meet the requirements of the effect description text semantically. This greatly improves the quality and diversity of the generated video, enabling it to better meet the personalized needs of merchants for the target video.

[0103] In addition, by embedding key frames in the noise data, the model can more effectively utilize this information during the denoising and decoding processes, thereby generating more realistic video content that meets the needs of merchants. This process not only optimizes the efficiency of video generation, but also reduces the difference between the generated video and the target video, improving the training effect and convergence speed of the model.

[0104] Generally speaking, the technical solution in the above embodiments significantly improves the performance and effect of the end-to-end generation model in the target video generation task through the introduction and replacement operations of key frames. This not only provides merchants with an efficient and flexible video generation solution, but also greatly improves the efficiency and effect of e-commerce marketing, and has important commercial application value.

[0105] Based on any embodiment of the method of the present application, publishing the target video to a product promotion platform for access and playback includes:

[0106] Step S3310, creating a video playback page corresponding to the target video on the product promotion platform;

[0107] The purpose of creating a video playback page is to provide a dedicated display space for the target video so that it can be presented to consumers in the best way. The video playback page usually includes a video player for playing the target video, and at the same time, it will also display information related to the video, such as detailed product descriptions, purchase links, user reviews, etc. These elements together constitute a complete shopping experience, which can not only attract consumers' attention but also promote sales conversion.

[0108] In one embodiment, the video playback page can be a dedicated product details page in the merchant's independent website. Merchant users can select to embed the target video into the details page of a specific product on the terminal device 90 through the operation interface. A video playback area can also be automatically generated on this page, and various functions of the video player, such as play, pause, volume control, etc., can be configured. In addition, the system will also add product information related to the video content, such as product name, price, promotional activities, etc., so that consumers can quickly understand the product details while watching the video.

[0109] In another embodiment, the video playback page can be a special page for a promotional activity. Merchant users can select to publish the target video to a specific promotional activity on the terminal device 90. According to the merchant's selection, a video playback area is created on the promotional activity page, and the target video is embedded therein. The special page usually displays multiple products and videos related to the promotional activity, attracting consumers' attention through centralized display. In addition, a navigation bar and a search function can also be set on the page to facilitate consumers to quickly find the videos and products they are interested in.

[0110] Step S3320: Upload the video file of the target video to the server of the product promotion platform;

[0111] The upload of the video file can be achieved through various technical means to ensure that the video file can be transmitted to the server efficiently and securely. In one embodiment, the FTP (File Transfer Protocol) or HTTP / HTTPS protocol can be used to upload the video file from the terminal device 90 of the merchant user to the server 80. These protocols can provide a stable data transmission channel to ensure that no data loss or damage occurs during the upload of the video file. For example, after the merchant user completes the generation of the target video on the terminal device 90, the user can select the upload option through the user interface, and the computer device automatically uses the FTP protocol to transmit the video file to the specified storage location of the server 80.

[0112] In another embodiment, the product promotion platform can provide dedicated API interfaces for receiving video files uploaded by merchant users. Merchant users can upload video files to the server by calling these API interfaces. The API interfaces can not only implement the upload of video files, but also provide functions such as file verification and format conversion to ensure that the uploaded video files meet the requirements of the platform. For example, the platform can require that the resolution, format, and duration of the video file meet specific standards. The API interfaces will automatically verify the files when receiving them, and files that do not meet the requirements will be rejected for upload.

[0113] In addition, to improve the loading speed and playback quality of the video, the video file can be compressed or its format can be converted during the upload process. For example, if the original video file has a high resolution, it can be compressed to a resolution suitable for web playback while retaining the main visual content of the video. This compression operation can reduce the size of the video file without significantly degrading the video quality, thereby accelerating the video loading speed and enhancing the user experience.

[0114] After the upload is completed, the server 80 will store the video file in a specific storage location and record the relevant information of the video file, such as the file path, file name, upload time, etc. This information will be used to set the access link on the video playback page in subsequent steps to ensure that consumer users can access and play the target video by clicking the link.

[0115] Step S3330: Set an access link on the video playback page so that users can access and play the target video through the access link.

[0116] The setting of the access link can be achieved through various technical means. In one embodiment, after receiving the uploaded target video file, the server will generate a unique access link and embed it in the video playback page. This link can directly point to the storage location of the video file or provide the video content through a video playback service. When a consumer user accesses the video playback page and clicks the link, the target video can be played in the video player on the page.

[0117] In another embodiment, the video playback page can adopt embedded video playback technology to directly embed the target video into the page without a separate access link. In this case, after the page is loaded, the target video will play automatically, or the user can manually play the video through the play button on the page. This embedded playback method can provide a smoother and seamless user experience and enhance the attractiveness of the video.

[0118] In addition, to further enhance the user experience, the access link can be configured to support different playback options. For example, functions such as full-screen playback, volume control, and playback progress bar can be provided, enabling users to adjust video playback according to their needs and preferences. At the same time, the access link can also interact with other elements on the video playback page (such as product purchase links, user comment areas, etc.) to provide users with a complete shopping experience.

[0119] When setting the access link, the video loading speed and playback quality can also be considered. To this end, CDN (Content Delivery Network) technology can be adopted to cache the video files on servers in multiple geographical locations, thus achieving fast distribution. When the user clicks the access link, the system will automatically select the server closest to the user to provide the video content, reducing the loading time and buffering time to ensure smooth video playback.

[0120] Through the above embodiments, the present application significantly improves the publishing efficiency and user experience of the target video through steps such as creating a video playback page, efficiently uploading video files, and setting access links. The creation of the video playback page provides an optimized display space for the target video, integrates video playback and related product information, enhances the shopping experience, and promotes sales conversion. At the same time, a variety of technical means are used to safely and efficiently upload the video files to the server, and optimize the video loading speed and playback quality through compression or format conversion. In addition, setting the access link or adopting embedded playback technology, combined with CDN optimization for distribution, ensures fast and smooth video playback, further enhancing the user experience. These technical advantages together provide merchants with an efficient, flexible, and user-friendly video marketing solution, enhancing the commercial value and market competitiveness of the e-commerce platform.

[0121] Based on any embodiment of the method of the present application, before responding to the user confirmation event, it includes:

[0122] Step S4100: Classify and identify the product image to determine the category to which the product belongs;

[0123] To achieve personalized target video generation, in this embodiment, the product image is first classified and identified to determine the category to which the product belongs, so as to call the video style template according to the product category to beautify the target video generated by the end-to-end generation model, ensuring that the generated video can accurately match the characteristics of the product, thereby enhancing the attractiveness and marketing effect of the video.

[0124] Specifically, classification and recognition can be achieved through various technical means. In one embodiment, a convolutional neural network (CNN) in deep learning can be used to extract features and classify product images. The CNN can automatically learn the key features in the image and map them to predefined product categories. For example, for a product image of a dress, the CNN can identify that it belongs to the "clothing" category and further subdivide it into the "dress" subcategory. This classification accuracy provides an accurate basis for subsequent video style selection and adjustment.

[0125] In another embodiment, traditional computer vision techniques can be combined with machine learning algorithms for classification and recognition. For example, by extracting the color histogram, texture features, and shape features of the image, and combining classification algorithms such as support vector machines (SVM) or decision trees, the product image can be classified. This method can provide faster processing speed in some cases and is suitable for application scenarios with high real-time requirements.

[0126] In addition, pre-trained classification models such as ResNet or Inception can be used to classify product images. These pre-trained models have been trained on large-scale image datasets and have strong generalization capabilities, and can be directly applied to the classification and recognition tasks of product images to further improve the accuracy and efficiency of classification.

[0127] Step S4200: According to the category information, select a video style template that matches the category from a preset style library;

[0128] According to the category information of the product image, selecting a video style template that matches the category from a preset style library can be used to achieve personalized target video generation. By selecting appropriate video styles for different categories of products, the attractiveness and marketing effect of the video can be enhanced.

[0129] Specifically, a variety of video style templates are stored in the preset style library, and each template is optimized for a specific product category to ensure that the generated video can accurately match the characteristics of the product. For example, for products in the "clothing" category, the style library may contain video templates in different styles such as fashion, elegance, and leisure; while for products in the "home decoration" category, there may be templates in styles such as warm, festive, and simple.

[0130] When selecting a video style template, according to the product category information determined in step S4100, retrieve the template that matches the category from the style library. This process can be achieved through a simple matching algorithm. For example, match the product category with the template categories in the style library one by one and select the template corresponding to the most suitable product category.

[0131] For example, if the product image is classified as "dress", select a video style template related to the "clothing" category from the style library, such as a template with a "fashionable and elegant" style. Such a template may include scene settings, color combinations, and dynamic effects suitable for displaying clothing, such as a catwalk scene of models on a fashion runway, bright colors, and smooth dynamic effects. If the product image is classified as "home decoration", select a video style template related to the "home" category, such as a template with a "warm and festive" style. Such a template may include scene settings suitable for displaying home decorations, such as beside a beautifully decorated Christmas tree, soft colors, and warm dynamic effects.

[0132] Step S4300: Adjust the configuration information in the video style template according to the semantic information of the effect description text, where the configuration information includes any one or more of rhythm, color, and dynamic effects;

[0133] Adjust the configuration information in the video style template according to the semantic information of the effect description text. These configuration information include rhythm, color, and dynamic effects, etc. The video style template is preset and used to generate a target video with specific visual and dynamic characteristics for different categories of products. These templates are selected according to the product category in step S4200, providing a basic style framework for subsequent video generation.

[0134] The video style template is stored in a preset style library, and each template is optimized for a specific product category. For example, templates for the "clothing" category may include styles such as fashionable, elegant, or casual, while templates for the "home decoration" category may include styles such as warm, festive, or minimalist. These templates define the basic visual and dynamic characteristics of the video, such as scene settings, color combinations, and dynamic effects, etc.

[0135] Considering the need to improve the personalization ability of the video style template, adjust the selected video style template according to the semantic information of the effect description text. The effect description text provides specific instructions on the playback display effect of the target video, such as scenes, actions, and atmospheres, etc. By parsing these semantic information, the configuration information in the video style template can be adjusted to ensure that the generated video can accurately reflect the requirements of the effect description text.

[0136] In one embodiment, if the effect description text emphasizes the rhythm of the video, such as "the model walks quickly in the sun", the rhythm configuration in the video style template can be adjusted to make the generated video have a faster editing rhythm and dynamic effects. If the text describes specific color requirements, such as "the overall color tone of the video is bright warm colors", the color configuration can be adjusted to make the color style of the video consistent with the description. Similarly, if the text mentions specific dynamic effects, such as "snowflakes slowly falling", the dynamic effect configuration can be adjusted to include the corresponding visual effects in the video.

[0137] Step S4400, beautify the target video according to the adjusted video style template.

[0138] In the previous steps, the classification and recognition of the product image have been completed, the video style template matching the product category has been selected, and the configuration information (such as rhythm, color, and dynamic effects) in the template has been adjusted according to the semantic information of the effect description text. In this step, these adjusted configuration information will be applied to the beautification process of the target video.

[0139] In one embodiment, assume that the product image is classified as "dress", and the effect description text requires the video to have the rhythm of "walking quickly in the sun" and the color style of "bright warm colors". In this step, the target video can be beautified according to the adjusted video style template. Specifically, the model will adjust the editing rhythm of the video to make it more in line with the dynamic effect of walking quickly; at the same time, adjust the color of the video so that its overall color tone tends to bright warm colors, thereby enhancing the visual attraction of the video.

[0140] In another embodiment, if the product image is classified as "home decoration", and the effect description text requires the video to have the dynamic effect of "snowflakes slowly falling" and the atmosphere of "warm festival", beautify the video according to these requirements. Specifically, the dynamic effect of snowflakes falling can be added to the video, and the color and scene settings of the video can be adjusted to make it more festive, thereby enhancing the emotional expression and attraction of the video.

[0141] In addition, it can also involve adjusting other visual elements of the video, such as light and shadow effects, background music, etc. For example, if the effect description text mentions specific light and shadow effects, the model can adjust the light and shadow in the video to make it more in line with the requirements in the description. Similarly, if the text mentions background music, the model can add the corresponding music to the video to further enhance the appeal of the video.

[0142] By performing the steps in the above embodiments, the technical solution of the present application has achieved significant technical advantages in personalized target video generation. First, by classifying and identifying product images, the category to which the product belongs can be accurately determined, providing an accurate basis for subsequent video style selection and adjustment. This classification and identification uses deep learning or traditional computer vision techniques to ensure high-precision and high-efficiency classification results. Second, by selecting a matching video style template from a preset style library according to the product category, personalized video generation for different product categories is realized, enhancing the attractiveness and marketing effect of the video. Further, according to the semantic information of the effect description text, the configuration information in the video style template, such as rhythm, color, and dynamic effects, is adjusted, so that the generated video can accurately reflect the needs and creativity of the merchant. Finally, through beautification processing, the adjusted configuration information is applied to the target video, enhancing the visual and emotional attractiveness of the video and ensuring a high degree of visual and semantic consistency of the generated video. These technical advantages together provide a merchant with an efficient, flexible, and personalized target video generation solution, greatly improving the efficiency and effect of e-commerce marketing.

[0143] Based on any embodiment of the method of the present application, before responding to the user confirmation event, it includes:

[0144] Step S5100: Perform a quality assessment on the target video to determine the dimension scores corresponding to multiple evaluation dimensions, where the evaluation dimensions include any one or any combination of the following: video smoothness, matching degree with the effect description text, and integrity of product display;

[0145] Performing a quality assessment on the generated target video can ensure that the video can meet the expected effects and quality standards of the merchant. The quality assessment is achieved by determining the dimension scores corresponding to multiple evaluation dimensions, and these evaluation dimensions include but are not limited to video smoothness, matching degree with the effect description text, and integrity of product display. Through the comprehensive evaluation of these dimensions, the quality and effect of the target video can be comprehensively measured.

[0146] In specific implementation, the evaluation of video smoothness can be carried out by analyzing whether the transition between video frames is natural and smooth. For example, video analysis tools or other corresponding smoothness scoring models can be used to detect whether there are problems such as stuttering, frame skipping or other issues affecting the viewing experience in the video, and output the corresponding quality score. The evaluation of the matching degree with the effect description text is completed through a pre-trained matching degree evaluation model. The matching degree evaluation model can parse the key information in the text through natural language processing technology and compare it with the visual elements in the video to determine the corresponding quality score. The evaluation of the integrity of product display can also be scored and determined through a corresponding preset integrity evaluation model. The integrity evaluation model mainly focuses on whether the video comprehensively displays all the important features of the product and outputs the corresponding quality score, and measures the integrity of product display through the quality score, such as whether all the key angles and functions of the product are displayed.

[0147] In one embodiment, a pre-integrated automated evaluation system can be used to evaluate the quality of the target video. The system will automatically calculate the score of each evaluation dimension according to the preset evaluation criteria. For example, for video smoothness, the system can analyze the stability of the video frame rate; for matching degree, the system can extract the key visual elements in the video and compare them with the text description; for the integrity of product display, the system can check whether all the important features of the product are included in the video.

[0148] In another embodiment, the quality evaluation can be carried out in combination with manual review. Professionals can subjectively evaluate the target video according to the evaluation criteria and give the quality score of each evaluation dimension. This manual review method can supplement the deficiencies of automated evaluation and ensure that the evaluation results are more accurate and comprehensive.

[0149] Step S5200, when the quality score corresponding to the evaluation dimension does not reach the preset threshold, call the best optimization prompt text among the multiple optimization prompt texts corresponding to this evaluation dimension, merge it into the effect description text, and re-call the end-to-end generation model to generate a target video according to the adjusted effect description text and the product image;

[0150] When the quality evaluation result shows that the quality score of a certain evaluation dimension (such as video smoothness, matching degree with the effect description text or integrity of product display) does not reach the preset threshold, call the best optimization prompt text among the multiple optimization prompt texts corresponding to this evaluation dimension. This optimization prompt text will be merged into the original effect description text, and then the end-to-end generation model will be re-called to regenerate the target video according to the adjusted effect description text and the product image.

[0151] In a specific implementation, the optimization prompt text is pre-prepared, and multiple optimization prompt texts under the same evaluation dimension can be sorted according to the user ratings provided by users who use the end-to-end generation model to generate target videos, so as to determine the best optimization prompt text among them. The optimization prompt text is used to provide specific optimization suggestions for problems in different evaluation dimensions. For example, if the video smoothness does not reach the threshold, the optimization prompt text may include suggestions for improving the video frame rate stability; if the matching degree is insufficient, the optimization prompt text may provide a more detailed description to help the model better understand the correspondence between the video content and the text description; if the integrity of the product display is insufficient, the optimization prompt text may emphasize the need to display specific angles or functions of the product.

[0152] In one embodiment, it is assumed that the score of the video smoothness does not reach the preset threshold, and the best optimization prompt text related to the video smoothness is selected from the preset optimization prompt text library. This optimization prompt text may include optimization suggestions for the video frame rate stability, encoding format, or network transmission conditions. These optimization suggestions are merged into the original effect description text, and then the end-to-end generation model is called again to generate a new target video.

[0153] In another embodiment, if the matching degree with the effect description text does not reach the threshold, the optimization prompt text related to the matching degree is called. These optimization prompt texts may include suggestions for enhancing the semantic consistency between the video content and the text description. These optimization suggestions are merged into the original text, and the video is regenerated to ensure that the video content can more accurately reflect the requirements of the effect description text.

[0154] In addition, the optimization prompt text can also be dynamically generated according to different evaluation dimensions and problem types. For example, through natural language processing technology, the evaluation results can be analyzed to automatically generate targeted optimization suggestions and merge them into the effect description text to guide the model to regenerate the video.

[0155] Step S5300: Obtain the user evaluation information corresponding to the regenerated target video, and determine the user rating of the best optimization prompt text according to the user evaluation information;

[0156] After regenerating the target video, the user's evaluation information on the regenerated target video can be obtained from the user who submitted the effect description text, and the user rating of the best optimization prompt text used can be determined according to this evaluation information. This process is an important link to ensure the continuous improvement of the video generation quality. It verifies the effectiveness of the optimization measures through the user's feedback and provides data support for subsequent optimizations.

[0157] In a specific implementation, user evaluation information can be collected in various ways. In one embodiment, a user feedback module can be set on the video playback page, allowing users to rate or comment on various aspects of the regenerated target video. For example, users can rate the smoothness of the video, the matching degree with the description text, and the integrity of the product display to determine the user ratings corresponding to each evaluation dimension.

[0158] In another embodiment, a questionnaire survey method can be adopted to send a questionnaire containing specific questions to the users who trigger the regeneration of the target video to obtain more detailed evaluation information. For example, the questionnaire can ask about the user's satisfaction with the product display in the target video, the consistency between the video content and the description text, and the smoothness of the viewing experience, etc. In this way, more comprehensive user feedback can be collected to quantify the user ratings corresponding to each evaluation dimension according to the user evaluation information.

[0159] According to the collected user evaluation information, the effect of each optimization prompt text can be quantitatively evaluated. For example, if the user is satisfied with the smoothness of a certain optimized video, the corresponding optimization prompt text will receive a higher user rating. On the contrary, if the user feedbacks that there are still problems in some aspects of the optimized video, the user rating of the relevant optimization prompt text will be lower. This user rating mechanism based on user feedback can ensure that the actual effect of the optimization measures is accurately evaluated, providing a reliable basis for subsequent optimization.

[0160] Step S5400: Re - sort the multiple optimization prompt texts for each evaluation dimension according to the historical user ratings corresponding to that evaluation dimension to re - determine the best optimization prompt text among them.

[0161] Re - sorting the optimization prompt texts according to the historical user ratings for each evaluation dimension to re - determine the best optimization prompt text ensures that the optimization prompt text can be dynamically adjusted according to the actual effect, thereby continuously improving the quality of the generated video.

[0162] The historical user ratings for each evaluation dimension (such as video smoothness, matching degree with the effect description text, and integrity of product display) are obtained by collecting the evaluation information of a large number of users on the videos generated multiple times. These ratings reflect the users' subjective feelings and satisfaction with the video quality, and indirectly also reflect the improvement expectations for the target videos generated according to the corresponding optimization prompt texts. Therefore, this kind of user rating is relevant for the valuable evaluation of the optimization prompt text. In a specific implementation, a rating record can be maintained for each optimization prompt text to record the corresponding user rating after each use. For example, for the evaluation dimension of video smoothness, each time the user rates the smoothness of the optimized video, this rating will be recorded in the rating record of the corresponding optimization prompt text.

[0163] In this case, for each optimization hint text in each evaluation dimension, the total score of each optimization hint text can be determined according to all its historical user scores. In one embodiment, a simple average score calculation method can be adopted. After adding up all the historical user scores and dividing by the number of scoring times, the average score of each optimization hint text is obtained as the total score. In another embodiment, a weighted average score method can be adopted, and different weights are given according to the timeliness or importance of the scores, so as to more accurately reflect the actual effect of the optimization hint text.

[0164] After determining the total score of each optimization hint text, all the optimization hint texts under the same evaluation dimension are sorted according to the total score. The sorting process can be implemented through a simple comparison algorithm. For example, all the optimization hint texts are arranged in descending order according to the total score. After sorting, the optimization hint text at the front of the list is the current best optimization hint text, which is used to serve subsequent users. This dynamic sorting mechanism can ensure that the optimization hint text is always adjusted according to the actual effect, so as to provide more effective optimization suggestions in the subsequent video generation process.

[0165] By executing the steps in the above embodiments, the technical solution of the present application has achieved significant technical advantages in the quality optimization of target video generation, including but not limited to:

[0166] First of all, through multi-dimensional quality evaluation of the generated target video, the fluency of the video, the matching degree with the effect description text, and the integrity of the product display can be comprehensively measured. This comprehensive evaluation method ensures that the video can meet high-quality standards in multiple key aspects and meet the expected effects and quality requirements of merchants.

[0167] Secondly, when the evaluation result shows that the quality score of a certain dimension does not reach the preset threshold, the best optimization hint text corresponding to this dimension is called and merged into the effect description text to regenerate the video. This process can not only specifically solve the problems in video generation, but also continuously improve the quality and effect of video generation by dynamically adjusting the optimization hint text.

[0168] Furthermore, by obtaining the evaluation information of users on the regenerated video and determining the user scores of the optimization hint text according to these evaluations, the optimization effect evaluation based on user feedback is realized. This mechanism can ensure that the actual effect of the optimization measures is accurately verified and provide reliable data support for subsequent optimizations.

[0169] Finally, re - rank the optimized prompt text based on the historical user ratings for each evaluation dimension, and dynamically determine the best optimized prompt text. This dynamic adjustment mechanism can ensure that the optimized prompt text is always optimized according to the actual effect, so as to provide more effective optimization suggestions in the subsequent video generation process, further improving the quality of the generated video and the user experience.

[0170] In summary, the above embodiments significantly improve the quality and efficiency of target video generation. Through the dynamic optimization and user feedback mechanism, it ensures high - quality performance of the generated video in multiple key dimensions, provides an efficient, flexible and user - friendly target video generation solution for merchants, and enhances the marketing effect and user experience of the e - commerce platform.

[0171] Please refer to Figure 3 , a video end - to - end generation device provided according to an aspect of the present application includes a data acquisition module 3100, a video generation module 3200, and a video publishing module 3300. Among them, the data acquisition module 3100 is configured to acquire a product image and an effect description text corresponding to the playback display effect indicating the target video to be generated; the video generation module 3200 is configured to input the product image and the effect description text into an end - to - end generation model, and the model generates a target video with the product image as the basic content and presented according to the playback display effect indicated by the effect description text; the video publishing module 3300 is configured to respond to a user confirmation event and publish the target video to a product promotion platform for access and playback.

[0172] Based on any embodiment of the method of the present application, prior to the data acquisition module 3100, the device further includes: a sample calling module configured to acquire video samples in a training dataset and an effect description text corresponding to the playback display effect describing each video sample; a training implementation module configured to call the video samples and the corresponding effect description text to implement training on the end - to - end generation model to obtain an inference video corresponding to the video sample; a training supervision module configured to supervise the training process of each video sample using a preset loss function and train the end - to - end generation model to a convergent state.

[0173] Based on any embodiment of the method of the present application, the training implementation module includes: a feature representation module configured to input the video samples in the training dataset and the effect description text into the image encoding branch and the text encoding branch of the end-to-end generation model respectively to extract their respective hidden layer features; a noise processing module configured to construct noise data corresponding to the hidden layer features for each image frame in the video sample, and use some image frames in the video sample to replace the noise frames corresponding in time sequence in the noise data to obtain controlled noise; a denoising inference module configured to input the controlled noise and the hidden layer features output by the text encoding branch into the diffusion model in the end-to-end generation model for denoising to obtain denoised features; a decoding and generation module configured to use the decoder in the end-to-end generation model to decode the denoised features to obtain an inference video presenting the playback display effect indicated by the effect description text.

[0174] Based on any embodiment of the method of the present application, the decoding and generation module includes: a video detection module configured to perform frame-by-frame detection on the video sample to determine at least one key frame, where the key frame is the first, middle, and / or last image frame in the video sample; a time sequence alignment module configured to determine the time sequence of the key frame in the video sample and determine the alignment position in the noise data according to this time sequence; a noise replacement module configured to replace the noise frame corresponding to the alignment position in the noise data with the corresponding key frame.

[0175] Based on any embodiment of the method of the present application, the video publishing module 3300 includes: a page creation module configured to create a video playback page corresponding to the target video on the product promotion platform; a video upload module configured to upload the video file of the target video to the server of the product promotion platform; an access configuration module configured to set an access link on the video playback page so that users can access and play the target video through the access link.

[0176] Based on any embodiment of the method of the present application, prior to the video publishing module 3300, the device further includes: a style classification module configured to classify and identify the product image to determine the category to which the product belongs; a template matching module configured to select a video style template matching the category from a preset style library according to the category information; a template configuration module configured to adjust the configuration information in the video style template according to the semantic information of the effect description text, where the configuration information includes any one or any combination of rhythm, color, and dynamic effects; a video beautification module configured to beautify the target video according to the adjusted video style template.

[0177] Based on any embodiment of the method of the present application, prior to the video publishing module 3300, the device further includes: a quality evaluation module configured to evaluate the quality of the target video and determine the dimension scores corresponding to multiple evaluation dimensions, where the evaluation dimensions include any one or more of the following: video smoothness, matching degree with the effect description text, and integrity of product display; a regeneration module configured to, when the quality score corresponding to the evaluation dimension does not reach the preset threshold, call the best optimization hint text among the multiple optimization hint texts corresponding to the evaluation dimension, merge it into the effect description text, and then call the end-to-end generation model again to generate a target video according to the adjusted effect description text and product images; a user evaluation module configured to obtain the user evaluation information corresponding to the regenerated target video and determine the user score of the best optimization hint text according to the user evaluation information; a sorting optimization module configured to re-sort the multiple optimization hint texts of each evaluation dimension according to the historical user scores corresponding to the evaluation dimension to re-determine the best optimization hint text among them.

[0178] Another embodiment of the present application further provides a video end-to-end generation device. As Figure 4 shown, it is a schematic internal structure diagram of the video end-to-end generation device. The video end-to-end generation device includes a processor, a computer-readable storage medium, a memory, and a network interface connected through a system bus. Among them, the computer-readable non-volatile storage medium of the video end-to-end generation device stores an operating system, a database, and computer-readable instructions. Information sequences can be stored in the database. When the computer-readable instructions are executed by the processor, the processor can implement a video end-to-end generation method.

[0179] The processor of the video end-to-end generation device is used to provide computing and control capabilities to support the operation of the entire video end-to-end generation device. Computer-readable instructions can be stored in the memory of the video end-to-end generation device. When the computer-readable instructions are executed by the processor, the processor can execute the video end-to-end generation method of the present application. The network interface of the video end-to-end generation device is used to connect and communicate with the terminal.

[0180] Those skilled in the art can understand that Figure 4 the structure shown in

[0181] is only a block diagram of some structures related to the solution of the present application and does not constitute a limitation on the video end-to-end generation device to which the solution of the present application is applied. The specific video end-to-end generation device may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0181] In this embodiment, the processor is used to execute Figure 3For the specific functions of each module in [the device], the memory stores program codes and various types of data required to execute the above-mentioned modules or sub-modules. The network interface is used to implement data transmission between user terminals or servers. In the non-volatile readable storage medium of this embodiment, the program codes and data required to execute all modules in the video end-to-end generation device of the present application are stored, and the server can call the program codes and data of the server to execute the functions of all modules.

[0182] The present application also provides a non-volatile readable storage medium storing computer-readable instructions. When the computer-readable instructions are executed by one or more processors, the one or more processors are caused to execute the steps of the video end-to-end generation method according to any embodiment of the present application.

[0183] The present application also provides a computer program product, including a computer program / instructions. When the computer program / instructions are executed by one or more processors, the steps of the method according to any embodiment of the present application are implemented.

[0184] Those of ordinary skill in the art can understand that to implement all or part of the processes in the methods of the above embodiments of the present application, it can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile readable storage medium. When the program is executed, it may include the processes of the embodiments of the above methods. Among them, the aforementioned storage medium can be a computer-readable storage medium such as a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM), etc.

[0185] In summary, the technical solution of the present application has significant application advantages in the field of e-commerce video generation. Through the end-to-end generation model, it can quickly and efficiently generate target videos that are accurately matched with the product images and play display effect descriptions, greatly reducing the video production cost and time, and improving the efficiency and quality of video generation. At the same time, this solution meets the diverse needs of different products and marketing scenarios by flexibly adjusting the video style and rhythm, enhancing the attractiveness and sales conversion rate of the target video. In addition, the present application has also optimized the publishing process of the target video, enabling merchants to more conveniently publish the generated videos to the product promotion platform, expanding the dissemination scope of the videos, and enhancing the market competitiveness of the products. Generally speaking, the technical solution of the present application not only provides an innovative video generation and promotion method for the e-commerce field, but also brings a more efficient, intuitive and vivid shopping experience to merchants and consumers, promoting the dual improvement of the commercial value and user experience of the e-commerce platform.

Claims

1. A video end-to-end generation method, characterized in that: include: Acquire a product image and an effect description text corresponding to the playback display effect of the target video to be generated; Input the product image and effect description text into an end-to-end generation model, and the model generates a target video based on the product image and presented according to the playback display effect indicated by the effect description text; In response to the user confirmation event, the target video is published to the product promotion platform for access and playback.

2. The video end-to-end generation method according to claim 1, characterized in that: Before obtaining the product image and the effect description text corresponding to the playback display effect of the video to be generated, it includes: Obtain video samples in the training data set and effect description text corresponding to the playback display effect of each video sample; Calling the video sample and the corresponding effect description text, training the end-to-end generation model, and obtaining the inference video corresponding to the video sample; A preset loss function is used to supervise the training process of each video sample, and the end-to-end generation model is trained to a convergence state.

3. The video end-to-end generation method according to claim 2, characterized in that: The video sample and the corresponding effect description text are called to train the end-to-end generation model to obtain the inference video corresponding to the video sample, including: Input the video samples in the training data set and the effect description text into the image encoding branch and the text encoding branch of the end-to-end generation model respectively to extract their respective hidden layer features; Constructing noise data corresponding to the hidden layer features corresponding to each image frame in the video sample, and replacing the noise frames corresponding to the time sequence in the noise data with some image frames in the video sample to obtain controlled noise; Inputting the controlled noise and the hidden layer features output by the text encoding branch into the diffusion model in the end-to-end generation model for denoising to obtain denoised features; The denoising features are decoded using a decoder in the end-to-end generation model to obtain an inference video presenting the playback display effect indicated by the effect description text.

4. The video end-to-end generation method according to claim 3, characterized in that: Replacing the noise frames corresponding to the time sequence in the noise data with some image frames in the video sample includes: Performing image frame detection on the video sample to determine at least one key frame, wherein the key frame is the first, middle and / or last image frame in the video sample; Determine the timing of the key frame in the video sample, and determine the reference position in the noise data according to the timing; The noise frame corresponding to the reference position in the noise data is replaced with a corresponding key frame.

5. The end-to-end video generation method according to any one of claims 1 to 4, characterized in that: Publishing the target video to the product promotion platform for access and playback, including: Creating a video playback page corresponding to the target video on the product promotion platform; Uploading the video file of the target video to the server of the commodity promotion platform; An access link is set on the video playing page so that the user can access and play the target video through the access link.

6. The end-to-end video generation method according to any one of claims 1 to 4, characterized in that: Before responding to user confirmation events, including: Classify and identify the commodity image to determine the category to which the commodity belongs; According to the category information, a video style template matching the category is selected from a preset style library; According to the semantic information of the effect description text, adjusting the configuration information in the video style template, the configuration information including any one or more of rhythm, color and dynamic effects; The target video is beautified according to the adjusted video style template.

7. The end-to-end video generation method according to any one of claims 1 to 4, characterized in that: Before responding to user confirmation events, including: Performing a quality assessment on the target video to determine dimension scores corresponding to a plurality of assessment dimensions, wherein the assessment dimensions include any one or more of the following: video fluency, matching degree with effect description text, and completeness of product display; When the quality score corresponding to the evaluation dimension does not reach a preset threshold, the best optimization prompt text among the multiple optimization prompt texts corresponding to the evaluation dimension is called, merged into the effect description text, and the end-to-end generation model is re-called to generate a target video according to the adjusted effect description text and the product image; Obtaining user evaluation information corresponding to the regenerated target video, and determining a user score for the best optimization prompt text according to the user evaluation information; According to the historical user scores corresponding to each evaluation dimension, multiple optimization prompt texts of the evaluation dimension are reordered to re-determine the best optimization prompt text therein.

8. A video end-to-end generation device, characterized in that: include: A data acquisition module, configured to acquire a product image and an effect description text corresponding to a playback display effect of a target video to be generated; A video generation module, configured to input the product image and effect description text into an end-to-end generation model, and the model generates a target video based on the product image and presented according to the playback display effect indicated by the effect description text; The video publishing model is configured to respond to a user confirmation event and publish the target video to a product promotion platform for access and playback.

9. A video end-to-end generation device, comprising a central processing unit and a memory, characterized in that: The central processing unit is used to call and run the computer program stored in the memory to execute the steps of the method according to any one of claims 1 to 7.

10. A non-volatile readable storage medium, characterized in that: It stores a computer program implemented according to the method described in any one of claims 1 to 7 in the form of computer-readable instructions, and when the computer program is called and executed by a computer, the steps included in the corresponding method are executed.

Images