Prompt word optimization method and device for generating video, electronic equipment and medium

Abstract

This invention provides a method, apparatus, electronic device, and medium for optimizing prompt words in video generation. The method includes: generating an initial generated video based on prompt words to be optimized; selecting reference videos from a high-quality video library based on the prompt words to be optimized; comparing the initial generated video and the reference videos to obtain a difference comparison result; and optimizing the prompt words to be optimized based on the difference comparison result to obtain optimized prompt words. The method provided by this invention generates an initial generated video using prompt words to be optimized; by comparing the initial generated video with the reference videos, objective difference comparison results are obtained, and the prompt words to be optimized are optimized accordingly. This effectively eliminates the gap between the short prompt words input by the user and the high-quality features trained on the text-based video model, achieving automated and fine-tuned adjustment of prompt words based on feedback from the generation effect, thereby significantly improving the generation quality of text-based videos and the accuracy of expressing user intent.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method, apparatus, electronic device, and medium for optimizing prompt words in video generation. Background Technology

[0002] Current mainstream solutions in the field of text-based video technology revolve around the training and inference of text-based video models. In the model training phase, a large-scale training dataset containing numerous text prompts and corresponding reference video samples is typically constructed first. Supervised learning is then used to teach the model the mapping relationship between text and video features to improve video generation quality. After training, high-quality models are selected for inference on a specific test dataset. In the inference generation phase, users input custom text prompts, which are then directly input into the trained model to generate and return videos. It's understandable that the prompts used in the training and testing phases are usually professionally designed, detailed text that conforms to the model's feature preferences, while the prompts actually input by users are often short, vague, and lack the key feature information required by the model. Currently, some technologies perform simple grammatical correction or keyword completion on the prompts to optimize them.

[0003] However, the optimization effect of simple grammatical correction or keyword completion of prompt words is limited, resulting in poor quality of the final video. Summary of the Invention

[0004] This invention provides a method, apparatus, electronic device, and medium for optimizing prompts in video generation, which addresses the shortcomings of existing technologies that simply optimize user-input prompts, resulting in limited optimization effects and ultimately poor video quality.

[0005] This invention provides a method for optimizing prompt words in video generation, comprising: Based on the prompts to be optimized, an initial video is generated. Based on the suggested words to be optimized, reference videos are selected from a high-quality video library; The initially generated video and the reference video are compared to obtain the difference comparison results; Based on the difference comparison results, the prompt words to be optimized are optimized to obtain optimized prompt words.

[0006] According to a method for optimizing prompt words in video generation provided by the present invention, the step of comparing the initially generated video and the reference video to obtain a difference comparison result includes: The initial generated video and the reference video are compared in terms of spatiotemporal features to obtain the spatiotemporal comparison result. The initially generated video is subjected to image perception quality assessment to obtain the quality assessment result; The semantic comparison between the prompt words to be optimized and the initially generated video is performed to obtain the semantic comparison results; The difference comparison result is obtained based on the quality assessment result and / or the semantic comparison result, as well as the spatiotemporal comparison result; The difference comparison results are used to characterize the degree of deviation of the initially generated video from the reference video in terms of multidimensional features.

[0007] According to a method for optimizing prompt words in video generation provided by the present invention, the step of performing spatiotemporal feature comparison between the initially generated video and the reference video to obtain a spatiotemporal comparison result includes: The initial generated video and the reference video are sampled respectively to obtain a first generated video frame set and a reference video frame set; Extract the spatiotemporal feature set of the first generated video frame set and the reference spatiotemporal feature set of the reference video frame set respectively; Calculate the distance between the generated spatiotemporal feature set and the reference spatiotemporal feature set to obtain the spatiotemporal alignment result; Arbitrary spatiotemporal feature sets are used to reflect the motion trajectories of objects and the patterns of scene changes in a video frame set.

[0008] According to a method for optimizing prompt words in video generation provided by the present invention, the step of performing image perception quality assessment on the initially generated video to obtain quality assessment results includes: The initially generated video is sampled to obtain a second set of generated video frames; The image perception quality is evaluated for each second generated video frame in the second generated video frame set to obtain the perception quality score for each second generated video frame. The quality assessment result is calculated based on the perceived quality score of each of the second generated video frames.

[0009] According to a method for optimizing prompt words in video generation provided by the present invention, the step of semantically comparing the prompt words to be optimized with the initially generated video to obtain a semantic comparison result includes: The initially generated video is sampled to obtain a third set of generated video frames; Extract the generated image features of each third generated video frame in the third generated video frame set, and extract the text features of the prompt word to be optimized; The semantic comparison result is calculated based on the similarity between each generated image feature and the text feature.

[0010] According to a method for optimizing prompt words in video generation provided by the present invention, the step of optimizing the prompt words to be optimized based on the difference comparison results to obtain optimized prompt words includes: If the difference comparison result does not meet the preset video quality threshold, the prompt word to be optimized is optimized based on the difference comparison result to obtain the intermediate optimized prompt word; Based on the intermediate optimization prompts, an intermediate generated video is regenerated. The intermediate generated video is then compared with the intermediate reference video obtained by re-filtering based on the intermediate optimization prompts to obtain an intermediate difference comparison result. If the intermediate difference comparison result still does not meet the preset video quality threshold, the optimization, generation, filtering, and comparison steps are repeated until the obtained intermediate difference comparison result meets the preset video quality threshold. Finally, the final intermediate optimization prompts are output as the optimization prompts.

[0011] According to a method for optimizing prompt words in video generation provided by the present invention, the step of optimizing the prompt words to be optimized based on the difference comparison results to obtain intermediate optimized prompt words includes: The suggestion word examples corresponding to the difference comparison results are obtained by matching from a high-quality suggestion word library; Based on the suggested word examples, the difference comparison results, and the suggested words to be optimized, a suggested text is constructed; The prompt text is input into the large language model to obtain the intermediate optimized prompt word output by the large language model.

[0012] The present invention also provides a device for optimizing prompt words in video generation, comprising: The generation unit generates an initial video based on the prompts to be optimized; The filtering unit selects reference videos from a high-quality video library based on the prompts to be optimized. The difference comparison unit compares the initially generated video and the reference video to obtain the difference comparison result; The optimization unit optimizes the prompt word to be optimized based on the difference comparison results to obtain the optimized prompt word.

[0013] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the prompt word optimization method for generating video as described above.

[0014] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the prompt word optimization method for generating video as described in any of the above.

[0015] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the prompt word optimization method for generating video as described above.

[0016] The present invention provides a method, apparatus, electronic device, and medium for optimizing prompt words in video generation. It generates an initial video using prompt words to be optimized. By comparing the initial video with a reference video, objective difference comparison results are obtained, and the prompt words to be optimized are then specifically optimized. This effectively eliminates the gap between the short prompt words input by the user and the high-quality features trained on the text-based video model. It achieves automated and fine-tuned adjustment of prompt words based on feedback from the generation effect, thereby significantly improving the generation quality of text-based videos and the accuracy of expressing user intent. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0018] Figure 1 This is one of the flowcharts illustrating the prompt word optimization method for generating videos provided by the present invention; Figure 2 This is a schematic diagram of the method for obtaining difference comparison results provided by the present invention; Figure 3 This is a schematic diagram of the method for obtaining the prompt words to be optimized provided by the present invention; Figure 4 This is the second flowchart of the prompt word optimization method for generating videos provided by the present invention; Figure 5 This is a schematic diagram of the device for optimizing prompts in video generation provided by the present invention; Figure 6 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0020] It should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, features defined with "first" and "second" may explicitly or implicitly include one or more of the stated features. In the description of embodiments of the present invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0021] To address the aforementioned issues, this invention provides a method for optimizing prompt words in video generation. It aims to solve the problem that simply optimizing user-input prompt words fails to adapt to the optimal generation state of the Text-to-Video (T2V) model by introducing high-quality reference standards and a dynamic feedback mechanism. Figure 1 This is one of the flowcharts illustrating the method for optimizing prompt words in video generation provided by the present invention, such as... Figure 1 As shown, the method includes: Step 110: Generate an initial video based on the prompts to be optimized.

[0022] Specifically, firstly, the prompt words to be optimized are obtained. The original prompt words input by the user can be acquired and used as the prompt words to be optimized. It is understandable that in real-world scenarios, user input is often brief and vague, such as simply inputting "a person dancing," lacking the key feature descriptions required for model generation, such as composition, lighting, and movement details. Therefore, directly inputting the original prompt words into the Wensheng video model may result in poor generation performance, as the original prompt words usually lack the key feature information required by the model. Preferably, the original prompt words input by the user can be enhanced, and the enhanced prompt words can be used as the prompt words to be optimized. For example, natural language processing techniques or Large Language Models (LLM) can be used to analyze the semantic intent of the prompt words to be optimized, and high-quality descriptive features associated with this semantic intent can be retrieved from a high-quality prompt word library, and these features can be fused into the prompt words to be optimized. For example, optimizing the brief "a person dancing" into an enhanced prompt word containing rich details such as clothing, scene, and lighting could result in "A person in casual clothing is dancing on a brightly lit stage, with smooth and graceful movements, and the camera follows the dancer." Therefore, the enhanced prompt words obtained can be used as prompt words for further optimization.

[0023] Here, the high-quality prompt vocabulary refers to a pre-constructed, pairwise associated high-quality dataset. Thus, by mapping the user's non-professional descriptions to a high-quality feature distribution familiar to the textual video model and seen during training, the optimal generative capability of the textual video model is stimulated.

[0024] Furthermore, the initial generated video can be obtained by inputting the prompt words to be optimized into the text-based video model.

[0025] Step 120: Based on the prompt words to be optimized, select reference videos from the high-quality video library.

[0026] The construction process of the high-quality prompt word library and high-quality video library includes: during the training or testing phase of the text-generated video model, samples that meet the generation performance standards are selected. The professionally designed prompt words corresponding to these samples are used as the high-quality prompt word library, and the corresponding generated videos are used as the high-quality video library. It should be noted that the high-quality prompt words stored in the high-quality prompt word library can exist in text form or in the form of high-quality text features obtained after feature extraction of the prompt words. Similarly, the high-quality videos in the high-quality video library can exist in video form or in the form of image features of key image frames of the high-quality videos.

[0027] For example, samples that meet the generation performance standards during the training and testing phases of the Wensheng video model can be selected. The generated videos are then manually evaluated from three dimensions: visual dimension, temporal consistency, and semantic relevance, with scores ranging from 0 to 10. Videos with an average score of 7 or higher across these three dimensions are selected, and their corresponding prompt words are considered high-quality prompt words. Next, feature extraction and deduplication can be performed on these high-quality prompt words. Specific steps include: first, performing Chinese word segmentation on the high-quality prompt words, breaking them down into core words such as subject, action, environment, shot, and lighting. Then, semantic features of the high-quality prompt words are extracted using a text semantic encoding model. Next, feature vectors can be stored according to scene categories and indexed. Redundant data is removed through feature similarity comparison, with a cosine similarity ≥ 95% indicating duplication. Finally, the semantic features of the deduplicated high-quality prompt words can be used to construct a high-quality prompt word library. The video feature vectors of the corresponding high-quality videos are bound and stored with the corresponding prompt word feature vectors, establishing a mapping relationship to construct a high-quality video library.

[0028] Here, reference videos refer to video samples that can serve as benchmarks for the current generation task in terms of visual quality, temporal coherence, or semantic style. It should be noted that the selected reference videos are not required to be completely identical to the video the user expects to generate, but rather emphasize their high similarity in scene category, action type, or style features, thereby providing a context-relevant quality reference.

[0029] Specifically, reference videos can be obtained by matching and searching a high-quality video library based on the semantic features of the prompt word to be optimized. For example, the feature similarity between the semantic features of the prompt word to be optimized and the prompt word features corresponding to each high-quality video in the high-quality video library can be calculated. The high-quality videos corresponding to the prompt word features with similarity higher than a preset similarity threshold are used as reference videos. The reference videos here include at least one high-quality video.

[0030] Understandably, the significance of introducing reference videos lies in providing a specific and quantifiable benchmark for evaluating the subsequent video generation effect, avoiding the problem of single subjective judgments or the failure of absolute indicators, thereby enabling dynamic optimization of prompt words to improve the optimization effect of prompt words.

[0031] Step 130: Compare the initially generated video and the reference video to obtain the difference comparison result.

[0032] Here, the difference comparison results refer to the quantitative data or qualitative analysis conclusions of the differences between the initially generated video and the reference video used as a benchmark in multiple dimensions.

[0033] Specifically, the initially generated video can be compared with the selected reference videos in multiple dimensions. It should be noted that this comparison can cover visual dimensions such as clarity and composition quality, temporal dimensions such as action coherence, and semantic dimensions such as content consistency.

[0034] Understandably, by comparing with high-quality reference videos, it is possible to accurately identify which aspects of the currently generated video have not yet reached the ideal standard, such as image clarity or motion smoothness, thereby providing data support for subsequent targeted optimization.

[0035] Step 140: Optimize the prompt word to be optimized based on the difference comparison results to obtain the optimized prompt word.

[0036] Specifically, based on the deficiencies pointed out in the difference comparison results, such as the difference comparison results showing that the initial generated video lacks a sense of depth in its lighting and shadows, a large language model or rule matching method can be used to add, delete, or modify the corresponding keywords or descriptive statements in the current prompts to be optimized, thereby generating new prompts for the next round of verification or output.

[0037] It should be noted that by comparing the differences between the initially generated video and the reference video, the prompt words to be optimized are optimized again, thus constructing a dynamic feedback loop of prompt word optimization - video generation - difference analysis - re-optimization. This changes the traditional text-based video generation technology model of one-time input and one-time generation, ensuring that the prompt words can be dynamically adjusted according to the gap between the actual generated effect and the high-quality standard.

[0038] The method provided in this invention generates an initial generated video using prompts to be optimized. By comparing the initial generated video with a reference video, objective difference comparison results are obtained, and the prompts to be optimized are optimized accordingly. This effectively eliminates the gap between the short prompts input by the user and the high-quality features trained on the text-based video model, and realizes automated fine-tuning of prompts based on feedback from the generation effect. This significantly improves the generation quality of the text-based video and the accuracy of expressing user intent.

[0039] To comprehensively evaluate whether the generated video meets expectations, quantitative analysis can be performed from three dimensions: temporal, visual, and semantic. Based on any of the above embodiments... Figure 2 This is a schematic diagram of the method for obtaining difference comparison results provided by the present invention, as shown below. Figure 2 As shown, the method includes: Step 210: Perform spatiotemporal feature comparison between the initially generated video and the reference video to obtain the spatiotemporal comparison result.

[0040] Here, the spatiotemporal comparison results primarily reflect the differences between the initial generated video and the reference video in terms of temporal distribution and dynamic coherence. Specifically, since the reference video represents a high-quality spatiotemporal feature distribution, comparison can measure whether the generated video approaches the high-quality standard in terms of motion smoothness and overall video rhythm. For example, the spatiotemporal feature comparison between the initial generated video and the reference video can be achieved by calculating the Fréchet Video Distance (FVD) value, thus obtaining the spatiotemporal comparison result. A pre-trained video classification model can be used to extract the spatiotemporal features of both the initial generated video and the reference video. Then, the distance between the spatiotemporal feature distributions of the initial generated video and the reference video is calculated to obtain the FVD value, which is the spatiotemporal comparison result. It can be understood that the lower the FVD value, the closer the initial generated video is to the reference video in terms of visual quality and temporal dynamics.

[0041] Step 220: Perform image perception quality assessment on the initially generated video to obtain the quality assessment result.

[0042] Here, the quality assessment results primarily reflect the spatial visual quality of the generated video itself, such as sharpness, resolution, and color fidelity. Specifically, a no-reference image quality assessment method is typically used, directly scoring the frames of the generated video to determine whether it meets the required image quality standards.

[0043] Step 230: Perform a semantic comparison between the prompt word to be optimized and the initially generated video to obtain the semantic comparison result.

[0044] Here, the semantic comparison results mainly reflect whether the generated content accurately executes the instructions of the prompt words.

[0045] Specifically, semantic comparison can be achieved by calculating the matching degree between the prompt words to be optimized and the video content of the initially generated video. The calculated matching degree is used as the semantic comparison result to ensure that no semantic drift or loss of key elements occurs during the generation process.

[0046] Step 340: Based on the quality assessment results and / or the semantic comparison results, and the spatiotemporal comparison results, the difference comparison results are obtained.

[0047] The difference comparison results are used to characterize the degree of deviation of the initially generated video from the reference video in multidimensional features, that is, to reflect the degree of deviation of the initially generated video from the reference video in the dimensions of image quality, semantics, and spatiotemporal.

[0048] Specifically, the difference comparison results for each dimension can be compared with the preset dimensional thresholds corresponding to each dimension to obtain difference comparison results in the form of an analysis report. For example, if the spatiotemporal comparison result is less than the preset FVD value, the spatiotemporal comparison result is marked as "Spatiotemporal consistency needs optimization". If the quality assessment result is lower than the preset image quality score, it is marked as "Image quality needs optimization". If the spatiotemporal comparison result is too far apart, it is marked as "Motion needs smoothing".

[0049] Finally, the quality assessment results and / or semantic comparison results, as well as the spatiotemporal comparison results, can be combined to obtain a complete difference comparison result, providing a clear optimization target for the subsequent targeted selection of second prompt word examples. It should be noted that steps 210, 220, and 230 above do not have a specific execution order; they can be executed simultaneously or in a specific order.

[0050] The method provided in this invention constructs a multi-dimensional evaluation system that includes spatiotemporal comparison results with temporal consistency, quality assessment results reflecting visual quality, and semantic comparison results reflecting semantic relevance. This avoids the one-sidedness of a single indicator and ensures that the quality of video generation can comprehensively approach high-quality reference standards.

[0051] Based on any of the above embodiments, step 210 includes: The initial generated video and the reference video are sampled respectively to obtain a first generated video frame set and a reference video frame set; Extract the spatiotemporal feature set of the first generated video frame set and the reference spatiotemporal feature set of the reference video frame set respectively; Calculate the distance between the generated spatiotemporal feature set and the reference spatiotemporal feature set to obtain the spatiotemporal alignment result; Arbitrary spatiotemporal feature sets are used to reflect the motion trajectories of objects and the patterns of scene changes in a video frame set.

[0052] Specifically, to capture the dynamic changes in video, the first step is to extract frame sequences from the continuous video stream. A uniform sampling strategy can be used to extract a fixed number of frames from both the initial generated video and the reference video, for example, sampling several frames per second to form the first generated video frame set and the reference video frame set.

[0053] It should be noted that the reference video frame set here can come from multiple high-quality reference videos of the same type to form a statistically significant distribution sample.

[0054] Then, a pre-trained video classification model can be used as a feature extractor. The aforementioned frame set is input into the video classification model, which extracts deep feature vectors containing temporal and spatial dimensions, thereby obtaining the generated spatiotemporal feature set and the reference spatiotemporal feature set. It can be understood that these features not only contain the content of the image but also implicitly contain the patterns of object motion trajectories and scene changes.

[0055] Furthermore, the distance between the generated spatiotemporal feature set and the reference spatiotemporal feature set can be calculated to obtain the spatiotemporal comparison result. For example, the Fréchet video distance can be used as a metric. The mean vector and covariance matrix of the generated spatiotemporal feature set and the reference spatiotemporal feature set are calculated respectively, and then the trace operator is used to calculate the geometric distance between the two high-dimensional distributions, i.e., the FVD value. It can be understood that the lower the FVD value, the closer the initially generated video is to the real high-quality video in terms of spatiotemporal dynamic characteristics. Here, the FVD value can be calculated using the following formula, as shown in the following equation: ; In the formula, and These represent reference videos. Compared with the initial generated video The mean of the spatiotemporal characteristic distribution; and Reference Video Compared with the initial generated video The spatiotemporal characteristic distribution covariance. Tr(...) represents the trace operator, which calculates the sum of the diagonal elements of a matrix. Here, it is used to transform a matrix representing the differences in the shape of the distribution into a single scalar value. The first term in the formula measures the difference at the distribution's center point, and the second term measures the difference in the distribution's shape and direction.

[0056] The method provided in this invention calculates the distance between the generated spatiotemporal feature set and the reference spatiotemporal feature set, quantifies the spatiotemporal comparison results, and can effectively measure the gap between the generated video and the high-quality video in terms of motion smoothness and temporal logic, so as to achieve accurate evaluation of the prompt words in terms of motion smoothness and temporal logic.

[0057] Based on any of the above embodiments, step 220 includes: The initially generated video is sampled to obtain a second set of generated video frames; The image perception quality is evaluated for each second generated video frame in the second generated video frame set to obtain the perception quality score for each second generated video frame. The quality assessment result is calculated based on the perceived quality score of each of the second generated video frames.

[0058] Specifically, to evaluate the overall image quality of the initially generated video, representative frames can be selected. For example, a preset number of frames can be uniformly sampled from the initially generated video, such as 16 frames, to form the second generated video frame set.

[0059] Then, image perceptual quality assessment is performed on each of the second generated video frames in the second generated video frame set to obtain a perceptual quality score for each second generated video frame. In specific implementations, a Multi-scale Image Quality Transformer (MUSIQ) model or other full-resolution image quality assessment models can be used to assess the image perceptual quality of each second generated video frame. It should be noted that this model can simulate the human visual system, perceptually scoring details such as sharpness, noise, and compression artifacts in each frame. Therefore, each second generated video frame can be input into the full-resolution image quality assessment model, which then outputs the corresponding perceptual quality score.

[0060] Finally, based on the perceptual quality scores of each second-generated video frame, the quality assessment result is calculated. In a specific implementation, the scores of the above frames are statistically aggregated, and their average value is usually calculated as the final quality score of the video. If this average value is lower than a preset quality threshold, such as 7 points, the quality assessment result indicates that the video has defects in the visual dimension.

[0061] The method provided in this invention can objectively reflect the level of resolution and detail performance of the generated video by scoring the perceived quality of video frames, and provides a direct basis for whether to add modifiers such as "high quality", "4K" and "rich in detail" to the prompts to improve visual quality.

[0062] Based on any of the above embodiments, step 330 includes: The initially generated video is sampled to obtain a third set of generated video frames; Extract the generated image features of each third generated video frame in the third generated video frame set, and extract the text features of the prompt word to be optimized; The semantic comparison result is calculated based on the similarity between each generated image feature and the text feature.

[0063] Specifically, the same uniform sampling method can be used to extract N frames from the initially generated video to form a third set of generated video frames, which can cover different time segments of the video.

[0064] Then, the generated image features of each of the third generated video frames in the third generated video frame set are extracted, along with the text features of the prompt words to be optimized. In a specific implementation, a contrastive language image pre-training model can be used for cross-modal feature extraction. Each frame in the third generated video frame set is input into the image encoder of the contrastive language image pre-training model to obtain a set of generated image feature vectors. Simultaneously, the prompt words to be optimized used to generate the video are input into the text encoder of the contrastive language image pre-training model to obtain a text feature vector, which serves as the text feature of the prompt words to be optimized.

[0065] Then, based on the similarity between the generated image features and the text features, the semantic comparison result is calculated. In a specific implementation, the cosine similarity between the generated image feature vector and the text feature vector of each frame can be calculated separately. For the ... Frame generated image feature vector Frame and text feature vectors Cosine similarity between It can be calculated using the following formula, as shown in the following equation: ; Then, the similarity scores of all frames are averaged to obtain the final CLIPScore score. This score reflects the degree of consistency between the video frame and the text description of the prompt word to be optimized. The CLIPScore score can be calculated using the following formula, as shown below: ; In the formula, N represents the total number of video frames in the third generated video frame set.

[0066] The method provided in this invention calculates semantic comparison results by measuring the similarity between generated image features and text features. This enables a quantitative assessment of whether the generated video content is faithful to the prompt words to be optimized. In this way, it can determine whether the main description or weight in the prompt words needs to be adjusted in multiple rounds of optimization, thus preventing the generated content from going off-topic.

[0067] It should be noted that existing technologies, after generating a video, can only return the result to the user, and cannot dynamically adjust and optimize the prompts based on the differences between the generated video and the high-quality videos from the training and testing phases. If the user is not satisfied with the generated result, they can only re-enter the text prompts, and the entire process lacks an effective feedback loop. To address this issue, based on any of the above embodiments, step 140 includes: If the difference comparison result does not meet the preset video quality threshold, the prompt word to be optimized is optimized based on the difference comparison result to obtain the intermediate optimized prompt word; Based on the intermediate optimization prompts, an intermediate generated video is regenerated. The intermediate generated video is then compared with the intermediate reference video obtained by re-filtering based on the intermediate optimization prompts to obtain an intermediate difference comparison result. If the intermediate difference comparison result still does not meet the preset video quality threshold, the optimization, generation, filtering, and comparison steps are repeated until the obtained intermediate difference comparison result meets the preset video quality threshold. Finally, the final intermediate optimization prompts are output as the optimization prompts.

[0068] Here, the preset video quality threshold refers to the threshold standard set to ensure that the generated video reaches the level of high-quality samples in the training set, and may include thresholds of at least one dimension or multiple dimensions.

[0069] Specifically, after obtaining the initial round of difference comparison results, a judgment is made based on the difference comparison results and the preset video quality threshold. It is understood that the difference comparison results include specific quantitative indicators for each dimension. Therefore, it can be determined whether the visual quality score is lower than the preset score, whether the semantic feature similarity between the generated video and the high-quality reference video set is lower than the preset percentage, or whether the temporal consistency indicator is higher than the maximum allowed distance value. If any of the above indicators in the difference comparison results fails to meet the standard, or the comprehensive weighted score fails to reach the passing grade, then the initially generated video in the initial round is determined to not meet the preset video quality threshold, i.e., it fails the verification.

[0070] Then, based on the difference comparison results, the suggestion words to be optimized are optimized to obtain intermediate optimized suggestion words. For example, suggestion word examples corresponding to the difference comparison results can be obtained by matching from a high-quality suggestion word library. Then, suggestion text is constructed based on suggestion word examples, difference comparison results, and suggestion words to be optimized. Further, the suggestion text can be input into a large language model to obtain the intermediate optimized suggestion words output by the large language model. Alternatively, the difference comparison results and suggestion words to be optimized can be directly input into the large language model, and the large language model outputs intermediate optimized suggestion words.

[0071] Then, using intermediate optimization prompts, an intermediate generated video is regenerated. For example, the intermediate optimization prompts can be input into a text-based video model to obtain the intermediate generated video output by the model. Next, the intermediate generated video can be compared with an intermediate reference video obtained by re-filtering based on the intermediate optimization prompts of the current round to obtain the intermediate difference comparison result. It can be understood that the intermediate difference comparison result here is also used to characterize the degree of deviation of the intermediate generated video from the intermediate reference video in terms of multidimensional features.

[0072] Furthermore, the intermediate difference comparison results are compared with the preset video quality threshold. If the intermediate difference comparison results still do not meet the preset video quality threshold, the optimization, generation, filtering and comparison steps are repeated until the intermediate difference comparison results meet the preset video quality threshold. Finally, the intermediate optimization prompt words are output as optimization prompt words.

[0073] The method provided in this invention, by constructing a dynamic feedback closed loop of video generation-difference analysis-prompt word re-optimization, further overcomes the limitations of one-time generation and lack of feedback correction in the prior art. It can progressively repair the specific generation defects in each round, ensuring that the feature distribution of prompt words gradually approaches the optimal state during model training, thereby significantly improving the video effect of the final generated video.

[0074] Based on any of the above embodiments, the prompt words to be optimized are optimized based on the difference comparison results to obtain intermediate optimized prompt words, including: The suggestion word examples corresponding to the difference comparison results are obtained by matching from a high-quality suggestion word library; Based on the suggested word examples, the difference comparison results, and the suggested words to be optimized, a suggested text is constructed; The prompt text is input into the large language model to obtain the intermediate optimized prompt word output by the large language model.

[0075] Specifically, firstly, sample prompts corresponding to the difference comparison results can be obtained from a high-quality prompt word library. It should be noted that the difference comparison results here are quantitative differences or qualitative evaluations obtained after comparing the previously generated video with the reference video across multiple dimensions. For example, the difference comparison results might indicate deficiencies in the current video regarding "image clarity," "motion coherence," or "realism of lighting and shadow." Then, high-quality prompts that can generate videos that excel in the aforementioned deficiencies can be retrieved from the high-quality prompt word library and used as sample prompts corresponding to the difference comparison results.

[0076] Specifically, based on the specific defect dimensions indicated in the difference comparison results, targeted searches are performed to match corresponding prompt word examples from a high-quality prompt word library. For example, if the difference comparison results show that the generated video has a poor temporal consistency score, then high-quality prompt words with excellent Fraser video distance scores for the corresponding video will be selected from the high-quality prompt word library as prompt word examples. It is understandable that the retrieved prompt word examples implicitly contain descriptive patterns or specialized vocabulary needed to solve specific generation defects, and thus the matched prompt word examples can be used to construct an optimization rule base for the current problem.

[0077] Subsequently, based on the aforementioned prompt word examples, difference comparison results, and prompt words to be optimized, a prompt text is constructed. Here, the prompt text refers to a structured instruction set (prompt) built to guide the large language model to complete a specific generation task. Its construction process aims to transform the logic of difference analysis into a natural language processing task.

[0078] Specifically, the prompt words to be optimized, the difference comparison results, and the prompt word examples can be combined according to a preset template. This template can include a role setting section, a task background section, and an input data section. For example, when constructing the prompt text, the role of the large language model is first set as "Wensheng Video Prompt Optimization Expert." Then, the prompt words to be optimized are marked as "current input," the difference comparison results are marked as "defects to be fixed," and the prompt word examples are marked as "reference examples." Next, the instruction logic is clearly defined in the prompt text: the description in the "reference examples" to achieve high-quality results is analyzed, and the feature description methods for the "defects to be fixed" are extracted and integrated into the "current input," while maintaining the original semantic content of the "current input."

[0079] Thus, by constructing prompt text, an abstract optimization requirement is transformed into a text rewriting task that can be performed by a large language model.

[0080] Next, the prompt text is input into the large language model to obtain the intermediate optimized prompt words output by the large language model. In specific implementation, the structured prompt text constructed above is input into the large language model. The large language model, based on its powerful semantic understanding and reasoning capabilities, parses the prompt text. First, the large language model analyzes the specific problems indicated in the difference comparison results, such as "blurry image" or "stiff movements"; then, it compares the prompt word examples to identify the specific vocabulary or sentence structures used in the examples to solve the problem, i.e., the so-called semantic enhancement patterns. For example, the large language model identifies that the example uses "fine texture rendering" to solve the "blurry image" problem. Finally, based on the large language model, these identified high-quality feature words or sentence structures are naturally embedded or replaced into the corresponding positions of the prompt words to be optimized, according to the original contextual logic, to generate the final intermediate optimized prompt words.

[0081] It should be noted that a dynamic adjustment mechanism based on difference feedback was established during the iterative optimization phase. By matching prompt word examples from the high-quality library according to specific generation defects, and applying the feature patterns in the high-quality examples to the correction of the current prompt words, targeted closed-loop optimization can be achieved. This ensures that each round of adjustment directly targets the weak links in video generation, thereby quickly converging the generation effect and making the final output video approach the high-quality reference standard.

[0082] The method provided in this invention fully utilizes the contextual learning capabilities of a large language model by constructing a structured prompt text that includes the object to be optimized, defect feedback, and reference standards. It goes beyond simple word concatenation, allowing the large language model to learn how to describe like high-quality prompt words, thereby achieving deep semantic reconstruction of the prompt words. This ensures that the generated intermediate optimized prompt words not only correct the specific defects in video generation but also maintain the fluency and logical coherence of the sentences.

[0083] It should also be noted that, based on any of the above embodiments, in the initial optimization round, the optimization of the user-inputted original prompt words to obtain the prompt words to be optimized can be achieved through the following steps. These steps include: retrieving initial prompt word samples from a high-quality prompt word library based on the semantic vectors of the original prompt words; extracting sample keywords from the initial prompt word samples. Then, inputting the sample keywords and the original prompt words into a large language model to obtain the prompt words to be optimized output by the large language model.

[0084] For example, in the initial stage of video generation when the user first inputs a prompt word, firstly, based on the semantic vector of the original prompt word, initial prompt word samples are retrieved from a high-quality prompt word library. Here, initial prompt word samples refer to high-quality prompt word samples in the pre-built high-quality prompt word library that are most semantically similar to the original prompt word input by the user.

[0085] In one embodiment, to obtain initial prompt word samples, the original prompt words input by the user first need to be converted into vector form. This can be done using a pre-trained text semantic encoding model, such as the text encoder in the Contrastive Language-Image Pre-training (CLIP) model, to encode the original prompt words and generate high-dimensional semantic feature vectors. Simultaneously, each high-quality prompt word in the high-quality prompt word library has already been pre-converted using the same encoding model and its corresponding feature vector has been stored before being added to the library.

[0086] Next, the cosine similarity between the semantic feature vector of the original suggestion word and the feature vectors of each high-quality suggestion word in the high-quality suggestion word library is calculated. The cosine similarity here can be calculated using the following formula, as shown below: ; In the formula, Let represent the cosine similarity between the feature vectors of the original prompt word A and any high-quality prompt word; where, , This indicates the total number of segmentations contained in the original prompt word. This represents the semantic feature vector of the first word segment in the original prompt word; , This indicates the total number of word segments contained in any given high-quality suggestion word. This represents the semantic feature vector of the first segment of any high-quality prompt word.

[0087] To ensure the relevance of search results, high-quality suggestion words can be sorted in descending order of cosine similarity, and several top-ranked high-quality suggestion words can be selected as initial suggestion word examples, such as the top 5 high-quality suggestion words. It is understood that the initial suggestion word examples typically contain detailed description features favored by the text-based video model during its training phase, features that can generate high-quality videos.

[0088] Subsequently, sample keywords are extracted from the initial prompt word examples. Specifically, natural language processing techniques can be used to segment and tag the selected initial prompt word examples. The extracted dimensions should cover the key elements of video generation, including scene element keywords, such as forest and street; action detail keywords, such as running and spinning; style modification keywords, such as cyberpunk and oil painting style; and camera movement keywords, such as push-pull and surround. By extracting these sample keywords, high-value features from high-quality samples are extracted from specific sentence structures and used as enhancement material.

[0089] It should be noted that the sample keywords here refer to the core elements that constitute the initial prompt sample, which can be considered as the key features that enable the original prompt to generate a high-quality video.

[0090] Finally, the sample keywords and original prompts are input into the large language model to obtain the prompts to be optimized. Specifically, leveraging the powerful semantic understanding and generation capabilities of the large language model, the extracted high-quality features are naturally integrated into the user's original intent. For example, the original prompts and extracted sample keywords can be combined to construct a prompt instruction, which is then input into the large language model. The large language model executes a dynamic feature mapping strategy: first, it identifies and retains the core semantics of the prompts to be optimized, ensuring that the generated content does not deviate from the user's original intent; then, it organically embeds the sample keywords into the description in an order that conforms to the video generation logic, such as the order of subject, action, environment, style, and shot. At the same time, it detects and removes any semantic conflicts or redundant words. The final output prompts to be optimized not only retain the user's original concept but also supplement the model's preferred details. For example, "a person dancing" is optimized to "a person in casual clothes dancing on a stage with dazzling lights and shadows, with smooth and natural movements and slow camera movement."

[0091] In one embodiment, Figure 3 This is a schematic diagram of the method for obtaining suggestion words to be optimized provided by the present invention, such as... Figure 3 As shown, the process mainly includes three core stages: syntax normalization, semantic normalization, and hybrid enhancement mechanism.

[0092] First, the raw text input by the user is subjected to grammatical normalization. Here, raw text refers to the original prompt words. This stage aims to eliminate formatting noise and logical errors in the input text. Specific operations include removing redundant spaces, standardizing letter case, and using natural language processing techniques to detect grammatical errors and ambiguous expressions, such as correcting parts of speech, tenses, and subject-verb-object relationships, thereby outputting a well-formatted and clearly meaningful basic text.

[0093] Next, the normalized text undergoes semantic standardization. In this stage, the normalized prompts from the previous stage are first input into a large language model, which then performs semantic completion based on a common-sense knowledge base. The completion process considers dimensions such as subject (person / object), scene (indoor / outdoor), action (static / dynamic), and style (realistic / fantasy). Then, referring to the styles of high-quality prompts commonly seen in the training phase, the prompts are standardized to possess complete descriptiveness. For example, the input prompt "a person is dancing" is standardized to "a person wearing casual clothes is dancing on stage." It should be noted that at this stage, the large language model can analyze the text's intent and complete the missing key descriptive dimensions, ensuring that the prompts fully cover core elements such as subject, scene, action, and style, forming standardized prompts with complete descriptiveness.

[0094] Finally, the hybrid enhancement mechanism stage is entered to generate the final enhanced prompt words, i.e., the prompt words to be optimized. This mechanism first uses similarity retrieval to match high-quality samples semantically similar to the standardized prompt words in a pre-built high-quality feature library. Then, keywords are extracted from the matched samples to obtain high-quality feature words. Next, dynamic feature mapping is performed to organically integrate the extracted keywords into the standardized prompt words. Finally, the output is the enhanced prompt words, which have undergone multiple optimizations, are rich in detail, and conform to the model's generation preferences; these are the prompt words to be optimized.

[0095] The method provided in this invention solves the problem of mismatch between user input and the training features of the text-based video model by retrieving a high-quality prompt word library and extracting sample keywords in the first round of optimization, and using a large language model to map the short prompt words input by the user into prompt words containing rich details. This ensures that the video generated in the first round has a high quality starting point and effectively avoids video generation failure caused by overly simple prompt words.

[0096] Based on any of the above embodiments, step 140 is followed by: The final generated video will be used as the video to be verified. If the verification result of the video to be verified is passed, the pre-built high-quality prompt word library and the high-quality video library are updated based on the final obtained intermediate optimized prompt words and the generated video, respectively.

[0097] Specifically, when the loop iteration meets the termination condition, such as reaching the maximum number of iterations or the difference comparison result meeting a preset threshold, the generated video output in the current round is locked and marked as a video to be verified. At the same time, the intermediate optimization prompts used to generate the video are marked as prompts to be verified.

[0098] Next, an automated quality check is performed on the video to be verified. If the verification passes, it indicates that the currently generated pair of prompt words and video data is a high-quality and highly matched successful case. At this point, the high-quality prompt word library and the high-quality video library can be updated. For example, the semantic vectors of the final intermediate optimized prompt words can be extracted, stored in the high-quality prompt word library, and indexed. At the same time, the corresponding generated video or extracted video feature vectors are stored in the high-quality video library, and a mapping relationship between the two is established.

[0099] It should be noted that, in order to ensure the diversity and validity of the data in the database, a deduplication operation can be performed before updating. For example, the cosine similarity between the new feature vector and the existing vectors in the database can be calculated. If the similarity is higher than a preset redundancy threshold, such as 95%, it is considered duplicate data and will not be added to the database.

[0100] The method provided in this invention, by collecting verified high-quality generated results into a library, means that as the number of times users use the product increases, the coverage of the high-quality prompt word library and the high-quality video library will continue to expand, thereby providing more accurate reference examples for future processing of similar prompt words to be optimized, forming a positive optimization effect.

[0101] Considering the diversity of user-generated intents, a single verification standard is often difficult to apply to all scenarios. Based on any of the above embodiments, the method for determining the verification result of the video to be verified includes: When the prompt word to be optimized is a non-realistic description, the style alignment score between the video to be verified and the final intermediate optimized prompt word is calculated to obtain the verification result. If the prompt word to be optimized is a realistic description, the similarity between the video to be verified and the final intermediate optimized prompt word is calculated to obtain the verification result.

[0102] Specifically, the first step is to classify the intent of the prompts to be optimized. When prompts contain non-realistic or fantasy tags such as "cyberpunk," "ink painting style," or "surrealism," they are classified as non-realistic descriptions. In this case, simply comparing the physical realism of the video content is meaningless; the focus of verification is on "style consistency."

[0103] Therefore, a style alignment algorithm can be used, such as the style alignment scoring mechanism of the CLIP model, to calculate the style alignment score between the video to be verified and the final intermediate optimized prompts. Specifically, the visual style feature vector of the initially generated video and the style description feature vector of the intermediate optimized prompts can be extracted, and the distance or correlation score between the two can be calculated as the style alignment score. If this score is higher than a preset style threshold, the verification is considered successful, ensuring that creative videos are not mistakenly deemed unqualified due to inconsistencies with real-world physical laws.

[0104] Furthermore, when the prompt to be optimized is a realistic description, the similarity between the video to be verified and the final intermediate optimized prompt can be calculated to obtain the verification result. Specifically, when the prompt describes a real-life scene, such as "a person is running in a park," the verification focuses on semantic accuracy and physical realism. Therefore, the feature similarity between the initially generated video and the intermediate optimized prompt can be calculated. For example, CLIPScore or other image-text matching algorithms can be used to calculate the cosine similarity between the video frame image features and the prompt text features. If the similarity reaches a preset threshold, such as 80%, the verification is considered successful, ensuring that the generated video accurately reproduces the subject, action, and environment described in the prompt.

[0105] The method provided in this invention distinguishes between realistic and non-realistic scenarios and adopts a dual-track verification standard of semantic similarity and style alignment scores. This effectively solves the problem that a single indicator is prone to failure when faced with users' imaginative creative inputs. It ensures the accuracy of realistic content while giving innovative content sufficient inclusiveness, thereby significantly improving the reliability and intelligence of the automated verification process.

[0106] Figure 4 This is the second flowchart illustrating the method for optimizing prompt words in video generation provided by this invention, as shown below. Figure 4 As shown, this method mainly includes four core steps.

[0107] First, in the prompt word preprocessing stage, the original prompt words input by the user are received, and their syntax is normalized and semantics are completed. Using a large language model and dynamic feature mapping strategy, the short and ambiguous user input is mapped into prompt words containing rich details to be optimized, so as to initially adapt to the feature preferences of the text-based video model.

[0108] Next, in the first round of video generation and difference analysis stage, the first round of videos is generated using prompts to be optimized, and reference videos are selected from the high-quality video library as quality benchmarks based on semantic features. By comparing the specific differences between the first round of videos and the reference videos in dimensions such as visual quality, temporal consistency and semantic relevance, a quantitative difference analysis report is generated.

[0109] Next, the process enters a multi-round prompt word optimization and video iteration generation stage. This stage constructs a dynamic feedback closed loop. Based on the specific defects pointed out in the difference analysis report, such as blurry image quality or disjointed actions, specific optimization rules or examples are retrieved from the high-quality prompt word library. The large language model is used to make targeted corrections to the prompt words of the previous round and generate new videos. This process is iterated repeatedly until the generation effect meets the preset requirements or reaches the maximum number of iterations.

[0110] Finally, the final stage of effect verification and output is carried out. The quality of the final generated video is automatically verified, and different verification standards are used to distinguish between realistic and non-realistic scenes, such as semantic similarity or style alignment scores. After the verification is passed, the final result is output to the user, and the generated high-quality prompt words and video pairs are updated to the high-quality prompt word library and the high-quality video library for subsequent learning.

[0111] The method provided in this invention aims to solve the technical problem that there is a gap between the effect of user input prompts in the generated video and the effect during the training and testing phases in existing text-based video technology. Specifically, it includes establishing a correlation between user input prompts and prompt features from the training and testing phases of the text-based video model to achieve accurate optimization of the prompts and make them conform to the model's feature preferences; constructing a dynamic feedback adjustment mechanism and performing multiple rounds of optimization on the prompts based on the differences between the generated video and the high-quality training and testing videos to gradually narrow the gap in generation effect; and guiding the text-based video model to call the high-quality feature generation mode from the training and testing phases during the inference phase to ensure that the quality of the generated video remains consistent with that of the training and testing phases.

[0112] Based on any of the above embodiments Figure 5 This is a schematic diagram of the device for optimizing prompt words in video generation provided by the present invention, as shown below. Figure 5 The device includes: Generation unit 510 generates an initial video based on the prompt words to be optimized; The filtering unit 520 filters reference videos from the high-quality video library based on the prompt words to be optimized. The difference comparison unit 530 compares the initially generated video and the reference video to obtain the difference comparison result; The optimization unit 540 optimizes the prompt word to be optimized based on the difference comparison results to obtain the optimized prompt word.

[0113] The apparatus provided in this invention generates an initial generated video using prompts to be optimized. By comparing the initial generated video with a reference video, objective difference comparison results are obtained, and the prompts to be optimized are optimized accordingly. This effectively eliminates the gap between the short prompts input by the user and the high-quality features trained on the text-based video model, and realizes automated fine-tuning of prompts based on feedback from the generation effect. This significantly improves the generation quality of the text-based video and the accuracy of expressing user intent.

[0114] Based on any of the above embodiments, the difference comparison unit is specifically used for: The initial generated video and the reference video are compared in terms of spatiotemporal features to obtain the spatiotemporal comparison result. The initially generated video is subjected to image perception quality assessment to obtain the quality assessment result; The semantic comparison between the prompt words to be optimized and the initially generated video is performed to obtain the semantic comparison results; The difference comparison result is obtained based on the quality assessment result and / or the semantic comparison result, as well as the spatiotemporal comparison result; The difference comparison results are used to characterize the degree of deviation of the initially generated video from the reference video in terms of multidimensional features.

[0115] Based on any of the above embodiments, the difference comparison unit is further specifically used for: The initial generated video and the reference video are sampled respectively to obtain a first generated video frame set and a reference video frame set; Extract the spatiotemporal feature set of the first generated video frame set and the reference spatiotemporal feature set of the reference video frame set respectively; Calculate the distance between the generated spatiotemporal feature set and the reference spatiotemporal feature set to obtain the spatiotemporal alignment result; Arbitrary spatiotemporal feature sets are used to reflect the motion trajectories of objects and the patterns of scene changes in a video frame set.

[0116] Based on any of the above embodiments, the difference comparison unit is further specifically used for: The initially generated video is sampled to obtain a second set of generated video frames; The image perception quality is evaluated for each second generated video frame in the second generated video frame set to obtain the perception quality score for each second generated video frame. The quality assessment result is calculated based on the perceived quality score of each of the second generated video frames.

[0117] Based on any of the above embodiments, the difference comparison unit is further specifically used for: The initially generated video is sampled to obtain a third set of generated video frames; Extract the generated image features of each third generated video frame in the third generated video frame set, and extract the text features of the prompt word to be optimized; The semantic comparison result is calculated based on the similarity between each generated image feature and the text feature.

[0118] Based on any of the above embodiments, the optimization unit is specifically used for: If the difference comparison result does not meet the preset video quality threshold, the prompt word to be optimized is optimized based on the difference comparison result to obtain the intermediate optimized prompt word; Based on the intermediate optimization prompts, an intermediate generated video is regenerated. The intermediate generated video is then compared with the intermediate reference video obtained by re-filtering based on the intermediate optimization prompts to obtain an intermediate difference comparison result. If the intermediate difference comparison result still does not meet the preset video quality threshold, the optimization, generation, filtering, and comparison steps are repeated until the obtained intermediate difference comparison result meets the preset video quality threshold. Finally, the final intermediate optimization prompts are output as the optimization prompts.

[0119] Based on any of the above embodiments, the optimization unit is further specifically used for: The suggestion word examples corresponding to the difference comparison results are obtained by matching from a high-quality suggestion word library; Based on the suggested word examples, the difference comparison results, and the suggested words to be optimized, a suggested text is constructed; The prompt text is input into the large language model to obtain the intermediate optimized prompt word output by the large language model.

[0120] Figure 6 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 6 As shown, the electronic device may include a processor 610, a communication interface 620, a memory 630, and a communication bus 640, wherein the processor 610, the communication interface 620, and the memory 630 communicate with each other via the communication bus 640. The processor 610 can call logical instructions in the memory 630 to execute a prompt word optimization method for generating video. This method includes: generating an initial generated video based on the prompt word to be optimized; selecting reference videos from a high-quality video library based on the prompt word to be optimized; comparing the initial generated video and the reference video to obtain a difference comparison result; and optimizing the prompt word to be optimized based on the difference comparison result to obtain an optimized prompt word.

[0121] Furthermore, the logical instructions in the aforementioned memory 630 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0122] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the prompt word optimization method for generating videos provided by the above methods. The method includes: generating an initial generated video based on the prompt words to be optimized; selecting reference videos from a high-quality video library based on the prompt words to be optimized; comparing the initial generated video and the reference videos to obtain a difference comparison result; and optimizing the prompt words to be optimized based on the difference comparison result to obtain optimized prompt words.

[0123] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon. When executed by a processor, the computer program implements a method for optimizing prompt words for generating videos, as provided by the methods described above. The method includes: generating an initial generated video based on the prompt words to be optimized; selecting reference videos from a high-quality video library based on the prompt words to be optimized; comparing the initial generated video and the reference videos to obtain a difference comparison result; and optimizing the prompt words to be optimized based on the difference comparison result to obtain optimized prompt words.

[0124] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0125] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0126] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for optimizing prompt words in video generation, characterized in that, include: Based on the prompts to be optimized, an initial video is generated. Based on the suggested words to be optimized, reference videos are selected from a high-quality video library; The initially generated video and the reference video are compared to obtain the difference comparison results; Based on the difference comparison results, the prompt words to be optimized are optimized to obtain optimized prompt words.

2. The method for optimizing prompt words for video generation according to claim 1, characterized in that, The step of comparing the initially generated video and the reference video to obtain the difference comparison result includes: The initial generated video and the reference video are compared in terms of spatiotemporal features to obtain the spatiotemporal comparison result. The initially generated video is subjected to image perception quality assessment to obtain the quality assessment result; The semantic comparison between the prompt words to be optimized and the initially generated video is performed to obtain the semantic comparison results; The difference comparison result is obtained based on the quality assessment result and / or the semantic comparison result, as well as the spatiotemporal comparison result; The difference comparison results are used to characterize the degree of deviation of the initially generated video from the reference video in terms of multidimensional features.

3. The method for optimizing prompt words for video generation according to claim 2, characterized in that, The step of performing spatiotemporal feature comparison between the initially generated video and the reference video to obtain spatiotemporal comparison results includes: The initial generated video and the reference video are sampled respectively to obtain a first generated video frame set and a reference video frame set; Extract the spatiotemporal feature set of the first generated video frame set and the reference spatiotemporal feature set of the reference video frame set respectively; Calculate the distance between the generated spatiotemporal feature set and the reference spatiotemporal feature set to obtain the spatiotemporal alignment result; Arbitrary spatiotemporal feature sets are used to reflect the motion trajectories of objects and the patterns of scene changes in a video frame set.

4. The method for optimizing prompt words for video generation according to claim 2, characterized in that, The step of performing image perception quality assessment on the initially generated video to obtain quality assessment results includes: The initially generated video is sampled to obtain a second set of generated video frames; The image perception quality is evaluated for each second generated video frame in the second generated video frame set to obtain the perception quality score for each second generated video frame. The quality assessment result is calculated based on the perceived quality score of each of the second generated video frames.

5. The method for optimizing prompt words for video generation according to claim 2, characterized in that, The step of performing a semantic comparison between the prompt word to be optimized and the initially generated video to obtain the semantic comparison result includes: The initially generated video is sampled to obtain a third set of generated video frames; Extract the generated image features of each third generated video frame in the third generated video frame set, and extract the text features of the prompt word to be optimized; The semantic comparison result is calculated based on the similarity between each generated image feature and the text feature.

6. The method for optimizing prompt words for generating video according to any one of claims 1 to 5, characterized in that, The optimization of the prompt words to be optimized based on the difference comparison results to obtain optimized prompt words includes: If the difference comparison result does not meet the preset video quality threshold, the prompt word to be optimized is optimized based on the difference comparison result to obtain the intermediate optimized prompt word; Based on the intermediate optimization prompts, an intermediate generated video is regenerated. The intermediate generated video is then compared with the intermediate reference video obtained by re-filtering based on the intermediate optimization prompts to obtain an intermediate difference comparison result. If the intermediate difference comparison result still does not meet the preset video quality threshold, the optimization, generation, filtering, and comparison steps are repeated until the obtained intermediate difference comparison result meets the preset video quality threshold. Finally, the final intermediate optimization prompts are output as the optimization prompts.

7. The method for optimizing prompt words for video generation according to claim 6, characterized in that, The optimization of the prompt words to be optimized based on the difference comparison results to obtain intermediate optimized prompt words includes: The suggestion word examples corresponding to the difference comparison results are obtained by matching from a high-quality suggestion word library; Based on the suggested word examples, the difference comparison results, and the suggested words to be optimized, a suggested text is constructed; The prompt text is input into the large language model to obtain the intermediate optimized prompt word output by the large language model.

8. A device for optimizing prompts in video generation, characterized in that, include: The generation unit generates an initial video based on the prompts to be optimized; The filtering unit selects reference videos from a high-quality video library based on the prompts to be optimized. The difference comparison unit compares the initially generated video and the reference video to obtain the difference comparison result; The optimization unit optimizes the prompt word to be optimized based on the difference comparison results to obtain the optimized prompt word.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the prompt word optimization method for generating video as described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the prompt word optimization method for generating video as described in any one of claims 1 to 7.

11. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the prompt word optimization method for generating video as described in any one of claims 1 to 7.

Images