Model fine-tuning method, device, electronic equipment, storage medium and program product

Abstract

The model fine-tuning method and device, the electronic device, the storage medium and the program product are disclosed. The prompt description data for the target weakness dimension is generated by using the video description content with the defect feature in the target weakness dimension and the video description content with the preset target feature, the prompt description data is expanded and enhanced in multiple types, a large number of target prompt data sets of multiple types are formed, the target training data set is generated according to the target prompt data sets of multiple types and diversities, the diversified training samples are constructed, and the fine-tuning training optimization of the to-be-fine-tuned model is performed after the training samples are evaluated. In this way, the data quality of the training data set is good, the training target can be focused, and the adaptation process of the to-be-fine-tuned model is accelerated. The target prompt data set of multiple types is generated, the training data of the to-be-fine-tuned model can be enriched, manual labeling is not needed, and the fine-tuning process of the model is accelerated.

Description

Technical Field

[0001] This disclosure relates to Internet technology, and in particular to a method, apparatus, electronic device, computer-readable storage medium, and computer program product for model fine-tuning. Background Technology

[0002] Diffusion models are probabilistic generative models that simulate the forward diffusion process of data into noise and its reverse denoising process to achieve high-quality data generation. They are widely used in image, video, and speech synthesis. Advances in diffusion models have significantly improved video generation quality. With the help of large-scale training data and resources, these pre-trained models perform well in surface attributes (such as aesthetics, temporal smoothness, and instruction consistency), but they still have shortcomings in some lower-level video quality dimensions, such as physical credibility and the maintenance of object presence. This often reduces the coherence and credibility of the generated content.

[0003] In related technologies, to address these shortcomings and deficiencies, it is necessary to fine-tune the pre-trained model to improve the generation quality in specific dimensions. Existing fine-tuning methods are generally divided into two categories: The first category is supervised fine-tuning, such as Wan (an open-source video generation and editing model) and HunyuanVideo (a text-based video model). After pre-training, it is necessary to collect and introduce large-scale, high-quality labeled data in specific dimensions and continue training in the same way. This fine-tuning method has a high computational cost. The second category uses a reinforcement learning-like training method. This type of method usually requires collecting manually labeled preference scores to train the reward model. The trained reward model is then used to achieve training optimization. It relies on manually labeled data and requires a large amount of human resources. Summary of the Invention

[0004] To address the technical problems in related technologies, embodiments of this disclosure provide a method, apparatus, electronic device, computer-readable storage medium, and computer program product for model fine-tuning.

[0005] According to a first aspect of the present disclosure, a method for model fine-tuning is provided, the method comprising:

[0006] Obtain a set of basic prompt content, which includes a first prompt content, a second prompt content, and a third prompt content. The first prompt content includes video description content generated by the model to be fine-tuned that has defect features in the target weakness dimension. The second prompt content includes video description content with preset target features in the target weakness dimension. The third prompt content includes video description content from preset source real videos.

[0007] Based on the first and second prompts, a multi-type target prompt dataset is generated. The target prompt dataset includes prompt description data that controls the fine-tuning training of the model to be fine-tuned for the target weakness dimension.

[0008] A first target training dataset is generated based on the target prompt dataset and the third prompt content. The first target training dataset includes a first synthetic video set and a second synthetic video set. The first synthetic video set includes synthetic video data that corresponds one-to-one with the prompt description data in the target prompt dataset and is used to optimize the target weakness dimension of the model to be fine-tuned. The second synthetic video set includes synthetic video data generated based on the third prompt content.

[0009] The synthetic video data in the first target training dataset is evaluated to obtain the evaluation results;

[0010] Based on the evaluation results and the first target training dataset, the network parameters of the model to be fine-tuned are optimized and updated to obtain the target model, which is the model to be fine-tuned after the network parameters have been optimized and updated.

[0011] As an optional embodiment, the generation of a multi-type target prompt dataset based on the first prompt content and the second prompt content, wherein the target prompt dataset includes prompt description data that controls the fine-tuning training of the model to be fine-tuned for the target weakness dimension, including:

[0012] Based on the first prompt, determine the key attributes for the target's weakness dimension;

[0013] Based on the second prompt content and the preset large language model, generate meta-prompt text for all the key attributes;

[0014] The meta-hint text is expanded using the preset large language model to generate multiple types of hint description data corresponding to the meta-hint text, thus obtaining the target hint dataset.

[0015] As an optional embodiment, the generation of a first target training dataset based on the target cue dataset and the third cue content, wherein the first target training dataset includes a first synthetic video set and a second synthetic video set, the first synthetic video set including synthetic video data that corresponds one-to-one with the cue description data in the target cue dataset and is used to optimize the target weakness dimension of the model to be fine-tuned, and the second synthetic video set including synthetic video data generated based on the third cue content, including:

[0016] The target cue dataset is used as the first input data of the model to be fine-tuned, so that the model to be fine-tuned generates synthetic video data corresponding to each type of cue description data in the target cue dataset, which is used to optimize the training of the target weakness dimension of the model to be fine-tuned, and thus the first synthetic video set is obtained.

[0017] The third prompt content is used as the second input data of the model to be fine-tuned, so that the model to be fine-tuned generates synthetic video data corresponding to the preset source real video according to the third prompt content, and the second synthetic video set is obtained.

[0018] The first synthetic video set and the second synthetic video set are determined as the first target training dataset.

[0019] As an optional embodiment, the evaluation of the synthetic video data in the first target training dataset to obtain the evaluation result includes:

[0020] For any synthetic video data, based on the prompt description data corresponding to the synthetic video data and the target weakness dimension corresponding to the synthetic video data, a dimension evaluation question-and-answer pair corresponding to the target weakness dimension is generated based on a preset video evaluation standard;

[0021] The synthesized video data is evaluated by combining the aforementioned dimensional evaluation questions and answers to obtain the evaluation result.

[0022] As an optional embodiment, the evaluation of the synthesized video data by combining the dimension evaluation question-answer pair to obtain the evaluation result includes:

[0023] For any synthetic video data, the content of the synthetic video data is compared with its corresponding dimension evaluation question-and-answer pair. In response to determining that the content of the synthetic video data is consistent with the content described by its corresponding dimension evaluation question-and-answer pair, an evaluation score is output, and the evaluation score is the evaluation result.

[0024] As an optional embodiment, the optimization and updating of the network parameters of the model to be fine-tuned based on the evaluation results and the first target training dataset to obtain the target model, wherein the target model is the model to be fine-tuned after the network parameter optimization and update, includes:

[0025] The synthetic video data corresponding to the first target training dataset is weighted according to the evaluation score to obtain the second target training dataset.

[0026] The model to be fine-tuned is trained using the second target training dataset to optimize and update the network parameters of the model to be fine-tuned, thereby obtaining the target model, which is the model to be fine-tuned after the network parameter optimization and update.

[0027] As an optional embodiment, the optimization and updating of the network parameters of the model to be fine-tuned based on the evaluation results and the first target training dataset to obtain the target model, wherein the target model is the model to be fine-tuned after the network parameter optimization and update, includes:

[0028] The synthetic video data corresponding to the first target training dataset is weighted according to the evaluation score to obtain the second target training dataset.

[0029] The model to be fine-tuned is trained using the second target training dataset, the second synthetic video set, and the preset source real videos corresponding to the second synthetic video set. The network parameters of the model to be fine-tuned are optimized and updated using the second target training dataset. The first optimization model is constrained and optimized based on the consistency evaluation results of the second synthetic video set and the preset source real videos corresponding to the second synthetic video set, to obtain the target model, which is the model to be fine-tuned after the network parameter optimization and update.

[0030] According to a second aspect of the present disclosure, an apparatus for model fine-tuning is provided, the apparatus comprising:

[0031] The basic prompt content set module is used to obtain a basic prompt content set, which includes a first prompt content, a second prompt content, and a third prompt content. The first prompt content includes video description content generated by the model to be fine-tuned that has defect features in the target weakness dimension. The second prompt content includes video description content with preset target features in the target weakness dimension. The third prompt content includes video description content from preset source real videos.

[0032] The first data generation module is used to generate multiple types of target prompt datasets based on the first prompt content and the second prompt content. The target prompt dataset includes prompt description data that controls the fine-tuning training of the model to be fine-tuned for the target weakness dimension.

[0033] The second data generation module is used to generate a first target training dataset based on the target prompt dataset and the third prompt content. The first target training dataset includes a first synthetic video set and a second synthetic video set. The first synthetic video set includes synthetic video data that corresponds one-to-one with the prompt description data in the target prompt dataset and is used to optimize the target weakness dimension of the model to be fine-tuned. The second synthetic video set includes synthetic video data generated based on the third prompt content.

[0034] The video data evaluation module is used to evaluate the synthetic video data in the first target training dataset and obtain the evaluation results;

[0035] The model fine-tuning and optimization module is used to optimize and update the network parameters of the model to be fine-tuned based on the evaluation results and the first target training dataset to obtain the target model, which is the model to be fine-tuned after the network parameter optimization and update.

[0036] As an optional embodiment, the first data generation module includes:

[0037] The first determining unit is used to determine the key attributes for the target weakness dimension based on the first prompt content.

[0038] The meta-prompt generation unit is used to generate meta-prompt text for all the key attributes based on the second prompt content and the preset large language model;

[0039] The prompt dataset generation unit is used to expand the meta-prompt text using the preset large language model to generate multiple types of prompt description data corresponding to the meta-prompt text, thereby obtaining the target prompt dataset.

[0040] As an optional embodiment, the second data generation module includes:

[0041] The first video synthesis unit is used to take the target cue dataset as the first input data of the model to be fine-tuned, so that the model to be fine-tuned generates synthetic video data corresponding to each type of cue description data in the target cue dataset, which is used to optimize the training of the target weakness dimension of the model to be fine-tuned, and obtains the first synthetic video set.

[0042] The second video synthesis unit is used to take the third prompt content as the second input data of the model to be fine-tuned, so that the model to be fine-tuned generates synthesized video data corresponding to the preset source real video according to the third prompt content, and obtains the second synthesized video set.

[0043] The training dataset determination unit is used to determine the first synthetic video set and the second synthetic video set as the first target training dataset.

[0044] As an optional embodiment, the video data evaluation module includes:

[0045] The first question-and-answer pair generation unit is used to generate a dimension evaluation question-and-answer pair corresponding to the target weakness dimension based on a preset video evaluation standard, according to the prompt description data corresponding to the synthetic video data and the target weakness dimension corresponding to the synthetic video data for any synthetic video data.

[0046] The first video evaluation unit is used to evaluate the synthesized video data by combining the dimension evaluation question-and-answer pair to obtain the evaluation result.

[0047] As an optional embodiment, the first video evaluation unit is further configured to:

[0048] For any synthetic video data, the content of the synthetic video data is compared with its corresponding dimension evaluation question-and-answer pair. In response to determining that the content of the synthetic video data is consistent with the content described by its corresponding dimension evaluation question-and-answer pair, an evaluation score is output, and the evaluation score is the evaluation result.

[0049] As an optional embodiment, the model fine-tuning optimization module includes:

[0050] The first weighted processing unit is used to perform weighted processing on the synthetic video data corresponding to the first target training dataset according to the evaluation score to obtain the second target training dataset.

[0051] The first optimization training unit is used to train the model to be fine-tuned using the second target training dataset to optimize and update the network parameters of the model to be fine-tuned, thereby obtaining a target model, which is the model to be fine-tuned after the network parameter optimization and update.

[0052] As an optional embodiment, the model fine-tuning and optimization module further includes:

[0053] The second weighted processing unit is used to perform weighted processing on the synthetic video data corresponding to the first target training dataset according to the evaluation score, so as to obtain the second target training dataset.

[0054] The second optimization training unit is used to train the model to be fine-tuned using the second target training dataset, the second synthetic video set, and the preset source real videos corresponding to the second synthetic video set. The network parameters of the model to be fine-tuned are optimized and updated using the second target training dataset. The first optimization model is constrained and optimized based on the consistency evaluation results of the second synthetic video set and the preset source real videos corresponding to the second synthetic video set, so as to obtain the target model, which is the model to be fine-tuned after the network parameter optimization and update.

[0055] According to a third aspect of the present disclosure, an electronic device is provided, comprising:

[0056] Memory, used to store computer program products;

[0057] A processor is configured to execute a computer program product stored in the memory, and when the computer program product is executed, to implement the method described in the first aspect above.

[0058] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided that stores computer program instructions thereon, which, when executed by a processor, implement the method described in the first aspect.

[0059] According to a fifth aspect of the present disclosure, a computer program product is provided, including computer program instructions that, when executed by a processor, implement the method described in the first aspect above.

[0060] In this embodiment, target weakness dimension-specific prompt description data is generated by combining video description content with defect features and video description content with preset target features. This prompt description data is then expanded and enhanced in various ways to form a large number of multi-type target prompt datasets. Based on these diverse target prompt datasets, a target training dataset (video data) is generated to construct diverse training samples. These training samples are then evaluated and used for fine-tuning and optimizing the model to be fine-tuned. In this way, the training dataset generated specifically for the target weakness dimension of the model to be fine-tuned has better training data quality, maintains focus on the training objective, and accelerates the adaptation process of the model. The generation of multi-type target prompt datasets increases the diversity of training data, enriches the training data for the model to be fine-tuned, and makes the generation of training data more efficient, thus accelerating the model fine-tuning process. Furthermore, it eliminates the need for manual annotation, resulting in better stability and robustness of the model to be fine-tuned.

[0061] The technical solutions of this disclosure will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0062] The accompanying drawings, which form part of this specification, illustrate embodiments of this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0063] This disclosure will become clearer with reference to the accompanying drawings and the following detailed description, wherein:

[0064] Figure 1 This is a flowchart illustrating one embodiment of a model fine-tuning method according to the present disclosure.

[0065] Figure 2 This is a second schematic flowchart illustrating a model fine-tuning method according to an embodiment of the method disclosed herein.

[0066] Figure 3 This is the third flowchart illustrating a model fine-tuning method according to one embodiment of the method disclosed herein.

[0067] Figure 4 This is a structural block diagram of a model fine-tuning device according to one embodiment of the present disclosure.

[0068] Figure 5 This is a structural block diagram of an electronic device according to an embodiment of the present disclosure. Detailed Implementation

[0069] In related technologies, model fine-tuning methods either rely on massive amounts of training data with enormous computational costs or heavily depend on costly manual annotation. Specifically, supervised fine-tuning (SFT)-based methods utilize large-scale, high-quality datasets to expand model training. While this effectively improves general generative capabilities, it requires significant computational resources and manual data processing, resulting in high costs and low efficiency. Reinforcement learning-based methods aim to make generation conform to human preferences through feedback-driven supervision, including reward-weighted regression and direct preference optimization. However, these methods still rely on human-annotated data to train reward models, also requiring substantial human resources.

[0070] To address the technical problems in related technologies, embodiments of this disclosure provide a method, apparatus, electronic device, computer-readable storage medium, and computer program product for model fine-tuning. On the one hand, diverse training samples (video data) can be constructed by generating and expanding cue description data specifically targeting the weaknesses (target weakness dimensions) of the model to be fine-tuned. On the other hand, from the perspective of fine-tuning optimization, model fine-tuning is performed based on the evaluation feedback (e.g., reward-oriented) of the training samples.

[0071] To enable those skilled in the art to clearly and accurately understand the specific technical solutions of the embodiments of this disclosure, the technical solutions of the embodiments of this disclosure will be described in detail below with reference to the accompanying drawings.

[0072] Figure 1 This is a flowchart illustrating one embodiment of a model fine-tuning method according to the present disclosure.

[0073] Step 101: Obtain a set of basic prompts, which includes a first prompt, a second prompt, and a third prompt. The first prompt includes video descriptions generated by the model to be fine-tuned that have defect features in the target weakness dimension. The second prompt includes video descriptions with preset target features in the target weakness dimension. The third prompt includes video descriptions from preset source real videos.

[0074] In this embodiment, the basic prompt content set includes, but is not limited to, a number of historical failure case data (i.e., first prompt content) and a small number of seed examples (i.e., second prompt content) of the model to be fine-tuned (e.g., a pre-trained text-based video model). The historical failure case data can be negative prompt content, which may come from defective results previously generated by the model to be fine-tuned, while the seed examples can be positive prompt content, which may come from a small number of one-time manually labeled positive prompts. It also includes video description content from preset source real videos (i.e., third prompt content).

[0075] In this embodiment, the preset source real video refers to video content obtained through real-world filming. The third prompt content refers to a precise textual description of the core content of the real-world filmed video, obtained through manual annotation or automatic analysis by AI (Artificial Intelligence). These textual descriptions (third prompt content) must meet two core objectives: 1. Objectively record the actual visual and spatiotemporal events that occur in the video; 2. Structure key elements to adapt to AI training requirements.

[0076] The target weakness dimension refers to the systematic and categorizable defects or deficiencies exhibited by the model to be fine-tuned in a specific aspect. It defines the specific directions and evaluation criteria for improvement of the model. It does not refer to overall poor performance of the model, but rather to poor performance in a specific area, specifically a recurring problem pattern under a particular type of input prompt, which can be identified and quantified by observing or analyzing the generated video data. For example, the model may frequently make errors when generating videos involving complex object collision physics, or the generated videos may show a significant lack of coherence and naturalness in the movements of characters.

[0077] Specifically, historical failure case data consists of video descriptions generated by the model to be fine-tuned that exhibit flawed characteristics in the target weakness dimension (e.g., defective video samples that do not conform to physical laws, logic, or visual consistency). Conversely, the second set of prompts includes video descriptions with preset target characteristics in the target weakness dimension (e.g., standard video samples that conform to physical laws, logic, or visual consistency), which are used to demonstrate the correct representation of the target weakness dimension. For example, the first prompt might be a video of water being poured out and then suspended in mid-air, which is a sample with kinematic distortion defects. Its annotation must correspond to the target weakness dimension, such as motion inconsistency. The second prompt, on the other hand, might be water flowing naturally into a cup, which is an ideal demonstration of the target weakness dimension (e.g., "smooth fluid motion," "dynamically consistent collisions," etc.). Its annotation must clearly indicate the target weakness dimension it interprets, such as fluid dynamics correctness.

[0078] Thus, by using a number of historical failure case data (first prompt content) and a small number of seed examples (second prompt content) as part of the basic prompt content set, the "capability blind spot" (i.e., the target weakness dimension) of the model to be fine-tuned can be accurately located. This provides an effective data foundation for subsequent targeted repair of the model's defects, avoids deviations from reality in the distribution of subsequently generated training videos, and provides basic reference data support for subsequent optimization directions through a small number of seed examples, preventing over-correction during subsequent optimization and the generation of new problems. In summary, this embodiment of the disclosure uses both positive and negative prompt content as the basic prompt content set, which can lay an effective data foundation for achieving accurate defect repair.

[0079] Step 102: Generate a multi-type target prompt dataset based on the first prompt content and the second prompt content. The target prompt dataset includes prompt description data that controls the fine-tuning training of the model to be fine-tuned for the target weakness dimension.

[0080] As an optional embodiment, key attributes targeting the weakness dimension can be determined based on the first prompt content. Meta prompt texts for all key attributes can be generated based on the second prompt content and a preset large language model. The meta prompt texts can be expanded using the preset large language model to generate multiple types of prompt description data corresponding to the meta prompt texts, thus obtaining the target prompt dataset.

[0081] The prompt description data can be a set of structured text instructions that precisely control the semantic space coordinates of a pre-defined large language model. This data can include target weakness dimension anchoring, key visual / temporal attribute constraints, and distribution control signals (used to guide generated samples to fall within the target distribution). In this embodiment, the prompt description data is a variant or extension of the meta-prompt text. Thus, the target prompt dataset is a collection of a large amount of prompt description data. The meta-prompt text is a structured text instruction template used to guide the pre-defined large language model to generate the target prompt dataset. It can be generated by fusing defect feature samples (i.e., the first prompt content) with the target constraint prototype (the second prompt content) to construct prompt generation rules on the target weakness dimension.

[0082] Based on the first prompt, the target weakness dimension (i.e., the specific defect dimension of the model to be improved or optimized, such as unreasonable object movement physics, stiff character movements, chaotic scene spatial relationships, poor temporal coherence, etc.) can be identified, allowing for focused optimization of weaknesses or defects. Key attributes of the target weakness dimension can be extracted from the first prompt, such as key visual attributes and / or key temporal attributes. Key visual attributes include, but are not limited to, attributes related to objects, scenes, colors, and textures, while key temporal attributes include, but are not limited to, attributes related to actions, movement trajectories, speed, duration, and the sequence of events. Then, based on the second prompt, the pre-defined large language model learns how to positively, clearly, and completely describe these key attributes of the target weakness dimension to generate meta-prompt text.

[0083] For example, suppose the improvement direction (target weakness dimension) of the model to be fine-tuned is "improving the physical plausibility of objects after a collision." Based on the first prompt content (e.g., it can include video and its original prompt), the key attributes of this improvement direction (target weakness dimension) are determined to include key visual attributes (e.g., 1. The prompt itself is vague: "The ball hits the wall," without explaining how it hits or what happens afterward; 2. The prompt omits key physical attributes: it doesn't explain the ball's material, weight, speed, or impact angle) and key temporal attributes (e.g., the prompt's time description is inaccurate or inconsistent: "The ball rolls over and then bounces," without explaining the connection between "rolling" and "bouncing"). Thus, a list of key visual / temporal attributes missing from the first prompt content is summarized. For example, for "collision physical plausibility," key attributes might include: object material, mass, initial velocity vector, impact point, impact angle, expected energy loss / rebound method, subsequent trajectory, and secondary interactions with other objects.

[0084] In addition, a second set of prompts is provided, such as 3 to 5 high-quality examples of prompt text for the areas to be improved in the model to be fine-tuned (target weakness dimension), demonstrating how to correctly describe the target weakness dimension and explicitly including the list of missing key visual / temporal attributes summarized from the first set of prompts. For example, assuming the area to be improved in the model to be fine-tuned (target weakness dimension) is "collision physics plausibility," an example of the second set of prompts is as follows:

[0085] Example 1: A heavy rubber ball (material, mass) strikes a hard brick wall (impact object property) at a moderate speed (velocity) and a 45-degree angle (angle). Show the ball undergoing noticeable deformation at the point of impact (visual detail), then bounces off along a smooth parabolic trajectory (motion trajectory) at a smaller angle and decreasing speed (result property), and rolls a distance on a rough surface (environment) before slowly coming to a stop (final state).

[0086] Example 2: A lightweight ping-pong ball (material, mass) is struck horizontally at high speed (velocity, direction) and hits the edge of a fixed wooden tabletop (point of impact, property). The ball is shown to bounce with almost no deformation at the point of contact (visual detail), at an angle close to the angle of incidence (physical laws) and with a slight decrease in velocity (energy loss), briefly flying through the air (temporal dynamics) before landing on the other side of the table.

[0087] Then, the key attributes for the target weakness dimension identified by the first prompt and the second prompt are used as input to the preset large language model, and prompt instructions are given to the preset large language model to generate meta-prompt text. For example, "Please analyze the structure and content of the above second prompt, especially how they describe [key attribute 1, key attribute 2...], and conceive a new scenario with similar descriptive depth and targeting" and / or "Please imitate the level of detail and clarity of excellent examples to generate entirely new text prompts. These prompts should focus on [target weakness dimension] and explicitly include descriptions of [key attribute 1, key attribute 2...]", thus guiding the preset large language model to generate meta-prompt text.

[0088] As one optional embodiment, the preset large language model can be guided to generate meta-prompt text and then output in the form of a list. As another optional embodiment, the meta-prompt text generated by the preset large language model can be further deduplicated. For example, by calculating the similarity between pairs of meta-prompt texts, any two pairs of meta-prompt texts with high similarity can be deduplicated to obtain the final meta-prompt text.

[0089] In this embodiment, to diversify the training data and improve the model's generalization ability and robustness, the meta-prompt text can be further extended using the preset large language model. This involves transforming the basic prompt (meta-prompt text) in terms of language style, scene atmosphere, expressive perspective, and detail granularity without altering the original core semantics (key visual / temporal attributes), generating numerous semantically equivalent but expressively different prompt text variants, thereby enhancing the diversity of the training data. For example, the prompt instruction "Please rewrite the following meta-prompt text in a [science fiction style] version" can be used. This causes the preset large language model to generate various extensions (or variants) of each meta-prompt text based on the aforementioned meta-prompt text. These extended meta-prompt text extensions constitute different styles / types of prompt description data corresponding to the original meta-prompt text. All the different styles / types of prompt description data corresponding to the meta-prompt text are used as the target prompt dataset.

[0090] In some embodiments, the contrastive learning mechanism of the first language model can also be used to generate meta-hint text. For example, the first language model can use its contrastive learning mechanism to perform difference analysis on the first and second hint content, and then output meta-hint text based on the difference analysis results. For instance, if the first hint content is "The metal ball bounces back according to the conservation of momentum after the collision," and the second hint content is "The metal ball remains viscous after the collision," then the meta-hint text output by the first language model would be: "Generate two polished metal balls that collide at a 45-degree angle, demonstrating the separation motion after energy transfer." This reduces the data processing steps for the first and second hint content, improving efficiency.

[0091] Step 103: Generate a first target training dataset based on the target prompt dataset and the third prompt content. The first target training dataset includes a first synthetic video set and a second synthetic video set. The first synthetic video set includes synthetic video data that corresponds one-to-one with the prompt description data in the target prompt dataset and is used to optimize the target weakness dimension of the model to be fine-tuned during training. The second synthetic video set includes synthetic video data generated based on the third prompt content.

[0092] The synthesized video data is video content that is automatically generated entirely by an artificial intelligence algorithm model (such as the model to be fine-tuned in the embodiments of this disclosure) rather than obtained through real-world shooting. In the embodiments of this disclosure, the synthesized video data is video content automatically created by the model to be fine-tuned (such as the Wensheng video model) according to input instructions (such as prompt description data, including but not limited to text prompts, parameters, blueprints, etc.) to repair the weaknesses and defects of the model to be fine-tuned.

[0093] In this embodiment, the target cue dataset can be used as the first input data for the model to be fine-tuned. The model generates synthetic video data corresponding to the target weakness dimensions of the model based on the cue description data of each type in the target cue dataset, thus optimizing the training of the model. This results in a first synthetic video set. Then, the third cue content is used as the second input data for the model to be fine-tuned. The model generates synthetic video data corresponding to a preset source real video based on the third cue content, resulting in a second synthetic video set. Finally, the first and second synthetic video sets are determined as the first target training dataset. This constructs a controlled and focused training data distribution, improving the diversity and efficiency of the training data, accelerating the model adaptation process, and preventing the model from overfitting the synthetic distribution or deviating from the attributes of the real video by using synthetic video data from preset source real videos.

[0094] To enable those skilled in the art to clearly and accurately understand the technical content of steps 101 to 103, the following example illustrates the entire process of generating synthetic video data regarding the weakness of the "physical rationality of liquid pouring" in the model to be fine-tuned:

[0095] The first step is to obtain the first and second prompts.

[0096] First prompt (failure case prompt): "Pour milk into a glass" (too blurry, causing physical distortion);

[0097] The second tip (manual seed example): "A bucket of viscous honey is poured from a height of 30cm into a wide-mouthed glass. The initial flow rate is fast, causing splashing on the glass wall. Then the flow rate slows down, forming a sticky liquid column. The liquid level in the glass gradually rises and is accompanied by slow shaking."

[0098] The second step is to generate a target prompt dataset using a pre-set large language model.

[0099] First, meta-prompt text is generated using a pre-defined large language model:

[0100] Original prompt text: "Pour 250ml of room temperature milk vertically from a height of 25cm into a conical wine glass. Initially, a continuous jet of water will splash against the bottom of the glass. In the middle stage, the liquid jet will become thinner and produce bubbles. In the later stage, the liquid surface will stabilize after shaking."

[0101] Then, the meta-prompt text is expanded and enhanced using a pre-defined large language model:

[0102] Extended prompt text or prompt description data 1 (technical document style): "Experimental record: Room temperature whole milk (density 1.03 g / ml) was freely dropped from 25 cm ± 0.5 cm into a conical wine glass (wall thickness 2 mm). High-speed photography shows that at t = 0.3 s, the liquid column diameter of 5 mm impacted the bottom of the glass, producing a crown-shaped splash; at t = 1.2 s, the liquid level rose to 50 mm and damped oscillations occurred."

[0103] Extended prompt text or prompt description data 2 (children's science style): "Look! The milk baby jumps from the tall milk waterfall into the glass slide cup, splashing water everywhere! Slowly, it turns into a thin milk line, and finally falls asleep swaying in the cup~".

[0104] In this way, a large amount of meta-hint text and its corresponding extended hint description data constitute the target hint dataset.

[0105] The third step involves generating the first target training dataset, i.e., generating synthetic video data, based on the target cue dataset. The cue description data from the target cue dataset is input into the model to be fine-tuned, which then generates synthetic video data based on this cue description data. For example:

[0106] Synthetic video data 1: Technical document style prompts (the aforementioned extended prompt text or prompt description data 1) → Generate scientific research documentary-style video:

[0107] Frames 1-10: The ruler shows a height of 25cm, and the milk column falls precisely;

[0108] Frame 15: Slow-motion splashing particles appear at the bottom of the cup (consistent with fluid dynamics simulation);

[0109] Frame 30: The liquid level reaches 50mm and wobbles slightly (amplitude 3mm).

[0110] Synthesized video data 2: Children's science-themed prompts (the aforementioned extended prompt text or prompt description data 2) → Generate cartoon-enhanced video:

[0111] The milk is personified as a smiling face, and when it splashes, it forms a teardrop-shaped "skirt". The fluctuation of the liquid surface is exaggerated, but it conforms to the law of energy decay.

[0112] In addition, corresponding synthetic video data is generated by using the video description content (third-party prompt content) of the preset source real video. For example, "The scene is set on a stove in the kitchen, a pot is placed on a black countertop...", thus generating a corresponding synthetic video data (such as...). Figure 3 The image in Figure A is a "realistically synthesized video" generated by a text-based image model.

[0113] Step 104: Evaluate the synthetic video data in the first target training dataset to obtain the evaluation results.

[0114] As an optional embodiment of this disclosure, video data in the first target training dataset and its corresponding prompt description data can be formed into data pairs such as "video-prompt". Then, this video-prompt data pair is sent to a preset large language model to utilize the preset large language model to transform abstract quality dimensions into concrete, verifiable standards based on world knowledge. Specifically, for any synthetic video data, based on the prompt description data corresponding to the synthetic video data and the target weakness dimension corresponding to the synthetic video data, a dimension evaluation question-and-answer pair corresponding to the target weakness dimension is generated based on preset video evaluation standards (i.e., concrete, verifiable standards based on world knowledge of the preset large language model, which can correctly understand physical laws, common sense, and predict what kind of expected / reasonable results, etc.). The synthetic video data is then evaluated using the dimension evaluation question-and-answer pair to obtain an evaluation result. Further, the content of the synthetic video data can be compared with its corresponding dimension evaluation question-and-answer pair to determine whether the content of the synthetic video data is consistent with the content described by its corresponding dimension evaluation question-and-answer pair. If it is determined that the content of the synthetic video data is consistent with the content described by its corresponding dimension evaluation question-and-answer pair, an evaluation score is output, and the evaluation score is determined as the evaluation result.

[0115] For example, for any synthetic video data, prompts and target evaluation dimensions (e.g., action consistency or physical credibility) are input into a pre-defined large language model (MLM) to instruct the MLM to evaluate the synthetic video data according to the target evaluation dimensions. The MLM generates a set of dimensional evaluation question-and-answer (QA) pairs according to the prompts. These QA pairs are typically in "yes / no" format, representing the expected result when the synthetic video data correctly demonstrates the expected behavior, serving as an explicit check of the target evaluation dimensions. The synthetic video data and its corresponding QA pairs are then input into a multimodal large language model (MLM). This MLM further evaluates whether the synthetic video data is consistent with the expected outcome posed by the questions in the QA pairs (which could be specific dimensional world knowledge encoded in the MLM). As an optional embodiment, the MLM does not return a specific evaluation answer but outputs an evaluation score that reflects the degree of consistency between the synthetic video data and the questions in the QA pairs. For example, synthetic video data that better matches the questions in the QA pairs receives a higher score. This allows for scalable evaluation without the need for manual annotation.

[0116] like Figure 3Taking the example of a cheese cheese image, the input to the pre-defined large language model is: "A time-lapse video of cheese changing with temperature. A piece of yellow cheese is placed on a table, and the temperature rises from xxx to xxxx. The environment is xxx, and the atmosphere is xxx. Based on the above, ask one question from a thermodynamic perspective and provide the corresponding correct answer. Requirements: The question focuses on the visual appearance of thermodynamics, and the answer is either yes or no. Only the question-answer pair itself is output; no other explanations are provided." Therefore, the pre-defined large language model outputs the dimension evaluation question-answer pair as: "Q: Is the cheese ultimately a solid state with slightly softened edges? A: Yes."

[0117] Step 105: Based on the evaluation results and the first target training dataset, optimize and update the network parameters of the model to be fine-tuned to obtain the target model, where the target model is the model to be fine-tuned after the network parameter optimization and update.

[0118] According to embodiments of this disclosure, the synthetic video data in the first target training dataset can be weighted according to the evaluation score to obtain the second target training dataset. The model to be fine-tuned is then trained using the second target training dataset to optimize and update the network parameters of the model, resulting in the target model. The target model is the model to be fine-tuned after the network parameter optimization and update. As an example, the weighting process can be based on the evaluation score, where the evaluation score is used as an automatic weighting value for the synthetic video data; a higher score represents a higher weight, and vice versa. As another example, corresponding weight values ​​can be set based on the evaluation score; synthetic video data with higher scores corresponds to higher weights, and vice versa. The weighted synthetic video data is then used as the training data for optimization and fine-tuning (the second target training dataset), so that during training, the model to be fine-tuned increases the gradient update intensity for synthetic video data with high weight values ​​and decreases the learning weight for synthetic video data with low weight values.

[0119] As an optional embodiment, the number of training iterations can be used as the termination condition for training iterations. That is, when the number of training iterations reaches a preset number, the training of the model to be fine-tuned is stopped, and the model to be fine-tuned at this time is the target model.

[0120] exist Figure 1 Based on the illustrated embodiments, this disclosure also provides another possible embodiment. Figure 2 This is a second schematic flowchart illustrating a model fine-tuning method according to an embodiment of the present disclosure, as shown below. Figure 2 As shown, a method for fine-tuning a model may also include the following steps:

[0121] Step 101: Obtain a set of basic prompts, which includes a first prompt, a second prompt, and a third prompt. The first prompt includes video descriptions generated by the model to be fine-tuned that have defect features in the target weakness dimension. The second prompt includes video descriptions with preset target features in the target weakness dimension. The third prompt includes video descriptions from preset source real videos.

[0122] Step 102: Generate a multi-type target prompt dataset based on the first prompt content and the second prompt content. The target prompt dataset includes prompt description data that controls the fine-tuning training of the model to be fine-tuned for the target weakness dimension.

[0123] Step 103: Generate a first target training dataset based on the target prompt dataset and the third prompt content. The first target training dataset includes a first synthetic video set and a second synthetic video set. The first synthetic video set includes synthetic video data that corresponds one-to-one with the prompt description data in the target prompt dataset and is used to optimize the target weakness dimension of the model to be fine-tuned during training. The second synthetic video set includes synthetic video data generated based on the third prompt content.

[0124] The aforementioned steps 102 to 103 can be found in the following: Figure 1 For the sake of brevity, the detailed description of the relevant steps in the illustrated embodiment will not be described here.

[0125] In this embodiment, step 104 can be implemented as follows:

[0126] Step 1041: For any synthesized video data, based on the prompt description data corresponding to the synthesized video data and the target weakness dimension corresponding to the synthesized video data, generate a dimension evaluation question-and-answer pair corresponding to the target weakness dimension based on the preset video evaluation criteria.

[0127] Step 1042: For any synthetic video data, compare the content of the synthetic video data with its corresponding dimension evaluation question-and-answer pair. In response to determining whether the content of the synthetic video data is consistent with the content described by its corresponding dimension evaluation question-and-answer pair, output an evaluation score. The evaluation score is the evaluation result.

[0128] Step 105 can be achieved through the following steps:

[0129] Step 1051: Based on the evaluation scores, the corresponding synthetic video data in the first target training dataset is weighted to obtain the second target training dataset.

[0130] Step 1052: The model to be fine-tuned is trained using the second target training dataset, the second synthetic video set, and the preset source real videos corresponding to the second synthetic video set. The network parameters of the model to be fine-tuned are optimized and updated using the second target training dataset, and the model to be fine-tuned is constrained and optimized based on the consistency evaluation results of the second synthetic video set and the preset source real videos corresponding to the second synthetic video set, to obtain the target model. The target model is the model to be fine-tuned after the network parameter optimization and update.

[0131] Among these methods, optimizing and updating the network parameters of the model to be fine-tuned using the second target training dataset can be found in [reference needed]. Figure 1 The relevant description of step 105 in the illustrated embodiment.

[0132] The first optimization model is constrained and optimized based on the consistency evaluation results between the second synthetic video set and the corresponding preset source real video. Specifically, the preset divergence loss function is, for example, the Kullback-Leibler Divergence Loss function. As an optional embodiment, for any synthetic video data in the second synthetic video set, video features (e.g., pixel features, semantic features, dynamic temporal sequence, etc.) can be extracted from the synthetic video data, and the video features of the corresponding preset source real video can be uniformly extracted. Then, the consistency evaluation score is calculated using, for example, the Kullback-Leibler Divergence Loss function, and it is determined as the consistency evaluation result. The KL divergence loss is used to measure the difference between two probability distributions (synthetic video data and target real video). In this embodiment of the disclosure, it can quantify the "distance" between the video distribution (synthetic video data) generated by the model to be fine-tuned (target model) and the real-world video distribution (target real video), forcing the two to maintain similarity. For example, when the distribution of the generated videos (synthetic video data) by the model to be fine-tuned deviates from the distribution of real-world videos (target real videos), the KL divergence loss increases and the training gradient is enhanced to strongly pull back the output distribution of the model to be fine-tuned.

[0133] In this embodiment of the disclosure, the model to be fine-tuned is trained using a second target training dataset, and the synthetic video data output by the model to be fine-tuned is obtained. During training, when the similarity between the synthetic video data output by the model to be fine-tuned and its corresponding target real video reaches a preset similarity, the training of the model to be fine-tuned is stopped, and the target model is output.

[0134] In order to enable those skilled in the art to clearly and accurately understand the technical solutions of the embodiments of this disclosure, the following is combined with Figure 3 The example shown is described in further detail. Figure 3This is the third flowchart illustrating a model fine-tuning method according to an embodiment of the present disclosure, as shown below. Figure 3 As shown, Figure 3 Figure A shows the generation process of the target prompt dataset, the first target training dataset, and the target real video. Figure B shows the evaluation process of the synthesized video and the fine-tuning and optimization training process. Figure A shows that a basic set of prompt content (such as "seed example: ...") and prompt instructions (such as "thermodynamics refers to ...") can be input into the large language model (i.e., the preset large language model) to instruct the large language model to generate meta-prompts (i.e., prompt description data; such as "cheese, 5℃ to 25℃") through category-based sentence construction. Further prompt instructions instructing the large language model to expand the meta-prompts are then input, enabling the large language model to enhance and expand the meta-prompts, generating multiple types of prompt description data (such as the synthetic prompts shown in the figure). As an optional embodiment, video prompt content (or video description data; such as "real video content as prompts"; corresponding to the third prompt content) is extracted from real video data (i.e., preset source real videos). The synthetic prompts and the prompt content from the real video data are then used as input to the model to be fine-tuned, generating a first synthetic video set (corresponding to synthetic prompts) and a second synthetic video set (corresponding to prompts for real video content or video description data of real video content), thus completing the construction of the first target training dataset. Then, the processing flow shown in Figure B proceeds. First, the first target training dataset is evaluated. For example, a large language model is used to generate dimensional evaluation question-answer pairs, such as "Q: Ultimately…; A: Yes" in Figure B. The synthetic data is then further evaluated using a multimodal large language model combined with these dimensional evaluation question-answer pairs to obtain evaluation scores. These scores are then used to weight the synthetic video data (as shown in the "weighted denoising loss" figure). The model to be fine-tuned is then trained based on the weighted synthetic video data (as shown in the text-to-image model; or the aforementioned pre-trained text-to-video model). Simultaneously, during training, for the synthetic video data in the second synthetic video set, the model to be fine-tuned is further constrained based on the corresponding preset source real videos and, for example, using KL divergence loss (as shown in Figure a, which compares the text-to-image model's generated image with the real video b, to determine the real-world constraint loss of the text-to-image model).

[0135] Thus, in this embodiment, the loss for fine-tuning the model can consist of two parts: first, encouraging the model to learn from controlled, target-vulnerability-driven synthetic video data, i.e., by performing score evaluation and weighting on the synthetic video data; and second, maintaining consistency with the distribution of real video data. This dual-objective loss can efficiently fine-tune challenging dimensions without affecting the overall generation quality. It can efficiently repair the weaknesses of the model while maintaining its natural style and generalization ability. In other words, it can specifically improve the weaknesses of the model without compromising its fundamental capabilities.

[0136] In summary, in this embodiment, prompt description data targeting the target weakness dimension is generated by combining video description content with defect features and video description content with preset target features. This prompt description data is then expanded and enhanced in multiple ways to form a large number of multi-type target prompt datasets. Based on these diverse target prompt datasets, a target training dataset (video data) is generated to construct diverse training samples. These training samples are then evaluated and used for fine-tuning and optimizing the model to be fine-tuned. Thus, the training dataset generated targeting the target weakness dimension of the model to be fine-tuned has better training data quality, maintains focus on the training objective, and accelerates the adaptation process of the model. The generation of multi-type target prompt datasets increases the diversity of training data, enriches the training data for the model to be fine-tuned, and makes the generation of training data more efficient, accelerating the model fine-tuning process. Furthermore, it eliminates the need for manual annotation, resulting in better stability and robustness of the model to be fine-tuned.

[0137] Correspondingly, this disclosure also provides device embodiments corresponding to the foregoing method embodiments. The device embodiments of this disclosure will be described in detail below with reference to the accompanying drawings. Figure 4 This is a structural block diagram of a model fine-tuning device according to one embodiment of the present disclosure. Figure 4 As shown, an apparatus for model fine-tuning according to an embodiment of this disclosure may include:

[0138] The basic prompt content set module 401 is used to obtain a basic prompt content set, which includes a first prompt content, a second prompt content, and a third prompt content. The first prompt content includes video description content generated by the model to be fine-tuned that has defect features in the target weakness dimension. The second prompt content includes video description content with preset target features in the target weakness dimension. The third prompt content includes video description content from a preset source real video.

[0139] The first data generation module 402 is used to generate a multi-type target prompt dataset based on the first prompt content and the second prompt content. The target prompt dataset includes prompt description data that controls the fine-tuning training of the model to be fine-tuned for the target weakness dimension.

[0140] The second data generation module 403 is used to generate a first target training dataset based on the target prompt dataset and the third prompt content. The first target training dataset includes a first synthetic video set and a second synthetic video set. The first synthetic video set includes synthetic video data that corresponds one-to-one with the prompt description data in the target prompt dataset and is used to optimize the target weakness dimension of the model to be fine-tuned. The second synthetic video set includes synthetic video data generated based on the third prompt content.

[0141] The video data evaluation module 404 is used to evaluate the synthetic video data in the first target training dataset and obtain the evaluation result;

[0142] The model fine-tuning optimization module 405 is used to optimize and update the network parameters of the model to be fine-tuned based on the evaluation results and the first target training dataset to obtain the target model, which is the model to be fine-tuned after the network parameter optimization and update.

[0143] In this embodiment, target weakness dimension-specific prompt description data is generated by combining video description content with defect features and video description content with preset target features. This prompt description data is then expanded and enhanced in various ways to form a large number of multi-type target prompt datasets. Based on these diverse target prompt datasets, a target training dataset (video data) is generated to construct diverse training samples. These training samples are then evaluated and used for fine-tuning and optimizing the model to be fine-tuned. In this way, the training dataset generated specifically for the target weakness dimension of the model to be fine-tuned has better training data quality, maintains focus on the training objective, and accelerates the adaptation process of the model. The generation of multi-type target prompt datasets increases the diversity of training data, enriches the training data for the model to be fine-tuned, and makes the generation of training data more efficient, thus accelerating the model fine-tuning process. Furthermore, it eliminates the need for manual annotation, resulting in better stability and robustness of the model to be fine-tuned.

[0144] exist Figure 4 Based on the embodiments shown, this disclosure also provides the following optional embodiments:

[0145] As an optional embodiment, the first data generation module includes:

[0146] The first determining unit is used to determine the key attributes for the target weakness dimension based on the first prompt content.

[0147] The meta-prompt generation unit is used to generate meta-prompt text for all the key attributes based on the second prompt content and the preset large language model;

[0148] The prompt dataset generation unit is used to expand the meta-prompt text using the preset large language model to generate multiple types of prompt description data corresponding to the meta-prompt text, thereby obtaining the target prompt dataset.

[0149] As an optional embodiment, the second data generation module includes:

[0150] The first video synthesis unit is used to take the target cue dataset as the first input data of the model to be fine-tuned, so that the model to be fine-tuned generates synthetic video data corresponding to each type of cue description data in the target cue dataset, which is used to optimize the training of the target weakness dimension of the model to be fine-tuned, and obtains the first synthetic video set.

[0151] The second video synthesis unit is used to take the third prompt content as the second input data of the model to be fine-tuned, so that the model to be fine-tuned generates synthesized video data corresponding to the preset source real video according to the third prompt content, and obtains the second synthesized video set.

[0152] The training dataset determination unit is used to determine the first synthetic video set and the second synthetic video set as the first target training dataset.

[0153] As an optional embodiment, the video data evaluation module includes:

[0154] The first question-and-answer pair generation unit is used to generate a dimension evaluation question-and-answer pair corresponding to the target weakness dimension based on a preset video evaluation standard, according to the prompt description data corresponding to the synthetic video data and the target weakness dimension corresponding to the synthetic video data for any synthetic video data.

[0155] The first video evaluation unit is used to evaluate the synthesized video data by combining the dimension evaluation question-and-answer pair to obtain the evaluation result.

[0156] As an optional embodiment, the first video evaluation unit is further configured to:

[0157] For any synthetic video data, the content of the synthetic video data is compared with its corresponding dimension evaluation question-and-answer pair. In response to determining that the content of the synthetic video data is consistent with the content described by its corresponding dimension evaluation question-and-answer pair, an evaluation score is output, and the evaluation score is the evaluation result.

[0158] As an optional embodiment, the model fine-tuning optimization module includes:

[0159] The first weighted processing unit is used to perform weighted processing on the synthetic video data corresponding to the first target training dataset according to the evaluation score to obtain the second target training dataset.

[0160] The first optimization training unit is used to train the model to be fine-tuned using the second target training dataset to optimize and update the network parameters of the model to be fine-tuned, thereby obtaining a target model, which is the model to be fine-tuned after the network parameter optimization and update.

[0161] As an optional embodiment, the model fine-tuning and optimization module further includes:

[0162] The second weighted processing unit is used to perform weighted processing on the synthetic video data corresponding to the first target training dataset according to the evaluation score, so as to obtain the second target training dataset.

[0163] The second optimization training unit is used to train the model to be fine-tuned using the second target training dataset, the second synthetic video set, and the preset source real videos corresponding to the second synthetic video set. The network parameters of the model to be fine-tuned are optimized and updated using the second target training dataset. The first optimization model is constrained and optimized based on the consistency evaluation results of the second synthetic video set and the preset source real videos corresponding to the second synthetic video set, so as to obtain the target model, which is the model to be fine-tuned after the network parameter optimization and update.

[0164] It is understood that the specific implementation process, functions and technical effects of the device embodiments provided in this disclosure correspond one-to-one with the aforementioned method embodiments. For details, please refer to the relevant descriptions of the specific embodiments in the aforementioned method embodiments. For the sake of brevity, these will not be repeated here.

[0165] Below, for reference Figure 5 This describes an electronic device according to embodiments of the present disclosure. The electronic device may be either or both of a first device and a second device, or a standalone device independent of them, which may communicate with the first device and the second device to receive acquired input signals from them.

[0166] Figure 5A block diagram of an electronic device according to an embodiment of the present disclosure is shown.

[0167] like Figure 5 As shown, the electronic device includes one or more processors and memory.

[0168] A processor can be a central processing unit (CPU) or other form of processing unit with data processing and / or instruction execution capabilities, and can control other components in an electronic device to perform desired functions.

[0169] The memory can store one or more computer program products, and the memory can include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program products can be stored on the computer-readable storage medium, and the processor can run the computer program products to implement the model fine-tuning methods of the various embodiments of this disclosure described above, and / or other desired functions.

[0170] In one example, the electronic device may also include input devices and output devices, which are interconnected via a bus system and / or other forms of connection mechanism (not shown).

[0171] In addition, the input device may also include, for example, a keyboard, a mouse, etc.

[0172] This output device can output various information to the outside, including determined distance information, direction information, etc. The output device may include, for example, a display, a speaker, a printer, and a communication network and its connected remote output devices, etc.

[0173] Of course, for the sake of simplicity, Figure 5 Only some of the components of the electronic device relevant to this disclosure are shown, omitting components such as buses, input / output interfaces, etc. In addition, the electronic device may include any other suitable components depending on the specific application.

[0174] In addition to the methods and apparatus described above, embodiments of this disclosure may also be computer program products comprising computer program instructions that, when executed by a processor, cause the processor to perform the steps of the model fine-tuning methods according to various embodiments of this disclosure as described in the foregoing portion of this specification.

[0175] The computer program product can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this disclosure. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on a user's computing device, partially on a user's computing device, as a standalone software package, partially on a user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0176] Furthermore, embodiments of this disclosure may also be computer-readable storage media storing computer program instructions that, when executed by a processor, cause the processor to perform the steps of the model fine-tuning methods according to various embodiments of this disclosure described in the foregoing portion of this specification.

[0177] The computer-readable storage medium may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof.

[0178] The basic principles of this disclosure have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this disclosure are merely examples and not limitations, and should not be considered as essential features of each embodiment of this disclosure. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the scope of this disclosure to the necessity of employing the aforementioned specific details for implementation.

[0179] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For system embodiments, since they largely correspond to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0180] The block diagrams of devices, apparatuses, devices, and systems disclosed herein are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or,” and are used interchangeably with them unless the context clearly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.

[0181] The methods and apparatus of this disclosure may be implemented in many ways. For example, they may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above-described order of steps for the methods is for illustrative purposes only, and the steps of the methods of this disclosure are not limited to the order specifically described above, unless otherwise specifically stated. Furthermore, in some embodiments, this disclosure may also be implemented as a program recorded on a recording medium, the program including machine-readable instructions for implementing the methods according to this disclosure. Thus, this disclosure also covers recording media storing programs for performing the methods according to this disclosure.

[0182] It should also be noted that in the apparatus, devices, and methods of this disclosure, the components or steps can be disassembled and / or recombined. These disassemblies and / or recombinations should be considered as equivalent solutions to this disclosure.

[0183] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of this disclosure. Therefore, this disclosure is not intended to be limited to the aspects shown herein, but rather to be carried out within the widest scope consistent with the principles and novel features disclosed herein.

[0184] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this disclosure to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations therein.

Claims

1. A method for fine-tuning a model, characterized in that, The method includes: Obtain a set of basic prompt content, which includes a first prompt content, a second prompt content, and a third prompt content. The first prompt content includes video description content generated by the model to be fine-tuned that has defect features in the target weakness dimension. The second prompt content includes video description content with preset target features in the target weakness dimension. The third prompt content includes video description content from preset source real videos. Based on the first and second prompts, a multi-type target prompt dataset is generated. The target prompt dataset includes prompt description data that controls the fine-tuning training of the model to be fine-tuned for the target weakness dimension. A first target training dataset is generated based on the target prompt dataset and the third prompt content. The first target training dataset includes a first synthetic video set and a second synthetic video set. The first synthetic video set includes synthetic video data that corresponds one-to-one with the prompt description data in the target prompt dataset and is used to optimize the target weakness dimension of the model to be fine-tuned. The second synthetic video set includes synthetic video data generated based on the third prompt content. The synthetic video data in the first target training dataset is evaluated to obtain the evaluation results; Based on the evaluation results and the first target training dataset, the network parameters of the model to be fine-tuned are optimized and updated to obtain the target model, which is the model to be fine-tuned after the network parameters have been optimized and updated.

2. The method according to claim 1, characterized in that, The generation of multiple types of target prompt datasets based on the first and second prompt contents, wherein the target prompt datasets include prompt description data that controls the fine-tuning training of the model to be fine-tuned for the target weakness dimension, including: Based on the first prompt, determine the key attributes for the target's weakness dimension; Based on the second prompt content and the preset large language model, generate meta-prompt text for all the key attributes; The meta-hint text is expanded using the preset large language model to generate multiple types of hint description data corresponding to the meta-hint text, thus obtaining the target hint dataset.

3. The method according to claim 2, characterized in that, The first target training dataset is generated based on the target cue dataset and the third cue content. The first target training dataset includes a first synthetic video set and a second synthetic video set. The first synthetic video set includes synthetic video data that corresponds one-to-one with the cue description data in the target cue dataset and is used to optimize the target weakness dimension of the model to be fine-tuned. The second synthetic video set includes synthetic video data generated based on the third cue content, including: The target cue dataset is used as the first input data of the model to be fine-tuned, so that the model to be fine-tuned generates synthetic video data corresponding to each type of cue description data in the target cue dataset, which is used to optimize the training of the target weakness dimension of the model to be fine-tuned, and thus the first synthetic video set is obtained. The third prompt content is used as the second input data of the model to be fine-tuned, so that the model to be fine-tuned generates synthetic video data corresponding to the preset source real video according to the third prompt content, and the second synthetic video set is obtained. The first synthetic video set and the second synthetic video set are determined as the first target training dataset.

4. The method according to claim 3, characterized in that, The evaluation of the synthetic video data in the first target training dataset to obtain the evaluation result includes: For any synthetic video data, based on the prompt description data corresponding to the synthetic video data and the target weakness dimension corresponding to the synthetic video data, a dimension evaluation question-and-answer pair corresponding to the target weakness dimension is generated based on a preset video evaluation standard; The synthesized video data is evaluated by combining the aforementioned dimensional evaluation questions and answers to obtain the evaluation result.

5. The method according to claim 4, characterized in that, The evaluation of the synthesized video data by combining the aforementioned dimensions with the question-and-answer pair yields the evaluation result, including: For any synthetic video data, the content of the synthetic video data is compared with its corresponding dimension evaluation question-and-answer pair. In response to determining that the content of the synthetic video data is consistent with the content described by its corresponding dimension evaluation question-and-answer pair, an evaluation score is output, and the evaluation score is the evaluation result.

6. The method according to claim 5, characterized in that, The process involves optimizing and updating the network parameters of the model to be fine-tuned based on the evaluation results and the first target training dataset to obtain the target model. The target model is the model to be fine-tuned after the network parameter optimization and update, including: The synthetic video data corresponding to the first target training dataset is weighted according to the evaluation score to obtain the second target training dataset. The model to be fine-tuned is trained using the second target training dataset to optimize and update the network parameters of the model to be fine-tuned, thereby obtaining the target model, which is the model to be fine-tuned after the network parameter optimization and update.

7. The method according to claim 5, characterized in that, The process involves optimizing and updating the network parameters of the model to be fine-tuned based on the evaluation results and the first target training dataset to obtain the target model. The target model is the model to be fine-tuned after the network parameter optimization and update, including: The synthetic video data corresponding to the first target training dataset is weighted according to the evaluation score to obtain the second target training dataset. The model to be fine-tuned is trained using the second target training dataset, the second synthetic video set, and the preset source real videos corresponding to the second synthetic video set. The network parameters of the model to be fine-tuned are optimized and updated using the second target training dataset. The model to be fine-tuned is constrained and optimized based on the consistency evaluation results of the second synthetic video set and the preset source real videos corresponding to the second synthetic video set, resulting in a target model, which is the model to be fine-tuned after the network parameter optimization and update.

8. A device for fine-tuning a model, characterized in that, The device includes: The basic prompt content set module is used to obtain a basic prompt content set, which includes a first prompt content, a second prompt content, and a third prompt content. The first prompt content includes video description content generated by the model to be fine-tuned that has defect features in the target weakness dimension. The second prompt content includes video description content with preset target features in the target weakness dimension. The third prompt content includes video description content from preset source real videos. The first data generation module is used to generate multiple types of target prompt datasets based on the first prompt content and the second prompt content. The target prompt datasets include prompt description data that controls the fine-tuning training of the model to be fine-tuned for the target weakness dimension. The second data generation module is used to generate a first target training dataset based on the target prompt dataset and the third prompt content. The first target training dataset includes a first synthetic video set and a second synthetic video set. The first synthetic video set includes synthetic video data that corresponds one-to-one with the prompt description data in the target prompt dataset and is used to optimize the target weakness dimension of the model to be fine-tuned. The second synthetic video set includes synthetic video data generated based on the third prompt content. The video data evaluation module is used to evaluate the synthetic video data in the first target training dataset and obtain the evaluation results; The model fine-tuning and optimization module is used to optimize and update the network parameters of the model to be fine-tuned based on the evaluation results and the first target training dataset to obtain the target model, which is the model to be fine-tuned after the network parameter optimization and update.

9. An electronic device, characterized in that, include: Memory, used to store computer program products; A processor for executing a computer program product stored in the memory, wherein when the computer program product is executed, it implements the method described in any one of claims 1-7.

10. A computer-readable storage medium having computer program instructions stored thereon, characterized in that, When the computer program instructions are executed by the processor, they implement the method described in any one of claims 1-7.

11. A computer program product comprising computer program instructions, characterized in that, When the computer program instructions are executed by the processor, they implement the method described in any one of claims 1-7.

Images