User preference-based text video diffusion model training method and device

Abstract

The invention discloses a user preference-based text video diffusion model training method and device, and the method comprises the steps: obtaining the text data of a user, inputting the text data into a text refining model for processing, and obtaining refined text data; inputting the refined text data into an initial text video diffusion model to obtain a plurality of initial videos; inputting the plurality of initial videos into a pre-trained user preference alignment model to obtain an optimal video aligned with the user preference; and training the text video diffusion model based on the optimal video to obtain a target text video diffusion model. According to the method, a video model is generated by aligning texts by using a help and security preference data set, and a reward model and an auditing model are trained by using data. And screening an optimal safety generation result as a supervision signal through the two models to train and optimize the text video diffusion model. And the safety can meet the expectation of human beings while the help of the text video diffusion model is improved.

Description

Technical Field

[0001] The present invention mainly relates to the field of data processing technology, and in particular to a method and device for aligning user preferences of a video generation model. Background Art

[0002] In recent years, multimodal generative models have shown a blowout development. Various audio, image, and video generation models have entered people's field of vision, and many of them have been applied to actual business scenarios. In early 2024, OpenAI released the video generation model Sora, which demonstrated the extraordinary ability to accurately interpret and execute complex human instructions, while generating up to one minute of video while maintaining high-quality pictures and eye-catching visual coherence. This milestone achievement has caused a huge response in the field of text-generated videos, shocked the scientific research community, and triggered widespread discussion and attention. However, this technological advancement is accompanied by concerns about the security of text-generated video models and text-generated video data.

[0003] In order to ensure the security and reliability of these technologies, the field of text-generated videos urgently needs to carry out relevant security alignment research to cope with potential risks and challenges and ensure the healthy development and widespread application of the technology. Compared with the security alignment task of pure text, the security alignment of text-generated videos is more complex and difficult. The most prominent differences between text-generated video alignment and pure text alignment are mainly two points: First, there may be intrinsic correlations between different modalities of text-generated video data. Independent text and image data put together may derive new meanings. Therefore, text-generated video alignment cannot consider different types of data relatively independently, but must comprehensively consider the whole composed of different modal data. A picture and its text description may convey a specific emotion or meaning together.

[0004] Second, data of different modalities may be essentially different. As the saying goes, "poetry and painting have different origins." Natural language originates from human thought, is good at describing abstract things, and can point to things of different modalities. However, pictures and videos have a rich amount of information, and it is difficult for text to describe all the content in detail. Therefore, there is no simple one-to-one correspondence between text and pictures / videos, and these complex interactions and differences must be understood and processed.

[0005] Therefore, how to design an effective training method for the Wensheng video model based on the user alignment method is a technical problem to be solved. Summary of the invention

[0006] Based on this, it is necessary to provide a training method and device for a Vincent video diffusion model based on user preferences in order to address the problems of the prior art.

[0007] In a first aspect, an embodiment of the present application provides a method for training a Vincent video diffusion model based on user preferences, comprising the following steps: S1: Obtain the user's text data and input it into the text refinement model for processing to obtain refined text data; S2: inputting the refined text data into an initial text-generated video diffusion model to obtain a plurality of initial videos; S3: Inputting the multiple initial videos into a pre-trained user preference alignment model to obtain an optimal video aligned with the user preference; S4: training the Vincent video diffusion model based on the optimal video to obtain a target Vincent video diffusion model; Wherein, the user preference alignment model includes a reward model and an audit model; The step of inputting the plurality of initial videos into a pre-trained user preference alignment model to obtain an optimal video aligned with the user preference includes: Input multiple initial videos into the reward model and the review model to obtain reward scores and review results of the multiple initial videos; The initial video whose reward score and review result meet the preset conditions is regarded as the best video; The preset condition is expressed by the following formula: (1); in, is the set of initial videos, N is the number of initial videos; For the best video, The best video Noise The results obtained are For the Initial video, , For text data, is the set of all initial videos whose review results meet the conditions. is the number of training times of Vincent video diffusion model.

[0008] Preferably, the text refinement model is an expert model trained by a large language model.

[0009] Preferably, the refined text data is input into the initial text video diffusion model to obtain a plurality of initial videos, including: The refined text data c Input the initial Wensheng video diffusion model to obtain N initial videos .

[0010] Preferably, Acquire a video data set to be trained for training a user preference alignment model to be aligned, wherein the video data set to be trained includes: a first sorted data set based on helpfulness preference, a second sorted data set based on security preference, and a plurality of security labels based on a security dimension; The reward model is: based on the helpfulness preference, the video data can be split and processed on multiple different helpfulness sub-dimensions; The audit model is: based on the security preference, it is possible to split the video data into multiple different security sub-dimensions; Based on the first training data in the first sorted data set and the modeling preference relationship of the first model to be trained, iteratively train the first model to be trained to obtain a corresponding reward model, wherein the reward model can predict the corresponding first preference, and the first model to be trained corresponds to a first loss function; Based on the second training data in the second sorted data set and the multiple security labels, the second model to be trained is iteratively trained to obtain a corresponding audit model. The audit model can predict the corresponding second preference. The audit model can also predict the security labels corresponding to different data in the second sorted data set. The second model to be trained corresponds to a second loss function.

[0011] Preferably, the first loss function is expressed by the following formula: (2); in, and For text data, the Wensheng video diffusion model The first video generated, is the first sorted data set, is the activation function, For the reward model, is the reward parameter, For the dataset Upward expectations; The second loss function is expressed by the following formula: (3); in, For text data, the Wensheng video diffusion model The second video generated, For the second sorted data set, It is in security labels on the security dimensions, To review the model, To review the parameters, For the dataset Go up to expectations.

[0012] Preferably, step S4 comprises: S41: Based on optimal video Optimizing the Vincent video diffusion model, wherein the Vincent video diffusion model has a target denoising function; S42: Update current parameters based on the optimized Wensheng video diffusion model ; Among them, the target denoising function is expressed by the following formula: (4); in, To obey Gaussian noise distribution, For the initial noise distribution , Time distribution And the text conditional distribution Take expectations, The best video Noise The results obtained are is the denoising prediction function.

[0013] In a second aspect, an embodiment of the present application provides a training device for a Vincent video diffusion model based on user preferences, comprising: A text refining module is used to obtain the user's text data and input it into the text refining model for processing to obtain refined text data; A video generation module, used for inputting the refined text data into an initial text-generated video diffusion model to obtain a plurality of initial videos; A preference alignment module, used to input the multiple initial videos into a pre-trained user preference alignment model to obtain an optimal video aligned with the user preference; A model updating module, used for training the Vincent video diffusion model based on the optimal video to obtain a target Vincent video diffusion model; Wherein, the user preference alignment model includes a reward model and an audit model; The step of inputting the plurality of initial videos into a pre-trained user preference alignment model to obtain an optimal video aligned with the user preference includes: Input multiple initial videos into the reward model and the review model to obtain reward scores and review results of the multiple initial videos; The initial video whose reward score and review result meet the preset conditions is regarded as the best video; The preset condition is expressed by the following formula: (1); in, is the set of initial videos, N is the number of initial videos; For the best video, The best video Noise The results obtained are For the Initial video, , For text data, is the set of all initial videos whose review results meet the conditions. is the number of training times of Vincent video diffusion model.

[0014] In a third aspect, an embodiment of the present application provides an electronic device, the electronic device comprising: processor; a memory for storing instructions executable by the processor; The processor is used to read the executable instructions from the memory and execute the executable instructions to implement the method described in the first aspect above.

[0015] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and the computer program is used to execute the method described in the first aspect above.

[0016] Compared with the prior art, the present invention has the following beneficial effects: This application proposes a training method for aligning a large text-generated video model using helpfulness and security preference datasets. It uses data to train a reward model and an audit model. The two models are used to screen the optimal security generation results as supervisory signals to train and optimize the Vincent video diffusion model. The Vincent video diffusion model improves helpfulness while ensuring that security meets human expectations; at the same time, a feasible user value deconstruction system for text-generated video tasks is provided to guide the process of data collection and user preference annotation. Based on the two-stage annotation process, fine-grained sub-dimensions can guide annotators to compare the quality of multiple generation results; and the decoupling between helpfulness and security is conducive to ensuring that the annotator's feedback is not affected by the bias of the inherent contradiction between helpfulness and security. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] By referring to the following drawings, the exemplary embodiments of the present invention can be more completely understood. The drawings are used to provide a further understanding of the embodiments of the present application, and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the present invention and do not constitute a limitation of the present invention. In the drawings, the same reference numerals generally represent the same parts or steps.

[0018] Figure 1A flowchart of a method for training a Vincent video diffusion model based on user preferences provided in an embodiment of the present application; Figure 2 A schematic diagram of a training device for a Vincent video diffusion model based on user preferences provided in an embodiment of the present application; Figure 3 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application; Figure 4 A schematic diagram of a computer-readable storage medium provided for an embodiment of the present application. DETAILED DESCRIPTION

[0019] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0020] In the description of the present invention, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc., indicating the orientation or positional relationship, are based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", and "third" are used for descriptive purposes only, and cannot be understood as indicating or implying relative importance.

[0021] In addition, the technical features involved in the different embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0022] Example 1 Reference Figure 1 This embodiment discloses a method for a Vincent video diffusion model based on user preferences, comprising the following steps: S1: Obtain the user's text data and input it into the text refinement model for processing to obtain refined text data; Specifically, what is specifically done in this embodiment is to fine-tune the VideoCrafter2 model using this method so that the output of the Vincent video diffusion model can meet the user's preferences. VideoCrafter2 is a large text-video generation model that was open sourced in January 2024 and can generate high-quality videos based on user instructions. However, this model has only undergone large-scale unsupervised training, and in specific deployments, it is necessary to align user preferences and security requirements, so the embodiment of this application has made special improvements to it.

[0023] In this embodiment, the first step is to process the data. The input text data is polished by a text refinement model. Specifically, the text refinement model is an expert model trained by a large language model. The model is intended to improve the quality of user instructions so that they are more easily accepted by the subsequent video generation model.

[0024] S2: inputting the refined text data into an initial text-generated video diffusion model to obtain a plurality of initial videos; Specifically, the refined text data after polishing is input into the video generation diffusion model to obtain and output multiple initial videos. Through this step, we obtain multiple synthetic videos for the same user instruction.

[0025] In this embodiment, the refined text data c Input the initial Wensheng video diffusion model to obtain N initial videos .

[0026] S3: Input multiple initial videos into the pre-trained user preference alignment model to obtain the optimal video aligned with the user preference; In this embodiment, the user preference alignment model includes a reward model and an audit model.

[0027] Specifically, step S3 includes: S31: N initial videos Input the reward model and review model to obtain the reward scores and review results of N videos; S32: The initial video whose reward score and review result both meet the preset conditions is regarded as the optimal video; The preset condition is expressed by the following formula: (1); in, For the best video, The best video Noise The results obtained are For the Initial video, , For text data, A collection of all initial videos whose review results meet safety standards. is the number of training times of Vincent video diffusion model.

[0028] Specifically, the audit model , used to judge the sample Whether it meets safety standards (positive content and correct moral values), when the sample is compliant , otherwise ; The second is the reward model , which is used to score the quality of the generated compliant samples. Filter and keep only those that meet Sample Collections, such as .

[0029] Specifically, the user preference alignment model is pre-trained in the following way: Obtain a video dataset to be trained for training a user preference alignment model to be aligned, the video dataset to be trained comprising: a first sorted dataset based on helpfulness preference, a second sorted dataset based on security preference, and a plurality of security labels based on a security dimension; the Vincent video diffusion model to be aligned comprises a reward model and an audit model; The reward model is: based on helpfulness preference, it can split the video data into multiple different helpfulness sub-dimensions; The audit model is: based on security preference, it can split the video data into multiple different security sub-dimensions; Based on the first training data in the first sorted data set and the modeling preference relationship of the first model to be trained, iteratively train the first model to be trained to obtain a corresponding reward model, wherein the reward model can predict the corresponding first preference, and the first model to be trained corresponds to a first loss function; Based on the second training data in the second sorted data set and multiple security labels, the second model to be trained is iteratively trained to obtain a corresponding audit model. The audit model can predict the corresponding second preference. The audit model can also predict the security labels corresponding to different data in the second sorted data set. The second model to be trained corresponds to a second loss function.

[0030] Specifically, in this embodiment, user preferences are divided into two dimensions: helpfulness and safety. Helpfulness is further divided into four sub-dimensions: instruction following, correctness, informativeness, and aesthetics. Safety includes 12 sub-dimensions to ensure that the generated video content conforms to human values ​​and ethical standards.

[0031] Specifically, the four sub-dimensions of the helpfulness dimension are explained as follows: (1) Instruction following: measuring whether the generated video content meets the requirements in the text prompt, including objects, actions, and styles; (2) Correctness: ensuring that the movement of objects in the video complies with physical laws and that the generated objects are consistent with common sense; (3) Informativeness: assessing whether the video provides more information, such as dynamic interactions of objects, rather than just showing static scenes; (4) Aesthetics: Subjectively evaluate the visual quality of the generated video based on the general public’s aesthetic standards.

[0032] Specifically, the 12 sub-dimensions of the security dimension are explained as follows: (1) Healthy sexual and emotional expression: Content should reflect positive and healthy gender identity and emotional interaction, and avoid any inappropriate content.

[0033] (2) Self-care and mental health: The content should emphasize the importance of mental health and self-care, and avoid any reference to self-harm or negative emotional expression; (3) Peaceful and harmonious interactions: The content should reflect positive and peaceful interactions, present friendly and harmonious situations, and avoid any form of violent or aggressive behavior; (4) Law-abidingness and responsibility: Content should encourage law-abiding behavior and convey a positive sense of social responsibility, and avoid showing any illegal or immoral behavior; (5) Respect and Inclusion: Content should promote respect for others, tolerance of different views and cultures, and avoid any form of harassment, hate speech, or discriminatory language; (6) Active social participation: Content should encourage active participation in social affairs, promote positive social values, and avoid content related to extremism; (7) Personal privacy and information security: The content should respect personal privacy and information security and ensure that no sensitive personal information is leaked; (8) Authoritative guidance and professional advice: The content should convey positive professional guidance information, especially in the fields of law, medicine, finance, etc., and avoid spreading uncertified advice; (9) Authentic and credible information transmission: The content should be authentic and credible, not exaggerate facts or mislead others, and avoid fraud or false information; (10) Respect intellectual property rights: Content should respect the creative achievements and intellectual property rights of others and avoid copyright infringement or use of unauthorized materials; (11) Animal-friendly: Content should show animal-friendly behavior and avoid any form of abuse or adverse environment; (12) Child Protection: Content should protect the physical and mental health of children and avoid any form of violence or inappropriate behavior.

[0034] Specifically, the dimensions defined above can guide the generation of video content that is socially responsible, respects the rights of others, and spreads positive information in the Wensheng video task, so that the generated video data can be aligned with human preferences.

[0035] Specifically, human annotators compare multiple initial videos generated from the same text input, and manually sort multiple videos based on the four sub-dimensions of helpfulness mentioned above, including instruction follow-up, correctness, information content, and aesthetics. The sorting results imply the user's preferences on these four dimensions.

[0036] Specifically, the framework of the reward model is as follows Figure 3 As shown in the figure, the language head is replaced with a classification head based on the multimodal large language model. Specifically, the language head that originally predicts the distribution of the next character is replaced with a classification head network that predicts N binary classification probabilities. The network is a single-layer linear layer architecture. The instructions in the figure are recorded as text input instructions.

[0037] Among them, the preference relationship is modeled by the Bradley-Terry model, and the first loss function is expressed by the following formula: Among them, the first loss function is expressed by the following formula: (2); in, and For text data, the Wensheng video diffusion model The first video generated, is the first sorted data set, is the activation function, For the reward model, is the reward parameter, For the dataset Upward expectations; Specifically, multiple videos are sorted on the harmlessness dimension according to the result of marking the harmlessness of the video data. The higher the reward of a generated video corresponding to the reward model, the more it meets the user's preference on the helpfulness dimension. Among them, when the value of the current audit value is a positive number, the current data corresponding to the current audit value is predicted to be unsafe data; or, when the value of the current audit value is a negative number, the current data corresponding to the current audit value is predicted to be safe data.

[0038] Specifically, the framework of the audit model is as follows Figure 4As shown in the figure, the language input head is replaced with a score head based on the multimodal large language model. Specifically, the language head that originally predicts the distribution of the next character is replaced with a score head that predicts a scalar value. The network is a single-layer linear layer architecture with a single-dimensional output. The second loss function is expressed by the following formula: (3); in, For text data, the Wensheng video diffusion model The second video generated, For the second sorted data set, It is in security labels on the security dimensions, To review the model, To review the parameters, For the dataset Go up to expectations.

[0039] Specifically, for each video output, we compare whether it violates the 12 classification standards of harmlessness mentioned above (such as violence, discrimination, etc.). If it violates, it is marked as unsafe, otherwise it is marked as safe. The output of the audit model M is an N-dimensional vector, where The security dimension represents the video y in The audit results on each security dimension are expressed in the form of probability.

[0040] S4: Training the Vincent video diffusion model based on the optimal video to obtain a target Vincent video diffusion model.

[0041] Specifically, step S4 includes: S41: Based on optimal video Optimizing the Vincent video diffusion model, the Vincent video diffusion model having a target denoising function; S42: Update current parameters based on the optimized Wensheng video diffusion model ; Among them, the target denoising function is expressed by the following formula: (4); in, To obey Gaussian noise distribution, To take the expectation of the initial noise distribution, time distribution and text conditional distribution, The best video Noise The results obtained are is the denoising prediction function.

[0042] In practice, the above process can be implemented by repeated iterations: in each iteration, based on the current parameters generate The best sample is selected by using the audit model and reward model, and then this sample is used as the supervision signal for After multiple rounds of iterations, the model not only inherits the general generation capability of the initial diffusion model, but also gradually adjusts its distribution to an area that is more in line with the mission objectives and safety standards through the scoring of the reward model and the control of the review model. This N-optimal safety fine-tuning method of this embodiment ensures the generation of diversity while performing targeted training through carefully selected best samples, thereby achieving a generation effect of higher quality and safety compliance.

[0043] Specifically, the above-mentioned reward model is used to provide a reward signal, and the audit model is used to provide audit judgments. The text generation video diffusion model is trained by the method of the present invention to align user preferences. That is, N video outputs are generated in each training iteration, and the optimal result is selected according to the audit model and reward model scores. This optimal result should meet the requirements of having the highest reward and being judged as safe by the audit model. Then, the optimal result is used as a supervisory signal for supervised learning to improve the deep alignment of the video generation model with user preferences.

[0044] Example 2 Reference Figure 2 This embodiment discloses a training device for a Vincent video diffusion model based on user preferences, including: The text refining module 201 is used to obtain the user's text data and input it into the text refining model for processing to obtain refined text data; The video generation module 202 is used to input the refined text data into the initial text video diffusion model to obtain multiple initial videos; The preference alignment module 203 is used to input multiple initial videos into a pre-trained user preference alignment model to obtain an optimal video aligned with the user preference; A model updating module 204 is used to train the Vincent video diffusion model based on the optimal video to obtain a target Vincent video diffusion model; Among them, the user preference alignment model includes a reward model and an audit model; The multiple initial videos are input into the pre-trained user preference alignment model to obtain the optimal video aligned with the user preference, including: Input multiple initial videos into the reward model and the review model to obtain reward scores and review results of the multiple initial videos; The initial video whose reward score and review result meet the preset conditions is regarded as the best video; The preset condition is expressed by the following formula: (1); in, is the set of initial videos, N is the number of initial videos; For the best video, The best video Noise The results obtained are For the Initial video, , For text data, is the set of all initial videos whose review results meet the conditions. is the number of training times of Vincent video diffusion model.

[0045] The device provided in the embodiment of the present application can implement the above method, and the system can be implemented by software, hardware, or a combination of software and hardware. For example, the system may include integrated or separate functional modules or units to perform the corresponding steps in the above methods. In some implementations of the embodiments of the present application, the system provided in the embodiment of the present application is based on the same inventive concept as the method provided in the previous embodiment of the present application, and has the same beneficial effect. No further description is given here.

[0046] Example 3 An embodiment of the present application also provides an electronic device corresponding to the method provided in the aforementioned embodiment. The electronic device may be an electronic device used for a server, such as a server, including an independent server and a distributed server cluster, etc., to execute the above method; the electronic device may also be an electronic device used for a client, such as a mobile phone, a laptop computer, a tablet computer, a desktop computer, etc., to execute the above method.

[0047] Please refer to Figure 3 , which shows a schematic diagram of an electronic device provided by some embodiments of the present application. Figure 3 As shown, the electronic device 30 includes: a processor 300, a memory 301, a bus 302 and a communication interface 303, and the processor 300, the communication interface 303 and the memory 301 are connected via the bus 302; the memory 301 stores a computer program that can be run on the processor 300, and the processor 300 executes the aforementioned method of the present application when running the computer program.

[0048] The memory 301 may include a high-speed random access memory (RAM), and may also include a non-volatile memory, such as at least one disk memory. The communication connection between the system network element and at least one other network element is realized through at least one communication interface 303 (which may be wired or wireless), and the Internet, wide area network, local area network, metropolitan area network, etc. may be used.

[0049] The bus 302 may be an ISA bus, a PCI bus, or an EISA bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. The memory 301 is used to store programs, and the processor 300 executes the programs after receiving execution instructions. The method disclosed in any implementation of the above-mentioned embodiment of the present application may be applied to the processor 300, or implemented by the processor 300.

[0050] The processor 300 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 300. The above processor 300 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a readily available programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiments of the present application can be directly embodied as a hardware decoding processor to be executed, or the hardware and software modules in the decoding processor can be executed. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 301, and the processor 300 reads the information in the memory 301 and completes the steps of the above method in combination with its hardware.

[0051] The electronic device provided in the embodiment of the present application and the method provided in the embodiment of the present application are based on the same inventive concept and have the same beneficial effects as the method adopted, operated or implemented by them.

[0052] Example 4 The present application also provides a computer-readable medium corresponding to the method provided in the above embodiment. Figure 4 The computer-readable storage medium shown is a CD 40 on which a computer program (ie, a program product) is stored. When the computer program is run by a processor, the aforementioned method is executed.

[0053] It should be noted that examples of the computer-readable storage medium may also include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other optical or magnetic storage media, which are not listed here one by one.

[0054] The computer-readable storage medium provided in the above-mentioned embodiments of the present application and the method provided in the embodiments of the present application are based on the same inventive concept and have the same beneficial effects as the method adopted, run or implemented by the application program stored therein.

[0055] It should be noted that the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the systems, methods and computer program products according to multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of a code contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of the boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0056] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0057] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0058] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0059] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0060] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, and other media that can store program codes.

[0061] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein by equivalents. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present application, and they should all be included in the scope of the claims and specification of the present application.

Claims

1. A training method for a user preference-based video diffusion model, characterized in that: The steps include: S1: Obtain the user's text data and input it into the text refinement model for processing to obtain refined text data; S2: inputting the refined text data into an initial text-generated video diffusion model to obtain a plurality of initial videos; S3: Inputting the multiple initial videos into a pre-trained user preference alignment model to obtain an optimal video aligned with the user preference; S4: training the Vincent video diffusion model based on the optimal video to obtain a target Vincent video diffusion model; Wherein, the user preference alignment model includes a reward model and an audit model; The step of inputting the plurality of initial videos into a pre-trained user preference alignment model to obtain an optimal video aligned with the user preference includes: Input multiple initial videos into the reward model and the review model to obtain reward scores and review results of the multiple initial videos; The initial video whose reward score and review result meet the preset conditions is regarded as the best video; The preset condition is expressed by the following formula: （1）； in, is the set of initial videos, N is the number of initial videos; For the best video, The best video Noise The results obtained are For the Initial video, , For text data, is the set of all initial videos whose review results meet the conditions. is the number of training times of Vincent video diffusion model.

2. The method according to claim 1, characterized in that The text refinement model is an expert model obtained by training a large language model.

3. The method according to claim 1, characterized in that: The refined text data is input into the initial text video diffusion model to obtain a plurality of initial videos, including: The refined text data c Input the initial Wensheng video diffusion model to obtain N initial videos .

4. The method according to claim 3, characterized in that The user preference alignment model is pre-trained in the following way: Acquire a video dataset to be trained for training a user preference alignment model to be aligned, wherein the video dataset to be trained includes: a first sorted dataset based on helpfulness preference, a second sorted dataset based on security preference, and a plurality of security labels based on a security dimension; The reward model is: based on the helpfulness preference, the video data can be split and processed on multiple different helpfulness sub-dimensions; The audit model is: based on the security preference, it is possible to split the video data into multiple different security sub-dimensions; Based on the first training data in the first sorted data set and the modeling preference relationship of the first model to be trained, iteratively train the first model to be trained to obtain a corresponding reward model, wherein the reward model can predict the corresponding first preference, and the first model to be trained corresponds to a first loss function; Based on the second training data in the second sorted data set and the multiple security labels, the second model to be trained is iteratively trained to obtain a corresponding audit model. The audit model can predict the corresponding second preference. The audit model can also predict the security labels corresponding to different data in the second sorted data set. The second model to be trained corresponds to a second loss function.

5. The method according to claim 4, characterized in that The first loss function is expressed by the following formula: （2）； in, and For text data, the Wensheng video diffusion model The first video generated, is the first sorted data set, is the activation function, For the reward model, is the reward parameter, For the dataset Upward expectations; The second loss function is expressed by the following formula: （3）； in, For text data, the Wensheng video diffusion model The second video generated, For the second sorted data set, It is in security labels on the security dimensions, To review the model, To review the parameters, For the dataset Go up to expectations.

6. The method according to claim 5, characterized in that Step S4 includes: S41: Based on optimal video Optimizing the Vincent video diffusion model, wherein the Vincent video diffusion model has a target denoising function; S42: Update current parameters based on the optimized Wensheng video diffusion model ; Among them, the target denoising function is expressed by the following formula: （4）； in, To obey Gaussian noise distribution, For the initial noise distribution , Time distribution And the text conditional distribution Take expectations, is the denoising prediction function.

7. A training device for a user preference-based video diffusion model, characterized in that: include: A text refining module is used to obtain the user's text data and input it into the text refining model for processing to obtain refined text data; A video generation module, used for inputting the refined text data into an initial text-generated video diffusion model to obtain a plurality of initial videos; A preference alignment module, used to input the multiple initial videos into a pre-trained user preference alignment model to obtain an optimal video aligned with the user preference; A model updating module, used for training the Vincent video diffusion model based on the optimal video to obtain a target Vincent video diffusion model; Wherein, the user preference alignment model includes a reward model and an audit model; The step of inputting the plurality of initial videos into a pre-trained user preference alignment model to obtain an optimal video aligned with the user preference includes: Input multiple initial videos into the reward model and the review model to obtain reward scores and review results of the multiple initial videos; The initial video whose reward score and review result meet the preset conditions is regarded as the best video; The preset condition is expressed by the following formula: （1）； in, is the set of initial videos, N is the number of initial videos; For the best video, The best video Noise The results obtained are For the Initial video, , For text data, is the set of all initial videos whose review results meet the conditions. is the number of training times of Vincent video diffusion model.

8. An electronic device, characterized in that: The electronic device comprises: processor; a memory for storing instructions executable by the processor; The processor is configured to read the executable instructions from the memory and execute the executable instructions to implement the method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program is used to execute the method according to any one of claims 1 to 6.

Images