Content generation method and related device

Abstract

Embodiments of the present application disclose a content generation method and related devices, which can include a content generation apparatus, an electronic device, a computer program product and a computer readable storage medium; after a set of reward information for a preset content generation model is obtained, the set of reward information including reward information corresponding to a plurality of task objectives, a combination weight information of the task objectives is determined based on a preset weight parameter, then the reward information is weighted according to the combination weight information to obtain target reward information corresponding to the task objectives, the preset content generation model is trained based on the target reward information to obtain an initial content generation model corresponding to the task objectives, then the initial content generation model is fused according to the combination weight information and preference information for the plurality of task objectives, and a target content satisfying the preference information is generated by using the fused model; the scheme can improve the accuracy of content generation.

Description

Technical Field

[0001] This invention relates to the field of content generation technology, specifically to a content generation method and related equipment, which may include a content generation apparatus, electronic equipment, computer program products, and computer-readable storage media. Background Technology

[0002] In recent years, with the rapid development of neural network technology, content generation has become increasingly convenient and accurate, especially in the field of multi-objective controllable content generation. Multi-objective controllable content generation tasks require considering multiple conflicting objectives simultaneously. By adjusting the weights of different objectives, dynamic control and precise optimization of the generated results can be achieved. Current multi-objective content generation methods often involve optimizing single-objective models to obtain a series of single-objective models, and then fusing these models during testing using linear interpolation. The fused model is then used to generate content that satisfies multiple objectives.

[0003] In the process of researching and practicing current technologies, the inventors of this application have found that in the process of model tuning for a single objective, the reward function corresponding to the single objective is usually only optimized for a specific objective and cannot fully explore the entire reward space. This limits the performance of the model on other objectives and makes it impossible to achieve the optimal solution on the Pareto Front. In addition, in the process of model fusion, the preference information is directly used as the fusion weight, ignoring the deep relationship between the internal parameters of the model and the target space. Therefore, the accuracy of content generation is low. Summary of the Invention

[0004] This application provides a content generation method and related equipment. The related equipment may include a content generation device, an electronic device, a computer program product, and a computer-readable storage medium, which can improve the accuracy of content generation.

[0005] A content generation method, comprising:

[0006] Obtain a set of reward information for generating a model based on preset content, wherein the set of reward information includes reward information corresponding to multiple task objectives;

[0007] Based on preset weight parameters, the combined weight information of the task objectives is determined, and the combined weight information indicates the importance of the task objectives among the multiple task objectives;

[0008] The reward information is weighted according to the combined weight information to obtain the target reward information corresponding to the task objective;

[0009] The preset content generation model is trained based on the target reward information to obtain the initial content generation model corresponding to the task target;

[0010] Based on the combined weight information and the preference information for the multiple task objectives, the initial content generation model is fused, and the fused model is used to generate target content that satisfies the preference information.

[0011] Accordingly, embodiments of this application provide a content generation apparatus, including:

[0012] The acquisition unit is used to acquire a set of reward information for generating a model based on preset content, wherein the set of reward information includes reward information corresponding to multiple task objectives;

[0013] The determining unit is configured to determine the combined weight information of the task objectives based on preset weight parameters, wherein the combined weight information indicates the importance of the task objectives among the multiple task objectives;

[0014] A weighting unit is used to weight the reward information according to the combined weight information to obtain the target reward information corresponding to the task objective;

[0015] The training unit is used to train the preset content generation model based on the target reward information to obtain the initial content generation model corresponding to the task target.

[0016] The generation unit is used to fuse the initial content generation model according to the combined weight information and the preference information for the multiple task objectives, and use the fused model to generate target content that satisfies the preference information.

[0017] In some embodiments, the determining unit may be specifically used to count the number of task objectives among the plurality of task objectives to obtain the number of objectives; determine the distribution parameters of the combined weight information in the target space according to the number of objectives and preset weight parameters; and generate the combined weight information of the task objectives based on the distribution parameters.

[0018] In some embodiments, the determining unit may be specifically used to construct a uniform distribution corresponding to the plurality of task objectives based on the number of objectives; identify the task order of the task objectives among the plurality of task objectives, and construct a deterministic distribution corresponding to the task objectives based on the task order; and mix the deterministic distribution with the uniform distribution according to the distribution parameters to obtain the combined weight information of the task objectives.

[0019] In some embodiments, the determining unit may be specifically used to determine, based on the distribution parameters, a first distribution weight of the deterministic distribution and a second distribution weight corresponding to the uniform distribution; weight the deterministic distribution based on the first distribution weight and weight the uniform distribution based on the second distribution weight; and fuse the weighted deterministic distribution and the weighted uniform distribution to obtain the combined weight information of the task objective.

[0020] In some embodiments, the weighting unit may be specifically used to identify the target weight of each reward information in the reward information set in the combined weight information, wherein the sum of the target weights is 1; to weight the reward information based on the target weights respectively, to obtain at least one weighted reward information corresponding to the combined weight information; and to fuse the weighted reward information to obtain the target reward information of the task objective.

[0021] In some embodiments, the generation unit may be specifically used to obtain the target object's preference information for the multiple task objectives; determine the fusion weight of the initial content generation model based on the preference information and the combined weight information; and fuse the initial content generation model according to the fusion weight to obtain the fused model.

[0022] In some embodiments, the generation unit may be specifically used to extract preference features from the preference information; construct weight features corresponding to the multiple task objectives based on the combined weight information; and calculate the linear relationship between the preference features and the weight features to obtain the fusion weight of the initial content generation model.

[0023] In some embodiments, the generation unit may be specifically used to obtain the current model parameters of the initial content generation model, and to weight the current model parameters based on the fusion weight; and to fuse the weighted current model parameters to obtain the fused model.

[0024] In some embodiments, the generation unit may be specifically used to fuse the weighted current model parameters to obtain fused model parameters; adjust the deviation of the fused model parameters based on preset extrapolation parameters to obtain target model parameters, wherein the preset extrapolation parameters are used to eliminate the deviation influence of the initial model parameters of the preset content generation model; and replace the current model parameters with the target model parameters in the initial content generation model to obtain the fused model.

[0025] In some embodiments, the generation unit may be specifically used to obtain the initial model parameters of the preset content generation model; determine the model parameter deviation value corresponding to the fusion model parameters based on the preset extrapolation parameters and the initial model parameters; and fuse the fusion model parameters with the model parameter deviation value to obtain the target model parameters.

[0026] In some embodiments, the generation unit may be specifically used to calculate the difference between the initial model parameters and the fused model parameters to obtain the model parameter difference; and to fuse the preset extrapolation parameters with the model parameter values ​​to obtain the model parameter deviation value corresponding to the fused model parameters.

[0027] In some embodiments, the generation unit may be specifically used to obtain the initial model parameters of the preset content generation model and the current model parameters of the initial content generation model; adjust the current model parameters based on the preset extrapolation parameters and the initial model parameters to obtain the adjusted model parameters; fuse the adjusted model parameters according to the fusion weight to obtain the target model parameters; and replace the current model parameters with the target model parameters in the initial content generation model to obtain the fused model.

[0028] Furthermore, embodiments of this application also provide an electronic device, including a processor and a memory, wherein the memory stores an application program, and the processor is used to run the application program in the memory to execute the content generation method provided in embodiments of this application.

[0029] Furthermore, embodiments of this application also provide a computer-readable storage medium storing a plurality of instructions adapted for loading by a processor to execute steps in any of the content generation methods provided in embodiments of this application.

[0030] Furthermore, embodiments of this application also provide a computer program product, including a computer program or instructions, which, when executed by a processor, implement the steps in the content generation method provided in embodiments of this application.

[0031] This application embodiment obtains a reward information set for a preset content generation model, which includes reward information corresponding to multiple task objectives. Based on preset weight parameters, it determines the combined weight information of the task objectives, indicating the importance of the task objective among multiple task objectives. Then, based on the combined weight information, the reward information is weighted to obtain the target reward information corresponding to the task objective. The preset content generation model is trained based on the target reward information to obtain the initial content generation model corresponding to the task objective. Then, based on the combined weight information and the preference information for multiple task objectives, the initial content generation model is fused, and the fused model is used to generate target content that satisfies the preference information. Since this scheme uses multi-objective reinforcement learning to weight the combined weight information with multiple reward information, thereby training a series of base models (initial content generation models), it ensures that the base models are closer to the Pareto optimal solution in the multi-objective space. In addition, by using the combined weight information and preference information to fuse the model, it achieves fine control of the model output, ensuring that the fused model can balance the trade-offs between different objectives and generate the optimal result. Therefore, it can improve the accuracy of content generation. Attached Figure Description

[0032] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0033] Figure 1 This is a schematic diagram illustrating a scenario of the content generation method provided in an embodiment of this application;

[0034] Figure 2 This is a flowchart illustrating the content generation method provided in an embodiment of this application;

[0035] Figure 3 This is a schematic diagram illustrating the fusion of the initial content generation model based on the fusion weights provided in an embodiment of this application;

[0036] Figure 4 This is a schematic diagram of the current model fusion process provided in the embodiments of this application;

[0037] Figure 5 This is a schematic diagram of the model fusion process in the solution provided in this application embodiment;

[0038] Figure 6 This is a schematic diagram illustrating the effects of the proposed solution and other multi-objective controllable generation methods in different dimensions, as provided in the embodiments of this application.

[0039] Figure 7 This is another flowchart illustrating the content generation method provided in the embodiments of this application;

[0040] Figure 8 This is a schematic diagram of the structure of the content generation apparatus provided in the embodiments of this application;

[0041] Figure 9 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0042] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0043] This application provides a content generation method and related equipment, which may include a content generation apparatus, an electronic device, a computer program product, and a computer-readable storage medium. The content generation apparatus may be integrated into an electronic device, which may be a server or a terminal, etc.

[0044] The server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery network (CDN), and big data and artificial intelligence platforms. The terminal can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, etc., but is not limited to these. The terminal and server can be directly or indirectly connected via wired or wireless communication, which is not limited herein.

[0045] For example, see Figure 1Taking the integration of a content generation device into an electronic device as an example, the electronic device can acquire a set of reward information for a preset content generation model. This set of reward information includes reward information corresponding to multiple task objectives. Based on preset weight parameters, it determines the combined weight information of the task objectives. This combined weight information indicates the importance of the task objective among multiple task objectives. Then, based on the combined weight information, the reward information is weighted to obtain the target reward information corresponding to the task objective. The preset content generation model is trained based on the target reward information to obtain the initial content generation model corresponding to the task objective. Then, based on the combined weight information and the preference information for multiple task objectives, the initial content generation model is fused, and the fused model is used to generate target content that satisfies the preference information, thereby improving the accuracy of content generation.

[0046] It is understood that, in the specific embodiments of this application, data related to object preference information or generated target content are involved. When the following embodiments of this application are applied to specific products or technologies, permission or consent is required, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0047] The following sections provide detailed descriptions of each example. It should be noted that the order in which the embodiments are described is not intended to limit the preferred order of the embodiments.

[0048] This embodiment will be described from the perspective of a content generation device, which can be integrated into an electronic device, such as a server or a terminal. The terminal can include tablet computers, laptops, personal computers (PCs), wearable devices, virtual reality devices, or other smart devices capable of content generation.

[0049] A content generation method, comprising:

[0050] A reward information set for a preset content generation model is obtained, which includes reward information corresponding to multiple task objectives. Based on preset weight parameters, combined weight information of task objectives is determined, which indicates the importance of a task objective among multiple task objectives. The reward information is weighted according to the combined weight information to obtain the target reward information corresponding to the task objective. The preset content generation model is trained based on the target reward information to obtain the initial content generation model corresponding to the task objective. The initial content generation model is fused according to the combined weight information and the preference information for multiple task objectives, and the fused model is used to generate target content that satisfies the preference information.

[0051] like Figure 2As shown, the specific process of this content generation method is as follows:

[0052] 101. Obtain the set of reward information for the model generated based on the preset content.

[0053] The preset content generation model can be understood as a pre-defined model or network used for content generation.

[0054] The reward information set can include reward information corresponding to multiple task objectives. A task objective can be understood as the goal in a multi-objective generation task. For example, taking the task of generating text content as an example, the task objectives could include the accuracy, relevance, completeness, comprehensiveness, conciseness, or other objectives of the generated text content, etc. The reward information can be understood as the reward model or reward function corresponding to the task objective. There are several ways to obtain the reward information set for a preset content generation model, as follows:

[0055] For example, it can directly receive a set of reward information for a preset content generation model uploaded by a terminal or client. Alternatively, it can filter out reward information corresponding to multiple task objectives from the preset reward information set to obtain a set of reward information for the preset content generation model. Or, it can receive a training request for the preset content generation model, which carries the storage address of reward information corresponding to multiple task objectives. Based on the storage address, it can obtain the reward information corresponding to multiple task objectives and use this reward information as the set of reward information for the preset content generation model, and so on.

[0056] 102. Determine the combined weight information of the task objectives based on preset weight parameters.

[0057] The combined weight information indicates the importance of the task objective among multiple task objectives. In multi-objective reinforcement learning, the combined weight information can be understood as a combined weight vector or basis vector.

[0058] The preset weight parameters can be understood as pre-defined parameters used to calculate the combined weight vector or basis vector. These preset weight parameters can also be understood as a dominance value β, used to ensure significant differences between the initial content generation models (base models) trained for different task objectives, avoiding excessive overlap in model capabilities, thereby improving linear mode connectivity (LMC) and enhancing the controllability and fusion effect of the initial content generation models during the fusion process. Furthermore, it should be noted that the sum of the combined weight vectors corresponding to different task objectives is 1, ensuring a stable weight distribution and enhancing the effectiveness of reward information.

[0059] There are several ways to determine the combined weight information of the task objectives based on preset weight parameters, as follows:

[0060] For example, the number of task objectives can be counted among multiple task objectives to obtain the number of objectives. Based on the number of objectives and preset weight parameters, the distribution parameters of the combined weight information in the objective space can be determined. Based on the distribution parameters, the combined weight information of the task objectives can be generated.

[0061] The distribution parameter can be understood as the weight information of the uniform distribution and the deterministic distribution of multiple task objectives in the target space. Based on the number of objectives and the preset weight parameters, the distribution parameter of the combined weight information in the target space can be determined as shown in formula (1), specifically as follows:

[0062]

[0063] Where ξ is the distribution parameter, β is the preset weight parameter, and n is the target number.

[0064] After determining the distribution parameters of the combined weight information in the target space based on the number of targets and preset weight parameters, the combined weight information of the task targets can be generated based on the distribution parameters. Methods for generating the combined weight information of task targets based on the distribution parameters may include: constructing a uniform distribution corresponding to multiple task targets based on the number of targets; identifying the task order among the multiple task targets; constructing a deterministic distribution corresponding to the task targets based on the task order; and mixing the deterministic distribution with the uniform distribution according to the distribution parameters to obtain the combined weight information of the task targets.

[0065] Taking a target quantity of n as an example, the distribution is uniform.

[0066] Taking the first of multiple task objectives as an example, the deterministic distribution corresponding to this task objective is...

[0067] After constructing uniform and deterministic distributions, the deterministic and uniform distributions can be mixed according to the distribution parameters to obtain the combined weight information of the task objective. The method of mixing the deterministic and uniform distributions according to the distribution parameters can include: determining the first distribution weight of the deterministic distribution and the second distribution weight of the uniform distribution according to the distribution parameters; weighting the deterministic distribution according to the first distribution weight and the uniform distribution according to the second distribution weight; and then fusing the weighted deterministic distribution and the weighted uniform distribution to obtain the combined weight information of the task objective. Taking the task objective as the first among multiple task objectives as an example, the process of determining the combined weight information of this task objective can be as shown in formula (2), specifically as follows:

[0068] w1=ξo1+(1-ξ)u (2)

[0069] Where w1 represents the combined weight information of the task objective, o1 represents the deterministic distribution of the task objective, and u represents the uniform distribution of multiple task objectives.

[0070] Each combination of weight information (combination weight vector / basis vector) has a dominance value (preset weight parameter). Other non-dominated components are set as follows: Taking the combined weight information w1 as an example, By setting a dominance value, significant differences between base models are ensured, and excessive overlap of model capabilities is avoided, thereby improving Linear Mode Connectivity (LMC) and enhancing the controllability and effectiveness of model fusion.

[0071] This scheme incorporates the EXP3 exploration strategy. By mixing the determinism of each task objective with the uniform distribution corresponding to multiple task objectives, it ensures that the combined weight information (basis vectors) not only possesses exploration capabilities but also remains stably distributed across the target space, avoiding the limitations of overfitting a single reward. Furthermore, experimental verification shows that the preset weight parameter β used in this scheme ranges from (0.5, 1), and the Pareto performance and controllability of the model are optimized through a small-scale validation dataset. Pareto performance can be understood as the performance of Pareto optimization. Pareto optimization is an important concept in multi-objective optimization, referring to the process of balancing multiple objectives to ensure that the generated result cannot be further improved on any one objective but improves the performance of other objectives, thus forming the so-called Pareto front.

[0072] 103. Based on the combined weight information, the reward information is weighted to obtain the target reward information corresponding to the task objective.

[0073] For example, the target weight of each reward information in the reward information set can be identified in the combined weight information. The reward information can be weighted according to the target weight to obtain at least one weighted reward information corresponding to the combined weight information. The weighted reward information can be merged to obtain the target reward information of the task objective.

[0074] The sum of the target weights is 1. The method for fusing the weighted reward information can include: accumulating the weighted reward information corresponding to the same combined weight information to obtain the target reward information of the task objective corresponding to that combined weight information, as shown in formula (3), specifically as follows:

[0075]

[0076] Among them, h i (s,a) represents the target reward information for the i-th task objective, w i For the combined weight information of the i-th task objective, r j (s,a) represents the j-th reward information.

[0077] Taking a scenario with multiple task objectives including three task objectives, a preset weight parameter of β, and a reward information set including the reward information of task objective R1, R2, and R3, the calculation process of the target reward information corresponding to each of the multiple task objectives can be shown in formula (4), and can be further described as follows:

[0078]

[0079] Where h1, h2, and h3 are the target reward information corresponding to the three task objectives respectively, β is the preset weight parameter, and R1, R2, and R3 are the reward information of the three task objectives respectively.

[0080] It should be noted that this scheme expands the original reward information (signal) into multiple reward information sets. By combining weight information w i Construct target reward information h i (s,a) (a new base reward function) introduces multi-objective reinforcement learning and reward combination mechanism in the process of generating multi-objective controllable tasks, which solves the problem that a single reward cannot achieve optimal performance and the problems of poor model generalization and limitations caused by training with a single reward.

[0081] 104. Train the preset content generation model based on the target reward information to obtain the initial content generation model corresponding to the task target.

[0082] For example, based on the target reward information, the preset content generation model can be trained using multi-objective reinforcement learning to obtain the initial content generation model corresponding to each task objective among multiple task objectives.

[0083] It should be noted that different target reward information (i.e., base reward functions) can be used to train the same preset content generation model, thereby obtaining the initial content generation model corresponding to each task objective.

[0084] 105. Based on the combined weight information and the preference information for multiple task objectives, the initial content generation model is fused, and the fused model is used to generate target content that satisfies the preference information.

[0085] Preference information can be understood as the subjective judgment or preference of the target (decision-maker) regarding the relative importance of multiple task objectives. Based on the combined weight information and preference information for multiple task objectives, there are several ways to fuse the initial content generation model, as follows:

[0086] For example, the target object's preference information for multiple task objectives can be obtained. Based on the preference information and combined weight information, the fusion weight of the initial content generation model can be determined. According to the fusion weight, the initial content generation model is fused to obtain the fused model.

[0087] There are several ways to obtain the target object's preference information for multiple task targets. For example, it can directly obtain the target object's preference information for multiple task targets uploaded by the terminal or client. Alternatively, it can display a preference information collection page, which includes preference parameter input controls for multiple task targets. In response to the target object's trigger operation for the preference parameter input controls, it can collect the target object's preference information for multiple task targets. Or, it can obtain the target object's object identifier, determine the target object's object type based on the object identifier, and filter the preference information corresponding to the object type from the preference information set to obtain the target object's preference information for multiple task targets, and so on.

[0088] After obtaining the target object's preference information for multiple task objectives, the fusion weights of the initial content generation model can be determined based on the preference information and combined weight information. Methods for determining the fusion weights of the initial content generation model based on preference information and combined weight information may include: extracting preference features from the preference information; constructing weight features corresponding to multiple task objectives based on the combined weight information; and calculating the linear relationship between the preference features and the weight features to obtain the fusion weights of the initial content generation model.

[0089] Here, the preference feature can be understood as the preference weights of multiple task features, and the sum of the preference weights is 1.

[0090] Among them, weight features can be understood as weight matrices containing combined weight information. Based on the combined weight information, the method of constructing weight features corresponding to multiple task objectives can include: constructing a symmetric cyclic matrix from the weights in the combined weight information, thereby obtaining the weight features corresponding to multiple task objectives.

[0091] Taking a preset weight parameter of β and a number of task objectives of n as an example, the symmetric cyclic matrix B can be expressed as shown in formula (5), specifically as follows:

[0092]

[0093] Where B represents the weighting feature, β represents the preset weighting parameter (also known as the dominance value), and n represents the number of task objectives. It should also be noted that B is invertible, thus allowing for the determination of unique fusion coefficients (weights) in subsequent fusion stages through linearly solving linear equations.

[0094] After constructing weight features corresponding to multiple task objectives based on combined weight information, the linear relationship between preference features and weight features can be calculated to obtain the fusion weights of the initial content generation model. Taking preference features as preference vectors as an example, the fusion weights can be determined by solving the linear equation μ = Bλ. Since the weight features (symmetric cyclic features B) are invertible, the fusion weights λ can be uniquely determined. The calculation process can be shown in formula (6), specifically as follows:

[0095] λ=B -1 μ (6)

[0096] Where λ is the fusion weight, B is the weight feature, and μ is the preference feature.

[0097] After determining the fusion weights of the initial content generation model based on preference information and combined weight information, the initial content generation model can be fused according to the fusion weights to obtain the fused model. The process of fusing the initial content generation model according to the fusion weights may include: obtaining the current model parameters of the initial content generation model, weighting the current model parameters according to the fusion weights, and fusing the weighted current model parameters to obtain the fused model.

[0098] The process of fusing the weighted current model to obtain the fused model may include: fusing the weighted current model parameters to obtain fused model parameters; adjusting the deviation of the fused model parameters based on preset extrapolation parameters to obtain target model parameters; and replacing the current model parameters of the initial content generation model with the target model parameters to obtain the fused model.

[0099] The process of fusing the weighted current model parameters to obtain the fused model parameters may include: summing the weighted current model parameters to obtain the fused model parameters, as shown in formula (7), specifically as follows:

[0100]

[0101] Where θ represents the parameters of the fusion model, θ i Generate the current model parameters for the model based on the i-th initial content, λ i Generate the fusion weights corresponding to the model for the i-th initial content.

[0102] Specifically, based on the fusion weights, the current model parameters of the initial content generation model are fused to obtain the fused model parameters. The specific process is as follows: Figure 3 As shown.

[0103] After fusing the weighted current model parameters, the fused model parameters can be adjusted for deviation based on preset extrapolation parameters to obtain the target model parameters. The preset extrapolation parameters are used to eliminate the influence of deviations in the initial model parameters of the preset content generation model. The methods for adjusting the deviations of the fused model parameters based on the preset extrapolation parameters include: obtaining the initial model parameters of the preset content generation model; determining the model parameter deviation values ​​corresponding to the fused model parameters based on the preset extrapolation parameters and the initial model parameters; and fusing the fused model parameters with the model parameter deviation values ​​to obtain the target model parameters.

[0104] The process of determining the model parameter deviation value corresponding to the fused model parameters based on the preset extrapolation parameters and the initial model parameters may include: calculating the difference between the initial model parameters and the fused model parameters to obtain the model parameter difference value, and fusing the preset extrapolation parameters and the model parameter difference value to obtain the model parameter deviation value corresponding to the fused model parameters.

[0105] After determining the model parameter deviation value corresponding to the fused model parameters based on the preset extrapolation parameters and the initial model parameters, the fused model parameters and the model deviation parameter value can be fused to obtain the target model parameters. The process of fusing the fused model parameters and the model deviation parameter value can include: adding the fused model parameters and the model parameter deviation value to obtain the target model parameters, as shown in formula (8), specifically as follows:

[0106]

[0107] in, Here, α represents the target model parameters, α represents the preset extrapolation parameters, and θ represents the fusion model parameters. sft These are the initial model parameters.

[0108] After adjusting the deviation of the fusion model parameters based on the preset extrapolation parameters, the current model parameters can be replaced with the target model parameters in the initial content generation model to obtain the fused model.

[0109] Optionally, in some embodiments, the process of fusing the initial content generation model according to the fusion weight to obtain the fused model may further include: obtaining the initial model parameters of the preset content generation model and the current model parameters of the initial content generation model; adjusting the deviation of the current model parameters based on the preset extrapolation parameters and the initial model parameters to obtain the adjusted model parameters; fusing the adjusted model parameters according to the fusion weight to obtain the target model parameters; and replacing the current model parameters with the target model parameters in the initial content generation model to obtain the fused model.

[0110] The method of adjusting the deviation of the current model parameters based on the preset extrapolation parameters and the initial model parameters can be similar to the method of adjusting the deviation of the fused model parameters, as described above, and will not be repeated here.

[0111] After adjusting the deviation of the current model parameters based on preset extrapolation parameters and initial model parameters, the adjusted model parameters can be fused according to the fusion weights to obtain the target model parameters. The process of fusing the adjusted model parameters according to the fusion weights may include: weighting the adjusted model parameters according to the fusion weights, and then fusing the weighted adjusted model parameters to obtain the target model parameters.

[0112] After fusing the adjusted model parameters according to the fusion weights to obtain the target model parameters, the current model parameters can be replaced with the target model parameters in the initial content generation model to obtain the fused model.

[0113] In order to further improve the generation quality, this scheme introduces an extrapolation optimization mechanism, which can weaken the influence of the preset content generation model (SFT model) on the fusion result, thereby effectively eliminating the bias of the preset content generation model and further improving the performance of the fused model in multi-objective generation tasks.

[0114] After fusing the initial content generation model based on combined weight information and preference information for multiple task objectives, the fused model can be used to generate target content that satisfies the preference information. The number of target contents can be one or more. The types of target content can also be varied, including text, audio, video, image, or content presented in other formats. There are several ways to generate target content that satisfies the preference information using the fused model. For example, it can involve obtaining content prompts corresponding to the target object, extracting content features from the prompts using the fused model, and generating at least one type of content that satisfies the preference information based on these features, thus obtaining the target content. Alternatively, it can involve obtaining content prompts for at least one modality corresponding to the target object, extracting initial content features for each modality using the fused model, fusing these initial content features to obtain multimodal target content features, and generating at least one type of content that satisfies the preference information based on these features, thus obtaining the target content, and so on.

[0115] Content hints can be understood as information indicating the aspect or type of content to be used. Content hints can be of the same type as the final generated target content; for example, both can be text, video, audio, image, or other content. Content hints can also be of a different type than the final generated target content; for example, content hints can be audio / video / image content, while the generated target content can be text content, and so on.

[0116] It should be noted that, in the multi-objective content generation task, taking the number of task objectives as 3 and the preference vector μ = (0.2, 0.5, 0.3) as an example, the current model fusion process can be as follows: Figure 4 As shown, a single reward (RM1 / RM2 / RM3) is used to fine-tune the model for each single objective, resulting in a series of initial content generation models for each single objective. During testing, preference information is directly used as fusion weights to fuse the models through linear interpolation. The current model fusion process is limited by not considering the mutual influence between different task objectives, causing the fused model to fail to effectively approach the Pareto optimal frontier. The model fusion process of this scheme can be described as follows: Figure 5As shown, a multi-objective reinforcement learning and reward combination mechanism is introduced. By combining weight information (combined weight vector) and multiple original reward information sets (reward model set), target reward information (new base reward functions h1 / h2 / h3) is constructed. The target reward information is then used to train the preset content generation models, resulting in an initial content generation model for each task objective. A symmetric cyclic matrix B is constructed using the combined weight information. Then, based on the cyclic matrix B and the target object's preference features (preference vector) for multiple task objectives, the linear equation μ = Bλ is solved to dynamically determine the fusion weights λ(λ1 / λ2 / λ3) of the initial content generation models for each task objective. Finally, the initial content generation models are fused based on the fusion weights to obtain the fused model. Furthermore, it should be noted that an extrapolation optimization mechanism is introduced during the fusion process to eliminate the bias influence of the preset content generation models.

[0117] This solution can be applied to various scenarios of multi-objective generation tasks, especially in AI search scenarios. Traditional search engines or question-answering systems often can only generate results based on a pre-set single objective (such as relevance or accuracy), making it difficult to simultaneously meet the diverse needs of users in different scenarios. This invention provides users with a method to dynamically adjust the generated content through multi-objective controllable generation technology. In specific applications, this solution can be directly applied to the answer generation module of an AI search system. Using this method, the AI ​​search system can more effectively understand and integrate complex information from multiple documents, overcoming the limitations of existing methods in cross-document information extraction and fusion. For example, when dealing with questions that require integrating information from multiple documents, this solution can dynamically adjust the model fusion parameters by reconstructing the base model, achieving an optimal balance between accuracy, relevance, and information completeness in the generated answer. Regardless of whether the user prioritizes the comprehensiveness of the answer (such as academic research) or its conciseness (such as fast querying), the system can adjust the generated results in real time according to preferences, ensuring that the output meets the user's needs. Furthermore, this solution can not only improve the quality of results in multi-document understanding tasks but also enhance the reliability and consistency of the generated results. For example, in scenarios where accuracy is paramount, such as medical Q&A or legal research, this invention can dynamically optimize generated content, avoiding answer bias caused by conflicting objectives and providing users with high-quality reference opinions. Therefore, this invention possesses the ability to extend multi-objective optimization technology from theory to practical application, providing a future-oriented intelligent solution for the field of AI search.

[0118] In the AI ​​search scenario, this solution overcomes the limitations of traditional model fusion methods that cannot simultaneously consider both objective conflict and user preferences by introducing a base model reconstruction technique based on multi-objective reinforcement learning (using combined weight information to construct a new base reward function), a dynamic fusion mechanism (determining fusion weights based on a symmetric cyclic matrix composed of combined weight information and preference information), and an extrapolation optimization mechanism (eliminating the bias caused by the initial model parameters of the preset content generation model based on preset external coefficients). This results in the following beneficial effects:

[0119] (1) Improve answer quality and user experience

[0120] In AI search scenarios, taking tasks with accuracy, relevance, and information completeness as examples, this solution dynamically adjusts the fusion weights to achieve an optimal balance among these three factors based on user needs, thereby generating high-quality answers. For instance, when processing information queries that require integrating multiple documents, the answers generated by this solution are not only comprehensive but also highlight the key points, significantly improving user satisfaction and overall experience.

[0121] (2) Improve the efficiency and reliability of multi-document understanding

[0122] Experiments show that this scheme, through optimized training of the base model (preset content generation model) and symmetric matrix mapping to determine the fusion weights of each model, can effectively improve the fusion model's ability to process complex information in documents. The performance of this scheme compared to other current multi-objective controllable generation schemes (RS / MOD / RiC) on different datasets (HH1 / HH2 / FP) is shown in Table 1. Figure 6 As shown, HH1 is the training data for usefulness and harmlessness (Helpful vs Harmless), HH2 is the training data for usefulness and humor, and FP is the training data for loyalty and preference, as detailed below:

[0123] Table 1

[0124]

[0125] Continued from Table 1

[0126]

[0127] Through Table 1 and Figure 6 The content shows that the Pareto front performance of this scheme is significantly better than that of existing technologies, the generated results are closer to the user's true preferences, and the inconsistency caused by goal conflict is reduced, thus improving the reliability of the answer.

[0128] (3) Enhance the dynamic adaptability of the search system

[0129] By introducing a linear solution mechanism in the fusion phase, this solution can dynamically adjust the parameters of the generated model based on the user's input preferences during testing, ensuring that the search system can respond to user needs in real time. This adaptability is particularly suitable for complex scenarios, such as medical question answering or legal retrieval, where user preferences may change depending on the question's background or context. This invention can quickly adjust the generated content to adapt to these dynamic needs.

[0130] As can be seen from the above, in this embodiment, after obtaining a set of reward information for a preset content generation model, which includes reward information corresponding to multiple task objectives, the combined weight information of the task objectives is determined based on preset weight parameters. This combined weight information indicates the importance of the task objective among multiple task objectives. Then, the reward information is weighted according to the combined weight information to obtain the target reward information corresponding to the task objective. The preset content generation model is trained based on the target reward information to obtain the initial content generation model corresponding to the task objective. Then, the initial content generation model is fused according to the combined weight information and the preference information for multiple task objectives, and the fused model is used to generate target content that satisfies the preference information. Since this scheme uses multi-objective reinforcement learning to weight the combined weight information with multiple reward information, thereby training a series of base models (initial content generation models), it ensures that the base models are closer to the Pareto optimal solution in the multi-objective space. In addition, the model fusion using combined weight information and preference information achieves fine control of the model output, ensuring that the fused model can balance the trade-offs between different objectives and generate the optimal result. Therefore, the accuracy of content generation can be improved.

[0131] Based on the method described in the above embodiments, the following examples will provide further detailed explanations.

[0132] In this embodiment, the content generation device will be specifically integrated into an electronic device, with the electronic device being a server, as an example for explanation.

[0133] like Figure 7 As shown, a content generation method has the following specific process:

[0134] 201. The server obtains a set of reward information for generating a model based on preset content.

[0135] For example, the server can directly receive a set of reward information for a preset content generation model uploaded by a terminal or client. Alternatively, it can filter out reward information corresponding to multiple task objectives from the preset reward information set to obtain a set of reward information for the preset content generation model. Or, it can receive a training request for the preset content generation model, which carries the storage address of reward information corresponding to multiple task objectives. Based on the storage address, it can obtain the reward information corresponding to multiple task objectives and use this reward information as the set of reward information for the preset content generation model, and so on.

[0136] 202. The server determines the combined weight information of the task objectives based on preset weight parameters.

[0137] For example, the server can count the number of task objectives among multiple task objectives to obtain the number of objectives. Based on the number of objectives and preset weight parameters, the distribution parameters of the combined weight information in the objective space can be determined as shown in formula (1).

[0138] The server can construct a uniform distribution corresponding to multiple task objectives based on the number of objectives. It then identifies the task order among these objectives and constructs a deterministic distribution based on this order. According to the distribution parameters, it determines a first distribution weight for the deterministic distribution and a second distribution weight for the uniform distribution. The deterministic distribution is weighted according to the first distribution weight, and the uniform distribution is weighted according to the second distribution weight. Finally, the weighted deterministic distribution and the weighted uniform distribution are merged to obtain the combined weight information for the task objectives.

[0139] 203. The server weights the reward information according to the combined weight information to obtain the target reward information corresponding to the task objective.

[0140] For example, the server can identify the target weight of each reward information in the reward information set from the combined weight information, weight the reward information according to the target weight, obtain at least one weighted reward information corresponding to the combined weight information, and fuse the weighted reward information to obtain the target reward information of the task objective.

[0141] 204. The server trains the preset content generation model based on the target reward information to obtain the initial content generation model corresponding to the task target.

[0142] For example, the server can train a preset content generation model using multi-objective reinforcement learning based on the target reward information, thereby obtaining the initial content generation model corresponding to each task objective among multiple task objectives.

[0143] 205. The server obtains the target object's preference information for multiple task objectives.

[0144] For example, there are several ways for a server to obtain the target object's preference information for multiple task objectives. For instance, it can directly obtain the target object's preference information for multiple task objectives uploaded by the terminal or client, or display a preference information collection page that includes preference parameter input controls for multiple task objectives. In response to the target object's trigger operation on the preference parameter input controls, the server can collect the target object's preference information for multiple task objectives. Alternatively, it can obtain the target object's object identifier, determine the target object's object type based on the object identifier, and filter the preference information corresponding to the object type from the preference information set to obtain the target object's preference information for multiple task objectives, and so on.

[0145] 206. The server determines the fusion weights of the initial content generation model based on preference information and combined weight information.

[0146] For example, the server can extract preference features from preference information, construct a symmetric cyclic matrix by combining the weights in the weight information, thereby obtaining the weight features corresponding to multiple task objectives, and calculate the linear relationship between preference features and weight features to obtain the fusion weights of the initial content generation model.

[0147] 207. The server merges the initial content generation model according to the fusion weight to obtain the merged model.

[0148] For example, the server can obtain the current model parameters of the initial content generation model, and weight the current model parameters based on the fusion weight. The weighted current model parameters are then summed to obtain the fusion model parameters, as shown in formula (7).

[0149] The server can obtain the initial model parameters of the preset content generation model, calculate the difference between the initial model parameters and the fused model parameters, obtain the model parameter difference, fuse the preset extrapolated parameters and the model parameter difference to obtain the model parameter deviation value corresponding to the fused model parameters, add the fused model parameters and the model parameter deviation value to obtain the target model parameters, as shown in formula (8). In the initial content generation model, the current model parameters are replaced with the target model parameters to obtain the fused model.

[0150] In some embodiments, the server can obtain the initial model parameters of a preset content generation model and the current model parameters of the initial content generation model. Based on preset extrapolation parameters and the initial model parameters, the current model parameters are adjusted to obtain adjusted model parameters. The adjusted model parameters are then weighted according to the fusion weights and fused to obtain the target model parameters. The current model parameters are then replaced with the target model parameters in the initial content generation model to obtain the fused model.

[0151] 208. The server uses the fused model to generate target content that satisfies the preference information.

[0152] For example, the server may include obtaining content prompt information corresponding to the target object, extracting content features from the content prompt information using a fusion model, and generating at least one type of content that satisfies the preference information based on the content features, thereby obtaining the target content. Alternatively, it may obtain content prompt information of at least one modality corresponding to the target object, extract initial content features corresponding to each modality from the content prompt information using a fusion model, fuse the initial content features to obtain multimodal target content features, and generate at least one type of content that satisfies the preference information based on the target content features, thereby obtaining the target content, and so on.

[0153] As can be seen from the above, in this embodiment, after the server obtains a set of reward information for a preset content generation model, which includes reward information corresponding to multiple task objectives, it determines the combined weight information of the task objectives based on preset weight parameters. This combined weight information indicates the importance of the task objective among multiple task objectives. Then, the reward information is weighted according to the combined weight information to obtain the target reward information corresponding to the task objective. The preset content generation model is trained based on the target reward information to obtain the initial content generation model corresponding to the task objective. Then, the initial content generation model is fused according to the combined weight information and the preference information for multiple task objectives, and the fused model is used to generate target content that satisfies the preference information. Since this scheme uses multi-objective reinforcement learning to weight the combined weight information with multiple reward information to train a series of base models (initial content generation models), it ensures that the base models are closer to the Pareto optimal solution in the multi-objective space. In addition, the model fusion using combined weight information and preference information enables fine control of the model output, ensuring that the fused model can balance the trade-offs between different objectives and generate the optimal result. Therefore, the accuracy of content generation can be improved.

[0154] To better implement the above methods, this application also provides a content generation device that can be integrated into an electronic device, such as a server or terminal. The terminal may include a tablet computer, a laptop computer, and / or a personal computer.

[0155] For example, such as Figure 8 As shown, the content generation device may include an acquisition unit 301, a determination unit 302, a weighting unit 303, a training unit 304, and a generation unit 305, as follows:

[0156] (1) Obtain unit 301;

[0157] The acquisition unit 301 is used to acquire a set of reward information for the preset content generation model, which includes reward information corresponding to multiple task objectives.

[0158] (2) Determine unit 302;

[0159] The determining unit 302 is used to determine the combined weight information of the task objectives based on preset weight parameters. The combined weight information indicates the importance of the task objectives among multiple task objectives.

[0160] For example, the determining unit 302 can be used to count the number of task objectives among multiple task objectives, obtain the number of objectives, determine the distribution parameters of the combined weight information in the target space based on the number of objectives and preset weight parameters, construct a uniform distribution corresponding to multiple task objectives based on the number of objectives, identify the task order of the task objectives among multiple task objectives, construct a deterministic distribution corresponding to the task objectives based on the task order, and mix the deterministic distribution and the uniform distribution according to the distribution parameters to obtain the combined weight information of the task objectives.

[0161] (3) Weighting unit 303;

[0162] Weighting unit 303 is used to weight the reward information according to the combined weight information to obtain the target reward information corresponding to the task target.

[0163] For example, the weighting unit 303 can be used to identify the target weight of each reward information in the reward information set in the combined weight information, weight the reward information according to the target weight, obtain at least one weighted reward information corresponding to the combined weight information, and fuse the weighted reward information to obtain the target reward information of the task objective.

[0164] (4) Training Unit 304;

[0165] Training unit 304 is used to train a preset content generation model based on target reward information to obtain an initial content generation model corresponding to the task target.

[0166] For example, training unit 304 can be used to train a preset content generation model for multi-objective reinforcement learning based on target reward information, thereby obtaining the initial content generation model corresponding to each task objective among multiple task objectives.

[0167] (5) Generation unit 305;

[0168] The generation unit 305 is used to fuse the initial content generation model based on the combined weight information and the preference information for multiple task objectives, and to use the fused model to generate target content that satisfies the preference information.

[0169] For example, the generation unit 305 can be used to obtain the target object's preference information for multiple task objectives, extract preference features from the preference information, construct weight features corresponding to multiple task objectives based on the combined weight information, calculate the linear relationship between preference features and weight features to obtain the fusion weight of the initial content generation model, fuse the initial content generation model according to the fusion weight to obtain the fused model, and use the fused model to generate target content that satisfies the preference information.

[0170] In practice, each of the above units can be implemented as an independent entity or can be arbitrarily combined to be implemented as the same or several entities. For the specific implementation of each of the above units, please refer to the previous method embodiments, which will not be repeated here.

[0171] As can be seen from the above, in this embodiment of the application, after the acquisition unit 301 acquires the reward information set for the preset content generation model, the reward information set includes reward information corresponding to multiple task objectives. The determination unit 302 determines the combined weight information of the task objectives based on preset weight parameters. This combined weight information indicates the importance of the task objective among multiple task objectives. Then, the weighting unit 303 weights the reward information according to the combined weight information to obtain the target reward information corresponding to the task objective. The training unit 304 trains the preset content generation model based on the target reward information to obtain the initial content generation model corresponding to the task objective. Then, the generation unit 305 generates the content based on the combined weight information. By combining weight information and preference information for multiple task objectives, the initial content generation model is fused, and the fused model is used to generate target content that satisfies the preference information. Since this scheme uses multi-objective reinforcement learning to weight the combined weight information with multiple reward information, a series of base models (initial content generation models) are trained, ensuring that the base models are closer to the Pareto optimal solution in the multi-objective space. In addition, by using the combined weight information and preference information to fuse the model, fine control of the model output is achieved, ensuring that the fused model can balance the trade-offs between different objectives (tasks) and generate the optimal result. Therefore, the accuracy of content generation can be improved.

[0172] This application also provides an electronic device, such as... Figure 9 As shown, it illustrates a structural schematic diagram of the electronic device involved in the embodiments of this application, specifically:

[0173] The electronic device may include components such as a processor 401 with one or more processing cores, a memory 402 with one or more computer-readable storage media, a power supply 403, and an input unit 404. Those skilled in the art will understand that... Figure 9 The electronic device structure shown does not constitute a limitation on the electronic device and may include more or fewer components than shown, or combine certain components, or have different component arrangements. Wherein:

[0174] The processor 401 is the control center of the electronic device, connecting various parts of the device via various interfaces and lines. It executes software programs and / or modules stored in the memory 402, and calls data stored in the memory 402, to perform various functions and process data. Optionally, the processor 401 may include one or more processing cores; preferably, the processor 401 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the modem processor may not be integrated into the processor 401.

[0175] The memory 402 can be used to store software programs and modules. The processor 401 executes various functional applications and data processing by running the software programs and modules stored in the memory 402. The memory 402 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, application programs required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 402 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, the memory 402 may also include a memory controller to provide the processor 401 with access to the memory 402.

[0176] The electronic device also includes a power supply 403 that supplies power to the various components. Preferably, the power supply 403 can be logically connected to the processor 401 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system. The power supply 403 may also include one or more DC or AC power supplies, recharging systems, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0177] The electronic device may also include an input unit 404, which can be used to receive input digital or character information, and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function control.

[0178] Although not shown, the electronic device may also include a display unit, etc., which will not be described in detail here. Specifically, in this embodiment, the processor 401 in the electronic device loads the executable files corresponding to the processes of one or more applications into the memory 402 according to the following instructions, and the processor 401 runs the applications stored in the memory 402 to realize various functions, as follows:

[0179] A reward information set for a preset content generation model is obtained, which includes reward information corresponding to multiple task objectives. Based on preset weight parameters, combined weight information of task objectives is determined, which indicates the importance of a task objective among multiple task objectives. The reward information is weighted according to the combined weight information to obtain the target reward information corresponding to the task objective. The preset content generation model is trained based on the target reward information to obtain the initial content generation model corresponding to the task objective. The initial content generation model is fused according to the combined weight information and the preference information for multiple task objectives, and the fused model is used to generate target content that satisfies the preference information.

[0180] For example, an electronic device can acquire a set of reward information for a preset content generation model. This set of reward information includes reward information corresponding to multiple task objectives. The number of task objectives is counted among the multiple task objectives, resulting in the target quantity. Based on the target quantity and preset weight parameters, the distribution parameters of the combined weight information in the target space are determined. A uniform distribution corresponding to the multiple task objectives is constructed based on the target quantity. The task order of the task objectives is identified among the multiple task objectives, and a deterministic distribution corresponding to the task objectives is constructed based on the task order. Based on the distribution parameters, the deterministic distribution and the uniform distribution are mixed to obtain the combined weight information of the task objectives. The target weight of each reward information in the reward information set is identified in the combined weight information. The reward information is weighted based on the target weight to obtain at least one weighted reward information corresponding to the combined weight information. The weighted reward information is then fused to obtain the target reward information of the task objective. Based on the target reward information, the preset content generation model is trained using multi-objective reinforcement learning, thereby obtaining the initial content generation model corresponding to each of the multiple task objectives. The system obtains the target object's preference information for multiple task objectives, extracts preference features from the preference information, constructs weight features corresponding to multiple task objectives based on combined weight information, calculates the linear relationship between preference features and weight features to obtain the fusion weight of the initial content generation model, and fuses the initial content generation model according to the fusion weight to obtain the fused model. The fused model is then used to generate target content that satisfies the preference information.

[0181] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.

[0182] As can be seen from the above, in this embodiment, after obtaining a set of reward information for a preset content generation model, which includes reward information corresponding to multiple task objectives, the combined weight information of the task objectives is determined based on preset weight parameters. This combined weight information indicates the importance of the task objective among multiple task objectives. Then, the reward information is weighted according to the combined weight information to obtain the target reward information corresponding to the task objective. The preset content generation model is trained based on the target reward information to obtain the initial content generation model corresponding to the task objective. Then, the initial content generation model is fused according to the combined weight information and the preference information for multiple task objectives, and the fused model is used to generate target content that satisfies the preference information. Since this scheme uses multi-objective reinforcement learning to weight the combined weight information with multiple reward information, thereby training a series of base models (initial content generation models), it ensures that the base models are closer to the Pareto optimal solution in the multi-objective space. In addition, the model fusion using combined weight information and preference information achieves fine control of the model output, ensuring that the fused model can balance the trade-offs between different objectives and generate the optimal result. Therefore, the accuracy of content generation can be improved.

[0183] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be performed by instructions, or by instructions controlling related hardware. These instructions can be stored in a computer-readable storage medium and loaded and executed by a processor.

[0184] Therefore, embodiments of this application provide a computer-readable storage medium storing a plurality of instructions that can be loaded by a processor to execute steps in any of the content generation methods provided in embodiments of this application. For example, the instructions can execute the following steps:

[0185] A reward information set for a preset content generation model is obtained, which includes reward information corresponding to multiple task objectives. Based on preset weight parameters, combined weight information of task objectives is determined, which indicates the importance of a task objective among multiple task objectives. The reward information is weighted according to the combined weight information to obtain the target reward information corresponding to the task objective. The preset content generation model is trained based on the target reward information to obtain the initial content generation model corresponding to the task objective. The initial content generation model is fused according to the combined weight information and the preference information for multiple task objectives, and the fused model is used to generate target content that satisfies the preference information.

[0186] For example, a reward information set for a preset content generation model is obtained, which includes reward information corresponding to multiple task objectives. The number of task objectives is counted among the multiple objectives, resulting in the objective count. Based on the objective count and preset weight parameters, the distribution parameters of the combined weight information in the objective space are determined. A uniform distribution corresponding to the multiple task objectives is constructed based on the objective count. The task order of the task objectives is identified among the multiple objectives, and a deterministic distribution corresponding to the task objectives is constructed based on the task order. The deterministic distribution and the uniform distribution are mixed according to the distribution parameters to obtain the combined weight information of the task objectives. The target weight of each reward information in the reward information set is identified in the combined weight information. The reward information is weighted based on the target weight to obtain at least one weighted reward information corresponding to the combined weight information. The weighted reward information is then fused to obtain the target reward information of the task objective. Based on the target reward information, the preset content generation model is trained using multi-objective reinforcement learning, thereby obtaining the initial content generation model corresponding to each of the multiple task objectives. The system obtains the target object's preference information for multiple task objectives, extracts preference features from the preference information, constructs weight features corresponding to multiple task objectives based on combined weight information, calculates the linear relationship between preference features and weight features to obtain the fusion weight of the initial content generation model, and fuses the initial content generation model according to the fusion weight to obtain the fused model. The fused model is then used to generate target content that satisfies the preference information.

[0187] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.

[0188] The computer-readable storage medium may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0189] Since the instructions stored in the computer-readable storage medium can execute the steps in any of the content generation methods provided in the embodiments of this application, the beneficial effects that any of the content generation methods provided in the embodiments of this application can achieve can be realized, as detailed in the preceding embodiments, and will not be repeated here.

[0190] According to one aspect of this application, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the electronic device to perform the methods provided in the various alternative implementations of the model fusion or content generation aspects described above.

[0191] The foregoing has provided a detailed description of a content generation method and related equipment provided in the embodiments of this application. The related equipment may include a content generation device, an electronic device, a computer program product, and a computer-readable storage medium. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A content generation method, characterized in that, include: Obtain a set of reward information for generating a model based on preset content, wherein the set of reward information includes reward information corresponding to multiple task objectives; Based on preset weight parameters, the combined weight information of the task objectives is determined, and the combined weight information indicates the importance of the task objectives among the multiple task objectives; The reward information is weighted according to the combined weight information to obtain the target reward information corresponding to the task objective; The preset content generation model is trained based on the target reward information to obtain the initial content generation model corresponding to the task target; Based on the combined weight information and the preference information for the multiple task objectives, the initial content generation model is fused, and the fused model is used to generate target content that satisfies the preference information.

2. The content generation method according to claim 1, characterized in that, The step of determining the combined weight information of the task objective based on preset weight parameters includes: The number of task objectives is counted among the multiple task objectives to obtain the number of objectives. Based on the target quantity and preset weight parameters, determine the distribution parameters of the combined weight information in the target space; Based on the distribution parameters, the combined weight information of the task objective is generated.

3. The content generation method according to claim 2, characterized in that, The step of generating the combined weight information of the task objective based on the distribution parameters includes: Based on the number of targets, construct a uniform distribution corresponding to the multiple task targets; Identify the task order of the multiple task objectives, and construct a deterministic distribution corresponding to the task objectives based on the task order; Based on the distribution parameters, the deterministic distribution and the uniform distribution are mixed to obtain the combined weight information of the task objective.

4. The content generation method according to claim 3, characterized in that, The step of mixing the deterministic distribution and the uniform distribution according to the distribution weights to obtain the combined weight information of the task objective includes: Based on the distribution parameters, determine the first distribution weight of the deterministic distribution and the second distribution weight corresponding to the uniform distribution; The deterministic distribution is weighted according to the first distribution weight, and the uniform distribution is weighted according to the second distribution weight; The weighted deterministic distribution and the weighted uniform distribution are fused to obtain the combined weight information of the task objective.

5. The content generation method according to claim 1, characterized in that, The step of weighting the reward information according to the combined weight information to obtain the target reward information corresponding to the task objective includes: The target weights of each reward information in the reward information set are identified in the combined weight information, and the sum of the target weights is 1; The reward information is weighted according to the target weight to obtain at least one weighted reward information corresponding to the combined weight information; The weighted reward information is then integrated to obtain the target reward information for the task objective.

6. The content generation method according to claim 1, characterized in that, The step of fusing the initial content generation model based on the combined weight information and the preference information for the multiple task objectives includes: Obtain the target object's preference information for the multiple task objectives; Based on the preference information and combined weight information, the fusion weights of the initial content generation model are determined; The initial content generation model is fused according to the fusion weights to obtain the fused model.

7. The content generation method according to claim 6, characterized in that, The step of determining the fusion weights of the initial content generation model based on the preference information and the combined weight information includes: Extract preference features from the preference information; Based on the combined weight information, construct the weight features corresponding to the multiple task objectives; Calculate the linear relationship between the preference features and the weight features to obtain the fusion weights of the initial content generation model.

8. The content generation method according to claim 6, characterized in that, The step of fusing the initial content generation model according to the fusion weight to obtain the fused model includes: Obtain the current model parameters of the initial content generation model, and weight the current model parameters based on the fusion weights; The weighted current model parameters are then fused to obtain the fused model.

9. The content generation method according to claim 8, characterized in that, The step of fusing the weighted current model parameters to obtain the fused model includes: The weighted current model parameters are then fused to obtain the fused model parameters; Based on preset extrapolation parameters, the parameters of the fusion model are adjusted to obtain the target model parameters. The preset extrapolation parameters are used to eliminate the influence of the deviation of the initial model parameters of the preset content generation model. In the initial content generation model, the current model parameters are replaced with the target model parameters to obtain the fused model.

10. The content generation method according to claim 9, characterized in that, The step of adjusting the deviation of the fusion model parameters based on preset extrapolation parameters to obtain the target model parameters includes: Obtain the initial model parameters of the preset content generation model; Based on the preset extrapolation parameters and the initial model parameters, determine the model parameter deviation value corresponding to the fusion model parameters; The target model parameters are obtained by fusing the fusion model parameters with the model parameter deviation values.

11. The content generation method according to claim 10, characterized in that, The step of determining the model parameter deviation value corresponding to the fused model parameters based on the preset extrapolation parameters and the initial model parameters includes: Calculate the difference between the initial model parameters and the fused model parameters to obtain the model parameter difference; The preset extrapolation parameters are fused with the model parameter values ​​to obtain the model parameter deviation value corresponding to the fused model parameters.

12. The content generation method according to claim 6, characterized in that, The step of fusing the initial content generation model according to the fusion weight to obtain the fused model includes: Obtain the initial model parameters of the preset content generation model and the current model parameters of the initial content generation model; Based on the preset extrapolation parameters and the initial model parameters, the current model parameters are adjusted to obtain the adjusted model parameters; The adjusted model parameters are fused according to the fusion weights to obtain the target model parameters; In the initial content generation model, the current model parameters are replaced with the target model parameters to obtain the fused model.

13. A content generation apparatus, characterized in that, include: The acquisition unit is used to acquire a set of reward information for generating a model based on preset content, wherein the set of reward information includes reward information corresponding to multiple task objectives; The determining unit is configured to determine the combined weight information of the task objectives based on preset weight parameters, wherein the combined weight information indicates the importance of the task objectives among the multiple task objectives; A weighting unit is used to weight the reward information according to the combined weight information to obtain the target reward information corresponding to the task objective; The training unit is used to train the preset content generation model based on the target reward information to obtain the initial content generation model corresponding to the task target. The generation unit is used to fuse the initial content generation model according to the combined weight information and the preference information for the multiple task objectives, and use the fused model to generate target content that satisfies the preference information.

14. An electronic device, characterized in that, It includes a processor and a memory, the memory storing an application program, and the processor running the application program within the memory to perform the steps of the content generation method according to any one of claims 1 to 13.

15. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the content generation method according to any one of claims 1 to 12.

16. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a plurality of instructions adapted for loading by a processor to perform the steps of the content generation method according to any one of claims 1 to 12.

Images