Model training methods and apparatus, content generation methods and apparatus

Abstract

This specification discloses a model training method and apparatus, and a content generation method and apparatus. The training method includes: generating first content that conforms to the content requirements of the original input information using a content model; and training a style model based on the first content and the original input information to further generate second predicted content that conforms to the style requirements based on the first content. The content generation method includes: generating target content that conforms to the content and style requirements of the input information using a content model and a style model, based on the input information.

Description

Technical Field

[0001] One or more embodiments of this specification relate to the field of artificial intelligence technology, and in particular to model training methods and apparatus, and content generation methods and apparatus. Background Technology

[0002] AIGC (Artificial Intelligence Generated Content) refers to content generated by artificial intelligence. AIGC technology can automatically analyze and understand various types of data, such as text, images, audio, and video, and automatically generate mathematical models based on the data, thereby achieving automated modeling.

[0003] Taking image data as an example, existing AIGC technology can generate content that conforms to the prompt (i.e., AI prompt words), but the generated content cannot simultaneously guarantee that it conforms to both the content described in the prompt and the style required by the prompt. Summary of the Invention

[0004] This specification describes one or more embodiments of a model training method and apparatus, and a content generation method and apparatus, which can train a style model based on a content model and generate content that simultaneously meets both content and style requirements when using the content model and style model to generate content.

[0005] Firstly, the embodiments of this specification provide a model training method, including:

[0006] Obtain training samples, which include original input information, original content corresponding to the original input information, and style annotation input information corresponding to the original input information;

[0007] The content extraction module of the content model is used to extract content input information from the original input information.

[0008] Input the aforementioned content information into the content base model of the content model, and output the first content;

[0009] Based on the first content and the original input information, the style model is trained. The training process is as follows:

[0010] The style extraction module of the style model is used to extract style prediction input information from the original input information;

[0011] Input the style prediction input information and the first content into the style base model of the style model, and output the second predicted content;

[0012] The style prediction input information and the second prediction content are input into the style verification module of the style model, and the style type matching score is output.

[0013] The style prediction input and style annotation input are predicted and judged using the first loss function, the second loss function is predicted and judged using the second prediction content and the original content, and the style type matching score output by the style verification module is maximized using the third loss function. When all the above loss functions meet the convergence condition, the style model is trained. Otherwise, the above training process is repeated.

[0014] Secondly, the embodiments of this specification provide a model training method, including:

[0015] Obtain training samples, which include original input information, original content corresponding to the original input information, style annotation input information corresponding to the original input information, and content annotation input information corresponding to the original input information;

[0016] The content model is trained based on the original input information. The training process is as follows:

[0017] The content extraction module of the content model is used to extract content prediction input information from the original input information.

[0018] The content prediction input information is input into the content base model of the content model, and the first predicted content is output.

[0019] The content matching module of the content model inputs the content prediction input information and the first predicted content inputs the content matching module to output the content matching score.

[0020] The content model is trained by using loss function one to predict and judge the content prediction input information and content annotation input information, loss function two to predict and judge the first predicted content and the original content, and loss function three to maximize the content matching score output by the content matching module. When all the above loss functions meet the convergence condition, the content model is trained. Otherwise, the above training process is repeated.

[0021] The content extraction module of the content model is used to extract content input information from the original input information.

[0022] Input the aforementioned content information into the content base model of the content model, and output the first content;

[0023] Based on the first content and the original input information, the style model is trained. The training process is as follows:

[0024] The style extraction module of the style model is used to extract style prediction input information from the original input information;

[0025] Input the style prediction input information and the first content into the style base model of the style model, and output the second predicted content;

[0026] The style prediction input information and the second prediction content are input into the style verification module of the style model, and the style type matching score is output.

[0027] The style prediction input and style annotation input are predicted and judged using the first loss function, the second loss function is predicted and judged using the second prediction content and the original content, and the style type matching score output by the style verification module is maximized using the third loss function. When all the above loss functions meet the convergence condition, the style model is trained. Otherwise, the above training process is repeated.

[0028] Thirdly, the embodiments of this specification provide a model training method. Based on the second aspect, during the training process of the content model and the style model, the method further includes: randomly replacing the style prediction input information and / or content prediction input information during the training process with the original input information.

[0029] Fourthly, the embodiments of this specification provide a model training method. Based on the second aspect, after the content model and the style model have been trained separately, the method further includes: jointly training the content model and the style model, wherein the style prediction input information and / or content prediction input information during the training process will be randomly replaced with the original input information; the training process of the content model and the style model in the joint training is the same as their individual training process.

[0030] Fifthly, embodiments of this specification provide a content generation method, implemented based on the content model and style model trained by the methods described in the second, third, and fourth aspects above; the content generation method includes:

[0031] Input the input information into the content extraction module of the content model, and output the content input information;

[0032] Input the input information into the content base model of the content model, and output the first target content;

[0033] Input the input information into the style extraction module in the style model, and output the style input information;

[0034] Input the style input information and the first target content into the style base model in the style model, and output the second target content.

[0035] Sixthly, embodiments of this specification provide a model training apparatus, including:

[0036] The sample acquisition unit is used to acquire training samples, which include original input information, original content corresponding to the original input information, and style annotation input information corresponding to the original input information.

[0037] The content model unit is used to extract content input information from the original input information using the content extraction module of the content model; it is also used to input the content input information into the content base model of the content model and output the first content.

[0038] The style model training unit is used to train the style model based on the first content and the original input information. The training process is as follows:

[0039] The style extraction module of the style model is used to extract style prediction input information from the original input information;

[0040] Input the style prediction input information and the first content into the style base model of the style model, and output the second predicted content;

[0041] The style prediction input information and the second prediction content are input into the style verification module of the style model, and the style type matching score is output.

[0042] The style prediction input and style annotation input are predicted and judged using the first loss function, the second loss function is predicted and judged using the second prediction content and the original content, and the style type matching score output by the style verification module is maximized using the third loss function. When all the above loss functions meet the convergence condition, the style model is trained. Otherwise, the above training process is repeated.

[0043] Seventhly, embodiments of this specification provide a model training apparatus, including:

[0044] The sample acquisition unit is used to acquire training samples, which include original input information, original content corresponding to the original input information, style annotation input information corresponding to the original input information, and content annotation input information corresponding to the original input information.

[0045] The content model training unit is used to train the content model based on the original input information before training the style model using the first content obtained from the content model. The training process is as follows:

[0046] The content extraction module of the content model is used to extract content prediction input information from the original input information.

[0047] The content prediction input information is input into the content base model of the content model, and the first predicted content is output.

[0048] The content matching module of the content model inputs the content prediction input information and the first predicted content inputs the content matching module to output the content matching score.

[0049] The content model is trained by using loss function one to predict and judge the content prediction input information and content annotation input information, loss function two to predict and judge the first predicted content and the original content, and loss function three to maximize the content matching score output by the content matching module. When all the above loss functions meet the convergence condition, the content model is trained and the trained content model is sent to the content model unit. Otherwise, the above training process is repeated.

[0050] The content model unit is used to extract content input information from the original input information using the content extraction module of the content model; it is also used to input the content input information into the content base model of the content model and output the first content.

[0051] The style model training unit is used to train the style model based on the first content and the original input information. The training process is as follows:

[0052] The style extraction module of the style model is used to extract style prediction input information from the original input information;

[0053] Input the style prediction input information and the first content into the style base model of the style model, and output the second predicted content;

[0054] The style prediction input information and the second prediction content are input into the style verification module of the style model, and the style type matching score is output.

[0055] The style prediction input and style annotation input are predicted and judged using the first loss function, the second loss function is predicted and judged using the second prediction content and the original content, and the style type matching score output by the style verification module is maximized using the third loss function. When all the above loss functions meet the convergence condition, the style model is trained. Otherwise, the above training process is repeated.

[0056] Eighthly, the embodiments of this specification provide a model training apparatus, which, based on the seventh aspect, further includes a random replacement unit, used to randomly replace the style prediction input information and / or content prediction input information during the training process of the content model and the style model with the original input information.

[0057] Ninthly, the embodiments of this specification provide a model training apparatus, which, based on the seventh aspect, further includes a random replacement unit for jointly training the content model and the style model after the content model and the style model have been trained separately. During the training process, the style prediction input information and / or content prediction input information will be randomly replaced with the original input information. The training process of the content model and the style model in the joint training is the same as their individual training process.

[0058] Tenthly, embodiments of this specification provide a content generation apparatus, implemented based on the content model and style model trained by the methods described in the second, third, and fourth methods; the apparatus includes:

[0059] The content input information extraction unit is used to input the input information into the content extraction module in the content model and output the content input information.

[0060] The first target content generation unit is used to input the input information into the content base model in the content model and output the first target content.

[0061] The style input information extraction unit is used to input the input information into the style extraction module in the style model and output the style input information.

[0062] The second target content generation unit is used to input the style input information and the first target content into the style base model in the style model, and output the second target content.

[0063] Eleventhly, embodiments of this specification provide an electronic device, including a processor and a memory;

[0064] The processor is connected to the memory;

[0065] The memory is used to store executable program code;

[0066] The processor runs a program corresponding to the executable program code stored in the memory to perform the method described in any of the above aspects.

[0067] In a twelfth aspect, embodiments of this specification provide a computer storage medium having a computer program stored thereon, which, when executed by a processor, implements the methods described in one or more embodiments of this specification.

[0068] The beneficial effects of the technical solutions provided in some embodiments of this specification include at least the following:

[0069] In one or more embodiments of this specification, the generation process of the target content is decoupled into two tasks, generated by a content model and a style model respectively, so that the target content can meet the dual requirements of content and style in the input information. During the training process, the style model is trained based on the content model. The trained style model can further stylize the target content based on the generated target content that meets the content requirements. Attached Figure Description

[0070] To more clearly illustrate the technical solutions in the embodiments of this specification, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0071] Figure 1 This is a global schematic diagram of the AIGC model in content generation application scenarios.

[0072] Figure 2 Generate user-oriented interactive diagrams for content application scenarios;

[0073] Figure 3 A schematic diagram of the scenario for model deployment;

[0074] Figure 4 A schematic flowchart illustrating a model training method provided in an embodiment of this specification;

[0075] Figure 5 A flowchart illustrating yet another model training method provided in the embodiments of this specification;

[0076] Figure 6 A flowchart illustrating yet another model training method provided in the embodiments of this specification;

[0077] Figure 7 A flowchart illustrating yet another model training method provided in the embodiments of this specification;

[0078] Figure 8 A flowchart illustrating a content generation method provided in an embodiment of this specification;

[0079] Figure 9 This is a schematic diagram of the structure of a model training device provided in the embodiments of this specification;

[0080] Figure 10 This is a schematic diagram of the structure of another model training device provided in the embodiments of this specification;

[0081] Figure 11 This is a schematic diagram of the structure of another model training device provided in the embodiments of this specification;

[0082] Figure 12 This is a schematic diagram of the structure of a content generation device provided in an embodiment of this specification;

[0083] Figure 13 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this specification. Detailed Implementation

[0084] The technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings.

[0085] The terms "first," "second," "third," etc., in the description, claims, and accompanying drawings are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to such processes, methods, products, or apparatus.

[0086] AIGC is a new artificial intelligence technology that uses AI models to automatically generate various types of text, images, audio, video, and other content based on given conditions such as topics, keywords, formats, and styles (inputted to the model in the form of AI prompts "Prompt").

[0087] Figure 1 The diagram illustrates an AIGC content generation application scenario. The AIGC model in the diagram can be loaded onto an AIGC platform or APP application on terminal devices (such as smartphones, tablets, desktops, laptops, ultra-mobile personal computers (UMPCs), handheld computers, PC devices, personal digital assistants (PDAs), virtual reality devices, etc.) for user use.

[0088] Users can choose according to their needs. Figure 2 Enter the AI ​​prompt word on the interface shown. The AIGC platform or APP application will intelligently process the received input information and generate content corresponding to the AI ​​prompt word. Figure 2 Taking image data as an example, when a user enters the text information "sunset at the seaside" on the input interface, the AIGC platform or APP application outputs the image information of "sunset at the seaside" on the interface based on the AI ​​prompt words. Figure 2 The purpose is to illustrate the interaction process in application scenarios, which is not limited to image data. That is, the generated content can be one or two or more types of data such as text, images, audio, and video (i.e., the original content, first predicted content, second predicted content, first target content, second target content, etc. mentioned in the embodiments of this specification).

[0089] See Figure 3 , Figure 3The diagram illustrates a scenario for deploying an AIGC model. This scenario includes a server 100 and terminal devices 200. The server 100 and the terminal devices 200 are communicatively linked. Multiple terminal devices 200 can be connected to the server 100. The server can store AIGC models, which can be single models or a group of models.

[0090] The server 100, on the one hand, can respond to instructions from the terminal device 200 and load the model onto the terminal device. For example, based on the user's actual operation, it is determined that a model needs to be retrieved from the server. If an AI prompt word is received, it is determined that a model needs to be retrieved from the server 100 and loaded onto the terminal device 200.

[0091] The server 100, on the other hand, can be used for model training. For example, the server 100 is equipped with a training platform that can train an initial model using a training set and a test set to obtain a target model. The trained model can be stored on the server or locally using a software development kit (SDK).

[0092] AIGC capabilities are provided by machine learning models, which are large models pre-trained on large amounts of data and are often referred to as foundation models. To balance content and style requirements, common AIGC image generation methods can be divided into two types. The first type is stylized content generation based on reference images. This method requires a reference image as a reference for the content, while the subsequent AIGC model focuses primarily on the stylization aspect. Using a reference image reduces the requirements on the AIGC foundation model, thus satisfying both content and style requirements. However, finding suitable reference images is challenging and has a high application threshold. The second type is image generation methods based on online training. This method uses several images of a specific style / content as training data to train the model, allowing it to "memorize" the specific style / content. After training, image generation can be performed based on the traditional "text-to-image" model. While it satisfies both style and content requirements, this method requires users to select images for online training, which is time-consuming and difficult to scale.

[0093] Based on this, this specification provides one or more embodiments to propose a model training scheme that can meet both content and style requirements, and a scheme for generating target content using the trained model, while avoiding the selection of reference images and online training of the model.

[0094] Please see Figure 4 , Figure 4 A schematic flowchart of a model training method provided in an embodiment of this specification is shown.

[0095] like Figure 4 As shown, the model training method may specifically include the following steps:

[0096] Step 102: Obtain training samples, which include original input information, original content corresponding to the original input information, and style annotation input information corresponding to the original input information;

[0097] Step 104: Use the content extraction module of the content model to extract content input information from the original input information;

[0098] Step 106: Input the content input information into the content base model of the content model, and output the first content;

[0099] Step 108: Based on the first content and the original input information, train the style model. The training process is as follows:

[0100] The style extraction module of the style model is used to extract style prediction input information from the original input information;

[0101] Input the style prediction input information and the first content into the style base model of the style model, and output the second predicted content;

[0102] The style prediction input information and the second prediction content are input into the style verification module of the style model, and the style type matching score is output.

[0103] The style prediction input and style annotation input are predicted and judged using the first loss function, the second loss function is predicted and judged using the second prediction content and the original content, and the style type matching score output by the style verification module is maximized using the third loss function. When all the above loss functions meet the convergence condition, the style model is trained. Otherwise, the above training process is repeated.

[0104] The model training method in the embodiments of this specification trains the style model based on the content model, so that the generated content meets both the content and style requirements of the input information.

[0105] The model training method in the embodiments of this specification is completed on the server, and the content model and the trained style model are stored on the server; alternatively, the training can be completed on a terminal device with super computing power, and the trained style model is stored on the terminal device. The content base model and style base model required during training can be obtained from the server.

[0106] The following examples and embodiments illustrate... Figure 4Each step in the process will be explained separately.

[0107] In step 102, training samples are constructed by collecting historical data. The historical data includes the original input information and the original content corresponding to the original input information. The original content can reflect the content and style requirements of the original input information.

[0108] The style annotation input information can be manually annotated using the original input information, with style as the target, resulting in style annotation input information that contains only style. Alternatively, the style annotation input information can be automatically labeled using a system based on certain standards, identifying fields that conform to the style, labeling them, and ultimately forming the style annotation input information.

[0109] In steps 104 and 106, the content model includes a content extraction module and a content base model. Both the content extraction module and the content base model are trained before the style model is trained. The content extraction module is mainly used to extract content input information that does not contain style from the original input information. For example, the original input information is "a landscape painting", where "landscape" is the content and "ink painting" is the style. After extraction by the content extraction module, the content input information is "a painting containing landscape". The content extraction module can use methods such as field recognition to extract fields from the input information. This module can be trained in conjunction with labeled data. The content base model is a diffusion model, which can be selected from a large number of open-source models and trained to generate a model that only meets the content requirements. For example, the content base model can be trained based on the original input information that only contains content and the original content that only presents the content.

[0110] In step 108, the style model includes a style extraction module, a style base model, and a style verification module. Training the style model mainly involves training the style extraction module and the style base model. The style extraction module is primarily used to extract style input information that does not contain content from the original input information. For example, the original input information is "a landscape painting," where "landscape" is the content and "ink painting" is the style. After extraction by the style extraction module, the style prediction input information is "ink painting." The style base model is a diffusion model, which can be selected from a large number of open-source models and trained to generate models that only meet the style requirements. Based on the style prediction input information and the first content generated using the aforementioned content base model, the style base model outputs a second predicted content. The second predicted content is the stylized content. Taking image data as an example, the first content is a painting containing landscape, and the second predicted content is an ink painting containing landscape.

[0111] For video data, refer to the example of image data. For audio data, frequency domain segmentation is used to generate first audio content with content requirements, and then second audio content with style requirements is generated based on the first audio content. Here, audio style mainly refers to a specific timbre, such as a loli voice.

[0112] During training, the style extraction module uses a first loss function to predict and judge the style prediction input information and the style annotation input information. The first loss function can be a similarity loss function, such as the KL loss function.

[0113] During training, the style base model uses a second loss function to predict and judge the consistency between the second predicted content and the original content. This second loss function can be a distance loss function, such as the L2 distance loss function.

[0114] Step 108, which involves inputting the style prediction input information and the second prediction content into the style verification module of the style model and outputting a style type matching score, includes:

[0115] Step A1: The style prediction input information is input into the style input information classification sub-model of the style verification module, and the style classification information is output; the second prediction content is input into the style content classification sub-model of the style verification module, and the content classification information is output.

[0116] Step A2: Using the matching submodule of the style verification module, calculate the style type matching score between the style classification information and the content classification information.

[0117] In step A1, the style input information classification sub-model and the style content classification sub-model remain unchanged during the style model training process. Both the style input information classification sub-model and the style content classification sub-model are pre-trained before being used. These two classification sub-models are trained using labeled original input information and original content of different styles. It is worth noting that the content classification information obtained by the style content classification sub-model refers to the classification of the second predicted content from a style perspective.

[0118] In step A2, the matching submodule is used to verify the degree of matching between the generated second predicted content and the style prediction input information in terms of style. The content classification information and the style classification information are each presented as classification probabilities. For example, if there are 10 styles, the classification results are presented as a probability distribution of length 10. Then, for the style with the highest probability value, the similarity value between the style classification information and the content classification information is calculated using the cosine similarity formula, thereby determining the style type matching score. In one example, the similarity value can be directly used as the style type matching score; in another example, the similarity value can be converted into a style type matching score based on a certain scoring system.

[0119] During style model training, the goal is to maximize the style type matching score. Simultaneously, the first and second loss functions are used to train both the style extraction module and the base style model. A higher style type matching score results in more accurate information output by both the base style model and the style extraction module.

[0120] Please see Figure 5 , Figure 5 A flowchart illustrating another model training method provided in the embodiments of this specification is shown.

[0121] like Figure 5 As shown, the model training method may specifically include the following steps:

[0122] Step 202: Obtain training samples, which include original input information, original content corresponding to the original input information, style annotation input information corresponding to the original input information, and content annotation input information corresponding to the original input information;

[0123] Step 204: Based on the original input information, train the content model. The training process is as follows:

[0124] The content extraction module of the content model is used to extract content prediction input information from the original input information.

[0125] The content prediction input information is input into the content base model of the content model, and the first predicted content is output.

[0126] The content matching module of the content model inputs the content prediction input information and the first predicted content inputs the content matching module to output the content matching score.

[0127] The content model is trained by using loss function one to predict and judge the content prediction input information and content annotation input information, loss function two to predict and judge the first predicted content and the original content, and loss function three to maximize the content matching score output by the content matching module. When all the above loss functions meet the convergence condition, the content model is trained. Otherwise, the above training process is repeated.

[0128] Step 206: Using the content extraction module of the trained content model, extract content input information from the original input information;

[0129] Step 208: Input the content input information into the content base model of the trained content model, and output the first content;

[0130] Step 210: Based on the first content and the original input information, train the style model. The training process is as follows:

[0131] The style extraction module of the style model is used to extract style prediction input information from the original input information;

[0132] Input the style prediction input information and the first content into the style base model of the style model, and output the second predicted content;

[0133] The style prediction input information and the second prediction content are input into the style verification module of the style model, and the style type matching score is output.

[0134] The style prediction input and style annotation input are predicted and judged using the first loss function, the second loss function is predicted and judged using the second prediction content and the original content, and the style type matching score output by the style verification module is maximized using the third loss function. When all the above loss functions meet the convergence condition, the style model is trained. Otherwise, the above training process is repeated.

[0135] The model training method in the embodiments of this specification trains a content model and a style model, wherein the style model is trained on the basis of the well-trained content model, so that the generated content meets the dual requirements of the input information in terms of both content and style.

[0136] The model training method in the embodiments of this specification is completed on the server, and the trained content model and style model are stored on the server; alternatively, training can be completed on a terminal device with super computing power, and the trained content model and style model are stored on the terminal device. The basic content model and style model required during training can be obtained from the server.

[0137] Figure 5 The example shown is the same as Figure 4 The difference in the examples shown is that, Figure 5 The training method shown in the example also includes the training process of the content model. Steps 202 and 206 to 210 are the same as steps 102 to 108, and will not be repeated here.

[0138] In step 204, the content model includes a content extraction module, a content base model, and a content matching module. Training the content model primarily involves training the content extraction module and the content base model. The content extraction module is mainly used to extract content input information that does not contain style from the original input information. For example, the original input information is "a landscape painting," where "landscape" is the content and "ink painting" is the style. After extraction by the content extraction module, the predicted content input information is "a painting containing landscape." The content base model is a diffusion model, which can be selected from a large number of open-source models and trained to generate models that only meet the content requirements. The content base model generates the first predicted content based on the predicted content input information. Taking image data as an example, the first predicted content is a painting containing landscape.

[0139] For video data, refer to the example of image data. For audio data, first predicted audio content with content requirements is generated by segmenting the frequency domain. Here, audio style mainly refers to a specific timbre, such as a "loli" voice.

[0140] During training, the content extraction module uses a loss function pair to predict and judge the content prediction input and the content annotation input. The content annotation input can be manually annotated based on the original input, focusing on the content itself to obtain content-only annotated input. Alternatively, the content annotation input can be automatically labeled based on certain standards, with the system identifying and labeling fields that match the content, ultimately forming the content annotation input. The loss function can be a similarity loss function, such as the KL loss function.

[0141] During training, the content-based model uses a second loss function to predict and judge the consistency between the first predicted content and the original content. This second loss function can be a distance loss function, such as the L2 distance loss function.

[0142] Step 204, which involves inputting the content prediction input information and the first predicted content into the content model's content matching module, and outputting a content matching score, includes:

[0143] B1, using the content segmentation submodule of the content matching module, the first predicted content is segmented into several first predicted sub-contents with the content object as the segmentation target;

[0144] B2 uses the matching submodule of the content matching module to calculate the content sub-matching score of each first predicted sub-content with the content prediction input information. Then, it sums all the content sub-matching scores and outputs the total content matching score.

[0145] In step B1, the content segmentation submodule employs a segmentation model to segment content objects. For example, if the first predicted content includes object A, object B, and object C, then the segmented first predicted sub-contents will be the first predicted sub-content containing object A, the first predicted sub-content containing object B, and the first predicted sub-content containing object C. Taking image data as an example, the objects can be people, landscapes, animals, etc. For video data examples, refer to the image data examples. For audio data as an example, the objects can be different types of musical instruments such as pianos, drum kits, and trumpets; or different vocal parts, etc. The segmentation model is pre-trained based on different types of data before being put into use.

[0146] In step B2, the matching submodule performs optimization calculations using existing matching algorithms (such as dynamic programming algorithms). For example, it matches and scores the first predicted sub-content containing object A and the content prediction input information to obtain a matching score 'a'; it matches and scores the first predicted sub-content containing object B and the content prediction input information to obtain a matching score 'b'; and it matches and scores the first predicted sub-content containing object C and the content prediction input information to obtain a matching score 'c'. The scores are then summed to obtain the total matching score.

[0147] During content model training, the goal is to maximize the content matching score. Simultaneously, loss functions one and two are used to train both the content extraction module and the basic content model. A higher content matching score results in more accurate output from both the basic content model and the content extraction module. Furthermore, the accuracy of the content generated by the basic content model also affects the training precision and convergence speed of subsequent style models.

[0148] Please see Figure 6 , Figure 6 A flowchart illustrating another model training method provided in the embodiments of this specification is shown.

[0149] like Figure 6 As shown, the model training method may specifically include the following steps:

[0150] Step 302: Obtain training samples, which include original input information, original content corresponding to the original input information, content-annotated input information corresponding to the original input information, and style-annotated input information corresponding to the original input information;

[0151] Step 304: Based on the original input information, train the content model. The training process is as follows:

[0152] The content extraction module of the content model is used to extract content prediction input information from the original input information.

[0153] The content prediction input information is input into the content base model of the content model, and the first predicted content is output; during this process, the content prediction input information is randomly replaced with the original input information;

[0154] The content matching module of the content model inputs the content prediction input information and the first predicted content inputs the content matching module to output the content matching score.

[0155] The content model is trained by using loss function one to predict and judge the content prediction input information and content annotation input information, loss function two to predict and judge the first predicted content and the original content, and loss function three to maximize the content matching score output by the content matching module. When all the above loss functions meet the convergence condition, the content model is trained. Otherwise, the above training process is repeated.

[0156] Step 306: Using the content extraction module of the trained content model, extract content input information from the original input information;

[0157] Step 308: Input the content input information into the content base model of the trained content model above, and output the first content;

[0158] Step 310: Based on the first content and the original input information, train the style model. The training process is as follows:

[0159] The style extraction module of the style model is used to extract style prediction input information from the original input information;

[0160] The style prediction input information and the first content are input into the style base model of the style model, and the second predicted content is output; during this process, the style prediction input information is randomly replaced with the original input information.

[0161] The style prediction input information and the second prediction content are input into the style verification module of the style model, and the style type matching score is output.

[0162] The style prediction input and style annotation input are predicted and judged using the first loss function, the second loss function is predicted and judged using the second prediction content and the original content, and the style type matching score output by the style verification module is maximized using the third loss function. When all the above loss functions meet the convergence condition, the style model is trained. Otherwise, the above training process is repeated.

[0163] Figure 6 The example shown is the same as Figure 5 The difference in the examples shown is that, Figure 6The training method in the example introduces random perturbation, meaning that during training, the style prediction input and / or content prediction input are randomly replaced with the original input. Other processes are the same as... Figure 5 The examples are the same, so I will not repeat them here.

[0164] The method described in this specification replaces the predicted content input or style prediction input during the training process with the original input information with a certain probability, thereby enhancing the robustness of the base model to predicted input information. This training method concatenates the content model and style model for joint optimization, improving the overall performance.

[0165] Please see Figure 7 , Figure 7 A flowchart illustrating another model training method provided in the embodiments of this specification is shown.

[0166] like Figure 7 As shown, the model training method may specifically include the following steps:

[0167] Step 402: Obtain training samples, which include original input information, original content corresponding to the original input information, content-annotated input information corresponding to the original input information, and style-annotated input information corresponding to the original input information;

[0168] Step 404: Based on the training samples, train the content model and style model separately, and output the trained content model and style model;

[0169] Step 406: Based on the training samples, jointly train the content model and style model trained in step 404.

[0170] Step 402 refers to step 102. Step 404 refers to steps 204 to 210. Step 406 refers to steps 304 to 310, and will not be described in detail here.

[0171] Figure 7 The example shown is the same as Figure 6 The difference in the examples shown is that, Figure 6 The example introduces a perturbation mechanism that randomly replaces input information during a training process, while... Figure 7 The example consists of two training processes: step 404 performs unperturbed training, and then step 406 performs perturbed training.

[0172] The method described in this specification replaces the predicted content input or style prediction input during the training process with the original input information with a certain probability, thereby enhancing the robustness of the base model to predicted input information. This training method concatenates the content model and style model for joint optimization, improving the overall performance.

[0173] Please see Figure 8 , Figure 8 A flowchart illustrating the content generation method provided in the embodiments of this specification is shown.

[0174] like Figure 8 The content generation method in the embodiments of this specification is based on the foregoing Figure 5 , Figure 6 , Figure 7 The example model training method yields the content model and style model implementations, including:

[0175] Step 502: Input the input information into the content extraction module in the content model and output the content input information;

[0176] Step 504: Input the input information into the content base model in the content model, and output the first target content;

[0177] Step 506: Input the input information into the style extraction module in the style model and output the style input information;

[0178] Step 508: Input the style input information and the first target content into the style base model in the style model, and output the second target content.

[0179] The content generation method in the embodiments of this specification generates target content based on a trained content model and style model, so that the generated content meets both the content and style requirements of the input information.

[0180] The content generation method described in this specification is deployed on a terminal device. When this method is executed, a content model and a style model can be obtained from a server and then loaded and processed on the terminal device. When the terminal device has powerful computing capabilities, the content model and style model are stored on the terminal device, and when this method is executed, the models stored on the terminal device are directly loaded and processed.

[0181] The content generation method described in this specification can generate second target content that meets both content and style requirements.

[0182] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0183] Please see Figure 9The diagram shown is a structural schematic of a model training device according to an embodiment of this specification.

[0184] like Figure 9 As shown, the model training device 1000 may include at least a sample acquisition unit 1001, a content model unit 1002, and a style model training unit 1003, wherein:

[0185] The sample acquisition unit 1001 is used to acquire training samples, which include original input information, original content corresponding to the original input information, and style annotation input information corresponding to the original input information.

[0186] Training samples are constructed by collecting historical data, which includes original input information and original content corresponding to the original input information. The original content is original content that can reflect the content and style requirements of the original input information.

[0187] The content model unit 1002 is used to extract content input information from the original input information using the content extraction module of the content model; it is also used to input the content input information into the content base model of the content model and output the first content.

[0188] The content model includes a content extraction module and a content base model. Both the content extraction module and the content base model are trained before the style model is trained. The content extraction module is mainly used to extract content input information that does not contain style from the original input information.

[0189] The content extraction module can extract fields from the input information using methods such as field recognition. This module can be trained in conjunction with labeled data. The basic content model is a diffusion model, which can be selected from a large number of open-source models and trained to generate models that only meet the content requirements.

[0190] The style model training unit 1003 is used to train the style model based on the first content and the original input information. The training process is as follows:

[0191] The style extraction module of the style model is used to extract style prediction input information from the original input information;

[0192] Input the style prediction input information and the first content into the style base model of the style model, and output the second predicted content;

[0193] The style prediction input information and the second prediction content are input into the style verification module of the style model, and the style type matching score is output.

[0194] The style prediction input and style annotation input are predicted and judged using the first loss function, the second loss function is predicted and judged using the second prediction content and the original content, and the style type matching score output by the style verification module is maximized using the third loss function. When all the above loss functions meet the convergence condition, the style model is trained. Otherwise, the above training process is repeated.

[0195] The training of the style model mainly includes the training of a style extraction module and a style base model. The style extraction module is primarily used to extract style input information that does not contain content from the original input information. The style base model is a diffusion model, which can be selected from a large number of open-source models and trained to generate only models that meet the style requirements. Based on the style prediction input information and the first content generated using the aforementioned content base model, the style base model outputs a second predicted content. The second predicted content is the stylized content.

[0196] The model training apparatus of this specification trains the style model based on the content model, so that the generated content meets both the content and style requirements of the input information.

[0197] During training, the style extraction module uses a first loss function to predict and judge the style prediction input and style annotation input. The style annotation input can be manually annotated with the original input, focusing on style, resulting in style-specific annotation input. Alternatively, the style annotation input can be automatically labeled based on certain standards, identifying and labeling "fields" that match the style, ultimately forming the style annotation input. The first loss function can be a similarity loss function, such as the KL loss function.

[0198] During training, the style base model uses a second loss function to predict and judge the consistency between the second predicted content and the original content. This second loss function can be a distance loss function, such as the L2 distance loss function.

[0199] The style model training unit, which inputs the style prediction input information and the second prediction content into the style model's style verification module and outputs a style type matching score, includes:

[0200] Step A1: The style prediction input information is input into the style input information classification sub-model of the style verification module, and the style classification information is output; the second prediction content is input into the style content classification sub-model of the style verification module, and the content classification information is output.

[0201] Step A2: Using the matching submodule of the style verification module, calculate the style type matching score between the style classification information and the content classification information.

[0202] During style model training, the goal is to maximize the style type matching score. Simultaneously, the first and second loss functions are used to train both the style extraction module and the base style model. A higher style type matching score results in more accurate information output by both the base style model and the style extraction module.

[0203] Please see Figure 10 The diagram shows a structural schematic of another model training device according to an embodiment of this specification.

[0204] like Figure 10 As shown, the model training device 2000 may include at least a sample acquisition unit 2001, a content model training unit 2002, a content model unit 2003, and a style model training unit 2004, wherein:

[0205] The sample acquisition unit 2001 is used to acquire training samples, which include original input information, original content corresponding to the original input information, content annotation input information corresponding to the original input information, and style annotation input information corresponding to the original input information.

[0206] The content model training unit 2002 is used to train the content model based on the original input information before training the style model using the first content obtained from the content model. The training process is as follows:

[0207] The content extraction module of the content model is used to extract content prediction input information from the original input information.

[0208] The content prediction input information is input into the content base model of the content model, and the first predicted content is output.

[0209] The content matching module of the content model inputs the content prediction input information and the first predicted content inputs the content matching module to output the content matching score.

[0210] The content model is trained by using loss function one to predict and judge the content prediction input information and content annotation input information, loss function two to predict and judge the first predicted content and the original content, and loss function three to maximize the content matching score output by the content matching module. When all the above loss functions meet the convergence condition, the content model is trained and the trained content model is sent to the content model unit. Otherwise, the above training process is repeated.

[0211] The content model unit 2003 is used to extract content input information from the original input information using the content extraction module of the content model; it is also used to input the content input information into the content base model of the content model and output the first content.

[0212] Style model training unit 2004 is used to train the style model based on the first content and the original input information. The training process is as follows:

[0213] The style extraction module of the style model is used to extract style prediction input information from the original input information;

[0214] Input the style prediction input information and the first content into the style base model of the style model, and output the second predicted content;

[0215] The style prediction input information and the second prediction content are input into the style verification module of the style model, and the style type matching score is output.

[0216] The style prediction input and style annotation input are predicted and judged using the first loss function, the second loss function is predicted and judged using the second prediction content and the original content, and the style type matching score output by the style verification module is maximized using the third loss function. When all the above loss functions meet the convergence condition, the style model is trained. Otherwise, the above training process is repeated.

[0217] The model training apparatus of this specification trains a content model and a style model, wherein the style model is trained based on the content model, so that the generated content meets both the content and style requirements of the input information.

[0218] Figure 10 The example shown is the same as Figure 9 The difference in the examples shown is that, Figure 10 The training apparatus shown in the example also includes a content model training unit. Other identical units will not be described further here.

[0219] The content model includes a content extraction module, a content base model, and a content matching module. Training the content model primarily involves training the content extraction module and the content base model. The content extraction module is mainly used to extract content input information, excluding style-related elements, from the raw input information. The content base model is a diffusion model, which can be selected from a large number of open-source models and trained to generate models that only meet the content requirements. The content base model generates the first predicted content based on the content prediction input information.

[0220] During training, the content extraction module uses a loss function pair to predict and judge the content prediction input and the content annotation input. The content annotation input can be manually annotated based on the original input, focusing on the content itself to obtain content-only annotated input. Alternatively, the content annotation input can be automatically labeled based on certain standards, with the system identifying and labeling fields that match the content, ultimately forming the content annotation input. The loss function can be a similarity loss function, such as the KL loss function.

[0221] During training, the content-based model uses a second loss function to predict and judge the consistency between the first predicted content and the original content. This second loss function can be a distance loss function, such as the L2 distance loss function.

[0222] The content matching module in the content model training unit, which inputs the content prediction input information and the first predicted content into the content model and outputs a content matching score, includes:

[0223] B1, using the content segmentation submodule of the content matching module, the first predicted content is segmented into several first predicted sub-contents with the content object as the segmentation target;

[0224] B2 uses the matching submodule of the content matching module to calculate the content sub-matching score of each first predicted sub-content with the content prediction input information. Then, it sums all the content sub-matching scores and outputs the total content matching score.

[0225] During content model training, the goal is to maximize the content matching score. Simultaneously, loss functions one and two are used to train both the content extraction module and the basic content model. A higher content matching score results in more accurate output from both the basic content model and the content extraction module. Furthermore, the accuracy of the content generated by the basic content model also affects the training precision and convergence speed of subsequent style models.

[0226] Please see Figure 11 The diagram shows a structural schematic of another model training device according to an embodiment of this specification.

[0227] like Figure 11 As shown, the model training device 3000 may include at least a sample acquisition unit 3001, a content model training unit 3002, a content model unit 3003, a style model training unit 3004, and a random replacement unit 3005.

[0228] Figure 11 Examples and Figure 10 The same units will not be described again. Next, the random replacement unit 3005 will be explained.

[0229] The random replacement unit 3005 is used to randomly replace the style prediction input information and / or content prediction input information during the training process of the content model and the style model with the original input information. Alternatively,

[0230] The random replacement unit 3005 is used to jointly train the content model and the style model after the content model and the style model have been trained separately. During the training process, the style prediction input information and / or content prediction input information will be randomly replaced with the original input information. The training process of the content model and the style model in the joint training is the same as their individual training process.

[0231] The former implementation method can be referred to Figure 6 The model training method shown in the example is executed. For a more detailed implementation, please refer to... Figure 7 The example shown demonstrates the execution of the model training method.

[0232] Please see Figure 12 , Figure 12 A schematic diagram of the structure of a content generation apparatus provided in an embodiment of this specification is shown.

[0233] like Figure 12 The content generation apparatus 4000 in this embodiment includes at least: a content input information extraction unit 4001, a first target content generation unit 4002, a style input information extraction unit 4003, and a second target content generation unit 4004, wherein:

[0234] The content input information extraction unit is used to input the input information into the content extraction module in the content model and output the content input information;

[0235] The first target content generation unit is used to input the input information into the content base model in the content model and output the first target content;

[0236] The style input information extraction unit is used to input the input information into the style extraction module in the style model and output the style input information.

[0237] The second target content generation unit is used to input style input information and the first target content into the style base model in the style model, and output the second target content.

[0238] The content model and style model used in the content generation apparatus of this specification embodiments are based on Figure 5 , Figure 6 , Figure 7 The example model was trained using the given training method.

[0239] The content generation apparatus of this specification generates target content based on a trained content model and style model, so that the generated second target content meets both the content and style requirements of the input information.

[0240] Please see Figure 13 The diagram shown is a structural schematic of an electronic device provided in an embodiment of this specification.

[0241] like Figure 13 As shown, the electronic device 1100 may include: at least one processor 1101, at least one network interface 1104, user interface 1103, memory 1105, and at least one communication bus 1102.

[0242] The communication bus 1102 can be used to realize the connection and communication of the above components.

[0243] The user interface 1103 may include buttons, and the optional user interface may also include a standard wired interface or a wireless interface.

[0244] The network interface 1104 may include, but is not limited to, Bluetooth modules, NFC modules, Wi-Fi modules, etc.

[0245] The processor 1101 may include one or more processing cores. The processor 1101 connects to various parts within the electronic device 1100 using various interfaces and lines. It executes instructions, programs, code sets, or instruction sets stored in the memory 1105, and calls data stored in the memory 1105 to perform various functions and process data within the routing device 1100. Optionally, the processor 1101 may be implemented using at least one hardware form of DSP, FPGA, or PLA. The processor 1101 may integrate one or more of the following: CPU, GPU, and modem. The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the content required for display; and the modem handles wireless communication. It is understood that the modem may also not be integrated into the processor 1101 and may be implemented as a separate chip.

[0246] The memory 1105 may include RAM or ROM. Optionally, the memory 1105 may include a non-transitory computer-readable medium. The memory 1105 may be used to store instructions, programs, code, code sets, or instruction sets. The memory 1105 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as touch function, sound playback function, image playback function, etc.), instructions for implementing the above-described method embodiments, etc.; the data storage area may store data involved in the above-described method embodiments, etc. Optionally, the memory 1105 may also be at least one storage device located remotely from the aforementioned processor 1101. As a computer storage medium, the memory 1105 may include an operating system, a network communication module, a user interface module, and application programs. The processor 1101 may be used to call the application programs stored in the memory 1105 and execute the methods in one or more of the above embodiments.

[0247] This specification also provides a computer-readable storage medium storing instructions that, when executed on a computer or processor, cause the computer or processor to perform the above-described instructions. Figure 4 or Figure 5 or Figure 6 or Figure 7 One or more steps in the illustrated embodiment. If the constituent modules of the above-described electronic device are implemented as software functional units and sold or used as independent products, they can be stored in the computer-readable storage medium.

[0248] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this specification are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in or transmitted through a computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, Digital Subscriber Line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., Digital Versatile Discs (DVDs)), or semiconductor media (e.g., Solid State Disks (SSDs)).

[0249] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the methods described above. The aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks. Unless otherwise specified, the technical features of this embodiment and its implementation can be combined arbitrarily.

[0250] The embodiments described above are merely preferred embodiments of this specification and are not intended to limit the scope of this specification. Any modifications and improvements made by those skilled in the art to the technical solutions of this specification without departing from the spirit of this specification should fall within the protection scope defined by the claims of this specification.

Claims

1. Model training methods, including: Obtain training samples, which include original input information, original content corresponding to the original input information, style annotation input information corresponding to the original input information, and content annotation input information corresponding to the original input information; The original input information is text information; the style annotation input information is information that annotates the original input information with style as the target; and the content annotation input information is information that annotates the original input information with content as the target. The content model is trained based on the original input information. The training process is as follows: The content extraction module of the content model is used to extract content prediction input information from the original input information. The content prediction input information is input into the content base model of the content model, and the first predicted content is output. The content matching module of the content model inputs the content prediction input information and the first predicted content inputs the content matching module to output the content matching score. The content model is trained by using loss function one to predict and judge the content prediction input information and content annotation input information, loss function two to predict and judge the first predicted content and the original content, and loss function three to maximize the content matching score output by the content matching module. When all the above loss functions meet the convergence condition, the content model is trained. Otherwise, the above training process is repeated. The content extraction module of the content model is used to extract content input information from the original input information. Input the aforementioned content information into the content base model of the content model, and output the first content; Based on the first content and the original input information, the style model is trained. The training process is as follows: The style extraction module of the style model is used to extract style prediction input information from the original input information; Input the style prediction input information and the first content into the style base model of the style model, and output the second predicted content; The style prediction input information and the second prediction content are input into the style verification module of the style model, and the style type matching score is output. The style prediction input and style annotation input are predicted and judged using the first loss function, the second loss function is predicted and judged using the second prediction content and the original content, and the style type matching score output by the style verification module is maximized using the third loss function. When all the above loss functions meet the convergence condition, the style model is trained. Otherwise, the above training process is repeated.

2. The method according to claim 1, further comprising the following steps during the training of the content model and the style model: The style prediction input and / or content prediction input during the training process are randomly replaced with the original input.

3. The method according to claim 1, further comprising, after the content model and the style model have been trained separately: The content model and style model are jointly trained, and the style prediction input information and / or content prediction input information during the training process will be randomly replaced with the original input information; The training process for the content model and style model in the joint training is the same as their individual training process.

4. The method according to claim 1, 2, or 3, wherein the step of inputting the style prediction input information and the second prediction content into the style verification module of the style model and outputting a style type matching score includes: The style prediction input information is input into the style input information classification sub-model of the style verification module, and the output is style classification information; the second prediction content is input into the style content classification sub-model of the style verification module, and the output is content classification information. The matching submodule of the style validation module is used to calculate the style type matching score between style classification information and content classification information.

5. The method according to claim 1, 2, or 3, wherein the step of inputting the content prediction input information and the first predicted content into the content matching module of the content model, and outputting a content matching score, includes: Using the content segmentation submodule of the content matching module, the first predicted content is divided into several first predicted sub-contents with the content object as the segmentation target; Using the matching submodule of the content matching module, calculate the content sub-matching score of each first predicted sub-content with the content prediction input information, then sum all the content sub-matching scores and output the total content matching score.

6. A content generation method, implemented based on a content model and a style model trained according to claim 1, 2, or 3; the content generation method includes: Input the input information into the content extraction module of the content model, and output the content input information; Input the input information into the content base model of the content model, and output the first target content; Input the input information into the style extraction module in the style model, and output the style input information; Input the style input information and the first target content into the style base model in the style model, and output the second target content.

7. A model training device, comprising: The sample acquisition unit is used to acquire training samples, which include original input information, original content corresponding to the original input information, style annotation input information corresponding to the original input information, and content annotation input information corresponding to the original input information. The original input information is text information; the style annotation input information is information that annotates the original input information with style as the target; and the content annotation input information is information that annotates the original input information with content as the target. The content model training unit is used to train the content model based on the original input information before training the style model using the first content obtained from the content model. The training process is as follows: The content extraction module of the content model is used to extract content prediction input information from the original input information. The content prediction input information is input into the content base model of the content model, and the first predicted content is output. The content matching module of the content model inputs the content prediction input information and the first predicted content inputs the content matching module to output the content matching score. The content model is trained by using loss function one to predict and judge the content prediction input information and content annotation input information, loss function two to predict and judge the first predicted content and the original content, and loss function three to maximize the content matching score output by the content matching module. When all the above loss functions meet the convergence condition, the content model is trained and the trained content model is sent to the content model unit. Otherwise, the above training process is repeated. The content model unit is used to extract content input information from the original input information using the content extraction module of the content model. It is also used to input the content input information into the content base model of the content model and output the first content; The style model training unit is used to train the style model based on the first content and the original input information. The training process is as follows: The style extraction module of the style model is used to extract style prediction input information from the original input information; Input the style prediction input information and the first content into the style base model of the style model, and output the second predicted content; The style prediction input information and the second prediction content are input into the style verification module of the style model, and the style type matching score is output. The style prediction input and style annotation input are predicted and judged using the first loss function, the second loss function is predicted and judged using the second prediction content and the original content, and the style type matching score output by the style verification module is maximized using the third loss function. When all the above loss functions meet the convergence condition, the style model is trained. Otherwise, the above training process is repeated.

8. The apparatus according to claim 7, further comprising: The random replacement unit is used to randomly replace the style prediction input information and / or content prediction input information during the training process of the content model and the style model with the original input information.

9. The apparatus according to claim 7, further comprising: The random replacement unit is used to jointly train the content model and the style model after the content model and the style model have been trained separately. During the training process, the style prediction input information and / or content prediction input information will be randomly replaced with the original input information. The training process for the content model and style model in the joint training is the same as their individual training process.

10. A content generation apparatus, implemented based on a content model and a style model trained according to the method of claim 1, 2, or 3; the apparatus comprising: The content input information extraction unit is used to input the input information into the content extraction module in the content model and output the content input information. The first target content generation unit is used to input the input information into the content base model in the content model and output the first target content. The style input information extraction unit is used to input the input information into the style extraction module in the style model and output the style input information. The second target content generation unit is used to input the style input information and the first target content into the style base model in the style model, and output the second target content.

11. Electronic devices, including processors and memory; The processor is connected to the memory; The memory is used to store executable program code; The processor runs a program corresponding to the executable program code stored in the memory to perform the method as described in any one of claims 1-6.

12. A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method as claimed in any one of claims 1-6.

Images