Model training and content generation methods and apparatuses

Abstract

The present disclosure provides a model training and content generation method and device, relating to the fields of artificial intelligence such as large models, deep learning, natural language processing and knowledge graphs. The AIGC model training method comprises: constructing a first training set, which comprises a first training sample, the first training sample comprising: a first training picture corresponding to any entity and picture description information of the first training picture; training the entity recognition capability of the first content generation model using the first training set to obtain a second content generation model; constructing a second training set, which comprises a second training sample, the second training sample comprising: modification relationship information between words in a target prompt word, and a second training picture matched with the target prompt word; training the entity detail description capability of the second content generation model using the second training set to obtain a third content generation model. The application of the scheme disclosed in the present disclosure can improve the instruction following capability of the content generation model.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, and in particular to methods and apparatus for model training and content generation in the fields of large models, deep learning, natural language processing and knowledge graphs. Background Technology

[0002] For cross-modal AIGC (Artificial Intelligence Generated Content) models, it is necessary to improve their instruction-following or prompt-following capabilities. Cross-modal refers to text-to-image or text-to-video generation, etc. Instruction-following capability refers to the ability to strictly follow given prompts to generate content. Summary of the Invention

[0003] This disclosure provides methods and apparatus for model training and content generation.

[0004] A content generation model training method, comprising:

[0005] Construct a first training set, which includes a first training sample, and the first training sample includes: a first training image corresponding to any entity and image description information of the first training image;

[0006] The entity recognition ability of the first content generation model is trained using the first training set to obtain the second content generation model;

[0007] Construct a second training set, which includes a second training sample. The second training sample includes: modification relationship information between words in the target prompt word, and a second training image that matches the target prompt word.

[0008] Using the second training set, the entity detail representation capability of the second content generation model is trained to obtain the third content generation model.

[0009] A content generation method, comprising:

[0010] Obtain the input prompt words to be processed, and obtain the modification relationship information between words in the prompt words to be processed;

[0011] The modification relationship information is used as input to the target model to obtain the content generation result corresponding to the prompt word to be processed. The target model is the third content generation model trained using the above method.

[0012] A method for training a modification relation discrimination model includes:

[0013] Construct a third training set, which includes a third training sample, and the third training sample includes: sample prompt words;

[0014] The pre-trained large language model is subjected to supervised fine-tuning training using the third training set to obtain the modification relation discrimination model. The modification relation discrimination model is used to construct a second training set, which includes a second training sample. The second training sample includes: modification relation information between words in the target prompt word determined by the modification relation discrimination model, and a second training image matching the target prompt word. The second training set is used to train the entity detail characterization ability of the second content generation model to obtain a third content generation model. The second content generation model is obtained by training the entity recognition ability of the first content generation model using the first training set. The first training set includes a first training sample, which includes: a first training image corresponding to any entity and image description information of the first training image. The modification relation discrimination model is also used to determine the modification relation information of the input prompt word when using the third content generation model for content generation.

[0015] A content generation model training device includes: a first training module and a second training module;

[0016] The first training module is used to construct a first training set, which includes a first training sample. The first training sample includes: a first training image corresponding to any entity and image description information of the first training image; and to train the entity recognition ability of the first content generation model using the first training set to obtain a second content generation model.

[0017] The second training module is used to construct a second training set, which includes a second training sample. The second training sample includes: modification relationship information between words in the target prompt word, and a second training image that matches the target prompt word. Using the second training set, the entity detail characterization ability of the second content generation model is trained to obtain a third content generation model.

[0018] A content generation apparatus includes: an acquisition module and a generation module;

[0019] The acquisition module is used to acquire the input prompt words to be processed, and to acquire the modification relationship information between words in the prompt words to be processed;

[0020] The generation module is used to take the modification relationship information as input to the target model to obtain the content generation result corresponding to the prompt word to be processed. The target model is the third content generation model trained using the above-mentioned device.

[0021] A training device for a modification relation discrimination model includes: a construction module and a training module;

[0022] The construction module is used to construct a third training set, which includes a third training sample, and the third training sample includes: sample prompt words;

[0023] The training module is used to perform supervised fine-tuning training on the pre-trained large language model using the third training set to obtain the modification relation discrimination model. The modification relation discrimination model is used to construct a second training set, which includes a second training sample. The second training sample includes: modification relation information between words in the target prompt word determined by the modification relation discrimination model, and a second training image matching the target prompt word. The second training set is used to train the entity detail characterization ability of the second content generation model to obtain a third content generation model. The second content generation model is obtained by training the entity recognition ability of the first content generation model using the first training set. The first training set includes a first training sample, which includes: a first training image corresponding to any entity and image description information of the first training image. The modification relation discrimination model is also used to determine the modification relation information of the input prompt word when using the third content generation model for content generation.

[0024] An electronic device, comprising:

[0025] At least one processor; and

[0026] A memory communicatively connected to the at least one processor; wherein,

[0027] The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method described above.

[0028] A non-transitory computer-readable storage medium storing computer instructions for causing a computer to perform the methods described above.

[0029] A computer program product includes a computer program / instructions that, when executed by a processor, implement the method described above.

[0030] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0031] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0032] Figure 1 This is a flowchart of an embodiment of the AIGC model training method described in this disclosure;

[0033] Figure 2 A flowchart illustrating an embodiment of the method for generating the content described in this disclosure;

[0034] Figure 3 This is a flowchart of an embodiment of the modification relation discrimination model training method described in this disclosure;

[0035] Figure 4 This is a schematic diagram illustrating the overall implementation process of model training and prediction as described in this disclosure;

[0036] Figure 5 This is a schematic diagram of the composition structure of Embodiment 500 of the AIGC model training device described in this disclosure;

[0037] Figure 6 This is a schematic diagram of the composition structure of Embodiment 600 of the content generation apparatus described in this disclosure;

[0038] Figure 7 This is a schematic diagram of the composition structure of Embodiment 700 of the modification relationship discrimination model training device described in this disclosure;

[0039] Figure 8 A schematic block diagram of an electronic device 800 that can be used to implement embodiments of the present disclosure is shown. Detailed Implementation

[0040] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0041] Furthermore, it should be understood that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0042] Figure 1 This is a flowchart illustrating an embodiment of the AIGC model training method described in this disclosure. Figure 1 As shown, the specific implementation methods are as follows.

[0043] In step 101, a first training set is constructed, which includes a first training sample. The first training sample includes: a first training image corresponding to any entity and image description information of the first training image.

[0044] In step 102, the entity recognition ability of the first AIGC model is trained using the first training set to obtain the second AIGC model.

[0045] In step 103, a second training set is constructed, which includes a second training sample. The second training sample includes: modification relationship information between words in the target prompt word, and a second training image that matches the target prompt word.

[0046] In step 104, the entity detail representation capability of the second AIGC model is trained using the second training set to obtain the third AIGC model.

[0047] By adopting the scheme described in the above method embodiments, the instruction compliance ability of the AIGC model can be visualized as entity cognition ability and entity detail depiction ability, and corresponding training sets can be constructed respectively. Then, the AIGC model can be trained using the constructed training sets, thereby achieving targeted optimization of instruction compliance ability (specific effect improvement), that is, improving the instruction compliance ability of the AIGC model, and thus improving the accuracy of content generated based on AIGC.

[0048] For ease of distinction, the AIGC models trained at different stages of the training process in the scheme described in this disclosure are referred to as the first AIGC model, the second AIGC model, and the third AIGC model, respectively.

[0049] The AIGC model is typically a cross-modal model such as a text-to-image model or a text-to-video model. The following explanation uses the text-to-image model as an example.

[0050] Among them, entity recognition ability refers to the ability of AIGC model to correctly recognize entities. For example, if a certain person needs to be drawn, if the model cannot even recognize the entity (the person) in the prompt words entered by the user, that is, it cannot even draw the main body correctly, then it must be a case of not following the user's instructions, resulting in a very poor user experience.

[0051] The ability to depict the details of an entity, as the name suggests, refers to the AIGC model's ability to depict the specific details of an entity, which can be expressed as the ability to understand the modifying relationships between words that describe an entity.

[0052] In some embodiments of this disclosure, the modification relationship may include: attribute binding, referential relationship information, sentence-free context, and spatial position relationship. These will be described in detail below.

[0053] 1) Property binding

[0054] For example, the prompt is: There is a red ball on a blue table, and a beautiful woman with big eyes is sitting in front of the table.

[0055] Attribute binding refers to the binding of adjectives to entities, such as:

[0056] Blue -> Decoration -> Table, Blue is bound to the table;

[0057] Red -> Decoration -> Ball, red is bound to the ball;

[0058] Big eyes -> Makeup -> Beautiful woman; big eyes are now associated with beautiful woman.

[0059] 2) Referential relationship

[0060] For example, the prompt is: "A graceful and elegant empress, holding a violin in one hand and a book in the other..."

[0061] Referential relationships can include: third-person referencing and specific body part referencing, such as:

[0062] She became the Empress;

[0063] violin -> one hand;

[0064] Book -> other hand.

[0065] 3) No sentence structure or context

[0066] For example, the prompt phrase is: "A noble princess Ragdoll cat, elegant and luxurious, fluffy, golden-haired and blue-eyed, foraging for food, sitting on a magnificent throne."

[0067] The absence of sentence structure or context means that it is impossible to determine which adjective modifies which entity through sentence structure. For example, in the above prompt, there are two entities, "princess" and "ragdoll cat". One entity is a person and the other is a cat, but the following adjectives do not clearly indicate which entity they modify.

[0068] But in reality, common sense tells us:

[0069] Elegant and luxurious -> likely to be modified -> princess;

[0070] Fluffy -> Likely to be modified -> Ragdoll cat;

[0071] Blonde hair and blue eyes -> likely to be modified -> princess;

[0072] Foraging -> Likely to be modified -> Ragdoll cat.

[0073] 4) Spatial relationship

[0074] For example, the prompt is: "Mountains in ink wash style, rivers at the foot of the mountains, farmland beside the rivers, large tractors in the farmland, and a drone parked to the right of the tractors."

[0075] Spatial positional relationship refers to the relationship that exists in spatial location, such as:

[0076] There is a river at the foot of the mountain;

[0077] River -> Beside it -> Farmland;

[0078] Farmland -> contains -> tractors;

[0079] Tractor -> There's a drone on the right.

[0080] These spatial relationships must be correct; otherwise, the generated image will show obvious signs of not following instructions.

[0081] Based on the above-described entity cognition ability and entity detail depiction ability, training sets can be constructed in a targeted manner, and the constructed training sets can be used to train the AIGC model to improve the AIGC model's instruction compliance ability.

[0082] As described in step 101, a first training set can be constructed first. The first training set includes a first training sample, which may include: a first training image corresponding to any entity and image description information of the first training image. Accordingly, in step 102, the entity recognition ability of the first AIGC model can be trained using the first training set.

[0083] In some embodiments of this disclosure, in order to construct a first training set, an entity recognition graph can be constructed first, and the following processing can be performed on all or part of the entities in the entity recognition graph: obtain the entity image corresponding to the entity, the first training image includes the entity image, and obtain the image description information of the first training image, and then generate a first training sample based on the first training image and the image description information.

[0084] Using the above processing method, an entity cognitive graph can be constructed first. Then, for all or part of the entities in the entity cognitive graph, corresponding first training samples can be constructed. Thus, the entity cognitive graph provides a data foundation for the construction of the first training samples, thereby improving the efficiency of sample construction.

[0085] In some embodiments of this disclosure, when constructing an entity cognitive graph, a first entity with a higher than preset first popularity can be obtained from a set of historical prompt words recorded in a predetermined business log, and an entity cognitive graph can be constructed based on at least the first entity. Alternatively, a second entity with a higher than preset second popularity can be obtained from other predetermined data sources outside the set of historical prompt words, and an entity cognitive graph can be constructed based on the first and second entities. The specific values ​​of the first and second popularity can be determined according to actual needs.

[0086] For example, the scheduled business log could be the Wenxin Yige business log, which can obtain a complete set of historical prompt words input by the user. Then, based on an entity linking tool, entities from each historical prompt word can be linked to a knowledge base. This allows for the calculation of the cumulative occurrence count of different entities, and entities with a cumulative occurrence count greater than the highest popularity can be designated as the primary entity. Furthermore, the required entity cognitive graph can be constructed based on the primary entity.

[0087] In addition to obtaining the first entity from the historical cue word set, the second entity can also be obtained from other predetermined data sources. Then, the required entity cognitive graph can be constructed based on the first and second entities. For example, other predetermined data sources could be encyclopedic data sources.

[0088] The first entities obtained from the historical prompt word set are all entities that have appeared in the prompt words entered by the user. However, if some entities have not appeared in the prompt words entered by the user, it does not mean that the AIGC model does not need to recognize these entities. Accordingly, the first entities obtained from the historical prompt word set can be effectively supplemented by mining other predetermined data sources, thereby constructing a more complete entity recognition graph.

[0089] For the constructed entity cognitive graph, corresponding first training samples can be built for all or some of the entities. For example, corresponding first training samples can be built for all entities, or entities with higher popularity can be considered to have higher priority. Accordingly, from the perspective of resource consumption, corresponding first training samples can be built only for some entities with higher priority. The specific implementation method is not limited. In addition, one or more first training samples can be built for the same entity.

[0090] Each entity in the entity cognitive graph may have a corresponding entity name and may also have one or more entity aliases. For example, an entity's entity name may be Li Si, and its entity alias may be Li Zi. Entity linking tools can be used to map entity names and entity aliases to the same entity. Alternatively, entity aliases corresponding to any entity name can be obtained by querying or searching relevant knowledge bases.

[0091] For any entity in the entity recognition graph, the corresponding image can be obtained by means of image search based on the entity name or alias of the entity. That is, the first training image corresponding to the entity can be obtained. The number of first training images can be one or more. If there are multiple images, then the corresponding first training sample can be constructed based on each image.

[0092] Furthermore, for the entity "Li Si," the first training image will typically only include Li Si and not other individuals. Moreover, the first training image should be selected based on its aesthetics and clarity to improve the subsequent model learning performance.

[0093] For the first training image, its image description information can also be obtained. In some embodiments of this disclosure, an image description generator (captioner) can first be used to generate initial description information for the first training image. Then, at least one of the entity name and entity alias of the entity corresponding to the first training image can be used to replace the target concept word in the initial description information to obtain the image description information. The target concept word is used to represent the entity corresponding to the first training image.

[0094] A captioner is a model tool that generates text for a given image. It takes any image as input and outputs textual description information (caption) about the content of the image. For ease of distinction, the generated caption is referred to as the initial description information in the scheme described in this disclosure.

[0095] Typically, the initial description information will not contain the entity name or alias of the entity. For example, the initial description information generated for the first training image corresponding to the entity "Li Si" might be: "A man wearing a black suit, holding a wine glass..." That is, the initial description information will only contain conceptual words such as "man," "boy," and "handsome guy," without directly including the entity name or alias. Therefore, it needs to be optimized. Specifically, at least one of the entity name and alias of the entity corresponding to the first training image should be used to replace the target conceptual words in the initial description information to obtain the desired image description information.

[0096] Assuming the entity in the first training image is Li Si, then the target concept word refers to the concept word used to represent Li Si, such as "man".

[0097] With the help of a captioner, the required initial descriptive information can be obtained efficiently and accurately. Then, by replacing the target concept words in the initial descriptive information, semantically clear and explicit image description information can be obtained, thus laying a good foundation for subsequent processing.

[0098] In some embodiments of this disclosure, when replacing a target concept word in the initial description information using at least one of the entity name and entity alias of the entity corresponding to the first training image, a predetermined target object can be used to replace the target concept word. The target object includes the entity name and entity alias. Alternatively, all target objects can be used to replace the target concept word in a tiling manner. Or, target objects can be randomly selected, and L selected target objects can be used to replace the target concept word, where L is a positive integer and less than P, and P is the total number of target objects.

[0099] In other words, the scheme described in this disclosure provides three ways to replace target concept words. Furthermore, for the first training image, one of the replacement methods can be randomly selected or selected according to a certain strategy to complete the replacement, which is very flexible and convenient.

[0100] Suppose the entity corresponding to the first training image is Li Si, that is, the entity name is Li Si, and the entity aliases include Li Zi and Si Ye, then one of the pre-determined Li Si, Li Zi, and Si Ye can be used to replace the target concept word. For example, Li Si can always be used to replace the target concept word, or, in a tiling form, such as "Li Si, Li Zi, Si Ye" or "Li Si / Li Zi / Si Ye", all target objects can be used to replace the target concept word, or, assuming that Li Zi and Si Ye are randomly selected from Li Si, Li Zi, and Si Ye, then in a tiling form, such as "Li Zi, Si Ye" or "Li Zi / Si Ye", the two selected target objects can be used to replace the target concept word.

[0101] By using the three replacement methods described above, the AIGC model can fully recognize the entity name and alias of the entity, thereby avoiding the situation where the entity cannot be correctly identified when its name is changed.

[0102] After obtaining the image description information of the first training image, a first training sample can be formed using the first training image and its corresponding image description information. In the same way, multiple first training samples can be constructed to form a first training set. Then, as described in step 102, the entity recognition ability of the first AIGC model can be trained using the first training set to obtain the second AIGC model.

[0103] In step 103, a second training set can also be constructed. The second training sample may include: modification relationship information between words in the target prompt word, and a second training image that matches the target prompt word. Then, as described in step 104, the entity detail depiction ability of the second AIGC model can be trained using the second training set.

[0104] In some embodiments of this disclosure, when constructing the second training set, a portion of historical prompt words can be obtained from the set of historical prompt words recorded in a predetermined business log, and target prompt words can be determined based on the portion of historical prompt words. Then, the second training images corresponding to each target prompt word and the modification relationship information of each target prompt word can be obtained respectively, and then the second training samples can be formed based on the modification relationship information and the corresponding second training images.

[0105] In some embodiments of this disclosure, the modification relationship may include M different types of modification relationships, where M is a positive integer greater than 1, such as 4. Accordingly, when obtaining some historical prompt words from the historical prompt word set, the modification relationship type to which each historical prompt word in the historical prompt word set belongs can be determined, and historical prompt words can be extracted according to the principle that the more historical prompt words there are, the more historical prompt words are extracted for different modification relationship types.

[0106] For different types of modification relationships, such as attribute binding, referential relationships, no sentence context, and spatial relationships, the modification relationship type of each historical prompt word in the historical prompt word set can be determined, thus obtaining the true distribution of historical prompt words belonging to different modification relationship types. Historical prompt words can then be extracted according to this true distribution; that is, for different modification relationship types, the principle is that the more historical prompt words belonging to a particular type, the more historical prompt words are extracted. Notably, the same historical prompt word may belong to one modification relationship type or multiple different modification relationship types simultaneously.

[0107] The above processing allows for the extraction of more historical prompt words from more diverse modification relationship types, thereby improving the targeting of subsequent model training and ultimately enhancing the model training effect.

[0108] In some embodiments of this disclosure, to obtain the target prompt word, a quality check can be performed on the portion of historical prompt words. In response to determining that there are target historical prompt words that have failed the check, the target historical prompt words can be modified, and the target prompt word can be determined. The target prompt word includes: historical prompt words that have passed the check and the modified target historical prompt word. In response to determining that there are no target historical prompt words that have failed the check, the target prompt word can include: historical prompt words that have passed the check. Alternatively, in some embodiments of this disclosure, the target prompt word can be directly determined, and the target prompt word includes: the portion of historical prompt words.

[0109] In other words, extracted historical prompts can be directly used as target prompts to simplify processing. Alternatively, considering that some extracted historical prompts may have quality issues and are unsuitable for direct use as target prompts, the extracted historical prompts can first be quality-verified according to a predetermined verification strategy. Target historical prompts that fail verification can be specifically modified. Then, the verified historical prompts and the modified target historical prompts can be used as target prompts, thereby improving the quality of the target prompts and consequently improving the subsequent model training effect. The specific verification strategy is not limited. Furthermore, if necessary, new prompts can be generated based on the extracted historical prompts and used as target prompts.

[0110] For each target prompt word, the corresponding second training image can be obtained. There are no restrictions on how to obtain it. For example, the image generated and determined to be correct based on the text-to-image model can be used as the required second training image.

[0111] In addition, the modification relationship information of each target prompt word can be obtained separately. Accordingly, for any obtained modification relationship information, a second training sample can be formed using the modification relationship information and the corresponding second training image.

[0112] In the same way, multiple second training samples can be constructed to form a second training set, which can then be used to train the entity detail depiction capability of the second AIGC model as described in step 104.

[0113] In some embodiments of this disclosure, for any second training image, the following processing may be performed: modifying the second training image using an image modification tool to obtain a modified image; modifying the target prompt word corresponding to the second training image based on the modified image to obtain a modified prompt word that matches the modified image; obtaining the modification relationship information of the modified prompt word; and using the modification relationship information of the modified prompt word and the modified image to form a second training sample.

[0114] For example, for any second training image, it can be quickly modified using tools such as ControlNet to generate one or more modified images that are similar to the second training image. For example, the spatial relationship can be modified, such as changing the position of object 1 to the right of object 2 instead of object 1 to object 2, or changing the position of Li Si from holding a wine glass in his left hand to holding a fan in his left hand instead of Li Si holding a wine glass in his left hand.

[0115] As the second training image is modified into the modified image, the corresponding target prompt words also need to be modified accordingly to obtain modified prompt words that match the modified image. In this way, the modification relationship information of the modified prompt words can be obtained, and the modification relationship information of the modified prompt words and the modified image can be used to form a second training sample and add it to the second training set.

[0116] The above processing can expand the second training set, thereby increasing the number of second training samples and further improving the training effect of the model.

[0117] It should be noted that in practical applications, when training the entity recognition ability of the first AIGC model using the first training set, the image description information (which can also be regarded as prompt words) of the first training image can also be obtained, and then the first training image and the corresponding modification relationship information can be used to form the first training sample.

[0118] In some embodiments of this disclosure, to obtain modification relationship information, taking the target prompt word as an example (the same applies to other prompt words), the following processing can be performed: using a pre-trained component analysis model, the component information of the words in the target prompt word is determined, and based on the component information, the modification relationship information is determined using a pre-trained modification relationship discrimination model.

[0119] In some embodiments of this disclosure, the component information of words in the target prompt word may include: subject words, limiting words, modifiers, topic words, and grammatical words in the target prompt word; the modification relationship may include: attribute binding, referential relationship, sentence-free context, and spatial position relationship.

[0120] The component information of the words in the target prompt refers to the components included in the image to be generated.

[0121] Assume the target prompt = noble princess Ragdoll, graceful, fluffy, blonde with blue eyes, foraging, the cat is on the left of the princess, the princess holds a wine glass in her left hand, wears black leather shoes on her feet, cyberpunk style.

[0122] Then we can obtain:

[0123] Subject words (main subjects of the picture): princess, Ragdoll, wine glass, leather shoes;

[0124] Limiting words (words modifying subject words): noble, graceful, fluffy, blonde with blue eyes, foraging, black;

[0125] Modifying words (for modifying the whole picture): cyberpunk style;

[0126] Grammatical words (punctuation marks, prepositions, auxiliary words, etc.): de, etc.;

[0127] Topic words (theme of the picture): abstract concepts such as catering industry, religion, etc.

[0128] The component information of words in the target prompt can be determined by using a component analysis model, as shown above. Then, according to said component information, the required modification relationship information can be determined by using a modification relationship discrimination model, which is to resolve what kind of modification relationship is needed to satisfy the instruction following capability.

[0129] For example, attribute binding:

[0130] noble, graceful, blonde with blue eyes -> modify -> princess;

[0131] fluffy, foraging -> modify -> Ragdoll;

[0132] black -> modify -> leather shoes.

[0133] Reference relationship:

[0134] cat -> personal reference -> Ragdoll;

[0135] wine glass -> reference inclusion (body part) -> left hand;

[0136] leather shoes -> reference inclusion (body part) -> foot.

[0137] No syntactic context:

[0138] graceful -> modify -> princess;

[0139] blonde with blue eyes -> modify -> princess;

[0140] fluffy -> modify -> Ragdoll;

[0141] foraging -> modify -> Ragdoll.

[0142] Spatial relationship:

[0143] Cat -> Left side -> Princess.

[0144] It should be noted that in practical applications, both component information and modification relationship information can include coordinates, which can refer to the position offset information in the text.

[0145] In addition, the modification relationship information can be organized into structured data in the form of JavaScript Object Notation (JSON), that is, the target prompt words are converted into intermediate structured data, and then combined with the corresponding image information and input into the AIGC model for training. This allows the AIGC model to obtain all the structured and accurate information of instruction compliance, thereby improving the instruction compliance ability of the AIGC model.

[0146] Once the AIGC model has been trained, it can be used for actual content generation, i.e., for prediction.

[0147] Figure 2 This is a flowchart illustrating an embodiment of the method for generating the content described in this disclosure. Figure 2 As shown, the specific implementation methods are as follows.

[0148] In step 201, the input prompt words to be processed are obtained, and the modification relationship information between words in the prompt words to be processed is obtained.

[0149] In step 202, the modification relationship information is used as input to the target model to obtain the content generation result corresponding to the prompt word to be processed. The target model is to utilize... Figure 1 The third AIGC model trained by the method in the illustrated embodiment.

[0150] By employing the scheme described in the above method embodiments, an AIGC model with improved instruction compliance capabilities, which is pre-trained, can be used to process the input prompt words, thereby improving the accuracy of the generated content.

[0151] In some embodiments of this disclosure, the method of obtaining the modification relationship information between words in the prompt word to be processed may include: using a pre-trained component analysis model to determine the component information of the words in the prompt word to be processed, and using a pre-trained modification relationship discrimination model to determine the modification relationship information based on the component information.

[0152] As can be seen, both model training and prediction can convert the prompt words into intermediate states and align them bidirectionally. In this way, no matter how the prompt words change, the information received by the model during training and prediction is consistent.

[0153] Furthermore, the target model described in this disclosure has the capability for multi-round detailed modification. For example, suppose a user needs to use the target model to generate an image. The user can interact with the target model multiple times. After initially inputting the prompt words to be processed, the user receives the content generated by the target model, i.e., an image. However, if the user is not satisfied with certain details of the image, they can input new prompt words to be processed to address the dissatisfaction, so that the target model can optimize the content generated result until the user is satisfied.

[0154] As can be seen, both model training and prediction require the use of a modification relation discrimination model. The following explains the training method of the modification relation discrimination model.

[0155] Figure 3 This is a flowchart illustrating an embodiment of the modification relation discrimination model training method described in this disclosure. Figure 3 As shown, the specific implementation methods are as follows.

[0156] In step 301, a third training set is constructed, which includes a third training sample, and the third training sample includes sample prompt words.

[0157] In step 302, the pre-trained Large Language Model (LLM) is trained using a third training set with supervised fine-tuning (SFT) to obtain a modification relation discrimination model. This modification relation discrimination model is used to construct a second training set, which includes second training samples. These samples include modification relation information between words in the target prompt word determined by the modification relation discrimination model, and a second training image matching the target prompt word. The second training set is used to train the entity detail representation ability of the second AIGC model to obtain a third AIGC model. The second AIGC model is obtained by training the first AIGC model's entity recognition ability using the first training set. The first training set includes first training samples, which include a first training image corresponding to any entity and image description information for the first training image. The modification relation discrimination model is also used to determine the modification relation information of the input prompt word when generating content using the third AIGC model.

[0158] As can be seen, by adopting the scheme described in the above method embodiment, the LLM+SFT architecture can be used to efficiently and accurately train the required modification relationship discrimination model based on the constructed third training set.

[0159] The specific type of LLM model is not limited; for example, it could be a text-based learning platform or a Generative Pre-trained Transformer (GPT) model. Depending on the number of parameters in the LLM model, a third training set of different sizes can be constructed for SFT training.

[0160] In some embodiments of this disclosure, when constructing the third training set, some historical prompt words can be obtained from the set of historical prompt words recorded in the predetermined business log, and corresponding sample prompt words, i.e., the third training samples, can be constructed according to each obtained historical prompt word.

[0161] The historical prompt word set usually contains a large number of historical prompt words, but in practice only a relatively small number of historical prompt words may be needed to construct the third training sample. Therefore, a certain strategy can be adopted to extract the historical prompt words in the historical prompt word set for subsequent processing.

[0162] In some embodiments of this disclosure, the learning-augmented k-means clustering algorithm can be used to cluster the historical prompt words in the historical prompt word set. For each cluster, the following processing can be performed: the historical prompt words in the cluster are sorted in ascending order of their distance from the center point of the cluster. Starting from the first historical prompt word after sorting, T historical prompt words are extracted from the cluster at intervals of W, where W and T are both positive integers, W is less than T, and T is less than the number of historical prompt words included in the cluster.

[0163] The specific values ​​of W and T can be determined according to actual needs. For example, suppose a cluster contains 100 historical prompt words, which are numbered as historical prompt words 1 to 100 after being sorted in ascending order of their distance from the center point of the cluster. If W is 1 and T is 20, then historical prompt words 1, 3, 5, 7, ..., a total of 20 historical prompt words can be extracted.

[0164] Different methods of extracting historical prompts can lead to significant differences in the subsequent model training results. Experiments show that using the above extraction methods can significantly improve the model training results compared to random extraction methods, such as a significant improvement in the overall index (F1).

[0165] Based on the extracted historical prompt words, corresponding sample prompt words can be constructed respectively. In some embodiments of this disclosure, for any extracted historical prompt word, the following processing can be performed: the component information of the words in the historical prompt word is determined using a pre-trained component analysis model; the default instruction template is filled according to the historical prompt word and the component information; and the sample prompt word corresponding to the historical prompt word is determined according to the filling result.

[0166] The component information of words in the historical prompt words may include: subject words, determiners, modifiers, thematic words, and grammatical words in the historical prompt words.

[0167] For example, the default instruction template could be:

[0168] The text is named '{text}', and the main words are {A}, {B}, {C}, etc., and the qualifiers are {D}, {E}, {F}, etc. What are the modifying relationships between them?

[0169] The contents within {} are all variables that can be filled in based on historical prompts and component information, making it very flexible and convenient. For example, "text" can be filled in with historical prompts, and A, B, and C can all be filled in with the main words in the component information. In addition, "The main words include {A}, {B}, {C}, etc., and the qualifiers include {D}, {E}, {F}, etc., ......," can be called reference information, which can help the modification relationship discrimination model to better learn the modification relationship between words.

[0170] In some embodiments of this disclosure, after obtaining the filling result corresponding to any historical prompt word, the determined sample prompt word corresponding to the historical prompt word may include: the filling result, or, the sample prompt word corresponding to the historical prompt word may be obtained by perturbation enhancement processing on the filling result, and the sample prompt word includes: the perturbation enhancement processing result.

[0171] In other words, the filling result can be directly used as the sample prompt word corresponding to the historical prompt word, or the filling result can be perturbed and enhanced, and the result of the perturbed enhancement can be used as the sample prompt word corresponding to the historical prompt word.

[0172] In some embodiments of this disclosure, the perturbation enhancement processing of the filling result may include any of the following: selecting one from a pre-defined N types of related words, where N is a positive integer, and adding the selected related word to the filling result to obtain the perturbation enhancement processing result; selecting one from a pre-defined S types of order adjustment methods, where S is a positive integer, and adjusting the order of the content in the filling result according to the selected order adjustment method to obtain the perturbation enhancement processing result; selecting one from a pre-defined Q types of instruction templates, where Q is a positive integer, and the Q types of instruction templates do not include the default instruction template, refilling the selected instruction template according to the historical prompt words and component information, and determining the refilled result as the perturbation enhancement processing result.

[0173] Assuming a total of 20,000 historical prompt words are extracted, for each historical prompt word, after obtaining the corresponding filling result, the filling result can be directly used as the sample prompt word corresponding to that historical prompt word. Alternatively, one of the three perturbation enhancement processing methods mentioned above can be selected to perturb the filling result. In this way, the final number of sample prompt words will still be 20,000, and the total number will not change.

[0174] In practical applications, a certain probability can be used to select whether to perform perturbation enhancement processing on the filling result. If it is determined that perturbation enhancement processing is required, one of the three perturbation enhancement processing methods mentioned above can be randomly selected, or one of the three perturbation enhancement processing methods mentioned above can be selected according to a certain strategy, and the specific strategy is not limited.

[0175] By using perturbation enhancement processing, the form and content of sample prompts can be enriched, thereby improving the perturbation resistance, robustness, and generalization ability of the modification relationship discrimination model obtained through subsequent training.

[0176] The three perturbation enhancement methods mentioned above can be referred to as subordinate perturbation, positioning perturbation, and paraphrasing perturbation, respectively.

[0177] 1) Subordinate relationship disturbance method

[0178] N types of related words can be preset, where N is a positive integer and the specific value can be determined according to actual needs. The related words can include "because-therefore", "if-then", etc. In this way, for any filling result, one of the N related words can be randomly selected, or one that matches the semantics of the filling result can be selected. Then the selected related word can be added to the filling result to obtain the required perturbation enhancement processing result.

[0179] For example, suppose the result of the fill is: the text is 'Girl wearing black-rimmed glasses', ... (representing omitted reference information), what is the modification relationship between them?

[0180] The result of perturbation enhancement processing could be: If the text is 'A girl is wearing black-rimmed glasses', ..., then what is the modification relationship between them?

[0181] 2) Location relationship disturbance mode

[0182] S order adjustment methods can be preset, where S is a positive integer and the specific value can be determined according to actual needs. In this way, for any filling result, one of the S order adjustment methods can be randomly selected, or a matching one can be selected according to the semantics of the filling result. Then, the content in the filling result can be adjusted in order according to the selected order adjustment method to obtain the required perturbation enhancement processing result.

[0183] For example, suppose the result is: the text is 'The girl is wearing black-rimmed glasses', ..., what is the modifying relationship between them?

[0184] So, the result of the perturbation enhancement process could be: What is the modification relationship between them? The text is 'The girl is wearing black-rimmed glasses', ...

[0185] 3) Methods of perturbing the semantic relationship

[0186] Q instruction templates can be preset, where Q is a positive integer and the specific value can be determined according to actual needs. The Q instruction templates may not include the default instruction template. In this way, for any filling result, one can be randomly selected from the Q instruction templates. Then, the selected instruction template can be refilled according to the historical prompt words and component information to obtain the required perturbation enhancement processing result.

[0187] For example, suppose the result is: the text is 'The girl is wearing black-rimmed glasses', ..., what is the modifying relationship between them?

[0188] The result of perturbation enhancement processing could be: the text is 'The girl is wearing black-rimmed glasses', ... How do you describe the modifiers in this text?

[0189] As can be seen, by adopting the above perturbation enhancement processing method, there is no need to modify the core content of the filling result. Only some minor changes are needed to obtain the required perturbation enhancement processing result. Accordingly, the robustness and generalization ability of the modification relationship discrimination model can be improved.

[0190] The input to the trained modification relation discrimination model can be a prompt word, and the output can be specific modification relation information, such as (a, b), (c, d), (e, above, f), etc. Among them, spatial position relations can be in the form of triples, and other modification relations can be in the form of binary pairs.

[0191] Based on the above introduction, Figure 4 This is a schematic diagram illustrating the overall implementation process of model training and prediction as described in this disclosure. Figure 4 As shown, the instruction-following ability of the AIGC model can be visualized as entity cognition ability and entity detail depiction ability (including multi-round detail modification ability). Specifically, for entity cognition ability, an entity cognition graph can be constructed, and a first training set can be constructed based on the entity cognition graph. Then, the first AIGC model can be trained using the first training set to obtain a second AIGC model. For entity detail depiction ability, a second training set can be constructed based on component analysis model and modification relationship discrimination model, and the second AIGC model can be trained again using the second training set to obtain a third AIGC model. Furthermore, actual content generation can be performed based on the third AIGC model.

[0192] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this disclosure is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this disclosure. Secondly, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this disclosure. Furthermore, for parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0193] The above is an introduction to the method embodiments. The following describes the solution described in this disclosure further through device embodiments.

[0194] Figure 5 This is a schematic diagram of the structural composition of Embodiment 500 of the AIGC model training device described in this disclosure. Figure 5 As shown, it includes: a first training module 501 and a second training module 502.

[0195] The first training module 501 is used to construct a first training set, which includes a first training sample. The first training sample includes: a first training image corresponding to any entity and image description information of the first training image; the entity recognition ability of the first AIGC model is trained using the first training set to obtain a second AIGC model.

[0196] The second training module 502 is used to construct a second training set, which includes second training samples. The second training samples include: modification relationship information between words in the target prompt word, and second training images that match the target prompt word. Using the second training set, the entity detail characterization ability of the second AIGC model is trained to obtain the third AIGC model.

[0197] By adopting the scheme described in the above device embodiment, the instruction compliance capability of the AIGC model can be visualized as entity recognition capability and entity detail depiction capability, and corresponding training sets can be constructed respectively. Then, the AIGC model can be trained using the constructed training sets, thereby achieving targeted optimization of instruction compliance capability, that is, improving the instruction compliance capability of the AIGC model, and thus improving the accuracy of content generated based on AIGC.

[0198] In some embodiments of this disclosure, the first training module 501 can construct an entity recognition graph and perform the following processing on all or part of the entities in the entity recognition graph: obtain the entity image corresponding to the entity, the first training image includes the entity image, obtain the image description information of the first training image, and generate a first training sample based on the first training image and the image description information.

[0199] In addition, in some embodiments of this disclosure, the first training module 501 may obtain a first entity with a higher preset first popularity from the set of historical prompt words recorded in the predetermined business log, and construct an entity cognitive graph based on the first entity. Alternatively, it may obtain a second entity with a higher preset second popularity from other predetermined data sources outside the set of historical prompt words, and construct an entity cognitive graph based on the first entity and the second entity.

[0200] For the first training image, its image description information can also be obtained. In some embodiments of this disclosure, the first training module 501 may first use an image description generator to generate initial description information for the first training image, and then use at least one of the entity name and entity alias of the entity corresponding to the first training image to replace the target concept word in the initial description information to obtain the image description information, where the target concept word is used to represent the entity corresponding to the first training image.

[0201] In some embodiments of this disclosure, when the first training module 501 replaces the target concept word in the initial description information using at least one of the entity name and entity alias of the entity corresponding to the first training image, it may replace the target concept word with a predetermined target object, wherein the target object includes the entity name and entity alias; or, it may replace the target concept word with all target objects in a tiling manner; or, it may randomly select target objects and replace the target concept word with L selected target objects, where L is a positive integer and less than P, and P is the total number of target objects.

[0202] After obtaining the image description information of the first training image, the first training module 501 can use the first training image and its corresponding image description information to form a first training sample. In the same way, multiple first training samples can be constructed to form a first training set. This first training set can then be used to train the entity recognition ability of the first AIGC model to obtain a second AIGC model.

[0203] In addition, the second training module 502 can also construct a second training set, which includes: modification relationship information between words in the target prompt word, and a second training image that matches the target prompt word. The second training set can then be used to train the entity detail representation ability of the second AIGC model.

[0204] In some embodiments of this disclosure, when constructing the second training set, the second training module 502 can obtain a portion of historical prompt words from the historical prompt word set recorded in the predetermined business log, and determine the target prompt words based on the portion of historical prompt words. Then, it can obtain the second training image corresponding to each target prompt word and the modification relationship information of each target prompt word, and then form the second training sample based on the modification relationship information and the corresponding second training image.

[0205] In some embodiments of this disclosure, the modification relationship may include M different types of modification relationships, where M is a positive integer greater than 1, such as 4. Accordingly, when the second training module 502 obtains some historical prompt words from the historical prompt word set, it can determine the modification relationship type to which each historical prompt word in the historical prompt word set belongs, and can extract historical prompt words according to the principle that the more historical prompt words there are, the more historical prompt words are extracted for different modification relationship types.

[0206] In some embodiments of this disclosure, the second training module 502 can perform quality verification on the portion of historical prompt words. In response to determining that there are target historical prompt words that have failed verification, the target historical prompt words are modified, and a target prompt word is determined. The target prompt word includes: historical prompt words that have passed verification and the modified target historical prompt word. In response to determining that there are no target historical prompt words that have failed verification, the target prompt word includes: historical prompt words that have passed verification. Alternatively, in some embodiments of this disclosure, the second training module 502 can directly determine the target prompt word, including: the portion of historical prompt words.

[0207] In some embodiments of this disclosure, the second training module 502 may further perform the following processing on any second training image: modify the second training image using an image modification tool to obtain a modified image; modify the target prompt word corresponding to the second training image based on the modified image to obtain a modified prompt word that matches the modified image; obtain the modification relationship information of the modified prompt word; and use the modification relationship information of the modified prompt word and the modified image to form a second training sample.

[0208] In some embodiments of this disclosure, the second training module 502, in order to obtain modification relationship information, taking the target prompt word as an example (the other prompt words are the same), can perform the following processing: using a pre-trained component analysis model, determine the component information of the words in the target prompt word, and based on the component information, use a pre-trained modification relationship discrimination model to determine the modification relationship information.

[0209] In some embodiments of this disclosure, the component information of words in the target prompt word may include: subject words, limit words, modifier words, topic words and grammatical words in the target prompt word; the modification relationship may include: attribute binding, referential relationship, sentence-independent context and spatial position relationship.

[0210] Figure 6 This is a schematic diagram of the structural composition of Embodiment 600 of the content generation apparatus described in this disclosure. Figure 6 As shown, it includes: an acquisition module 601 and a generation module 602.

[0211] The acquisition module 601 is used to acquire the input prompt words to be processed and to acquire the modification relationship information between words in the prompt words to be processed.

[0212] The generation module 602 is used to take the modification relationship information as input to the target model to obtain the content generation result corresponding to the prompt word to be processed. The target model can be used to... Figure 5 The third AIGC model trained by the device in the illustrated embodiment.

[0213] By employing the scheme described in the above-described device embodiment, an AIGC model with improved instruction compliance capabilities, which is pre-trained, can be used to process the input prompt words, thereby improving the accuracy of the generated content.

[0214] In some embodiments of this disclosure, the method by which the acquisition module 601 acquires the modification relationship information between words in the prompt word to be processed may include: using a pre-trained component analysis model to determine the component information of the words in the prompt word to be processed, and using a pre-trained modification relationship discrimination model to determine the modification relationship information based on the component information.

[0215] Figure 7 This is a schematic diagram of the structural composition of an embodiment 700 of the modification relationship discrimination model training device described in this disclosure. Figure 7 As shown, it includes: a construction module 701 and a training module 702.

[0216] Module 701 is used to construct a third training set, which includes third training samples, including sample prompt words.

[0217] Training module 702 is used to perform SFT training on the pre-trained large language model using a third training set to obtain a modification relation discrimination model. The modification relation discrimination model is used to construct a second training set, which includes a second training sample. The second training sample includes: modification relation information between words in the target prompt word determined by the modification relation discrimination model, and a second training image matching the target prompt word. The second training set is used to train the entity detail characterization ability of the second AIGC model to obtain a third AIGC model. The second AIGC model is obtained by training the entity recognition ability of the first AIGC model using the first training set. The first training set includes a first training sample, which includes: a first training image corresponding to any entity and image description information of the first training image. The modification relation discrimination model is also used to determine the modification relation information of the input prompt word when using the third AIGC model for content generation.

[0218] Using the scheme described in the above device embodiment, an LLM+SFT architecture can be adopted to efficiently and accurately train the required modification relationship discrimination model based on the constructed third training set.

[0219] In some embodiments of this disclosure, when constructing the third training set, the construction module 701 can obtain some historical prompt words from the historical prompt word set recorded in the predetermined business log, and construct corresponding sample prompt words, i.e., the third training samples, based on each obtained historical prompt word.

[0220] In some embodiments of this disclosure, the construction module 701 may use the Learning-augmented k-means algorithm to cluster the historical prompt words in the historical prompt word set, and may perform the following processing on each obtained cluster: sort the historical prompt words located in the cluster in ascending order of distance from the center point of the cluster, and starting from the first historical prompt word after sorting, extract T historical prompt words from the cluster in a manner that one historical prompt word is extracted every W historical prompt words, where W and T are both positive integers, W is less than T, and T is less than the number of historical prompt words included in the cluster.

[0221] In some embodiments of this disclosure, the construction module 701 may perform the following processing on any extracted historical prompt word: determine the component information of the words in the historical prompt word using a pre-trained component analysis model, fill the default instruction template according to the historical prompt word and the component information, and determine the sample prompt word corresponding to the historical prompt word according to the filling result.

[0222] The component information of words in the history prompt can include: subject words, determiners, modifiers, thematic words, and grammatical words in the history prompt.

[0223] In some embodiments of this disclosure, after obtaining the filling result corresponding to any historical prompt word, the construction module 701 determines the sample prompt word corresponding to the historical prompt word, which may include: the filling result, or, the sample prompt word corresponding to the historical prompt word may be obtained by perturbation enhancement processing on the filling result, wherein the sample prompt word includes: the perturbation enhancement processing result.

[0224] In addition, in some embodiments of this disclosure, the construction module 701 may perform perturbation enhancement processing on the filling result in the following ways: selecting one from N pre-set associated words, where N is a positive integer, and adding the selected associated word to the filling result to obtain the perturbation enhancement processing result; or selecting one from S pre-set order adjustment methods, where S is a positive integer, and adjusting the order of the content in the filling result according to the selected order adjustment method to obtain the perturbation enhancement processing result; or selecting one from Q pre-set instruction templates, where Q is a positive integer, and the Q instruction templates do not include the default instruction template, refilling the selected instruction template according to the historical prompt words and component information, and determining the refilling result as the perturbation enhancement processing result.

[0225] The specific workflow of each of the above device embodiments can be found in the relevant descriptions in the foregoing method embodiments, and will not be repeated here.

[0226] The solutions described in this disclosure can be applied to the field of artificial intelligence, particularly large-scale models, deep learning, natural language processing, and knowledge graphs. Artificial intelligence is the study of enabling computers to simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, and planning). It involves both hardware and software technologies. Artificial intelligence hardware technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, and big data processing. Artificial intelligence software technologies mainly include computer vision, speech recognition, natural language processing, machine learning / deep learning, big data processing, and knowledge graph technologies.

[0227] The prompts and images in the embodiments described in this disclosure are not targeted at any specific user and do not reflect the personal information of any specific user. The collection, storage, use, processing, transmission, provision, and disclosure of user personal information involved in the technical solutions of this disclosure all comply with relevant laws and regulations and do not violate public order and good morals.

[0228] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0229] Figure 8 A schematic block diagram of an electronic device 800 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workbenches, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0230] like Figure 8 As shown, device 800 includes a computing unit 801, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 802 or a computer program loaded from storage unit 808 into random access memory (RAM) 803. RAM 803 may also store various programs and data required for the operation of device 800. The computing unit 801, ROM 802, and RAM 803 are interconnected via bus 804. Input / output (I / O) interface 805 is also connected to bus 804.

[0231] Multiple components in device 800 are connected to I / O interface 805, including: input unit 806, such as keyboard, mouse, etc.; output unit 807, such as various types of monitors, speakers, etc.; storage unit 808, such as disk, optical disk, etc.; and communication unit 809, such as network card, modem, wireless transceiver, etc. Communication unit 809 allows device 800 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0232] The computing unit 801 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 801 performs the various methods and processes described above, such as those described in this disclosure. For example, in some embodiments, the methods described in this disclosure can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 808. In some embodiments, part or all of the computer program can be loaded and / or installed on device 800 via ROM 802 and / or communication unit 809. When the computer program is loaded into RAM 803 and executed by the computing unit 801, one or more steps of the methods described in this disclosure can be performed. Alternatively, in other embodiments, the computing unit 801 can be configured to perform the methods described in this disclosure by any other suitable means (e.g., by means of firmware).

[0233] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0234] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0235] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0236] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0237] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0238] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0239] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0240] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A method for training a content generation model, comprising: Construct a first training set, which includes a first training sample, and the first training sample includes: a first training image corresponding to any entity and image description information of the first training image; The entity recognition ability of the first content generation model is trained using the first training set to obtain the second content generation model; A second training set is constructed, comprising a second training sample, which includes: modification relationship information between words in the target prompt words, and a second training image matching the target prompt words. Constructing the second training set includes: acquiring each target prompt word, acquiring the second training image corresponding to each target prompt word, and acquiring the modification relationship information of each target prompt word; and forming the second training sample based on the modification relationship information and the corresponding second training image. Acquiring the modification relationship information of each target prompt word includes: for any target prompt word, using a pre-trained component analysis model to determine the component information of the words in the target prompt word, and using a pre-trained modification relationship discrimination model to determine the modification relationship information based on the component information. The component information of the words in the target prompt word includes: subject words, limiting words, modifying words, topic words, and grammatical words in the target prompt word; the modification relationship includes: attribute binding, referential relationship, no-contextuality relationship, and spatial position relationship. Using the second training set, the entity detail representation capability of the second content generation model is trained to obtain the third content generation model.

2. The method according to claim 1, wherein, The construction of the first training set includes: Constructing entity cognitive graphs; For all or part of the entities in the entity recognition graph, the following processes are performed respectively: obtaining the entity image corresponding to the entity, wherein the first training image includes the entity image; obtaining the image description information of the first training image; and generating the first training sample based on the first training image and the image description information.

3. The method according to claim 2, wherein, The construction of the entity cognitive graph includes: Retrieve the first entity with a higher preset popularity from the set of historical prompts recorded in the pre-defined business logs; At least the entity cognitive graph is constructed based on the first entity.

4. The method according to claim 3, wherein, The construction of the entity cognitive graph also includes: Obtain a second entity with a higher preset second popularity from other predetermined data sources besides the historical prompt word set; The entity cognitive graph is constructed based on the first entity and the second entity.

5. The method according to claim 2, wherein, The step of obtaining the image description information of the first training image includes: Using an image description generator, generate initial description information for the first training image; The target concept word in the initial description information is replaced by at least one of the entity name and entity alias of the entity corresponding to the first training image to obtain the image description information, wherein the target concept word is used to represent the entity corresponding to the first training image.

6. The method according to claim 5, wherein, The step of replacing the target concept word in the initial description information using at least one of the entity name and entity alias of the entity corresponding to the first training image includes any one of the following: Replace the target concept term with a predetermined target object, wherein the target object includes the entity name and the entity alias; By tiling, the target concept word is replaced with all target objects; The target objects are randomly selected, and the target concept words are replaced by L selected target objects, where L is a positive integer and less than P, and P is the total number of target objects.

7. The method according to any one of claims 1 to 6, wherein, The acquisition of each target prompt word includes: Retrieve a subset of historical prompt words from the historical prompt word set recorded in the scheduled business log; The target prompt word is determined based on the aforementioned historical prompt words.

8. The method according to claim 7, wherein, The modification relations include: M different types of modification relations, where M is a positive integer greater than 1; The step of retrieving a subset of historical prompt words from the set of historical prompt words recorded in the pre-defined business logs includes: Determine the modification relationship type of each historical prompt word in the historical prompt word set; For different types of modification relationships, historical prompts are extracted according to the principle that the more historical prompts there are, the more historical prompts are extracted.

9. The method according to claim 7, wherein, The step of determining the target prompt word based on the aforementioned historical prompt words includes: The quality of the aforementioned historical suggestion words was verified; In response to the determination that there are target historical prompt words that have failed verification, the target historical prompt words are modified, and the target prompt words are determined. The target prompt words include: historical prompt words that have passed verification and the modified target historical prompt words. In response to determining that there are no target historical prompt words that failed verification, the target prompt entity includes: the historical prompt words that passed verification.

10. The method according to claim 7, wherein, The step of determining the target prompt word based on the aforementioned historical prompt words includes: The identified target prompt words include: a portion of the historical prompt words.

11. The method of claim 7, further comprising: For any second training image, perform the following processing: The second training image was modified using an image editing tool to obtain the modified image; Based on the modified image, the target prompt word corresponding to the second training image is modified to obtain a modified prompt word that matches the modified image; Obtain the modification relationship information of the modified prompt word; The modification relationship information of the modified prompt words and the modified image are used to form the second training sample.

12. A content generation method, comprising: Obtain the input prompt words to be processed, and obtain the modification relationship information between words in the prompt words to be processed; The modification relationship information is used as input to the target model to obtain the content generation result corresponding to the prompt word to be processed. The target model is the third content generation model trained using the method of any one of claims 1 to 11.

13. The method according to claim 12, wherein, The step of obtaining the modification relationship information between words in the prompt word to be processed includes: Using a pre-trained component analysis model, the component information of the words in the prompt words to be processed is determined; Based on the component information, the modification relationship information is determined using a pre-trained modification relationship discrimination model.

14. A method for training a modification relation discrimination model, comprising: Construct a third training set, which includes a third training sample, and the third training sample includes: sample prompt words; The pre-trained large language model is subjected to supervised fine-tuning training using the third training set to obtain the modification relation discrimination model. The modification relation discrimination model is used to construct a second training set, which includes a second training sample. The second training sample includes: modification relation information between words in the target prompt word determined by the modification relation discrimination model, and a second training image matching the target prompt word. The second training set is used to train the entity detail characterization ability of the second content generation model to obtain a third content generation model. The second content generation model is obtained by training the entity recognition ability of the first content generation model using the first training set. The first training set includes a first training sample, which includes: a first training image corresponding to any entity and image description information of the first training image. The modification relation discrimination model is also used to determine the modification relation information of the input prompt word when using the third content generation model for content generation.

15. The method according to claim 14, wherein, The construction of the third training set includes: Retrieve a subset of historical prompt words from the historical prompt word set recorded in the scheduled business log; Based on the obtained historical prompts, corresponding sample prompts are constructed.

16. The method according to claim 15, wherein, The step of retrieving a subset of historical prompt words from the set of historical prompt words recorded in the pre-defined business logs includes: The K-means clustering algorithm of reinforcement learning is used to cluster the historical prompt words in the historical prompt word set; For each obtained cluster, the following processing is performed: the historical prompt words located in the cluster are sorted in ascending order of their distance from the center point of the cluster, and starting from the first historical prompt word after sorting, T historical prompt words are extracted from the cluster by extracting one historical prompt word every W historical prompt words, where W and T are both positive integers, W is less than T, and T is less than the number of historical prompt words included in the cluster.

17. The method according to claim 15, wherein, The step of constructing corresponding sample prompt words based on the obtained historical prompt words includes: For each extracted historical suggestion word, the following processing is performed: The component information of the words in the historical prompt words is determined using a pre-trained component analysis model; The default instruction template is filled in based on the historical prompt words and the component information, and the sample prompt words corresponding to the historical prompt words are determined based on the filling results.

18. The method according to claim 17, wherein, The step of determining the sample prompt words corresponding to the historical prompt words based on the filling results includes: The sample prompts corresponding to the identified historical prompts include: the filling results; Alternatively, by performing perturbation enhancement processing on the filling result, sample prompt words corresponding to the historical prompt words can be obtained, and the sample prompt words include: the perturbation enhancement processing result.

19. The method according to claim 18, wherein, The perturbation enhancement process applied to the filling result includes any one of the following: Select one of the pre-defined N related words, where N is a positive integer, and add the selected related word to the filling result to obtain the perturbation enhancement processing result; Select one of the pre-set S order adjustment methods, where S is a positive integer, and adjust the order of the contents in the filling result according to the selected order adjustment method to obtain the disturbance enhancement processing result; One instruction template is selected from a set of Q pre-defined instruction templates, where Q is a positive integer. The default instruction template is not included in the set of Q instruction templates. The selected instruction template is then refilled based on the historical prompts and the component information. The refilled result is then determined as the perturbation enhancement processing result.

20. A content generation model training device, comprising: The first training module and the second training module; The first training module is used to construct a first training set, which includes a first training sample. The first training sample includes: a first training image corresponding to any entity and image description information of the first training image; and to train the entity recognition ability of the first content generation model using the first training set to obtain a second content generation model. The second training module is used to construct a second training set, which includes a second training sample. The second training sample includes: modification relationship information between words in the target prompt words, and a second training image matching the target prompt words. Constructing the second training set includes: acquiring each target prompt word, acquiring the second training image corresponding to each target prompt word, and acquiring the modification relationship information of each target prompt word; and forming the second training sample based on the modification relationship information and the corresponding second training image. Acquiring the modification relationship information of each target prompt word includes: for any target prompt word, using a pre-trained component analysis model to determine the component information of the words in the target prompt word, and using a pre-trained modification relationship discrimination model to determine the modification relationship information based on the component information. The component information of the words in the target prompt word includes: subject words, limiting words, modifying words, topic words, and grammatical words in the target prompt word; the modification relationship includes: attribute binding, referential relationship, no-sentence context, and spatial position relationship. Using the second training set, the entity detail characterization ability of the second content generation model is trained to obtain a third content generation model.

21. The apparatus according to claim 20, wherein, The first training module constructs an entity recognition graph and performs the following processing on all or part of the entities in the entity recognition graph: obtaining entity images corresponding to the entities, the first training image including the entity images, obtaining image description information of the first training image, and generating the first training sample based on the first training image and the image description information.

22. The apparatus according to claim 21, wherein, The first training module obtains a first entity with a higher than preset first popularity from the set of historical prompt words recorded in the predetermined business log, and constructs the entity cognitive graph based on at least the first entity.

23. The apparatus according to claim 22, wherein, The first training module is further configured to obtain a second entity with a higher than preset second popularity from other predetermined data sources outside the set of historical prompt words, and construct the entity cognitive graph based on the first entity and the second entity.

24. The apparatus according to claim 21, wherein, The first training module uses an image description generator to generate initial description information for the first training image, and uses at least one of the entity name and entity alias of the entity corresponding to the first training image to replace the target concept word in the initial description information to obtain the image description information. The target concept word is used to represent the entity corresponding to the first training image.

25. The apparatus according to claim 24, wherein, The first training module replaces the target concept word with a predetermined target object, the target object including the entity name and the entity alias, or replaces the target concept word with all target objects in a tiling manner, or randomly selects the target objects and replaces the target concept word with L selected target objects, where L is a positive integer and less than P, and P is the total number of target objects.

26. The apparatus according to any one of claims 20 to 25, wherein, The second training module obtains a portion of historical prompt words from the set of historical prompt words recorded in the predetermined business log, and determines the target prompt word based on the portion of historical prompt words.

27. The apparatus according to claim 26, wherein, The modification relations include: M different types of modification relations, where M is a positive integer greater than 1; The second training module determines the modification relationship type of each historical prompt word in the historical prompt word set. For different modification relationship types, historical prompt words are extracted according to the principle that the more historical prompt words there are, the more historical prompt words are extracted.

28. The apparatus according to claim 26, wherein, The second training module performs quality verification on the aforementioned historical prompt words. In response to determining that there are target historical prompt words that have failed verification, the target historical prompt words are modified, and the target prompt words are determined. The target prompt words include: historical prompt words that have passed verification and the modified target historical prompt words. In response to determining that there are no target historical prompt words that have failed verification, the target prompt word includes: the historical prompt words that have passed verification.

29. The apparatus according to claim 26, wherein, The target prompt words determined by the second training module include: a portion of the historical prompt words.

30. The apparatus according to claim 26, wherein, The second training module is further configured to perform the following processing on any second training image: modify the second training image using an image modification tool to obtain a modified image; modify the target prompt word corresponding to the second training image based on the modified image to obtain a modified prompt word that matches the modified image; obtain the modification relationship information of the modified prompt word; and use the modification relationship information of the modified prompt word and the modified image to form the second training sample.

31. A content generation apparatus, comprising: Acquire modules and generate modules; The acquisition module is used to acquire the input prompt words to be processed, and to acquire the modification relationship information between words in the prompt words to be processed; The generation module is used to take the modification relationship information as input to the target model to obtain the content generation result corresponding to the prompt word to be processed, wherein the target model is the third content generation model trained using the device according to any one of claims 20 to 30.

32. The apparatus according to claim 31, wherein, The acquisition module uses a pre-trained component analysis model to determine the component information of the words in the prompt words to be processed, and uses a pre-trained modification relationship discrimination model to determine the modification relationship information based on the component information.

33. A training device for a modification relation discrimination model, comprising: Build modules and training modules; The construction module is used to construct a third training set, which includes a third training sample, and the third training sample includes: sample prompt words; The training module is used to perform supervised fine-tuning training on the pre-trained large language model using the third training set to obtain the modification relation discrimination model. The modification relation discrimination model is used to construct a second training set, which includes a second training sample. The second training sample includes: modification relation information between words in the target prompt word determined by the modification relation discrimination model, and a second training image matching the target prompt word. The second training set is used to train the entity detail characterization ability of the second content generation model to obtain a third content generation model. The second content generation model is obtained by training the entity recognition ability of the first content generation model using the first training set. The first training set includes a first training sample, which includes: a first training image corresponding to any entity and image description information of the first training image. The modification relation discrimination model is also used to determine the modification relation information of the input prompt word when using the third content generation model for content generation.

34. The apparatus according to claim 33, wherein, The construction module obtains a portion of historical prompt words from the set of historical prompt words recorded in the predetermined business log, and constructs corresponding sample prompt words based on each obtained historical prompt word.

35. The apparatus according to claim 34, wherein, The construction module uses the K-means clustering algorithm of reinforcement learning to cluster the historical prompt words in the historical prompt word set. For each cluster, the following processing is performed: the historical prompt words in the cluster are sorted in ascending order of distance from the center point of the cluster. Starting from the first historical prompt word after sorting, T historical prompt words are extracted from the cluster at intervals of W, where W and T are both positive integers, W is less than T, and T is less than the number of historical prompt words included in the cluster.

36. The apparatus according to claim 34, wherein, For any extracted historical prompt word, the construction module performs the following processing: using a pre-trained component analysis model to determine the component information of the words in the historical prompt word; filling the default instruction template according to the historical prompt word and the component information; and determining the sample prompt word corresponding to the historical prompt word based on the filling result.

37. The apparatus according to claim 36, wherein, The sample prompt words corresponding to the historical prompt words determined by the construction module include: the filling result, or, by performing perturbation enhancement processing on the filling result, the sample prompt words corresponding to the historical prompt words are obtained, and the sample prompt words include: the perturbation enhancement processing result.

38. The apparatus according to claim 37, wherein, The construction module selects one from N pre-defined related words, where N is a positive integer, and adds the selected related word to the filling result to obtain the perturbation enhancement processing result. Alternatively, it selects one from S pre-defined order adjustment methods, where S is a positive integer, and adjusts the order of the contents in the filling result according to the selected order adjustment method to obtain the perturbation enhancement processing result. Or, it selects one from Q pre-defined instruction templates, where Q is a positive integer, and the default instruction template is not included in the Q instruction templates. The selected instruction template is refilled according to the historical prompt words and the component information, and the refilled result is determined as the perturbation enhancement processing result.

39. An electronic device comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-19.

40. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-19.

41. A computer program product comprising a computer program / instructions that, when executed by a processor, implement the method of any one of claims 1-19.

Images