A creative material generation method and device based on task arrangement

Abstract

This invention discloses a method and apparatus for generating creative materials based on task orchestration. The method includes: collecting initial generation requirements and initial reference materials; decomposing and encapsulating the initial generation requirements and initial reference materials based on a preset task object pattern to form a target task; generating creative prompts by extracting reference factors from the initial reference materials; executing the target task based on the creative prompts and a multimodal large language model to obtain task results; analyzing and judging the task results based on various result evaluation dimensions to provide a target evaluation result; and optimizing the creative prompts based on the target evaluation result to obtain target creative materials. By extracting reference factors from the initial reference materials, complex creative requirements are expressed in a unified structure, enabling accurate control over the style and differences of batch-generated requirements, increasing the probability of using the generated results, and ensuring the stable application and implementation efficiency of creative materials.

Description

Technical Field

[0001] This invention belongs to the technical field of artificial intelligence and digital content generation, specifically relating to a method and apparatus for generating creative materials based on task arrangement. Background Technology

[0002] With the development of generative artificial intelligence technology, technologies such as text-to-image generation, image editing, and image-to-video generation have begun to be applied in scenarios such as advertising creative design, product poster production, and marketing material generation. Existing technologies typically rely on user-input natural language prompts to invoke image or video generation models to generate corresponding images, videos, and other creative content. However, depending on users to organize their own prompts makes it difficult to transform the various constraints within those prompts into a unified, stable, and executable generation task, leading to discrepancies between the generated results and actual needs. The natural language understanding module, visual semantic understanding module, image generation model, video generation model, task scheduling module, and material management module in existing technologies are usually independent. Users need to complete tasks such as prompt writing, model selection, reference material organization, result downloading, post-processing cropping and compression, and material archiving separately, resulting in fragmented processes and low automation. Furthermore, the generated results lack a complete correlation with the model version used, sampling parameters, random seed, reference materials, prompts, and post-processing pipeline, making the generation process difficult to reproduce and track.

[0003] In scenarios where reference materials are used in the generation process, existing technologies often struggle to accurately distinguish the user's true intent. For example, a user might upload an image simply to reference its style, or they might want to perform partial editing based on the original image, replace the background, merge multiple images, batch modify historically generated images, or use the image as the first or last frame of a video. If the system cannot select the appropriate model invocation method and parameter organization method based on different intents, it can easily lead to inaccurate use of reference materials, affecting the quality of the generated content.

[0004] For example, patent application CN122288790A discloses a method, apparatus, electronic device, and storage medium for generating promotional information. The method includes: responding to an input command triggered on a promotional information generation interface, presenting the input business promotion requirements, and initiating multiple intelligent agents to collaboratively execute a promotional information generation task; during task execution, the multiple intelligent agents collaboratively generate promotional information based on the business promotion requirements and at least one reference promotional material; each reference promotional material is selected from a preset material library based on the dissemination potential assessment value of each promotional material in the preset material library; presenting and continuously updating the generation progress of the promotional information on the promotional information generation interface, the generation progress being used to reflect the task completion status of the multiple intelligent agents; and presenting the target promotional information and interactive prompts for the target audience to evaluate the usability of the target promotional information upon task completion, thereby improving the quality of the generated promotional information.

[0005] The existing technologies mentioned above all focus on how to generate materials, rather than on breaking down and extracting user needs.

[0006] How to accurately control the style and differences of batch-generated requirements under complex creative needs, form a traceable, reproducible and reusable management process, and ensure the stable application and implementation efficiency of creative materials is a problem that needs to be solved. Summary of the Invention

[0007] To address the shortcomings of the existing technologies, this invention provides a method and apparatus for generating creative materials based on task orchestration. The method includes: collecting initial generation requirements and initial reference materials input by the user; decomposing and encapsulating the initial generation requirements and initial reference materials based on a preset task object pattern to form a target task; generating creative prompts by extracting reference factors from the initial reference materials; executing the target task based on the creative prompts and combining a multimodal large language model to obtain task results; analyzing and judging the task results based on various result evaluation dimensions to provide target evaluation results; and optimizing the creative prompts based on the target evaluation results to obtain target creative materials.

[0008] By extracting reference factors from initial reference materials to generate creative prompts, and then executing target tasks based on these prompts, the results are analyzed and evaluated using multiple evaluation dimensions. This process optimizes the creative prompts to obtain the target creative materials. By extracting reference factors from initial reference materials, complex creative requirements are expressed in a unified, structured manner. This allows for accurate control over the style and diversity of batch-generated requirements, increasing the usability of the generated results and establishing a traceable and reproducible management process. This ensures the stable application and efficient implementation of creative materials.

[0009] In a first aspect, the present invention provides a method for generating creative materials based on task arrangement, specifically including the following steps: Collect initial generation requirements and initial reference materials from users; Based on the preset task object pattern, the initial generation requirements and initial reference materials are decomposed and encapsulated to form the target task; Creative prompts are generated by extracting reference factors from the initial reference materials. The reference factors are reusable design baseline elements obtained from the initial reference materials through joint representation vector extraction and parametric reconstruction by variational autoencoder. The creative prompts are obtained by combining and analyzing the reference factors through graph neural networks and prompt generation models. Based on the creative prompts and combined with a multimodal large language model, the target task is executed to obtain the task results; Based on each result evaluation dimension, the task results are analyzed and judged, and the target evaluation results are given. Based on the target evaluation results, optimize the creative prompts to obtain the target creative materials.

[0010] Furthermore, the task object pattern includes reference materials, advertising platforms, brand information, product selling points, target audience, generation quantity, and delivery specifications; Based on a preset task object pattern, the initial generation requirements and initial reference materials are decomposed and encapsulated to form the target task, which specifically includes: Based on the modality of the initial generated requirements and initial reference materials, the requirement text data and requirement image data in the initial generated requirements and initial reference materials are analyzed respectively to determine the requirement phrases and reference images; The required phrases, reference images and task object patterns are matched and the data is structured to form a primary task object; Iterate through the primary task objects, combine them with the task content constraints, and generate feedback instructions; Based on feedback instructions, the initial task object is supplemented and adjusted to determine the target task.

[0011] Furthermore, by extracting reference factors from the initial reference materials, creative prompts are generated, specifically including: Multimodal analysis was performed on the initial reference material to obtain the joint characterization vector; A parameterized abstraction network based on variational autoencoder is used to design and reconstruct the joint representation vector in a parameterized manner to obtain the reference factor parameter set; Based on the correlation between reference factors and the reference factor parameter set, a reference factor map is constructed. By using a graph neural network, reasoning is performed in the reference factor map to determine multiple reference factor combinations; A prompt word generation model is used to analyze and process the combination of reference factors to obtain creative prompt words.

[0012] Furthermore, multimodal analysis is performed on the initial reference material to obtain a joint representation vector, specifically including: A visual encoder is used to extract multi-scale visual feature maps from the initial reference material. A text encoder is used to extract text representations from the initial reference material, resulting in multi-granular text representations. Cross-modal alignment of multi-scale visual feature maps and multi-granular text representations is performed, and an image-text alignment matrix is ​​given. Based on a pre-built multi-label emotion classifier, emotion perception is performed on multi-granular text representations to obtain text emotion labels; Extract digital entities, problem entities, and effect entities from multi-granular text representations to construct core triples; By using a multimodal fusion gating network, the image-text alignment matrix, text sentiment tags, and core triples are projected into a unified latent space to determine the joint representation vector.

[0013] Furthermore, the graph neural network is a heterogeneous graph attention network, which includes a node type mapping layer, a heterogeneous graph attention layer, and a combined output layer; The node type mapping layer is used to receive the reference factor map and perform feature projection according to the type of each graph node in the reference factor map, mapping the reference factor parameters of different dimensions to a unified latent vector space and outputting the initial latent vector features of each reference factor. The heterogeneous graph attention layer is used to aggregate neighborhood messages based on the initial latent vector features of each reference factor, calculate the attention coefficients between different reference factors, and give the final latent state vector of each reference factor. The combined output layer is used to analyze and evaluate the combination of reference factors based on the final hidden state vector of each reference factor and the edge weights between different reference factors, to give a comprehensive best-selling score and to screen and determine the combination of reference factors.

[0014] Furthermore, through graph neural networks, inference is performed on the reference factor map to determine multiple combinations of reference factors, specifically including: Traverse each node in the reference factor graph, extract the reference factor parameters of each reference factor, and determine the original feature vector of each reference factor. Feature mapping is performed on the original feature vectors of each reference factor to unify the feature dimensions of each original feature vector, thereby obtaining the initial latent vector features of each reference factor. Based on the first-order neighbor set of each reference factor, and combined with multiple pre-constructed independent attention heads, neighborhood aggregation is performed in each attention head to obtain the attention coefficient of each reference factor in each independent attention head; For each reference factor, the initial latent vector features of each reference factor are fused by combining the attention coefficients of each reference factor in each independent attention head to obtain the final latent state vector of each reference factor. The final hidden state vector of each reference factor is evaluated by a pre-constructed factor evaluation function. The comprehensive best-selling score of each candidate combination of reference factors is determined by combining the edge weights between each reference factor. Based on a pre-constructed screening strategy, the comprehensive best-selling scores of the candidate combinations of reference factors are judged to obtain the reference factor combinations.

[0015] Furthermore, the overall best-selling score is determined through the following steps: By using the factor evaluation function, the final hidden state vector of each reference factor in the candidate combination of reference factors is analyzed and evaluated, the average fitness score of the reference factors in the candidate combination of reference factors is given, and the single factor fitness item is determined. Based on the edge weights of each reference factor in the candidate combination of reference factors, the mean correlation strength of each reference factor is analyzed to obtain the factor combination compatibility term. By balancing the single-factor fit term and the factor combination compatibility term, the comprehensive best-selling score of each reference factor candidate combination is obtained.

[0016] Furthermore, a prompt word generation model is used to analyze and process the combination of reference factors to obtain creative prompt words, specifically including: A prompt word generation model is used to semantically supplement the combination of reference factors to form initial task prompt words; The differences between the initial task prompts are analyzed, and a difference score is given, specifically as follows: Among them, S diff The difference score for each task prompt word is given, where n is the number of task prompt words, and D is the score for each prompt word. pq Let u be the cosine distance between the current task cue word p and the current task cue word q. p Let u be the semantic vector of the current task cue word p. q Let q be the semantic vector of the current task prompt word, and ||·||2 be the second norm. By judging the difference score, a random seed is introduced to make differential adjustments to the initial task prompt words to obtain the current task prompt words; The process of differential analysis of task prompts is repeated until the difference score reaches the preset difference range, thus generating creative prompts.

[0017] Furthermore, the evaluation dimensions include text-image semantic matching, core entity, visual perception, and compliance constraints. Based on the various evaluation dimensions, the task results are analyzed and judged, and the target evaluation results are given, including: Based on the semantic matching dimension of images and text, the similarity between the task results and creative prompts is calculated, and the image and text matching score of the task results is given. Based on the core entity dimension, a distortion discrimination model and a text recognition model are used to analyze and judge the distortion of core entities and text in the task results, and give the entity distortion score. The visual perception prediction model is used to judge the task results from the perspective of visual perception and obtain a comprehensive visual perception score. Based on the compliance constraint dimension, the non-compliant elements in the task results are analyzed and extracted to determine the compliance dimension score; The scores for image-text matching, entity distortion, overall visual perception, and compliance dimensions of the task results are integrated to form the target evaluation result.

[0018] Secondly, the present invention also provides a creative material generation device based on task arrangement, employing the creative material generation method based on task arrangement as described above, including: The data acquisition module is used to collect the initial generation requirements and initial reference materials input by the user; The task generation module is used to decompose and encapsulate the initial generation requirements and initial reference materials based on the preset task object pattern to form the target task; The creative extraction module is used to generate creative prompts by extracting reference factors from the initial reference materials. The reference factors are reusable design baseline elements obtained from the initial reference materials through joint representation vector extraction and parametric reconstruction by variational autoencoder. The creative prompts are obtained by combining and analyzing the reference factors through graph neural networks and prompt generation models. The task execution module is used to execute the target task and obtain the task result based on the creative prompts and combined with the multimodal large language model. The task evaluation module is used to analyze and judge the task results based on various result evaluation dimensions and provide the target evaluation results. The task optimization module is used to optimize creative prompts based on the target evaluation results, thereby obtaining target creative materials.

[0019] The present invention provides a method and apparatus for generating creative materials based on task orchestration, which has at least the following beneficial effects: (1) By extracting reference factors from the initial reference materials to generate creative prompts and executing the target task based on the creative prompts, the task results are obtained. Combining multiple result evaluation dimensions, the task results are analyzed and judged, and the creative prompts are optimized to obtain the target creative materials. By extracting reference factors from the initial reference materials, complex creative requirements are expressed in a unified structure, enabling accurate control over the style and differences of batch-generated requirements, improving the probability of using the generated results, forming a traceable and reproducible management process, and ensuring the stable application and implementation efficiency of creative materials.

[0020] (2) By extracting multimodal information, decoupling factors, combining reasoning and generating prompt words from the initial reference materials, the creative paradigm and the pixel content of the original materials are decoupled. Through VAE parameterized reconstruction, the concrete information in the initial reference materials is stripped away, the general reference factors for design are extracted, and the relationship between the factors is learned by using graph and graph neural network to verify the design logic of the reference factors, thus providing a data foundation for reorganizing and obtaining differentiated creative directions.

[0021] (3) By modeling the complex heterogeneous relationships between different types of reference factors, a reference factor map is formed. The heterogeneous graph attention network performs differentiated processing on different types of reference factors and edges with different semantics. At the same time, it considers the quality of the single factor itself and the compatibility of the pairing of factors, outputs highly adaptable reference factor combinations, and improves the overall usability of the output creative direction.

[0022] (4) By using heterogeneous graph attention network to aggregate neighbor features, we can not only learn the relationship between directly connected reference factors, but also explore the indirect potential matching relationship of "factor of factor", explore the indirect implicit matching rules of factors, expand the creative combination space, and enhance the richness of creativity.

[0023] (5) By setting up multiple independent attention heads, different attention heads can capture factor associations in different dimensions, and multiple attention heads can learn the association rules of the reference factor map from different perspectives. Attached Figure Description

[0024] Figure 1 A flowchart illustrating a creative material generation method based on task orchestration provided in an embodiment of the present invention; Figure 2 A flowchart for generating creative prompts provided in an embodiment of the present invention; Figure 3 A flowchart for obtaining the joint representation vector provided in an embodiment of the present invention; Figure 4 A flowchart illustrating the combination of reference factors is provided for embodiments of the present invention; Figure 5 This is a structural block diagram of a creative material generation device based on task orchestration provided in an embodiment of the present invention.

[0025] The modules are as follows: 201. Data Acquisition Module; 202. Task Generation Module; 203. Creative Extraction Module; 204. Task Execution Module; 205. Task Evaluation Module; 206. Task Optimization Module. Detailed Implementation

[0026] To better understand the above technical solutions, a detailed description of the solutions will be provided below in conjunction with the accompanying drawings and specific embodiments. Obviously, the described embodiments are merely some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0027] The terminology used in the embodiments of this invention is for the purpose of describing particular embodiments only and is not intended to limit the invention. The singular forms “a,” “the,” and “the” as used in the embodiments of this invention and the appended claims are also intended to include the plural forms, and “multiple” generally includes at least two unless the context clearly indicates otherwise.

[0028] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that an article or device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such an article or device. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the article or device that includes said element.

[0029] This invention provides a method for generating creative materials based on task orchestration. The method includes collecting initial generation requirements and initial reference materials input by the user; decomposing and encapsulating the initial generation requirements and initial reference materials based on a preset task object pattern to form a target task; generating creative prompts by extracting reference factors from the initial reference materials; executing the target task based on the creative prompts and combining a multimodal large language model to obtain task results; analyzing and judging the task results based on various result evaluation dimensions to provide target evaluation results; and optimizing the creative prompts based on the target evaluation results to obtain target creative materials.

[0030] By abstracting advertising creative needs into an "advertising creative task object," and completing a full process around this task object, including need understanding, knowledge retrieval, extraction of best-selling patterns, prompt keyword planning, model invocation, result evaluation, material post-processing, asset archiving, and process tracking, complex creative needs are expressed in a unified structure. This enables accurate control over the style and differences of batch-generated needs, increases the probability of using the generated results, and forms a traceable, reproducible, and reusable management process, ensuring the stable application and efficient implementation of creative materials.

[0031] like Figure 1 As shown in the figure, this embodiment of the invention provides a method for generating creative materials based on task orchestration, and the specific steps are as follows: S101: Collect initial generation requirements and initial reference materials input by the user.

[0032] The initial generation requirement consists of all relevant text data input by the user, and the initial reference material consists of image data input by the user. Both the initial generation requirement and the initial reference material are all relevant text and image data input multiple times by the user within a continuous period. For example, a user might input: "Generate 6 summer ad images suitable for Xiaohongshu (Little Red Book) for a sugar-free sparkling tea, referencing the refreshing style of a competitor's popular poster, but without directly copying the layout and copy," and upload a product image and a competitor's popular poster as initial reference material. Another example is a user uploading a product image as initial reference material and then inputting prompts such as "Generate a summer beverage ad image."

[0033] S102: Based on the preset task object pattern, the initial generation requirements and initial reference materials are decomposed and encapsulated to form the target task.

[0034] Among them, the task object pattern is a data encapsulation pattern used for creative material generation tasks. The task object pattern encapsulates multiple business parameters required for task execution, including reference materials, advertising platforms, brand information, product selling points, target audience, generation quantity, delivery specifications, etc. By summarizing all business parameters through the task object pattern, the business requirements are structured so that the generated results match the requirements of actual business advertising. Based on a preset task object pattern, the initial generation requirements and initial reference materials are decomposed and encapsulated to form the target task, which specifically includes: Based on the modality of the initial generated requirements and initial reference materials, the requirement text data and requirement image data in the initial generated requirements and initial reference materials are analyzed respectively to determine the requirement phrases and reference images; The required phrases, reference images and task object patterns are matched and the data is structured to form a primary task object; Iterate through the primary task objects, combine them with the task content constraints, and generate feedback instructions; Based on feedback instructions, the initial task object is supplemented and adjusted to determine the target task.

[0035] In one specific implementation, the initial generated requirements and initial reference materials include requirement text data and requirement image data. Based on the different modalities of the initial generated requirements and initial reference materials, the initial generated requirements are analyzed to obtain corresponding requirement phrases and reference images. For example, requirement phrases extracted from the requirement text data include "0-sugar sparkling tea," "Xiaohongshu," "6 images," "summer," "advertising image," "refreshing style," "refer to competitor's best-selling posters," and "do not copy the layout and copy." The reference image is the "competitor's best-selling poster" uploaded by the user. Matching each requirement phrase with the reference materials, placement platform, brand information, product selling points, target audience, generation quantity, and delivery specifications in the task object mode yields the following: the reference material is the user-uploaded "competitor's best-selling poster," the placement platform is "Xiaohongshu," the brand information is "0-sugar sparkling tea," the generation quantity is "6 images," but the delivery specifications, product selling points, and target audience do not match the corresponding requirement phrases, forming a primary task object. Since the task content in the primary task object is incomplete, the task content is judged according to the task content constraints. If there is task content that does not conform to the task content constraints, a feedback instruction is generated. The initial task object is supplemented by providing feedback instructions to the user or by extracting and integrating information from reference images and other required phrases, resulting in the target task. The task content constraints limit the value range of each task content in the task object pattern. For example, "Generation Quantity" must be non-empty and ≤100. Specific constraints include various conditions related to task content such as creative description, brand information, product selling points, target audience, visual style, layout requirements, reference materials, generation quantity, finished product size, and file size. These are set according to actual circumstances and are not limited here.

[0036] S103: Generate creative prompts by extracting reference factors from the initial reference materials.

[0037] Among them, the reference factors are reusable design baseline elements obtained from the initial reference materials through joint representation vector extraction and variational autoencoder parameterized reconstruction, and the creative prompts are obtained by combining and analyzing the reference factors through graph neural networks and prompt generation models.

[0038] Specifically, refer to Figure 2 By extracting reference factors from initial reference materials, creative prompts are generated, including: Multimodal analysis was performed on the initial reference material to obtain the joint representation vector; A parameterized abstraction network based on variational autoencoder is used to design and reconstruct the joint representation vector in a parameterized manner to obtain the reference factor parameter set; Based on the correlation between reference factors and the reference factor parameter set, a reference factor map is constructed. By using a graph neural network, reasoning is performed in the reference factor map to determine multiple reference factor combinations; A prompt word generation model is used to analyze and process the combination of reference factors to obtain creative prompt words.

[0039] It's important to understand that prompt word generation models are a type of specialized generative model designed for image / video generation tasks. They take brief, coarse-grained natural language input from users, and after semantic parsing, detail expansion, and style completion, output professional prompt words adapted to the input format of text-to-image and text-to-video models. These typically include positive prompt words and can simultaneously output negative prompt words. Prompt word generation models transform simple descriptions into structured prompt text that the generation model can efficiently understand, eliminating the problem of poor generation results caused by users' insufficient prompt word writing skills. In this example, the prompt word generation model is built on an instruction-tuned architecture based on a large language model or a multimodal large language model.

[0040] In one specific implementation, the initial reference materials can be multiple popular posters, product promotional images, and short video covers. These materials include image modalities as well as text modalities such as text, tags, titles, and product descriptions. A visual encoder converts the image modalities into image feature vectors. A text encoder extracts the text modalities from the initial reference materials, such as poster text and tag semantics, to obtain text feature vectors. The image and text feature vectors are then fused through cross-modal attention to achieve multimodal joint representation fusion, outputting a joint representation vector. In a specific example, the initial reference materials are three popular summer sunscreen skincare product posters: Poster A: Light blue refreshing background, model holding sunscreen, text: "Lightweight and non-comedogenic, summer sun protection"; Poster B: Beach scene, high-saturation blue sky, text: SPF50+ outdoor protection; Poster C: Minimalist Instagram style, close-up of a white glass bottle product, text: Hydrating and nourishing sunscreen. A visual encoder extracts image feature vectors from the poster, containing information such as a refreshing light blue color scheme, a seaside scene, and a close-up product composition. Simultaneously, a text encoder extracts text feature vectors from the poster, containing information such as summer, SPF50+, lightweight and non-comedogenic properties, and moisturizing and skin-nourishing effects. Finally, cross-modal attention fusion is used to combine the visual and textual features, resulting in a unified joint representation vector that records both the visual style and the semantic meaning of the text.

[0041] Next, the joint representation vector is input into the Variational Auto-Encoder (VAE). The VAE encoder maps the high-dimensional joint representation vector to a Gaussian distribution, sampling to obtain the latent space vector. Simultaneously, the VAE decoder performs parameterized reconstruction, decoupling and decomposing multiple independent reference factors from the latent space vector, outputting a set of reference factor parameters. For example, reference factors include visual style factors, composition factors, lighting factors, color factors, textual semantic factors, and scene factors.

[0042] The reference factor parameter set is specifically represented as follows: Where Z is the joint representation vector, VAE encoder () represents the VAE encoder, and Sample[] represents the re-parameter sampling. decoder () represents the VAE decoder, and F represents the set of reference factor parameters.

[0043] In a specific example, the input joint representation vector is decoupled and reconstructed by VAE, and the output reference factor parameter set includes color factors (main color #87CEEB light sky blue, low saturation refreshing color scheme), scene factors (summer beach, outdoor sunlight environment), composition factors (product center close-up + people in the background), light and shadow factors (bright natural light, soft diffuse reflection), and text semantic factors (lightweight, non-comedogenic, SPF50+, moisturizing and skin-nourishing), etc. Each reference factor is accompanied by a parameter weight, which represents the intensity of the corresponding reference factor in the initial reference material.

[0044] Each reference factor is defined as a graph node, and the node attributes are the parameters and weights corresponding to the reference factor. Based on a pre-built library of trending content, the co-occurrence probability of each reference factor is analyzed, and edges between graph nodes are constructed. The edge weights represent the correlation strength between two reference factors. If the edge between two reference factors is strongly correlated, it means that the two reference factors frequently appear together; if the edge between two reference factors is weakly correlated, it means that the two reference factors rarely appear together, ultimately forming a reference factor graph.

[0045] In a specific example, the graph node set includes five nodes: Node 1: light sky blue color scheme; Node 2: summer seaside scene; Node 3: centered close-up of the product; Node 4: bright natural light; Node 5: lightweight and non-comedogenic semantics. The edge weight between Node 1 and Node 2 is 0.9, indicating that the light sky blue color scheme and the summer seaside scene appear frequently together, showing a strong positive correlation; the edge weight between Node 2 and Node 4 is 0.85, indicating that the seaside scene is often paired with bright natural light; the edge weight between Node 3 and Node 5 is 0.7, indicating that the product close-up image is often paired with the copywriting "lightweight and hydrating". The resulting reference factor graph records each reference factor and the strength of the pairings between them.

[0046] This study employs a heterogeneous graph attention network within graph neural networks to perform inference within a reference factor graph. For each graph node, features from neighboring nodes are aggregated to learn the compatibility between reference factors. Based on the scores of each graph node and edge association weights, multiple subsets of highly suitable reference factors are selected, resulting in multiple sets of reference factor combinations. The heterogeneous graph attention network automatically avoids weakly correlated factors, preventing strongly conflicting factors from being placed in the same combination.

[0047] In a specific example, given a reference factor map for sunscreen, the heterogeneous graph attention network inference outputs three sets of reference factor combinations: Reference factor combination 1 is {light sky blue color scheme, summer scene at the beach, bright natural light, composition of people + product, SPF50+ protection copy}, corresponding to the creative direction of an outdoor sunscreen route at the beach; Reference factor combination 2 is {light blue and white tone, soft light by the window indoors, close-up of the product in the center, copy for lightweight and non-comedogenic, moisturizing and nourishing}, corresponding to the creative direction of a refreshing skincare route at home; Reference factor combination 3 is {low saturation Morandi blue, solid color background, close-up of the product, extremely short copy}, corresponding to the creative direction of a minimalist and sophisticated Instagram route.

[0048] Finally, a prompt word generation model is used to translate the structured reference factor parameters into natural language, supplementing details, optimizing word order, and adapting to the format of the text-to-image model, outputting creative prompt words that can be directly used for AI drawing. For example, inputting reference factor combination 2 into the prompt word generation model, with the following structured factor parameters: color: light blue and white tone; scene: indoor windowside; lighting: soft windowside light; composition: centered close-up of sunscreen product; text semantics: lightweight, non-comedogenic, moisturizing and nourishing. The final creative prompt word is: "E-commerce poster, summer sunscreen product poster, light blue and white refreshing color scheme, indoor windowside scene, soft natural light diffused lighting, centered close-up of sunscreen glass bottle product, clean and simple, Instagram fresh style, text highlighting lightweight and non-comedogenic, moisturizing and nourishing, high-definition commercial photography."

[0049] Furthermore, referring to Figure 3 Multimodal analysis is performed on the initial reference material to obtain a joint representation vector, which specifically includes: A visual encoder is used to extract multi-scale visual feature maps from the initial reference material. A text encoder is used to extract text representations from the initial reference material, resulting in multi-granular text representations. Cross-modal alignment of multi-scale visual feature maps and multi-granular text representations is performed, and an image-text alignment matrix is ​​given. Based on a pre-built multi-label emotion classifier, emotion perception is performed on multi-granular text representations to obtain text emotion labels; Extract digital entities, problem entities, and effect entities from multi-granular text representations to construct core triples; By using a multimodal fusion gating network, the image-text alignment matrix, text sentiment tags, and core triples are projected into a unified latent space to determine the joint representation vector.

[0050] In one specific implementation, the visual encoder employs ViT, simultaneously extracting visual feature maps of different resolutions from the initial reference material: shallow (edges, colors, composition), mid-layer (objects, elements), and deep (overall style, lighting and atmosphere), while preserving spatial location information. The shallow visual feature map includes color blocks, outlines, and lighting; the mid-layer visual feature map includes the product itself and decorative elements; and the deep visual feature map includes the overall atmosphere and compositional layout. In other examples, the visual encoder can also be other models such as convolutional neural networks, without limitation. In a specific example, the initial reference material is a beauty poster image, with a foundation product as the main image, a soft pink background, a close-up of a person's face, and a centered composition. The pink hue, highlight areas, and outline edges are extracted from the initial reference material to obtain the shallow visual feature map. The foundation bottle, the person, and the text placement areas are located to obtain the mid-layer visual feature map. The deep visual feature map represents the style characteristics of "soft lighting, centered main visual, and atmospheric beauty." By extracting the accompanying text (poster copy, promotional slogans) from the initial reference materials, we output representation vectors at the character, word, and sentence levels, taking into account both keyword entities and overall sentence semantics. For example, the poster copy: "24-hour makeup wear, say goodbye to cakey and patchy makeup, create a clear and creamy complexion," corresponds to the multi-granular text representations {makeup wear, 24 hours, cakey and patchy makeup, clear and creamy complexion}.

[0051] The semantic similarity between each spatial feature in the visual feature map and each token in the text representation is calculated. Cross-modal alignment is performed on the multi-scale visual feature map and the multi-granular text representation to obtain the image-text alignment matrix. Each element in the image-text alignment matrix represents the degree of matching between each position in the visual feature map and the corresponding text. For example, text tokens: 24 hours, cakey makeup, creamy skin; visual feature maps: product bottle, facial skin area, background pink soft light area. The facial skin area feature has a high similarity to the "creamy skin" token, and the corresponding value in the matrix is ​​large; the product bottle area has a high matching degree with the "makeup lasts" token. The image-text alignment matrix is ​​formed to record the correspondence between each area and each text in the initial reference material, establishing a mapping relationship between image and text elements. The image-text alignment matrix distinguishes between the "effect of the text description" and the "object displayed in the image," avoiding a disconnect between image and text information.

[0052] A multi-label emotion classifier is a pre-trained classification model that takes a text representation as input and outputs corresponding emotion labels simultaneously. It doesn't just output a single emotion; the text emotion labels represent the psychological feelings the text conveys to the audience. For example, if the input text representation is "24-hour makeup wear, say goodbye to cakey and patchy makeup, create a clear, creamy complexion," the multi-label emotion classifier will output the text emotion labels as {expectation, reassurance, pleasure, pain relief}. Here, expectation corresponds to "clear, creamy complexion," reassurance corresponds to "24-hour makeup wear," and pain relief corresponds to "say goodbye to cakey and patchy makeup." The multi-label emotion classifier is a classification model that allows a single sample to output multiple emotion labels from text, speech, or image samples. It outputs a set of 0-1 Boolean labels or the confidence probability of each emotion category. Multi-label emotion classifiers can be trained using model architectures such as BERT-based, TextCNN, and Bi-LSTM+Attention.

[0053] Entity recognition is performed on the text representation to extract digital entities, problem entities, and effect entities. Digital entities include numerical values, durations, specifications, etc. Problem entities represent user concerns, and effect entities represent the product's promised results. Core triples are constructed. For example, if the text representation is "24-hour makeup wear, say goodbye to cakey and patchy makeup, create a clear and creamy complexion", the extracted digital entities are 24 hours, the problem entity is cakey and patchy makeup, and the effect entities are "wearing makeup, clear and creamy complexion". Core triples are constructed as [cakey and patchy makeup, solution, (24-hour makeup wear, clear and creamy complexion)].

[0054] By setting gating weights in the multimodal fusion gating network, the proportion of each type of information is adaptively controlled. The image-text alignment matrix, text sentiment labels, and core triples are mapped to the same-dimensional latent vector space by combining the projection layer. The gating unit automatically adjusts the weights of visual alignment information, sentiment information, and core triple information according to the material and fuses them to obtain the final joint representation vector.

[0055] By identifying information such as title structure, selling point order, emotional triggers, main product position, background style, color scheme, text area, and overall composition in the initial reference materials, and combining this with a parametric abstraction network based on a variational autoencoder, this information is abstracted into multiple reusable but not directly copyable "reference factors." Through sequential visual layer analysis → composition layer analysis → color and lighting analysis → marketing copy structure analysis → scene and emotion analysis → factor desensitization and structuring, multi-layered structural analysis is performed on the initial reference materials to form reference factors including composition structure factors, main layout factors, color matching factors, lighting texture factors, atmosphere element factors, and advertising level factors. For example, extracting reference factors from the initial reference materials and describing them verbally results in phrases such as "use ice cubes and water droplets to enhance the feeling of coolness," "light-colored background highlights the low-burden selling point," "the title first presents the benefit and then connects to the usage scenario," and "the main product is located in the lower center of the image."

[0056] In a specific example, the SAM image segmentation model and subject detection model are used to perform structured analysis on the initial reference materials. This identifies the coordinate proportions and center / left / right positioning patterns of the product subject, figures, and decorative elements in the initial reference materials, outputting structured position labels such as "subject centered," "bottom left white space," and "top atmosphere area." Simultaneously, it distinguishes between the foreground product layer, the midground atmosphere decoration layer, and the background solid color / gradient layer, identifying the visual stacking structure in the reference materials to form subject layout factors. Through image spatial transformation analysis, golden ratio detection, and grid layout recognition, compositional paradigms in the reference materials are extracted. Combined with image saliency detection algorithms, a heatmap is calculated to locate the user's visual focus, determining the white space distribution patterns and visual guidance patterns of the reference materials, forming compositional structural factors. For example, compositional paradigms include symmetrical composition, rule of thirds composition, central focus composition, and diagonal dynamic composition; white space distribution patterns include top white space for titles, bottom white space for selling points, and side white space for atmosphere extension; and visual guidance patterns include light, gradients, and element arrangement pointing towards the product subject, forming a visual focus path. By employing color mean clustering and HSV color space quantization extraction, the primary, secondary, and accent colors of the initial reference materials are collected to form a matching color chart for the reference materials, resulting in color matching factors. Light and shadow texture factors are obtained by identifying exposure styles such as soft light, hard light, side-backlight, transparent lighting, and high transparency in the reference materials. The marketing information structure in the initial reference materials is extracted, including title position, the order of selling points, the presentation format of benefits, and the hierarchical priority of brand logo, product display, scene atmosphere, and copywriting information, forming advertising hierarchy factors. For example, the creative rule corresponding to the advertising hierarchy factors is the visual communication order of atmosphere first, then product, and finally selling points. A multi-label emotion classifier is used to output text emotion labels to determine atmosphere element factors, such as refreshing summer, minimalist sophistication, healing and soft, trendy and youthful, and light luxury texture, which are used to constrain the style of subsequent prompts.

[0057] By using a parametric abstraction network based on variational autoencoders, the joint representation vector is desensitized and structurally encapsulated. Instead of extracting specific elements, patterns, or text from the reference materials, it extracts only structural patterns, layout paradigms, color logic, lighting rules, and marketing levels to obtain different types of reference factors.

[0058] Furthermore, the graph neural network is a heterogeneous graph attention network, which includes a node type mapping layer, a heterogeneous graph attention layer, and a combined output layer; The node type mapping layer is used to receive the reference factor map and perform feature projection according to the type of each graph node in the reference factor map, mapping the reference factor parameters of different dimensions to a unified latent vector space and outputting the initial latent vector features of each reference factor. The heterogeneous graph attention layer is used to aggregate neighborhood messages based on the initial latent vector features of each reference factor, calculate the attention coefficients between different reference factors, and give the final latent state vector of each reference factor. The combined output layer is used to analyze and evaluate the combination of reference factors based on the final hidden state vector of each reference factor and the edge weights between different reference factors, to give a comprehensive best-selling score and to screen and determine the combination of reference factors.

[0059] By employing a heterogeneous graph attention network for graph analysis, for each reference factor, the features of compatible neighbors are absorbed and the features of conflicting factors are suppressed, outputting the final hidden state vector containing collocation knowledge.

[0060] Furthermore, referring to Figure 4 Through graph neural networks, inference is performed on the reference factor map to determine multiple combinations of reference factors, specifically including: Traverse each node in the reference factor graph, extract the reference factor parameters of each reference factor, and determine the original feature vector of each reference factor. Feature mapping is performed on the original feature vectors of each reference factor to unify the feature dimensions of each original feature vector, thereby obtaining the initial latent vector features of each reference factor. Based on the first-order neighbor set of each reference factor, and combined with multiple pre-constructed independent attention heads, neighborhood aggregation is performed in each attention head to obtain the attention coefficient of each reference factor in each independent attention head; For each reference factor, the initial latent vector features of each reference factor are fused by combining the attention coefficients of each reference factor in each independent attention head to obtain the final latent state vector of each reference factor. The final hidden state vector of each reference factor is evaluated by a pre-constructed factor evaluation function. The comprehensive best-selling score of each candidate combination of reference factors is determined by combining the edge weights between each reference factor. Based on a pre-constructed screening strategy, the comprehensive best-selling scores of the candidate combinations of reference factors are judged to obtain the reference factor combinations.

[0061] In a specific example, the reference factor parameters for graph nodes include RGB color, scene label vector, composition parameters, weights, etc. Different types of graph nodes have different dimensions in their original feature vectors, which need to be projected to a unified dimension to form initial latent vector features.

[0062] The initial latent vector features of each reference factor are specifically represented as follows: in, Let τ(v) be the initial latent vector feature of the i-th reference factor. i ) represents graph node vi type For node type τ(v) i The corresponding learnable projection matrix, For graph node v i The original feature vector, For node type τ(v) i The corresponding bias vector.

[0063] Attention coefficient, specifically expressed as: in, Let exp() be the attention coefficients of reference factor i and reference factor j under the k-th independent attention head, and let exp() be the exponential function. Let N be the unnormalized attention scores of reference factor i and reference factor j under the k-th independent attention head. i Let a be the set of first-order neighbors of reference factor i, where i, j, and z are the indices of each reference factor in the reference factor map, and a T(k) Let be the attention vector under the k-th independent attention head, and be learnable parameters. Let be the initial latent vector features of the i-th reference factor. For the initial latent vector features of the j-th reference factor, w ij Let represent the edge weights of the i-th and j-th reference factors in the reference factor graph. LeakyReLU() is a leaky rectified linear unit, which is an activation function. [·||·] represents the vector concatenation operation.

[0064] By simultaneously integrating node features and graph edge weights during the calculation of attention coefficients, the ability to model the correlation between reference factors is improved.

[0065] The final hidden state vector of each reference factor is specifically represented as follows: in, Let N be the final hidden state vector of the i-th reference factor, K be the total number of independent attention heads, k be the number of the independent attention head, σ() be the activation function, and N be the number of independent attention heads. i Let i be the set of first-order neighbors of the reference factor i. Let i be the attention coefficients of reference factor i and reference factor j under the k-th independent attention head. Let be the initial latent vector features of the j-th reference factor. This means concatenating the vectors output by K independent attention heads.

[0066] By setting up multiple independent attention heads, different attention heads can capture factor relationships in different dimensions. For example, one attention head focuses on capturing the matching relationship between color and scene; another attention head focuses on capturing the constraint relationship between light and shadow and composition; multiple heads learn the correlation rules of the reference factor map from different perspectives.

[0067] The overall best-selling score is determined through the following steps: By using the factor evaluation function, the final hidden state vector of each reference factor in the candidate combination of reference factors is analyzed and evaluated, the average fitness score of the reference factors in the candidate combination of reference factors is given, and the single factor fitness item is determined. Based on the edge weights of each reference factor in the candidate combination of reference factors, the mean correlation strength of each reference factor is analyzed to obtain the factor combination compatibility term. By balancing the single-factor fit term and the factor combination compatibility term, the comprehensive best-selling score of each reference factor candidate combination is obtained.

[0068] The overall best-selling score is expressed as follows: Where Score(S) is the comprehensive best-selling score of the reference factor candidate combination S, |N S | represents the number of reference factors in the candidate combination S, V i This is the i-th reference factor (corresponding to the i-th graph node). Let w be the final hidden state vector of the i-th reference factor, φ() be the factor evaluation function, and w be the value of the reference factor. ij Let E be the edge weights of the i-th and j-th reference factors in the reference factor graph. S Let S be the set of associated edges in the candidate combination of reference factors, |N E | represents the number of associated edges in the candidate combination S of the reference factor, and λ is the balance coefficient used to control the weights of "single factor quality" and "compatibility between factors". In this example, the factor evaluation function uses an MLP network, taking the final hidden state vector as input and outputting the fit score of a single factor. In other examples, the factor evaluation function can use other network architectures, and there are no restrictions on this. For single-factor fitting terms, This is a compatibility term for the factor.

[0069] It should be noted that |N E |=0 means that the factor combination compatibility term is 0 when there are no associated edges in the candidate combination of reference factors.

[0070] In one specific implementation, the original feature vector of graph node 1 [light sky blue color scheme, type: color factor] is [135, 206, 235, 0.92], including RGB values ​​and factor importance weights; the original feature vector of graph node 2 [seaside summer scene, type: scene factor] is [0.9, 0.8, 0.1], including seaside label score, summer label score, and city scene score. The two graph nodes have different feature dimensions in their original feature vectors. They are projected using the learnable projection matrix and bias vector corresponding to their node types, respectively, to output initial latent feature vectors of the same dimension, which then enter the heterogeneous graph attention layer.

[0071] In a specific example, taking graph node v2 [summer seaside scene, type: scene factor] as the starting node, the corresponding first-order neighbor set includes graph node v1 [light sky blue color scheme, w21=0.90], graph node v4 [bright natural light, w24=0.85], and graph node vj [dark lighting, w2j=-0.80]. The unnormalized attention score [e21, e24, e2j] for each neighbor pair is calculated, where the value of e2j is very small, and the attention coefficient has almost no effect after normalization. By calculating the attention coefficient, graph node v2 receives features from the two neighbor nodes, light sky blue and bright natural light, and fuses them to update the final hidden state vector. The final hidden state vector encodes the knowledge that "a seaside scene should be paired with light sky blue and bright light, avoiding dark lighting." For each reference factor, features from compatible neighbors are absorbed, features from conflicting factors are suppressed, and the final hidden state vector containing matching knowledge is output.

[0072] Finally, the final hidden state vectors of each reference factor are input into a pre-constructed factor evaluation function for evaluation. Combining the edge weights between each reference factor, the overall best-selling score for each candidate combination of reference factors is determined. A screening strategy is used to determine the reference factor combinations. For example, the candidate combination S1 {light sky blue, summer at the beach, bright natural light, portrait + product composition, SPF50 + copywriting} has an overall best-selling score of Score(S1) = 0.87; the candidate combination S2 {summer at the beach, dark lighting} has an overall best-selling score of Score(S2) = 0.21 and is filtered out. The final high-scoring set is then fed into the prompt word generation model to generate creative prompt words.

[0073] The reference factor parameter set is output through a VAE, and co-occurrence associations of factors are fused to construct a reference factor map. Then, a heterogeneous graph attention network is used to project different types of reference factors, and combined with multi-head heterogeneous graph attention message aggregation, the compatibility and conflict between reference factors are learned to obtain multiple sets of reference factor combinations. Finally, these are fed into a prompt word generation model to translate the structured factor set into creative prompt words.

[0074] Furthermore, a prompt word generation model is used to analyze and process the combination of reference factors to obtain creative prompt words, specifically including: A prompt word generation model is used to semantically supplement the combination of reference factors to form initial task prompt words; The differences between the initial task prompts are analyzed, and a difference score is given. By judging the difference score, a random seed is introduced to make differential adjustments to the initial task prompt words to obtain the current task prompt words; The process of differential analysis of task prompts is repeated until the difference score reaches the preset difference range, thus generating creative prompts.

[0075] In one specific implementation, the reference factor combination includes scene factors, composition factors, lighting factors, emotion factors, and core triples. The cue word generation model generally employs a finely tuned large language model. The cue word generation model receives the reference factor combination, performs semantic expansion, modifier completion, word order organization, and supplements style modifications and detailed descriptions, transforming the discrete structured factors into initial task cue words in natural language form.

[0076] In a specific example, the reference factor combination is: {Scene factor: beauty foundation poster, e-commerce main image; Composition factor: centered composition, product occupies 1 / 3 of the frame; Light and shadow factor: soft pink tone, soft diffused light; Emotion factor: anticipation, pain point relief; Core triplet: (creases, solution, 24-hour wear)}. The prompt word generation model provides semantic supplementation and outputs 3 initial task prompt words: ① E-commerce beauty foundation poster, centered composition, product occupies one-third of the frame, soft pink tone with diffused light, conveying anticipation, showcasing 24-hour wear, improving creases, high-definition commercial photography. ② Commercial poster, foundation product placed in the center, pink soft light atmosphere, showing 24-hour wear effect, solving the pain point of creases, high-definition realism. ③ Beauty e-commerce main visual, centered composition, soft pink light, highlighting the foundation's 24-hour wear, alleviating creases, high-end commercial photography. The three initial task prompts mentioned above are largely similar in meaning, with some differences, but the overall differences are insufficient.

[0077] Based on the semantic vectors corresponding to each initial task prompt, the cosine distance between each pair of semantic vectors is calculated as a basic difference metric. Then, the average of the basic difference metrics is used as the overall difference score for each initial task prompt.

[0078] The difference score is specifically expressed as follows: Among them, S diff The difference score for each task prompt word is given, where n is the number of task prompt words, and D is the score for each prompt word. pqLet u be the cosine distance between the current task cue word p and the current task cue word q. p Let u be the semantic vector of the current task prompt word p. q Let q be the semantic vector of the current task prompt word, and ||·||2 be the second norm.

[0079] By quantifying the semantic differences between pairs of multiple task prompts, a difference score is calculated. A difference score close to 0 indicates a high degree of semantic overlap between the task prompts, resulting in highly similar generated images; a larger difference score indicates a higher degree of creative differentiation between the task prompts. A preset difference range is set to judge the difference score; excessively high difference scores will deviate from the reference factor.

[0080] During the inference process of the prompt word generation model, different random seeds change the sampling distribution. While preserving the original reference factors, modifiers are randomly replaced, sentence structures are varied, and different detailed elements are added to diversify the prompt words. The current task prompt word, reference factor combination, and random seed are fed into the prompt word generation model. The model constraint must retain the reference factor combination, randomly modify secondary modifier descriptions, composition details, and atmosphere descriptions, and output the adjusted task prompt words. For example, if the difference score = 0.193 < 0.25, a difference adjustment is triggered, and a random seed is set. The model retains the reference factors: foundation liquid poster, centered composition, soft pink light, 24-hour wear, and solves the problem of creasing and patchiness; it changes the secondary descriptions and outputs the adjusted task prompt words for this round as: ① E-commerce foundation liquid commercial poster, centered composition, product placed in 1 / 3 of the image, soft pink diffused light, dreamy atmosphere, highlighting 24-hour wear effect, improving creasing and patchiness, delicate skin texture, 8K commercial photography. ② A Xiaohongshu-style beauty main image, featuring the foundation product centered in a soft pink-purple lighting environment, highlighting its long-lasting wear and addressing the issues of creasing and patchiness, resulting in a translucent, creamy skin texture, reminiscent of high-end Instagram photography. ③ An e-commerce product promotional poster, centering on the foundation, with a soft pink lighting, emphasizing 24-hour long-lasting wear, eliminating creasing and patchiness, with a clean, minimalist background, creating a magazine-quality feel. After adjustments, the task prompts for this round are obtained. The difference scores for each task prompt are recalculated. If the difference scores meet the preset range, the current task prompts are used as creative prompts. If the difference scores do not meet the preset range, the current task prompts are adjusted based on a random seed.

[0081] By retaining reference factors and ensuring creative direction, multiple distinctive creative prompts are generated through quantification of difference scores and random seed iteration, thus avoiding the generation of a large number of highly similar posters.

[0082] In one specific implementation, based on the combination of reference factors in the initial task prompts, multiple random seed variables are configured for each reference factor. Random seed variables of different dimensions are randomly selected to form the prompts for each reference factor. In a specific example, the random seed variables include random scenes, random perspectives, random composition, random lighting, and random elements. Random scenes include outdoor summer street scenes, indoor desktop scenes, pure white minimalist studio scenes, transparent water mist scenes, minimalist Instagram-style scenes, and cold drink bar scenes; random perspectives include eye-level close-ups, 45° overhead shots, low-angle shots, partial detail compositions, and full-screen atmospheric compositions; random composition includes central compositions, rule-of-thirds compositions, minimalist compositions with negative space, and diagonal dynamic compositions; random lighting includes soft, transparent light, hard sunlight, cool-toned clear light, frosted diffused light, and Instagram-style creamy light; random elements include ice cubes, bubbles, water mist, crushed ice, light-colored gradient backgrounds, and bokeh. Controllable and differentiated constraints are configured for the extraction of random seed variables to avoid invalid randomness. Controllable differentiation constraints include fixed invariant items (product subject, core selling points, brand color scheme, platform specifications), completely random items (scene, perspective, atmosphere, decoration, lighting, composition details), and mutually exclusive constraints, meaning that highly similar compositions and similar scene combinations will not appear simultaneously in the same batch, automatically removing duplicates. Random seed variables are extracted to form a set of random dimension combinations. Based on each set of random dimension combinations, they are spliced ​​and merged to output corresponding creative prompts. For example, the resulting reference factor combinations include: ① Pure white minimalist studio shot + centered composition + transparent soft light + clean and bright image; ② Summer window table scene + 45° overhead shot + natural sunlight + slight water mist atmosphere; ③ Cool-toned icy scene + low-angle upward shot + hard light contrast + ice cube decoration; ④ Instagram light color gradient background + rule of thirds composition + creamy soft light + minimalist white space; ⑤ Outdoor tree shade dappled light scene + dynamic diagonal composition + bokeh + fresh summer feel; ⑥ Transparent water mist full-screen atmosphere + close-up detail composition + high transparency texture + minimalist high-end style. Subsequently, by combining user-owned brands, product images, Xiaohongshu platform rules, and the aforementioned reference factors, multiple sets of differentiated creative prompts are generated. For example, six random seeds can be generated: "Quenching thirst and controlling sugar in the office afternoon," "Refreshing hydration outdoors in summer," "Lightweight drink after a workout," "Sugar-free alternative for parties," "Lightweight afternoon tea pairing," and "Refreshing choice for stocking up in the refrigerator," each generating corresponding independent creative prompts. These six sets of creative prompts achieve zero-homogeneity batch image generation through random scenes, random perspectives, random lighting, and accompanying structural images.

[0083] S104: Based on the creative prompts and combined with a multimodal large language model, perform the target task and obtain the task results.

[0084] In one specific implementation, creative prompts are input into a multimodal large language model. The multimodal large language model parses the creative prompts, verifies their semantic completeness, and detects any missing key business constraints. It then assembles the textual prompts, composition parameters, color constraints, and multiple business parameters from the target task into a complete request body recognizable by the image generation model. Negative prompts tailored to the scenario are added, and distorted, aberrant, and low-quality risks are filtered out. The assembled request is then sent to the multimodal large language model to execute the target image generation task, yielding the corresponding task result.

[0085] It's important to understand that the Multimodal Large Language Model (MLLM) is a multimodal generative foundational model built upon the Large Language Model (LLM). It uses the LLM as its backbone network, adding modal encoders for images, audio, and other modalities, along with modal projection modules. In addition to processing text tokens, it converts non-text signals such as images, audio, and video into embedded features aligned with the language model by adding visual encoders, audio encoders, and other modal encoders. These image / audio / video features are mapped to the semantic space of the language model, and then processed by the LLM backbone for understanding, reasoning, and generation. It can simultaneously receive, understand, and output multiple modalities, including text and images, achieving cross-modal perception and generation.

[0086] Among them, the Large Language Model (MLM) is a large-scale deep learning model based on the Transformer decoder architecture, pre-trained on massive text corpora. By learning the probability distribution and semantic logic between words, sentences, and paragraphs in the text, it can understand natural language input and generate coherent text output that conforms to human language habits. It can perform various language tasks such as understanding, generation, reasoning, induction, rewriting, and question answering, and is a core foundational model of generative artificial intelligence. The MLM typically performs specific tasks under the guidance of cue text. Cue text is an initial text or text fragment provided to the MLM to stimulate the model to produce corresponding output. Through cue text, the expected task can be explicitly told to the MLM, such as answering a question or writing an article. At the same time, cue text can provide the MLM with necessary background information and context, enabling it to understand the logic, style, theme, or stance that should be followed when generating content. Furthermore, cue text can also stimulate the MLM to demonstrate its inherent knowledge reserves or specific language abilities, such as explaining complex concepts or citing regulations.

[0087] S105: Analyze and judge the task results based on each result evaluation dimension, and give the target evaluation results.

[0088] The evaluation dimensions include text-image semantic matching, core entity, visual perception, and compliance constraints. Specifically, based on each outcome evaluation dimension, the task results are analyzed and judged, and the target evaluation results are given, including: Based on the semantic matching dimension of images and text, the similarity between the task results and creative prompts is calculated, and the image and text matching score of the task results is given. Based on the core entity dimension, a distortion discrimination model and a text recognition model are used to analyze and judge the distortion of core entities and text in the task results, and give the entity distortion score. The visual perception prediction model is used to judge the task results from the perspective of visual perception and obtain a comprehensive visual perception score. Based on the compliance constraint dimension, the non-compliant elements in the task results are analyzed and extracted to determine the compliance dimension score; The scores for image-text matching, entity distortion, overall visual perception, and compliance dimensions of the task results are integrated to form the target evaluation result.

[0089] In one specific implementation, an image encoder encodes the task result to obtain the corresponding image code, and a text encoder encodes the creative prompt to obtain the corresponding text vector. The cosine similarity between the image vector and the text vector is calculated to give the image-text matching score of the task result. By calculating the cosine similarity, the semantic matching dimension of image and text is evaluated to measure the degree of alignment between the task result and the creative prompt, avoiding problems such as mismatched images and text or off-topic creative content.

[0090] In the core entity dimension, an object detection model is used to detect entities in the task results to identify target product entities. Then, a pre-trained distortion discrimination model is used to detect distortion in the target product entities in the task results, outputting an image entity distortion score. A higher image entity distortion score indicates more severe distortion of the target product entity. Simultaneously, OCR is used to recognize the text in the task results, obtaining a set of text entities, and the text recognition accuracy is calculated. The text recognition accuracy and image entity distortion score are weighted and fused to obtain the entity distortion score. By verifying the existence, reasonableness, and absence of distortion of product entities in the task results, and the recognizability of key selling point text, common problems such as product distortion, object disappearance, and garbled text are resolved.

[0091] In the visual perception dimension, the LAION Aesthetics Predictor (LAP) model is used to evaluate the commercial visual perception performance of the task results, including image composition, lighting, color, texture, and depth of field, to obtain a comprehensive visual perception score. The LAION visual perception prediction model is a regression model for image visual perception evaluation. The model architecture consists of a frozen CLIP ViT-L / 14 image encoder connected in series with a trainable multilayer perceptron regression head.

[0092] In terms of compliance constraints, a content moderation model is used to extract image violation elements from the task results, outputting the image violation confidence score; OCR is used to scan and extract text violation elements from the task results, outputting the text violation confidence score. The image violation confidence scores and text violation confidence scores are weighted and fused to give a compliance dimension score.

[0093] By combining the pre-set fusion coefficients, the text-image matching score, entity distortion score, comprehensive visual perception score, and compliance dimension score of the task results are weighted and fused to form the target evaluation result.

[0094] S106: Optimize creative prompts based on the target evaluation results to obtain target creative materials.

[0095] Specifically, based on the target evaluation results, the creative prompts are optimized to obtain target creative materials, including: Based on the pre-set standard material thresholds, the target evaluation results are judged, and a first judgment result is given; Based on the first judgment result, extract the dimensional scores of each result evaluation dimension; The scores of each evaluation dimension are assessed to determine the core adjustment direction; By adjusting the core direction, optimizing creative prompts, and repeatedly generating task results and obtaining target evaluation results, the first judgment result is qualified, and the target creative material is obtained.

[0096] In one specific implementation, if the target evaluation result is greater than or equal to the standard material threshold, the first judgment result is qualified; if the target evaluation result is less than the standard material threshold, it is further judged whether the target evaluation result is greater than the discard material threshold. If so, the first judgment result is a local defect; if not, the first judgment result is unqualified. When the first judgment result is qualified, the corresponding task result is output as the target creative material; if the first judgment result is unqualified, the corresponding task result is fed back to the multimodal large language model and prompt word generation model as a global constraint, and the corresponding task result generation process is re-executed; if the first judgment result is a local defect, the dimension scores of each result evaluation dimension (image and text matching score, entity distortion score, comprehensive visual perception score, and compliance dimension score) are extracted during the calculation of the target evaluation result. Combined with the evaluation threshold of each evaluation dimension, the scores of each dimension are judged, and unqualified evaluation dimensions are screened out to determine the core adjustment direction. According to the core adjustment direction, the creative prompt words are optimized, and the process of generating task results and obtaining target evaluation results is repeated until the first judgment result is qualified, thus obtaining the target creative material.

[0097] In a specific example, the unqualified evaluation dimension is the core entity dimension, and the corresponding local defect is "product entity distortion". A mandatory constraint description for the physical shape of the foundation bottle is added to the creative prompt. For example, adding "The outline of the foundation glass bottle is regular and complete, the object proportions are normal, there is no distortion or deformation, and the entity structure is rigorous" yields the adjusted creative prompt. The optimized creative prompt is then fed into a multimodal large language model to perform the target image generation task, resulting in a new task result. The process of evaluating the task result and optimizing the creative prompt, generating new task results, is repeated until the first judgment result is qualified, completing the generation of all target creative materials.

[0098] For task results that pass the target evaluation, size adjustments, format conversions, and compression are performed based on the delivery specifications to output directly deployable creative materials, i.e., target creative materials. For task results with local defects, creative prompts can be adjusted based on the reasons for failure. For task results that fail the target evaluation, the corresponding task result is discarded and fed back into the multimodal large language model and prompt generation model as a global constraint, and the generation process for the corresponding task result is re-executed. Simultaneously, the prompt version, model version, random seed, reference material hash, reference factor source, post-processing parameters, and the final target creative material delivered for this task result are recorded, making the generation process traceable, reproducible, and reusable.

[0099] By breaking down the task of generating multiple images or videos into multiple parallel tasks, and improving the stability of batch generation through concurrency control, failure retry, random seed management, prompt word adjustment, and quality scoring mechanisms, the system ensures the diversity of results when batch generation or exploration of multiple solutions is required. This is achieved through different creative directions, different random seeds, and semantic difference constraints.

[0100] Reference Figure 5 This invention provides a creative material generation device based on task orchestration, comprising: The data acquisition module 201 is used to collect the initial generation requirements and initial reference materials input by the user; Task generation module 202 is used to decompose and encapsulate the initial generation requirements and initial reference materials based on a preset task object pattern to form a target task; The creative extraction module 203 is used to generate creative prompts by extracting reference factors from the initial reference materials. The reference factors are reusable design baseline elements obtained from the initial reference materials through joint representation vector extraction and parametric reconstruction by variational autoencoder. The creative prompts are obtained by combining and analyzing the reference factors through graph neural networks and prompt generation models. The task execution module 204 is used to execute the target task and obtain the task result based on the creative prompts and combined with the multimodal large language model. The task evaluation module 205 is used to analyze and judge the task results based on various result evaluation dimensions and give the target evaluation results; Task optimization module 206 is used to optimize creative prompts based on the target evaluation results to obtain target creative materials.

[0101] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process of the described module can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0102] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention. Clearly, those skilled in the art can make various alterations and modifications to the invention without departing from its spirit and scope. Thus, if these modifications and modifications of the invention fall within the scope of the claims and their equivalents, the invention is also intended to include these modifications and modifications.

Claims

1. A method for generating creative materials based on task arrangement, characterized in that, include: Collect initial generation requirements and initial reference materials from users; Based on the preset task object pattern, the initial generation requirements and initial reference materials are decomposed and encapsulated to form the target task; Creative prompts are generated by extracting reference factors from the initial reference materials. The reference factors are reusable design baseline elements obtained from the initial reference materials through joint representation vector extraction and parametric reconstruction by variational autoencoder. The creative prompts are obtained by combining and analyzing the reference factors through graph neural networks and prompt generation models. Based on the creative prompts and combined with a multimodal large language model, the target task is executed to obtain the task results; Based on each result evaluation dimension, the task results are analyzed and judged, and the target evaluation results are given. Based on the target evaluation results, optimize the creative prompts to obtain the target creative materials.

2. The creative material generation method based on task arrangement as described in claim 1, characterized in that, Based on a preset task object pattern, the initial generation requirements and initial reference materials are decomposed and encapsulated to form the target task, which specifically includes: Based on the modality of the initial generated requirements and initial reference materials, the requirement text data and requirement image data in the initial generated requirements and initial reference materials are analyzed respectively to determine the requirement phrases and reference images; The required phrases, reference images and task object patterns are matched and the data is structured to form a primary task object; Iterate through the primary task objects, combine them with the task content constraints, and generate feedback instructions; Based on feedback instructions, the initial task object is supplemented and adjusted to determine the target task.

3. The creative material generation method based on task arrangement as described in claim 1, characterized in that, By extracting reference factors from initial reference materials, creative prompts are generated, specifically including: Multimodal analysis was performed on the initial reference material to obtain the joint representation vector; A parameterized abstraction network based on variational autoencoder is used to design and reconstruct the joint representation vector in a parameterized manner to obtain the reference factor parameter set; Based on the correlation between reference factors and the reference factor parameter set, a reference factor map is constructed. By using a graph neural network, reasoning is performed in the reference factor map to determine multiple reference factor combinations; A prompt word generation model is used to analyze and process the combination of reference factors to obtain creative prompt words.

4. The creative material generation method based on task arrangement as described in claim 3, characterized in that, Multimodal analysis is performed on the initial reference material to obtain the joint representation vector, which specifically includes: A visual encoder is used to extract multi-scale visual feature maps from the initial reference material. A text encoder is used to extract text representations from the initial reference material, resulting in multi-granular text representations. Cross-modal alignment of multi-scale visual feature maps and multi-granular text representations is performed, and an image-text alignment matrix is ​​given. Based on a pre-built multi-label emotion classifier, emotion perception is performed on multi-granular text representations to obtain text emotion labels; Extract digital entities, problem entities, and effect entities from multi-granular text representations to construct core triples; By using a multimodal fusion gating network, the image-text alignment matrix, text sentiment tags, and core triples are projected into a unified latent space to determine the joint representation vector.

5. The creative material generation method based on task arrangement as described in claim 3, characterized in that, Graph neural networks are heterogeneous graph attention networks, which include a node type mapping layer, a heterogeneous graph attention layer, and a combined output layer. The node type mapping layer is used to receive the reference factor map and perform feature projection according to the type of each graph node in the reference factor map, mapping the reference factor parameters of different dimensions to a unified latent vector space and outputting the initial latent vector features of each reference factor. The heterogeneous graph attention layer is used to aggregate neighborhood messages based on the initial latent vector features of each reference factor, calculate the attention coefficients between different reference factors, and give the final latent state vector of each reference factor. The combined output layer is used to analyze and evaluate the combination of reference factors based on the final hidden state vector of each reference factor and the edge weights between different reference factors, to give a comprehensive best-selling score and to screen and determine the combination of reference factors.

6. The creative material generation method based on task arrangement as described in claim 5, characterized in that, By using a graph neural network, inference is performed on the reference factor map to determine multiple combinations of reference factors, specifically including: Traverse each node in the reference factor graph, extract the reference factor parameters of each reference factor, and determine the original feature vector of each reference factor. Feature mapping is performed on the original feature vectors of each reference factor to unify the feature dimensions of each original feature vector, thereby obtaining the initial latent vector features of each reference factor. Based on the first-order neighbor set of each reference factor, and combined with multiple pre-constructed independent attention heads, neighborhood aggregation is performed in each attention head to obtain the attention coefficient of each reference factor in each independent attention head; For each reference factor, the initial latent vector features of each reference factor are fused by combining the attention coefficients of each reference factor in each independent attention head to obtain the final latent state vector of each reference factor. The final hidden state vector of each reference factor is evaluated by a pre-constructed factor evaluation function. The comprehensive best-selling score of each candidate combination of reference factors is determined by combining the edge weights between each reference factor. Based on a pre-constructed screening strategy, the comprehensive best-selling scores of the candidate combinations of reference factors are judged to obtain the reference factor combinations.

7. The creative material generation method based on task arrangement as described in claim 6, characterized in that, The overall best-selling score is determined through the following steps: By using the factor evaluation function, the final hidden state vector of each reference factor in the candidate combination of reference factors is analyzed and evaluated, the average fitness score of the reference factors in the candidate combination of reference factors is given, and the single factor fitness item is determined. Based on the edge weights of each reference factor in the candidate combination of reference factors, the mean correlation strength of each reference factor is analyzed to obtain the factor combination compatibility term. By balancing the single-factor fit term and the factor combination compatibility term, the comprehensive best-selling score of each reference factor candidate combination is obtained.

8. The creative material generation method based on task arrangement as described in claim 3, characterized in that, A prompt word generation model is used to analyze and process the combination of reference factors to obtain creative prompt words, specifically including: A prompt word generation model is used to semantically supplement the combination of reference factors to form initial task prompt words; The differences between the initial task prompts are analyzed, and a difference score is given, specifically as follows: ； Among them, S diff The difference score for each task prompt word is given, where n is the number of task prompt words, and D is the score for each prompt word. pq Let u be the cosine distance between the current task cue word p and the current task cue word q. p Let u be the semantic vector of the current task prompt word p. q Let q be the semantic vector of the current task prompt word, and ||·||2 be the second norm. By judging the difference score, a random seed is introduced to make differential adjustments to the initial task prompt words to obtain the current task prompt words; The process of differential analysis of task prompts is repeated until the difference score reaches the preset difference range, thus generating creative prompts.

9. The creative material generation method based on task arrangement as described in claim 1, characterized in that, in, The evaluation dimensions include text-image semantic matching, core entity, visual perception, and compliance constraints. Based on the various evaluation dimensions, the task results are analyzed and judged, and the target evaluation results are given, including: Based on the semantic matching dimension of images and text, the similarity between the task results and creative prompts is calculated, and the image and text matching score of the task results is given. Based on the core entity dimension, a distortion discrimination model and a text recognition model are used to analyze and judge the distortion of core entities and text in the task results, and give the entity distortion score. The visual perception prediction model is used to judge the task results from the perspective of visual perception and obtain a comprehensive visual perception score. Based on the compliance constraint dimension, the non-compliant elements in the task results are analyzed and extracted to determine the compliance dimension score; The scores for image-text matching, entity distortion, overall visual perception, and compliance dimensions of the task results are integrated to form the target evaluation result.

10. A creative material generation device based on task orchestration, characterized in that, The creative material generation method based on task arrangement as described in any one of claims 1-9 includes: The data acquisition module is used to collect the initial generation requirements and initial reference materials input by the user; The task generation module is used to decompose and encapsulate the initial generation requirements and initial reference materials based on the preset task object pattern to form the target task; The creative extraction module is used to generate creative prompts by extracting reference factors from the initial reference materials. The reference factors are reusable design baseline elements obtained from the initial reference materials through joint representation vector extraction and parametric reconstruction by variational autoencoder. The creative prompts are obtained by combining and analyzing the reference factors through graph neural networks and prompt generation models. The task execution module is used to execute the target task and obtain the task result based on the creative prompts and combined with the multimodal large language model. The task evaluation module is used to analyze and judge the task results based on various result evaluation dimensions and provide the target evaluation results. The task optimization module is used to optimize creative prompts based on the target evaluation results, thereby obtaining target creative materials.

Images