Image generation method, training method of reward model

Abstract

Embodiments of the present application provide an image generation method, a reward model training method, an electronic device, a storage medium and a computer program product. The method comprises: obtaining object description text of a target object, original object image and scene information matched with a to-be-generated image carried by an image generation instruction; taking the scene information, the object description text and the original object image as input information, calling a pre-tuned visual language model to generate background image design description text adapted to the scene information, wherein when the scene information is a weak signal in the input information, the background image design description text can reflect scene preference characteristics matched with the scene information; calling a preset image generation model to generate an image of the target object matching the scene information under the constraint of the background image design description text, the image of the target object in the original object image and layout information. The image generated by the method is more matched with the scene preference corresponding to the scene information.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an image generation method, a training method for a reward model, a training method for a visual language model, a method for constructing a training sample set for a reward model, an electronic device, a storage medium, and a computer program product. Background Technology

[0002] Multimodal generation and understanding technologies based on visual-language models have been widely applied to tasks such as image description, visual question answering, and content generation. However, existing visual-language models perform poorly when generating images for tasks where the instructions contain strong predictive variables with weak signals. For example, in the task of "selecting a more popular advertising image for a specific country market based on the product attributes and image of product A," the key variable "country" is weakened relative to the text; this phenomenon is defined as "weakening of strong predictive variables with weak signals." When performing the above task, existing image generation methods often ignore the key textual clue of "country" and over-rely on high-frequency features such as image content or product attributes, resulting in output product advertising images that deviate from the actual country preferences.

[0003] It is evident that existing image generation methods still need improvement. Summary of the Invention

[0004] This application provides an image generation method that can effectively improve the matching degree between the generated image and the scene preference.

[0005] Accordingly, embodiments of this application also provide a training method for a reward model, a method for constructing a training sample set for a reward model, a training method for a visual language model, an electronic device, a storage medium, and a computer program product, to ensure the implementation and application of the above-mentioned image generation method.

[0006] To solve the above-mentioned technical problems, this application is implemented as follows: In a first aspect, embodiments of this application provide an image generation method, the method comprising: The multimodal information of the target object carried by the image generation instruction and the scene information matching the image to be generated are obtained, wherein the multimodal information includes: the object description text of the target object and the original object image; Using the scene information, the object description text, and the original object image as input information, a pre-tuned visual language model is invoked to generate a background image design description text that adapts to the scene information. When the scene information is a weak signal in the input information, the background image design description text can reflect the scene preference features matched by the scene information. Using the background image design description text and the original object image as input information, a preset image generation model is triggered to generate an image of the target object with a background that matches the scene information, under the constraints of the background image design description text, the image of the target object in the original object image, and the layout information.

[0007] Secondly, embodiments of this application provide a training method for a reward model, wherein the reward model includes: a text encoding module, a visual encoding module, a multimodal feature fusion module, a modality alignment module, and a classification prediction module, and the method includes: The text encoding module performs text encoding processing on the scene information, enhanced answer text, and prompt instruction text corresponding to each training sample in the pre-acquired training sample set, and obtains the enhanced answer feature vector, instruction feature vector, and scene feature vector corresponding to the training sample. The visual encoding module performs visual encoding processing on the first image and the second image of the object corresponding to the prompt instruction text to obtain the first visual feature vector corresponding to the first image and the second visual feature vector corresponding to the second image. The multimodal feature fusion module modulates the first visual feature vector and the second visual feature vector using the scene feature vector to obtain the first fusion feature vector corresponding to the first image and the second fusion feature vector corresponding to the second image. The modality alignment module performs inference processing on the instruction feature vector, the first fused feature vector, and the second fused feature vector to obtain the hidden layer output. The classification prediction module performs classification prediction based on the hidden layer output and the enhanced answer feature vector to obtain the prediction result of the training sample; The model loss is calculated based on the attention intensity penalty loss of each preset key feature in the prompt instruction text and the classification prediction loss corresponding to the prediction result. The model parameters of the reward model are iteratively optimized with the convergence of the model loss as the goal.

[0008] Thirdly, embodiments of this application provide a method for training a visual language model, the method comprising: The visual language model to be fine-tuned is trained using a supervised fine-tuning method. The model is designed to generate background images based on object description text and original object images to design description text, thus obtaining an initial fine-tuned visual language model. Using the initially fine-tuned visual language model as an agent, a reward signal is constructed based on the scene preference evaluation results of the generated image corresponding to the agent's actions according to a preset reward model. The initially fine-tuned visual language model is then fine-tuned using a reinforcement learning method to obtain the visual language model. The reward model is sensitive to weak scene signals, which are strong predictive variables.

[0009] Fourthly, embodiments of this application provide a method for constructing a training sample set for a reward model, including: Based on the preset query text, obtain multiple different advertising images of the same object; Obtain real user click preference data for the multiple different advertising images in multiple scenarios; Based on the real user click preference data, preference pair data is constructed. Each preference pair data includes: the query text, at least two different advertising images of the same object, and user click preferences of the at least two different advertising images in multiple scenarios. The preference data is deduplicated and subjected to multi-dimensional balancing to obtain the training sample set of the reward model. The multi-dimensional balancing includes scene dimension and object category dimension.

[0010] Fifthly, embodiments of this application provide an electronic device including a processor, a memory, and a program or instructions stored in the memory and executable on the processor, wherein the program or instructions, when executed by the processor, implement the method described in the first, second, third, or fourth aspects.

[0011] In a sixth aspect, embodiments of this application provide a readable storage medium on which a program or instructions are stored, which, when executed by a processor, implement the steps of the method described in the first, second, third, or fourth aspect.

[0012] In a seventh aspect, embodiments of this application provide a computer program product, including a computer program / computer executable instructions, which, when executed by a processor in an electronic device, implement the method described in the first, second, third, or fourth aspect.

[0013] Compared with the prior art, the embodiments of this application have the following advantages: The process involves acquiring multimodal information of the target object carried by the image generation instruction and scene information matching the image to be generated. The multimodal information includes: object description text of the target object and an original object image. Using the scene information, the object description text, and the original object image as input, a pre-tuned visual language model is invoked to generate a background image design description text adapted to the scene information. When the scene information is a weak signal in the input information, the background image design description text can reflect the scene preference features matching the scene information. Using the background image design description text and the original object image as input, a preset image generation model is invoked, under the constraints of the target object's image and layout information in the background image design description text and the original object image, to generate an image of the target object whose background matches the scene information, effectively improving the matching degree between the generated image and the scene preference corresponding to the scene information. Attached Figure Description

[0014] Figure 1 This is a flowchart of the image generation method disclosed in the embodiments of this application; Figure 2 This is a schematic diagram illustrating the working principle of the reward model disclosed in the embodiments of this application; Figure 3 This is a schematic diagram of the training method of the reward model disclosed in the embodiments of this application; Figure 4 This is a flowchart of the steps of the training method for the visual language model disclosed in the embodiments of this application; Figure 5 This is a schematic diagram illustrating the implementation principle of the training method for the visual language model disclosed in the embodiments of this application; Figure 6 This is a flowchart illustrating the steps of the method for constructing the training sample set of the reward model disclosed in the embodiments of this application; Figure 7 This is a schematic diagram of the structure of an exemplary device provided in one embodiment of this application. Detailed Implementation

[0015] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0016] This application implements the disclosed technical solution by training a reward model sensitive to weak scene signals as strong predictive variables. The trained reward model is then applied to fine-tune a visual language model, ensuring that the fine-tuned model avoids ignoring weak strong predictive variables in the instruction information when generating images based on multimodal input information. This solution, through the collaborative work of the two models, forms a complete closed loop from "understanding preferences" to "generated content," effectively solving the problem of "weakening of strong predictive variables with weak signals" during image generation based on the visual language model.

[0017] like Figure 1 As shown, the image generation method disclosed in this embodiment includes steps 102 to 106.

[0018] Step 102: Obtain the multimodal information of the target object and the scene information matching the image to be generated carried by the image generation instruction. The multimodal information includes: the object description text of the target object and the original object image.

[0019] Optionally, the scenario information includes, but is not limited to, country, user group, and usage scenario. The specific values ​​of the scenario information are determined according to application requirements.

[0020] Optionally, the target object includes, but is not limited to, any of the following: items (such as goods on an e-commerce platform), people, animals, etc.

[0021] Taking an image generation instruction for generating advertising images of goods placed in a designated country on an e-commerce platform as an example, the image generation task carries at least the following information: target country, product name, original product image, and product description text. Specifically, the image generation instruction could be: "Generate an advertising image for the H country market based on <product attributes> and <product image>". Correspondingly, the scenario information is "H country", the target object is the product belonging to the "product attributes", the object description text includes "product attributes", and the original object image is the "product image".

[0022] In the embodiments of this application, a two-stage generation method is used to generate images that meet the requirements of the image generation task. Specifically, in the first stage, image design description text for the background image of the image to be generated is generated based on information carried in the image generation instruction; in the second stage, an object image with a background matching the scene information is generated based on the image design description text and the original object image.

[0023] Step 104: Using the scene information, the object description text, and the original object image as input information, call the pre-tuned visual language model to generate background image design description text adapted to the scene information.

[0024] Where the scene information is a weak signal in the input information, the background image design description text can reflect the scene preference characteristics matched by the scene information. The weak signal includes, but is not limited to, signals described with very little text in the input information. For example, for input information with a length of 20 characters, a country name described with only two characters can be considered a weak signal.

[0025] Optionally, the visual language model is obtained through reinforcement learning using the output of a reward model that is sensitive to weak scene signals as strong predictor variables. Here, the strong predictor variable refers to variable input information that serves as an important constraint on the output of the visual language model.

[0026] Optionally, using the scene information, the object description text of the target object, and the original object image as input information, a prompt message is generated according to a preset template. Then, based on the prompt message, a pre-tuned visual language model is invoked, which generates a background image design description text adapted to the scene information.

[0027] Taking the country "Japan" as an example, the visual language model will output a detailed text description describing the advertising background design suitable for the target country market, such as "In front of a Japanese garden with cherry blossoms in full bloom, the product is placed on a simple wooden display stand".

[0028] Optionally, the visual language model is fine-tuned using the following method: A supervised fine-tuning method is used to train the visual language model to be fine-tuned to generate background images and design descriptive text based on object description text and original object images, resulting in an initial fine-tuned visual language model; using the initial fine-tuned visual language model as an agent, a reward signal is constructed based on a preset reward model to evaluate the scene preference of the generated images corresponding to the agent's actions; a reinforcement learning method is then used to fine-tune the initial fine-tuned visual language model to obtain the visual language model. The reward model is sensitive to weak scene signals as strong predictive variables, and the actions include generating background images and designing descriptive text.

[0029] The training data used when training the visual language model to be fine-tuned using the supervised fine-tuning method includes: object description text, original object images, and background image design description text used as sample labels.

[0030] The visual language model to be fine-tuned is trained using a supervised fine-tuning method to generate background images and design descriptive text based on object description text and original object images. The specific implementation of the initial fine-tuning visual language model is obtained, as described in the relevant embodiments below, and will not be repeated here.

[0031] Because the reward model is sensitive to weak scene signals as strong predictors, the visual language model trained by reinforcement learning based on the reward model can pay attention to weak scene signals when generating background image design description text based on object description text and original object image, thereby making the generated image more match the scene preference of the target scene.

[0032] The training data used when fine-tuning the initial fine-tuned visual language model using reinforcement learning methods includes: scene information, object description text, and original object images.

[0033] The training method for the reward model is described in the relevant description in the embodiments below, and will not be repeated here.

[0034] Step 106: Using the background image design description text and the original object image as input information, a preset image generation model is invoked to generate an image of the target object whose background matches the scene information, under the constraints of the background image design description text, the image of the target object in the original object image, and layout information.

[0035] Next, using the background image design description text and the original object image as input information, prompt information is generated. Based on the generated prompt information, a preset image generation model is guided to generate an image according to the prompt information. The preset image generation model includes a diffusion model and a control network, wherein the control network is used to determine the image and layout information of the target object based on the original object image. The preset image generation model is used to perform a controlled text-to-image generation operation. The layout information includes the outline of the target object and the position information of the target object in the original object image.

[0036] When the preset image generation model generates an object image based on the background image, the design description text, and the image and layout of the target object in the original object image, the control network performs edge detection on the original object image to obtain an edge map. Based on this edge map, the image, edge information, and layout information of the target object in the original object image are determined. Then, the control network ensures that the image generated by the diffusion model strictly adheres to the contour and layout of the target object in the original object image, based on the edge information and layout information. The diffusion model generates a background image based on the background image, the design description text, and the edge detection information. Finally, the diffusion model merges the generated background image and the image of the target object to generate a new object image. The object contour and layout in the object image generated in this way are consistent with the original object image.

[0037] In summary, the image generation method disclosed in this application obtains multimodal information of the target object and scene information matching the image to be generated, carried by the image generation instruction. The multimodal information includes: object description text of the target object and the original object image. Using the scene information, the object description text, and the original object image as input, a pre-tuned visual language model is invoked to generate a background image design description text adapted to the scene information. When the scene information is a weak signal in the input information, the background image design description text can reflect the scene preference features matching the scene information. Using the background image design description text and the original object image as input, a preset image generation model is invoked to generate an image of the target object with a background matching the scene information, constrained by the image and layout information of the target object in the background image design description text and the original object image. This method utilizes the output of a reward model sensitive to weak scene signals as strong predictive variables when training the visual language model, allowing the visual language model to fully focus on weak scene signals as strong predictive variables, thereby improving the preference matching degree between the generated image and the scene corresponding to the scene information.

[0038] To implement the above embodiments, this application also discloses a training method for training a reward model. For example... Figure 2 As shown, the reward model includes: a text encoding module, a visual encoding module, a multimodal feature fusion module, a modality alignment module, and a classification prediction module. Figure 3 As shown, the training method includes steps 302 to 312.

[0039] The following is combined with Figure 2 The schematic diagram of the reward model shown illustrates the specific implementation of the training method.

[0040] Step 302: The text encoding module performs text encoding processing on the scene information, enhanced answer text, and prompt instruction text corresponding to each training sample in the pre-acquired training sample set to obtain the enhanced answer feature vector, instruction feature vector, and scene feature vector corresponding to the training sample.

[0041] Each training sample in the training sample set consists of two different images of the same object, along with real user preference data from multiple scenarios. The training sample set constructs comparative scenarios where high-quantity features (such as product description text and images) remain unchanged while the strong predictor variable (target country) shows a weak signal, serving as the foundation for systematically studying the weakening of strong predictor variables with weak signals. Furthermore, the training sample set uses real preference data (such as click-through rates) as preference signals, and through rigorous object and image deduplication and meticulous balancing across scenario and object category dimensions, it ensures the authenticity, purity, and unbiasedness of the data. This provides an indispensable foundation for reliably training and validating solutions for the weakening of strong predictor variables with weak signals.

[0042] Each training sample includes, but is not limited to, one or more of the following information: query text, a first image, and a second image. The first image and the second image are two object images of the same object obtained by retrieving data from a preset object database based on the query text. Each training sample corresponds to a set of information, which includes, but is not limited to, scene information, enhanced answer text, and prompt instruction text. As mentioned above, the scene information includes, but is not limited to, any of the following: country, user group, usage scenario, etc. The enhanced answer text is obtained using retrieval enhancement techniques to indicate the preference of the first image and the second image for the scene information. For example, the enhanced answer text could be expressed as: "Based on experience, the first image is more suitable for the market of country H." The prompt instruction text is text describing the reward model performing a scene-related image classification task, for example, it could be expressed as: "For the market of country H, based on <object title> and <object description text>, compare the product images below (A / B) and select the image that is more attractive to consumers and can improve the conversion rate. Please consider factors such as visual appeal and background design. The answer must be a single uppercase letter (A / B) and located on the last line."

[0043] For a detailed implementation of the text encoding module, please refer to the prior art. After different types of text are input into the text encoding module, the module encodes the input text sequence to obtain vector representations of each input text. For example, it obtains the scene feature vector of the scene information, the enhanced answer feature vector of the enhanced answer text, and the instruction feature vector of the prompt instruction text.

[0044] Step 304: The visual encoding module performs visual encoding processing on the first image and the second image of the object corresponding to the prompt instruction text to obtain the first visual feature vector corresponding to the first image and the second visual feature vector corresponding to the second image.

[0045] The first image and the second image are object images of the same object indicated by the prompt instruction text.

[0046] After the first image is input into the visual encoding module, the visual encoding module encodes the input first image to obtain a vector representation of the first image, denoted as the "first visual feature vector". After the second image is input into the visual encoding module, the visual encoding module encodes the input second image to obtain a vector representation of the second image, denoted as the "second visual feature vector".

[0047] Optionally, the specific implementation of the visual encoding module can be found in the prior art. During the training of the reward model, the model parameters of the visual encoding module can be frozen, and the model parameters other than those of the visual encoding module can be optimized.

[0048] Step 306: Through the multimodal feature fusion module, the scene feature vector is used to perform visual feature modulation on the first visual feature vector and the second visual feature vector respectively to obtain the first fusion feature vector corresponding to the first image and the second fusion feature vector corresponding to the second image.

[0049] Optionally, the multimodal feature fusion module uses the scene feature vector to perform visual feature modulation on the first visual feature vector and the second visual feature vector respectively to obtain the first fused feature vector corresponding to the first image and the second fused feature vector corresponding to the second image, including sub-step S1 and sub-step S2.

[0050] Sub-step S1: The scene feature vector is mapped to affine transformation parameters through the multimodal feature fusion module.

[0051] The affine transformation parameters include scaling parameters and offset parameters.

[0052] Optionally, the multimodal feature fusion module can be implemented based on a feedforward neural network. For example, a modified linear unit can be used to perform a linear transformation on the scene feature vector to obtain a transformed vector. The feedforward neural network can be represented as follows: Optional, ,in, This represents the scene feature vector. , , and These are the model parameters.

[0053] Next, the transformation vector is decomposed into the affine transformation parameters. For example, the transformation vector can be divided into scaling and offset parameters using the following function: ,in, This represents the scaling parameter. This represents the offset parameter. The initial value of the scaling parameter is approximately 1, and the initial value of the offset parameter is approximately 0. This is to ensure that the pre-trained visual features are not altered when the reward model begins training.

[0054] Sub-step S2 involves performing an element-wise affine transformation on the first visual feature vector using the affine transformation parameters to obtain a first fused feature vector corresponding to the first image; and performing an element-wise affine transformation on the second visual feature vector using the affine transformation parameters to obtain a second fused feature vector corresponding to the second image.

[0055] For example, using formulas Perform an element-wise affine transformation on the visual feature vector, where... It can be either the first visual feature vector or the second visual feature vector, where "⊙" indicates element-wise multiplication. The first and second fused feature vectors obtained after the visual feature vectors undergo radial transformation carry visual preference information corresponding to the scene information.

[0056] Step 308: The modality alignment module performs inference processing on the instruction feature vector, the first fused feature vector, and the second fused feature vector to obtain the hidden layer output.

[0057] Optionally, the modality alignment module can be implemented based on the backbone network of a generative language model. The step of performing inference processing on the instruction feature vector, the first fused feature vector, and the second fused feature vector through the modality alignment module to obtain the hidden layer output includes: performing inference on the feature vectors of preset type inputs, the first fused feature vector, and the second fused feature vector in the input sequence corresponding to the prompt instruction text based on an attention mechanism to obtain the hidden layer output; wherein, the preset type input includes: scene and object description text and object image. The hidden layer output is used to represent the vector representation of the inference result, and the vector representation includes the vector representation of the token (such as an image identifier) ​​corresponding to the image (i.e., the first image or the second image) selected by the generative language model that is more consistent with the prompt instruction text. Taking the image identifier of the first image as "A" and the image identifier of the second image as "B" as an example, the hidden layer output includes the vector representation of image identifier A or B.

[0058] The output position of the inference result can be specified in the prompt instruction text. For example, the output position of the token corresponding to the image that better matches the prompt instruction text in the inference result can be specified (such as the last line). Accordingly, based on the position of the token corresponding to the image that better matches the prompt instruction text in the inference result, the vector representation of the image is obtained from the last hidden layer output of the modality alignment module, for example, denoted as "h". last ".

[0059] The modality alignment module uses an attention mechanism to align the instruction feature vector, the first fused feature vector, and the second fused feature vector to the hidden layer vectors corresponding to the scene information, the object description text, and the image, until the last hidden layer output of the modality alignment module. During vector mapping, the starting or key position t corresponding to the scene information, object description text, and image in the input sequence is first determined. c t p and t i Next, attention weights corresponding to different types of information are calculated based on the determined positions. Then, different weights are used to perform feature mapping on the hidden layer vectors of different types of information. Optionally, the weights corresponding to different types of information can be calculated using the following formula: ; ; ; Where T represents the length of the input sequence, and i represents the word index in the input sequence. This represents the hidden layer vector corresponding to the scene information. The hidden vector corresponding to the object description text. This represents the hidden layer vector corresponding to the image. The L2 norm of the j-th hidden layer vector The attention weights represent the text describing the object. This represents the attention weights corresponding to scene information. This represents the attention weights corresponding to the image.

[0060] Step 310: The classification prediction module performs classification prediction based on the hidden layer output and the enhanced answer feature vector to obtain the prediction result of the training sample.

[0061] The prediction result is used to indicate the classification result prediction value of the first image and the second image as the preferred image for matching the scene information.

[0062] Optionally, the hidden layer output and the enhanced answer feature vector are concatenated and then input into the classification prediction module. The classification prediction module performs classification prediction on the input vector to obtain the prediction result for the current training sample. The classification function is expressed as follows: For example, the classification result can be represented as: .(e aug + h last ), where e aug h represents the enhanced answer feature vector. last The image identifier represents the image that best matches the prompt instruction text. The classification result is processed by an activation function to obtain the prediction result.

[0063] Step 312: Calculate the model loss based on the attention intensity penalty loss of each preset key feature in the prompt instruction text and the classification prediction loss corresponding to the prediction result, and iteratively optimize the model parameters of the reward model with the goal of model loss convergence.

[0064] The preset key features include: scene features, object description text features, and object image features.

[0065] The prediction result for each training sample is obtained using the method described above. Next, based on the prediction result and prediction process data for each training sample, the loss value of the reward model is calculated. In the embodiments of this application, a novel loss function is introduced. When the model makes a prediction error, this function calculates the model's "attention intensity" or "feature contribution" to key variables (such as scene, object description text, and image), and imposes additional penalties on variables with insufficient attention. This forces the model to actively and evenly utilize all key information during decision-making, thereby compelling it to achieve the optimization objective.

[0066] Optionally, the step of calculating the model loss based on the attention intensity penalty loss of each preset key feature in the prompt instruction text and the classification prediction loss corresponding to the prediction result, and iteratively optimizing the model parameters of the reward model with the goal of model loss convergence, includes: obtaining the attention intensity penalty loss corresponding to the misclassified training sample based on the attention intensity of the reward model to each preset key feature in the prompt instruction text; calculating the classification prediction loss corresponding to the training sample based on the difference between the prediction result and the true value of the preset classification result of the training sample; weighted summing of the attention intensity penalty loss and the classification prediction loss corresponding to each training sample to obtain the model loss, wherein the attention intensity penalty loss corresponding to the correctly classified training sample is zero; and iteratively optimizing the model parameters of the reward model with the goal of model loss convergence.

[0067] Optionally, based on the attention intensity of each preset key feature in the prompt instruction text according to the reward model, the attention intensity penalty loss corresponding to the misclassified training sample is obtained, including: for the misclassified training sample, obtaining the attention intensity corresponding to each preset key feature based on the proportion of the L2 norm of the hidden state of the output layer of the modality alignment module in the reward model to the total norm of each preset key feature in the prompt instruction text; and performing an exponential operation on the sum of the complements of the attention intensities corresponding to each preset key feature to obtain the attention intensity penalty loss corresponding to the training sample.

[0068] For example, the attention intensity of scene features can be calculated using the formula described above. Attention intensity of object description text Attention intensity corresponding to the image Then, the penalty term corresponding to each preset key feature is applied using the following formula: , , ,in, This represents the penalty term corresponding to the scene features. This indicates the penalty item corresponding to the object description text. This represents the penalty term corresponding to the image.

[0069] Next, assuming Let i be an indicator function for whether a training sample i is predicted incorrectly. This indicator function is activated when a prediction is incorrect. For a batch of training samples, the attention intensity penalty loss for that batch of training samples is calculated using the following formula. : Wherein, for training samples that are predicted incorrectly, when training sample i is predicted incorrectly... A value of 1 indicates that the training sample was predicted correctly. The value is 0.

[0070] Optionally, the classification prediction loss can be the binary cross-entropy loss between the prediction result and the preset true value of the classification result.

[0071] Then, the formula can be used. Calculate the model loss, where, Indicates the classification prediction loss. Indicates hyperparameters, for example .

[0072] During iterative training, the parameters of the reward model are optimized with the goal of model loss convergence, until a preset number of iterations is reached or the model loss converges.

[0073] Optionally, the training sample includes: query text and a first image and a second image. The enhanced answer text corresponding to the training sample is obtained through the following method: a first-stage screening of objects in a preset object set based on text similarity is performed to obtain a first candidate object set; wherein, the object description text of the objects in the first candidate object set and the query text of the training sample satisfy a preset text similarity condition; a second-stage screening of objects in the first candidate object set is performed based on image similarity to obtain a second candidate object set and a third candidate object set; wherein, the object images of each object in the second candidate object set and the first image satisfy a preset image similarity condition, and the object images of each object in the third candidate object set and the second image satisfy the preset image similarity condition; a preference score for the first image is calculated based on a preset evaluation index for each object in the second candidate object set; and a preference score for the second image is calculated based on the preset evaluation index for each object in the third candidate object set; and the enhanced answer text corresponding to the training sample is generated based on the comparison result of the preference scores.

[0074] The query text is used to retrieve target objects from a preset database, such as products on an e-commerce platform. The preset object set is a collection of target objects retrieved from the preset database based on the query text.

[0075] Optionally, the first-stage screening of objects in the preset object set based on text similarity to obtain a first candidate object set includes: obtaining the text similarity between the object description text of each object in the preset object set and the query text of the training sample; selecting the first M objects whose text similarity is less than a first preset similarity threshold in descending order of text similarity to form the first candidate object set, where M is a natural number greater than 1.

[0076] First, obtain the embedding vectors of the query text. The text feature embedding vector matrix of the object description text of each object in the preset object set. Image feature embedding vector matrix of the object image ,in, , N represents the number of objects in the preset object set, and d represents the feature length of the embedding vector. Then, based on the embedding vector of the query text... The text feature embedding vector matrix of the object description text of each object in the preset object set. Calculate the text similarity between the query text and the text of each object in the preset object set.

[0077] Optionally, the first preset similarity threshold can be set to 0.7. Then, objects in the preset object set whose text similarity is less than the first preset similarity threshold are selected to obtain a preliminary object set. Next, the top M objects with the highest text similarity in the preliminary object set are selected to form a first candidate object set, where M is a natural number greater than 1.

[0078] Optionally, a second-stage screening is performed on the objects in the first candidate object set based on image similarity to obtain a second candidate object set and a third candidate object set. This includes: obtaining the first image similarity between the object images of the objects in the first candidate object set and the first image, and the second image similarity between the object images and the second image; selecting the first M objects whose first image similarity is less than a second preset similarity threshold in descending order of the first image similarity to form the second candidate object set; and selecting the first M objects whose second image similarity is less than the second preset similarity threshold in descending order of the second image similarity to form the third candidate object set, where M is a natural number greater than 1.

[0079] Next, we first obtain the image feature embedding vectors of the object images of each object in the first candidate object set. The image feature embedding vector of the first image Image feature embedding vector of the second image Next, the image feature embedding vector of the first image is calculated. and image feature embedding vector The vector similarity is used as the first image similarity between the corresponding object in the first candidate object set and the first image. Similarly, the second image similarity between each object in the first candidate object set and the second image is calculated.

[0080] Optionally, the second preset similarity threshold can be set to 0.7. Then, objects in the first candidate object set whose first image similarity is less than the second preset similarity threshold are selected to obtain a first refined object set. Next, the top M objects in the first refined object set with the highest first image similarity are selected to form a second candidate object set. Using the same method, objects in the first candidate object set whose second image similarity is less than the second preset similarity threshold are selected to obtain a second refined object set. Then, the top M objects in the second refined object set with the highest second image similarity are selected to form a third candidate object set.

[0081] Next, the preference scores for the first and second objects are calculated separately.

[0082] The preset evaluation indicators are used to evaluate users' preferences for objects, and these indicators are determined based on specific application scenarios. Taking products on an e-commerce platform as an example, the preset evaluation indicator could be click-through rate.

[0083] Optionally, calculating the preference score of the first image based on the preset evaluation indicators of each object in the second candidate object set includes: weighting the preset evaluation indicators of each object in the second candidate object set using image similarity, and aggregating to obtain the preference score of the first image. Optionally, weights are set for the preset evaluation indicators of the corresponding objects according to the first image similarity of each object in the second candidate object set; the greater the similarity, the greater the weight. For example, the objects in the second candidate object set can be sorted from front to back according to the first image similarity from largest to smallest, and then the weights of the preset evaluation indicators of each object in the second candidate object set can be set according to the sorting position.

[0084] Optionally, the preference score for the first image can be calculated using the following formula. : ;in, This represents the weight of object i in the second candidate object set. This represents the total number of objects in the second candidate object set. Represents the second set of candidate objects Each of the objects described in the middle Click-through rate (i.e., preset evaluation metric). Represents the second set of candidate objects Each of the objects described in the middle Click-through rate (i.e., preset evaluation metric). This indicates an indicator function.

[0085] Optionally, calculating the preference score of the second image based on the preset evaluation index of each object in the third candidate object set includes: weighting the preset evaluation index of each object in the third candidate object set using image similarity, and aggregating to obtain the preference score of the second image. The method for setting the weights of the preset evaluation indexes of each object in the third candidate object set is the same as the method for setting the weights of the preset evaluation indexes of each object in the second candidate object set.

[0086] Optionally, the preference score for the second image can be calculated using the following formula. : ;in, This represents the weight of object i in the third candidate object set. This represents the total number of objects in the third candidate object set.

[0087] After calculating the preference scores of the first image and the second image, the preference scores of the first image and the second image are compared. If the preference score of the first image is greater than the preference score of the second image, the image identifier of the first image can be used as the enhanced answer text corresponding to the query text (for example, the enhanced answer text of the training sample is set to "A"). Otherwise, the image identifier of the second image can be used as the enhanced answer text corresponding to the query text (for example, the enhanced answer text of the training sample is set to "B").

[0088] In the embodiments of this application, a market-oriented "enhanced answer" is retrieved and generated for the target sample by utilizing historical preference data (such as click rate) of a specified scenario. External, specific market knowledge is injected into the training process as a priori signal, aiming to strengthen the reward model's cognition and reliance on the key variable of scenario information from the data source.

[0089] In summary, the training method for the reward model disclosed in this application includes a text encoding module, a visual encoding module, a multimodal feature fusion module, a modal alignment module, and a classification prediction module. The method includes: using the text encoding module to perform text encoding processing on the scene information, enhanced answer text, and prompt instruction text corresponding to each training sample in a pre-acquired training sample set, obtaining enhanced answer feature vectors, instruction feature vectors, and scene feature vectors corresponding to the training samples; using the visual encoding module to perform visual encoding processing on the first image and the second image of the object corresponding to the prompt instruction text, obtaining a first visual feature vector corresponding to the first image and a second visual feature vector corresponding to the second image; and using the multimodal feature fusion module to... The first visual feature vector and the second visual feature vector are respectively modulated using the scene feature vector to obtain a first fused feature vector corresponding to the first image and a second fused feature vector corresponding to the second image. The instruction feature vector, the first fused feature vector, and the second fused feature vector are inferred using the modality alignment module to obtain the hidden layer output. The classification prediction module performs classification prediction based on the hidden layer output and the enhanced answer feature vector to obtain the prediction result of the training sample. The model loss is calculated based on the attention intensity penalty loss of each preset key feature in the prompt instruction text and the classification prediction loss corresponding to the prediction result. The model parameters of the reward model are iteratively optimized with the goal of model loss convergence. In the process of training the reward model, this method modulates the first visual feature vector and the second visual feature vector using the scene feature vector, so that the fused feature vector carries the visual preferences of a specific scene. This allows the visual content to be differentiated according to the semantic embedding of different scenes, achieving "scene-conditional" adaptation of the reward model at the feature level. Furthermore, when calculating the model loss, attention intensity penalty loss is introduced for scene features, object description text features, and image features. This imposes an additional penalty on variables that are not given enough attention (such as scene information), making the reward model fully consider the "feature contribution" of scene information. This forces the model to actively and evenly utilize all key information when making decisions, which helps to solve the problem of inaccurate classification results caused by weak scene information as a strong predictor variable.

[0090] After ablation experiments, when the enhanced answer text input, the multimodal fusion module, or the attention intensity penalty loss were removed, the accuracy of the reward model decreased by 3.16%, 2.39%, and 3.42%, respectively, and the sensitivity decreased by 3.44%, 1.89%, and 3.37%, respectively. The training method disclosed in this application, through the above three mechanisms, enables the reward model to be guided by scene information at the data level (input enhanced answer text), the feature level (the multimodal fusion module modulates visual features based on scene information), and the optimization objective level (introducing attention intensity penalty loss). It incorporates weak scene variables (such as country) into its decision core, so that the reward signal output by the reward model can accurately reflect the preference differences caused by scene variables. In cross-scene (such as cross-country, cross-user group) preference prediction tasks, the accuracy and sensitivity are significantly improved.

[0091] Based on the foregoing embodiments, this application discloses a method for training a visual language model, such as... Figure 4 The method includes steps 402 and 404.

[0092] Step 402: The visual language model to be fine-tuned is trained using a supervised fine-tuning method to generate background images and design descriptive text based on object description text and original object images, thus obtaining the initially fine-tuned visual language model.

[0093] For example, a visual language model can be fine-tuned based on training data from a specified domain to train its ability to generate background image design description text based on multimodal inputs. The multimodal inputs include: object description text from the specified domain and object images, with each multimodal input corresponding to a set of background image design description text.

[0094] For specific implementation methods of training a visual language model to be fine-tuned using training data from a specified domain, please refer to the prior art, and will not be repeated in the embodiments of this application.

[0095] Step 404: Using the initially fine-tuned visual language model as an agent, construct a reward signal based on the scene preference evaluation result of the generated image corresponding to the agent's action according to the preset reward model, and fine-tune the initially fine-tuned visual language model using reinforcement learning method to obtain the visual language model, wherein the reward model is sensitive to weak scene signals as strong predictive variables.

[0096] The following is combined with Figure 5 The schematic diagram shown illustrates the training method for the visual language model.

[0097] like Figure 5In the embodiments of this application, the training and optimization system for the visual language model includes: a visual language model to be optimized, a preset image generation model, and a reward model. These three models collaborate to optimize the visual language model. Specifically, the visual language model serves as a background design model, used to design background images based on the instruction description text carried by the image generation instructions and the original object image, generating two sets of background image design schemes for the same object. Each background image design scheme is represented by a background image design description text. This instruction description text includes scene information and object description text. The preset image generation model generates object images of the same object with different backgrounds based on each set of background image design schemes and the original object image. The reward model evaluates the two object images to obtain an evaluation result. Finally, the gradient update strategy of the visual language model is adjusted based on the evaluation result, and the visual language model is optimized through reinforcement learning.

[0098] As mentioned above, the preset image generation model can be built based on a diffusion model and a control network. For specific implementation methods of generating object images based on background images, descriptive text, and original object images, please refer to existing technologies; these will not be repeated in the embodiments of this application.

[0099] Taking the optimization of an advertising campaign for a new blue water bottle targeting the F country market as an example, the reinforcement training process is as follows: The first step involves inputting the object description text ("blue thermos"), the original object image, and the target scene ("Country F") into the visual language model to be fine-tuned. The visual language model then samples and generates two different background image design schemes. (e.g., "by the window of a Parisian café") and (e.g., "alongside a hiking trail in the Alps").

[0100] The second step is to design the background image. and The images are input into a preset image generation model to perform controlled image generation operations. Combined with the object edge information in the original object image, two complete advertising images are rendered. and .

[0101] The third step is to add the advertising image. and The object description text and scene information (such as "Country F") are input into the reward model, which is sensitive to weak scene signals as strong predictive variables. The reward model determines which advertising image is more attractive to users in Country F, that is, more in line with the scene preferences corresponding to the scene information, and determines the winning background image design scheme and the losing background image design scheme based on the judgment result.

[0102] The fourth step is to update the model parameters of the visual language model using the Direct Preference Optimization (DPO) algorithm. The DPO loss function encourages the model to increase the probability of generating winning background image designs while decreasing the probability of generating suboptimal background image designs.

[0103] By iterating through steps one through four, the reward model becomes sensitive to scene information. Through continuous iteration and optimization, the visual language model gradually learns to generate highly attractive background image design schemes for different scenes and objects.

[0104] The training method for the reward model is described in the previous embodiments and will not be repeated here.

[0105] In summary, the visual language model training method disclosed in this embodiment applies a reward model trained using the aforementioned method that is sensitive to weak scene signals as strong predictors to a reinforcement learning framework, driving the optimization of the visual language model. Because the reward model overcomes the weakening problem of strong predictors with weak signals, it can provide accurate and unbiased cross-scene preference reward signals. Therefore, when the reward model is used to drive the optimization of the visual language model within a reinforcement learning framework, the optimization direction provided by the reward model is correct and fine-grained, capable of distinguishing preference differences in different scenarios (such as different countries, different user groups, and different usage scenarios). Since the reward model, sensitive to weak scene signals as strong predictors, can distinguish preferences in different scenarios to a certain extent, when it evaluates the generated images corresponding to the two designs generated by the visual language model, the reward it provides truly reflects which design better suits the preferences of the target scenario. Only after direct preference optimization based on this signal can the generation strategy of the visual language model be effectively adjusted to a distribution that conforms to the preferences of a specific scenario. Tests showed that the generated images corresponding to the background image description text generated by the visual language model trained using this method significantly improved the scene adaptability score under the evaluation of the reward model, which is sensitive to weak scene signals as strong predictors. This proves that the reinforcement learning process successfully transformed the accurate scene preference judgment ability of the reward model, which is sensitive to weak scene signals as strong predictors, into the scene adaptive generation ability of the visual language model.

[0106] This method innovatively uses a reward model capable of accurately perceiving scene preferences as the reward signal for reinforcement learning to fine-tune a visual language model. This allows the fine-tuned model to generate background images and descriptive text that align with the preferences of the target scene, based on the original object image, object description text, and the target scene. Furthermore, combined with a controllable text-to-image model, an end-to-end "generation-evaluation-optimization" closed loop is formed, ultimately automating the production of object images optimized for specific scenes, thus improving the efficiency of generating object images for different scenarios.

[0107] To implement the above-described training method for the reward model, this application also discloses a method for constructing a training sample set for the reward model, such as... Figure 6 As shown, the method includes steps 602 to 608.

[0108] Step 602: Based on the preset query text, obtain multiple different advertising images of the same object.

[0109] The query text is a description of the query intent used to query objects in a preset object library.

[0110] Step 604: Obtain real user click preference data for the multiple different advertising images in multiple scenarios.

[0111] For specific implementation methods of obtaining real user click preference data for multiple different advertising images in multiple scenarios, please refer to the prior art, and will not be repeated in the embodiments of this application. Taking the acquisition of real user click preference data for products on an e-commerce platform in different national markets as an example, the real user click-through rate of the product's advertising images in each designated country can be obtained through the service interface opened by the e-commerce platform, which serves as the real user click preference data for each national market.

[0112] Step 606: Construct preference pair data based on the real user click preference data. Each preference pair data includes: the query text, at least two different advertising images of the same object, and user click preferences of the at least two different advertising images in multiple scenarios.

[0113] For example, a preference pair can be constructed based on a query text, two different advertising images of an object recalled by the query text, and the actual user click-through rate of each image in the same scene (such as the same country).

[0114] Step 608: Perform deduplication and multi-dimensional balancing on the preference data to obtain the training sample set of the reward model, wherein the multi-dimensional balancing includes scene dimension balancing and object category dimension balancing.

[0115] The deduplication process includes, but is not limited to: deleting redundant preference pairs corresponding to query text; the scenario-dimensional balancing process includes, but is not limited to: balancing the number of preference pairs in different scenarios; and the object category-dimensional balancing process includes, but is not limited to: balancing the number of preference pairs in different object categories.

[0116] The reward model is used to determine, based on each of the preference pairs, which of the at least two different advertising images matches the user's click preference for a specified scenario.

[0117] Furthermore, before training the reward model based on the training sample set, it is also necessary to obtain the scene information, enhanced answer text, and prompt instruction text corresponding to each training sample in the training sample set.

[0118] The method for obtaining the enhanced answer text is described in the previous embodiments and will not be repeated here.

[0119] Optionally, the scene information corresponds to the real user click preference data and can be obtained synchronously when acquiring the real user click preference data. The prompt instruction text can be generated according to a preset prompt instruction template.

[0120] The method for constructing the training sample set of the reward model disclosed in this application constructs each sample consisting of the same object, at least two different advertising images, and real user click preferences in multiple scenarios (such as national markets). This structure creates a contrasting scenario where high-quantity features (object description text, advertising images) remain unchanged while strong predictive variables (such as country) with weak signals change, providing an ideal experimental base for systematically studying the weakening problem of strong predictive variables with weak signals. Furthermore, the training samples incorporate real user preference data (such as click-through rate) as preference signals, and through rigorous object and image deduplication and balancing processing in the scenarios (such as country) and object category dimensions, ensure the authenticity, purity, and unbiasedness of the data, thus providing an indispensable foundation for reliably training and validating reward models sensitive to strong predictive variables with weak signals.

[0121] Based on the above embodiments, this embodiment also provides an image generation apparatus, the apparatus comprising: The input information acquisition module is used to acquire the multimodal information of the target object carried by the image generation instruction and the scene information matching the image to be generated, wherein the multimodal information includes: the object description text of the target object and the original object image; The first-stage generation module is used to take the scene information, the object description text, and the original object image as input information, and call a pre-fine-tuned visual language model to generate a background image design description text that is adapted to the scene information. When the scene information is a weak signal in the input information, the background image design description text can reflect the scene preference features matched by the scene information. The second-stage generation module is used to take the background image design description text and the original object image as input information, and call a preset image generation model to generate an image of the target object whose background matches the scene information, under the constraints of the background image design description text, the image of the target object in the original object image, and the layout information.

[0122] Optionally, the visual language model is obtained by reinforcement learning using the output of a reward model that is sensitive to weak scene signals as strong predictor variables as the reward signal.

[0123] Optionally, the visual language model is fine-tuned using the following method: The visual language model to be fine-tuned is trained using a supervised fine-tuning method. The model is designed to generate background images based on object description text and original object images to design description text, thus obtaining an initial fine-tuned visual language model. Using the initially fine-tuned visual language model as an agent, a reward signal is constructed based on the scene preference evaluation result of the generated image corresponding to the agent's action according to the preset reward model. The initially fine-tuned visual language model is then fine-tuned using a reinforcement learning method to obtain the visual language model. The reward model is sensitive to weak scene signals as strong predictive variables. The action includes generating a background image and designing descriptive text.

[0124] Optionally, the preset image generation model includes: a diffusion model and a control network, wherein the control network is used to determine the image and layout information of the target object based on the original object image.

[0125] The training method for the reward model is described in the previous embodiments and will not be repeated here.

[0126] The image generation apparatus disclosed in this application is used to implement the above-described image generation method. For the specific implementation of each module of the apparatus, please refer to the specific implementation of the corresponding steps in the foregoing method embodiments, which will not be repeated here.

[0127] In summary, the image generation apparatus disclosed in this application acquires multimodal information of the target object carried by the image generation instruction and scene information matching the image to be generated. The multimodal information includes: object description text of the target object and an original object image. Using the scene information, the object description text, and the original object image as input information, a pre-tuned visual language model is invoked to generate a background image design description text adapted to the scene information. When the scene information is a weak signal in the input information, the background image design description text can reflect the scene preference features matching the scene information. Using the background image design description text and the original object image as input information, a preset image generation model is invoked to generate an image of the target object with a background matching the scene information, under the constraints of the image and layout information of the target object in the background image design description text and the original object image. When training the visual language model, this apparatus uses the output of a reward model sensitive to weak scene signals as strong predictive variables as a reward signal, enabling the visual language model to fully focus on weak scene signals as strong predictive variables, thereby improving the preference matching degree between the generated image and the scene corresponding to the scene information.

[0128] Based on the above embodiments, this embodiment also provides a training device for a reward model, used to train a reward model sensitive to weak scene signals as strong predictive variables. The reward model includes: a text encoding module, a visual encoding module, a multimodal feature fusion module, a modality alignment module, and a classification prediction module. The device includes: The first encoding unit is used to perform text encoding processing on the scene information, enhanced answer text and prompt instruction text corresponding to each training sample in the pre-acquired training sample set through the text encoding module, so as to obtain the enhanced answer feature vector, instruction feature vector and scene feature vector corresponding to the training sample; The second encoding unit is used to perform visual encoding processing on the first image and the second image of the object corresponding to the prompt instruction text through the visual encoding module, so as to obtain the first visual feature vector corresponding to the first image and the second visual feature vector corresponding to the second image. The feature fusion unit is used to perform visual feature modulation on the first visual feature vector and the second visual feature vector respectively using the scene feature vector through the multimodal feature fusion module to obtain a first fused feature vector corresponding to the first image and a second fused feature vector corresponding to the second image. The inference unit is used to perform inference processing on the instruction feature vector, the first fused feature vector, and the second fused feature vector through the modality alignment module to obtain the hidden layer output; The prediction unit is used to perform classification prediction based on the hidden layer output and the enhanced answer feature vector through the classification prediction module to obtain the prediction result of the training sample; The iterative judgment unit is used to calculate the model loss based on the attention intensity penalty loss of each preset key feature in the prompt instruction text and the classification prediction loss corresponding to the prediction result, and to iteratively optimize the model parameters of the reward model with the goal of model loss convergence.

[0129] Optionally, the step of using the multimodal feature fusion module to perform visual feature modulation on the first visual feature vector and the second visual feature vector respectively using the scene feature vector to obtain the first fused feature vector corresponding to the first image and the second fused feature vector corresponding to the second image includes: The scene feature vector is mapped to affine transformation parameters through the multimodal feature fusion module. The first visual feature vector is subjected to an element-wise affine transformation using the affine transformation parameters to obtain a first fused feature vector corresponding to the first image; and the second visual feature vector is subjected to an element-wise affine transformation using the affine transformation parameters to obtain a second fused feature vector corresponding to the second image.

[0130] Optionally, the step of calculating the model loss based on the attention intensity penalty loss of each preset key feature in the prompt instruction text and the classification prediction loss corresponding to the prediction result, and iteratively optimizing the model parameters of the reward model with the goal of model loss convergence, includes: Based on the attention intensity of each preset key feature in the prompt instruction text according to the reward model, the attention intensity penalty loss corresponding to the training sample with the incorrect classification is obtained; Based on the difference between the predicted result of the training sample and the true value of the preset classification result, the classification prediction loss corresponding to the training sample is calculated. The model loss is obtained by weighted summing of the attention intensity penalty loss and the classification prediction loss corresponding to each training sample, wherein the attention intensity penalty loss corresponding to the correctly classified training sample is zero. With the goal of achieving model loss convergence, the model parameters of the reward model are iteratively optimized.

[0131] Optionally, the training samples include: query text, a first image, and a second image, and the enhanced answer text corresponding to the training samples is obtained through the following method: The objects in the preset object set are filtered in the first stage based on text similarity to obtain a first candidate object set; wherein the object description text of the objects in the first candidate object set and the query text of the training sample satisfy the preset text similarity condition. Based on image similarity, the objects in the first candidate object set are filtered in a second stage to obtain a second candidate object set and a third candidate object set; wherein, the object image of each object in the second candidate object set satisfies the preset image similarity condition with the first image, and the object image of each object in the third candidate object set satisfies the preset image similarity condition with the second image; Based on the preset evaluation index of each object in the second candidate object set, the preference score of the first image is calculated; and based on the preset evaluation index of each object in the third candidate object set, the preference score of the second image is calculated. Based on the comparison results of the preference scores, the enhanced answer text corresponding to the training samples is generated.

[0132] Optionally, the first-stage filtering of objects in the preset object set based on text similarity to obtain a first candidate object set includes: Obtain the text similarity between the object description text of each object in the preset object set and the query text of the training sample; The first M objects whose text similarity is less than a first preset similarity threshold are selected in descending order of text similarity to form a first candidate object set, where M is a natural number greater than 1.

[0133] Optionally, the second-stage filtering of objects in the first candidate object set based on image similarity to obtain a second candidate object set and a third candidate object set includes: Obtain the similarity between the object images of the objects in the first candidate object set and the first image of the first image, and the similarity between the object images and the second image of the second image; The first M objects whose first image similarity is less than the second preset similarity threshold are selected in descending order of the first image similarity to form a second candidate object set, and the first M objects whose second image similarity is less than the second preset similarity threshold are selected in descending order of the second image similarity to form a third candidate object set, where M is a natural number greater than 1.

[0134] Optionally, calculating the preference score of the first image based on the preset evaluation index of each object in the second candidate object set includes: Image similarity is used to weight the voting on the preset evaluation indicators of each object in the second candidate object set, and the aggregated results are used to obtain the preference score of the first image; The step of calculating the preference score of the second image based on the preset evaluation index of each object in the third candidate object set includes: The preference score of the second image is obtained by weighting the preset evaluation index of each object in the third candidate object set using image similarity. The device includes: The training apparatus for the reward model disclosed in this application is used to implement the training method of the above-mentioned reward model. For the specific implementation of each module of the apparatus, please refer to the specific implementation of the corresponding steps in the foregoing method embodiments, which will not be repeated here.

[0135] In summary, the training device for the reward model disclosed in this embodiment modulates the first and second visual feature vectors using the scene feature vectors during the training process. This allows the fused feature vectors to carry the visual preferences of a specific scene, enabling visual content to be differentiated based on the semantic embedding of different scenes. This achieves "scene-conditional" adaptation of the reward model at the feature level. Furthermore, when calculating the model loss, attention intensity penalty losses for scene features, object description text features, and image features are introduced. Additional penalties are imposed on variables that receive insufficient attention (such as scene information), ensuring that the reward model fully considers the "feature contribution" of scene information. This forces the model to actively and evenly utilize all key information during decision-making, helping to solve the problem of inaccurate classification results caused by weak scene information as a strong predictor variable.

[0136] To implement the above-described training method for the reward model, this application also discloses an apparatus for constructing a training sample set for the reward model, comprising: The image acquisition module is used to acquire multiple different advertising images of the same object based on preset query text; The real user click preference data acquisition module is used to acquire real user click preference data for the multiple different advertising images in multiple scenarios; The preference pair data construction module is used to construct preference pair data based on the real user click preference data. Each preference pair data includes: the query text, at least two different advertising images of the same object, and user click preferences of the at least two different advertising images in multiple scenarios. A training sample set construction module is used to perform deduplication and multi-dimensional balancing processing on the preference pair data to obtain the training sample set for the reward model. The multi-dimensional balancing processing includes scene dimension balancing processing and object category dimension balancing processing. Based on the above embodiments, this embodiment also provides a training device for a visual language model, the device comprising: The initial fine-tuning module is used to train the visual language model to be fine-tuned using a supervised fine-tuning method. The model is designed to generate background images based on object description text and original object images, and thus obtain the initial fine-tuned visual language model. The reinforcement training module is used to use the initially fine-tuned visual language model as an agent, construct a reward signal based on the scene preference evaluation result of the generated image corresponding to the agent's action according to the preset reward model, and fine-tune the initially fine-tuned visual language model using reinforcement learning method to obtain the visual language model. The reward model is sensitive to weak scene signals as strong predictive variables.

[0137] The training method for the reward model is described in the previous embodiments and will not be repeated here.

[0138] The visual language model training device disclosed in this embodiment is used to implement the above-mentioned visual language model training method. For the specific implementation of each module of the device, please refer to the specific implementation of the corresponding steps in the foregoing method embodiment, which will not be repeated here.

[0139] In summary, the visual language model training device disclosed in this application drives the optimization of the visual language model by applying a reward model trained using the aforementioned method that is sensitive to weak scene signals as strong predictors within a reinforcement learning framework. Because the reward model overcomes the weakening problem of strong predictors with weak signals, it can provide accurate and unbiased cross-scene preference reward signals. Therefore, when the reward model is used to drive the optimization of the visual language model within a reinforcement learning framework, the optimization direction provided by the reward model is correct and fine-grained, capable of distinguishing preference differences in different scenarios (such as different countries, different user groups, and different usage scenarios). Since the reward model, sensitive to weak scene signals as strong predictors, can distinguish preferences in different scenarios to a certain extent, when it evaluates the generated images corresponding to the two designs generated by the visual language model, the reward it provides truly reflects which design is more in line with the preference of the target scenario. Only after direct preference optimization based on this signal can the generation strategy of the visual language model be effectively adjusted to a distribution that conforms to the preferences of a specific scenario.

[0140] This application also provides a non-volatile readable storage medium storing one or more modules (programs). When these modules are applied to a device, they enable the device to execute the instructions for the method steps in this application.

[0141] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the methods described in this application.

[0142] This application also provides an electronic device, including: a processor and a memory communicatively connected to the processor; the memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory to implement the method described in this application embodiment. In this application embodiment, the electronic device includes devices such as servers and terminal devices.

[0143] This application also discloses a computer program product, including a computer program / computer executable instructions, which, when executed by a processor in an electronic device, implement the method described in this application.

[0144] Embodiments of this disclosure can be implemented as an apparatus with any suitable hardware, firmware, software, or any combination thereof, configured as desired, and the apparatus may include electronic devices such as servers (clusters) and terminals. Figure 7 An exemplary apparatus 700 is schematically shown that can be used to implement the various embodiments described in this application.

[0145] In one embodiment, Figure 7 An exemplary device 700 is shown, which includes one or more processors 702, a control module (chipset) 704 coupled to at least one of the processors 702, a memory 706 coupled to the control module 704, a non-volatile memory (NVM) / storage device 608 coupled to the control module 704, one or more input / output devices 710 coupled to the control module 704, and a network interface 712 coupled to the control module 704.

[0146] Processor 702 may include one or more single-core or multi-core processors, and processor 702 may include any combination of general-purpose processors or special-purpose processors (e.g., graphics processors, application processors, baseband processors, etc.). In some embodiments, device 700 can serve as a server, terminal, or other device as described in the embodiments of this application.

[0147] In some embodiments, the apparatus 700 may include one or more computer-readable media (e.g., memory 706 or NVM / storage device 708) having instructions 714 and one or more processors 702 that are combined with the one or more computer-readable media and configured to execute the instructions 714 to implement the module and thus perform the actions described in this disclosure.

[0148] In one embodiment, the control module 704 may include any suitable interface controller to provide any suitable interface to at least one of the processors 702 and / or any suitable device or component communicating with the control module 704.

[0149] The control module 704 may include a memory controller module to provide an interface to the memory 706. The memory controller module may be a hardware module, a software module, and / or a firmware module.

[0150] Memory 706 may be used, for example, to load and store data and / or instructions 714 for device 700. In one embodiment, memory 706 may include any suitable volatile memory, such as suitable DRAM. In some embodiments, memory 706 may include double data rate type 4 synchronous dynamic random access memory (DDR4 SDRAM).

[0151] In one embodiment, the control module 704 may include one or more input / output controllers to provide an interface to the NVM / storage device 708 and (one or more) input / output devices 710.

[0152] For example, NVM / storage device 708 may be used to store data and / or instructions 714. NVM / storage device 708 may include any suitable non-volatile memory (e.g., flash memory) and / or may include any suitable (one or more) non-volatile storage devices (e.g., one or more hard disk drives (HDDs), one or more optical disc drives (CDs), and / or one or more digital universal optical disc (DVD) drives).

[0153] NVM / storage device 708 may include storage resources that are part of a device on which device 700 is mounted, or that are accessible to the device but do not necessarily have to be part of the device. For example, NVM / storage device 708 may be accessed via a network through one or more input / output devices 710.

[0154] One or more input / output devices 710 may provide an interface for device 700 to communicate with any other suitable device. Input / output devices 710 may include communication components, audio components, sensor components, etc. A network interface 712 may provide an interface for device 700 to communicate via one or more networks. Device 700 may wirelessly communicate with one or more components of a wireless network according to any of one or more wireless network standards and / or protocols, such as accessing wireless networks based on communication standards, such as Bluetooth, WiFi, 2G, 3G, 4G, 5G, etc., or combinations thereof.

[0155] In one embodiment, at least one of the processors 702 may be logically packaged with one or more controllers (e.g., memory controller modules) of the control module 704. In one embodiment, at least one of the processors 702 may be logically packaged with one or more controllers of the control module 704 to form a system-in-package (SiP). In one embodiment, at least one of the processors 702 may be integrated with the logic of one or more controllers of the control module 704 on the same die. In one embodiment, at least one of the processors 702 may be integrated with the logic of one or more controllers of the control module 704 on the same die to form a system-on-a-chip (SoC).

[0156] In various embodiments, device 700 may be, but is not limited to, a component, integrated circuit, or chip in a terminal. The device may be a mobile electronic device or a non-mobile electronic device. For example, mobile electronic devices may be mobile phones, tablets, laptops, PDAs, in-vehicle electronic devices, wearable devices, ultra-mobile personal computers (UMPCs), netbooks, or personal digital assistants (PDAs), etc., while non-mobile electronic devices may be servers, network-attached storage (NAS), personal computers (PCs), televisions (TVs), ATMs, or self-service machines, etc. The embodiments of this application do not impose specific limitations.

[0157] In various embodiments, device 700 may have more or fewer components and / or different architectures. For example, in some embodiments, device 700 includes one or more cameras, a keyboard, a liquid crystal display (LCD) screen (including a touch screen display), a non-volatile memory port, multiple antennas, a graphics chip, an application-specific integrated circuit (ASIC), and a speaker.

[0158] The device can use a main control chip as a processor or control module, and sensor data, location information, etc. can be stored in a memory or NVM / storage device. The sensor group can be used as an input / output device, and the communication interface can include a network interface.

[0159] This application also provides an electronic device, including: a processor; and a memory storing executable code thereon. When the executable code is executed, the processor performs one or more methods as described in this application embodiment. In this application embodiment, the memory can store various types of data, such as target files, file-application association data, and user behavior data, thereby providing a data foundation for various processing operations.

[0160] This application also provides one or more machine-readable media having executable code stored thereon, which, when executed, causes a processor to perform one or more of the methods described in this application.

[0161] As the device embodiment is basically similar to the method embodiment, the description is relatively simple, and relevant parts can be found in the description of the method embodiment.

[0162] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0163] This application describes embodiments with reference to flowchart illustrations and / or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0164] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1The function specified in one or more boxes.

[0165] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal equipment, causing a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0166] Although preferred embodiments of the present application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the present application.

[0167] Finally, it should be noted that the terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one…" does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element. In the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following related objects are in an "or" relationship.

[0168] The foregoing has provided a detailed description of an image generation method, a reward model training method, a reward model training sample set construction method, a visual language model training method, an electronic device, a storage medium, and a computer program product provided by this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. An image generation method, characterized in that, The method includes: The multimodal information of the target object carried by the image generation instruction and the scene information matching the image to be generated are obtained, wherein the multimodal information includes: the object description text of the target object and the original object image; Using the scene information, the object description text, and the original object image as input information, a pre-tuned visual language model is invoked to generate a background image design description text that adapts to the scene information. When the scene information is a weak signal in the input information, the background image design description text can reflect the scene preference features matched by the scene information. Using the background image design description text and the original object image as input information, a preset image generation model is invoked to generate an image of the target object whose background matches the scene information, under the constraints of the target object's image and layout information in the background image design description text and the original object image.

2. The method according to claim 1, characterized in that, The visual language model is obtained by reinforcement learning using the output of a reward model that is sensitive to weak scene signals, which are strong predictors, as the reward signal.

3. The method according to claim 1, characterized in that, The visual language model was fine-tuned using the following method: The visual language model to be fine-tuned is trained using a supervised fine-tuning method. The model is designed to generate background images based on object description text and original object images to design description text, thus obtaining an initial fine-tuned visual language model. Using the initially fine-tuned visual language model as an agent, a reward signal is constructed based on the scene preference evaluation result of the generated image corresponding to the agent's action according to the preset reward model. The initially fine-tuned visual language model is then fine-tuned using a reinforcement learning method to obtain the visual language model. The reward model is sensitive to weak scene signals as strong predictive variables. The action includes generating a background image and designing descriptive text.

4. The method according to claim 1, characterized in that, The preset image generation model includes a diffusion model and a control network, wherein the control network is used to determine the image and layout information of the target object based on the original object image.

5. A training method for a reward model, characterized in that, The reward model includes: a text encoding module, a visual encoding module, a multimodal feature fusion module, a modality alignment module, and a classification prediction module; the method includes: The text encoding module performs text encoding processing on the scene information, enhanced answer text, and prompt instruction text corresponding to each training sample in the pre-acquired training sample set, and obtains the enhanced answer feature vector, instruction feature vector, and scene feature vector corresponding to the training sample. The visual encoding module performs visual encoding processing on the first image and the second image of the object corresponding to the prompt instruction text to obtain the first visual feature vector corresponding to the first image and the second visual feature vector corresponding to the second image. The multimodal feature fusion module modulates the first visual feature vector and the second visual feature vector using the scene feature vector to obtain the first fusion feature vector corresponding to the first image and the second fusion feature vector corresponding to the second image. The modality alignment module performs inference processing on the instruction feature vector, the first fused feature vector, and the second fused feature vector to obtain the hidden layer output. The classification prediction module performs classification prediction based on the hidden layer output and the enhanced answer feature vector to obtain the prediction result of the training sample; The model loss is calculated based on the attention intensity penalty loss of each preset key feature in the prompt instruction text and the classification prediction loss corresponding to the prediction result. The model parameters of the reward model are iteratively optimized with the convergence of the model loss as the goal.

6. The method according to claim 5, characterized in that, The step of using the multimodal feature fusion module to perform visual feature modulation on the first visual feature vector and the second visual feature vector respectively using the scene feature vector to obtain a first fused feature vector corresponding to the first image and a second fused feature vector corresponding to the second image includes: The scene feature vector is mapped to affine transformation parameters through the multimodal feature fusion module. The first visual feature vector is subjected to an element-wise affine transformation using the affine transformation parameters to obtain a first fused feature vector corresponding to the first image; and the second visual feature vector is subjected to an element-wise affine transformation using the affine transformation parameters to obtain a second fused feature vector corresponding to the second image.

7. The method according to claim 5, characterized in that, The model loss is calculated based on the attention intensity penalty loss of each preset key feature in the prompt instruction text and the classification prediction loss corresponding to the prediction result. The model parameters of the reward model are iteratively optimized with the goal of model loss convergence, including: Based on the attention intensity of each preset key feature in the prompt instruction text according to the reward model, the attention intensity penalty loss corresponding to the training sample with the incorrect classification is obtained; Based on the difference between the predicted result of the training sample and the true value of the preset classification result, the classification prediction loss corresponding to the training sample is calculated. The model loss is obtained by weighted summing of the attention intensity penalty loss and the classification prediction loss corresponding to each training sample, wherein the attention intensity penalty loss corresponding to the correctly classified training sample is zero. With the goal of achieving model loss convergence, the model parameters of the reward model are iteratively optimized.

8. The method according to claim 5, characterized in that, The training samples include: query text, a first image, and a second image. The enhanced answer text corresponding to the training samples is obtained through the following method: The objects in the preset object set are filtered in the first stage based on text similarity to obtain a first candidate object set; wherein the object description text of the objects in the first candidate object set and the query text of the training sample satisfy the preset text similarity condition. Based on image similarity, the objects in the first candidate object set are filtered in a second stage to obtain a second candidate object set and a third candidate object set; wherein, the object image of each object in the second candidate object set satisfies the preset image similarity condition with the first image, and the object image of each object in the third candidate object set satisfies the preset image similarity condition with the second image; Based on the preset evaluation index of each object in the second candidate object set, the preference score of the first image is calculated; and based on the preset evaluation index of each object in the third candidate object set, the preference score of the second image is calculated. Based on the comparison results of the preference scores, the enhanced answer text corresponding to the training samples is generated.

9. The method according to claim 8, characterized in that, The first-stage screening of objects in the preset object set based on text similarity to obtain a first candidate object set includes: Obtain the text similarity between the object description text of each object in the preset object set and the query text of the training sample; The first M objects whose text similarity is less than a first preset similarity threshold are selected in descending order of text similarity to form a first candidate object set, where M is a natural number greater than 1.

10. The method according to claim 8, characterized in that, The second-stage filtering of objects in the first candidate object set based on image similarity to obtain a second candidate object set and a third candidate object set includes: Obtain the similarity between the object images of the objects in the first candidate object set and the first image of the first image, and the similarity between the object images and the second image of the second image; The first M objects whose first image similarity is less than the second preset similarity threshold are selected in descending order of the first image similarity to form a second candidate object set, and the first M objects whose second image similarity is less than the second preset similarity threshold are selected in descending order of the second image similarity to form a third candidate object set, where M is a natural number greater than 1.

11. The method according to claim 8, characterized in that, The step of calculating the preference score of the first image based on the preset evaluation index of each object in the second candidate object set includes: Image similarity is used to weight the voting on the preset evaluation indicators of each object in the second candidate object set, and the aggregated results are used to obtain the preference score of the first image; The step of calculating the preference score of the second image based on the preset evaluation index of each object in the third candidate object set includes: Image similarity is used to weight the preset evaluation index of each object in the third candidate object set, and the preference score of the second image is obtained by aggregation.

12. A training method for a visual language model, characterized in that, include: The visual language model to be fine-tuned is trained using a supervised fine-tuning method. The model is designed to generate background images based on object description text and original object images to design description text, thus obtaining an initial fine-tuned visual language model. Using the initially fine-tuned visual language model as an agent, a reward signal is constructed based on the scene preference evaluation results of the generated image corresponding to the agent's actions according to a preset reward model. The initially fine-tuned visual language model is then fine-tuned using a reinforcement learning method to obtain the visual language model. The reward model is sensitive to weak scene signals, which are strong predictive variables.

13. A method for constructing a training sample set for a reward model, characterized in that, include: Based on the preset query text, obtain multiple different advertising images of the same object; Obtain real user click preference data for the multiple different advertising images in multiple scenarios; Based on the real user click preference data, preference pair data is constructed. Each preference pair data includes: the query text, at least two different advertising images of the same object, and user click preferences of the at least two different advertising images in multiple scenarios. The preference data is deduplicated and subjected to multi-dimensional balancing to obtain the training sample set of the reward model. The multi-dimensional balancing includes scene dimension and object category dimension.

14. An electronic device, characterized in that, include: The electronic device includes a processor, a memory, and a program or instructions stored in the memory and executable on the processor, wherein the program or instructions, when executed by the processor, implement the steps of the method as described in any one of claims 1-13.

15. A computer-readable storage medium, characterized in that, The readable storage medium stores a program or instructions that, when executed by a processor, implement the method as described in any one of claims 1-13.

16. A computer program product comprising a computer program / computer-executable instructions, characterized in that, When the computer program / computer executable instructions are executed by a processor in an electronic device, the method of any one of claims 1-13 is implemented.

Images