Article display graph generation model training method and article display graph generation method and device

Abstract

The invention discloses a training method of an article display graph generation model and an article display graph generation method and device, and relates to the technical field of computers. A specific embodiment of the training method for the article display graph generation model comprises the steps of determining image preference features based on acquired historical click behavior data of each historical user for historical articles, and then performing clustering processing on a historical user set to obtain historical crowd features; according to the article image feature, the article description feature and the historical crowd feature corresponding to each historical article, training to obtain an article display graph generation model; image preferences of different historical users can be effectively identified, and a training result adapts to complex user requirements. A specific embodiment of the article display graph generation method comprises the steps of inputting an article image feature, an article description feature, a target crowd feature and a display graph generation cue word corresponding to an obtained to-be-processed article into an article display graph generation model to obtain a target display graph; the user click rate and the conversion rate can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, and in particular, to a method and device for training an item display image generation model. BACKGROUND

[0002] The content of the item display image is crucial to attract consumers and sales growth. The existing solution is to use user click rate as a reward indicator to improve the attractiveness of the generated item display image. However, the above-mentioned method considers that all user click preferences are the same when training the model, and trains the model by using fixed item attributes and user attributes, which leads to the difficulty of the model training result to adapt to complex user needs, and the user click rate and conversion rate of the generated item display image are not high. SUMMARY

[0003] Therefore, the embodiments of the present application provide a method and device for training an item display image generation model, which analyzes the historical click behavior data of each historical user in the historical user set with respect to the historical item, determines the image preference feature of each historical user, clusters the historical user set according to the image preference feature, obtains the historical crowd feature, and then trains the model using the historical crowd feature, which can effectively identify the image preference of different historical users and adaptively aggregate the historical users into a crowd, so that the model training result can adapt to complex user needs and improve the user click rate and conversion rate.

[0004] To achieve the above-mentioned purpose, according to an aspect of the embodiments of the present application, a method for training an item display image generation model is provided, comprising: obtaining a historical user set corresponding to a historical item and historical click behavior data of each historical user in the historical user set with respect to the historical item, and determining an image preference feature of each historical user based on the historical click behavior data of each historical user with respect to the historical item; clustering each historical user in the historical user set according to the image preference feature of each historical user to obtain a historical crowd feature; training an item display image generation model according to the item image feature, the item description feature and the historical crowd feature corresponding to each historical item.

[0005] Optionally, determining the image preference feature of each historical user based on the historical click behavior data of each historical user with respect to the historical item comprises: determining the user attribute information of each historical user and the item image information and item description information of the historical item based on the historical click behavior data of each historical user; The user attribute information, the item image information and the item description information are encoded respectively to obtain user attribute features, item image features and item description features; According to a pre-configured interaction attention mechanism, the user attribute features, the item image features and the item description features are interacted to obtain image preference features of each historical user.

[0006] Optionally, each historical user in the historical user set is clustered according to the image preference features of each historical user to obtain historical crowd features, including: Each historical user in the historical user set is clustered based on the image preference features of each historical user to obtain a historical user group; According to the image preference features of each historical user in the historical user group, a feature center point corresponding to the historical user group is determined; Based on a preset proportion coefficient and the feature center point, the historical user group is sampled to obtain a preference feature point; The historical crowd features are constructed according to the feature center point and the preference feature point.

[0007] Optionally, an item display graph generation model is trained according to the item image features, the item description features and the historical crowd features corresponding to each historical item, including: The item image features, the item description features and the historical crowd features corresponding to each historical item are spliced to obtain a multi-modal input sequence corresponding to each historical item; The multi-modal input sequence corresponding to each historical item is input into a multi-modal large language model, and the multi-modal large language model is trained according to a pre-set pre-training task to obtain the item display graph generation model.

[0008] Optionally, after the item display graph generation model is trained according to the item image features, the item description features and the historical crowd features corresponding to each historical item, including: An item display graph control group corresponding to the historical item and a crowd perception prompt word are constructed; The historical crowd features, the item display graph control group and the crowd perception prompt word are input into a pre-trained user click rate prediction model to obtain a user click rate prediction result corresponding to the item display graph control group; The item display graph generation model is optimized based on the user click rate prediction result.

[0009] According to a second aspect of the embodiment of the present application, an item display graph generation method is provided, including: Item image features, item description features, target crowd features and display graph generation prompt words corresponding to a to-be-processed item are obtained; input the item image feature, the item description feature, the target crowd feature, and the display image generation prompt word into a pre-trained item display image generation model to obtain a target display image corresponding to the to-be-processed item; the item display image generation model is obtained by any of the methods in the first aspect of the embodiments of the present application.

[0010] According to a third aspect of the embodiments of the present application, a training device of an item display image generation model is provided, comprising: A determination module is configured to obtain a historical user set corresponding to historical items and historical click behavior data of each historical user in the historical user set with respect to the historical items, and determine an image preference feature of each historical user based on the historical click behavior data of each historical user with respect to the historical items. A clustering module is configured to perform clustering processing on each historical user in the historical user set according to the image preference feature of each historical user to obtain a historical crowd feature. A training module is configured to train an item display image generation model according to the item image feature, the item description feature, and the historical crowd feature corresponding to each historical item.

[0011] According to a fourth aspect of the embodiments of the present application, an item display image generation device is provided, comprising: An acquisition module is configured to acquire an item image feature, an item description feature, a target crowd feature, and a display image generation prompt word corresponding to a to-be-processed item. A generation module is configured to input the item image feature, the item description feature, the target crowd feature, and the display image generation prompt word into a pre-trained item display image generation model to obtain a target display image corresponding to the to-be-processed item; the item display image generation model is obtained by any of the methods in the first aspect of the embodiments of the present application.

[0012] According to a fifth aspect of the embodiments of the present application, an electronic device is provided, comprising: one or more processors; a storage device configured to store one or more programs, when the one or more programs are executed by the one or more processors, the one or more processors implement the method of any of the above embodiments.

[0013] According to a sixth aspect of the embodiments of the present application, a computer readable medium having a computer program stored thereon is provided, the program being executed by a processor to implement the method of any of the above embodiments.

[0014] According to a seventh aspect of the embodiments of the present application, a computer program product is provided, comprising a computer program, the computer program being executed by a processor to implement the method of any of the above embodiments.

[0015] An embodiment of the above application has the following advantages or beneficial effects: by obtaining a historical user set corresponding to a historical item and historical click behavior data of each historical user in the historical user set for the historical item, determining an image preference feature of each historical user based on the historical click behavior data of each historical user for the historical item; clustering each historical user in the historical user set according to the image preference feature of each historical user to obtain a historical population feature; and training an item display image generation model according to the item image feature, the item description feature and the historical population feature of each historical item, the embodiment determines the image preference feature of each historical user by analyzing the historical click behavior data of each historical user in the historical user set for the historical item, clusters the historical user set according to the image preference feature to obtain the historical population feature, and then trains the model using the historical population feature, which can effectively identify the image preference of different historical users and adaptively aggregate the historical users into a population, so that the model training result can adapt to complex user demands, improving the click rate and conversion rate of users. By obtaining an item image feature, an item description feature, a target population feature and a display image generation prompt word corresponding to a to-be-processed item; inputting the item image feature, the item description feature, the target population feature and the display image generation prompt word into a pre-trained item display image generation model to obtain a target display image corresponding to the to-be-processed item; wherein the item display image generation model is obtained by any method of the first aspect of the embodiment of the application; the user click rate and conversion rate can be improved.

[0016] Further effects of the above non-conventional optional mode will be described in the following with reference to the specific embodiments. BRIEF DESCRIPTION OF DRAWINGS

[0017] The accompanying drawings are used to better understand the application and do not constitute an improper limitation on the application. Among them: Figure 1 is a schematic diagram of the main process of the training method of the item display image generation model according to the embodiment of the application; Figure 2 is a schematic diagram of the main process of the item display image generation method according to the embodiment of the application; Figure 3 is a schematic diagram of the main modules of the training device of the item display image generation model according to the embodiment of the application; Figure 4 is a schematic diagram of the main modules of the item display image generation device according to the embodiment of the application; Figure 5 is an exemplary system architecture diagram to which the embodiment of the application can be applied; Figure 6Fig. 1 is a structural schematic diagram of a computer system of a terminal device or a server suitable for implementing embodiments of the present application. DETAILED DESCRIPTION

[0018] Exemplary embodiments of the present application are described herein with reference to the accompanying drawings, in which various details of the embodiments of the present application are set forth in order to provide a thorough understanding of the present application. It will be apparent to those skilled in the art that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present application. Also, the description set forth herein with reference to the accompanying drawings is merely illustrative and is not intended to limit the present application as described herein.

[0019] It should be noted that the acquisition, storage and application of personal information and the like involved in the embodiments of the present application comply with relevant laws and regulations and do not violate public order and good customs.

[0020] The content of the item display image is crucial to attracting consumers and increasing sales. The existing solution is to use user click rate as a reward indicator to improve the attractiveness of the generated item display image. However, the above method considers that all users have the same click preference when training the model, and the model is trained by fixed item attributes and user attributes, which makes it difficult for the model to adapt to complex user needs, and the user click rate and conversion rate of the generated item display image are not high.

[0021] Therefore, according to an aspect of an embodiment of the present application, a method for training an item display image generation model is provided.

[0022] Figure 1 Fig. 1 is a structural schematic diagram of a computer system of a terminal device or a server suitable for implementing embodiments of the present application. Figure 1 As shown in Fig. 1, the method for training an item display image generation model according to an embodiment of the present application mainly includes the following steps S101-S103.

[0023] In step S101, historical user sets corresponding to historical items and historical click behavior data of each historical user in the historical user sets for the historical items are obtained, and image preference features of each historical user are determined based on the historical click behavior data of each historical user for the historical items.

[0024] In the embodiment, the historical item is an item that has completed display, promotion or sales on the platform within a preset historical time window. The historical user is a user individual who has generated browsing, clicking or other interaction behaviors on the historical item within the historical time window, and the historical user set is composed of all users satisfying the above condition. The historical clicking behavior data is the clicking record data generated by the historical user in the process of browsing the above historical item display image, which includes the number of clicks, click time, corresponding display image identifier and user identifier and the like, and can be used to reflect the preference degree of the historical user to different item display images.

[0025] When the historical user set corresponding to the historical item and the historical clicking behavior data of each historical user in the historical user set for the historical item are obtained, the historical user behaviors within the historical time window can be filtered and associated based on platform log data or behavior embedding data, the clicking relationship mapping between the historical item and the historical user is established, and thus the clicking behavior data set of each historical user for different historical items is obtained. When the image preference feature of each historical user is determined based on the above historical clicking behavior data, the image features corresponding to the item display images clicked by the historical user can be weighted and aggregated, wherein the number of clicks or the clicking frequency is used as the weight to form a feature vector capable of representing the overall image preference of the user. Alternatively, the user-image preference sample can also be constructed based on the historical clicking behavior data corresponding to each historical user, and the above sample is encoded by the pre-trained preference modeling network, so as to extract the image preference features reflecting the preferences of the historical user in color, composition, style and visual elements and the like.

[0026] In step S102, each historical user in the historical user set is clustered according to the image preference feature of each historical user, and the historical crowd feature is obtained.

[0027] The historical crowd feature is a comprehensive representation of the crowd composed of multiple historical users in the visual preference level of the item display image, and is used to depict the common features of the same type of users in style preference, visual focus and clicking tendency and the like. The historical crowd feature is not a simple superposition of single user features, but a whole expression of the core preference features and their diversity distribution within the same crowd.

[0028] When clustering each historical user in the historical user set according to the image preference feature of each historical user, the similarity between the historical users can be calculated in the feature space based on the image preference feature of the historical users, and a clustering algorithm can be used to automatically group the historical users with high similarity, so that the historical users in the same group have similar image click preferences, thereby forming multiple historical crowds. The user image preference features in each historical crowd are aggregated to generate corresponding historical crowd features. In addition, a low-dimensional representation space can also be constructed based on the image preference features of the historical users, the preference difference is enhanced through feature mapping or attention mechanism, and then clustering processing is performed in the representation space, so that the clustering result can more accurately reflect the difference of users in visual style and element preference, and the historical crowd features are constructed based on the clustering center and the surrounding representative features.

[0029] In step S103, an item display image generation model is trained according to the item image feature, the item description feature and the historical crowd feature corresponding to each historical item.

[0030] The item image feature is a visual semantic feature extracted from the item display image corresponding to the historical item, which is used to represent the information of the item in terms of color distribution, structure layout, texture style and visual style. The item image feature can be obtained by encoding the item display image using a pre-trained image feature extraction network. The item description feature refers to a feature representation obtained by modeling the semantic information of the text description of the historical item. The text description information includes item title, attribute description, brand information and marketing copy information. The item description feature can be obtained by semantic vectorization through a text encoding model to reflect the semantic content and key information of the item.

[0031] During training of the item display image generation model, the item image feature, the item description feature and the corresponding historical crowd feature can be jointly encoded to form a multi-modal input, and the high click rate display image of the historical item or the corresponding generation prompt information is used as a supervision signal to perform end-to-end training on the item display image generation model, so that the model learns to generate an item display image that meets the crowd preference under the given item content and target crowd condition. In addition, a preference-driven optimization strategy can also be used to generate a candidate item display image based on the item image feature and the item description feature, and then evaluate the candidate item display image in combination with the historical crowd feature. The item display image generation model is iteratively updated through the click rate prediction or preference ranking result, so that the model gradually aligns the visual preference features of different historical crowds.

[0032] The embodiment obtains a historical user set corresponding to a historical item and historical click behavior data of each historical user in the historical user set for the historical item, determines an image preference feature of each historical user based on the historical click behavior data of each historical user for the historical item, performs clustering processing on each historical user in the historical user set according to the image preference feature of each historical user, and obtains a historical population feature; and the embodiment trains an item display image generation model according to the item image feature, the item description feature and the historical population feature of each historical item. The embodiment analyzes the historical click behavior data of each historical user in the historical user set for the historical item, determines the image preference feature of each historical user, and then clusters the historical user set according to the image preference feature to obtain the historical population feature, and further trains the model using the historical population feature, which can effectively identify the image preference of different historical users, and adaptively aggregates the historical users into a population, so that the model training result can adapt to complex user demands, and the user click rate and conversion rate are improved.

[0033] Optionally, determining the image preference feature of each historical user based on the historical click behavior data of each historical user for the historical item comprises: determining user attribute information of each historical user and item image information and item description information of the historical item based on the historical click behavior data of each historical user; performing encoding processing on the user attribute information, the item image information and the item description information respectively to obtain user attribute features, item image features and item description features; and interacting the user attribute features, the item image features and the item description features according to a pre-configured interaction attention mechanism to obtain the image preference feature of each historical user.

[0034] In the embodiment, first, the user behavior log is parsed to establish the click association relationship between the historical users and the historical items. On this basis, the user attribute information of each historical user is extracted from the user side, and the user attribute information includes but is not limited to the basic attributes of the user, the interest label, and the behavior statistical characteristics, etc.; the item image information and the item description information corresponding to the historical click behavior are obtained from the item side, and the item image information can be the display image of the historical item, and the item description information can be the text data used to describe the content of the item. Then, the user attribute information, the item image information, and the item description information are encoded respectively, wherein the user attribute information can generate user attribute characteristics through attribute embedding and feature fusion network, the item image information can generate item image characteristics through image feature extraction network, and the item description information can generate item description characteristics through a text encoding model, so as to map the multi-source heterogeneous data to the same feature space. Based on the pre-configured interaction attention mechanism, the user attribute characteristics are taken as the query vector, and the item image characteristics and the item description characteristics are taken as the context information for feature interaction modeling, so that the generated image preference characteristics can perceive the user attribute difference and the item content characteristics at the same time, thereby accurately reflecting the visual click preference of the historical user in the specific item scene. In addition, the user-item preference sample can be constructed based on the historical click behavior, and the image preference characteristics of the user are directly learned through the deep preference network jointly modeling the user attribute characteristics and the item multi-modal characteristics. The embodiment can fully integrate the user attributes, the item visual information, and the semantic information, etc., and improve the expression ability of the image preference characteristics.

[0035] Optionally, each historical user in the historical user set is clustered based on the image preference characteristics of each historical user to obtain historical crowd characteristics, including: clustering each historical user in the historical user set based on the image preference characteristics of each historical user to obtain a historical user group; determining a feature center point corresponding to the historical user group according to the image preference characteristics of each historical user in the historical user group; sampling the historical user group based on a preset proportion coefficient and the feature center point to obtain a preference feature point; and constructing the historical crowd characteristics according to the feature center point and the preference feature point.

[0036] In this embodiment, the image preference features of each historical user are taken as clustering inputs, similarity analysis is performed on the historical users in the historical user set, and the historical users with similar image preference features are automatically divided into the same historical user group in the feature space, so that the users in the same historical user group have high consistency in visual style, composition preference, click tendency and the like. After clustering is completed, statistical modeling is performed on the image preference features of all historical users in each historical user group, a feature center point that can represent the overall preference trend of the historical user group is calculated through feature aggregation, and the feature center point is used to depict the core visual preference of the historical user group. On this basis, to avoid that a single feature center point cannot comprehensively reflect the preference differences within the group, a preset proportion coefficient is introduced, the image preference features in the historical user group are sampled within different feature distance ranges based on the feature center point, and a plurality of representative preference feature points are obtained. The above preference feature points and the feature center point jointly constitute a multi-level description of the preference distribution of the historical user group, and finally combine to form a historical population feature, so that the feature contains the core preference of the group and retains the diversity information within the historical user group. In addition to the above implementation manners, the historical users can also be directly grouped through a clustering method based on density or hierarchical structure, and a historical population feature is constructed in combination with the statistical results of the feature distribution within the group.

[0037] The embodiment can effectively improve the expression ability of the population feature to the real user preference, provide more stable and fine population condition constraints for the training of a subsequent display image generation model, and thus improve the adaptation effect of the generation result to different populations.

[0038] Optionally, the item display image generation model is trained according to the item image feature, the item description feature and the historical population feature corresponding to each historical item, including: splicing the item image feature, the item description feature and the historical population feature corresponding to each historical item to obtain a multi-modal input sequence corresponding to each historical item; inputting the multi-modal input sequence corresponding to each historical item into a multi-modal large language model, and training and processing the multi-modal large language model according to a pre-set pre-training task to obtain the item display image generation model.

[0039] In this embodiment, the item image feature, the item description feature and the historical population feature corresponding to each historical item are processed in terms of dimension alignment, format unification and the like, so that they can be jointly modeled in the same feature space. Then, the item image feature, the item description feature and the historical population feature are spliced in a preset order to construct a multi-modal input sequence corresponding to each historical item, and the multi-modal input sequence can represent visual information, semantic information and preference features of a target population of the item and the like.

[0040] In the model training stage, the multi-modal input sequence is input into the multi-modal large language model, and the multi-modal large language model is trained and processed based on the pre-set pre-training task. The pre-training task includes but is not limited to modeling the understanding of the crowd characteristics and learning the generation of related prompt information of the item display graph, so that the multi-modal large language model gradually possesses the ability to perceive the differences of the crowd and generate display graphs that meet the preferences of the crowd while fusing multi-modal information. Through the above training process, the multi-modal large language model is optimized as an item display graph generation model. The embodiment can realize the deep fusion of item content information and crowd preferences, and improve the adaptability and click effect of the generated item display graph under different crowds.

[0041] In other embodiments, a phased training strategy can also be used. First, the multi-modal large language model is pre-trained based on the item image features and item description features, and then the historical crowd features are introduced as conditional information to fine-tune the model, thereby further enhancing the model's ability to distinguish different crowd preferences.

[0042] Optionally, after training the item display graph generation model according to the item image features, item description features and historical crowd features corresponding to each historical item, the method further includes: constructing an item display graph control group corresponding to the historical item and a crowd perception prompt word; inputting the historical crowd features, the item display graph control group and the crowd perception prompt word into a pre-trained user click rate prediction model to obtain a user click rate prediction result corresponding to the item display graph control group; and optimizing the item display graph generation model based on the user click rate prediction result.

[0043] After the initial training of the item display graph generation model is completed, an optimization process based on crowd preferences can be further introduced. First, an item display graph control group is constructed for each historical item, which includes multiple candidate display graphs obtained by different display graph generation strategies or different generation prompt conditions. According to the historical crowd characteristics, crowd perception prompt words related to the target crowd preferences are generated to explicitly express the crowd conditions and preference constraints faced by the current display graph evaluation. Then, the historical crowd characteristics, item display graph control group and crowd perception prompt words are jointly input into the pre-trained user click rate prediction model. The user click rate prediction model evaluates the click tendency of each item display graph in the item display graph control group based on the matching degree between the crowd characteristics and the display graph content, and outputs the corresponding user click rate prediction result or relative click rate ranking result, thereby reflecting the expected performance difference of different item display graphs under the target crowd. On this basis, the item display graph generation model is optimized and updated according to the user click rate prediction result, for example, by reinforcement learning or preference alignment strategy, increasing the weight of the generation path corresponding to the high click rate item display graph, and suppressing the generation result corresponding to the low click rate item display graph, so that the model gradually aligns the visual preferences of the target crowd. In addition to the above-mentioned manner, preference sample pairs can also be constructed based on the prediction result to optimize the item display graph generation model through a contrast learning method. The embodiment can effectively improve the click effect of the generated item display graph in different crowd scenarios and reduce the conflict of cross-crowd preferences.

[0044] According to a second aspect of an embodiment of the present application, an item display graph generation method is provided.

[0045] Figure 2 is a schematic diagram of the main process of the item display graph generation method according to an embodiment of the present application; as Figure 2 shown, the item display graph generation method according to an embodiment of the present application mainly includes the following steps S201 to S202.

[0046] Step S201, obtaining item image features, item description features, target crowd characteristics and display graph generation prompt words corresponding to a to-be-processed item.

[0047] Step S202, inputting the item image features, item description features, target crowd characteristics and display graph generation prompt words into a pre-trained item display graph generation model to obtain a target display graph corresponding to the to-be-processed item; wherein the item display graph generation model is obtained by any method of the first aspect of the present application.

[0048] In this embodiment, for a to-be-processed item, first, multi-source input information corresponding to the to-be-processed item is acquired, including item image features, item description features, target crowd features, and display graph generation prompt words, and the like. The item image features can be obtained by encoding an original item image of the to-be-processed item through an image feature extraction network, and are used to represent appearance information of the item at a visual level; the item description features can be generated by a text encoding model from text description information of the to-be-processed item, and are used to reflect semantic attributes and key information of the item. The target crowd features can be determined from historical user clustering results or online crowd recognition results, and are used to depict overall visual preferences of a target user group in a current display scene. The display graph generation prompt words are used to constrain the generation direction of the item display graph, and the content thereof can include style requirements, composition guidelines, or marketing appeals, and is matched with the target crowd features.

[0049] After the above information acquisition is completed, the item image features, the item description features, the target crowd features, and the display graph generation prompt words are uniformly organized to form a multi-modal input sequence conforming to a model input format, and the multi-modal input sequence is input to a pre-trained item display graph generation model. The item display graph generation model can be obtained based on the training method constructed in the first aspect embodiment, and has learned a mapping relationship between item content information and crowd preference features inside, so as to generate a target display graph corresponding to the to-be-processed item on the basis of comprehensively considering item visual semantics, text semantics, and target crowd preferences. In other embodiments, multi-candidate sampling can also be performed on the generation result in the model inference stage, and the candidate display graphs are sorted or filtered in combination with the target crowd features, so as to select a display graph most conforming to the target crowd preferences as the final output.

[0050] Through the technical solution of this embodiment, adaptive generation of item display graphs in different crowd scenes can be realized, the matching degree of the display graph and the visual preferences of the target crowd is improved, and therefore the click rate and the overall display effect of the display graph are effectively improved.

[0051] According to a preferred embodiment of the present application, in a large-scale e-commerce scene, the visual preferences of different user groups are simultaneously satisfied through a unified generation model, so as to improve the click rate of the generated item display graph. In this preferred embodiment, first, an adaptive user aggregation process of item perception is performed. Image preference features are extracted based on image click behaviors of historical users, and the historical user click behaviors are decomposed into user attribute information A, item image information I, and item description information T, and the like. The item image information I and the item description information T are respectively extracted by using a dedicated encoder E I and E T to extract corresponding semantic features:

[0052] wherein, is an image feature of the item, is a description feature of the item.

[0053] For user attribute information, an embedding layer is used to project each attribute, and then each attribute is spliced to input a multi-layer perception to obtain the interaction of the attribute, thereby obtaining the user attribute representation .

[0054] In order to enable the historical user preference to perceive the specific item context, first based on the interaction between the user attribute feature and the item description feature:

[0055] Wherein, CA represents an interaction attention layer, is able to encode the user's preference according to the specific item content.

[0056] Further, the representation is interacted with the item image feature:

[0057] The final obtained representation, which represents the image click preference of the user under the condition of specific item and specific image. Based on the above representation, each branch uses a specific prediction head h to produce a scalar:

[0058] By minimizing the cross-entropy of the click label y and the predicted , the parameters of the model are trained end-to-end, so that the obtained image preference feature has good discrimination ability.

[0059] On this basis, for the same item, the historical users with similar image preference features are adaptively aggregated into the same historical group. Specifically, for the item s, K-means clustering is used to find the group aggregation mode with the smallest intra-class distance. Since the preferences within the group are not completely unified, it is difficult to completely describe the preference of the whole group using only a feature center point. Therefore, the embodiment constructs a comprehensive historical group feature by sampling preference feature points with different distances from the feature center point:

[0060] Wherein, represents the feature points sampled according to the percentile, and u k represents the feature center point. It is ensured that both the core preference feature and the surrounding preference feature are extracted, so that the subsequent preference alignment stage is more accurate.

[0061] After completing the historical population feature construction, enter the e-commerce knowledge and population feature pre-training phase. The embodiment proposes a multi-modal large language model G-MLLM that perceives the population, injects historical population features as explicit conditions into the input sequence of the multi-modal large language model, so that the model can explicitly perceive the preference characteristics of the target population while processing the image features and description features of the item. Through the design of user understanding tasks and prompt word generation tasks, the model is pre-trained to enable the model to understand the commonalities and differences between different populations, and to generate high-quality display image generation prompt words that meet the advertising creative scene, thereby laying the foundation for the subsequent generation process. The embodiment defines the following two pre-training tasks: one is the user understanding task, which aims to help G-MLLM understand the representation of users and the commonalities between users. G-MLLM first learns to describe the user's portrait, for example, in the form of natural language. Then it predicts the potential users who click on a specific item, which enables the model to extract users with the same click preference. The second is the prompt word generation task, which aims to train G-MLLM to generate diverse prompt words to describe the advertising creative image. The input of this task contains diverse item attributes, including item see-through images, titles, brands, etc. These tasks can help the model adapt to multi-modal elements in the advertising scene.

[0062] After pre-training, a population preference alignment mechanism is further introduced to make the model output consistent with the real-world population click preference. Based on the text-to-image model, multiple sets of advertising display images are generated for the same item, and a user click rate prediction model that perceives the population is constructed based on the pre-trained model. The model takes historical population features, display image pairs, and population perception prompt words as input to predict the relative click rate of different display images under the target population. Based on the prediction results, preference samples containing high click rate prompt words and low click rate prompt words are constructed, and an optimization strategy based on population preference is used to update the multi-modal large language model, so that the model aligns the preference characteristics of different populations while optimizing the click rate target. Specifically, first, ControlNet and Stable diffusion are combined as a text-to-image model to generate display images for items. Then, based on G-MLLM, it is extended to a population-perceiving reward model GRM. The model takes historical population features, display image pairs, and corresponding instructions X T to obtain implicit representations of preferences . Then, through a classification head , the relative click rate of the two item display images is predicted:

[0063] where, represents when facing the population click rate of the nth picture. The relative click rate prediction result of the GRM is used to construct a preference tuple (s, y and the preference tuple (s, , y w , y l ), wherein y w and y l represent prompt words corresponding to the image display of the item s with higher and lower CTR. The G-MLLM is followed by a Group-DPO to maximize the probability of the winning prompt word, directly optimize the G-MLLM, and generate prompt words with higher CTR for a specific target group, effectively solving the preference conflict across groups.

[0064] The embodiment realizes personalized generation of image displays of different items and different groups under the unified generation model framework, effectively alleviates the multi-group preference conflict problem, and improves the click effect of the generated image display under the target group.

[0065] According to a third aspect of an embodiment of the present application, a device for training an image display generation model is provided.

[0066] Figure 3 is a schematic diagram of main modules of the device for training the image display generation model according to an embodiment of the present application, as shown in Figure 3 The device for training the image display generation model 300 comprises: A determination module 301 is configured to obtain a historical user set corresponding to historical items and historical click behavior data of each historical user in the historical user set for the historical items, and determine image preference features of each historical user based on the historical click behavior data of each historical user for the historical items. A clustering module 302 is configured to perform clustering processing on each historical user in the historical user set according to the image preference features of each historical user, and obtain historical group features. A training module 303 is configured to train an image display generation model according to item image features, item description features, and historical group features corresponding to each historical item.

[0067] Optionally, the determination module 301 is configured to: determine user attribute information of each historical user and item image information and item description information of the historical items based on the historical click behavior data of each historical user; perform encoding processing on the user attribute information, the item image information, and the item description information respectively to obtain user attribute features, item image features, and item description features; According to the pre-configured interaction attention mechanism, the user attribute feature, the item image feature and the item description feature are interacted to obtain an image preference feature of each historical user.

[0068] Optionally, the clustering module 302 is further configured to: Based on the image preference feature of each historical user, each historical user in the historical user set is clustered to obtain a historical user group; According to the image preference feature of each historical user in the historical user group, a feature center point corresponding to the historical user group is determined; Based on the preset proportion coefficient and the feature center point, the historical user group is sampled to obtain a preference feature point; The historical crowd feature is constructed according to the feature center point and the preference feature point.

[0069] Optionally, the training module 303 is further configured to: The item image feature, the item description feature and the historical crowd feature corresponding to each historical item are spliced to obtain a multi-modal input sequence corresponding to each historical item; The multi-modal input sequence corresponding to each historical item is input into a multi-modal large language model, and the multi-modal large language model is trained according to a pre-set pre-training task to obtain an item display graph generation model.

[0070] Optionally, the training device 300 further comprises an optimization module, and the optimization module is configured to: An item display graph control group and a crowd perception prompt word corresponding to the historical item are constructed; The historical crowd feature, the item display graph control group and the crowd perception prompt word are input into a pre-trained user click rate prediction model to obtain a user click rate prediction result corresponding to the item display graph control group; The item display graph generation model is optimized based on the user click rate prediction result.

[0071] It should be noted that the specific implementation content of the training device of the item display graph generation model in the embodiment of the application has been described in detail in the above training method of the item display graph generation model, and therefore the repeated content will not be described here.

[0072] According to a fourth aspect of the embodiment of the application, an item display graph generation device is provided.

[0073] Figure 4 is a schematic diagram of main modules of the item display graph generation device according to the embodiment of the application, as Figure 4 shown, an item display graph generation device 400 comprises: The acquisition module 401 is configured to acquire an item image feature corresponding to the to-be-processed item, an item description feature, a target crowd feature, and a display map generation prompt word. The generation module 402 is configured to input the item image feature, the item description feature, the target crowd feature, and the display map generation prompt word into a pre-trained item display map generation model to obtain a target display map corresponding to the to-be-processed item. The item display map generation model is obtained by any of the methods in the first aspect of the embodiments of the present application.

[0074] It should be noted that the specific implementation of the item display map generation device in the embodiments of the present application has been described in detail in the above item display map generation method, and therefore repeated content will not be described here.

[0075] According to a fifth aspect of the embodiments of the present application, an electronic device is provided, including: one or more processors; a storage device configured to store one or more programs, when the one or more programs are executed by the one or more processors, the one or more processors implement the method provided by the first aspect and / or the second aspect of the embodiments of the present application.

[0076] According to a sixth aspect of the embodiments of the present application, a computer readable medium is provided, which stores a computer program. The program is executed by a processor to implement the method provided by the first aspect and / or the second aspect of the embodiments of the present application.

[0077] According to a seventh aspect of the embodiments of the present application, a computer program product is provided, which includes a computer program. The computer program is executed by a processor to implement the method provided by the first aspect and / or the second aspect of the embodiments of the present application.

[0078] Figure 5 An exemplary system architecture 500 is shown, which can apply the training method of the item display map generation model or the training device of the item display map generation model of the embodiments of the present application.

[0079] Figure 5 An exemplary system architecture 500 is shown, which can apply the item display map generation method or the item display map generation device of the embodiments of the present application.

[0080] As shown in Figure 5 The system architecture 500 can include terminal devices 501, 502, 503, a network 504, and a server 505. The network 504 is used as a medium to provide a communication link between the terminal devices 501, 502, 503 and the server 505. The network 504 can include various connection types, such as wired, wireless communication links, or optical fiber cables, etc.

[0081] The user can use the terminal devices 501, 502, 503 to interact with the server 505 through the network 504 to receive or send messages, etc. Various communication user end applications can be installed on the terminal devices 501, 502, 503, such as shopping applications, web browser applications, search applications, instant messaging tools, mailbox user ends, social platform software, etc. (only as examples).

[0082] The terminal devices 501, 502, 503 can be various electronic devices with display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, etc.

[0083] The server 505 can be a server providing various services, such as a background management server supporting a shopping website browsed by the user using the terminal device 501, 502, 503 (only as an example). The background management server can analyze and process received model training requests and other data, and feed back the processing results (item display image generation model - only as an example) to the terminal device.

[0084] It should be noted that the training method of the item display image generation model provided by the embodiment of the application is generally executed by the server 505, and correspondingly, the training device of the item display image generation model is generally provided in the server 505. The training method of the item display image generation model provided by the embodiment of the application can also be executed by the terminal device 501, 502, 503, and correspondingly, the training device of the item display image generation model can be provided in the terminal device 501, 502, 503.

[0085] It should be noted that the item display image generation method provided by the embodiment of the application is generally executed by the server 505, and correspondingly, the item display image generation device is generally provided in the server 505. The item display image generation method provided by the embodiment of the application can also be executed by the terminal device 501, 502, 503, and correspondingly, the item display image generation device can be provided in the terminal device 501, 502, 503.

[0086] It should be understood that Figure 5 The number of terminal devices, networks and servers in the above description is only illustrative. According to the needs of implementation, there can be any number of terminal devices, networks and servers.

[0087] The following refers to Figure 6 which shows a structural schematic diagram of a computer system 600 of a terminal device suitable for implementing the embodiments of the application. Figure 6 The terminal device shown is only an example and should not impose any limitation on the functions and use range of the embodiments of the application.

[0088] like Figure 6 As shown, the computer system 600 includes a central processing unit (CPU) 601, which can perform various appropriate actions and processes based on programs stored in read-only memory (ROM) 602 or programs loaded from storage section 608 into random access memory (RAM) 603. The RAM 603 also stores various programs and data required for the operation of the system 600. The CPU 601, ROM 602, and RAM 603 are interconnected via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0089] The following components are connected to I / O interface 605: an input section 606 including a keyboard, mouse, etc.; an output section 607 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 608 including a hard disk, etc.; and a communication section 609 including a network interface card such as a LAN card, modem, etc. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to I / O interface 605 as needed. A removable medium 611, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 610 as needed so that computer programs read from it can be installed into storage section 608 as needed.

[0090] In particular, according to the embodiments disclosed in this invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this invention include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 609, and / or installed from removable medium 611. When the computer program is executed by central processing unit (CPU) 601, it performs the functions defined above in the system of this invention.

[0091] It should be noted that the computer-readable medium shown in the present application can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may, for example, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or component, or any combination of the above. More specific examples of computer-readable storage media can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, the computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device or component. In the present application, the computer-readable signal medium can include a data signal carried in a baseband or as a part of a carrier wave, which carries computer-readable program code. Such a propagated data signal can take various forms, including but not limited to an electromagnetic signal, an optical signal or any suitable combination of the above. The computer-readable signal medium can also be any computer-readable medium other than the computer-readable storage medium, which can send, propagate or transmit a program for use by or in conjunction with an instruction execution system, device or component. The program code contained on the computer-readable medium can be transmitted by any suitable medium, including but not limited to wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0092] The flowcharts and block diagrams in the drawings illustrate the possible implementation architectures, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. In this regard, each block in the flowcharts or block diagrams can represent a module, a program segment or a part of code containing one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions noted in the blocks can occur in different order than that shown in the drawings. For example, two blocks that are shown in succession can actually be executed substantially in parallel, and sometimes in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams or flowcharts, and the combination of blocks in the block diagrams or flowcharts, can be implemented by a dedicated hardware-based system that performs the specified functions or operations, or can be implemented by a combination of dedicated hardware and computer instructions.

[0093] The modules described in the embodiments of the present application can be implemented in the form of software or in the form of hardware. The described modules can also be arranged in a processor, for example, a processor can be described as including a determination module, a clustering module, and a training module, wherein the names of these modules do not constitute a limitation on the modules themselves in some cases, for example, the training module can also be described as "a module for training an item display graph generation model according to item image features, item description features, and historical crowd features corresponding to each historical item". Alternatively, a processor includes an acquisition module and a generation module, wherein the names of these modules do not constitute a limitation on the modules themselves in some cases, for example, the generation module can also be described as "a module for inputting item image features, item description features, target crowd features, and display graph generation prompt words into a pre-trained item display graph generation model to obtain a target display graph corresponding to a to-be-processed item".

[0094] As another aspect, the present application also provides a computer readable medium, which can be included in the device described in the above embodiments, or can exist independently without being assembled into the device. The computer readable medium carries one or more programs, and when the one or more programs are executed by the device, the device implements the following method: acquiring a historical user set corresponding to a historical item and historical click behavior data of each historical user in the historical user set for the historical item, determining image preference features of each historical user based on the historical click behavior data of each historical user for the historical item; performing clustering processing on each historical user in the historical user set according to the image preference features of each historical user, to obtain historical crowd features; and training an item display graph generation model according to item image features, item description features, and historical crowd features corresponding to each historical item. Alternatively, the device implements the following method: acquiring item image features, item description features, target crowd features, and display graph generation prompt words corresponding to a to-be-processed item; inputting the item image features, item description features, target crowd features, and display graph generation prompt words into a pre-trained item display graph generation model to obtain a target display graph corresponding to the to-be-processed item; wherein the item display graph generation model is obtained by any of the methods in the first aspect of the present application.

[0095] The computer program product provided by the embodiments of the present application includes a computer program, and the computer program implements the training method of the item display graph generation model in the first aspect of the present application and / or the item display graph generation method in the second aspect of the present application when executed by a processor.

[0096] According to the technical scheme of the embodiment of the present application, the following advantages or beneficial effects are achieved: the historical user set corresponding to the historical item and the historical click behavior data of each historical user in the historical user set for the historical item are obtained, the image preference feature of each historical user is determined based on the historical click behavior data of each historical user for the historical item, each historical user in the historical user set is clustered according to the image preference feature of each historical user, and the historical crowd feature is obtained, the item display image generation model is trained according to the item image feature, the item description feature and the historical crowd feature of each historical item, the image preference feature of each historical user is determined by analyzing the historical click behavior data of each historical user in the historical user set for the historical item, the historical user set is clustered according to the image preference feature, the historical crowd feature is obtained, and then the model is trained by using the historical crowd feature, which can effectively identify the image preference of different historical users, and the historical users are adaptively aggregated into a crowd, so that the model training result can adapt to complex user demand, and the user click rate and conversion rate are improved. The item image feature, the item description feature, the target crowd feature and the display image generation prompt word corresponding to the to-be-processed item are obtained, the item image feature, the item description feature, the target crowd feature and the display image generation prompt word are input into the pre-trained item display image generation model, and the target display image corresponding to the to-be-processed item is obtained, wherein the item display image generation model is obtained by any method in the first aspect of the embodiment of the present application, and the user click rate and conversion rate can be improved.

[0097] The above specific embodiments do not constitute a limitation on the protection scope of the present application. Those skilled in the art should understand that various modifications, combinations, sub-combinations and substitutions can be made depending on design requirements and other factors. Any modification, equivalent replacement and improvement made within the spirit and principles of the present application should be included in the protection scope of the present application.

[0098] It should be noted that in the technical scheme of the present disclosure, the acquisition, storage and application of user personal information involved comply with relevant laws and regulations and do not violate public order and good customs.

Claims

1. A method of training an article display map generation model, characterized by, The method comprises the following steps: acquiring a historical user set corresponding to a historical item and historical click behavior data of each historical user in the historical user set for the historical item, and determining an image preference feature of each historical user based on the historical click behavior data of each historical user for the historical item; performing clustering processing on each historical user in the historical user set according to the image preference feature of each historical user, to obtain a historical crowd feature; training an item display graph generation model based on an item image feature, an item description feature and the historical crowd feature corresponding to each historical item.

2. The method of claim 1, wherein, The method for determining the image preference feature of each historical user based on the historical click behavior data of each historical user for the historical item comprises the following steps: determining user attribute information of each historical user and item image information and item description information of the historical item based on the historical click behavior data of each historical user; performing encoding processing on the user attribute information, the item image information and the item description information respectively, to obtain a user attribute feature, an item image feature and an item description feature; interacting the user attribute feature, the item image feature and the item description feature according to a pre-configured interaction attention mechanism, to obtain the image preference feature of each historical user.

3. The method of claim 1, wherein, The method for performing clustering processing on each historical user in the historical user set according to the image preference feature of each historical user comprises the following steps: performing clustering on each historical user in the historical user set based on the image preference feature of each historical user, to obtain a historical user group; determining a feature center point corresponding to the historical user group according to the image preference feature of each historical user in the historical user group; performing sampling processing on the historical user group based on a pre-set proportion coefficient and the feature center point, to obtain a preference feature point; constructing the historical crowd feature according to the feature center point and the preference feature point.

4. The method of claim 1, wherein, The method for training the item display graph generation model based on an item image feature, an item description feature and the historical crowd feature corresponding to each historical item comprises the following steps: splicing the item image feature, the item description feature and the historical crowd feature corresponding to each historical item, to obtain a multi-modal input sequence corresponding to each historical item; inputting the multi-modal input sequence corresponding to each historical item into a multi-modal large language model, and performing training processing on the multi-modal large language model according to a pre-set pre-training task, to obtain the item display graph generation model.

5. The method of claim 1, wherein, After the item display graph generation model is trained based on an item image feature, an item description feature and the historical crowd feature corresponding to each historical item, the method comprises the following steps: constructing an item display graph control group and a crowd perception prompt word corresponding to the historical item; inputting the historical crowd feature, the item display graph control group and the crowd perception prompt word into a pre-trained user click rate prediction model, to obtain a user click rate prediction result corresponding to the item display graph control group; Optimize the item display graph generation model based on the user click rate prediction result.

6. An article display map generation method characterized by comprising: The method comprises the following steps: obtain the item image feature, item description feature, target crowd feature, and display graph generation prompt word corresponding to the to-be-processed item; input the item image feature, item description feature, target crowd feature, and display graph generation prompt word into a pre-trained item display graph generation model to obtain a target display graph corresponding to the to-be-processed item; wherein the item display graph generation model is obtained by the method in any one of claims 1 to 5.

7. A training device for generating an item display image model, characterized in that, The method comprises the following steps: determine the historical user set corresponding to the historical item and the historical click behavior data of each historical user in the historical user set for the historical item, determine the image preference feature of each historical user based on the historical click behavior data of each historical user for the historical item; cluster each historical user in the historical user set according to the image preference feature of each historical user to obtain a historical crowd feature; train the item display graph generation model according to the item image feature, item description feature, and historical crowd feature corresponding to each historical item.

8. An article display map generation apparatus characterized by comprising: The method comprises the following steps: obtain the item image feature, item description feature, target crowd feature, and display graph generation prompt word corresponding to the to-be-processed item; input the item image feature, item description feature, target crowd feature, and display graph generation prompt word into a pre-trained item display graph generation model to obtain a target display graph corresponding to the to-be-processed item; wherein the item display graph generation model is obtained by the method in any one of claims 1 to 5.

9. An electronic device, comprising: The method comprises the following steps: one or more processors; a storage device for storing one or more programs, when the one or more programs are executed by the one or more processors, the one or more processors implement the method in any one of claims 1-6.

10. A computer readable medium having stored thereon a computer program, characterized in that, The program is executed by the processor to implement the method in any one of claims 1-6.

11. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement the method in any one of claims 1-6.

Images