Poster layout generation model training method and device, equipment and readable storage medium

Abstract

The application discloses a poster layout generation model training method and device, equipment and a readable storage medium, and relates to the technical field of artificial intelligence and computer vision. The method comprises the following steps: acquiring a training set, the training set comprising a plurality of poster images to be generated and a description text corresponding to each poster image, the description text being used for describing the design requirements of the poster layout in a thought chain; selecting the poster image and the description text corresponding to the poster image from the training set, converting the poster image into a first image vector, and converting the description text into a text vector; inputting the first image vector and the text vector into a visual language model, obtaining an output result in a thought chain after fusion processing; updating the model parameters in the visual language model based on a first loss function determined by the output result in the thought chain, and obtaining a poster layout generation model. The application can organically combine multi-modal information such as text and images, and accurately model the content of the poster layout.

Description

Technical Field

[0001] This application belongs to the field of artificial intelligence and computer vision technology, specifically relating to a poster layout generation model training method, apparatus, device and readable storage medium. Background Technology

[0002] In the field of poster layout generation, various solutions based on deep learning models such as Generative Adversarial Networks (GANs) and Variational Autoencoders (VAEs) have been proposed. These methods can achieve automatic poster layout generation to a certain extent by modeling the visual content of the poster. For example, LayoutGAN uses a generative adversarial network to generate layouts with fixed scene frames, and it can generate layouts with geometric labels. Similarly, LayoutTransformer uses a self-attention mechanism to learn the contextual relationships between layout elements, thereby generating new layouts.

[0003] Existing poster layout generation methods mostly focus on modeling visual content, resulting in poster layouts that are not accurate enough in terms of content expression and fail to meet the actual needs of users. Summary of the Invention

[0004] This application provides a poster layout generation model training method, apparatus, device, and readable storage medium to solve the problem of poor accuracy of poster layout content generated by existing poster layout generation methods.

[0005] In a first aspect, embodiments of this application provide a poster layout generation model training method, including: Obtain a training set, which includes multiple poster images of layouts to be generated and descriptive text corresponding to each poster image. The descriptive text is used to describe the requirements of the mind chain-style poster layout design. Poster images and corresponding descriptive texts are selected from the training set. The poster images are converted into first image vectors, and the descriptive texts are converted into text vectors. The first image vector and the text vectors have the same dimension. The first image vector and the text vector are input into the visual language model, and after fusion processing, a thought chain-like output result is obtained. Based on the output of the thought chain, a first loss function is determined; and based on the first loss function, the model parameters in the visual language model are updated to obtain a poster layout generation model, which is used for poster layout generation.

[0006] In some embodiments, converting the poster image into a first image vector includes: The poster image is visually encoded to obtain a second image vector; The second image vector is sequentially processed through ReLU activation, a dropout layer, normalization, and linear transformation to obtain the first image vector, wherein the dimension of the first image vector is lower than that of the second image vector.

[0007] In some embodiments, the step of inputting the first image vector and the text vector into a visual language model, and obtaining a thought chain-like output result after fusion processing, includes: The first image vector and the text vector are input into the visual language model. A linear transformation is performed on the first image vector to obtain the query matrix of the poster image. A linear transformation is also performed on the text vector to obtain the key matrix and value matrix of the descriptive text. Based on the query matrix of the poster image and the key matrix of the descriptive text, normalization processing is performed to obtain the attention weights; Based on the attention weights, the value matrix of the descriptive text is weighted and fused to obtain a thought chain-like output result.

[0008] In some embodiments, determining the first loss function based on the output of the thought chain includes: Based on the output of the thought chain, the overlap between the ground truth bounding box and the predicted bounding box is calculated. The output of the thought chain includes the position, size, category, and description of the predicted bounding box, and the ground truth bounding box is included in the validation set. Obtain the values ​​of the real labels, and determine the cross-entropy loss based on the values ​​of the real labels and the model prediction probability, wherein the model prediction probability is calculated by the forward propagation of the visual language model; The first loss function is determined based on the overlap and the cross-entropy loss.

[0009] In some embodiments, obtaining the value of the real label includes: If the overlapping area between the real bounding box and the predicted bounding box is less than a preset threshold, the predicted bounding box is determined to be invalid, and the real label is set to 0. If the overlapping area between the real bounding box and the predicted bounding box is greater than or equal to the preset threshold, the predicted bounding box is determined to be valid, and the real label is set to 1.

[0010] In some embodiments, updating the model parameters in the visual language model based on the first loss function includes: After fixing the original weight matrix of the visual language model, the matrix parameters of the first low-rank matrix and the second low-rank matrix of the fully connected layer in the visual language model are updated based on the first loss function. The increment matrix obtained by matrix multiplication of the first low-rank matrix and the second low-rank matrix is ​​used to update the original weight matrix.

[0011] Secondly, embodiments of this application also provide a poster layout generation model training device, comprising: The acquisition module is used to acquire a training set, which includes multiple poster images of layouts to be generated and descriptive text corresponding to each poster image. The descriptive text is used to describe the design requirements of the mind chain-style poster layout. The first processing module is configured to select poster images and corresponding descriptive text from the training set, convert the poster images into first image vectors, and convert the descriptive text into text vectors; wherein the first image vector and the text vector have the same dimension. The second processing module is used to input the first image vector and the text vector into the visual language model, and after fusion processing, obtain the thought chain-like output result. The third processing module is used to determine a first loss function based on the output of the thought chain; and update the model parameters in the visual language model based on the first loss function to obtain a poster layout generation model, which is used for poster layout generation.

[0012] Thirdly, this application also provides a poster layout generation model training device, including a processor and a transceiver, wherein the transceiver receives and transmits data under the control of the processor, and the processor is used to perform the following operations: Obtain a training set, which includes multiple poster images of layouts to be generated and descriptive text corresponding to each poster image. The descriptive text is used to describe the requirements of the mind chain-style poster layout design. Poster images and corresponding descriptive texts are selected from the training set. The poster images are converted into first image vectors, and the descriptive texts are converted into text vectors. The first image vector and the text vectors have the same dimension. The first image vector and the text vector are input into the visual language model, and after fusion processing, a thought chain-like output result is obtained. Based on the output of the thought chain, a first loss function is determined; and based on the first loss function, the model parameters in the visual language model are updated to obtain a poster layout generation model, which is used for poster layout generation.

[0013] Fourthly, embodiments of this application also provide a poster layout generation model training device, including a memory, a processor, and a computer program stored in the memory and running on it. When the processor executes the program, it implements the poster layout generation model training method as described in the first aspect above.

[0014] Fifthly, embodiments of this application also provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in the poster layout generation model training method described in the first aspect above.

[0015] In a sixth aspect, embodiments of this application also provide a computer program product, including computer instructions, which, when executed by a processor, implement the steps in the poster layout generation model training method described in the first aspect above.

[0016] The above-described technical solution of the present invention has at least the following beneficial effects: In this embodiment of the invention, a training set is obtained, which includes multiple poster images of layouts to be generated and descriptive text corresponding to each poster image. The descriptive text describes the requirements of the thought chain-style poster layout design. Poster images and their corresponding descriptive texts are selected from the training set. The poster images are converted into first image vectors, and the descriptive texts are converted into text vectors. The first image vectors and the text vectors have the same dimension. The first image vectors and the text vectors are input into a visual language model. After fusion processing, a thought chain-style output result is obtained. A first loss function is determined based on the thought chain-style output result. Based on the first loss function, the model parameters in the visual language model are updated to obtain a poster layout generation model. This poster layout generation model is used for poster layout generation. In this way, by introducing a chain-style thinking enhancement strategy, multimodal information such as text and images can be organically combined to achieve accurate modeling of the poster layout content. Consequently, the poster layout content generated by the subsequent application model has high accuracy and can improve the semantic consistency and visual aesthetics of the generated layout. Attached Figure Description

[0017] Figure 1 This is a flowchart illustrating the poster layout generation model training method according to an embodiment of this application; Figure 2 This is an overall implementation block diagram of the poster layout generation model training method according to an embodiment of this application; Figure 3 This is a schematic diagram of the modules of the poster layout generation model training device according to an embodiment of this application; Figure 4 This is a schematic diagram of the hardware structure of the poster layout generation model training device according to an embodiment of this application. Detailed Implementation

[0018] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0019] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and are not used to describe a specified order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first" and "second" are generally of the same class, not limited in number; for example, a first object can be one or more. Furthermore, in the specification and claims, "and" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0020] While existing GAN and VAE models can generate layouts, they often fall short when handling complex and diverse layout requirements. These models typically rely on a large number of predefined tags and frames, making it difficult to flexibly adapt to new scenarios or specific user needs. Furthermore, when generating layouts, these models can usually only handle simple visual content descriptions and cannot generate the text and background descriptions that should be filled in the corresponding boxes, resulting in layouts that are lacking in visual aesthetics and content expression.

[0021] While existing research on multimodal large language models has achieved some success in tasks such as document understanding and visual question answering, its application in poster layout generation is still in the early stages of exploration. These models typically rely on planar text input during pre-training and supervised fine-tuning (SFT), lacking sufficient learning and modeling of document layout details. Furthermore, existing datasets often only provide basic frame and category information, lacking detailed descriptions of image content, which limits the model's performance when generating complex layouts.

[0022] In summary, existing technologies face challenges in automatically generating poster layouts, such as imperfect model structures and insufficient dataset information, making it difficult to meet the requirements for layout accuracy and generation efficiency in practical applications.

[0023] To address the aforementioned technical problems, embodiments of this application provide a poster layout generation model training method, apparatus, device, and readable storage medium. The method and apparatus are based on the same concept, and since the principles underlying the problems solved are similar, their implementations can be mutually referenced; repeated details will not be elaborated further.

[0024] like Figure 1 The diagram shown is a flowchart illustrating the poster layout generation model training method according to an embodiment of this application. The method may include: Step 101: Obtain the training set, which includes multiple poster images of layouts to be generated and descriptive text corresponding to each poster image. The descriptive text is used to describe the requirements of the mind chain-style poster layout design. Here, the descriptive text corresponding to the poster image typically refers to the prompt word input by the model. This descriptive text describes the requirements for a chain-of-thought poster layout design. A chain of thought (COT) refers to a cognitive process or a series of thought steps through which an individual or intelligent system solves problems or makes decisions.

[0025] In one example, the prompt word for the model input could be: You are now an assistant responsible for designing a poster layout. The poster will contain three types of boxes: text boxes, logo boxes, and background image boxes. Please complete the layout design by following these four steps.

[0026] 1. Image Description: Provide a detailed description of the poster image, including its visual elements and any text or graphics.

[0027] 2. Define categories and corresponding quantities: Based on the content and structure of the poster, identify and specify the number of boxes (text boxes, logo boxes, background image boxes) for each category.

[0028] 3. Describe each box and its corresponding category: Provide a description for each box, including its category (text, logo, background image) and the content it should contain.

[0029] 4. Define the position, width, height, category, and description of each box, and provide the final design: Specify the coordinates (x, y), width (w), height (h), category, and description for each box. Summarize these results into the final layout design format.

[0030] The prompt (i.e., the descriptive text corresponding to the poster image) inputted into the model describes the poster layout design requirements through a series of thought steps (the four steps mentioned above), which can guide the model to output the poster layout result according to the design requirements.

[0031] It's important to note that the training set includes a large number of poster images for the layouts to be generated and corresponding descriptive text for each poster image. In other words, the training data includes multimodal data such as images and text. Through model training and deep learning on this multimodal data, the flexibility and adaptability of layout generation can be improved. In the subsequent model application stage, it can dynamically generate creative and aesthetically valuable poster layouts based on different user needs and scenarios (i.e., users can select poster images and input their desired poster layout design requirements, which means users can input the descriptive text corresponding to the poster images, i.e., the prompt input by the model).

[0032] Step 102: Select poster images and corresponding descriptive text from the training set; convert the poster images into first image vectors and the descriptive text into text vectors; wherein the first image vector and the text vector have the same dimension. Here, the poster image is converted into a first image vector, and the descriptive text is converted into a text vector. This is to transform unstructured data into a numerical representation that the model can compute, while preserving key semantic features, thereby achieving efficient and accurate machine learning modeling and supporting end-to-end training.

[0033] Step 103: Input the first image vector and the text vector into the visual language model, and after fusion processing, obtain the thought chain-like output result; Here, the first image vector and the text vector have the same dimension. The purpose is to map features from different modalities into a unified semantic space, achieve effective matching and fusion of cross-modal information, and thus improve the model's ability to understand complex scenes.

[0034] Here, during each training iteration, a batch of poster images and their corresponding descriptive texts are selected from the training set. For each poster image and its corresponding descriptive text, the poster image is converted into a first image vector, and the descriptive text is converted into a text vector. Then, the first image vectors obtained from each poster image are concatenated, and the text vectors obtained from each descriptive text are concatenated. Finally, the concatenated image vectors and text vectors are input into the visual language model. After fusion processing, a chain-like output is obtained. This chain-like output includes the position, size, category, and description of the predicted bounding box for each poster image in the batch.

[0035] It should be noted that Vision-Language Models (VLMs) are a type of artificial intelligence model that deeply integrates visual and linguistic information. Their core value lies in breaking down the information barriers of a single modality, accurately understanding the semantic relationships between images and text, and thus efficiently completing cross-modal tasks such as image description generation, image-text matching verification, visual question answering (VQA), image content editing, and long video understanding. They have become a key technology carrier connecting virtual information and real-world scenarios.

[0036] Optionally, the visual language model of this application is Qwen2-VL, which adopts a cascaded architecture of "ViT visual encoder + Qwen2 language decoder", and uses the Naive Dynamic Resolution technology to realize dynamic token conversion of images of arbitrary resolution (requiring only 4 tokens at the minimum). It also strengthens the positional association and semantic alignment between modalities through Multimodal Rotation Position Encoding (M-ROPE) to maintain stable visual feature extraction capabilities.

[0037] Since the descriptive text of the poster image is used to describe the mind chain-style poster layout design requirements, guiding the model to output the poster layout result according to the design requirements, the output result of the visual language model is a mind chain-style output result.

[0038] It should be noted that in the field of artificial intelligence, especially in machine learning and natural language processing, COT (Cognitive Chain of Thought) represents the step-by-step reasoning or logical process employed by a model to reach a conclusion. The COT method is particularly suitable for tasks requiring multi-step reasoning. By breaking down complex problems into simpler components, the performance and interpretability of the model are significantly improved. In this application, the visual language model generates the final poster layout format result through four distinct steps. The visual language model will output the final result through four steps, guided by a COT approach. An example illustrating the COT-style output is provided below: Step 1: Describe the image This image showcases a marketing advertisement for a men's shampoo. A blue bottle of the anti-dandruff shampoo stands out against a dark background, surrounded by splashes of green and yellow liquid, symbolizing the product's refreshing and cleansing properties. The text "2X Anti-Dandruff Power 3X Oil Control Power" highlights the shampoo's enhanced dandruff and oil-control effects. The overall design conveys a sense of freshness and cleanliness.

[0039] Step 2: Define the boxes corresponding to each category The layout includes: 1 sign box 2 text boxes 2 base map frames Step 3: Provide a description for each box Logo frame: Contains the text "xx Men's Shampoo".

[0040] Text box: Contains the text "2X anti-dandruff power" and "3X oil control power".

[0041] Background frame: Both background frames are represented as "underlined, bright, green".

[0042] Step 4: Define the position, width, height, category, and description of each box. The final layout design is shown below: <cate> logo< / cate> <box> (0.0332,0.0238),(0.5498,0.0857)< / box> Description: {xxMen's Shampoo} <cate> text< / cate> <box> (0.13837,0.10357),(0.7011,0.1869)< / box> Description: {2X dandruff removal power} <cate> text< / cate> <box> (0.33579,0.1988),(0.8948,0.28095)< / box> Description: {3X Oil Control Power} <cate> Base map< / cate> <box> (0.0811,0.10238),(0.6974,0.20119)< / box> Description: {underlined, bright, green} <cate> Base map< / cate> <box> (0.33579,0.196428),(0.9428,0.30119)< / box> Description: {underlined, bright, green} Step 104: Determine the first loss function based on the output of the thought chain; and update the model parameters in the visual language model based on the first loss function to obtain the poster layout generation model, which is used for poster layout generation.

[0043] It should be noted that after each iteration of training, the model parameters in the visual language model are updated based on the first loss function until the iteration termination condition is met, thus obtaining the poster layout generation model. Here, the poster layout generation model is obtained by training the visual language model.

[0044] The iteration termination condition can be either the decrease of the first loss function in adjacent iterations being lower than a set threshold, or the training count reaching the maximum number of iterations.

[0045] The poster layout generation model training method of this application embodiment obtains a training set, which includes multiple poster images of layouts to be generated and descriptive text corresponding to each poster image. The descriptive text is used to describe the poster layout design requirements of a thought chain. Poster images and their corresponding descriptive texts are selected from the training set. The poster images are converted into first image vectors, and the descriptive texts are converted into text vectors. The first image vectors and the text vectors have the same dimension. The first image vectors and the text vectors are input into a visual language model, and after fusion processing, a thought chain output result is obtained. A first loss function is determined based on the thought chain output result. Based on the first loss function, the model parameters in the visual language model are updated to obtain a poster layout generation model. This poster layout generation model is used for poster layout generation. Thus, by introducing a chain-like thinking enhancement strategy, multimodal information such as text and images can be organically combined to achieve accurate modeling of the poster layout content. Consequently, the poster layout content generated by the subsequent application model has high accuracy and can improve the semantic consistency and visual aesthetics of the generated layout.

[0046] As an optional implementation, step 102 above, converting the poster image into a first image vector, includes: The poster image is visually encoded to obtain a second image vector; Specifically, a visual encoder can be used to visually encode the poster image to obtain a second image vector. It should be noted that the second image vector is a high-dimensional image vector V (with dimensions D). V ×M). Here, D V The visual feature dimension represents the number of dimensions of the feature vector obtained after encoding each image patch; M represents the number of image patches, i.e., the number of small patches into which the poster image is divided. The poster image is first visually encoded to obtain a high-dimensional image vector in order to extract key features of the image.

[0047] The second image vector is sequentially processed through ReLU activation, a dropout layer, normalization, and linear transformation to obtain the first image vector, wherein the dimension of the first image vector is lower than that of the second image vector.

[0048] Specifically, the high-dimensional image vector (i.e., the second image vector) V can be processed through ReLU activation, Dropout layer, and Normal normalization, and then processed through dimension d×d v The weight matrix W p A linear transformation maps the image to a vector with the same feature dimensions as the visual language model, i.e., the first image vector. (Dimension d), achieving dimension alignment between image vectors and text vectors. Here, the linear mapping of the neural network projection layer can be represented as: (1) Among them, W p This represents the weight matrix of the projection layer of the neural network, with dimensions d×d. v , This represents the mapped features, i.e., the first image vector, with dimension d.

[0049] Here, the second image vector can be linearly transformed using a neural network projection layer. This process yields the first image vector. In other words, to align the dimensions of the image vector with the text vector, the second image vector undergoes a linear transformation using a neural network projection layer, mapping the high-dimensional image vector to a vector with the same feature dimensions as the visual language model, i.e., the first image vector.

[0050] In some embodiments, converting the descriptive text into a text vector specifically includes: using a Python library to convert the descriptive text into a text vector.

[0051] As an optional implementation, in step 103 above, the first image vector and the text vector are input into the visual language model, and after fusion processing, a thought chain-like output result is obtained, including: The first image vector and the text vector are input into the visual language model. A linear transformation is performed on the first image vector to obtain the query matrix of the poster image. A linear transformation is also performed on the text vector to obtain the key matrix and value matrix of the descriptive text. Here, within the visual language model, the dual-modal information of the first image vector and text vector is fused through a cross-attention mechanism.

[0052] Specifically, the first image vector Where H is the height of the poster image, W is the width of the poster image, and C is the number of channels. Text vector , where L is the length of the text description and D is the embedding dimension of each word.

[0053] The first image vector I is transformed into the query matrix Q of the poster image through three different linear transformations. I Key matrix K I Sum matrix V I The query matrix Q describing the text is obtained by performing three different linear transformations on the text vector T. T Key matrix K T Sum matrix V T .

[0054] Based on the query matrix of the poster image and the key matrix of the descriptive text, normalization processing is performed to obtain the attention weights; Specifically, based on the query matrix Q of the poster image I and the key matrix K describing the text T The attention weight A is obtained by normalizing the data using the following formula. V .

[0055] (2) in, K represents the key matrix for describing the text. T The transpose of the poster image, the query matrix Q I With the key matrix K describing the text T Obtained through matrix multiplication Each element in this matrix represents the similarity score between a certain location in the image and a certain word in the text. This represents the dimension of the key matrix describing the text (and also the dimension of the query matrix for the poster image). It is a scaling factor, mainly used to maintain numerical stability and ensure that the input of softmax falls within the region of reasonable gradient.

[0056] Here, attention weight A V Used to indicate the degree of attention each image location gives to each text word.

[0057] Based on the attention weights, the value matrix of the descriptive text is weighted and fused to obtain a thought chain-like output result.

[0058] This step can be expressed mathematically as follows: (3) Here, the attention weight A V With the value matrix V describing the text T Multiplication (i.e. weighted fusion) yields the output of each image location fused with text information. The meaning of the above formula (3) is that each part of the image retrieves the most relevant information from the text, and then the retrieved text information is fused into the image, so that the model can understand cross-modal semantic relationships.

[0059] In some embodiments, step 104 above, determining the first loss function based on the output of the thought chain, includes: Based on the output of the thought chain, the overlap between the ground truth bounding box and the predicted bounding box is calculated. The output of the thought chain includes the position, size, category, and description of the predicted bounding box, and the ground truth bounding box is included in the validation set. Here, the output includes predicted bounding boxes for each predicted category, such as predicted label boxes, text boxes, and background image boxes. For each predicted category, the Intersection over Union (IoU) between the ground truth bounding box and the predicted bounding box for that category is calculated. Where, intersection is the area of ​​the intersection between the predicted bounding box and the ground truth bounding box (i.e., the overlap area), and union is the area of ​​the union between the predicted bounding box and the ground truth bounding box. (4) (5) in, It predicts the value (0 or 1) of pixel (i,j) in the mask image. It is the value (0 or 1) of pixel (i,j) in the actual mask image; Obtain the values ​​of the real labels, and determine the cross-entropy loss based on the values ​​of the real labels and the model prediction probability, wherein the model prediction probability is calculated by the forward propagation of the visual language model; As an optional implementation, the values ​​of the actual tags are obtained, including: If the overlapping area between the real bounding box and the predicted bounding box is less than a preset threshold, the predicted bounding box is determined to be invalid, and the real label is set to 0. If the overlapping area between the real bounding box and the predicted bounding box is greater than or equal to the preset threshold, the predicted bounding box is determined to be valid, and the real label is set to 1.

[0060] It should be noted that the overlap area between the ground truth bounding box and the predicted bounding box... ,in, This represents the right boundary of the intersection of the ground truth bounding box and the predicted bounding box. This represents the left boundary of the intersection of the ground truth bounding box and the predicted bounding box. This represents the lower boundary of the intersection of the ground truth bounding box and the predicted bounding box. This represents the upper boundary of the intersection of the ground truth bounding box and the predicted bounding box. If or If the ground truth bounding box and the predicted bounding box do not overlap, the overlap area between the ground truth bounding box and the predicted bounding box is 0.

[0061] If the overlap area between the ground truth bounding box and the predicted bounding box is less than the preset threshold (e.g., 384.75), the predicted bounding box is determined to be invalid, and the value of the ground truth label (y) is 0.

[0062] If the overlap area between the ground truth bounding box and the predicted bounding box is greater than or equal to the preset threshold, the predicted bounding box is determined to be valid, and the value of the ground truth label (y) is 1.

[0063] Here, based on the true label value y and the model predicted probability p, the mathematical representation of cross entropy loss can be determined as follows: (6) Among them, the values ​​of the real labels The model prediction probability p is calculated from the model's forward propagation.

[0064] The first loss function is determined based on the overlap and the cross-entropy loss. Here, the mathematical expression of the first loss function, Loss, is as follows: (7) Wherein, λ is an adjustable parameter. Optionally, λ is 0.2.

[0065] Here, a counter `empty_elem` is used to count the area overlap between the predicted bounding box and the ground truth bounding box. If the overlap area is less than a preset threshold, the predicted bounding box is invalid, and the counter value is incremented by 1. Specifically, `empty_elem += 1 if area < threshold`. The smaller the `empty_elem` metric, the more accurate the model's prediction.

[0066] In some embodiments, step 104 above, updating the model parameters in the visual language model based on the first loss function, includes: After fixing the original weight matrix of the visual language model, the matrix parameters of the first low-rank matrix and the second low-rank matrix of the fully connected layer in the visual language model are updated based on the first loss function. The increment matrix obtained by matrix multiplication of the first low-rank matrix and the second low-rank matrix is ​​used to update the original weight matrix.

[0067] Low-Rank Adaptation (LoRA) is an effective model fine-tuning technique, primarily used for fine-tuning large neural networks in resource-constrained environments. This embodiment employs LoRA for model fine-tuning. By introducing the concept of low-rank matrices, while keeping the original model parameters (the matrix parameters of the original weight matrix) frozen, the adaptive enhancement of the model is achieved by adjusting a small number of parameters (the matrix parameters of the first and second low-rank matrices). This method can significantly reduce the number of training parameters, thereby reducing the demand for computational resources, while retaining good performance on the fine-tuned model.

[0068] In this embodiment, the first low-rank matrix A is used to reduce the dimensionality of the input features from the original high-dimensional space (d-dimensional) to the low-rank latent space (r-dimensional); the second low-rank matrix B is used to remap the information in the low-rank latent space back to the original high-dimensional space (d-dimensional).

[0069] In the fully connected layer of the visual language model, a first low-rank matrix A (d×r) and a second low-rank matrix B (r×d) are introduced. During the backpropagation phase, the original weight matrix of the visual language model is fixed, and the parameters of the two matrices are updated based on the first loss function. During the forward propagation, the increment matrix ΔW = A×B is calculated and superimposed on the original weight matrix W0 to obtain the updated weight matrix W = W0 + ΔW.

[0070] As an optional implementation, the method of this application also includes: Get the first poster image input by the user and the corresponding descriptive text of the first poster image; The first poster image and its corresponding descriptive text are input into the poster layout generation model to generate the poster layout content.

[0071] In this implementation, the poster layout generated by the above-mentioned poster layout generation model has high accuracy and can improve the semantic consistency and visual aesthetics of the generated layout.

[0072] See below. Figure 2 The overall implementation process of the method in this application is illustrated by an example.

[0073] First, the user inputs a descriptive text (prompt) about the desired poster layout effect and a poster background image. In the preprocessing module, the poster background image is vectorized using a neural network projection layer. The descriptive text is vectorized using a Python library.

[0074] Subsequently, within the Qwen2-VL visual language model, after fusing bimodal information (the image vector of the poster background and the text vector of the descriptive text) through a cross-attention mechanism, the model outputs information step by step in the form of a thought chain: first, describe the image elements, then define the category and number of boxes, then supplement the description of the box content, and finally clarify the coordinates and width and height of the boxes to ensure the rationality of the poster layout generation.

[0075] Subsequently, the behavior learning module performs feedback learning: In positive feedback, if the IoU overlap meets the standard, the Val algorithm determines that the overlapping area is ≥384.75 (label=1), and the cross entropy loss and the first loss function Loss meet the requirements, the judgment result is valid and included in the training set to strengthen the model; In negative feedback, if the above indicators do not meet the standard (e.g., overlapping area <384.75, label=0), the labeling result is invalid, and the loss signal is transmitted through backpropagation to guide the model adjustment.

[0076] The model training adopts the LoRA approach: the original weight matrix W0 of Qwen2-VL is frozen, and low-rank matrices A (d×r) and B (r×d) are introduced only in the fully connected layer. The parameters of these two matrices are optimized during training, and finally the incremental matrix ΔW=A×B is superimposed on W0. This reduces the consumption of computational resources while retaining the basic capabilities of the model and supports rapid iteration.

[0077] like Figure 3 As shown in the embodiments of this application, a poster layout generation model training device is also provided, which may include: The acquisition module 301 is used to acquire a training set, which includes multiple poster images of layouts to be generated and descriptive text corresponding to each poster image. The descriptive text is used to describe the design requirements of the mind chain-style poster layout. The first processing module 302 selects a poster image and the corresponding descriptive text from the training set, converts the poster image into a first image vector, and converts the descriptive text into a text vector; wherein the first image vector and the text vector have the same dimension. The second processing module 303 is used to input the first image vector and the text vector into the visual language model, and after fusion processing, obtain a thought chain-like output result. The third processing module 304 is used to determine a first loss function based on the output of the thought chain; and update the model parameters in the visual language model based on the first loss function to obtain a poster layout generation model, which is used for poster layout generation.

[0078] In some embodiments, the first processing module 302 includes: The first processing unit is used to perform visual encoding on the poster image to obtain a second image vector; The second processing unit is used to sequentially process the second image vector through ReLU activation, a dropout layer, normalization, and linear transformation to obtain the first image vector, wherein the dimension of the first image vector is lower than the dimension of the second image vector.

[0079] In some embodiments, the second processing module 303 includes: The third processing unit is used to input the first image vector and the text vector into the visual language model, perform a linear transformation on the first image vector to obtain the query matrix of the poster image, and perform a linear transformation on the text vector to obtain the key matrix and value matrix of the descriptive text. The fourth processing unit is used to perform normalization processing based on the query matrix of the poster image and the key matrix of the descriptive text to obtain attention weights; The fifth processing unit is used to perform weighted fusion of the value matrix of the descriptive text according to the attention weights to obtain a thought chain-like output result.

[0080] In some embodiments, the third processing module 304 includes: The sixth processing unit is used to calculate the overlap between the ground truth boxes and the predicted boxes based on the output of the thought chain. The output of the thought chain includes the position, size, category, and description of the predicted boxes, and the ground truth boxes are included in the validation set. The seventh processing unit is used to obtain the value of the real label and determine the cross-entropy loss based on the value of the real label and the model prediction probability, wherein the model prediction probability is calculated by the forward propagation of the visual language model. The eighth processing unit is used to determine the first loss function based on the overlap and the cross-entropy loss.

[0081] In some embodiments, the seventh processing unit is specifically used for: If the overlapping area between the real bounding box and the predicted bounding box is less than a preset threshold, the predicted bounding box is determined to be invalid, and the real label is set to 0. If the overlapping area between the real bounding box and the predicted bounding box is greater than or equal to the preset threshold, the predicted bounding box is determined to be valid, and the real label is set to 1.

[0082] In some embodiments, the third processing module 304 includes: The ninth processing unit is used to fix the original weight matrix of the visual language model, and then update the matrix parameters of the first low-rank matrix and the second low-rank matrix of the fully connected layer in the visual language model based on the first loss function. The increment matrix obtained by matrix multiplication of the first low-rank matrix and the second low-rank matrix is ​​used to update the original weight matrix.

[0083] The poster layout generation model training device of this application embodiment acquires a training set, which includes multiple poster images of layouts to be generated and descriptive text corresponding to each poster image. The descriptive text describes the poster layout design requirements of a thought chain. It selects poster images and their corresponding descriptive texts from the training set, converts the poster images into first image vectors, and converts the descriptive texts into text vectors. The first image vectors and the text vectors have the same dimension. The first image vectors and the text vectors are input into a visual language model, and after fusion processing, a thought chain output result is obtained. A first loss function is determined based on the thought chain output result. Based on the first loss function, the model parameters in the visual language model are updated to obtain a poster layout generation model. This poster layout generation model is used for poster layout generation. Thus, by introducing a chain-thinking enhancement strategy, multimodal information such as text and images can be organically combined to achieve accurate modeling of the poster layout content. Consequently, the poster layout content generated by the subsequent application model has high accuracy and can improve the semantic consistency and visual aesthetics of the generated layout.

[0084] like Figure 4 As shown in the illustration, this application embodiment also provides a poster layout generation model training device, including a processor 400 and a transceiver 410. The transceiver 410 receives and transmits data under the control of the processor 400, and the processor 400 is used to perform the following operations: Obtain a training set, which includes multiple poster images of layouts to be generated and descriptive text corresponding to each poster image. The descriptive text is used to describe the requirements of the mind chain-style poster layout design. Poster images and corresponding descriptive texts are selected from the training set. The poster images are converted into first image vectors, and the descriptive texts are converted into text vectors. The first image vector and the text vectors have the same dimension. The first image vector and the text vector are input into the visual language model, and after fusion processing, a thought chain-like output result is obtained. Based on the output of the thought chain, a first loss function is determined; and based on the first loss function, the model parameters in the visual language model are updated to obtain the poster layout generation model, which is used for poster layout generation.

[0085] It should be understood that the bus interface between the processor 400 and the transceiver 410 in the diagram is a high-speed, reliable data and control channel established between the processor 400 and the transceiver 410. The bus interface can be used for: transmitting information to be sent and received; transmitting processor instructions and reporting the transceiver's status; and providing a unified clock standard to ensure communication synchronization.

[0086] In some embodiments, the processor 400 is further configured to: The poster image is visually encoded to obtain a second image vector; The second image vector is sequentially processed through ReLU activation, a dropout layer, normalization, and linear transformation to obtain the first image vector, wherein the dimension of the first image vector is lower than that of the second image vector.

[0087] In some embodiments, the processor 400 is further configured to: The first image vector and the text vector are input into the visual language model. A linear transformation is performed on the first image vector to obtain the query matrix of the poster image. A linear transformation is also performed on the text vector to obtain the key matrix and value matrix of the descriptive text. Based on the query matrix of the poster image and the key matrix of the descriptive text, normalization processing is performed to obtain the attention weights; Based on the attention weights, the value matrix of the descriptive text is weighted and fused to obtain a thought chain-like output result.

[0088] In some embodiments, the processor 400 is further configured to: Based on the output of the thought chain, the overlap between the ground truth bounding box and the predicted bounding box is calculated. The output of the thought chain includes the position, size, category, and description of the predicted bounding box, and the ground truth bounding box is included in the validation set. Obtain the values ​​of the real labels, and determine the cross-entropy loss based on the values ​​of the real labels and the model prediction probability, wherein the model prediction probability is calculated by the forward propagation of the visual language model; The first loss function is determined based on the overlap and the cross-entropy loss.

[0089] In some embodiments, the processor 400 is further configured to: If the overlapping area between the real bounding box and the predicted bounding box is less than a preset threshold, the predicted bounding box is determined to be invalid, and the real label is set to 0. If the overlapping area between the real bounding box and the predicted bounding box is greater than or equal to the preset threshold, the predicted bounding box is determined to be valid, and the real label is set to 1.

[0090] In some embodiments, the processor 400 is further configured to: After fixing the original weight matrix of the visual language model, the matrix parameters of the first low-rank matrix and the second low-rank matrix of the fully connected layer in the visual language model are updated based on the first loss function. The increment matrix obtained by matrix multiplication of the first low-rank matrix and the second low-rank matrix is ​​used to update the original weight matrix.

[0091] The poster layout generation model training device of this application embodiment acquires a training set, which includes multiple poster images of layouts to be generated and descriptive text corresponding to each poster image. The descriptive text is used to describe the poster layout design requirements of a thought chain. Poster images and their corresponding descriptive texts are selected from the training set. The poster images are converted into first image vectors, and the descriptive texts are converted into text vectors. The first image vectors and the text vectors have the same dimension. The first image vectors and the text vectors are input into a visual language model, and after fusion processing, a thought chain output result is obtained. A first loss function is determined based on the thought chain output result. Based on the first loss function, the model parameters in the visual language model are updated to obtain a poster layout generation model. This poster layout generation model is used for poster layout generation. Thus, by introducing a chain-like thinking enhancement strategy, multimodal information such as text and images can be organically combined to achieve accurate modeling of the poster layout content. Consequently, the poster layout content generated by the subsequent application model has high accuracy and can improve the semantic consistency and visual aesthetics of the generated layout.

[0092] This invention also provides a poster layout generation model training device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it performs the functions described above. Figure 1 The various processes in the poster layout generation model training method embodiment shown are all capable of achieving the same technical effect, and will not be repeated here to avoid repetition.

[0093] This invention also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the above-described functionality. Figure 1 The various processes in the poster layout generation model training method embodiment shown herein achieve the same technical effect, and will not be described again here to avoid repetition. The computer-readable storage medium mentioned herein includes, for example, read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0094] This invention also provides a computer program product, including computer instructions, which, when executed by a processor, implement the above-described functionality. Figure 1 The steps in the poster layout generation model training method are shown.

[0095] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-readable storage media (including, but not limited to, disk storage and optical storage) containing computer-usable program code.

[0096] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 A device for one or more processes and / or the functions specified in one or more boxes.

[0097] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce a paper article including an instruction means, the instruction means being implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0098] These computer program instructions can also be loaded onto a computer or other programmable data processing equipment, causing the computer or other programmable equipment to perform a series of operational steps to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0099] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A method for training a poster layout generation model, characterized in that, include: Obtain a training set, which includes multiple poster images of layouts to be generated and descriptive text corresponding to each poster image. The descriptive text is used to describe the requirements of the mind chain-style poster layout design. Poster images and corresponding descriptive texts are selected from the training set. The poster images are converted into first image vectors, and the descriptive texts are converted into text vectors. The first image vector and the text vectors have the same dimension. The first image vector and the text vector are input into the visual language model, and after fusion processing, a thought chain-like output result is obtained. Based on the output of the thought chain, a first loss function is determined; and based on the first loss function, the model parameters in the visual language model are updated to obtain a poster layout generation model, which is used for poster layout generation.

2. The method according to claim 1, characterized in that, The step of converting the poster image into a first image vector includes: The poster image is visually encoded to obtain a second image vector; The second image vector is sequentially processed through ReLU activation, a dropout layer, normalization, and linear transformation to obtain the first image vector, wherein the dimension of the first image vector is lower than that of the second image vector.

3. The method according to claim 1, characterized in that, The process of inputting the first image vector and the text vector into the visual language model, and then fusing them to obtain a thought chain-like output result, includes: The first image vector and the text vector are input into the visual language model. A linear transformation is performed on the first image vector to obtain the query matrix of the poster image. A linear transformation is also performed on the text vector to obtain the key matrix and value matrix of the descriptive text. Based on the query matrix of the poster image and the key matrix of the descriptive text, normalization processing is performed to obtain the attention weights; Based on the attention weights, the value matrix of the descriptive text is weighted and fused to obtain a thought chain-like output result.

4. The method according to claim 1, characterized in that, The step of determining the first loss function based on the output of the thought chain includes: Based on the output of the thought chain, the overlap between the ground truth bounding box and the predicted bounding box is calculated. The output of the thought chain includes the position, size, category, and description of the predicted bounding box, and the ground truth bounding box is included in the validation set. Obtain the values ​​of the real labels, and determine the cross-entropy loss based on the values ​​of the real labels and the model prediction probability, wherein the model prediction probability is calculated by the forward propagation of the visual language model; The first loss function is determined based on the overlap and the cross-entropy loss.

5. The method according to claim 4, characterized in that, The process of obtaining the value of the real label includes: If the overlapping area between the real bounding box and the predicted bounding box is less than a preset threshold, the predicted bounding box is determined to be invalid, and the real label is set to 0. If the overlapping area between the real bounding box and the predicted bounding box is greater than or equal to the preset threshold, the predicted bounding box is determined to be valid, and the real label is set to 1.

6. The method according to claim 1, characterized in that, The step of updating the model parameters in the visual language model based on the first loss function includes: After fixing the original weight matrix of the visual language model, the matrix parameters of the first low-rank matrix and the second low-rank matrix of the fully connected layer in the visual language model are updated based on the first loss function. The increment matrix obtained by matrix multiplication of the first low-rank matrix and the second low-rank matrix is ​​used to update the original weight matrix.

7. A poster layout generation model training device, characterized in that, include: The acquisition module is used to acquire a training set, which includes multiple poster images of layouts to be generated and descriptive text corresponding to each poster image. The descriptive text is used to describe the design requirements of the mind chain-style poster layout. The first processing module is configured to select poster images and corresponding descriptive text from the training set, convert the poster images into first image vectors, and convert the descriptive text into text vectors; wherein the first image vector and the text vector have the same dimension. The second processing module is used to input the first image vector and the text vector into the visual language model, and after fusion processing, obtain the thought chain-like output result. The third processing module is used to determine a first loss function based on the output of the thought chain; and update the model parameters in the visual language model based on the first loss function to obtain a poster layout generation model, which is used for poster layout generation.

8. A poster layout generation model training device, characterized in that, Includes a processor and a transceiver, the transceiver receiving and sending data under the control of the processor, the processor being used to perform the following operations: Obtain a training set, which includes multiple poster images of layouts to be generated and descriptive text corresponding to each poster image. The descriptive text is used to describe the requirements of the mind chain-style poster layout design. Poster images and corresponding descriptive texts are selected from the training set. The poster images are converted into first image vectors, and the descriptive texts are converted into text vectors. The first image vector and the text vectors have the same dimension. The first image vector and the text vector are input into the visual language model, and after fusion processing, a thought chain-like output result is obtained. Based on the output of the thought chain, a first loss function is determined; and based on the first loss function, the model parameters in the visual language model are updated to obtain a poster layout generation model, which is used for poster layout generation.

9. A poster layout generation model training device, comprising a memory, a processor, and a computer program stored in the memory and running thereon, characterized in that, When the processor executes the program, it implements the poster layout generation model training method as described in any one of claims 1 to 6.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps in the poster layout generation model training method as described in any one of claims 1 to 6.

11. A computer program product, characterized in that, It includes computer instructions that, when executed by a processor, implement the steps in the poster layout generation model training method as described in any one of claims 1 to 6.

Images