Image layout processing method, computer program product and medium

By extracting and encoding the poster background image and layout information, combining multimodal fusion and conditional denoising models, poster layout is automatically generated, which solves the problem that existing poster design relies on manual design, and realizes efficient and intelligent poster layout generation .

CN118982603APending Publication Date: 2024-11-19BEIJING SANKUAI NETWORK TECH CO LTD
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
CN202411053135.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-01
Publication Date
2024-11-19

AI Technical Summary

Technical Problem

The existing poster production methods rely on manual design, which is time-consuming and labor-intensive, and it is difficult to meet the needs of large-scale poster design, and it also requires high aesthetic standards of designers.

Method used

By obtaining background image and layout information, the visual representation module, layout information representation module and multi-modal fusion module are used for feature extraction and encoding, and multiple rounds of denoising processing are performed in combination with the conditional denoising model to automatically generate poster layout.

Benefits of technology

It significantly improves the diversity and adaptability of poster layout, reduces manual intervention, reduces human resources and time costs, improves design efficiency, and achieves efficient and intelligent poster layout generation.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118982603A_ABST
    Figure CN118982603A_ABST
Patent Text Reader

Abstract

The invention discloses an image layout processing method, which comprises the following steps: acquiring a background image, and inputting the background image into a visual representation module for image processing and feature extraction to obtain a first visual feature sequence; obtaining layout information to be subjected to layout display in the background image, and inputting the layout information into a layout information representation module for encoding to obtain a first layout feature sequence; inputting the first visual feature sequence and the first layout feature sequence into a multi-modal fusion module for fusion to obtain a multi-modal condition control sequence; and inputting the multi-modal condition control sequence as a control condition into the condition denoising model, and obtaining the layout position of the layout information in the background image through multiple rounds of denoising processing. According to the method, the design and processing efficiency of layout of information such as characters in the image can be improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The embodiments of the present application relate to the field of Internet technology, and in particular to an image layout processing method, a computer program product, and a computer-readable storage medium. Background Art

[0002] With the development of Internet business, the demand for promotional pictures, especially posters, is increasing. At present, the main way to make posters is for designers to layout pictures and texts based on experience and aesthetics, and complete the poster production by manually adjusting the position and size of elements. At present, some software tools can provide some basic poster layout templates and element adjustment functions to assist designers in layout creation, but the requirements for designers are still very high. Therefore, the existing poster production method relies on the manual design process, which is time-consuming and labor-intensive, and requires designers to have a high level of aesthetics, which is difficult to meet the needs of large-scale poster design. Summary of the invention

[0003] The embodiment of the present application provides an image layout processing method, a computer program product and a computer readable storage medium. The technical solution is as follows:

[0004] In a first aspect, an embodiment of the present application provides an image layout processing method, the method comprising: obtaining a background image, inputting the background image into a visual representation module for image processing and feature extraction, and obtaining a first visual feature sequence; obtaining layout information to be displayed in the background image, inputting the layout information into a layout information representation module for encoding, and obtaining a first layout feature sequence; inputting the first visual feature sequence and the first layout feature sequence into a multimodal fusion module for fusion, and obtaining a multimodal conditional control sequence; inputting the multimodal conditional control sequence as a control condition into a conditional denoising model, and obtaining the layout position of the layout information in the background image through multiple rounds of denoising processing.

[0005] In one possible implementation, the visual representation module includes a convolutional neural network and a first transform encoder connected in series, wherein the step of acquiring a background image and inputting the background image into the visual representation module for image processing and feature extraction to obtain a first visual feature sequence includes: inputting the background image into a convolutional neural network, generating visual features through the convolutional neural network, flattening and position encoding to obtain a first vector, and integrating and feature extracting the first vector through a first transform encoder to obtain a first visual feature sequence.

[0006] In a possible implementation, the method further includes: obtaining a background image, inputting the background image into a visual representation module for image processing and feature extraction, and the step of obtaining a first visual feature sequence further includes: obtaining a background image, generating a saliency detection map based on the background image, splicing and downsampling the background image and the saliency detection map, and then inputting them into the visual representation module.

[0007] In one possible implementation, the layout information representation module includes a bidirectional encoder, an embedding encoder, and a second transform encoder, and the output ends of the bidirectional encoder and the embedding encoder are connected to the input end of the second transform encoder, wherein the layout information to be displayed in the background image is obtained, and the layout information is input into the layout information representation module for encoding, and the step of obtaining the first layout feature sequence includes: inputting the content information in the layout information into the bidirectional encoder for semantic recognition to obtain a semantic feature vector, inputting the attribute information in the layout information into the embedding encoder for attribute embedding processing to obtain an embedding vector, and encoding the semantic feature vector and the embedding vector through the second transform encoder to obtain the first layout feature sequence.

[0008] In a possible implementation, the first visual feature sequence and the first layout feature sequence are input into a multimodal fusion module for fusion to obtain a multimodal conditional control sequence, and the step includes: using the first layout feature sequence as a query condition, performing cross-attention processing using the first visual feature sequence, and obtaining a second layout feature sequence; using the first visual feature sequence as a query condition, performing cross-attention processing using the second layout feature sequence, and obtaining a second visual feature sequence; using the second layout feature sequence as a query condition, performing cross-attention processing using the second visual feature sequence, and obtaining a third layout feature sequence; and splicing the second visual feature sequence and the third layout feature sequence to obtain a multimodal conditional control sequence.

[0009] In one possible implementation, the conditional denoising model includes a normalization layer, a self-attention mechanism layer, a first addition and normalization layer, a cross attention mechanism layer, a second addition and normalization layer, and a feedforward neural network layer connected in sequence, wherein the input of the normalization layer is also simultaneously used as the input of the first addition and normalization layer, and the output of the first addition and normalization layer is also simultaneously used as the input of the second addition and normalization layer, and the multimodal conditional control sequence serves as the control condition of the cross attention mechanism layer.

[0010] In one possible implementation, the initial input of the conditional denoising model is a white noise coordinate sequence, which includes element types and element coordinates. When a multimodal conditional control sequence is used as a control condition, the denoised coordinate sequence is output, and the conditional denoising model is used to perform cyclic denoising for a predetermined number of times to obtain the position coordinates of the layout information in the background image.

[0011] In one possible implementation, the training process of the conditional denoising model includes: inputting the original value of the element type and the original value of the element coordinate of the training image into the conditional denoising model, outputting a white noise coordinate sequence through a forward diffusion process, outputting the element type estimated value and the element coordinate estimated value of the training image through a reverse denoising process, and repeatedly training so that the element type estimated value and the element coordinate estimated value are close to the original value of the element type and the original value of the element coordinate.

[0012] In a second aspect, the present application provides a computer program product, including a computer program, characterized in that the computer program implements any of the above methods when executed by a processor.

[0013] In a third aspect, the present application provides a computer-readable storage medium, in which at least one program code is stored. The at least one program code is loaded and executed by a processor to enable a computer to implement any of the above methods.

[0014] The technical solution of this application brings at least the following beneficial effects: using machine learning technology to better understand and respond to user needs, effectively process massive data, and automatically generate high-quality picture and text layouts, significantly improving the diversity and adaptability of layouts to meet diverse design needs. In addition, this technology can also significantly reduce manual intervention, reduce human resources and time costs, thereby improving design efficiency and achieving more efficient and intelligent poster layout generation. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the drawings below without paying any creative work.

[0016] Figure 1 It is a flowchart of an image layout processing method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0017] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0018] The implementation environment of an image layout processing method provided in an embodiment of the present application may include: an electronic device. Optionally, the implementation environment may include an electronic device and a server. The electronic device may be at least one of a smart phone, a game console, a desktop computer, a tablet computer, and a device supporting an electronic painting function. The electronic device and the server may communicate with each other.

[0019] Based on the above implementation environment, the present application embodiment provides an image layout processing method. Figure 1 The flowchart of an image layout processing method provided by the embodiment of the present application is shown as an example. Figure 1 As shown, the method may include the following steps:

[0020] In step 100, a background image is obtained, and the background image is input into a visual representation module for image processing and feature extraction to obtain a first visual feature sequence.

[0021] In the promotional image production scenario of this application, there is a wide range of choices for background images, including natural scenery, abstract patterns, portraits, animal pictures, cultural elements, technological elements, etc., and can even be photo walls, art illustrations, etc. The colors can be color or black and white, etc., and there is no restriction here.

[0022] In an exemplary embodiment of the present application, the visual representation module may include a convolutional neural network and a first transform encoder connected in series.

[0023] Convolutional Neural Network (CNN) is a deep learning model used in image recognition, video analysis, natural language processing and other fields. In this application, it is used to automatically learn the features of images and identify object categories and scenes in images. The first transform encoder (Transformer) can capture the global context and high-level semantic information in the image.

[0024] In one possible implementation, the input of the visual representation module can be a background image. First, a convolutional neural network or a convolutional neural network backbone network, such as VGG, ResNet, etc., is used to extract deep features of the image and generate visual features. CNN has been pre-trained on large-scale image datasets and can effectively extract high-level semantic information of the image. The feature map output by CNN is flattened, and the multi-dimensional feature map is converted into a vector form for subsequent processing. Position encoding is added to the flattened feature vector, and the position information is encoded into the feature vector to obtain the first vector. Position encoding can effectively help the model understand the features at different positions in the image.

[0025] The first vector is passed through a first transform encoder, such as BERT or ViT, and the self-attention mechanism is used to further integrate and extract features from the flattened feature vector to obtain a first visual feature sequence F V Among them, the self-attention mechanism can help the visual representation module capture the mutual relationship between feature vectors, so as to better understand the image content.

[0026] Through the above steps, the visual representation module can effectively convert the background image into a computer-understandable visual feature sequence, providing important visual information for subsequent poster layout generation.

[0027] In a possible implementation, the input of the visual representation module may be a background image and a saliency detection map of the background image. The background image is obtained, a saliency detection map is generated based on the background image, and the background image and the saliency detection map are concatenated and downsampled before being input into the visual representation module.

[0028] The background image can be identified by a saliency detection algorithm, such as U-Net, DeepLab series, SalGAN, etc., to identify the most important areas in the image and generate a corresponding saliency detection map. The saliency detection map is usually represented by a grayscale image, where areas with higher brightness represent areas with higher saliency, and areas with lower brightness represent areas with lower saliency.

[0029] After obtaining the saliency detection map, the background image and the saliency detection map are spliced ​​according to the channels. For example, the three color channels of the background image are spliced ​​with the grayscale channel of the saliency detection map to form four-channel input data. The spliced ​​image is downsampled to reduce the image resolution, reduce the amount of calculation, and retain the important features of the image. The result is then input into the visual representation module for processing.

[0030] This method can enable the model to better understand the important areas in the image by fusing the saliency detection map, so as to generate layout more effectively. At the same time, it can help the model better deal with interference factors such as occlusion and noise in the image, and improve the robustness of layout generation.

[0031] In step 200, layout information to be displayed in the background image is obtained, and the layout information is input into a layout information representation module for encoding to obtain a first layout feature sequence.

[0032] In an exemplary embodiment of the present application, the layout information representation module may include a bidirectional encoder, an embedded encoder, and a second transform encoder, and the output ends of the bidirectional encoder and the embedded encoder are connected to the input end of the second transform encoder.

[0033] Among them, the bidirectional encoder can be a BERT model. In the field of natural language processing, the BERT model can simultaneously process the contextual information of the text sequence, more accurately understand the meaning of the text, and apply various language tasks on this basis. The embedding encoder is used to process the attribute information and convert the information into an embedding vector. The second transform encoder further extracts and integrates features from the information. The 6-layer Transformer encoder is an implementation of the second transform encoder, which can capture the contextual relationship in the feature sequence and generate a feature representation that represents the layout information.

[0034] In a possible implementation, the layout information to be displayed is obtained by a user through an electronic device input or other means. The layout information may include content information and attribute information. The attribute information may include length information and / or type information of the layout. In a possible implementation, the layout information includes three elements: logo, text, and substrate, and each element has three attributes: content, length, and type. If there is no specific content, the content attribute of the element can be set to empty. The text can be the text content that you want to display in the background image. The substrate can be the background color of the logo or text when it is laid out in the background image.

[0035] In a possible implementation, the attribute information includes the length information and type information of the layout. The input of the layout information representation module can be defined as: Where i represents the i-th element, N represents the number of elements in the layout, and c i Represents the type information of the layout, m i Indicates the length information of the layout, s i Indicates content information.

[0036] In a preferred embodiment, m i It can be expressed as the maximum aspect ratio:

[0037]

[0038] Among them, w i =r i -l i and h i =b i -t i . (l i ,t i )、(r i ,b i ) represent the coordinates of the top-left and bottom-right corners of the bounding box of the layout content to be displayed in the background image. This length construction method is independent of the length of the layout and is not affected by font or language.

[0039] The content information is input into the bidirectional encoder, and the model is used to capture the semantics of the content information, such as sentence structure, lexical meaning, etc., and encode it into a vector to obtain a semantic feature vector.

[0040] Input the attribute information into the embedding encoder. The embedding encoder usually uses a neural network, such as a multi-layer perceptron, to map the original information into a high-dimensional vector space. The embedding encoder performs attribute embedding processing on the attribute information to obtain an embedding vector.

[0041] Optionally, the three types of information of each layout information may be arranged in order and then connected to form an ID sequence R = (c1, M1, s1, ..., c N ,m N ,s N ), using the embedding encoder to i and m i Processing to get the embedding vector, use the bidirectional encoder to transform s i Converted into a semantic feature vector. At this time, the dimension of the obtained vector is C, and the original ID sequence is mapped to the vector space to obtain the vector E R ∈R 3N×C Then, the layout information is encoded using a second transform encoder, which mainly includes: rearranging the vector E according to the order of the input of the layout information representation module R ; Use the 6-layer Transformer encoder to capture the contextual relationship in the layout information sequence and obtain the first layout feature sequence E R ∈R 3N×C And output.

[0042] Optionally, the content information and attribute information of each layout information may be processed separately to obtain a semantic feature vector and an embedding vector, and the two vectors may be concatenated and encoded through a second transform encoder to obtain a first layout feature sequence.

[0043] After the layout information is processed by the layout information representation module, a series of first layout feature sequences that can represent the layout information are generated. This sequence not only contains the contextual relationship of the layout, but also integrates its attribute information, providing rich guidance for the generation of poster layout. This processing flow converts layout information into a feature sequence suitable for model understanding, allowing the machine to efficiently and accurately understand the layout input, thereby significantly improving the quality and efficiency of poster layout design.

[0044] In step 300, the first visual feature sequence and the first layout feature sequence are input into a multimodal fusion module for fusion to obtain a multimodal conditional control sequence.

[0045] The multimodal fusion module includes a plurality of cross attention processing modules. In a possible implementation, the multimodal fusion module includes three cross attention processing modules connected in series. The multimodal fusion module uses the first visual feature sequence F V and the first layout feature sequence F R As input, the two are fused to obtain the final multimodal conditional control sequence F U .

[0046] In one possible implementation, a multi-layer perceptron (MLP) may be used to aggregate the first layout feature sequence F R The element c i ,m i ,s i The multilayer perceptron can be used to aggregate and compress the information in the first layout feature sequence. The multilayer perceptron first converts each element of the first layout feature sequence into a format more suitable for neural network processing; then the converted information is passed through multiple hidden layers, each of which is processed by a nonlinear activation function such as ReLU to extract and combine features at different levels; finally, the output layer of the multilayer perceptron merges the information of all elements to generate a compact vector, which represents the first layout feature sequence F. R The comprehensive characteristics of each element in.

[0047] In the first cross-attention processing module, F R As a query condition, V Perform cross-attention query and processing to update F R , get the second layout feature sequence F' R :

[0048] F′ R =CrossAttention(Q=F R ,K=F V ,V=F V )

[0049] Among them, CrossAttention represents the cross attention operation, Q, K and V represent query (Query), key (Key) and value (Value) respectively.

[0050] The cross-attention mechanism allows the model to focus on different parts of another related sequence when processing a sequence, such as text, image features, etc. Its working process includes:

[0051] Calculating attention score: Using F R Sequence as query condition, F V The sequence is used as the key to calculate the attention score, which represents F REach element in the sequence is related to F V The correlation between each element in the sequence.

[0052] Calculate the probability distribution: The calculated attention score is converted into a probability distribution through the Softmax function. The probability represents F V Each element in the sequence is R The relative importance of the query vector in the sequence.

[0053] Generate weighted value vector: Use the obtained attention weights to F V The values ​​in the sequence are weighted and summed to obtain a weighted value vector, and F is obtained. R A contextual representation of each element in the sequence.

[0054] Update F R :Use the calculated weighted value vector as the update F R Sequence basis, these vectors and the original F R The sequences are merged to obtain the updated F' R sequence.

[0055] Through the cross attention mechanism, F representing the layout information to be laid out can be R and F, which represents visual features V Combined, they generate a more comprehensive and accurate multimodal representation for guiding the generation of poster layouts.

[0056] In the second cross-attention processing module, the first visual feature sequence F V As the query condition, the second layout feature sequence F' is used R Perform cross-attention query and processing to update F V , get the second visual feature sequence F' V :

[0057] F' V =CrossAttention(Q=F V , K = F′ R , V=F R )

[0058] In the third cross-attention processing module, the second layout feature sequence F' R As the query condition, using the second visual feature sequence F' V Perform cross-attention query and processing to update F' R , and obtain the third layout feature sequence F″ R :

[0059] F″ R =CrossAttention(Q=FR ,K=F′ V ,V=F′ V )

[0060] Finally, the second visual feature sequence and the third layout feature sequence are concatenated to obtain a multimodal conditional control sequence F U :

[0061] F U =Concatenate(F′ V ,F″ R )

[0062] By performing multiple rounds of attention operations on layout features and visual features, the two features can be deeply fused, and visual features and layout features can be better integrated to provide more comprehensive constraints for the layout. Through multiple rounds of iterative optimization, each round of calculation provides new contextual information for the next round, gradually eliminating irrelevant information, so that the model gradually generates a poster layout that better meets user needs. At the same time, the multi-round attention mechanism enables the model to capture the changes in contextual information in different rounds, refine and optimize the layout information, and generate richer and more accurate multimodal representations, thereby improving the generation quality of the poster layout.

[0063] In step 400, the multimodal conditional control sequence is input as a control condition into a conditional denoising model, and through multiple rounds of denoising processing, the layout position of the layout information in the background image is obtained.

[0064] The conditional denoising model includes a normalization layer, a self-attention mechanism layer, a first addition and normalization layer, a cross attention mechanism layer, a second addition and normalization layer, and a feedforward neural network layer connected in sequence, wherein the input of the normalization layer is also used as the input of the first addition and normalization layer, and the output of the first addition and normalization layer is also used as the input of the second addition and normalization layer, and the multimodal conditional control sequence is used as the control condition of the cross attention mechanism layer.

[0065] Preferably, the above model structure can also be used as a model unit. The conditional denoising model can include one or more such model units. When actually used, multiple such units can be connected and combined. In a possible implementation, six such model units can be connected in series.

[0066] The training process of the conditional denoising model includes:

[0067] Forward diffusion process: The element type and element coordinates of the training image are input into the conditional denoising model. Through the forward diffusion process, a white noise coordinate sequence is finally output, including the element type and element coordinates of the white noise.

[0068] Given a layout sequence x0, with a step size of t and a total time step of T, the sequence gradually generates a series of latent variables with increasing noise through the following process. For a single token in the sequence x0, the forward diffusion process can be expressed as:

[0069]

[0070] Where q represents the transition probability. v(x t ) is a one-hot column vector. i and j represent the position in the sequence. Q t is the transfer matrix, representing the transition from x t-1 to x t The transition probability, and the transition probability [Q t ] ij =q(x t =j|x t-1 =i). Due to the characteristics of Markov chains, it can be deduced that in

[0071] For type c in the sequence i , preferably, an absorption state MASK marker can be introduced. In the noise process, c i Gradually transforms into MASK, where MASK is a special marker used to indicate elements that gradually disappear in the noise process.

[0072] Reverse denoising process: The reverse denoising process is the opposite of the forward diffusion process. The input of the reverse denoising process is a white noise coordinate sequence, and the output is the element type estimate and element coordinate estimate of the training image. Repeated training makes the element type estimate and the element coordinate estimate close to the original value of the element type and the original value of the element coordinate.

[0073] In one possible implementation, the reverse denoising process uses a conditional denoising model containing multiple model units connected in series to estimate the posterior distribution p θ (x t-1 |x t ,y). Using the reparameterization technique, in each iteration, the model gradually predicts the initial sequence distribution p θ (x0|x t ,y). The posterior distribution expression is:

[0074]

[0075] Among them, y is the multimodal conditional control sequence, x is the layout sequence, t is the step size, and q is the transition probability.

[0076] The loss function mainly includes the commonly used variational lower bound objective and auxiliary denoising objective:

[0077]

[0078] in, is a variational lower bound objective that measures the difference between the predicted distribution and the true data distribution. λ is a hyperparameter. logp θ (x0|x t ,y) is the auxiliary denoising target, which is used to predict the original data distribution.

[0079] The conditional denoising model provides an efficient and flexible method to process noisy data and recover high-quality original data from the noise. The model combines the diffusion process and the denoising process to learn how to recover the original data from the noisy data through training. It supports multimodal processing, has strong scalability, high training efficiency, and can generate high-quality original data.

[0080] The use process of the conditional denoising model includes: the initial input of the model is a white noise coordinate sequence, which includes element types and element coordinates. Under the condition of a multimodal conditional control sequence as a control condition, the denoised coordinate sequence is output, and the conditional denoising model is used to perform a predetermined number of cyclic denoising to obtain the position coordinates of the layout information to be laid out in the background image.

[0081] In actual use, the user can upload a background image on the user device and input the layout information to be displayed in the background image, including the content, length, attributes and other information of the layout. The method can automatically output the position information of the layout in the background image. The specific form of the position information can be one or more input boxes, the coordinate values ​​of the layout information in the background image, or a new image obtained by laying out the layout content in the background image, etc., which is not limited here.

[0082] The solution of this application builds a poster layout generation model by integrating multiple models, achieving the expansion of data volume and the improvement of generation quality. At the same time, through multi-modal condition input, accurate and controllable layout generation is achieved, reducing manual intervention, improving the level of automation, and greatly saving human resources and time. This solution adopts advanced algorithm strategies and provides an efficient solution, bringing breakthrough improvements to poster layout design.

[0083] Another embodiment of the present application further provides a structural schematic diagram of an image layout processing device, the device comprising:

[0084] A background image processing module is used to obtain a background image, and input the background image into the visual representation module for image processing and feature extraction to obtain a first visual feature sequence;

[0085] A layout information processing module, used for acquiring layout information to be displayed in the background image, inputting the layout information into the layout information representation module for encoding, and obtaining a first layout feature sequence;

[0086] A multimodal fusion processing module, used for inputting the first visual feature sequence and the first layout feature sequence into a multimodal fusion module for fusion to obtain a multimodal condition control sequence;

[0087] The layout prediction module is used to input the multimodal conditional control sequence as a control condition into the conditional denoising model, and obtain the layout position of the layout information in the background image through multiple rounds of denoising processing.

[0088] It should be understood that when the above-mentioned device realizes its functions, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the device and method embodiments provided in the above-mentioned embodiments belong to the same concept. For other specific implementation processes of the method embodiments, please refer to the method embodiments for details, which will not be repeated here.

[0089] In an exemplary embodiment, a computer-readable storage medium is further provided, in which at least one program code is stored. The at least one program code is loaded and executed by a processor to enable a computer to implement any of the above-mentioned image layout processing methods.

[0090] Optionally, the computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a compact disc (CD-ROM), a magnetic tape, a floppy disk, an optical data storage device, etc.

[0091] In an exemplary embodiment, a computer program or a computer program product is also provided, wherein at least one computer instruction is stored in the computer program or the computer program product, and the at least one computer instruction is loaded and executed by a processor to enable a computer to implement any of the above-mentioned image layout processing methods.

[0092] It should be understood that the "plurality" mentioned in this article refers to two or more. "And / or" describes the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the associated objects are in an "or" relationship.

[0093] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0094] The above are merely exemplary embodiments of the present application and are not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A method for processing image layout, characterized in that: The method comprises: Acquire a background image, and input the background image into a visual representation module for image processing and feature extraction to obtain a first visual feature sequence; Acquire layout information to be displayed in the background image, input the layout information into a layout information representation module for encoding, and obtain a first layout feature sequence; Inputting the first visual feature sequence and the first layout feature sequence into a multimodal fusion module for fusion to obtain a multimodal condition control sequence; The multimodal conditional control sequence is input as a control condition into a conditional denoising model, and the layout position of the layout information in the background image is obtained through multiple rounds of denoising processing.

2. The method according to claim 1, characterized in that The visual representation module includes a convolutional neural network and a first transform encoder connected in series, wherein the step of obtaining a background image, inputting the background image into the visual representation module for image processing and feature extraction, and obtaining a first visual feature sequence includes: The background image is input into a convolutional neural network, visual features are generated by the convolutional neural network, and flattening and position encoding are performed to obtain a first vector, and the first vector is integrated and feature extracted by a first transform encoder to obtain a first visual feature sequence.

3. The method according to claim 2, characterized in that The step of obtaining a background image and inputting the background image into a visual representation module for image processing and feature extraction to obtain a first visual feature sequence also includes: A background image is obtained, a saliency detection map is generated based on the background image, and the background image and the saliency detection map are spliced ​​and downsampled before being input into a visual representation module.

4. The method according to claim 1, characterized in that: The layout information representation module includes a bidirectional encoder, an embedded encoder, and a second transform encoder, and the output ends of the bidirectional encoder and the embedded encoder are connected to the input end of the second transform encoder, wherein the step of acquiring the layout information to be displayed in the background image, inputting the layout information into the layout information representation module for encoding, and obtaining the first layout feature sequence includes: The content information in the layout information is input into a bidirectional encoder for semantic recognition to obtain a semantic feature vector, the attribute information in the layout information is input into an embedding encoder for attribute embedding processing to obtain an embedding vector, and the semantic feature vector and the embedding vector are encoded through a second transform encoder to obtain a first layout feature sequence.

5. The method according to claim 1, characterized in that The step of inputting the first visual feature sequence and the first layout feature sequence into a multimodal fusion module for fusion to obtain a multimodal condition control sequence comprises: Using the first layout feature sequence as a query condition, performing cross-attention processing using the first visual feature sequence, and obtaining a second layout feature sequence; Using the first visual feature sequence as a query condition, and performing cross-attention processing using the second layout feature sequence to obtain a second visual feature sequence; Using the second layout feature sequence as a query condition, performing cross attention processing using the second visual feature sequence, and obtaining a third layout feature sequence; The second visual feature sequence and the third layout feature sequence are spliced ​​to obtain a multimodal conditional control sequence.

6. The method according to claim 1, characterized in that The conditional denoising model includes a normalization layer, a self-attention mechanism layer, a first addition and normalization layer, a cross attention mechanism layer, a second addition and normalization layer, and a feedforward neural network layer connected in sequence, wherein the input of the normalization layer is also used as the input of the first addition and normalization layer, and the output of the first addition and normalization layer is also used as the input of the second addition and normalization layer, and the multimodal conditional control sequence is used as the control condition of the cross attention mechanism layer.

7. The method according to any one of claims 1 to 6, characterized in that: The initial input of the conditional denoising model is a white noise coordinate sequence, which includes element types and element coordinates. When a multimodal conditional control sequence is used as a control condition, the denoised coordinate sequence is output, and the conditional denoising model is used to perform cyclic denoising for a predetermined number of times to obtain the position coordinates of the layout information in the background image.

8. The method according to claim 7, characterized in that The training process of the conditional denoising model includes: inputting the original values ​​of the element type and the original values ​​of the element coordinates of the training image into the conditional denoising model, outputting a white noise coordinate sequence through a forward diffusion process, outputting the element type estimated value and the element coordinate estimated value of the training image through a reverse denoising process, and repeatedly training so that the element type estimated value and the element coordinate estimated value are close to the original values ​​of the element type and the original values ​​of the element coordinates.

9. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores at least one program code, and the at least one program code is loaded and executed by a processor so that the computer implements the method according to any one of claims 1 to 8.

Citation Information

Patent Citations

  • Data processing method and device, electronic equipment and storage medium

    CN115115913A

  • Aesthetics quality evaluation model and method based on multi-modal learning

    CN115601772A

  • Fine-grained editing method for character and image styles

    CN116563422A