Poster text layout, poster generation method and related devices

Abstract

This application discloses a poster text layout, poster generation method, and related devices. By applying the technical solution of this application, a cascaded autoencoder can be used to select a usable area from a smooth area within a poster background image. This area is then iteratively refined using another cascaded autoencoder to automatically determine the text box within it and fill it with text information, thereby intelligently generating a poster image. This avoids the problem faced by related technologies that rely solely on manual design to generate posters and fail to meet the high demand for high-quality posters.

Description

Technical Field

[0001] The present application relates to image data processing technology, in particular to a poster text layout, a poster generation method and related devices. Background Art

[0002] As a medium that combines both artistic and functional qualities, posters have been widely used in many commercial and non-commercial scenarios to promote and disseminate information. For example, e-commerce platforms use attractive posters to promote their own products.

[0003] In the related art, existing poster creation methods typically rely on professional artists to process and typeset individual product images and information. This process involves manually selecting the poster's background image, manually laying out the text within the poster, and manually determining the text style.

[0004] However, this time-consuming and subjective process of manually designing and generating posters cannot meet the large and rapidly growing demand for high-quality posters in real-world applications, thereby reducing the efficiency of information dissemination. Therefore, the problem of designing a technical solution that can automatically implement this poster generation process using pre-trained models has become a problem that needs to be solved. Summary of the Invention

[0005] The embodiments of the present application provide a poster text layout, a poster generation method, and related devices to solve the problem in the related art of reducing information dissemination efficiency due to manual poster generation.

[0006] According to one aspect of the embodiments of the present application, a poster text layout method is provided, including:

[0007] Determine a smooth region in the poster background image corresponding to the poster description text, and select a usable region in the smooth region using a first autoencoder;

[0008] The available area is sampled, and a text box corresponding to the available area is generated based on the corresponding sampling result and the text information corresponding to the poster description text for adding to the poster, so as to obtain a target text layout area in the poster background image for writing the text information.

[0009] Optionally, in another embodiment of the above method of the present application, determining a smooth area in the poster background image corresponding to the poster description text includes:

[0010] Selecting a plurality of candidate frames in the poster background image; and generating a salient image corresponding to the poster background image using a spectral residual algorithm, wherein there are overlapping candidate frames among the candidate frames;

[0011] Determining a candidate value corresponding to each candidate frame based on the saliency average value of the saliency image, the number of pixels of the candidate frame, and a preset offset;

[0012] A candidate frame whose candidate value is lower than a preset candidate threshold is selected as a target candidate frame, and a smooth area in the poster background image is determined based on the target candidate frame, wherein the positions of each target candidate frame in the poster background image do not overlap.

[0013] Optionally, in another embodiment of the above method of the present application, selecting a candidate frame whose candidate value is lower than a preset candidate threshold as a target candidate frame, and determining a smooth area in the poster background image based on the target candidate frame includes:

[0014] Based on the non-maximum suppression method, a candidate frame whose candidate value is lower than a preset candidate threshold is selected as a target candidate frame;

[0015] Determine candidate frame areas in the poster background image where all target candidate frames are located;

[0016] The candidate frame area in the poster background image is converted into a binary image, and the converted binary image area is used as a smooth area in the poster background image.

[0017] Optionally, in another embodiment of the method of the present application, selecting a usable area in the smooth area using the first autoencoder includes:

[0018] Using each object candidate box corresponding to the smooth area as an encoder end input of the first autoencoder to obtain an encoder output result, wherein the encoder end is constructed by stacking CNNs;

[0019] After connecting the encoder output result with the position embedding map, it is used as the input of the decoder side of the first autoencoder to obtain the available area in the smooth area as the initial text layout result, wherein the decoder side is constructed by stacking Transposed-CNN.

[0020] Optionally, in another embodiment of the above method of the present application, after generating the text box corresponding to the available area, the method further includes:

[0021] The second autoencoder is used to perform refinement layout processing on the text boxes corresponding to each of the available areas in an autoregressive manner to obtain the target text boxes corresponding to each of the available areas, so as to form a target text layout area in the poster background image for writing the text information.

[0022] Optionally, in another embodiment of the above method of the present application, both the first autoencoder and the second autoencoder are layout predictors having a cascaded autoencoder architecture.

[0023] Optionally, in another embodiment of the above method of the present application, a plurality of sample images are obtained, wherein each sample image includes a text area marked with a corresponding text description field;

[0024] Extracting a sample background image from each sample image, and determining a sample smooth image region in the sample background image using a smooth region detector;

[0025] Merging the text description field, the text area, the sample background image, and the sample smoothed image area into an encoder training dataset;

[0026] The first autoencoder and the second autoencoder are independently trained using the encoder training dataset, so that the first autoencoder is used to predict the probability distribution of text layout and the second autoencoder is used to refine the layout box.

[0027] Optionally, in another embodiment based on the above method of the present application, the second automatic encoder

[0028] The method is trained using a self-supervised learning strategy, wherein the encoder end of the second autoencoder is constructed by stacked CNNs, and the decoder end of the second autoencoder is constructed by a 2-layer bidirectional LSTM.

[0029] According to one aspect of the embodiments of the present application, a poster generation method is provided, including:

[0030] Obtaining a target text layout area based on the above poster text layout method;

[0031] Fill the text information in the text layout area to generate a target poster corresponding to the text information.

[0032] Optionally, in another embodiment of the above method of the present application, before obtaining the target text layout area, the method further includes:

[0033] Obtaining a poster description text, wherein the poster description text includes text information to be added to the poster;

[0034] Based on a pre-trained visual text model, a candidate background image that is weakly correlated with the text information is selected from a pre-acquired set of candidate background images as the poster background image.

[0035] Optionally, in another embodiment of the above method of the present application, filling the text information in the text layout area to generate a poster corresponding to the text information includes:

[0036] Extracting text features of the text information and detecting the background color of the text layout area;

[0037] Based on the background color of the text layout area, selecting a target text style that matches the text feature and the background color from a preset text style database;

[0038] The text information is converted according to the target text style and then filled into the text layout area to obtain a target poster corresponding to the text information.

[0039] According to another aspect of the embodiments of the present application, a poster generation device is provided, characterized by comprising:

[0040] a determination module for determining a smooth region in the poster background image corresponding to the poster description text, and selecting a usable region in the smooth region using a first autoencoder;

[0041] A generation module samples the available area and generates a text box corresponding to the available area based on the corresponding sampling results and the text information corresponding to the poster description text for adding to the poster, so as to obtain a target text layout area in the poster background image for writing the text information.

[0042] According to another aspect of the embodiments of the present application, an electronic device is provided, including:

[0043] a memory for storing executable instructions; and

[0044] The display is used to execute the executable instructions together with the memory to complete the operation of any of the above-mentioned poster generation methods.

[0045] According to another aspect of the embodiments of the present application, a computer-readable storage medium is provided for storing computer-readable instructions, which, when executed, perform the operations of any of the above-mentioned poster generation methods.

[0046] In the present application, a smooth area in the poster background image corresponding to the poster description text can be determined, and a first autoencoder can be used to select an available area in the smooth area; the available area is sampled, and a text box corresponding to the available area is generated based on the corresponding sampling results and the text information corresponding to the poster description text for adding to the poster, so as to obtain a target text layout area in the poster background image for writing text information. By applying the technical solution of the present application, a cascaded autoencoder can be used to select an available area from the smooth area in the poster background image, and then the text box is automatically determined and the text information is filled in after the available area is iteratively refined according to another cascaded autoencoder, thereby intelligently generating a poster image. This avoids the problem faced by the related art of relying solely on manual design to generate posters and failing to meet the large demand for high-quality posters.

[0047] The technical solution of the present application is further described in detail below through the accompanying drawings and examples. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments of the present application and, together with the description, serve to explain the principles of the present application.

[0049] The present application can be more clearly understood from the following detailed description with reference to the accompanying drawings, in which:

[0050] Figure 1 A schematic diagram of a poster text layout method proposed in this application;

[0051] Figure 2 A flowchart of a poster text layout proposed in this application;

[0052] Figure 3 A schematic diagram of the process of generating a poster proposed in this application;

[0053] Figure 4 Refer to the figure for an example of selecting a poster background image proposed in this application;

[0054] Figure 5 A schematic diagram comparing a text layout method proposed in this application with a text layout method in a prior art process;

[0055] Figure 6 A schematic diagram of the process of selecting a poster background image proposed in this application;

[0056] Figure 7 This is a schematic diagram of the overall process of a poster generation method proposed in this application;

[0057] Figure 8This is a schematic diagram of the structure of an electronic device for poster text layout proposed in this application;

[0058] Figure 9 This is a schematic structural diagram of an electronic device for poster text layout proposed in this application. DETAILED DESCRIPTION

[0059] Various exemplary embodiments of the present application will now be described in detail with reference to the accompanying drawings. It should be noted that unless otherwise specifically stated, the relative arrangements of components and steps, numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present application.

[0060] At the same time, it should be understood that for the convenience of description, the sizes of the various parts shown in the drawings are not drawn according to the actual proportional relationship.

[0061] The following description of at least one exemplary embodiment is merely illustrative in nature and is not intended to limit the present disclosure, its application, or uses.

[0062] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered part of the specification.

[0063] It should be noted that like reference numerals and letters refer to like items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0064] In addition, the technical solutions between the various embodiments of the present application can be combined with each other, but it must be based on the fact that ordinary technicians in this field can implement it. When the combination of technical solutions is mutually contradictory or cannot be implemented, it should be deemed that such a combination of technical solutions does not exist and is not within the scope of protection required by this application.

[0065] It should be noted that all directional indications in the embodiments of the present application (such as up, down, left, right, front, back, etc.) are only used to explain the relative position relationship, movement status, etc. between the various components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indication will also change accordingly.

[0066] The following combination Figure 1-Figure 7 The following describes a method for generating a poster according to an exemplary embodiment of the present application. It should be noted that the following application scenarios are only provided to facilitate understanding of the spirit and principles of the present application, and the embodiments of the present application are not limited in this respect. On the contrary, the embodiments of the present application can be applied to any applicable scenario.

[0067] This application also proposes a poster text layout, a poster generation method and related devices.

[0068] Figure 1 The flowchart of a method for generating a poster according to an embodiment of the present application is schematically shown. Figure 1 As shown, the method includes:

[0069] S101, determining a smooth area in a poster background image corresponding to a poster description text, and selecting an available area in the smooth area using a first automatic encoder.

[0070] In the related art, posters, as a medium that combines both artistic and functional aspects, have been widely used in many commercial and non-commercial scenarios for promoting and disseminating information. For example, e-commerce platforms use attractive posters to promote their products. Websites for social events such as conferences are often decorated with beautiful and content-rich posters. These high-quality posters are generated by embedding stylized text into suitable background images. This generation process requires a lot of manual editing work and difficult-to-quantify artistic design experience. However, such a time-consuming and subjective manual design process cannot meet the large and rapidly growing demand for high-quality posters in real-world applications, thereby reducing the efficiency of information dissemination and resulting in poor promotional effects.

[0071] In one approach, the poster generation process includes at least three steps, including:

[0072] Step 1: Select the background image of the poster;

[0073] Step 2: Lay out the text within the poster background image;

[0074] Step 3: Determine the style of the text on the poster and fill it into the background image to generate the final poster image.

[0075] Based on the above process, for step 1, in order to realize automatic search of the background image of the poster, the existing classic retrieval method generally searches for a suitable image by matching the poster content text with the background image annotation text.

[0076] However, methods in large-scale correlation techniques typically only consider unimodal features and therefore often fail to bridge the semantic gap between visual and textual modalities, resulting in highly biased retrieval results. This means that the background image selected does not match the user's current interest.

[0077] Furthermore, for step 2, conventional rule-based methods for text layout prediction typically select the text layout from a limited number of predefined layout templates, but this approach is limited in flexibility. Consequently, the background image content is often ignored when arranging the text, potentially resulting in suboptimal poster generation.

[0078] Furthermore, in order to solve the above-mentioned problem of mismatched text box selection in the poster background image, the present application proposes a technical solution for poster text layout.

[0079] In summary, the present application can first determine the smooth area in the poster background image corresponding to the poster description text, and use the first automatic encoder to select the available area in the smooth area (that is, the available area is a part of the smooth area). In addition, the embodiment of the present application can also sample the available area, so that according to the corresponding sampling results and the text information corresponding to the poster description text for adding to the poster, a text box corresponding to the available area is generated (that is, the text box is obtained in the available area), thereby finally obtaining the target text layout area in the poster background image for writing the text information.

[0080] In one approach, the present application may also first adopt a large-scale pre-trained visual text model, and select an image as the background of the target poster from a large number of candidate images based on the descriptive text of a given target poster, and then iteratively arrange the text on the background image through multiple cascaded autoencoders, and finally stylize the text through a matching-based method to synthesize the target poster image.

[0081] In one embodiment, the present application can implement the following steps in the process of determining the smooth area in the poster background image corresponding to the poster description text:

[0082] This application needs to first generate several regions (candidate boxes) of different sizes in the background image I. There is overlap in these regions, which are recorded as Then, the spectral residual method is applied to generate the saliency map corresponding to the background image I, denoted as S.

[0083] For each candidate box, a candidate value v is assigned to each candidate box by calculating the average value of the saliency map S. i .Right now

[0084] Among them, S(p) refers to the saliency of the text layout prediction result p, |A i |It is A i is the number of pixels in , and λ is an offset that is sensitive to the size of the candidate box.

[0085] In a preferred embodiment, the present invention can select a candidate frame whose candidate value is lower than a preset candidate threshold as the target candidate frame. The candidate frame region in the poster background image where the target candidate frame is located is determined, and the candidate frame region in the poster background image is converted into a binary image. Finally, the converted binary image region is used as the smoothed image region in the poster background image.

[0086] For example, Figure 2 As shown, the central portion of image (a) is the salient image S of the background image I, which can be displayed in a first color (e.g., blue) in practical applications. The rectangular area near the edge of image (a) is the smooth area image A identified by this step, which can be displayed in a second color (e.g., red) that is clearly different from the first color.

[0087] Furthermore, the present application requires using a first autoencoder to determine at least one usable region in the smooth image region.

[0088] In one approach, the present application can use the first cascaded autoencoder g1 to predict the probability distribution for text layout based on the smooth area A obtained in the above steps, denoted as L. For each pixel p, L(p)∈[0,1] represents the probability that p belongs to a text box.

[0089] The first cascaded autoencoder g1 used in this application has an autoencoder architecture, where the encoder f1 is a stacked CNN and the decoder h1 is a stacked Transposed-CNN. The input to the decoder h1 is then constructed by concatenating the output of the encoder f1 with a learnable position embedding graph (denoted as E).

[0090] For example Figure 2 The white part in image (b) shows the probability distribution prediction result L of text layout in the poster background image I, where the brighter the color, the higher the layout probability.

[0091] S102: Sampling the available area, generating a text box corresponding to the available area based on the sampling result and the text information corresponding to the poster description text for adding to the poster, so as to obtain a target text layout area for writing text information in the poster background image.

[0092] In one embodiment, the present application may utilize a second autoencoder to iteratively refine the layout of at least one available area in an autoregressive manner to obtain a text box corresponding to the available area.

[0093] The layout prediction probability distribution result L obtained in the above steps is the initial area screening for text layout, which needs to be refined to form clear text boxes for each available area.

[0094] That is, first, based on L, the layout is initialized by sampling each from L. The unnormalized coordinates are used as the upper left corner coordinates of the i-th text box and the box is initialized. Its size is determined by the length and attributes of the corresponding text information. Finally, the position and size of each text box for writing each text information are determined in the background image I, which is the text layout prediction result.

[0095] In a specific implementation, the present application can use the second-level league autoencoder to perform text layout refinement processing on the available area to obtain the final text layout area, which includes:

[0096] P (k+1) =g2(Concat(A,L),P (k) ), k=0,...,K-1.

[0097] Where k is the number of iterations, g2 is the second autoencoder, A is the smoothed image area, L is the probability distribution of the available area, and P is the pixel in the poster background image.

[0098] It should be noted that for the second-level league autoencoder, its encoder can be a stacked CNN and its decoder can be a 2-layer bidirectional LSTM.

[0099] In a preferred embodiment, the first cascaded autoencoder used for available area layout prediction in the above step and the second cascaded autoencoder used for text layout refinement in the above step can both adopt a layout predictor with a cascaded autoencoder architecture, thereby achieving the purpose of effectively imitating the manual image editing process.

[0100] In the present application, a smooth area in the poster background image corresponding to the poster description text can be determined, and a first autoencoder can be used to select an available area in the smooth area; the available area is sampled, and a text box corresponding to the available area is generated based on the corresponding sampling results and the text information corresponding to the poster description text for adding to the poster, so as to obtain a target text layout area in the poster background image for writing text information. By applying the technical solution of the present application, a cascaded autoencoder can be used to select an available area from the smooth area in the poster background image, and then the text box is automatically determined and the text information is filled in after the available area is iteratively refined according to another cascaded autoencoder, thereby intelligently generating a poster image. This avoids the problem faced by the related art of relying solely on manual design to generate posters and failing to meet the large demand for high-quality posters.

[0101] Optionally, in another embodiment of the above method of the present application, determining a smooth area in the poster background image corresponding to the poster description text includes:

[0102] Selecting multiple candidate frames in the poster background image; and generating a salient image corresponding to the poster background image using a spectral residual algorithm, wherein there are overlapping candidate frames among the candidate frames;

[0103] Determine a candidate value corresponding to each candidate frame based on the saliency average value of the saliency image, the number of pixels of the candidate frame, and a preset offset;

[0104] A candidate frame whose candidate value is lower than a preset candidate threshold is selected as a target candidate frame, and a smooth area in the poster background image is determined based on the target candidate frame, wherein the position of each target candidate frame in the poster background image does not overlap.

[0105] Optionally, in another embodiment of the above method of the present application, selecting a candidate frame whose candidate value is lower than a preset candidate threshold as a target candidate frame, and determining a smooth area in the poster background image based on the target candidate frame includes:

[0106] Based on the non-maximum suppression method, the candidate box with a candidate value lower than the preset candidate threshold is selected as the target candidate box;

[0107] Determine the candidate frame area of ​​all target candidate frames in the poster background image;

[0108] The candidate box area in the poster background image is converted into a binary image, and the converted binary image area is used as the smooth area in the poster background image.

[0109] In one embodiment, the present application can implement the following steps in the process of determining the smooth area in the poster background image corresponding to the poster description text:

[0110] This application needs to first generate several regions (candidate boxes) of different sizes in the background image I. There is overlap in these regions, which are recorded as Then, the spectral residual method is applied to generate the saliency map corresponding to the background image I, denoted as S.

[0111] For each candidate box, a candidate value v is assigned to each candidate box by calculating the average value of the saliency map S. i .Right now

[0112] Among them, S(p) refers to the saliency of the text layout prediction result p, |A i |It is A i is the number of pixels in , and λ is an offset that is sensitive to the size of the candidate box.

[0113] In a preferred embodiment, the present invention can select a candidate frame whose candidate value is lower than a preset candidate threshold as the target candidate frame. The candidate frame region in the poster background image where the target candidate frame is located is determined, and the candidate frame region in the poster background image is converted into a binary image. Finally, the converted binary image region is used as the smoothed image region in the poster background image.

[0114] It should be noted that the preset candidate threshold can be adaptively set according to different images, that is, the threshold can be set according to the average value of the values ​​corresponding to each area of ​​the current background image I. For example, the threshold can be adaptively set to 1.4×mean{}, and the non-maximum suppression method NMS is applied to ensure that the final selected areas from the aforementioned generated areas do not overlap (for example, 1000 candidate boxes are generated in the early stage, and these boxes may overlap. But it is necessary to ensure that the final selected areas do not overlap).

[0115] Optionally, in another embodiment of the method of the present application, selecting a usable area in the smooth area using the first autoencoder includes:

[0116] Each target candidate box corresponding to the smooth area is used as the encoder input of the first autoencoder to obtain the encoder output result, where the encoder end is constructed by stacking CNNs;

[0117] After connecting the encoder output result with the position embedding map, it is used as the input of the decoder side of the first autoencoder to obtain the available area in the smooth area as the initial text layout result, where the decoder side is constructed by stacking Transposed-CNN.

[0118] The first cascaded autoencoder g1 used in this application has an autoencoder architecture, where the encoder f1 is a stacked CNN and the decoder h1 is a stacked Transposed-CNN. The input to the decoder h1 is then constructed by concatenating the output of the encoder f1 with a learnable position embedding graph (denoted as E).

[0119] Specifically, the present application can construct the input of the decoder h1 by connecting the output of the encoder f1 with the learnable position embedding map (denoted as E), thereby obtaining the probability distribution prediction result L of the text layout that can be performed in the background image I (that is, the available area of ​​the initial text layout result). The formula is as follows:

[0120] L=g1(A)=h1(Concat(f1(A),E)).

[0121] Among them, A represents the smooth area and E represents the position embedding map.

[0122] Optionally, in another embodiment of the above method of the present application, after generating the text box corresponding to the available area, the method further includes:

[0123] The second autoencoder is used to perform refinement layout processing on the text boxes corresponding to each available area in an autoregressive manner to obtain the target text boxes corresponding to each available area, so as to form the target text layout area for writing text information in the poster background image.

[0124] The second autoencoder is used to iteratively refine the layout of at least one available area in an autoregressive manner to obtain a target text layout area.

[0125] In one embodiment, the layout prediction probability distribution result L obtained by the present application in the above steps is the initial area screening for text layout, which needs to be refined to form clear text boxes for each available area.

[0126] That is, first, based on L, the layout is initialized by sampling each from L. The unnormalized coordinates are used as the upper left corner coordinates of the i-th text box and the box is initialized. Its size is determined by the length and attributes of the corresponding text information. Finally, the position and size of each text box for writing each text information are determined in the background image I, which is the text layout prediction result.

[0127] In a specific implementation, the present application can use the second-level league autoencoder to perform text layout refinement processing on the available area to obtain the final text layout area, which includes:

[0128] P (k+1) =g2(Concat(A,L),P (k) ), k=0,...,K-1.

[0129] Where k is the number of iterations, g2 is the second autoencoder, A is the smoothed image area, L is the probability distribution of the available area, and P is the pixel in the poster background image.

[0130] It should be noted that for the second-level league autoencoder, its encoder can be a stacked CNN and its decoder can be a 2-layer bidirectional LSTM.

[0131] Optionally, in another embodiment of the above method of the present application, both the first autoencoder and the second autoencoder are layout predictors having a cascaded autoencoder architecture.

[0132] In a preferred embodiment, the first cascaded autoencoder for available area layout prediction mentioned in this application and the second cascaded autoencoder for text layout refinement mentioned in this application can both adopt a layout predictor with a cascaded autoencoder architecture, thereby achieving the purpose of effectively imitating the manual image editing process.

[0133] Optionally, in another embodiment of the above method of the present application, a plurality of sample images are obtained, wherein each sample image includes a text area marked with a corresponding text description field;

[0134] Extracting a sample background image from each sample image, and determining a sample smooth image region in the sample background image using a smooth region detector;

[0135] Merge the text description field, text region, sample background image, and sample smooth image region into the encoder training dataset;

[0136] The first autoencoder and the second autoencoder are independently trained using the encoder training dataset, so that the first autoencoder is used to predict the probability distribution of the text layout and the second autoencoder is used to refine the layout box.

[0137] The process of constructing an encoder training dataset in this embodiment of the present application may include the following steps:

[0138] Step 1: In one method, poster images of various categories may be collected from image material websites, etc., to form a sample image collection.

[0139] Step 2: For each sample image in the sample image collection, first apply the OCR tool to detect its text and the corresponding text box And generate the corresponding binary layout diagram based on these text boxes, expressed as

[0140] Step 3: Use the binary layout image to block the corresponding poster image text area, and use the image restoration method to fill the blocked area to obtain the sample background image corresponding to each sample image in the sample image set D.

[0141] Step 4: Applying a smooth region detector can obtain the background image of each sample accordingly The corresponding sample smoothing area maps

[0142] In summary, the encoder training dataset D proposed in this application can be expressed as:

[0143]

[0144] in, Represents a text description field, Represents a text area, Representative sample background images and Represents the sample smoothed image region.

[0145] Optionally, in another embodiment based on the above method of the present application, the second automatic encoder

[0146] It is trained using a self-supervised learning strategy, where the encoder side of the second autoencoder is constructed by stacked CNNs, and the decoder side of the second autoencoder is constructed by a 2-layer bidirectional LSTM.

[0147] Furthermore, for constructing the first autoencoder g1, it can be obtained based on the following formula:

[0148]

[0149] Furthermore, for constructing the second autoencoder g2, it can be obtained based on the following formula:

[0150]

[0151] in, The number of text boxes representing the nth poster image.

[0152] It should be noted that, in the process of training the second autoencoder, the present application may utilize a self-supervised learning strategy to train the second autoencoder.

[0153] In one way, you can to To the initial position Sampling is performed. The perturbation Δ=[0.1,0.1]T controls the variance between the initial position and the target position. Based on real layout Sampling realizes the self-supervision mechanism.

[0154] It should be noted that in the embodiment of the present application, in the process of using the encoder training data set to independently train the first autoencoder and the second autoencoder, in addition to being used in the initial model construction stage, the model will also continue to be trained when the model is subsequently updated and optimized.

[0155] That is, this step can be performed before S101 to independently train the first autoencoder and the second autoencoder using the encoder training data set. It can also be performed at S102, S103 or any other time point to optimize the two autoencoders based on the accumulated updated training data.

[0156] By applying the technical solution of this application, a cascaded autoencoder can be used to select a usable area from the smooth area of ​​the poster background image. After iteratively refining the usable area using another cascaded autoencoder, the text box within it is automatically determined and filled with text information, thereby intelligently generating a poster image. This avoids the problem faced by related technologies that rely solely on manual design to generate posters and cannot meet the large demand for high-quality posters.

[0157] Figure 3 The flowchart of a method for generating a poster according to an embodiment of the present application is schematically shown. Figure 3 As shown, the method includes:

[0158] S201: Obtain a target text layout area based on the above poster text layout method.

[0159] In related technologies, the poster generation process includes at least three steps, including:

[0160] Step 1: Select the background image of the poster;

[0161] Step 2: Lay out the text within the poster background image;

[0162] Step 3: Determine the style of the text on the poster and fill it into the background image to generate the final poster image.

[0163] Based on the above process, for step 1, in order to realize automatic search of the background image of the poster, the existing classic retrieval method generally searches for a suitable image by matching the poster content text with the background image annotation text.

[0164] However, methods in large-scale correlation techniques typically only consider unimodal features and therefore often fail to bridge the semantic gap between visual and textual modalities, resulting in highly biased retrieval results. This means that the background image selected does not match the user's current interest.

[0165] Furthermore, for step 2, conventional rule-based methods for text layout prediction typically select the text layout from a limited number of predefined layout templates, but this approach is limited in flexibility. Consequently, the background image content is often ignored when arranging the text, potentially resulting in suboptimal poster generation.

[0166] Furthermore, in order to solve the above-mentioned problem that the poster background image selection does not match the direction of the user's current interest, this application proposes a technical solution for a poster background image selection method based on a pre-trained visual text model of Text2Poster.

[0167] In summary, the present application can first obtain a large-scale pre-trained visual text model, and select a candidate background image that matches the poster description text from a large number of candidate background images based on the poster description text given by the user as the poster background image, and then iteratively arrange the text on the image through a cascaded autoencoder, and finally stylize the text through a matching-based method to synthesize the target poster.

[0168] In one approach, this application can also optimize each module of the framework through weakly supervised and self-supervised learning strategies, thereby reducing dependence on labeled data. This allows the technical solution based on the data-driven framework Text2Poster to achieve even better performance in generating posters.

[0169] Specifically, the application can first obtain a user-generated poster description text containing text information to be added to the poster. It is understandable that the poster description text is a description of the poster information that the user wants to generate. It can include multiple parameters, such as title information, scene information, implementation information, character information, etc.

[0170] S202: Based on the pre-trained visual text model, select a candidate background image that is weakly correlated with the text information from the pre-acquired candidate background image set as the poster background image.

[0171] Optionally, in a poster generation method provided in this application, it is first necessary to first select a target background poster image from multiple background poster images based on a poster background image selection method, thereby generating a corresponding target poster image. Specifically, the poster background image selection method in this application may include the following steps:

[0172] Obtaining a poster description text, wherein the poster description text includes text information to be added to the poster;

[0173] Based on a pre-trained visual text model, a candidate background image that is weakly correlated with the text information is selected from a pre-acquired set of candidate background images as the poster background image.

[0174] Optionally, the selecting, based on the pre-trained visual text model, a candidate background image that is weakly correlated with the text information from a pre-acquired set of candidate background images as the poster background image includes:

[0175] Based on the pre-trained text encoder, the pre-trained text encoder performs feature extraction on the text information to obtain text features; and

[0176] Performing feature extraction on each candidate background image based on the pre-trained image encoder to obtain image features corresponding to each candidate background image;

[0177] The weakly correlated feature similarities between the text feature and the image features corresponding to each of the candidate background images are calculated, and the candidate background image with the highest weakly correlated feature similarity to the text feature is used as the poster background image.

[0178] Optionally, the pre-trained text encoder includes: an encoder of RoBERTa-Large in a Chinese pre-trained model;

[0179] The pre-trained text encoder is used to extract features from the text information to obtain text features, including:

[0180] The text information is input into the RoBERTa-Large encoder in the Chinese pre-training model, so that the encoder extracts features from the text information and outputs corresponding text features.

[0181] Optionally, the pre-trained image encoder includes: pre-trained Faster R-CNN and EfficientNet;

[0182] The pre-trained image encoder extracts features from each candidate background image to obtain image features corresponding to each candidate background image, including:

[0183] Each of the candidate background images is input into the pre-trained image encoder so that the pre-trained Faster R-CNN performs visual object detection processing on each of the candidate background images, and then based on the EfficientNet, feature extraction is performed on each of the candidate background images after the visual object detection processing to obtain image features corresponding to each of the candidate background images.

[0184] Optionally, calculating the weakly correlated feature similarity between the text feature and the image features corresponding to each of the candidate background images, and selecting the candidate background image having the highest weakly correlated feature similarity with the text feature to be compared as the poster background image, includes:

[0185] Based on the preset weakly supervised learning method and the InfoNCE loss function constructed by the contrastive learning method CPC, the weakly correlated feature similarities between the text features and the image features corresponding to each of the candidate background images are determined.

[0186] Optionally, the poster generation method provided in this application also requires a visual text model training method for selecting a poster background image, thereby achieving the purpose of selecting a target background poster image from multiple background poster images. Specifically, the visual text model training method for selecting a poster background image in this application may include the following steps:

[0187] Acquire a plurality of weakly correlated image-text pairs, wherein the weakly correlated image-text pairs are used to represent a set of weakly correlated candidate background images and historical text information;

[0188] The preset visual text model is pre-trained and trained based on the plurality of weakly correlated image-text pairs to obtain a visual text model for selecting a poster background image that is weakly correlated with text information.

[0189] Optional, pre-trained visual text models include: BriVL, which includes a pre-trained image encoder and a pre-trained text encoder;

[0190] The pre-trained text encoder includes: the RoBERTa-Large encoder in the Chinese pre-trained model, and the pre-trained image encoder includes: the pre-trained Faster R-CNN and EfficientNet;

[0191] The RoBERTa-Large encoder is used to extract features from each of the historical text information and output corresponding text features;

[0192] The pre-trained Faster R-CNN is used to perform visual object detection processing on each of the candidate background images, and the EfficientNet is used to perform feature extraction on each of the candidate background images after the visual object detection processing to obtain image features corresponding to each of the candidate background images;

[0193] The BriVL determines the weakly correlated feature similarity between the text features and the image features corresponding to each of the candidate background images based on the InfoNCE loss function constructed by the preset weakly supervised learning method and the contrastive learning method CPC.

[0194] In one approach, in order to improve the high quality of posters, in an embodiment of the present application, when searching for background images during the poster generation process, the aim is to find images that have a weak correlation match with text information (i.e., text information contained in the poster description text and used to be added to the poster).

[0195] Specifically, a weakly correlated matching approach is a metaphorical matching approach. For example, when searching for background images based on the phrase "Bob and Alice's wedding," this application tends to find images with more metaphorical meanings, rather than images of specific wedding scenes (i.e., not strongly correlated images), such as a picture of a white church under a blue sky, which symbolizes love.

[0196] Optional, Figure 4 A schematic diagram of a method proposed by the present application for finding a background image that has a weak correlation match with text information (i.e., text information included in the poster description text and used to be added to the poster). The background image corresponding to each text information is shown.

[0197] Furthermore, in order to select a candidate background image that is weakly correlated with text information from the candidate background image set as the poster background image, the present application can use BriVL, one of the pre-trained visual text models, to select a background image from the candidate images based on text information.

[0198] Specifically, for example Figure 5 As shown, BriVL is the visual text model proposed in this application. It can be seen that it consists of an image encoder and a text encoder, which are represented by f and I (i.e., image encoder) and f T (i.e. text encoder).

[0199] Among them, the image encoder f I First, the pre-trained Faster R-CNN model is used to detect visual objects, and then the EfficientNet model is applied as its visual main framework to extract image features corresponding to each background candidate image.

[0200] In addition, the text encoder f T The encoder of the RoBERTa-Large pre-trained model for Chinese can be used as its main text framework. Based on the output of the above main framework model, BriVL stacks multiple layers of Transformers to derive text features corresponding to the text information.

[0201] It should be noted that the visual text model BriVL in this application needs to be pre-trained on multiple (for example, 30 million) weakly correlated "sample image-text pairs" collected from the Internet, so that it can meet the weak correlation artistry of the poster generation results proposed in this application.

[0202] As you can understand, this application uses a weakly supervised learning strategy and the InfoNCE loss function to align text features with image features. The loss function constructed by the contrastive predictive coding (CPC) method is InfoNCE, where NCE stands for noise contrast estimation.

[0203] In one approach, the present application can collect multiple high-quality images from image material websites and other sources as candidate background images, so that the poster background image can be selected based on the visual text model BriVL mentioned in the present application.

[0204] Specifically, since the present application extracts the image features corresponding to each candidate background image and the text features corresponding to the text information, the present application can calculate the weakly correlated feature similarity between the text features and the image features corresponding to each candidate background image, and select the candidate background image with the highest weakly correlated feature similarity to the text features as the poster background image.

[0205] Optional, Figure 5 This is a schematic diagram comparing the poster text layout method proposed in this application with other text layout methods in the prior art, wherein the corresponding target poster images under each poster text layout are shown.

[0206] In this application, a poster description text generated by a user can be obtained, which contains text information for adding to the poster; a set of candidate background images is obtained, and based on a visual text model, a candidate background image that is weakly correlated with the text information is selected from the candidate background image set as the poster background image, wherein the visual text model is trained by multiple sample images annotated with weakly correlated text pairs; after determining the text layout area in the poster background image, the text information is filled in the text layout area to obtain the target poster image.

[0207] By applying the technical solution of this application, a preset visual text model can be trained with multiple weakly correlated image-text pairs. The trained visual text model can then be used to automatically select a poster background image that is weakly correlated with the textual information the user is interested in. The final poster image is then generated based on this automatically selected poster background image. This avoids the problem faced by related technologies that rely solely on manual poster design and generation, which cannot meet the high demand for high-quality posters.

[0208] Optionally, in another embodiment of the above method of the present application, before obtaining the target text layout area, the method further includes:

[0209] Get the poster description text, which contains text information to be added to the poster;

[0210] Based on the pre-trained visual text model, a candidate background image that is weakly correlated with the text information is selected from the pre-acquired candidate background image set as the poster background image.

[0211] Still with the above Figure 6 For example, BriVL is the visual text model proposed in this application. It can be seen that it consists of an image encoder and a text encoder, which are represented by f I (i.e., image encoder) and f T (i.e. text encoder).

[0212] Among them, the image encoder f I First, the pre-trained Faster R-CNN model is used to detect visual objects, and then the EfficientNet model is applied as its visual main framework to extract image features corresponding to each background candidate image.

[0213] In addition, the text encoder f T The encoder of the RoBERTa-Large pre-trained model for Chinese can be used as its main text framework. Based on the output of the above main framework model, BriVL stacks multiple layers of Transformers to derive text features corresponding to the text information.

[0214] Specifically, in the embodiment of the present application, after obtaining the image features corresponding to each candidate background image and the text features corresponding to the text information contained in the poster description text, the multiple features can be converted into corresponding encoding vector values ​​for representation.

[0215] For example, when the encoding vector r of the text feature is obtained T' and the encoding vector of each candidate background image After that, you can use r T' =f T (Ui T i ), To express.

[0216] In one embodiment, the present application can calculate r T' With each The cosine similarity between them is calculated, and the candidate background image with the highest similarity is selected as the poster background image I.

[0217] The formula for calculating the cosine similarity between feature vectors can be:

[0218]

[0219] Optionally, in another embodiment of the method of the present application, the pre-trained text encoder includes: an encoder of RoBERTa-Large in a Chinese pre-trained model;

[0220] Based on a pre-trained text encoder, the pre-trained text encoder extracts features from text information to obtain text features, including:

[0221] The text information is input into the encoder of RoBERTa-Large in the Chinese pre-trained model, so that the encoder extracts features from the text information and outputs corresponding text features.

[0222] Optionally, in another embodiment of the above method of the present application, the pre-trained image encoder includes: pre-trained Faster R-CNN and EfficientNet;

[0223] Based on the pre-trained image encoder, feature extraction is performed on each candidate background image to obtain the image features corresponding to each candidate background image, including:

[0224] Each candidate background image is input into a pre-trained image encoder so that the pre-trained Faster R-CNN performs visual object detection on each candidate background image. Then, based on EfficientNet, feature extraction is performed on each candidate background image after visual object detection to obtain image features corresponding to each candidate background image.

[0225] Optionally, in another embodiment of the method of the present application, calculating the weakly correlated feature similarity between the text feature and the image features corresponding to each candidate background image, and selecting the candidate background image having the highest weakly correlated feature similarity with the text feature to be compared as the poster background image, includes:

[0226] Based on the preset weakly supervised learning method and the InfoNCE loss function constructed by the contrastive learning method CPC, the weakly correlated feature similarity between the text features and the image features corresponding to each candidate background image is determined.

[0227] It should be noted that the visual text model BriVL in this application needs to be pre-trained on multiple (for example, 30 million) weakly correlated "sample image-text pairs" collected from the Internet, so that it can meet the weak correlation artistry of the poster generation results proposed in this application.

[0228] like Figure 7 As shown, this is the overall process architecture diagram of the poster generation method proposed in this application. In one way, in the process of using the image features and text features of the candidate background image for weak correlation matching in this application, the candidate background image with the highest similarity to the weakly correlated features of the text features to be compared can be used as the poster background image. Specifically, this application needs to apply the weakly supervised learning strategy and the InfoNCE loss function to perform feature alignment operations on the text features and the image features of each candidate background image. And after the feature alignment, the two are then subjected to correlation matching.

[0229] Among them, for the InfoNCE loss function, the present application can use the loss function constructed by the contrastive learning method CPC (Contrastive Predictive Coding) as the InfoNCE, where NCE refers to noise contrast estimation (Noise Contrastive Estimation).

[0230] Optionally, in another embodiment of the method of the present application, filling text information in a text layout area to generate a poster corresponding to the text information includes:

[0231] Extract text features of text information and detect the background color of the text layout area;

[0232] Based on the background color of the text layout area, a target text style that matches the text features and the background color is selected from a preset text style database;

[0233] The text information is converted according to the target text style and then filled into the text layout area to obtain the target poster corresponding to the text information.

[0234] Furthermore, the present application can be used to describe each text information T in the poster description text. i ∈T', it is necessary to extract the corresponding text feature r=f T (T), and get the background color of the text feature as c I =I(p). Where pi It is a text message T i The corresponding text layout area.

[0235] In addition, based on (r,c T ), the target text style that matches the text feature and background color can be searched from the preset text style database F under the cosine similarity, thereby determining each T i and fill it into the corresponding text layout area.

[0236] A) Each convolutional layer used in g1 contains 16 convolution kernels of size 9×9; the encoder of g1 ultimately outputs a 64-dimensional feature vector.

[0237] B) Each convolutional layer of g2’s encoder contains 64 convolutional kernels of size 5×5; the dimension of the hidden layer of g2’s 2-layer bidirectional LSTM (decoder) is set to 200.

[0238] C) When training the two autoencoders, you can split the training and validation sets. For example, you can split dataset D into 138,013 training poster images and 16,000 validation poster images. Each poster image can be resized to 300×400. The Adam algorithm can be used to optimize the autoencoders g1 and g2, with a learning rate of 0.05 and a batch size of 512. On four V100 GPUs, the two autoencoders are trained for 4 hours (g1) and 48 hours (g2).

[0239] Finally, to demonstrate the effectiveness of the Text2Poster framework proposed in this application, the following section also includes corresponding verifications, which are detailed as follows:

[0240] 1) Verification for image retrieval:

[0241] First, we verify the rationality of (i) the image retrieval method. For comparison purposes, in addition to the BriVL method provided by this application, this application also references two search methods as control groups: (a) using the unsplash.com search engine and (b) matching input text with image tags in this application's image retrieval library.

[0242] Among them, this application shows representative retrieval results obtained by three different search methods. Compared with the other two methods, the images retrieved by the BriVL method provided by this application do contain metaphors corresponding to the input text. For example, for the given text "Campus Charity Sale", the search engine of unsplash.com is more inclined to look for images with clear concepts, such as "sale" and "campus", while the BriVL method provided by this application retrieves images with growing trees, rainbows and colorful handprints, whose content is metaphorical but suitable for the semantics of "Campus Charity Sale". Even for challenging abstract descriptions such as "See the world together" and "Dreams never stop", the BriVL method provided by this application can still find suitable images.

[0243] In the subjective evaluation, given 50 text queries, this application retrieved the top 5 images for each query using different methods. This application invited three volunteers to rate the quality of the retrieved images from 0 (very poor) to 4 (very good). The mean and standard deviation of the score of Unsplash.com's search engine was 2.17±0.10, the mean and standard deviation of the score of the tag-based matching method was 1.64±0.16, and the mean and standard deviation of the score of the BriVL method provided by this application was 2.38±0.13, which further demonstrated the superiority of the method of this application.

[0244] 2) Verification of text layout prediction:

[0245] We quantitatively and qualitatively evaluate the proposed layout predictor and compare it with the following control groups:

[0246] A) The most advanced learning-based method, LayoutGAN++;

[0247] B) the most advanced rule-based methods IUI and DeSal;

[0248] C)LUBAN, a commercial poster generator at https: / / luban.aliyun.com.

[0249] In addition, in order to demonstrate the usefulness of the iterative layout optimization strategy of the present application, for the layout predictor of the present application, the present application sets K to 1, 5 and 30 respectively.

[0250] This application collects 16,000 posters from huaban.com to construct a reference dataset, and prepares three background image sets: Unsplash2K, Unsplash10K, and PSD1.6K. Unsplash2K and Unsplash10K contain 2,000 and 10,000 background images from unsplash.com, respectively. PSD1.6K contains 1,637 background images extracted from poster files in PSD format. For each image set, this application arranges the input text on the background image and generates a poster using various methods. This application calculates the initial distance FID between the poster and the reference dataset according to the work in LayoutGAN++. The results in Table 1 show that the method of this application consistently outperforms the above-mentioned control group, and its performance improves with the increase of K, which verifies the rationality of the iterative refinement strategy of this application.

[0251] Table 1

[0252]

[0253] Table 1 shows the objective and subjective evaluation of various layout prediction methods. LUBAN in C) only provides paid services and cannot be used for large-scale quantitative evaluation.

[0254] In addition, this application manually selected 50 text sets, each of which contains a title and several subtitles or descriptions.

[0255] For each layout prediction method, we first retrieved five background images for each set of text and generated 250 posters accordingly. We asked three volunteers to rate the layout aesthetics of these generated posters from 0 (very poor) to 4 (very good). The mean and standard deviation of the scores for each method are shown in Table 1.

[0256] Optionally, in another embodiment of the present application, Figure 8 As shown, the present application also provides a poster generation device. It includes:

[0257] Determining module 301, determining a smooth area in the poster background image corresponding to the poster description text, and selecting a usable area in the smooth area using a first automatic encoder;

[0258] The generation module 302 samples the available area and generates a text box corresponding to the available area based on the corresponding sampling results and the text information corresponding to the poster description text for adding to the poster, so as to obtain a target text layout area in the poster background image for writing the text information.

[0259] In the present application, a smooth area in the poster background image corresponding to the poster description text can be determined, and a first autoencoder can be used to select an available area in the smooth area; the available area is sampled, and a text box corresponding to the available area is generated based on the corresponding sampling results and the text information corresponding to the poster description text for adding to the poster, so as to obtain a target text layout area in the poster background image for writing text information. By applying the technical solution of the present application, a cascaded autoencoder can be used to select an available area from the smooth area in the poster background image, and then the text box is automatically determined and the text information is filled in after the available area is iteratively refined according to another cascaded autoencoder, thereby intelligently generating a poster image. This avoids the problem faced by the related art of relying solely on manual design to generate posters and failing to meet the large demand for high-quality posters.

[0260] In another embodiment of the present application, the determining module 301 is configured to perform the following steps:

[0261] Selecting a plurality of candidate frames in the poster background image; and generating a salient image corresponding to the poster background image using a spectral residual algorithm, wherein there are overlapping candidate frames among the candidate frames;

[0262] Determining a candidate value corresponding to each candidate frame based on the salient average value of the salient image, the number of pixels of the candidate frame, and a preset offset;

[0263] A candidate frame whose candidate value is lower than a preset candidate threshold is selected as a target candidate frame, and a smooth area in the poster background image is determined based on the target candidate frame, wherein the positions of each target candidate frame in the poster background image do not overlap.

[0264] In another embodiment of the present application, the determining module 301 is configured to perform the following steps:

[0265] Based on the non-maximum suppression method, a candidate frame whose candidate value is lower than a preset candidate threshold is selected as a target candidate frame;

[0266] Determine candidate frame areas in the poster background image where all target candidate frames are located;

[0267] The candidate frame area in the poster background image is converted into a binary image, and the converted binary image area is used as a smooth area in the poster background image.

[0268] In another embodiment of the present application, the determining module 301 is configured to perform the following steps:

[0269] Using each object candidate box corresponding to the smooth area as an encoder end input of the first autoencoder to obtain an encoder output result, wherein the encoder end is constructed by stacking CNNs;

[0270] After connecting the encoder output result with the position embedding map, it is used as the input of the decoder side of the first autoencoder to obtain the available area in the smooth area as the initial text layout result, wherein the decoder side is constructed by stacking Transposed-CNN.

[0271] In another embodiment of the present application, the generation module 302 is configured to perform the following steps:

[0272] The second autoencoder is used to perform refinement layout processing on the text boxes corresponding to each of the available areas in an autoregressive manner to obtain the target text boxes corresponding to each of the available areas, so as to form a target text layout area in the poster background image for writing the text information.

[0273] In another embodiment of the present application, the determining module 301 is configured to perform the following steps:

[0274] The first autoencoder and the second autoencoder are both layout predictors with a cascaded autoencoder architecture.

[0275] In another embodiment of the present application, the generation module 302 is configured to perform the following steps:

[0276] Acquire multiple sample images, wherein each sample image includes a text area marked with a corresponding text description field;

[0277] Extracting a sample background image from each sample image, and determining a sample smooth image region in the sample background image using a smooth region detector;

[0278] Merging the text description field, the text area, the sample background image, and the sample smoothed image area into an encoder training dataset;

[0279] The first autoencoder and the second autoencoder are independently trained using the encoder training dataset, so that the first autoencoder is used to predict the probability distribution of text layout and the second autoencoder is used to refine the layout box.

[0280] In another embodiment of the present application, the generation module 302 is configured to perform the following steps:

[0281] The method is trained using a self-supervised learning strategy, wherein the encoder end of the second autoencoder is constructed by stacked CNNs, and the decoder end of the second autoencoder is constructed by a 2-layer bidirectional LSTM.

[0282] Figure 9 4 is a block diagram of a logical structure of an electronic device according to an exemplary embodiment. For example, the electronic device 400 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0283] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory including instructions, and the above instructions can be executed by an electronic device processor to complete the above poster generation method, the method comprising: determining a smooth area in the poster background image corresponding to the poster description text, and selecting an available area in the smooth area using a first automatic encoder; sampling the available area, and generating a text box corresponding to the available area based on the corresponding sampling results and the text information corresponding to the poster description text for adding to the poster, so as to obtain a target text layout area in the poster background image for writing the text information. Optionally, the above instructions can also be executed by the processor of the electronic device to complete the other steps involved in the above exemplary embodiment. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0284] In an exemplary embodiment, an application / computer program product is also provided, comprising one or more instructions that can be executed by a processor of an electronic device to perform the aforementioned poster generation method. The method comprises: determining a smooth region within a poster background image corresponding to the poster description text, and selecting a usable region within the smooth region using a first autoencoder; sampling the usable region, and generating a text box corresponding to the usable region based on the sampling result and the textual information to be added to the poster corresponding to the poster description text, thereby obtaining a target text layout region within the poster background image for writing the textual information. Optionally, the aforementioned instructions can also be executed by a processor of an electronic device to perform the other steps involved in the aforementioned exemplary embodiment.

[0285] Figure 9 FIG is an example diagram of an electronic device 400. A person skilled in the art will understand that Figure 9It is only an example of the electronic device 400 and does not constitute a limitation of the electronic device 400. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the electronic device 400 may also include input and output devices, network access devices, buses, etc.

[0286] The processor 402 may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor, or the processor 402 may be any conventional processor. The processor 402 is the control center of the electronic device 400 and connects various parts of the entire electronic device 400 using various interfaces and lines.

[0287] Memory 401 can be used to store computer-readable instructions 403. Processor 402 implements various functions of electronic device 400 by running or executing computer-readable instructions or modules stored in memory 401 and accessing data stored in memory 401. Memory 401 may primarily include a program storage area and a data storage area. The program storage area may store an operating system and at least one application required for a function (such as a sound playback function or an image playback function); the data storage area may store data generated based on the use of electronic device 400. Furthermore, memory 401 may include a hard disk, internal memory, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, at least one disk storage device, a flash memory device, a read-only memory (ROM), a random access memory (RAM), or other non-volatile / volatile storage devices.

[0288] If the modules integrated into electronic device 400 are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the processes in the above-mentioned method embodiments, and can also use computer-readable instructions to instruct related hardware to complete the above-mentioned computer-readable instructions. The computer-readable instructions can be stored in a computer-readable storage medium, and when executed by a processor, the computer-readable instructions can implement the steps of each of the above-mentioned method embodiments.

[0289] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered as exemplary only, and the true scope and spirit of the present application are indicated by the following claims.

[0290] It should be understood that the present application is not limited to the exact structures described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A poster text layout method, characterized in that: include: Determine a smooth region in the poster background image corresponding to the poster description text, and select a usable region in the smooth region using a first autoencoder; Sampling the available area, and generating a text box corresponding to the available area according to the sampling result and the text information corresponding to the poster description text for adding to the poster, so as to obtain a target text layout area for writing the text information in the poster background image; The second autoencoder is used to perform refinement layout processing on the text boxes corresponding to each of the available areas in an autoregressive manner to obtain the target text boxes corresponding to each of the available areas, so as to form a target text layout area in the poster background image for writing the text information.

2. The poster text layout method according to claim 1, wherein: The determining of the smooth area in the poster background image corresponding to the poster description text includes: Selecting a plurality of candidate frames in the poster background image; and generating a salient image corresponding to the poster background image using a spectral residual algorithm, wherein there are overlapping candidate frames among the candidate frames; Determining a candidate value corresponding to each candidate frame based on the salient average value of the salient image, the number of pixels of the candidate frame, and a preset offset; A candidate frame whose candidate value is lower than a preset candidate threshold is selected as a target candidate frame, and a smooth area in the poster background image is determined based on the target candidate frame, wherein the positions of each target candidate frame in the poster background image do not overlap.

3. The poster text layout method according to claim 2, wherein: The selecting a candidate frame whose candidate value is lower than a preset candidate threshold as a target candidate frame, and determining a smooth area in the poster background image based on the target candidate frame, includes: Based on the non-maximum suppression method, a candidate frame whose candidate value is lower than a preset candidate threshold is selected as a target candidate frame; Determine candidate frame areas in the poster background image where all target candidate frames are located; The candidate frame area in the poster background image is converted into a binary image, and the converted binary image area is used as a smooth area in the poster background image.

4. The poster text layout method according to claim 1, wherein: The selecting of the available area in the smooth area by using the first automatic encoder comprises: Using each object candidate box corresponding to the smooth area as an encoder end input of the first autoencoder to obtain an encoder output result, wherein the encoder end is constructed by stacking CNNs; After connecting the encoder output result with the position embedding map, it is used as the input of the decoder side of the first autoencoder to obtain the available area in the smooth area as the initial text layout result, wherein the decoder side is constructed by stacking Transposed-CNN.

5. The poster text layout method according to claim 1, wherein: The first autoencoder and the second autoencoder are both layout predictors with a cascaded autoencoder architecture.

6. The poster text layout method according to claim 5, wherein: Also includes: Acquire multiple sample images, wherein each sample image includes a text area marked with a corresponding text description field; Extracting a sample background image from each sample image, and determining a sample smooth image region in the sample background image using a smooth region detector; Merging the text description field, the text area, the sample background image, and the sample smoothed image area into an encoder training dataset; The first autoencoder and the second autoencoder are independently trained using the encoder training dataset, so that the first autoencoder is used to predict the probability distribution of text layout and the second autoencoder is used to refine the layout box.

7. The poster text layout method according to any one of claims 1, 5 and 6, characterized in that: The second autoencoder further comprises: The method is trained using a self-supervised learning strategy, wherein the encoder end of the second autoencoder is constructed by stacked CNNs, and the decoder end of the second autoencoder is constructed by a 2-layer bidirectional LSTM.

8. A poster generation method, characterized in that: include: Acquire a target text layout area based on the poster text layout method according to any one of claims 1 to 7; Fill the text information in the text layout area to generate a target poster corresponding to the text information.

9. The poster generation method according to claim 8, wherein: Before obtaining the target text layout area, the method further includes: Obtaining a poster description text, wherein the poster description text includes text information to be added to the poster; Based on a pre-trained visual text model, a candidate background image that is weakly correlated with the text information is selected from a pre-acquired set of candidate background images as the poster background image.

10. The poster generation method according to claim 8, wherein: Filling the text information in the text layout area to generate a poster corresponding to the text information includes: Extracting text features of the text information and detecting the background color of the text layout area; Based on the background color of the text layout area, selecting a target text style that matches the text feature and the background color from a preset text style database; The text information is converted according to the target text style and then filled into the text layout area to obtain a target poster corresponding to the text information.

11. A poster text layout device, characterized in that: include: a determination module for determining a smooth region in the poster background image corresponding to the poster description text, and selecting a usable region in the smooth region using a first autoencoder; a generating module for sampling the available area and generating a text box corresponding to the available area based on the corresponding sampling result and the text information corresponding to the poster description text for adding to the poster, so as to obtain a target text layout area in the poster background image for writing the text information; Furthermore, a second autoencoder is used to perform refinement layout processing on the text boxes corresponding to each of the available areas in an autoregressive manner to obtain a target text box corresponding to each of the available areas, so as to form a target text layout area in the poster background image for writing the text information.

12. An electronic device, characterized in that: include: a memory for storing executable instructions; as well as, A processor is configured to execute the executable instructions together with the memory to complete the operation of the poster text layout method described in any one of claims 1 to 7, or to complete the operation of the poster generation method described in any one of claims 8 to 10.

13. A computer-readable storage medium for storing computer-readable instructions, characterized in that: When the instruction is executed, the operation of the poster text layout method described in any one of claims 1-7 is performed, or the operation of the poster generation method described in any one of claims 8-10 is completed.

Images