Poster automatic generation method based on multi-dimensional feature extraction
By employing multi-dimensional feature extraction and template matching methods, the limitations of existing poster generation technologies in terms of application scenarios and personalization capabilities are addressed, enabling the rapid generation of personalized posters.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-11-29
- Publication Date
- 2026-03-24
AI Technical Summary
Existing poster generation technologies require simultaneous input of text and images, limiting their application scenarios and resulting in posters with low personalization capabilities, failing to meet the real-time needs of social media dissemination and museum visits.
The target subject of the cultural relic image is obtained by edge detection and background segmentation. Multi-dimensional visual feature encoding is performed to generate poster subtitles. The K-Means feature clustering algorithm is used to retrieve matching templates from a preset template library. The poster is then generated by editing the attention center region.
Posters can be generated from a single image, saving manpower and making them suitable for cultural dissemination and cultural relic promotion, while also enhancing the personalization and real-time nature of poster generation.
Smart Images
Figure CN115861485B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of data processing technology, and in particular to a method for automatically generating posters based on multi-dimensional feature extraction. Background Technology
[0002] Posters are an art form of information dissemination and a popular promotional tool. Automatic poster generation technology can automatically generate stylized themed posters based on input image or text data through retrieval or generation. Existing image-text-based poster generation technologies require simultaneous input of target images and text, ultimately generating aesthetically pleasing posters through layout optimization and style enhancement. This approach requires multimodal data input. In the field of cultural relics, adding text to posters for cultural relics requires a relatively specialized knowledge background, which significantly limits the application of image-text poster generation solutions. Furthermore, due to the richness of the input data, the personalized capabilities of the generated posters are relatively low. For poster generation technologies using text-based image search, the backgrounds generated are limited by the background library, resulting in high repetition and making it difficult to generate personalized and stylized posters. For solutions using generative adversarial networks to generate posters, the quality of background images generated based on input text is significantly lower than that of existing aesthetically pleasing posters.
[0003] Traditional poster generation methods require simultaneous input of text and poster elements, limiting their application scenarios. Furthermore, deep learning-based layout optimization algorithms or text-based automatic poster background generation algorithms incur significant time overhead, failing to meet the real-time demands of social media dissemination and museum visits. Additionally, layout optimization algorithms require creators to provide a large amount of poster elements and data beforehand, and backgrounds generated by automatic background generation algorithms differ greatly from manually created poster backgrounds. These factors render existing algorithms unsuitable for social poster applications in museum and social media settings. Summary of the Invention
[0004] Therefore, it is necessary to provide a poster automatic generation method based on multi-dimensional feature extraction to address the aforementioned technical problems.
[0005] A method for automatically generating posters based on multi-dimensional feature extraction, the method comprising:
[0006] The image of the cultural relic is acquired, and edge detection and background segmentation are performed on the image to obtain the target subject, and a transparent image of the target subject is constructed.
[0007] The target image is encoded with visual features in multiple dimensions to obtain shallow multi-dimensional visual features and deep multi-dimensional visual features. The shallow multi-dimensional visual features include visual features of color dimension, contour dimension and texture dimension, and the deep multi-dimensional visual features include visual features of cultural semantic dimension and craft technique dimension.
[0008] Based on the shallow multidimensional visual features, the deep multidimensional visual features, and the multimodal subtitle probability distribution map, poster subtitles are predicted and output. The multimodal subtitle probability distribution map is used to guide image subtitle generation. The multimodal subtitle probability distribution map is obtained by encoding shallow and deep multidimensional visual features of multiple training artifact images, then decoding the obtained shallow and deep multidimensional visual features, using the obtained visual feature vector as input and the text feature vector as output, and training in a supervised learning manner.
[0009] The VGG network with the middle layer removed is used to extract the attention center of the target subject image, and the attention center area is image edited to obtain the poster background texture.
[0010] Based on the attention center, the shallow multi-dimensional visual features, and the deep multi-dimensional visual features, the K-Means feature clustering algorithm is used to search in a preset poster template library to obtain a template that matches the attention center.
[0011] A cultural relic poster is generated based on the target subject, the poster subtitles, and a template that matches the center of attention.
[0012] In one embodiment, the method further includes constructing an author watermark based on the creator's personal information, configuring it in a suitable position on the cultural relic poster, and generating the final cultural relic poster.
[0013] In one embodiment, an image of a cultural relic is acquired, and edge detection and background segmentation are performed on the image to obtain the target subject. A transparent image of the target subject is then constructed, including:
[0014] Obtain images of cultural relics.
[0015] The U2-Net network is used to perform edge detection and background segmentation on the cultural relic image to obtain the target subject; the U2-Net network includes a codec with a 6-reSidual U-block architecture, a 1×1 convolutional layer and a sigmoid activation function.
[0016] The target subject is superimposed on the image of the cultural relic to obtain the complement of the intersection of the target subject and the image of the cultural relic.
[0017] The third channel parameter of the corresponding region of the complement is set to 0 to obtain the target subject image with a transparent background.
[0018] In one embodiment, the target subject image is encoded with visual features in multiple dimensions to obtain shallow multi-dimensional visual features and deep multi-dimensional visual features, including:
[0019] A shallow multidimensional visual feature encoder based on a convolutional neural network is used to encode visual features in the color dimension, contour dimension and texture dimension to obtain shallow multidimensional visual features; the shallow multidimensional visual feature encoder includes the heads of three parallel VGG16 networks and two fully connected layers.
[0020] A deep multidimensional feature encoder based on a convolutional neural network is used to encode the visual features of the target subject image in the dimensions of cultural semantics and craft techniques to obtain deep multidimensional visual features. The deep multidimensional visual feature encoder includes the heads of two parallel VGG16 networks and two fully connected layers.
[0021] In one embodiment, the multimodal caption probability distribution map includes: a shallow multimodal caption probability distribution map and a deep multimodal caption probability distribution map.
[0022] Based on the shallow multidimensional visual features, the deep multidimensional visual features, and the multimodal subtitle probability distribution map, the poster subtitles are predicted and output. The step of generating the multimodal subtitle probability distribution map includes:
[0023] Obtain multiple training images of cultural relics and the corresponding text data for each training image.
[0024] The multi-dimensional features of the training artifact images are encoded to obtain shallow multi-dimensional visual features and deep multi-dimensional visual features of the training artifact images.
[0025] Based on the shallow multi-dimensional visual features, the deep multi-dimensional visual features, and the text data, a multi-dimensional semantic feature encoder based on an LSTM network is used to extract multi-dimensional semantic features and encode them to obtain shallow multi-dimensional semantic features and deep multi-dimensional semantic features. The shallow multi-dimensional semantic features include semantic features of color dimension, contour dimension, and texture dimension. The deep multi-dimensional semantic features include semantic features of cultural semantic dimension and craft technique dimension.
[0026] After fusing the shallow multidimensional visual features and the shallow multidimensional semantic features, a shallow multidimensional decoder is used for decoding to obtain a shallow multimodal subtitle probability distribution map corresponding to the shallow semantic description; the shallow multidimensional decoder includes two fully connected layers and a Softmax activation function.
[0027] After fusing the deep multidimensional visual features and the deep multidimensional semantic features, a deep multidimensional decoder is used for decoding to obtain a deep multimodal subtitle probability distribution map corresponding to the deep semantic description; the deep multidimensional decoder has the same structure as the shallow multidimensional decoder.
[0028] In one embodiment, the multidimensional semantic feature encoder includes a shallow multidimensional semantic feature encoder and a deep multidimensional semantic feature encoder with the same structure. The shallow multidimensional semantic feature encoder includes a word embedding layer, a nonlinear transformation layer, an LSTM network, and a fully connected layer.
[0029] Based on the shallow multi-dimensional visual features, the deep multi-dimensional visual features, and the text data, a multi-dimensional semantic feature encoder based on an LSTM network is used to extract multi-dimensional semantic features and encode them to obtain shallow multi-dimensional semantic features and deep multi-dimensional semantic features, including:
[0030] The text data is segmented, and the text corresponding to the shallow multi-dimensional visual features is extracted and the data is cleaned.
[0031] After data cleaning, the results are input into the word embedding layer for encoding and then processed by a nonlinear transformation layer to obtain word vectors. The resulting LSTM network is then input into a fully connected layer to obtain shallow multi-dimensional semantic feature encoding.
[0032] The text data is segmented, and the text corresponding to the deep multi-dimensional visual features is extracted and the data is cleaned.
[0033] After data cleaning, the results are input into the word embedding layer for encoding and then processed by a nonlinear transformation layer to obtain word vectors in an LSTM network. The results are then input into a fully connected layer to obtain deep multi-dimensional semantic feature encoding of the cultural relic image.
[0034] In one embodiment, the multimodal caption probability distribution map includes: a shallow multimodal caption probability distribution map and a deep multimodal caption probability distribution map.
[0035] Based on the shallow multi-dimensional visual features, the deep multi-dimensional visual features, and the multimodal subtitle probability distribution map, poster subtitles are predicted and output, including:
[0036] Based on the shallow multidimensional visual features and the shallow multimodal caption probability distribution map, a shallow semantic description is predicted and output.
[0037] Based on the deep multidimensional visual features and the deep multimodal caption probability distribution map, predict and output deep semantic descriptions;
[0038] By fusing the shallow semantic description and the deep semantic description, a textual description of the cultural relic image is obtained, which becomes the poster subtitle.
[0039] In one embodiment, based on the attention center, the shallow multi-dimensional visual features, and the deep multi-dimensional visual features, a K-Means feature clustering algorithm is used to search a preset poster template library to obtain templates that match the attention center. The preset poster template library is constructed based on a dataset of cultural relic photographs. The steps for constructing the preset poster template library include:
[0040] Five types of poster template layouts were designed based on the style and type of cultural relics; each type of poster template layout includes eight specific layout parameters.
[0041] Seven color schemes were designed based on the color types of the cultural relics; among them, there are 12 color combinations.
[0042] Forty font styles were designed based on the shapes and types of cultural relics; the font styles include font and font size.
[0043] In one embodiment, based on the attention center, the shallow multi-dimensional visual features, and the deep multi-dimensional visual features, a K-Means feature clustering algorithm is used to search a preset poster template library to obtain a template that matches the attention center, including:
[0044] Feature tuple pairs are constructed based on a preset poster template library. The feature tuple pairs are as follows:
[0045] {H,C(I),S(I)}
[0046] Where H is the coordinate of the attention center, C(I) is the color feature vector, and S(I) is the feature composed of shallow multi-dimensional visual features and deep multi-dimensional data features.
[0047] A center label is constructed based on the relative position of the attention center.
[0048] Based on the central label and the feature tuple pair, the K-Means clustering algorithm is used to obtain the poster template layout that matches the central label.
[0049] Based on the feature tuple pairs and the color dimension features and blending features of the cultural relic image, the color scheme and font style of the template are determined using the same method as that used to determine the poster template layout; the blending features are features resulting from the combination of shallow multi-dimensional visual features and deep multi-dimensional data features.
[0050] Based on the poster template layout, color scheme, and font style, a template matching the center of attention is obtained.
[0051] The aforementioned poster automatic generation method based on multi-dimensional feature extraction acquires images of cultural relics and employs edge detection, background segmentation, and multi-dimensional visual feature encoding techniques to obtain shallow and deep multi-dimensional visual feature codes for the target subject of the cultural relic image. Based on these shallow and deep multi-dimensional visual feature codes and a multimodal subtitle probability distribution map, poster subtitles are generated. A VGG network with its intermediate layers removed is used to extract the attention center of the target subject image, and image editing is performed on the attention center region to obtain the poster background texture. Based on the attention center of the target subject image, a K-means feature clustering algorithm is used to retrieve templates matching the target subject's attention center from a pre-set poster template library. Finally, a poster of the cultural relic is generated based on the target subject, poster subtitles, and the template matching the target subject's attention center. This method requires only one image to generate a poster, greatly saving manpower, and is of great significance for cultural dissemination, value transmission, and cultural relic promotion. Attached Figure Description
[0052] Figure 1 This is a flowchart illustrating an automatic poster generation method based on multi-dimensional feature extraction in one embodiment.
[0053] Figure 2 This is a data flow diagram of an automatic poster generation method based on multi-dimensional feature extraction in another embodiment;
[0054] Figure 3 This is a network structure diagram of a shallow multidimensional visual feature encoder in another embodiment;
[0055] Figure 4 Here is a network structure diagram of a deep multidimensional visual feature encoder in another embodiment;
[0056] Figure 5 A data flow graph generated for a multimodal caption probability distribution map in another embodiment;
[0057] Figure 6 This is a network structure diagram of a shallow multidimensional semantic feature encoder in another embodiment. Detailed Implementation
[0058] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.
[0059] In one embodiment, such as Figure 1 , Figure 2 As shown, an automatic poster generation method based on multi-dimensional feature extraction is provided. This method includes the following steps:
[0060] Step 100: Obtain the image of the cultural relic, perform edge detection and background segmentation on the image of the cultural relic to obtain the target subject, and construct a transparent image of the target subject.
[0061] Specifically, the image of the cultural relic can be a photograph containing the cultural relic.
[0062] The target subject is the cultural relic for which poster images are to be produced.
[0063] The artifact in the photo is the main element needed to generate the poster. Edge detection and background segmentation can be performed using VGG, ResNet, or U2-Net networks based on convolutional neural networks.
[0064] Step 102: Encode the visual features of the target image in multiple dimensions to obtain shallow multi-dimensional visual features and deep multi-dimensional visual features.
[0065] Shallow multidimensional visual features include visual features in the dimensions of color, outline, and texture.
[0066] Deep, multi-dimensional visual features include visual features in terms of cultural semantics and craft techniques.
[0067] Specifically, for image visual feature encoding, the input image is first preprocessed, mainly including contour cropping and regularization. Next, convolution and pooling activation operations are performed on the preprocessed 224*224 three-channel image to obtain a 4096-element one-dimensional vector as a representation of the image's shallow visual features. Then, this shallow visual representation is fed into a fully connected layer for symmetric gradient dimensionality reduction, ultimately generating 256-element image visual features.
[0068] The shallow multidimensional visual feature encoder is used to extract the visual features of cultural relics images in three dimensions: color, texture, and contour, using three parallel feature extraction branches (composed of several convolution and pooling activation operations). The three-dimensional features are then encoded using a fully connected layer to obtain the shallow multidimensional visual feature encoding.
[0069] The deep multidimensional visual feature encoder uses two parallel feature extraction branches (composed of several convolution and pooling activation operations) to extract the visual features of cultural semantics and craftsmanship techniques from the images of cultural relics. It then uses a fully connected layer to encode the features of the two dimensions, thus obtaining the deep multidimensional visual feature encoding.
[0070] Step 104: Based on shallow multi-dimensional visual features, deep multi-dimensional visual features, and multimodal subtitle probability distribution map, predict and output poster subtitles.
[0071] The multimodal caption probability distribution map is used to guide image caption generation. The multimodal caption probability distribution map is obtained by encoding shallow and deep multidimensional visual features of multiple training images of cultural relics, then decoding the obtained shallow and deep multidimensional visual features, using the obtained visual feature vector as input and the text feature vector as output, and training in a supervised learning manner.
[0072] Specifically, shallow and deep multidimensional visual feature encodings were performed on multiple training images of cultural relics.
[0073] The process of obtaining shallow and deep multidimensional visual feature encoding of the image during the multimodal caption probability distribution map generation is the same as the process of multidimensional visual feature encoding in step 102.
[0074] Text feature vectors are vectors obtained by embedding the text information corresponding to shallow multi-dimensional visual features and deep multi-dimensional visual features.
[0075] Step 106: Use the VGG network with the middle layer removed to extract the attention center of the target subject image, and perform image editing on the attention center area to obtain the poster background texture.
[0076] Specifically, the output vector after stripping the intermediate layer of VGG (preferably the 13th layer) is regularized to obtain the peak point of the model's attention, and this peak point is used as the equivalent center of the image to generate background textures and match poster templates.
[0077] Image editing techniques, such as magnification and blurring, are applied to the central area of attention and used as the background texture for the poster.
[0078] Step 108: Based on the attention center, shallow multi-dimensional visual features, and deep multi-dimensional visual features, the K-Means feature clustering algorithm is used to search in the preset poster template library to obtain templates that match the attention center.
[0079] Step 110: Generate a cultural relic poster based on the target subject, poster text, and a template that matches the center of attention.
[0080] The aforementioned poster automatic generation method based on multi-dimensional feature extraction acquires images of cultural relics and employs edge detection, background segmentation, and multi-dimensional visual feature encoding techniques to obtain shallow and deep multi-dimensional visual feature codes for the target subject of the cultural relic image. Based on these shallow and deep multi-dimensional visual feature codes and a multimodal subtitle probability distribution map, poster subtitles are generated. A VGG network with its intermediate layers removed is used to extract the attention center of the target subject image, and image editing is performed on the attention center region to obtain the poster background texture. Based on the attention center of the target subject image, a K-means feature clustering algorithm is used to retrieve templates matching the target subject's attention center from a pre-set poster template library. Finally, a poster of the cultural relic is generated based on the target subject, poster subtitles, and the template matching the target subject's attention center. This method requires only one image to generate a poster, greatly saving manpower, and is of great significance for cultural dissemination, value transmission, and cultural relic promotion.
[0081] In one embodiment, the method further includes constructing an author watermark based on the creator's personal information, configuring it in an appropriate position on the cultural relic poster, and generating the final cultural relic poster.
[0082] In one embodiment, step 100 includes: acquiring an image of a cultural relic; performing edge detection and background segmentation on the image of the cultural relic using a U2-Net network to obtain a target subject; the U2-Net network includes a codec with a 6-reSidual U-block architecture, a 1×1 convolutional layer, and a sigmoid activation function; superimposing the target subject with the image of the cultural relic to obtain the complement of the intersection of the target subject and the image of the cultural relic; setting the third channel parameter of the region corresponding to the complement to 0 to obtain a target subject image with a transparent background.
[0083] Specifically, existing image segmentation models based on VGG and ResNet were originally designed for image classification tasks. They all extract image features in a semantic-based manner, and these networks require pre-training on large-scale image datasets, such as ImageNet. When the distribution of the target data differs significantly from that of the ImageNet dataset, the pre-trained models are inefficient and unreliable. Therefore, U2-Net is used for edge detection and background segmentation.
[0084] U2-Net does not use any pre-trained backbone network from image classification, allowing it to be trained from scratch and achieve high-resolution object segmentation with low memory and computational costs. For the input image Iin, U2-Net obtains the outputs of each layer through a six-layer ReSidual U-block encoder-decoder architecture. The feature maps of the six layers are concatenated, and finally, a 1×1 convolutional layer and a sigmoid activation function are used to obtain the output prediction Iout, thus achieving the experimental results of object segmentation. Next, the segmentation result is superimposed with the original image to obtain the complement Icut = Cu[Iout∩Iin], and the third channel parameter of the region corresponding to Icut is set to 0. This yields a noise-removed image of the cultural relic, which will serve as the input for the target image elements and feature encoding in the poster.
[0085] In one embodiment, step 102 includes: using a shallow multi-dimensional visual feature encoder based on a convolutional neural network to encode visual features in the color dimension, contour dimension, and texture dimension to obtain shallow multi-dimensional visual features; such as Figure 3 As shown, the shallow multi-dimensional visual feature encoder includes the heads of three parallel VGG16 networks and two fully connected layers; a deep multi-dimensional feature encoder based on a convolutional neural network is used to encode the visual features of the target subject image in the dimensions of cultural semantics and craft techniques, thus obtaining deep multi-dimensional visual features; as shown... Figure 4 As shown, the deep multidimensional visual feature encoder consists of two parallel VGG16 network heads and two fully connected layers.
[0086] In one embodiment, the data flow for acquiring the multimodal caption probability distribution map is as follows: Figure 5 As shown. The multimodal subtitle probability distribution map includes: a shallow multimodal subtitle probability distribution map and a deep multimodal subtitle probability distribution map; the multimodal subtitle probability distribution map in step 104 is generated through the following steps, specifically including:
[0087] Step 300: Obtain multiple training images of cultural relics and the corresponding text data for each training image.
[0088] Step 302: Encode the multi-dimensional features of the training artifact images to obtain the shallow multi-dimensional visual features and deep multi-dimensional visual features of the training artifact images.
[0089] Step 304: Based on the shallow multi-dimensional visual features, deep multi-dimensional visual features, and text data, a multi-dimensional semantic feature encoder based on an LSTM network is used to extract multi-dimensional semantic features and encode them to obtain shallow multi-dimensional semantic features and deep multi-dimensional semantic features. The shallow multi-dimensional semantic features include semantic features of color dimension, contour dimension, and texture dimension. The deep multi-dimensional semantic features include semantic features of cultural semantic dimension and craft technique dimension.
[0090] Step 306: After fusing the shallow multidimensional visual features and the shallow multidimensional semantic features, the shallow multidimensional decoder is used for decoding to obtain the shallow multimodal subtitle probability distribution map corresponding to the shallow semantic description; the shallow multidimensional decoder includes two fully connected layers and a Softmax activation function.
[0091] Step 308: After fusing the deep multi-dimensional visual features and the deep multi-dimensional semantic features, the deep multi-dimensional decoder is used for decoding to obtain the deep multi-modal subtitle probability distribution map corresponding to the deep semantic description; the deep multi-dimensional decoder has the same structure as the shallow multi-dimensional decoder.
[0092] Specifically, for multi-dimensional mixed visual features, they are fed into a multimodal feature decoder to generate image descriptions.
[0093] In this process, for each dimension of visual and linguistic features (color, pattern, shape, form), drop regularization is first applied to reduce the dimensionality of the model and prevent overfitting during training. Next, a random image sequence is introduced (this random sequence is randomly selected from the training set according to a preset ratio, preferably 70%). This random image sequence initializes the image list for training before each iteration, guiding the model to load the necessary images. Then, the mixed element vector is first fed into a Dense256-neuron layer, and then into the final output Dense layer to generate each word of the image captions. Finally, for each input image, the sentence with the highest probability distribution is selected as its description.
[0094] In this process, our goal is to achieve the highest probability of correct description given the input data:
[0095]
[0096] Where θ represents the model parameters of the fine-grained fusion feature decoder, F1 and F2 are the image feature vector and text feature vector after dropout dimensionality reduction, respectively, and S i Let p represent the correct description, and let p represent the probability distribution of the i-th description appearing in the sentence.
[0097] During the prediction process, the word vector with the highest probability is always used as the output. The probability of generating this sentence can be calculated as follows:
[0098]
[0099] Where, N s It is the maximum output length. logp(S i |F1,F2) represents the probability of generating a sentence, S it The (t+1)th word is the correct description.
[0100] Finally, for each input image, the sentence with the highest probability distribution generated by the decoder is selected as its description (word S). i ).
[0101] In addition, random image sequences were introduced during the training process, and a random sampling training mode was adopted. This eliminated the need to load all images in each round, shortened the training time, reduced memory consumption, and improved training efficiency. Most importantly, the introduction of random image sequences effectively avoided the impact of data distribution issues on the training results.
[0102] In one embodiment, the multi-dimensional semantic feature encoder includes a shallow multi-dimensional semantic feature encoder and a deep multi-dimensional semantic feature encoder with identical structures; such as Figure 6 As shown, the shallow multi-dimensional semantic feature encoder includes: a word embedding layer, a nonlinear transform layer, an LSTM network, and a fully connected layer; step 304 includes: segmenting the text data, extracting the text corresponding to the shallow multi-dimensional visual features, and cleaning the data; inputting the cleaned data into the word embedding layer for encoding, and then processing it with the nonlinear transform layer to obtain the word vector LSTM network, and inputting the obtained result into the fully connected layer to obtain the shallow multi-dimensional semantic feature encoding; segmenting the text data, extracting the text corresponding to the deep multi-dimensional visual features, and cleaning the data; inputting the cleaned data into the word embedding layer for encoding, and then processing it with the nonlinear transform layer to obtain the word vector LSTM network, and inputting the obtained result into the fully connected layer to obtain the deep multi-dimensional semantic feature encoding of the cultural relic image.
[0103] Specifically, the word embedding layer is the embedding module.
[0104] Drawing inspiration from feedforward neural network language models, the nonlinear transformation layer is implemented using a feedforward neural network to generate a word probability distribution model, thereby improving the accuracy of encoding prediction.
[0105] For semantic feature encoding, the text data is first segmented, the text corresponding to the visual features of the image is extracted and cleaned, then word segmentation technology is used to split each word and encode each word. Finally, the encoded words are fed into an LSTM language encoder for pre-training, and then through a fully connected layer with 256 memory units, a 256-element text embedding vector is finally generated, specifically including:
[0106] (1) First, the encoder is guided by a start code “startCap”.
[0107] (2) Traverse all images, take the visual features of the images and the first word embedding vector of the visual description as mixed input, and generate the second word vector through the nonlinear transformation of the feedforward neural network;
[0108] (3) Combine the first two word vectors together and then combine them with image features as a new model input to generate the next word vector;
[0109] (4) Repeat the process until a complete image description is generated and “endCap” is used as the ending word.
[0110] The output is fed into a fully connected layer with 256 neural units, ultimately generating a 256-element text embedding vector as a multi-dimensional semantic feature encoding of the cultural relic image.
[0111] In one embodiment, the multimodal caption probability distribution map includes: a shallow multimodal caption probability distribution map and a deep multimodal caption probability distribution map; step 104 includes: predicting and outputting a shallow semantic description based on the shallow multidimensional visual features and the shallow multimodal caption probability distribution map; predicting and outputting a deep semantic description based on the deep multidimensional visual features and the deep multimodal caption probability distribution map; and fusing the shallow semantic description and the deep semantic description to obtain a text description of the cultural relic image, which is the caption for the poster.
[0112] In one embodiment, the preset poster template library in step 108 is constructed based on the cultural relic photo dataset; the construction steps of the preset poster template library include: designing 5 types of poster template layouts according to the style of the cultural relic; wherein each type of poster template layout includes 8 specific layout parameters; designing 7 color systems according to the color of the cultural relic; wherein the color system includes 12 color schemes; and designing 40 sets of font styles according to the shape of the cultural relic; wherein the font style includes font and font size.
[0113] Specifically, existing automatic layout generation methods offer limited flexibility, typically presenting all elements completely in the image, which doesn't meet practical requirements. For cultural relic posters, leaving some white space enhances the poster's artistry and atmosphere. Sometimes, to highlight specific features of the relic, the entire object isn't shown in the poster. Furthermore, existing algorithms suffer from long latency issues. However, users have high aesthetic expectations and low tolerance for errors in cultural relic poster generation tasks.
[0114] To generate posters as accurately and quickly as possible, we researched and discussed a dataset of artifact photographs with artists, and designed poster templates and constructed the dataset Poster_layout according to the following principles:
[0115] Five types of template layouts (layout groups) were designed based on the style and type of cultural relics, with each group containing eight specific layout parameters.
[0116] Seven color schemes were designed based on the color types of the cultural relics, with 12 color combinations under each color scheme.
[0117] Forty sets of font styles (including font and font size) were designed based on the shape types of cultural relics (the shape types correspond to the characteristics of the artifacts).
[0118] In one embodiment, step 108 includes: constructing feature tuple pairs based on a preset poster template library, wherein the feature tuple pairs are:
[0119] {H,C(I),S(I)}
[0120] Where H is the coordinate of the attention center, C(I) is the color feature vector, and S(I) is the feature composed of shallow multi-dimensional visual features and deep multi-dimensional data features.
[0121] A center label is constructed based on the relative position of the attention center; based on the center label and feature tuple pairs, the K-Means clustering algorithm is used to obtain a poster template layout that matches the center label; based on the feature tuple pairs and the color dimension features and hybrid features of the cultural relic image, the color scheme and font style of the template are determined using the same method as determining the poster template layout; the hybrid features are features composed of shallow multi-dimensional visual features and deep multi-dimensional data features; based on the poster template layout, the color scheme and font style of the template, a template that matches the attention center is obtained.
[0122] In another embodiment, a method for automatically generating posters is provided, comprising the following steps:
[0123] Step 1: Use edge detection and contour extraction techniques to obtain the target subject of the image and construct a subject image with a transparent background.
[0124] Step 2: Use multimodal feature encoding techniques to obtain the visual feature encoding and semantic feature encoding of the target image.
[0125] Step 3: The multimodal feature encodings from Step 2 are fused and represented, and then decoded using multimodal feature decoding technology to generate an image description.
[0126] Step 4: Obtain the attention center of the target image by stripping the VGG output layer, and use the K-means feature clustering algorithm to retrieve templates from the poster template library that match the center position of the target subject.
[0127] The VGG output layer, mainly the output vector of layer 12, is stripped, and after regularization, the peak point of the model's attention is obtained. This peak point is then used as the equivalent center of the image to generate background textures and match poster templates.
[0128] Step 5: Use text stylization and template filling to combine poster elements to generate an aesthetic poster.
[0129] It should be understood that, although Figure 1 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 1 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.
[0130] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0131] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.
Claims
1. A method for automatically generating posters based on multi-dimensional feature extraction, characterized in that, The method includes: Acquire images of cultural relics, perform edge detection and background segmentation on the images to obtain the target subject, and construct a transparent image of the target subject; The target image is encoded with visual features in multiple dimensions to obtain shallow multi-dimensional visual features and deep multi-dimensional visual features. The shallow multi-dimensional visual features include visual features in color, outline and texture dimensions, and the deep multi-dimensional visual features include visual features in cultural semantic dimension and craft technique dimension. Based on the shallow multidimensional visual features, the deep multidimensional visual features, and the multimodal subtitle probability distribution map, poster subtitles are predicted and output. The multimodal subtitle probability distribution map is used to guide image subtitle generation. The multimodal subtitle probability distribution map is obtained by encoding shallow and deep multidimensional visual features of multiple training artifact images, then decoding the obtained shallow and deep multidimensional visual features, using the obtained visual feature vector as input and the text feature vector as output, and training it using a supervised learning method. The attention center of the target subject image is extracted using a VGG network with the middle layer removed, and the attention center area is image-edited to obtain the poster background texture. Based on the attention center, the shallow multi-dimensional visual features, and the deep multi-dimensional visual features, the K-Means feature clustering algorithm is used to search in a preset poster template library to obtain a template that matches the attention center. A cultural relic poster is generated based on the target subject, the poster subtitles, and a template that matches the center of attention.
2. The method according to claim 1, characterized in that, The method also includes, An author watermark is created based on the creator's personal information, placed in an appropriate position on the cultural relic poster, and the final cultural relic poster is generated.
3. The method according to claim 1, characterized in that, Acquire an image of the cultural relic, perform edge detection and background segmentation on the image to obtain the target subject, and construct a transparent image of the target subject, including: Acquire images of cultural relics; The U2-Net network is used to perform edge detection and background segmentation on the cultural relic image to obtain the target subject; the U2-Net network includes an encoder and decoder with a 6-reSidual U-block architecture, a 1×1 convolutional layer and a sigmoid activation function; The target subject is superimposed on the image of the cultural relic to obtain the complement of the intersection of the target subject and the image of the cultural relic; The third channel parameter of the corresponding region of the complement is set to 0 to obtain the target subject image with a transparent background.
4. The method according to claim 1, characterized in that, The target image is encoded with visual features in multiple dimensions to obtain shallow multi-dimensional visual features and deep multi-dimensional visual features, including: A shallow multidimensional visual feature encoder based on a convolutional neural network is used to encode visual features in the color dimension, contour dimension and texture dimension to obtain shallow multidimensional visual features; the shallow multidimensional visual feature encoder includes the heads of three parallel VGG16 networks and two fully connected layers. A deep multidimensional feature encoder based on a convolutional neural network is used to encode the visual features of the target subject image in the dimensions of cultural semantics and craft techniques to obtain deep multidimensional visual features. The deep multidimensional visual feature encoder includes the heads of two parallel VGG16 networks and two fully connected layers.
5. The method according to claim 1, characterized in that, The multimodal subtitle probability distribution map includes: a shallow multimodal subtitle probability distribution map and a deep multimodal subtitle probability distribution map; Based on the shallow multi-dimensional visual features, the deep multi-dimensional visual features, and the multimodal subtitle probability distribution map, poster subtitles are predicted and output; the steps for generating the multimodal subtitle probability distribution map include: Acquire multiple images of cultural relics for training and the corresponding text data for each image; The multi-dimensional features of the training artifact images are encoded to obtain shallow and deep multi-dimensional visual features. The text data is then processed using an LSTM-based multi-dimensional semantic feature encoder to extract and encode multi-dimensional semantic features, resulting in shallow and deep multi-dimensional semantic features. The shallow multi-dimensional semantic features include semantic features in color, contour, and texture dimensions. The deep multi-dimensional semantic features include semantic features in cultural and craft techniques dimensions. After fusing the shallow multidimensional visual features and the shallow multidimensional semantic features, a shallow multidimensional decoder is used for decoding to obtain a shallow multimodal subtitle probability distribution map corresponding to the shallow semantic description; the shallow multidimensional decoder includes two fully connected layers and a Softmax activation function; After fusing the deep multidimensional visual features and the deep multidimensional semantic features, a deep multidimensional decoder is used for decoding to obtain a deep multimodal subtitle probability distribution map corresponding to the deep semantic description; the deep multidimensional decoder has the same structure as the shallow multidimensional decoder.
6. The method according to claim 5, characterized in that, The multi-dimensional semantic feature encoder includes a shallow multi-dimensional semantic feature encoder and a deep multi-dimensional semantic feature encoder with the same structure. The shallow multi-dimensional semantic feature encoder includes: a word embedding layer, a nonlinear transformation layer, an LSTM network, and a fully connected layer. Based on the shallow multi-dimensional visual features, the deep multi-dimensional visual features, and the text data, a multi-dimensional semantic feature encoder based on an LSTM network is used to extract multi-dimensional semantic features and encode them to obtain shallow multi-dimensional semantic features and deep multi-dimensional semantic features, including: The text data is segmented, and the text corresponding to the shallow multi-dimensional visual features is extracted and the data is cleaned. After the data is cleaned, the result is input into the word embedding layer for encoding and then processed by the nonlinear transformation layer. The resulting word vectors are input into the LSTM network, and the result is input into the fully connected layer to obtain shallow multi-dimensional semantic feature encoding. The text data is segmented, and the text corresponding to the deep multi-dimensional visual features is extracted and the data is cleaned. After data cleaning, the results are input into the word embedding layer for encoding and then processed by the nonlinear transformation layer. The resulting word vectors are input into the LSTM network, and the results are input into the fully connected layer to obtain the deep multi-dimensional semantic feature encoding of the cultural relic image.
7. The method according to claim 1, characterized in that, The multimodal subtitle probability distribution map includes: a shallow multimodal subtitle probability distribution map and a deep multimodal subtitle probability distribution map; Based on the shallow multi-dimensional visual features, the deep multi-dimensional visual features, and the multimodal subtitle probability distribution map, poster subtitles are predicted and output, including: Based on the shallow multidimensional visual features and the shallow multimodal caption probability distribution map, predict and output shallow semantic descriptions; Based on the deep multidimensional visual features and the deep multimodal caption probability distribution map, predict and output deep semantic descriptions; By fusing the shallow semantic description and the deep semantic description, a textual description of the cultural relic image is obtained, which becomes the poster subtitle.
8. The method according to claim 1, characterized in that, Based on the attention center, the shallow multi-dimensional visual features, and the deep multi-dimensional visual features, the K-Means feature clustering algorithm is used to search in a preset poster template library to obtain templates that match the attention center. The preset poster template library is constructed based on a dataset of cultural relic photographs. The steps for constructing the preset poster template library include: Five types of poster template layouts were designed based on the styles and types of cultural relics; each type of poster template layout includes eight specific layout parameters; Seven color schemes were designed based on the color types of the cultural relics; each color scheme includes 12 color combinations. Forty font styles were designed based on the shapes and types of cultural relics; the font styles include font and font size.
9. The method according to claim 8, characterized in that, Based on the attention center, the shallow multi-dimensional visual features, and the deep multi-dimensional visual features, the K-Means feature clustering algorithm is used to search in a preset poster template library to obtain templates that match the attention center, including: Feature tuple pairs are constructed based on a preset poster template library. The feature tuple pairs are as follows: in, These are the coordinates of the attention center. It is a color feature vector. It is a feature composed of shallow multi-dimensional visual features and deep multi-dimensional data features; Construct a center label based on the relative position of the attention center; Based on the central label and the feature tuple pair, the K-Means clustering algorithm is used to obtain the poster template layout that matches the central label; Based on the feature tuple pairs and the color dimension features and blending features of the cultural relic image, the color scheme and font style of the template are determined using the same method as for determining the poster template layout; the blending features are features resulting from a combination of shallow multi-dimensional visual features and deep multi-dimensional data features. Based on the poster template layout, color scheme, and font style, a template matching the center of attention is obtained.