Method for image artistic conception mining and artistic conception conversion of chinese ancient poems based on deep learning

Abstract

The application provides a method for image artistic conception mining and artistic conception conversion Chinese ancient poems based on deep learning, receives a user task of generating an ancient poem by a user, and extracts a target picture submitted by the user from the user task; the target picture is input into a pre-trained cross-modal poem generator to generate a corresponding ancient poem for the image submitted by the user; a CNN-based image feature extractor of the application extracts feature vectors of a static image in three aspects of objects, scenes and emotions, and adds the position coding to serve as the input of an encoder of the cross-modal poem generator; model parameters of a pre-trained single-modal language model are introduced into a decoder of the cross-modal poem generator to learn grammar information, so as to improve the generation quality of the cross-modal poem generator; finally, the cross-modal poem generator is trained on a cross-modal image-poem pairing data set, and the generation result is evaluated through the cosine similarity between the generated ancient poem vector and the real ancient poem vector.

Description

Technical Field

[0001] This invention belongs to the field of deep learning recognition and generation technology, specifically relating to a method for image mood mining and mood transformation of ancient Chinese poems based on deep learning. Background Technology

[0002] Chinese classical poetry is a treasure of traditional Chinese culture, with a long history dating back over two thousand years to the *Classic of Poetry*. As the primary vehicle for ancient working people to record events and express emotions, Chinese classical poetry, under a unique historical and cultural background, has developed an aesthetic pursuit that combines form and spirit, and blends emotion and scenery. The *Opinions on Implementing the Project for the Inheritance and Development of Excellent Traditional Chinese Culture* points out that inheriting the spirit of Chinese humanism is one of the main contents of this project. Therefore, promoting and popularizing classical poetry culture has significant practical implications. With the continuous development of artificial intelligence technology, its applications in various fields are numerous, but research on cross-modal image-to-poetry generation is still relatively limited. Single-modal methods for generating classical poetry also suffer from problems such as theme drift, poor consistency, poor flexibility, poor self-learning ability, and the lack of an automatic evaluation system. How to extract image mood information from images and generate higher-quality Chinese classical poems based on this mood is a significant challenge in current research.

[0003] Existing unimodal methods for generating classical Chinese poems include traditional methods and deep learning-based methods. Traditional methods include template-based methods, pattern-based methods, instance-based reasoning methods, genetic algorithm-based methods, summary generation methods, and statistical machine translation-based methods. Deep learning-based methods include generating classical Chinese poems using sequence models such as RNNs.

[0004] Existing cross-modal image-to-poem generation methods fall into two categories. The first is based on manually collecting image-poem pairing datasets and retrieving poems from images using template matching. The second method mainly extracts keywords from images and uses these keywords as input information for existing unimodal poem generation methods. The main steps of this technique are as follows:

[0005] 1) Using existing ancient poetry datasets, collect matching image resources to form a paired dataset;

[0006] 2) Use a CNN network to extract the main objects in the image as keywords for generating ancient poems;

[0007] 3) Use traditional ancient poetry generation methods or RNN models and their variant LSTM models as ancient poetry generators, take the keywords obtained in the previous step as input information, and train them using a paired dataset;

[0008] 4) Use the trained image-ancient poem generator to generate ancient poems for the image and evaluate them on metrics commonly used in natural language processing such as BLEU.

[0009] The prior art has the following problems:

[0010] 1) Manually collecting the image-ancient poem paired dataset has no unified matching standard, is subjective, time-consuming, and laborious, and it is difficult to form a dataset with a large scale;

[0011] 2) Generating ancient poems from pictures is more challenging than generating ancient poems from the first sentence or keywords. The image contains rich visual information. If only the method of recognizing the main scenery as the recognition keyword is adopted, a large amount of information will be lost;

[0012] 3) Traditional ancient poem generation methods, such as template-based methods, pattern-based methods, case-based reasoning methods, genetic algorithm-based methods, summary generation-based methods, and statistical machine translation-based methods, etc., all rely heavily on templates or datasets, have no strong self-learning ability, lack connectivity in context, and it is also very difficult to conform to basic grammar; Generating ancient poems based on the RNN model is more flexible than traditional methods and gets rid of the dependence on artificial templates and rules. However, it also has certain problems. Limited by the gradient vanishing and gradient explosion phenomena of the RNN model, as the length of the generated ancient poem increases, the association between the later part and the previous part will become less and less tight, and there is a problem of topic drift;

[0013] 4) Metrics such as BLEU were originally invented to evaluate the results of machine translation. However, the ancient poem generation task is not equivalent to the machine translation task. For an image, there may be different interpretations from multiple angles, and two completely different words may also express the same meaning. For example, "cuì wēi" and "qīng shān" both refer to the green mountain. Therefore, the BLEU value is not suitable as an automatic evaluation standard for the ancient poem generation problem;

[0014] 5) Lack of more interactive, interesting, and aesthetic application forms, and there is a certain programming threshold for ordinary people to directly use the trained model. Summary of the Invention

[0015] In order to solve the above problems existing in the prior art, the present invention provides a method for mining image artistic conception and converting it into Chinese ancient poems based on deep learning. The technical problems to be solved by the present invention are realized through the following technical solutions:

[0016] The present invention provides a method for mining image artistic conception and converting it into Chinese ancient poems based on deep learning, including:

[0017] Step 1, receive the user task of generating ancient poems and extract the target picture submitted by the user from the user task;

[0018] Step 2: Input the target image into a trained cross-modal poetry generator to generate the corresponding ancient poem for the image submitted by the user;

[0019] The training process of the pre-trained cross-modal poetry generator is as follows:

[0020] (1) Obtain an open-source ancient poetry dataset from an open-source database and perform keyword preprocessing to obtain high-frequency keywords after filtering; wherein, the open-source ancient poetry dataset includes relevant information about the ancient poems and multiple keywords corresponding to each ancient poem; (2) Match the multiple keywords corresponding to each ancient poem with the high-frequency keywords. If only one keyword does not match, retain the relevant information about the ancient poem and the matched keywords to obtain a dataset to be paired; (3) Concatenate the keywords of the same ancient poem in the dataset to be paired as search keywords, and obtain multiple initial images that are paired with the ancient poems from the network according to the search keywords. (4) The initial matching images are filtered according to the relevant information of the ancient poems to obtain the images paired with each ancient poem according to the degree of correlation. Each ancient poem and its paired image are used as a sample data, and all sample data are combined into an image-ancient poem pairing dataset. (5) The multi-dimensional feature vectors of the image-ancient poem pairing dataset are extracted and reshaped into feature sequences. The feature sequences are used as the input of the cross-modal poetry generator. The model parameters of the pre-trained single-modal language model are imported into the preset cross-modal poetry generator. Then, the cross-modal poetry generator is fine-tuned and trained to complete the pre-training process of the cross-modal poetry generator.

[0021] The present invention has at least one of the following beneficial effects:

[0022] 1. Multiple CNN networks are used to mine clues for writing poetry from different aspects. Since writing poetry based on the content of an image requires understanding the image first, this invention uses three networks, VGGNet, Place205-VGGNet, and DCAN, to extract feature vectors of the image in multiple aspects such as objects, scenes, and emotions, thus retaining more comprehensive information.

[0023] 2. Using the Transformer model as a text generator for poetry, the machine not only needs to learn the poetic quality of the language and the structural features of the meter during the poetry generation process, but also needs to consider the consistency of the theme to avoid theme drift. The Transformer model used in this invention is currently the best performing model in many sub-problems in the field of NLP, and it can also achieve better results than previous studies in the problem of image-to-poetry generation.

[0024] 3. This invention proposes text similarity as an automatic evaluation metric for poetry generation. Because Chinese is a high-context language, two words with vastly different original meanings may have the same extended meaning in specific contexts. Therefore, evaluation metrics commonly used in machine translation or natural language processing, such as BLEU, are not suitable for generating classical Chinese poetry. This invention uses a BERT-CCPoem model trained on a poetry corpus dataset to generate feature vectors for the entire poem, and uses the cosine similarity between two poem vectors as an automatic evaluation metric for comparing the quality of poetry generation, which is more reasonable.

[0025] 4. The image-ancient poem matching dataset collection method proposed in this invention can greatly save the time cost of collecting datasets, collect data in batches, and expand the scale of the dataset.

[0026] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0027] Figure 1 This is a flowchart illustrating a method for extracting and transforming the artistic conception of images from ancient Chinese poems based on deep learning, provided by this invention.

[0028] Figure 2 This is a diagram of the IPGCT model structure;

[0029] Figure 3 This is a diagram of an image feature extractor based on CNN;

[0030] Figure 4 This is a diagram of the structure of a pre-trained unimodal language model;

[0031] Figure 5 This is a diagram of the cross-modal poetry generator structure;

[0032] Figure 6 This is an example diagram illustrating the method of extracting artistic conception and generating poetry according to the present invention;

[0033] Figure 7 This is an example diagram of a pre-trained unimodal language model;

[0034] Figure 8 The images show the poetry writing results of the method proposed in this invention and two comparative methods.

[0035] Figure 9 It is a confusion matrix diagram of manual evaluation;

[0036] Figure 10 This is a graph showing the experimental results of Grad-CAM. Detailed Implementation

[0037] The present invention will be further described in detail below with reference to specific embodiments, but the implementation of the present invention is not limited thereto.

[0038] Image-to-poetry generation comprises two sub-tasks: image understanding and natural language generation. First, it requires understanding the semantic features of the input image, and then generating poetic descriptive statements based on these features. The overall process of this invention's method for image-based mood mining and mood transformation of classical Chinese poetry is as follows: Figure 1 As shown.

[0039] like Figure 1 As shown, the present invention provides a method for image-based mood mining and mood transformation of ancient Chinese poems based on deep learning, which includes:

[0040] Step 1: Receive the user task of generating an ancient poem and extract the target image submitted by the user from the user task;

[0041] Step 2: Input the target image into a trained cross-modal poetry generator to generate the corresponding ancient poem for the image submitted by the user;

[0042] The training process for the cross-modal poetry generator is as follows:

[0043] (1) Obtain an open-source ancient poetry dataset from an open-source database and perform keyword preprocessing to obtain high-frequency keywords after filtering; wherein, the open-source ancient poetry dataset includes relevant information about the ancient poems and multiple keywords corresponding to each ancient poem; (2) Match the multiple keywords corresponding to each ancient poem with the high-frequency keywords. If only one keyword does not match, retain the relevant information about the ancient poem and the matched keywords to obtain a dataset to be paired; (3) Concatenate the keywords of the same ancient poem in the dataset to be paired as search keywords, and obtain multiple initial images that are paired with the ancient poems from the network according to the search keywords. (4) The initial matching images are filtered according to the relevant information of the ancient poems to obtain the images paired with each ancient poem according to the degree of correlation. Each ancient poem and its paired image are used as a sample data, and all sample data are combined into an image-ancient poem pairing dataset. (5) The multi-dimensional feature vectors of the image-ancient poem pairing dataset are extracted and reshaped into feature sequences. The feature sequences are used as the input of the cross-modal poetry generator. The model parameters of the pre-trained single-modal language model are imported into the preset cross-modal poetry generator. Then, the cross-modal poetry generator is fine-tuned and trained to complete the training process of the cross-modal poetry generator.

[0044] The feature vectors of each sample data extracted by the three feature extraction networks are concatenated and reshaped to obtain the feature sequence X = (X1, X2, ..., X...). n );

[0045] in, n is the number of feature sequences, and l is the length of the feature sequences;

[0046] The vector representation of the actual paired ancient poems is as follows:

[0047] Y = (Y1, Y2, ..., Y) m (1)

[0048]

[0049] Where m is the total number of characters in the ancient poem, and s is the size of the vocabulary list;

[0050] The predicted vector representation of the ancient poems is as follows:

[0051]

[0052]

[0053] Will The character with the highest median probability is used as the output character, and m characters are concatenated together to generate an ancient poem.

[0054] To address the challenges of extracting imagery and thematic drift inherent in existing poetry generation methods, and to accomplish the task of generating poetry from images, this invention proposes an Image to Poem Generator based on CNN and Transformer (IPGCT) framework. This framework uses a CNN-based image feature extractor, a pre-trained unimodal language model, and a Transformer-based cross-modal poetry generator. The overall framework is as follows: Figure 2 As shown.

[0055] First, a CNN-based image feature extractor extracts feature vectors from static images in three aspects: object, scene, and emotion. These vectors are then added to positional encodings and used as input to the cross-modal poetry generator's encoder. A pre-trained unimodal language model is trained on a unimodal poetry dataset. The model parameters of its decoder are extracted and imported into the cross-modal poetry generator's decoder. The learned grammatical information improves the generation quality of the cross-modal poetry generator. Finally, the cross-modal poetry generator is trained on a cross-modal image-poetry pairing dataset to complete the image-to-poetry task.

[0056] In one specific embodiment, (1) includes:

[0057] (11) Obtain an open-source ancient poetry dataset from an open-source database;

[0058] (12) Count the frequency of each keyword in the open-source ancient poetry dataset and select high-frequency keywords according to the proportion;

[0059] (13) Replace the uncommon keywords in the open-source ancient poetry dataset with keywords with the same meaning;

[0060] (14) Delete the high-frequency keywords abstracted from the open-source ancient poetry dataset to obtain the filtered high-frequency keywords.

[0061] In one specific embodiment, (3) includes:

[0062] (31) Concatenate the keywords of the same ancient poem in the dataset to be paired, and use the concatenated keywords as search keywords;

[0063] (32) Based on the search keywords, crawler technology is used to crawl multiple initial images that are paired with ancient poems from publicly available image databases on the Internet.

[0064] In one specific embodiment, (4) includes:

[0065] (41) The initial matching images are manually screened according to the relevant information of the ancient poems to obtain images matched according to the relevance of each ancient poem;

[0066] (42) Each ancient poem and its paired image are used as a sample data, and all sample data are combined into an image-ancient poem pairing dataset.

[0067] In one specific embodiment, the three feature extraction networks in (5) are implemented through the following steps:

[0068] a. Obtain the Imagent-1k dataset, Place205 dataset, and Sentibank dataset respectively;

[0069] b. Train a pre-defined VGG-16 network on the Imagent-1k dataset, a pre-defined Place205-VGGNet on the Place205 dataset, and a pre-defined DCAN network on the Sentibank dataset to obtain three feature extraction networks.

[0070] The main extraction processes of the three feature extraction networks are as follows: Figure 3As shown, creating an ancient poem often requires specific objects as images, which are then combined with the scene and the poet's emotions to create artistic conception. The VGG-16 network trained on the Imagent-1k dataset, the Place205-VGGNet trained on the Place205 dataset, and the DCAN network trained on the Sentibank dataset can extract information from images from aspects such as objects, scenes, and emotions. This invention uses these three CNN networks as image feature extractors.

[0071] In one specific embodiment, (5) includes:

[0072] (51) Use three feature extraction networks to extract the feature vector of each sample data respectively;

[0073] (52) The three feature vectors of the same sample data are concatenated to reshape the concatenated feature vectors into a feature sequence.

[0074] The reshaped shape is (1, 24, 512), and the cascade reshaping process is shown below:

[0075] V=Concat{CNN1(I),CNN2(I),CNN3(I)} (5)

[0076] TE = Reshape(V) (6)

[0077] Where I represents the original image, V represents the concatenated feature vector, and TE represents the reconstructed features;

[0078] (53) Calculate the position code for each sample data, add the position code to the reconstructed vector, and use the sum as the input to the preset cross-modal poetry generator;

[0079] The formula for calculating the location code is:

[0080]

[0081]

[0082] IE = PE + TE (9)

[0083] Where pos represents the position of the token in the entire sequence, i represents the i-th position of the position code, 2i represents all even positions, 2i+1 represents all odd positions, and d model The dimension represents the model, which is 512 dimensions in this case. PE represents positional encoding, and IE represents the input of the encoder to the cross-modal poetry generator.

[0084] (54) Select seven-character quatrains from the open-source ancient poetry dataset and clean the selected seven-character quatrains to obtain a single-modal ancient poetry dataset.

[0085] (55) The previous line of each ancient poem in the single-modal ancient poem dataset is used as the input of the encoder of the pre-set single-modal language model, and the next line is used as the input of the decoder of the single-modal language model. The single-modal language model is trained to learn the grammatical information of the ancient poems to obtain the pre-trained single-modal language model. The parameters of the self-attention layer and the feedforward neural network layer of the decoder in the pre-trained single-modal language model are retained.

[0086] This invention addresses the problem of insufficient training of the cross-modal poetry generator due to the small size of the collected multimodal image-ancient poetry dataset. The model structure diagram is shown below. Figure 4 As shown.

[0087] In this model, the previous line of each poem in the preprocessed unimodal ancient poetry dataset (containing over 70,000 quatrains) is used as the input to the Encoder, and the next line is used as the input to the Decoder. The process is as follows:

[0088] WE=σ{φ(SE)}+PE (10)

[0089] WD=σ{φ(SE)}+PE (11)

[0090] O = Trans(WE, WD) (12)

[0091] Where WE is the input of the Encoder in the Transformer network, WD is the input of the Decoder in the Transformer network, σ(·) is the bidirectional mapping function between Chinese characters and IDs, φ(·) is the word embedding function, SE is the original poem, PE is the positional encoding, O is the output of the Transformer network, and Trans(·) is the Transformer network.

[0092] Through training, the decoder of a unimodal language model can learn the grammatical information of classical Chinese poems. By saving the parameters of its self-attention layer and feedforward neural network layer and importing them into a multimodal model, the quality of poems generated by the multimodal model can be improved.

[0093] (56) The model parameters of the pre-trained single-modal language model are imported into the preset cross-modal poetry generator, and then the cross-modal poetry generator is fine-tuned and trained to complete the training process of the cross-modal poetry generator.

[0094] The cross-modal poetry generator is implemented based on the Transformer network, and its overall structure is as follows: Figure 5 As shown. In the Transformer, the input to the Encoder is IE, and the input to the Decoder is the pre-processed paired poem, that is:

[0095] PO = Trans(IE,WD) (13)

[0096] In one specific embodiment, the method for image mood mining and mood transformation of ancient Chinese poems based on deep learning provided by the present invention further includes:

[0097] (6) Use the test set input to train a cross-modal poetry generator to generate ancient poems and automatically evaluate the quality of the ancient poems.

[0098] (6) Includes:

[0099] (61) Obtain a test set consisting of multiple images;

[0100] Each image is paired with a real ancient poem;

[0101] (62) A trained cross-modal poetry generator was used to generate an ancient poem for each image in the test set, and the open-source BERT-CCPoem model was called to convert the ancient poems into 512-dimensional ancient poem vectors. i ;

[0102] (63), calculate and generate the ancient poem vector. i and real ancient Chinese poem vectors (poem realvector) i cosine similarity (cos sim) i ;

[0103]

[0104] (64) Calculate the cosine similarity (cossim) for each image in the entire test set. i The summation and average are used as the automatic evaluation standard for the quality of the generated ancient poems (auto evaluate).

[0105]

[0106] (65) The quality of the generated ancient poems is evaluated using the autoevaluate standard.

[0107] This invention provides a method for extracting and transforming the image's artistic conception into the artistic conception of classical Chinese poetry based on deep learning. The method receives a user task to generate a classical Chinese poem and extracts the target image submitted by the user from the task. The target image is then input into a pre-trained cross-modal poetry generator to generate a corresponding classical Chinese poem for the submitted image. The invention uses a CNN-based image feature extractor to extract feature vectors from static images in three aspects: object, scene, and emotion. These feature vectors are added to the positional encoding and used as input to the encoder of the cross-modal poetry generator. The model parameters of a pre-trained unimodal language model are imported into the decoder of the cross-modal poetry generator to learn grammatical information, thereby improving the generation quality of the cross-modal poetry generator. Finally, the cross-modal poetry generator is trained on a cross-modal image-poem pairing dataset, and the generation results are evaluated by the cosine similarity between the generated classical Chinese poem vector and the real classical Chinese poem vector.

[0108] The technical effects of this invention are illustrated below through simulation experiments.

[0109] The experimental environment for this invention was Windows 10, the GPU was an NVIDIA GeForce GTX 1050, the CPU was an Intel(R) Core(TM) i5-8300H CPU@2.30GHz, and the deep learning framework used was TensorFlow 2.6.0.

[0110] The experiment first collected the first multimodal image-poem pairing dataset, which contains over 1700 image-poem pairs. Then, poems from the public dataset THU-CCPC were filtered to form a single-modal poetry dataset. Next, this invention uses three networks—VGG-16 trained on the Imagent-1k dataset, Place205-VGGNet trained on the Place205 dataset, and DCAN trained on the Sentibank dataset—to extract features from images in terms of objects, scenes, and emotions, respectively. Then, a single-modal language model was trained on the single-modal poetry dataset, and a cross-modal poetry generator was trained on the multimodal image-poem pairing dataset. Finally, this invention uses BERT-CCPoem to automatically evaluate the poems generated by the proposed method and two comparative methods. A questionnaire containing randomly selected real and generated poems was used to conduct a double-blind experiment to manually evaluate the poems generated by the proposed method. Ablation experiments were conducted to investigate the impact of the three feature CNNs, the number of Transformer layers, and the pre-trained model on the quality of the generated poems.

[0111] One of the generated examples is as follows Figure 6As shown, when the test images are input into three CNN networks, they can recognize the mood information such as objects, scenes, and emotions in the images, such as "plum blossoms", "children", "courtyard", "spring", and "happy" here. Instead of directly using label information, this invention extracts the features of the "fc7" layer of these three CNNs, adds them to the position encoding, and inputs them into the encoder of the Transformer, and uses the trained cross-modal poem generator to obtain the generated ancient poems.

[0112] (1) Experimental setup

[0113] Since the research task of this invention has not been explored by predecessors and is a highly pioneering work, there is no open-source image-ancient poem pairing dataset published by predecessors on various platforms. Therefore, it is necessary to design a scheme to collect and organize the dataset by ourselves. The main idea of collecting the dataset is as follows:

[0114] The open-source ancient poem dataset THU-CCPC contains the relevant information of more than 130,000 jueju poems, and four corresponding keywords are given for each jueju poem. Conduct a word frequency statistics on the keywords of these jueju poems, and save the top 500 keywords with the highest occurrence frequencies. Then conduct an artificial screening on them, and save the part of the keywords that are convenient to search in the search engine, such as "white clouds", "green mountains", "plum blossoms", etc. Replace the relatively rare keywords according to their semantics, such as replacing "cuiwei" with "green mountains", etc. Delete some relatively abstract keywords, such as "lonely", "clear", etc.

[0115] Match the keywords in the original dataset with 220 high-frequency keywords. If at least three of the four keywords of a poem belong to high-frequency keywords, then retain the poem and its keywords. Concatenate the keywords of the same poem as the keywords for crawling on the "Visual China" website, and crawl three preliminary paired images for each poem. Then conduct an artificial screening on them, and retain the highly relevant pairs as the sample data in the image-ancient poem pairing dataset. After artificial screening, a multi-modal dataset containing 1,691 image-poem pairs is finally obtained.

[0116] The relevant parameter settings of the Transformer network in the method of this invention are shown in Table 1:

[0117] Table 1 Transformer model parameter settings

[0118]

[0119]

[0120] This invention uses Adam Optimizer as the optimizer of the model, and its parameter settings are shown in Table 2:

[0121] Table 2 Adam Optimizer Parameter Settings

[0122]

[0123] The learning rate is continuously updated during training, and its definition formula is shown below.

[0124]

[0125] (2) Quantitative analysis

[0126] By using the open-source BERT-CCPoem model, a 512-dimensional poem vector can be generated for each poem. This invention generates a 512-dimensional poem vector for both the real and generated poems corresponding to each image in the test set, and then calculates the cosine similarity.

[0127] The results from all test sets are then summed and averaged to serve as an automatic evaluation standard for the quality of the generated poems.

[0128] Since the problem studied in this invention has not been addressed by predecessors, the present invention has selected the following two comparative methods.

[0129] a) LSTM model for generating ancient poems based on keywords: The most probable results identified by three CNNs from the image are used as keywords and input into the LSTM-based poetry writing model.

[0130] b) Vision Transformer (ViT) model: The ViT model divides an image into several patches of a fixed size, adds positional encoding, and uses these patches as input to the Encoder of the Transformer, while using the corresponding ancient poem as input to the Decoder.

[0131] The experimental results of the proposed method and two comparative methods on automatic evaluation indicators are shown in Table 3.

[0132] Table 3. Results of the Automatic Evaluation Experiment

[0133]

[0134] As can be seen from the results in Table 4, the method proposed in this paper can achieve results that are closer to the real poems than the two comparison methods in terms of automatic evaluation indicators.

[0135] (3) Qualitative analysis

[0136] 3.1 Generating Poems

[0137] Figure 7One of the results of writing poetry using a pre-trained unimodal language model: Figure 7 In the examples, the imagery of "rain" and "the sound of the stream" in the generated sentences matches the original poems, and the overall emotional tone is one of homesickness and sadness. The generated poems have neat parallel structures, such as "one night" paired with "a curtain." The tones rhyme, such as "cold" paired with "sound."

[0138] This demonstrates that pre-trained unimodal language models possess the ability to accurately grasp the emotional tone and artistic conception of poetry, and to output beautiful and melodious poems.

[0139] To compare the poetry writing effect of the method proposed in this invention with the aforementioned comparative methods, four examples were selected for demonstration, as shown in Figure 8.

[0140] exist Figure 8 In the examples, compared to the two contrasting methods, IPGCT is able to capture more of the image's atmosphere. For instance, in the first image result, "Jiangnan" corresponds to the atmosphere of the image background, "Jianghu" corresponds to the lake in the image, "a single oar" corresponds to the oar in the image, and "evening smoke" and "rain song" correspond to the misty rain on the lake in the image. Furthermore, the poems generated by the IPGCT method are also stronger in terms of coherence and fluency.

[0141] This invention randomly selected eight lines of poetry generated by the IPGCT model and eight lines of poetry written by ancient people for a double-blind experiment. A questionnaire survey was conducted among a non-professional population, asking participants to determine whether each line of poetry was machine-generated or written by an ancient person. Based on the completed questionnaires, this invention calculated a confusion matrix, the results of which are as follows: Figure 9 As shown.

[0142] As can be seen from the above figure, for the average person, the probability that the poem written by the model of this invention will be recognized as machine-generated is even lower than that of human-written poetry, and vice versa. This indicates that the current poetry-writing results have reached a relatively good level for non-professionals.

[0143] 3.2 Visualization of Feature Extraction by Different Pre-trained CNNs

[0144] To verify that the three CNN networks respectively focus on different aspects of image feature information, this invention... Figure 9 Grad-CAM experiments were conducted on the examples in the example, and the experimental results are as follows: Figure 10 As shown, from Figure 10As can be seen, the three CNNs—object, scene, and emotion—focus on different regions. Object CNN focuses on local areas, such as the location of the "oar" in the image; scene CNN focuses on the background information of the image, such as the lake surface; while emotion CNN understands the implied emotional color of the image from a global perspective.

[0145] 3.3 Ablation Experiment

[0146] To further investigate the impact of the three CNNs, the number of Transformer layers, and the pre-trained unimodal language model on the generated results in the method of this invention, the experimental setup was kept the same in all other respects, and the method was tested on the same test set using automatic evaluation metrics. The specific comparison results are shown in Table 4.

[0147] Table 4 Ablation Experiment Results

[0148]

[0149] Comparing the results in Table 4, it can be seen that using different CNNs to extract image context has a positive impact on improving the correlation between the generated poem and the image. As the number of layers in the Transformer network increases, the quality of the generated poem first improves and then decreases, achieving the best generation effect when the Transformer network has 3 layers. Finally, the addition of a pre-trained unimodal language model also significantly improves the quality of the generated poem.

[0150] in conclusion

[0151] Generating classical Chinese poems from images is a highly innovative and challenging topic. This invention proposes a method for generating poems from images by combining multiple CNNs and Transformer networks. Since this topic has not been explored before, this invention collects the first image-poem pairing dataset and proposes average semantic similarity as a new automatic evaluation metric for the quality of generated poems. Experimental results show that the average semantic similarity between the poems generated by the proposed method and the original paired poems can reach over 75%. Furthermore, this invention conducts a double-blind experiment with a non-professional population through questionnaires. Human evaluation results show that the machine-generated poems are already indistinguishable to non-professionals.

[0152] The method of this invention cannot yet control the style and genre of the generated poems. In future work, a variational autoencoder will be introduced to convert high-dimensional data into a low-dimensional manifold. By continuously changing the low-dimensional manifold, the style of the generated poems can be controlled, achieving a more human-like effect in poetry generation.

[0153] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0154] Although this application has been described herein in conjunction with various embodiments, those skilled in the art will understand and implement other variations of the disclosed embodiments by reviewing the accompanying drawings, the disclosure, and the appended claims in carrying out the claimed application. In the claims, the word "comprising" does not exclude other components or steps, and "a" or "an" does not exclude a plurality.

[0155] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various simple deductions or substitutions can be made without departing from the concept of the present invention, and all such modifications and substitutions should be considered within the scope of protection of the present invention.

Claims

1. A method for image-based mood extraction and mood transformation in classical Chinese poetry based on deep learning, characterized in that... include: Step 1: Receive the user task of generating an ancient poem and extract the target image submitted by the user from the user task; Step 2: Input the target image into a trained cross-modal poetry generator to generate the corresponding ancient poem for the image submitted by the user; The training process of the pre-trained cross-modal poetry generator is as follows: (1) Obtain an open-source ancient poetry dataset from an open-source database and perform keyword preprocessing to obtain high-frequency keywords after filtering; wherein, the open-source ancient poetry dataset includes relevant information about the ancient poems and multiple keywords corresponding to each ancient poem; (2) Match the multiple keywords corresponding to each ancient poem with the high-frequency keywords. If only one keyword does not match, retain the relevant information about the ancient poem and the matched keywords to obtain a dataset to be paired; (3) Concatenate the keywords of the same ancient poem in the dataset to be paired as search keywords, and obtain multiple initial images that are paired with the ancient poems from the network according to the search keywords. (4) The initial matching images are filtered according to the relevant information of the ancient poems to obtain the images paired with each ancient poem according to the degree of correlation. Each ancient poem and its paired image are used as a sample data, and all sample data are combined into an image-ancient poem pairing dataset. (5) The multi-dimensional feature vectors of the image-ancient poem pairing dataset are extracted and reshaped into feature sequences. The feature sequences are used as the input of the cross-modal poetry generator. The model parameters of the pre-trained single-modal language model are imported into the preset cross-modal poetry generator. Then, the cross-modal poetry generator is fine-tuned and trained to complete the pre-training process of the cross-modal poetry generator.

2. The method for image-based mood extraction and mood transformation of classical Chinese poetry based on deep learning according to claim 1, characterized in that, (1) Includes: (11) Obtain an open-source ancient poetry dataset from an open-source database; (12) Count the frequency of each keyword in the open-source ancient poetry dataset and select high-frequency keywords according to the proportion; (13) Replace the uncommon keywords in the open-source ancient poetry dataset with keywords with the same meaning; (14) Delete the high-frequency keywords abstracted from the open-source ancient poetry dataset to obtain the filtered high-frequency keywords.

3. The method for image-based artistic conception mining and artistic conception transformation of ancient Chinese poetry based on deep learning according to claim 1, characterized in that, (3) Includes: (31) Concatenate the keywords of the same ancient poem in the dataset to be paired, and use the concatenated keywords as search keywords; (32) Based on the search keywords, crawler technology is used to crawl multiple initial images that are paired with ancient poems from publicly available image databases on the Internet.

4. The method for image-based artistic conception mining and artistic conception transformation of ancient Chinese poetry based on deep learning according to claim 1, characterized in that, (4) Includes: (41) The initial matching images are manually screened according to the relevant information of the ancient poems to obtain images matched according to the relevance of each ancient poem; (42) Each ancient poem and its paired image are used as a sample data, and all sample data are combined into an image-ancient poem pairing dataset.

5. The method for image-based mood extraction and mood transformation of classical Chinese poetry based on deep learning according to claim 1, characterized in that, The three feature extraction networks in (4) are implemented through the following steps: a. Obtain the Imagent-1k dataset, Place205 dataset, and Sentibank dataset respectively; b. Train a pre-defined VGG-16 network on the Imagent-1k dataset, a pre-defined Place205-VGGNet on the Place205 dataset, and a pre-defined DCAN network on the Sentibank dataset to obtain three feature extraction networks.

6. The method for image-based mood mining and mood transformation of classical Chinese poetry based on deep learning according to claim 5, characterized in that, (5) Includes: (51) Use three feature extraction networks to extract the feature vector of each sample data respectively; (52) The three feature vectors of the same sample data are concatenated to reshape the concatenated feature vectors into a feature sequence. (53) Calculate the position code for each sample data, add the position code to the reconstructed vector, and use the sum as the input to the preset cross-modal poetry generator; (54) Select seven-character quatrains from the open-source ancient poetry dataset and clean the selected seven-character quatrains to obtain a single-modal ancient poetry dataset. (55) The previous line of each ancient poem in the single-modal ancient poem dataset is used as the input of the encoder of the pre-set single-modal language model, and the next line is used as the input of the decoder of the single-modal language model. The single-modal language model is trained to learn the grammatical information of the ancient poems to obtain the pre-trained single-modal language model. The parameters of the self-attention layer and the feedforward neural network layer of the decoder in the pre-trained single-modal language model are retained. (56) The model parameters of the pre-trained single-modal language model are imported into the preset cross-modal poetry generator, and then the cross-modal poetry generator is fine-tuned and trained to complete the training process of the cross-modal poetry generator.

7. The method for image-based artistic conception mining and artistic conception transformation of ancient Chinese poetry based on deep learning according to claim 6, characterized in that, The feature vectors of each sample data extracted by the three feature extraction networks are concatenated and reshaped to obtain the feature sequence X = (X1, X2, ..., X...). n ); in, n is the number of feature sequences, and l is the length of the feature sequences; The vector representation of the actual paired ancient poems is as follows: Y=(Y1,Y2,…,Y m ) (1) Where m is the total number of characters in the ancient poem, and s is the size of the vocabulary list; The predicted vector representation of the ancient poems is as follows: Will The character corresponding to the element with the largest median value is used as the output character, and the m characters are concatenated together to generate the ancient poem.

8. The method for image-based mood mining and mood transformation of classical Chinese poetry based on deep learning according to claim 7, characterized in that, In (52), the reshaped shape is (1,24,512), and the cascade reshaping process is shown below: V=Concat{CNN1(I),CNN2(I),CNN3(I)} (5) TE = Reshape(V) (6) Where I represents the original image, V represents the concatenated feature vector, and TE represents the reconstructed features; The formula for calculating the location code in (53) is: IE = PE + TE (9) Where pos represents the position of the token in the entire sequence, i represents the i-th position of the position code, 2i represents all even positions, 2i+1 represents all odd positions, and d model The dimension represents the model, which is 512 dimensions in this case. PE represents positional encoding, and IE represents the input of the encoder to the cross-modal poetry generator.

9. The method for image-based artistic conception mining and artistic conception transformation of ancient Chinese poetry based on deep learning according to claim 1, characterized in that, The method for image-based mood extraction and mood transformation of classical Chinese poetry based on deep learning also includes: (6) Use the test set input to train a cross-modal poetry generator to generate ancient poems and automatically evaluate the quality of the ancient poems.

10. The method for image-based artistic conception mining and artistic conception transformation of ancient Chinese poetry based on deep learning according to claim 9, characterized in that, (6) Includes: (61) Obtain a test set consisting of multiple images; Each image is paired with a real ancient poem; (62) A trained cross-modal poetry generator was used to generate an ancient poem for each image in the test set, and the open-source BERT-CCPoem model was called to convert the ancient poems into 512-dimensional ancient poem vectors. i ; (63), calculate and generate the ancient poem vector. i and real ancient Chinese poem vectors i cosine similarity (cos sim) i ; (64) Calculate the cosine similarity (cos sim) for each sample in all test sets. i The summation and average are used as the automatic evaluation standard for the quality of the generated ancient poems (auto evaluate). (65) The quality of the generated ancient poems is evaluated using the auto evaluate standard.

Images