Ancient poetry generation method, device and storage medium
The method addresses the challenges of generating coherent and flexible Chinese ancient poems by pre-training a text generation model with an unsupervised corpus and fine-tuning it with picture poetry samples, using Faster Mask-RCNN and GPT2 with self-attention, enhancing conversion efficiency and poem quality.
Patent Information
- Application Number
- CN202011025268.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-09-25
- Publication Date
- 2024-12-13
- Estimated Expiration
- 2040-09-25
AI Technical Summary
Existing machine learning technologies for generating Chinese ancient poems face challenges such as high keyword selection requirements, lack of coherence, and rigidity in generated poems, due to the use of deep convolutional neural networks and long short-term memory networks that are prone to gradient vanishing and require costly large-scale training data.
A method involving pre-training a text generation model with an unsupervised text corpus and fine-tuning it with picture poetry samples, using a detection model like Faster Mask-RCNN to extract feature vectors, and employing a GPT2 model with a self-attention mechanism to generate coherent and flexible ancient poems.
The method improves conversion efficiency and generates ancient poems with good coherence and flexibility by leveraging unsupervised training and self-attention mechanisms, reducing the need for costly large-scale training data.
Smart Images
Figure CN112036192B_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the technical field of machine learning, and in particular to a method, device and storage medium for generating ancient poems. Background Art
[0002] In the field of machine learning technology, machines automatically generate Chinese ancient poems and image descriptions, but there are problems such as high requirements for the selection of keywords, which creates obstacles for many ordinary users, lack of coherence between the generated sentences, and the generated target poems are too rigid and lack flexibility.
[0003] The related technology uses deep convolutional neural networks and long short-term memory networks to establish a multimodal recurrent neural network, so that the computer can automatically convert pictures into ancient Chinese poems that can describe the pictures. Summary of the invention
[0004] In order to overcome the problems existing in the related art, the present disclosure provides a method, device and storage medium for generating ancient poems.
[0005] According to a first aspect of an embodiment of the present disclosure, there is provided a method for generating ancient poems, comprising:
[0006] Get the image;
[0007] Extracting a feature vector of a target object in the image;
[0008] The feature vector is input into a text generation model to obtain ancient poems output by the text generation model, wherein the text generation model is pre-trained with unsupervised text corpus and trained with picture poetry samples, and the picture poetry samples include picture samples and ancient poems serving as labels of the picture samples.
[0009] Optionally, the training of the text generation model includes:
[0010] Obtaining the unsupervised text corpus and picture poetry samples, wherein the unsupervised text corpus includes ancient poetry;
[0011] Performing unsupervised pre-training on a text generation model using the unsupervised text corpus;
[0012] After the pre-training is completed, the text generation model is trained using the picture poetry samples until the text generation model converges to obtain a trained text generation model.
[0013] Optionally, the picture poetry samples include a training set and a verification set, and fine-tuning the text generation model by using the picture poetry samples until the text generation model converges includes:
[0014] Training the text generation model for a preset number of times using the training set, and obtaining the cross entropy of the text generation model after each training using the validation set;
[0015] The text generation model corresponding to the smallest cross entropy is used as the trained text generation model.
[0016] Optionally, the text generation model is a GPT2 model, and the decoder in the GPT2 model is a decoder using a self-attention mechanism.
[0017] Optionally, extracting a feature vector of the target object in the image includes:
[0018] The image is input into a detection model to obtain a feature vector of a target object in the image output by the detection model, wherein the detection model is used to detect the target object in the image and extract the feature vector of the target object.
[0019] Optionally, the detection model is a Faster Mask-RCNN model, and the Faster Mask-RCNN model extracts the feature vector of the target object including: extracting feature vectors for multiple target objects according to a mean pooling method.
[0020] According to a second aspect of an embodiment of the present disclosure, there is provided an ancient poetry generation device, comprising:
[0021] An acquisition module, configured to acquire images;
[0022] An execution module, configured to extract a feature vector of a target object in the image;
[0023] The control module is configured to input the feature vector into a text generation model to obtain ancient poems output by the text generation model, wherein the text generation model is pre-trained with unsupervised text corpus and trained with picture poetry samples, and the picture poetry samples include picture samples and ancient poems serving as labels of the picture samples.
[0024] Optionally, the device further comprises a training module, wherein the training module is configured to obtain the unsupervised text corpus and picture poetry samples, wherein the unsupervised text corpus comprises ancient poetry;
[0025] Performing unsupervised pre-training on a text generation model using the unsupervised text corpus;
[0026] After the pre-training is completed, the text generation model is trained using the picture poetry samples until the text generation model converges to obtain a trained text generation model.
[0027] Optionally, the training module is configured to train the text generation model for a preset number of times through the training set of the picture poetry samples, and obtain the cross entropy of the text generation model after each training through the validation set of the picture samples;
[0028] The text generation model corresponding to the smallest cross entropy is used as the trained text generation model.
[0029] Optionally, the text generation model is a GPT2 model, and the decoder in the GPT2 model is a decoder using a self-attention mechanism.
[0030] Optionally, the execution module is configured to input the image into a detection model to obtain a feature vector of the target object in the image output by the detection model, wherein the detection model is used to detect the target object in the image and extract the feature vector of the target object.
[0031] Optionally, the detection model is a Faster Mask-RCNN model, and the Faster Mask-RCNN model extracts the feature vector of the target object including: extracting feature vectors for multiple target objects according to a mean pooling method.
[0032] According to a third aspect of an embodiment of the present disclosure, there is provided an ancient poetry generation device, comprising:
[0033] processor;
[0034] a memory for storing processor-executable instructions;
[0035] Wherein, the processor is configured to:
[0036] Get the image;
[0037] Extracting a feature vector of a target object in the image;
[0038] The feature vector is input into a text generation model to obtain ancient poems output by the text generation model, wherein the text generation model is pre-trained with unsupervised text corpus and trained with picture poetry samples, and the picture poetry samples include picture samples and ancient poems serving as labels of the picture samples.
[0039] According to a fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided, on which computer program instructions are stored. When the program instructions are executed by a processor, the steps of the ancient poetry generation method provided in the first aspect of the present disclosure are implemented.
[0040] The technical solution provided by the embodiments of the present disclosure may include the following beneficial effects: first extract the feature vector of the target object in the image, input the feature vector into the text generation model, so that the text generation model can perform targeted conversion between the target object and the picture and ancient poetry, improve the conversion efficiency, get rid of the limitation of the subject words, and obtain the ancient poetry output by the text generation model. The text generation model is pre-trained with unsupervised text corpus and trained with picture poetry samples, so that the ancient poetry output by the text generation model has good coherence and flexibility.
[0041] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS
[0042] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.
[0043] Figure 1 The present invention is a flowchart of a method for generating ancient poems according to an exemplary embodiment.
[0044] Figure 2 The figure is a schematic diagram of a device for using an ancient poetry generation method according to an exemplary embodiment.
[0045] Figure 3 The present invention is a block diagram of a device for generating ancient poems according to an exemplary embodiment.
[0046] Figure 4 The present invention is a block diagram of a device for generating ancient poems according to an exemplary embodiment. DETAILED DESCRIPTION
[0047] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.
[0048] It should be noted that in the present disclosure, the terms "S110", "S120" and the like in the specification, claims and drawings are used to distinguish steps, and do not necessarily mean that the method steps are performed in a specific order or sequential sequence.
[0049] Regarding the automatic generation of Chinese ancient poems and image descriptions by machines, the inventors found that the related technology uses a deep convolutional neural network and a long short-term memory network to establish a multimodal recurrent neural network, so that the computer can automatically convert pictures into Chinese ancient poems that can describe the pictures. Although it gets rid of the limitation of keywords, the LSTM decoder used to establish the multimodal virtual neural network is prone to the gradient vanishing problem of the LSTM decoder during the image-text conversion process, making the multimodal virtual neural network unable to accurately capture the information in the picture. In addition, the LSTM decoder does not use large-scale corpus for training, resulting in the poems of the multimodal virtual neural network conversion results being too rigid and lacking in flexibility. The need to obtain a large number of pictures and corpus describing the sentences corresponding to the picture sources will result in excessively high construction costs of the multimodal virtual neural network.
[0050] In view of this, the embodiments of the present disclosure provide a method, device and storage medium for generating ancient poems, wherein the execution subject of the method for generating ancient poems can be a terminal device, such as a smart phone, a tablet device, a personal computer, etc. The execution subject of the method for generating ancient poems can also be a cloud server.
[0051] Figure 1 is a flow chart of a method for generating ancient poems according to an exemplary embodiment. Figure 1 As shown, the following steps are included.
[0052] In step S110, a picture is obtained.
[0053] Wherein, when the execution subject is a terminal device, step S110 may be a picture taken by the terminal device through its own camera, or may be a picture sent to the terminal device by other devices through network communication. When the execution subject is a server, step S110 may be a picture received by the server from other devices, such as mobile phones, tablet computers and other terminal devices.
[0054] In step S120, a feature vector of the target object in the image is extracted.
[0055] In step S130, the feature vector is input into a text generation model to obtain ancient poems output by the text generation model, wherein the text generation model is pre-trained with unsupervised text corpus and trained with picture poetry samples, and the picture poetry samples include picture samples and ancient poems serving as labels of the picture samples.
[0056] Specifically, the text generation model is pre-trained with unsupervised text corpus and trained with picture poetry samples. It can be understood that the output layer of the text generation model trained with a large amount of unsupervised corpus is removed, and the remaining network is used as a feature extractor as a model for training icon poetry samples.
[0057] Specifically, after pre-training, training the text generation model through picture and poetry samples is to fine-tune the text generation model. It can be understood that the model structure of the pre-trained model remains unchanged, all the weights of the model are randomly reinitialized, and the pre-trained text generation model is partially trained according to the picture and poetry samples. The weights of some layers of the pre-trained text generation model are locked and kept unchanged, and the unlocked layers in the text generation model are retrained to obtain new weights to achieve fine-tuning of the text generation model.
[0058] The unsupervised text corpus may be a large number of ancient poems or classical Chinese texts including various types; the picture poem samples may be a small number of picture samples and ancient poems or classical Chinese texts as labels of the picture samples, or picture samples and classical Chinese texts as labels of the picture samples (i.e., supervised text corpus, such as <picture, ancient poem> pairs). This disclosure does not make specific limitations on this.
[0059] The ancient poetry generation method disclosed in this embodiment extracts the feature vector of the target object in the image and inputs the feature vector into the text generation model, so that the text generation model can perform targeted conversion between the target object and the image and ancient poetry, improve the conversion efficiency, get rid of the limitation of the subject words, and obtain the ancient poetry output by the text generation model. The text generation model is pre-trained with unsupervised text corpus and trained with picture poetry samples, so that the ancient poetry output by the text generation model has good coherence and flexibility.
[0060] Optionally, in step S120, extracting a feature vector of the target object in the image may include:
[0061] The image is input into a detection model to obtain a feature vector of a target object in the image output by the detection model, wherein the detection model is used to detect the target object in the image and extract the feature vector of the target object.
[0062] The target object may be a person, such as a girl, a young man, an old man, etc.; or it may be an object, such as a pine tree, a boat, a lotus, a loft, etc., and the present disclosure does not make any specific limitation on this.
[0063] Optionally, the detection model in step S120 may be a Faster Mask-RCNN model, and extracting the feature vector of the target object by the Faster Mask-RCNN model may include: extracting feature vectors for multiple target objects according to a mean pooling method.
[0064] Specifically, when the Faster Mask-RCNN model detects that there are multiple target objects in the picture, the Faster Mask-RCNN model can extract feature vectors of the multiple target objects according to the mean pooling method.
[0065] For example, a picture containing a lotus, a loft, and a girl is input into the Faster Mask-RCNN model. The Faster Mask-RCNN model detects that there are three target objects in the picture: lotus, loft, and girl. The Faster Mask-RCNN model can extract the feature vectors of the three target objects according to the mean pooling method, and output the feature vectors of the three target objects: lotus, loft, and girl.
[0066] Because it is costly and labor-intensive to obtain a large number of picture and poetry samples to implement a text generation model, while it is costly and convenient to obtain a large amount of unsupervised corpus, the training process of the text generation model in the ancient poetry generation method includes: first pre-training the text generation model using a large amount of unsupervised corpus, and then fine-tuning the pre-trained text generation model using a small amount of picture and poetry samples to complete the training of the text generation model.
[0067] Optionally, the training of the text generation model in step S130 may include:
[0068] Obtaining the unsupervised text corpus and picture poetry samples, wherein the unsupervised text corpus includes ancient poetry;
[0069] Performing unsupervised pre-training on a text generation model using the unsupervised text corpus;
[0070] After the pre-training is completed, the text generation model is trained using the picture poetry samples until the text generation model converges to obtain a trained text generation model.
[0071] For fine-tuning of the text generation model, the structure of the pre-trained text generation model can be kept unchanged, the output layer of the model can be removed, the remaining network can be used as a feature extractor, all the weights of the model can be randomly re-initialized, and the pre-trained text generation model can be partially trained according to the picture and poetry samples to achieve fine-tuning of the text generation model.
[0072] Specifically, the text generation model includes an input layer, an encoding component, a decoding component, a linear transformation layer, a logistic regression layer and an output layer.
[0073] The encoding component includes a plurality of encoders, each of which includes a self-attention layer and a feedforward layer; the decoding component includes a plurality of decoders, each of which includes a self-attention layer, an attention layer, and a feedforward layer. The number of encoders in the encoding component is the same as the number of decoders in the decoding component, and this is not specifically limited in the present disclosure.
[0074] For example, when the unsupervised text corpus includes a large number of various ancient poems, the unsupervised text corpus is input into the text generation model so that the text generation model can output a variety of ancient poems and realize unsupervised pre-training of the text generation model through the unsupervised text corpus. The pre-trained text generation model can generate various types of ancient poems and get rid of the limitation of subject words; the model structure of the pre-trained text generation model remains unchanged, the output layer of the model is removed, and the network consisting of the remaining input layer, encoding component, decoding component, linear transformation layer and logistic regression layer is regarded as a A feature extractor is used to randomly initialize all the weights of the input layer, encoding component, decoding component, linear transformation layer and logistic regression layer in the model, and the picture poetry samples are input into the pre-trained text generation model. The weights of the input layer, encoding component, decoding component, linear transformation layer and logistic regression layer are adjusted according to the picture poetry samples, so that the text generation model can output a variety of ancient poetry texts according to the pictures, and fine-tune the pre-trained text generation model according to the picture poetry samples to obtain a trained text generation model, so that the text generation model can generate various ancient poems according to pictures.
[0075] For fine-tuning of the text generation model, the weights of some layers of the pre-trained text generation model can also be locked to remain unchanged, and the unlocked layers in the text generation model can be retrained to obtain new weights to achieve fine-tuning of the text generation model.
[0076] For example, when the unsupervised text corpus includes a large number of various ancient poems, the unsupervised text corpus is input into the text generation model so that the text generation model can output a variety of ancient poetry texts, and unsupervised pre-training of the text generation model is achieved through the unsupervised text corpus, so that the pre-trained text generation model can generate various types of ancient poems and get rid of the limitation of keywords; the weights of the starting layer (such as the self-attention layer of the decoder) of the pre-trained text generation model are locked and remain unchanged, and the picture poetry samples are input into the pre-trained text generation model, and the weights of the unlocked layers in the text generation model (such as the input layer, decoding component, linear transformation layer, logistic regression layer and output layer) are readjusted to obtain new weights, so as to achieve fine-tuning of the text generation model so that the text generation model can generate various types of ancient poems based on pictures.
[0077] Optionally, the picture poetry samples may include a training set and a verification set.
[0078] Optionally, fine-tuning the text generation model using the picture poetry samples until the text generation model converges may include:
[0079] Training the text generation model for a preset number of times using the training set, and obtaining the cross entropy of the text generation model after each training using the validation set;
[0080] The text generation model corresponding to the smallest cross entropy is used as the trained text generation model.
[0081] Among them, training the text generation model for a preset number of times through the training set, and verifying the text generation model after each training through the verification set can be performed simultaneously or alternately. It is also possible that after the text generation model is trained for a certain number of times with the training set, the text generation model can be verified through the verification set until the text generation model is trained for a preset number of times with the training set.
[0082] Among them, the preset number of times can be preset according to the normal number of conversions between pictures and ancient poems, and the present disclosure does not make specific limitations on this.
[0083] For example, when the preset training times of the text generation model is 3000 times, the text generation model is trained 1000 times with the training set, and each training of the text generation model is verified with the verification set. The text generation model is continued to be trained 1000 times with the training set, and each training of the text generation model is verified with the verification set. Finally, the text generation model is continued to be trained 1000 times with the training set, and each training of the text generation model is verified with the verification set. The cross entropy of the text generation model after each training is obtained, and the text generation model obtained from the 3000 trainings of the text generation model with the smallest cross entropy is used as the trained text generation model. At this time, the text generation model reaches convergence, and the effect of ancient poetry obtained by the text generation model is best.
[0084] Optionally, the text generation model may be a GPT2 model, and the decoder in the GPT2 model may be a decoder using a self-attention mechanism.
[0085] Specifically, the decoder of the GPT2 (Generative Pre-Training 2, GPT2 for short) model adopts a self-attention mechanism decoder, which can better overcome the problem of gradient vanishing in the process of converting pictures and ancient poems. That is, it ensures that each word generated by the GPT2 model will be connected with the already generated words in O(1), so that the ancient poems output by the text generation model have good coherence.
[0086] Take smartphones as an example. Figure 2 As shown, the camera of the smartphone captures a picture as shown in the upper part of the display interface, which includes a fisherman wearing a straw raincoat and a pine tree on a hillside, and the ancient poem generated corresponding to the picture is displayed in the text box below the display interface of the smartphone, "An old man in a straw raincoat and a straw hat, fishing alone in the cold mountain snow."
[0087] Figure 3 1 is a block diagram of an ancient poetry generation device according to an exemplary embodiment. The ancient poetry generation device can be a terminal device or a server, and is used to execute the ancient poetry generation method provided by the above method embodiment. Figure 3 The ancient poetry generation device 200 includes an acquisition module 201, an execution module 202 and a control module 203.
[0088] The acquisition module 201 is configured to acquire a picture;
[0089] The execution module 202 is configured to extract a feature vector of a target object in the image;
[0090] The control module 203 is configured to input the feature vector into a text generation model to obtain ancient poetry output by the text generation model, wherein the text generation model is pre-trained with unsupervised text corpus and trained with picture poetry samples, and the picture poetry samples include picture samples and ancient poetry serving as labels of the picture samples.
[0091] The ancient poetry generation device disclosed in this embodiment extracts the feature vector of the target object in the image and inputs the feature vector into the text generation model, so that the text generation model can perform targeted conversion between the target object and the ancient poetry, improve the conversion efficiency, get rid of the limitation of the subject words, and obtain the ancient poetry output by the text generation model. The text generation model is pre-trained with unsupervised text corpus and trained with picture poetry samples, so that the ancient poetry output by the text generation model has good coherence and flexibility.
[0092] Optionally, the text generation model 200 is a GPT2 model, and the decoder in the GPT2 model is a decoder using a self-attention mechanism.
[0093] Optionally, the execution module 202 is configured to input the image into a detection model to obtain a feature vector of the target object in the image output by the detection model, wherein the detection model is used to detect the target object in the image and extract the feature vector of the target object.
[0094] Optionally, the detection model is a Faster Mask-RCNN model, and the Faster Mask-RCNN model extracts the feature vector of the target object including: extracting feature vectors for multiple target objects according to a mean pooling method.
[0095] Optionally, the device further comprises a training module, wherein the training module is configured to obtain the unsupervised text corpus and picture poetry samples, wherein the unsupervised text corpus comprises ancient poetry;
[0096] Performing unsupervised pre-training on a text generation model using the unsupervised text corpus;
[0097] After the pre-training is completed, the text generation model is trained using the picture poetry samples until the text generation model converges to obtain a trained text generation model.
[0098] Optionally, the training module is configured to train the text generation model for a preset number of times through the training set of the picture poetry samples, and obtain the cross entropy of the text generation model after each training through the validation set of the picture samples;
[0099] The text generation model corresponding to the smallest cross entropy is used as the trained text generation model.
[0100] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.
[0101] The present disclosure also provides a computer-readable storage medium on which computer program instructions are stored. When the program instructions are executed by a processor, the steps of the ancient poetry generation method provided by the present disclosure are implemented.
[0102] This embodiment provides an ancient poetry generation device, which may be a terminal device or a server, and includes:
[0103] processor;
[0104] a memory for storing processor-executable instructions;
[0105] Wherein, the processor is configured to:
[0106] Get the image;
[0107] Extracting a feature vector of a target object in the image;
[0108] The feature vector is input into a text generation model to obtain ancient poems output by the text generation model, wherein the text generation model is pre-trained with unsupervised text corpus and trained with picture poetry samples, and the picture poetry samples include picture samples and ancient poems serving as labels of the picture samples.
[0109] The ancient poetry generation device disclosed in this embodiment extracts the feature vector of the target object in the image and inputs the feature vector into the text generation model, so that the text generation model can perform targeted conversion between the target object and the ancient poetry, improve the conversion efficiency, get rid of the limitation of the subject words, and obtain the ancient poetry output by the text generation model. The text generation model is pre-trained with unsupervised text corpus and trained with picture poetry samples, so that the ancient poetry output by the text generation model has good coherence and flexibility.
[0110] Figure 4 is a block diagram of an ancient poetry generation device 300 according to an exemplary embodiment. Figure 4 , the device 300 may include one or more of the following components: a processing component 302 , a memory 304 , a power component 306 , a multimedia component 308 , an audio component 310 , an input / output (I / O) interface 312 , a sensor component 314 , and a communication component 316 .
[0111] The processing component 302 generally controls the overall operation of the device 300, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 302 may include one or more processors 320 to execute instructions to complete all or part of the steps of the above-mentioned ancient poetry generation method. In addition, the processing component 302 may include one or more modules to facilitate the interaction between the processing component 302 and other components. For example, the processing component 302 may include a multimedia module to facilitate the interaction between the multimedia component 308 and the processing component 302.
[0112] The memory 304 is configured to store various types of data to support the operation of the device 300. Examples of such data include instructions for any application or method operating on the device 300, unsupervised text corpus pre-training, and poetry samples through pictures, pictures, etc. The memory 304 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.
[0113] The power component 306 provides power to the various components of the device 300. The power component 306 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the device 300.
[0114] The multimedia component 308 includes a screen that provides an output interface between the device 300 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to display the ancient poems output by the text generation model. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 308 includes a front camera and / or a rear camera. When the device 300 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and the rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.
[0115] The audio component 310 is configured to output and / or input audio signals. For example, the audio component 310 includes a microphone (MIC), and when the device 300 is in an operating mode, such as a camera mode or an ancient poetry browsing mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 304 or sent via the communication component 316. In some embodiments, the audio component 310 also includes a speaker for outputting audio signals.
[0116] I / O interface 312 provides an interface between processing component 302 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: a home button, a volume button, a start button, and a lock button.
[0117] Sensor assembly 314 includes one or more sensors for providing various aspects of status assessment for device 300. For example, sensor assembly 314 can detect the open / closed state of device 300, the relative positioning of components, such as the display and keypad of device 300, the position change of device 300 or a component of device 300, and the presence or absence of user contact with device 300. Sensor assembly 314 can also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications.
[0118] The communication component 316 is configured to facilitate wired or wireless communication between the device 300 and other devices. The device 300 can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component 316 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 316 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.
[0119] In an exemplary embodiment, the device 300 can be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors or other electronic components to execute the above-mentioned ancient poetry generation method.
[0120] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 304 including instructions, and the instructions can be executed by the processor 320 of the device 300 to complete the above-mentioned ancient poetry generation method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.
[0121] In another exemplary embodiment, a computer program product is also provided, which includes a computer program that can be executed by a programmable device, and the computer program has a code portion for executing the above-mentioned ancient poetry generation method when executed by the programmable device.
[0122] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the present disclosure. This application is intended to cover any variations, uses or adaptations of the present disclosure, which follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not disclosed in the present disclosure. The specification and examples are to be considered as exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.
[0123] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.
Claims
1. A method for generating ancient poems, characterized in that: The method comprises: Get the image; Extracting a feature vector of a target object in the image; Inputting the feature vector into a text generation model to obtain an ancient poem output by the text generation model, wherein the text generation model is pre-trained with unsupervised text corpus and trained with picture poem samples, wherein the picture poem samples include picture samples and ancient poems as labels of the picture samples; The training of the text generation model includes: Acquire the unsupervised text corpus and the picture poetry samples, wherein the unsupervised text corpus includes a first number of ancient poems and / or a first number of classical Chinese texts, and the picture poetry samples include a second number of picture samples and ancient poems as labels of the picture samples, and / or a second number of picture samples and classical Chinese texts as labels of the picture samples, wherein the first number is greater than the second number; Performing unsupervised pre-training on a text generation model using the unsupervised text corpus; After the pre-training is completed, the text generation model is trained using the picture poetry samples until the text generation model converges to obtain a trained text generation model.
2. The method according to claim 1, characterized in that: The picture poetry samples include a training set and a verification set, and fine-tuning the text generation model by using the picture poetry samples until the text generation model converges includes: Training the text generation model for a preset number of times using the training set, and obtaining the cross entropy of the text generation model after each training using the validation set; The text generation model corresponding to the smallest cross entropy is used as the trained text generation model.
3. The method according to claim 1 or 2, characterized in that: The text generation model is a GPT2 model, and the decoder in the GPT2 model is a decoder using a self-attention mechanism.
4. The method according to claim 1 or 2, characterized in that: The extracting a feature vector of the target object in the picture includes: The image is input into a detection model to obtain a feature vector of a target object in the image output by the detection model, wherein the detection model is used to detect the target object in the image and extract the feature vector of the target object.
5. The method according to claim 4, characterized in that The detection model is a Faster Mask-RCNN model, and the Faster Mask-RCNN model extracts the feature vector of the target object, including: extracting feature vectors for multiple target objects according to a mean pooling method.
6. An ancient poetry generation device, characterized in that: The device comprises: An acquisition module, configured to acquire images; An execution module, configured to extract a feature vector of a target object in the image; A control module is configured to input the feature vector into a text generation model to obtain an ancient poem output by the text generation model, wherein the text generation model is pre-trained with unsupervised text corpus and trained with picture poem samples, wherein the picture poem samples include picture samples and ancient poems as labels of the picture samples; The apparatus further includes a training module, wherein the training module is configured to obtain the unsupervised text corpus and the picture poetry samples, wherein the unsupervised text corpus includes a first number of ancient poems and / or a first number of classical Chinese texts, and the picture poetry samples include a second number of picture samples and ancient poems as labels of the picture samples, and / or a second number of picture samples and classical Chinese texts as labels of the picture samples, wherein the first number is greater than the second number; Performing unsupervised pre-training on a text generation model using the unsupervised text corpus; After the pre-training is completed, the text generation model is trained using the picture poetry samples until the text generation model converges to obtain a trained text generation model.
7. The device according to claim 6, characterized in that The training module is configured to train the text generation model for a preset number of times through the training set of the picture poetry samples, and obtain the cross entropy of the text generation model after each training through the validation set of the picture samples; The text generation model corresponding to the smallest cross entropy is used as the trained text generation model.
8. The device according to claim 6 or 7, characterized in that The text generation model is a GPT2 model, and the decoder in the GPT2 model is a decoder using a self-attention mechanism.
9. The device according to any one of claims 6 or 7, characterized in that The execution module is configured to input the image into a detection model to obtain a feature vector of a target object in the image output by the detection model, wherein the detection model is used to detect the target object in the image and extract the feature vector of the target object.
10. The device according to claim 9, characterized in that The detection model is a Faster Mask-RCNN model, and the Faster Mask-RCNN model extracts the feature vector of the target object, including: extracting feature vectors for multiple target objects according to a mean pooling method.
11. An ancient poetry generation device, characterized in that: include: processor; a memory for storing processor-executable instructions; Wherein, the processor is configured to: Get the image; Extracting a feature vector of a target object in the image; Inputting the feature vector into a text generation model to obtain an ancient poem output by the text generation model, wherein the text generation model is pre-trained with unsupervised text corpus and trained with picture poem samples, wherein the picture poem samples include picture samples and ancient poems as labels of the picture samples; The training of the text generation model includes: Acquire the unsupervised text corpus and the picture poetry samples, wherein the unsupervised text corpus includes a first number of ancient poems and / or a first number of classical Chinese texts, and the picture poetry samples include a second number of picture samples and ancient poems as labels of the picture samples, and / or a second number of picture samples and classical Chinese texts as labels of the picture samples, wherein the first number is greater than the second number; Performing unsupervised pre-training on a text generation model using the unsupervised text corpus; After the pre-training is completed, the text generation model is trained using the picture poetry samples until the text generation model converges to obtain a trained text generation model.
12. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the program instructions are executed by a processor, the steps of the ancient poetry generation method described in any one of claims 1-5 are implemented.