Poem generation method, poem generation device, electronic equipment and readable medium

Abstract

The invention provides a poem generation method and device, electronic equipment and a computer readable medium, and belongs to the technical field of computers. The method comprises the following steps: performing visual information extraction on a target image of a poem to be generated to obtain image description information corresponding to the target image; extracting the geographic position of the target image to obtain geographic description information corresponding to the target image; inputting the image description information and the geographic description information into a preset historical description generation model to obtain historical description information corresponding to the target image; and inputting the image description information, the geographic description information and the historical description information into a preset poem generation model to generate a target poem corresponding to the target image. According to the embodiment of the invention, when the poem is generated, the multi-source information related to the target image can be fused, so that the deep relation and the corresponding relation between the generated target poem and the target image are improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a poetry generation method, a poetry generation device, an electronic device, and a computer-readable storage medium. Background Art

[0002] Image-based poetry generation refers to automatically generating poems that match an image by analyzing and understanding the content of an input image. In traditional poetry generation methods, only visual features in an image can be extracted, and then poems related to the image can be generated based on the visual features.

[0003] However, the above-mentioned poetry generation method can only generate corresponding poems based on the visual content contained in the image itself, resulting in the generated poems containing single information and lacking deep connection with the image, resulting in poor generation effect. Summary of the invention

[0004] The present application provides a poetry generation method, a poetry generation device, an electronic device, and a computer-readable storage medium, which can enhance the deep-level connection and correspondence between the generated target poetry and the target image.

[0005] In a first aspect, the present application provides a poetry generation method, which includes: extracting visual information from a target image for generating poetry, and obtaining image description information corresponding to the target image, wherein the image description information is used to characterize the visual information contained in the target image; extracting the geographic location of the target image, and obtaining geographic description information corresponding to the target image, wherein the geographic description information is used to characterize the geographic entity corresponding to the target image; inputting the image description information and the geographic description information into a preset historical description generation model, and obtaining historical description information corresponding to the target image, wherein the historical description information is used to characterize the historical and cultural information corresponding to the target image; inputting the image description information, the geographic description information and the historical description information into a preset poetry generation model, and generating a target poem corresponding to the target image.

[0006] In a second aspect, the present application provides a poetry generation device, which includes: an image description generation module, which is used to extract visual information from a target image for generating poetry, and obtain image description information corresponding to the target image, wherein the image description information is used to characterize the visual information contained in the target image; a geographic description generation module, which is used to extract the geographic location of the target image, and obtain geographic description information corresponding to the target image, wherein the geographic description information is used to characterize the geographic entity corresponding to the target image; a historical description generation module, which is used to input the image description information and the geographic description information into a preset historical description generation model, and obtain the historical description information corresponding to the target image, wherein the historical description information is used to characterize the historical and cultural information corresponding to the target image; a poetry generation module, which is used to input the image description information, the geographic description information and the historical description information into a preset poetry generation model, and generate a target poem corresponding to the target image.

[0007] In a third aspect, the present application provides an electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores one or more computer programs executable by the at least one processor, and one or more of the computer programs are executed by the at least one processor so that the at least one processor can execute the above-mentioned poetry generation method.

[0008] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the above-mentioned poetry generation method when executed by a processor / processing core.

[0009] In the embodiment provided by the present application, first, visual information is extracted from the target image for generating poetry to obtain image description information corresponding to the target image, wherein the image description information is used to characterize the visual information contained in the target image; secondly, the geographic location is extracted from the target image to obtain geographic description information corresponding to the target image, wherein the geographic description information is used to characterize the geographic entity corresponding to the target image; then, the image description information and the geographic description information are input into a preset historical description generation model to obtain historical description information corresponding to the target image, wherein the historical description information is used to characterize the historical and cultural information corresponding to the target image; finally, the image description information, the geographic description information and the historical description information are input into a preset poetry generation model to generate the target poetry corresponding to the target image. As can be seen from the above, when generating poetry, the embodiment of the present application not only combines the image description information corresponding to the target image, but also obtains the historical description information corresponding to the target image based on the image description information and the geographic description information corresponding to the target image, thereby generating poetry based on the image description information, the geographic description information and the historical description information, so that the generated target poetry can integrate the multi-source information related to the target image, thereby improving the deep connection and correspondence between the generated target poetry and the target image.

[0010] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present application, nor is it intended to limit the scope of the present application. Other features of the present application will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The accompanying drawings are used to provide a further understanding of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the present application and do not constitute a limitation of the present application. The above and other features and advantages will become more apparent to those skilled in the art by describing detailed example embodiments with reference to the accompanying drawings. In the accompanying drawings:

[0012] Figure 1 A flowchart of a poetry generation method provided in an embodiment of the present application;

[0013] Figure 2 A flowchart of a poetry generation method provided in an embodiment of the present application;

[0014] Figure 3 A block diagram of a poetry generation device provided in an embodiment of the present application;

[0015] Figure 4 A block diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0016] In order to enable those skilled in the art to better understand the technical solution of the present application, the exemplary embodiments of the present application are described below in conjunction with the accompanying drawings, including various details of the embodiments of the present application to facilitate understanding, which should be considered as merely exemplary. Therefore, those of ordinary skill in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present application. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0017] In the absence of conflict, the embodiments of the present application and the features therein may be combined with each other.

[0018] As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items.

[0019] The terms used herein are only used to describe specific embodiments and are not intended to limit the present application. As used herein, the singular forms "a" and "the" are also intended to include plural forms, unless the context clearly indicates otherwise. It will also be understood that when the terms "including" and / or "made of" are used in this specification, the presence of the features, wholes, steps, operations, elements and / or components is specified, but the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or groups thereof is not excluded. "Connected" or "connected" and similar words are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect.

[0020] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by those of ordinary skill in the art. It will also be understood that terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and this application, and will not be interpreted as having an idealized or overly formal meaning unless explicitly defined herein.

[0021] In the related art, an image recognition-based method is usually used to generate poems corresponding to an image, that is, by extracting visual features in an image, poems related to the image are generated based on the visual features.

[0022] However, the above-mentioned poetry generation method can only generate corresponding poems based on the visual content contained in the image itself, resulting in the generated poems containing single information and lacking deep connection with the image.

[0023] In view of this, the embodiments of the present application provide a poetry generation method, a poetry generation device, an electronic device, and a readable medium. According to the poetry generation method of the embodiments of the present application, when generating poetry, it is possible to not only combine the image description information corresponding to the target image, but also obtain the historical description information corresponding to the target image based on the image description information and the geographical description information corresponding to the target image, so as to generate poetry based on the image description information, the geographical description information, and the historical description information, so that the generated target poetry can integrate the multi-source information related to the target image, thereby improving the deep connection and correspondence between the generated target poetry and the target image.

[0024] The poetry generation method according to the embodiment of the present application can be performed by an electronic device such as a terminal device or a server, and the terminal device can be a user equipment (UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (PDA), a handheld device, a computing device, a vehicle-mounted device, a wearable device, etc. The method can be implemented by a processor calling a computer-readable program instruction stored in a memory. The server can be an independent physical server, a server cluster composed of multiple servers, or a cloud server capable of cloud computing.

[0025] Figure 1 A flowchart of a poetry generation method provided in an embodiment of the present application. Figure 1 , the method comprising:

[0026] Step S110: extract visual information from the target image for generating poetry, and obtain image description information corresponding to the target image.

[0027] The target image refers to the image of the poem to be generated. Specifically, the target image can be captured by an image acquisition device such as a camera. The visual information in the target image refers to the information presented in a visual form contained in the target image, for example, visual entities such as people, objects, scenes, etc. contained in the target image. By extracting visual information from the target image, the visual information contained in the target image can be extracted, so that the image description information corresponding to the target image can be obtained based on the visual information.

[0028] The image description information is used to characterize the visual information contained in the target image. Specifically, it can describe the key features corresponding to the visual information contained in the target image.

[0029] In an optional implementation, the image description information may describe the visual features corresponding to the visual entities in the visual information and the associated features between different visual entities. For example, the image description information may describe the visual features corresponding to the visual entities, such as the shape, size, length, color, etc. corresponding to the visual entities. For another example, the image description information may describe the associated features between different visual entities, such as the relative positional relationship between different visual entities, etc., thereby describing the key visual information contained in the target image in text form, and obtaining image description information with accurate and rich meanings.

[0030] For example, for a certain target image, the corresponding image description information may be "there is a wide river in the picture, with green trees on the river bank and a temple on the opposite bank".

[0031] In an optional implementation, in order to improve the correlation between the image description information and the target image, visual information of the target image may be extracted based on a preset image-to-text model to obtain image description information corresponding to the target image.

[0032] The preset graph-to-text model refers to a model used to generate image description information corresponding to an image. The preset graph-to-text model can be an existing pre-trained graph-to-text large model such as a Stable Diffusion model, or an optimized model obtained by fine-tuning the pre-trained large model, which is not limited in the present embodiment of the application.

[0033] Step S120: extracting the geographic location of the target image to obtain geographic description information corresponding to the target image.

[0034] The geographic description information is used to characterize the geographic entity corresponding to the target image. Specifically, the geographic entity refers to the geographic area corresponding to the target image in the real world.

[0035] It should be noted that geographic entities can be divided into multiple geographic entity types according to actual needs, such as urban geographic entity types, landscape geographic entity types, etc. Among them, when the geographic entity corresponding to the target image is an urban geographic entity type, the geographic entity refers to the administrative region corresponding to the target object in the real world, such as a municipality, a prefecture-level city, etc. When the geographic entity corresponding to the target image is a landscape geographic entity type, the geographic entity refers to the landscape area corresponding to the target image in the real world, such as an ancient building, a mountain range, a grassland, etc.

[0036] Exemplarily, the geographic description information corresponding to a target image may be the Grand Canal of Beijing, wherein Beijing is a geographic entity of the city geographic entity type corresponding to the target image, and the Grand Canal of Beijing is a geographic entity of the landscape geographic entity type corresponding to the target image.

[0037] The extraction of the geographical location of the target image refers to extracting the geographical location of the shooting location corresponding to the target image in the real world.

[0038] In an optional implementation, in order to improve the accuracy of geographic description information, geographic location data corresponding to the target image can be obtained from the image metadata, so as to accurately match the geographic description information corresponding to the target image in a preset geographic database based on the geographic location data. Accordingly, the geographic location of the target image is extracted to obtain the geographic description information corresponding to the target image, including: extracting the geographic location data corresponding to the target image from the image metadata corresponding to the target image; querying the preset geographic database for first geographic description information matching the geographic location data based on the geographic location data and a preset geographic entity type, wherein the first geographic description information corresponds to the preset geographic entity type; and obtaining the geographic description information corresponding to the target image based on the first geographic description information corresponding to the preset geographic entity type.

[0039] Image metadata refers to the metadata generated by the image acquisition device when shooting the target image. Image metadata includes descriptive data such as shooting parameters, shooting date, shooting location, image size, resolution, etc.

[0040] Accordingly, by extracting the data corresponding to the shooting location from the image metadata, the geographical location data corresponding to the target image can be obtained. The geographical location data represents the geographical location of the shooting location corresponding to the target image in the real world, such as the latitude and longitude coordinates of the shooting location corresponding to the target image.

[0041] The preset geographic entity type refers to the geographic entity type corresponding to the first geographic description information, such as a city geographic entity type, a landscape geographic entity type, and the like.

[0042] The first geographic description information is information corresponding to a preset geographic entity type obtained by matching and querying the geographic location data in a preset geographic database. For example, by matching and querying the longitude and latitude coordinates in the preset geographic database, the city information and scenic spot information corresponding to the longitude and latitude coordinates are obtained. Accordingly, the city information queried is the first geographic description information corresponding to the city geographic entity type, and the scenic spot information queried is the first geographic description information corresponding to the landscape geographic entity type.

[0043] The preset geographic database refers to a preset query database containing different cities, different scenic spots and their corresponding geographic location coordinate information.

[0044] It should be noted that the number of preset geographic entity types may be multiple, so that when there are multiple preset geographic entity types, multiple corresponding first geographic description information can be queried in the preset geographic database.

[0045] It should also be noted that when generating geographic description information corresponding to the target image, when there are multiple first geographic description information, the multiple first geographic description information can be merged to obtain the geographic description information corresponding to the target image. When there is only one first geographic description information, the first geographic description information can be directly used as the geographic description information corresponding to the target image.

[0046] Exemplarily, according to the geographic location data, the first geographic description information corresponding to the urban geographic entity type is Beijing, and the first geographic description information corresponding to the landscape geographic entity type is the Grand Canal. Therefore, the geographic description information finally obtained is "Beijing-Grand Canal".

[0047] In an embodiment of the present application, by directly extracting the geographic location data corresponding to the target image from the image metadata corresponding to the target image, the accurate geographic location of the shooting location corresponding to the target image can be obtained, and by querying the geographic location data in a preset geographic database, the first geographic description information corresponding to the preset geographic entity type can be obtained. Finally, the geographic description information corresponding to the target image can be obtained based on the first geographic description information, which not only improves the accuracy of the geographic description information corresponding to the target image, but also improves the information richness of the geographic description information corresponding to the target image.

[0048] Step S130: input the image description information and the geographic description information into a preset historical description generation model to obtain the historical description information corresponding to the target image.

[0049] The historical description information is used to represent the historical and cultural information corresponding to the target image, for example, historical events, literary poems, and other information corresponding to the target image.

[0050] In an optional implementation, in order to improve the information diversity and richness of the historical description information, the historical description information can be divided according to different historical description types. The historical description type refers to the type corresponding to the content described by the historical description information. Accordingly, the historical description type can include a historical background description type, a historical poetry description type, a historical event description type, etc.

[0051] Specifically, the historical description information corresponding to the historical background description type is used to describe the historical background information corresponding to the target image, such as the background, meaning, origin, etc. of the geographical entity and visual entity corresponding to the target image. The historical description information corresponding to the historical poetry description type is used to describe the historical poetry information corresponding to the target image, such as the historical poetry related to the geographical entity and visual entity corresponding to the target image. The historical description information corresponding to the historical event description type is used to describe the historical event information corresponding to the target image, such as the historical events and character activities related to the geographical entity and visual entity corresponding to the target image.

[0052] For example, for a target image related to Mount Tai, the historical description information corresponding to its historical background description type may be: "Mount Tai is known as the "first of the Five Mountains"... "; the historical description information corresponding to its historical poetry description type may be: "'Viewing the Mountain'... "; the historical description information corresponding to its historical event description type may be: "Mount Tai has a profound cultural heritage, and many travelers have climbed Mount Tai..."

[0053] The preset historical description generation model refers to a model used to generate historical description information. The preset historical description generation model can be obtained by fine-tuning the large language model according to the image description training information, the geographic description training information and the corresponding historical description tags.

[0054] By inputting the image description information and the geographic description information into a preset historical description generation model, the historical description information corresponding to the target image can be generated, that is, the description information containing the historical and cultural background corresponding to the target image can be obtained.

[0055] Step S140: Input the image description information, geographic description information and historical description information into a preset poetry generation model to generate a target poem corresponding to the target image.

[0056] The preset poetry generation model refers to a model for generating target poetry based on image description information, geographic description information, and historical description information. The preset poetry generation model can be obtained by fine-tuning the large language model based on the image description training information, geographic description training information, historical description training information, and the corresponding labeled poetry.

[0057] Among them, the target poem is a poem corresponding to the target image generated by a preset poem generation model based on image description information, geographical description information and historical description information.

[0058] Specifically, when the preset poetry generation model generates the target poetry corresponding to the target image, it can combine the image description information, geographical description information and historical description information corresponding to the target image, so that the target poetry can not only correspond to the content contained in the target image, but also contain the geographical location corresponding to the target image and the historical culture corresponding to the target image, making the content covered by the target poetry more rich and diverse.

[0059] According to the poetry generation method of the embodiment of the present application, when generating poetry, it is possible not only to combine the image description information corresponding to the target image, but also to obtain the historical description information corresponding to the target image based on the image description information and the geographical description information corresponding to the target image, so as to generate poetry based on the image description information, the geographical description information and the historical description information, so that the generated target poetry can integrate the multi-source information related to the target image, thereby improving the deep connection and correspondence between the generated target poetry and the target image.

[0060] In addition, those skilled in the art may also make various modifications and variations to the method in this application:

[0061] In an optional implementation, in order to enable a preset historical description generation model to combine image description information and geographic description information to generate historical description information corresponding to different historical description types, so that the historical description information covers rich information content, prompt information can be constructed according to the image description information, geographic description information and historical description type, so that the preset historical description generation model can output the corresponding historical description information according to the prompt information. Accordingly, the image description information and the geographic description information are input into the preset historical description generation model to obtain the historical description information corresponding to the target image, including: determining the preset historical description type corresponding to the historical description information to be generated, and generating the first task description information according to the preset historical description type; constructing the first prompt information for the historical description information generation task according to the image description information, the geographic description information and the first task description information; inputting the first prompt information into the preset historical description generation model to obtain the output result of the preset historical description generation model, and using the output result as the historical description information corresponding to the target image.

[0062] The preset history description type refers to the history description type corresponding to the history description information to be generated. The history description type has been explained above and will not be repeated here. The preset history description type can be adaptively set according to the actual need for generating history description information, and the embodiment of the present application does not limit this.

[0063] It should be noted that, when the preset history description type includes multiple history description types, corresponding history description information will be generated for each preset history description type, and these information will be merged to obtain the history description information corresponding to the target image.

[0064] Exemplarily, when the preset historical description types include historical background description types and historical poetry description types, the historical description information corresponding to the obtained target image includes both historical description information corresponding to the historical background description type and historical description information corresponding to the historical poetry description type.

[0065] The first task description information refers to the task description information representing the task of generating historical description information, and includes the task requirements corresponding to the task of generating historical description information, such as the input and output of a preset historical description generation model.

[0066] It should be noted that the first task description information can be adaptively adjusted according to changes in the preset history description type, so that the preset history description generation model can generate history description information corresponding to different history description types, thereby increasing the flexibility of history description information generation.

[0067] Exemplarily, when the preset historical description type is the historical background description type, the first task description information generated according to the preset historical description type may be “Task: Output relevant historical background based on given geographical description information and picture description information”, and when the preset historical description type is the historical background description type or the historical poetry description type, the first task description information generated according to the preset historical description type may be “Task: Output relevant historical background and historical poetry based on given geographical description information and picture description information”.

[0068] It can be seen that by determining the preset historical description type corresponding to the historical description information to be generated, the corresponding first task description information can be adaptively generated according to the preset historical description type, so that the model can generate historical description information corresponding to the preset historical description type according to the first task description information.

[0069] The first prompt information refers to the prompt information input to the preset history description generation model, so that the model can perform the corresponding history description information generation task according to the prompt information.

[0070] It should be noted that each first prompt information corresponds to a historical description information generation task. Accordingly, after the first prompt information is input into the preset historical description generation model, the output result of the preset historical description generation model is the task execution result of the preset historical description generation model for the historical description information generation task corresponding to the first prompt information.

[0071] The first prompt information includes image description information, geographic description information, and first task description information.

[0072] It should be noted that, in order to achieve formatted output of the preset history description generation model, the first prompt information may also include information for describing output requirements.

[0073] Exemplarily, the first prompt information may be:

[0074] “Task: Based on the given geographical description information and picture description information, provide relevant historical background information and historical poetry information.

[0075] Geographical description information: Beijing Tongzhou Grand Canal.

[0076] Image description: There is a wide river in the picture, with green trees on the river bank and a temple on the opposite bank.

[0077] Output requirements: Provide relevant historical background information and historical poetry information, within 300 words.

[0078] Output format: {'answer': historical background information: ''; historical poetry information: ''}".

[0079] It can be seen that by including information for describing output requirements, such as output requirement information and output format information, in the first prompt information, it is possible to achieve standardized output of the model and obtain a standardized output result.

[0080] Furthermore, by inputting the first prompt information constructed by different geographic description information and picture description information, the model can generate relevant historical description information according to the different geographic description information and picture description information, thereby improving the accuracy of generating the historical description information.

[0081] The output result includes the historical description information generated by the model according to the image description information and the geographic description information, so that the historical description information corresponding to the target image can be obtained according to the output result.

[0082] Exemplarily, after the first prompt information is input into the preset historical description generation model, the corresponding output result obtained may be:

[0083] {"answer": historical background information: "The Beijing-Hangzhou Grand Canal reaches Zhuo County in the north and has undergone a long period of historical changes and development..."; historical poetry information: "Related poems include: "Two Poems on the Bian River (One)" - Ten thousand dragon boats are among the green silk, carrying all the goods to Yangzhou and never returning. It must be God's will to open the Bian River, and there are no mountains for more than a thousand miles. "Occasional Poem on the Dike" - The canal transports grain to the capital, and I ride my horse on the dike in the spring breeze. Jiuligang is close to Yuhuangba, I have no long-term strategy but am only frightened."}.

[0084] In an embodiment of the present application, by determining the historical description type corresponding to the historical description information to be generated, the corresponding task description information is constructed, and prompt information is constructed based on the task description information, image description information and geographic description information, so that the prompt information is input into the historical description generation model, so that the historical description generation model can combine the image description information and the geographic description information to generate historical description information corresponding to different historical description types, so that the generated historical description information covers rich information content and improves the flexibility of generating historical description information.

[0085] In an optional implementation, in order to improve the accuracy of the historical description information generated by the preset historical description generation model, the historical description generation model can be trained by the semantic similarity and thematic similarity between the historical description information output by the historical description generation model and the historical description tag, so as to obtain the preset historical description generation model. Accordingly, the preset historical description generation model can be trained in the following manner: constructing first prompt training information for the historical description information generation task according to the first image description training information, the first geographic description training information and the first task description training information corresponding to the preset historical description type; inputting the first prompt training information into the historical description generation model to be trained, so as to obtain the first historical description information corresponding to the preset historical description type output by the historical description generation model to be trained; respectively calculating the semantic similarity and thematic similarity between the first historical description information and the first historical description tag corresponding to the first historical description information, so as to obtain the loss value corresponding to the first historical description information; adjusting the model parameters of the historical description generation model to be trained according to the loss value corresponding to the first historical description information, so as to obtain the preset historical description generation model.

[0086] Among them, the historical description generation model to be trained refers to a pre-trained large language model, such as the Qwen2 large model.

[0087] The first image description training information refers to the image description information used for training the historical description generation model. The first geographic description training information refers to the geographic description information used for training the historical description generation model. The first image description training information and the first geographic description training information have a corresponding relationship.

[0088] Among them, the first task description training information refers to the task description information constructed according to the preset historical description type and used for historical description generation model training.

[0089] It should be noted that, since different geographic description training information can be obtained according to different geographic description types, the first image description training information can correspond to multiple related first geographic description training information at the same time. For example, the first image description training information simultaneously corresponds to the first geographic description training information corresponding to the urban geographic entity type, the first geographic description training information corresponding to the landscape geographic entity type, and so on.

[0090] Furthermore, since the preset historical description type may include multiple historical description types, multiple first task description training information can be determined according to different preset historical description types, for example, the first task description training information corresponding to the historical background description type, the first task description training information corresponding to the historical poetry description type, and the first task description training information corresponding to the historical background description type and the historical poetry description type.

[0091] Correspondingly, since the first image description training information can correspond to multiple related first geographic description training information at the same time, and the number of first task description training information can be multiple, for a certain first image description training information, by combining it with the corresponding first geographic description training information and the first task description training information, it is possible to construct multiple corresponding first prompt training information, further increasing the data amount of the first prompt training information.

[0092] The first historical description information refers to the historical description information corresponding to the preset historical description type in the first prompt training information output by the historical description generation model to be trained after the first prompt training information is input into the historical description generation model to be trained.

[0093] Specifically, when the first prompt training information contains the first task description training information of "Task: Give relevant historical background information based on the given geographical description information and picture description information", the historical description generation model to be trained will output the first historical description information corresponding to the historical background description type.

[0094] Among them, the first historical description label corresponding to the first historical description information refers to the preset historical description information. The first historical description label is the same as the preset historical description type corresponding to the first historical description information. For example, when the first historical description information corresponds to the historical poem description type, the first historical description label also corresponds to the historical poem description type. At this time, the first historical description information is the poem information output by the historical description generation model to be trained according to the first prompt training information, and the first historical description label is the preset poem corresponding to the first prompt training information.

[0095] By calculating the similarity between the first history description information and the first history description tag, the generation effect of the history description information of the history description generation model to be trained can be determined, so as to optimize and adjust the model parameters of the history description generation model to be trained.

[0096] Calculating the similarity between the first history description information and the first history description tag may include calculating the semantic similarity and thematic similarity between the first history description information and the first history description tag.

[0097] Specifically, the semantic similarity is used to characterize the semantic relevance between the first historical description information and the first historical description tag. The semantic similarity between the two can be obtained by calculating the word similarity between the participles of the first historical description information and the participles of the first historical description tag. The semantic similarity between the two can also be obtained by calculating the ratio of the number of identical words between the first historical description information and the first historical description tag to the total number of words in the first historical description tag information. The embodiments of the present application are not limited to this.

[0098] In an optional implementation, calculating the semantic similarity between the first historical description information and the first historical description tag may include: calculating the word similarity between the participles of the first historical description information and the participles of the first historical description tag; calculating the ratio of the number of identical characters between the first historical description information and the first historical description tag to the total number of characters in the first historical description tag to obtain the word similarity; and obtaining the semantic similarity between the first historical description information and the first historical description tag based on the word similarity and the word similarity.

[0099] The word segmentation of the first historical description information is the word segmentation obtained by dividing the first historical description information into words, and the word segmentation of the first historical description tag is the word segmentation obtained by dividing the first historical description tag into words. The word similarity between the word segmentations can be obtained according to the vector distance between the word vectors corresponding to the word segmentations. The closer the vector distance, the higher the word similarity between the word segmentations.

[0100] The ratio of the number of identical characters between the first history description information and the first history description tag to the total number of characters in the first history description tag is used to represent the ratio of the number of repeated characters between the first history description information and the first history description tag. The larger the ratio, the higher the character similarity between the two.

[0101] When obtaining the semantic similarity between the first historical description information and the first historical description label based on word similarity and character similarity, different weight values ​​can be set for the word similarity and character similarity according to actual training needs, so as to obtain the final semantic similarity according to the corresponding weight values, thereby realizing the comprehensive calculation of semantic similarity from the word segmentation perspective and the character perspective.

[0102] Among them, the topic similarity is used to characterize the topic consistency between the first historical description information and the first historical description tag. For example, for a first historical description tag, its corresponding topic is used to describe the origin and significance of the Grand Canal, while the corresponding topic of the first historical description information is the changes in the historical relics of the Grand Canal. At this time, the topic consistency between the two is relatively low.

[0103] When calculating the topic similarity between the first historical description information and the first historical description tag, the first historical description information and the first historical description tag may be input into a topic evaluation model for topic extraction, thereby calculating the similarity between the topics corresponding to the two and obtaining the topic similarity. Accordingly, calculating the topic similarity between the first historical description information and the first historical description tag may include: inputting the first historical description information and the first historical description tag into a topic evaluation model, extracting the first topic information corresponding to the first historical description information and the second topic information corresponding to the first historical description tag; calculating the similarity between the first topic information and the second topic information, and obtaining the topic similarity between the first historical description information and the first historical description tag.

[0104] Among them, the topic evaluation model can be a pre-trained large language model, or it can be an optimized model obtained after fine-tuning and training the large language model, and the embodiments of the present application are not limited to this.

[0105] By inputting the first historical description information and the first historical description label into the topic evaluation model, the corresponding topic information, namely the first topic information and the second topic information, can be obtained after the model performs topic extraction on the first historical description information and the first historical description label.

[0106] Among them, the first topic information represents the main description content of the first historical description information, and the second topic information represents the main description content of the first historical description tag. Therefore, by calculating the similarity between the first topic information and the second topic information, the topic similarity between the first historical description information and the first historical description tag can be obtained.

[0107] After calculating the semantic similarity and topic similarity between the first historical description information and the first historical description tag corresponding to the first historical description information, the loss value corresponding to the first historical description information can be obtained. Specifically, corresponding weight values ​​can be set for the semantic similarity and the topic similarity, so as to obtain the loss value corresponding to the first historical description information according to the weight values, and the model parameters of the historical description generation model to be trained can be optimized and adjusted according to the loss values, so as to finally obtain a trained historical description generation model, that is, a preset historical description generation model.

[0108] In an embodiment of the present application, by calculating the semantic similarity and topic similarity between the first historical description information output by the historical description generation model to be trained and the first historical description label, and constructing a loss value based on the similarity calculation result, and then adjusting the model parameters of the historical description generation model based on the loss value, the generation effect of the historical description information by the preset historical description generation model obtained through training can be effectively improved.

[0109] In an optional implementation, in order to enable the preset poetry generation model to combine the image description information, geographic description information and historical description information to generate target poetry that meets the preset poetry format requirements, the corresponding task description information can be generated according to the preset poetry format, so as to construct prompt information according to the image description information, geographic description information, historical description information and the second task description information, and input the prompt information into the preset poetry generation model. Accordingly, the image description information, geographic description information and historical description information are input into the preset poetry generation model to generate the target poetry corresponding to the target image, including: generating the second task description information according to the preset poetry format corresponding to the target poetry to be generated; constructing the second prompt information for the poetry generation task according to the image description information, geographic description information, historical description information and the second task description information; and inputting the second prompt information into the preset poetry generation model so that the preset poetry generation model can generate the target poetry corresponding to the target image.

[0110] Among them, the preset poetry format is used to represent the format corresponding to the target poetry, such as the number of words in the target poetry, whether it contains a title, five-character or seven-character poetry, etc.

[0111] Among them, the second task description information refers to the task description information that characterizes the poetry generation task. The second task description information includes the task requirements corresponding to the poetry generation task, such as the input and output requirements of the preset poetry generation model. Exemplarily, the second task description information can be "Task: Combine image description information, geographic description information, and historical description information to generate a poem. Requirements: The number of words in the poem is within 30, and the poem has a title."

[0112] It should be noted that the second task description information can be adaptively adjusted according to changes in the preset poetry format, so that the preset poetry generation model can generate target poems corresponding to different preset poetry formats, thereby increasing the flexibility of target poetry generation.

[0113] The second prompt information refers to the prompt information input to the preset poetry generation model so that the model can perform the corresponding poetry generation task according to the prompt information. It should be noted that each second prompt information corresponds to a poetry generation task.

[0114] The second prompt information includes image description information, geographic description information, historical description information and second task description information. By constructing the second prompt information based on the image description information, geographic description information, historical description information and second task description information, the model can combine the image description information, geographic description information and historical description information to generate target poems related thereto.

[0115] It should be noted that, in order to achieve formatted output of the preset poetry generation model, information for describing the output format may also be included in the second prompt information.

[0116] For example, the second prompt information may be: “Task: Combine geographical description information and historical description information to generate a poem for the given picture description information.

[0117] Geographical description information: Beijing Tongzhou Grand Canal.

[0118] Image description: There is a wide river in the picture, with green trees on the river bank and a temple on the opposite bank.

[0119] Historical description information: The Grand Canal from Beijing to Hangzhou reaches Zhuo County in the north and has undergone a long period of historical changes and development. Related poems include: "Two poems about the Bian River (one)" - Ten thousand dragon boats are among the green silk, and they carry everything to Yangzhou and never return. It must be God's will to open the Bian River, and there are no mountains for more than a thousand miles. "Occasional Poems on the Dike" - The canal transfers grain to the capital, and horses ride on the dike in the spring breeze. Jiuligang is close to Yuhuangba, and I have no long-term strategy but only fear.

[0120] Output requirements: Give the poem and title, within 30 words.

[0121] Output format: {'title': '', 'text': ''}".

[0122] It can be seen that by including information used to describe the output format in the second prompt information, such as the "output format" in the above example, it is possible to achieve standardized output of the model and generate target poems with a standardized format.

[0123] In an embodiment of the present application, a preset poetry format corresponding to the target poetry is set, and prompt information is constructed according to the preset poetry format, image description information, geographical description information, and historical description information and input into a preset poetry generation model, so that the preset poetry generation model can fully combine the content contained in the prompt information to generate a target poem that meets the requirements.

[0124] In an optional implementation, in order to improve the generation effect of the preset poetry generation model on the target poetry, when training the poetry generation model, the similarity between the poetry sample output by the poetry generation model and the label poetry, the second image description training information, the second geographic description training information and the historical description training information corresponding to the poetry sample can be calculated, so as to adjust the model parameters of the poetry generation model according to the similarity result. Accordingly, the preset poetry generation model is trained in the following manner: constructing the second prompt training information for the poetry generation task according to the second image description training information, the second geographic description training information, the historical description training information and the second task description training information corresponding to the preset poetry format; inputting the second prompt training information into the poetry generation model to be trained to obtain the poetry sample output by the poetry generation model to be trained; calculating the similarity between the poetry sample and the label poetry, the second image description training information, the second geographic description training information and the historical description training information corresponding to the poetry sample, and obtaining the similarity calculation result; adjusting the model parameters of the poetry generation model to be trained according to the similarity calculation result to obtain the preset poetry generation model.

[0125] The second image description training information refers to the image description information used for poetry generation model training. The second geographic description training information refers to the geographic description information used for poetry generation model training.

[0126] The historical description training information refers to the historical description information used for poetry generation model training. The historical description training information can be generated by inputting the second image description training information and the second geographic description training information into a preset historical description generation model, or can be preset historical description information corresponding to the second image description training information and the second geographic description training information, which is not limited in the present embodiment of the application.

[0127] It should be noted that there is a corresponding relationship between the second image description training information, the second geographic description training information, and the history description training information, and they all correspond to the same training image.

[0128] Among them, the second task description training information refers to the task description information constructed according to the preset poetry format for poetry generation model training.

[0129] The second prompt training information refers to prompt information for inputting the poem generation model to be trained so that the poem generation model to be trained performs the poem generation task. The poem generation task refers to the poem generation model generating poems for the training image corresponding to the second prompt training information according to the second image description training information, the second geographical description training information, and the historical description training information contained in the second prompt training information.

[0130] Accordingly, when constructing the second prompt training information, it is necessary to construct the prompt information according to the second image description training information, the second geographic description training information, the history description training information and the second task description training information.

[0131] Among them, the poetry sample refers to the poetry generated by the poetry generation model to be trained for the training image corresponding to the second prompt training information after the second prompt training information is input into the poetry generation model to be trained.

[0132] For example, the poetry sample may be {"title":"Ancient Rhyme of the Canal","text":"The green bank of the Tongzhou Canal is long, and the ancient temples are reflected in the water. The Sui Dynasty embankment has connected the north and the south for thousands of years, and its vitality is endless and will last forever."}.

[0133] The label poem corresponding to the poem sample refers to a poem sample preset for the training image. Specifically, the label poem contains not only the image description information of the training image, but also the historical description information and geographical description information corresponding to the training image.

[0134] Furthermore, by calculating the similarity between the poem sample and the label poem corresponding to the poem sample, the content correlation between the poem sample and the label poem can be obtained. Moreover, by calculating the similarity between the poem sample and the second image description training information, the second geographic description training information and the historical description training information, the content correlation between the poem sample and the image description information, the geographic description information and the historical description information corresponding to the training image can be further obtained, so that the similarity calculation result obtained by the similarity calculation can fully reflect the correspondence between the poem sample output by the model and the training image, and whether the poem sample can fully reflect the information contained in the training image.

[0135] In an optional implementation, in order to improve the accuracy of similarity calculation between poetry samples and labeled poetry, second image description training information, second geographic description training information and historical description training information, similarity calculation is performed on the poetry samples with the labeled poetry, second image description training information, second geographic description training information and historical description training information corresponding to the poetry samples, respectively, to obtain a similarity calculation result, which may include: calculating the poetry similarity between the poetry sample and the labeled poetry corresponding to the poetry sample, and obtaining a first loss value corresponding to the poetry sample; calculating the word similarity between the segmentation corresponding to the poetry sample and the first keyword corresponding to the second image description training information, and obtaining a second loss value corresponding to the poetry sample; calculating the word similarity between the segmentation corresponding to the poetry sample and the second keyword corresponding to the second geographic description training information, and obtaining a third loss value corresponding to the poetry sample; calculating the word similarity between the segmentation corresponding to the poetry sample and the third keyword corresponding to the historical description training information, and obtaining a fourth loss value corresponding to the poetry sample; obtaining the similarity calculation result based on the first loss value, the second loss value, the third loss value and the fourth loss value.

[0136] Among them, the poetry similarity between the poetry sample and the labeled poetry can be obtained by calculating the vector distance between the sample vector corresponding to the poetry sample and the label vector corresponding to the labeled poetry. After calculating the poetry similarity, the first loss value corresponding to the poetry sample can be obtained.

[0137] Among them, the word segmentations corresponding to the poetry sample are the word segmentations obtained after word segmentation of the poetry sample. For example, for a sentence in the poetry sample "Green bank Tongzhou canal is long", the corresponding word segmentation can be "green, bank, Tongzhou, canal, long".

[0138] The first keyword corresponding to the second image description training information is a word with key information contained in the second image description training information, for example, a word describing a visual feature contained in the training image.

[0139] Exemplarily, the training information for describing the second image is “there is a wide river in the picture, with green trees on the river bank and a temple on the opposite bank”, and its corresponding keywords may be “wide, river, river bank, green trees, opposite bank, temple”.

[0140] When calculating the word similarity between the word segmentation corresponding to the poetry sample and the first keyword corresponding to the second image description training information, for any first keyword, the word vector of the word segmentation corresponding to the poetry sample can be respectively calculated with the corresponding word vector, so as to select the highest similarity value as the similarity value corresponding to the first keyword.

[0141] Furthermore, by taking the average of the similarity values ​​corresponding to all the first keywords, the second loss value corresponding to the poetry sample is obtained.

[0142] By calculating the word similarity between the word segmentation corresponding to the poetry sample and the first keyword corresponding to the second image description training information, it is possible to determine whether the poetry sample contains the key information of the second image description training information, thereby effectively judging the correlation between the poetry sample and the second image description training information.

[0143] The second keyword corresponding to the second geographic description training information is a word used to represent a geographic entity in the second geographic description training information.

[0144] Exemplarily, for the second geographical description training information “Beijing Tongzhou Grand Canal”, the corresponding keywords may be “Beijing, Tongzhou, Grand Canal”.

[0145] When calculating the word similarity between the word segmentation corresponding to the poetry sample and the second keyword corresponding to the second geographic description training information, for any second keyword, the word vector of the word segmentation corresponding to the poetry sample can be respectively calculated with the corresponding word vector, so as to select the highest similarity value as the similarity value corresponding to the second keyword.

[0146] Furthermore, after calculating the word similarity between the word segmentation corresponding to the poetry sample and the second keyword corresponding to the second geographic description training information, the third loss value corresponding to the poetry sample can be obtained by taking the average of the similarity values ​​corresponding to all second keywords.

[0147] By calculating the word similarity between the word segmentation corresponding to the poetry sample and the second keyword corresponding to the second geographic description training information, it is possible to determine whether the poetry sample contains the geographic entity contained in the second geographic description training information, thereby effectively judging the correlation between the poetry sample and the second geographic description training information.

[0148] Among them, the third keyword corresponding to the historical description training information is the words with key information contained in the historical description training information, for example, nouns, verbs, adjectives and other content words contained in the historical description training information, which can express the main meaning of the sentences in the historical description training information.

[0149] For example, for a sentence in the historical description training information, "The history of the Grand Canal can be traced back to the Spring and Autumn Period and the Warring States Period, but it was not until the Sui Dynasty that it truly became a north-south connection", its corresponding keywords may be "Beijing-Hangzhou, Grand Canal, trace back, Spring and Autumn Period and the Warring States Period, north-south connection, Sui Dynasty".

[0150] When calculating the word similarity between the word segmentation corresponding to the poetry sample and the third keyword corresponding to the historical description training information, for any third keyword, the word vector of the word segmentation corresponding to the poetry sample can be respectively calculated with the corresponding word vector, so as to select the highest similarity value as the similarity value corresponding to the third keyword.

[0151] Furthermore, after calculating the word similarity between the word segmentation corresponding to the poetry sample and the third keyword corresponding to the historical description training information, the fourth loss value corresponding to the poetry sample can be obtained by taking the average of the similarity values ​​corresponding to all third keywords.

[0152] By calculating the word similarity between the word segmentation corresponding to the poetry sample and the third keyword corresponding to the historical description training information, it is possible to determine whether the poetry sample contains the key information in the historical description training information, thereby effectively judging the correlation between the poetry sample and the historical description training information.

[0153] It should be noted that when obtaining the similarity calculation result based on the first loss value, the second loss value, the third loss value and the fourth loss value, the corresponding weight values ​​can be determined for the first loss value, the second loss value, the third loss value and the fourth loss value, so as to obtain the similarity calculation result by weighted calculation according to the weight values, or the similarity calculation result can be obtained by directly adding up the loss values ​​directly. The embodiment of the present application does not limit this.

[0154] In an embodiment of the present application, by calculating the poetry similarity between the poetry sample and the labeled poetry, the correlation between the poetry sample and the labeled poetry can be obtained, and by performing keyword extraction on the image description training information, the geographical description training information, and the historical description training information, the word similarity between the word segmentation corresponding to the poetry sample and different keywords can be calculated, so that the degree of association between the poetry sample and the image description training information, the geographical description training information, and the historical description training information can be determined, and finally the similarity calculation result is obtained, so that the poetry generated by the poetry generation model trained according to the similarity calculation result can contain the visual information, geographical entities and historical and cultural information corresponding to the image, so that the generated poetry can have a deep correspondence with the image, and its text content is richer and has depth.

[0155] For ease of understanding, the technical implementation details in the above embodiment are described in detail below using a specific example:

[0156] In the related art, when generating poems corresponding to images, methods based on image recognition and image description are usually adopted. However, these methods have the following problems when generating poems matching images:

[0157] 1) Limited information content: Traditional methods often only generate poetry and text based on the content of the image itself, lacking additional dimensional information input such as geographic location information, historical and cultural information, resulting in the generated poetry and text being monotonous and lacking depth and breadth.

[0158] 2) Inharmonious artistic conception: Due to the lack of deep-level connection and correspondence between the generated poetry text and the image, the generated poetry text and the image artistic conception are often inconsistent, making it difficult to meet the requirements of artistry.

[0159] In order to solve the above problems, a poetry generation method is proposed in this example. By combining the geographical location information, historical and cultural information and picture description content corresponding to the image, poetry generation is performed. Its corresponding advantages are as follows:

[0160] 1) Improve the information content of generated poems: By introducing additional dimensions of information such as geographic location information, historical and cultural information into the poem generation process, the generated poem text content is made richer and more diverse, and can more comprehensively reflect the information contained in the image.

[0161] 2) Enhance the coordination between the artistic conception of poetry and images: Through deep learning and large model technology, the generated poetry and images can have a deep connection and correspondence, ensuring that the generated poetry and images are coordinated and meet the artistic requirements.

[0162] Figure 2 FIG. 2 shows a flow chart of the poetry generation method in this example. Figure 2 As shown, the example includes the following steps:

[0163] Step S200: extract visual information from the target image for generating poetry, and obtain image description information corresponding to the target image.

[0164] The target image is an image captured by an image acquisition device such as a camera.

[0165] Among them, for the target image to be generated into poetry, such as a camera image, a graph-to-text model (such as a StableDiffusion model) can be used to perform content analysis on the camera image, thereby extracting key elements or scene information in the camera image, such as ancient buildings, bridges, rivers, etc., and obtaining the corresponding image description information.

[0166] It should be noted that, in order to enhance the image description information generation capability of the graph-to-text model, a high-quality data set including images and image descriptions can be constructed, the pre-trained graph-to-text model can be fine-tuned, and the generalization capability of the graph-to-text model can be improved by optimizing the training strategy (such as adjusting the learning rate and data enhancement). The embodiments of the present application do not impose any restrictions on this.

[0167] Step S201: extracting geographic location data corresponding to a target image from image metadata corresponding to the target image.

[0168] The image metadata is metadata obtained when the target image is photographed. By extracting geographic location information (such as longitude and latitude information) from the metadata of the target image, the geographic location data corresponding to the target image can be obtained.

[0169] Step S202: According to the geographic location data and the preset geographic entity type, first geographic description information matching the geographic location data is searched in a preset geographic database, wherein the first geographic description information corresponds to the preset geographic entity type.

[0170] Among them, the preset geographic database can be a geopy library, and the first geographic description information can be queried in the preset geographic database through the geographic location data, wherein the first geographic description information corresponds to the preset geographic entity type, such as the first geographic description information is the first geographic description information corresponding to the urban geographic entity type, the first geographic description information corresponding to the landscape geographic entity type, etc.

[0171] Step S203: Obtaining geographic description information corresponding to the target image according to the first geographic description information corresponding to the preset geographic entity type.

[0172] Step S204: determining a preset history description type corresponding to the history description information to be generated, and generating first task description information according to the preset history description type.

[0173] Among them, the preset description types may include historical background description type, historical poetry description type, historical event description type, etc.

[0174] Step S205: constructing first prompt information for the task of generating historical description information according to the image description information, the geographic description information and the first task description information.

[0175] Step S206: inputting the first prompt information into a preset history description generation model to obtain an output result of the preset history description generation model, and using the output result as the history description information corresponding to the target image.

[0176] Among them, the preset historical description generation model can be an optimized model obtained after fine-tuning and training a large language model such as the Qwen2 large model.

[0177] Specifically, fine-tuning training can be carried out by constructing a large number of data sets (including geographic description training information, image description training information as preset input, historical description information as preset output, and historical description labels as label data corresponding to the historical description information), and the effect of generating historical description information after model training can be evaluated by means of automated evaluation, manual evaluation and other means.

[0178] The first prompt information prompt_history is constructed by combining the image description information, the geographic description information and the first task description information, and is used as input information of a preset history description generation model, so that the model can further output historical description information corresponding to the related historical background description type, historical poetry description type, etc.

[0179] The first prompt information enables the preset historical description generation model to clarify the historical description generation task, thereby analyzing the correlation between the image description information and the geographic description information, and outputting the corresponding historical description information.

[0180] For example, the first prompt information may be prompt_history="

[0181] Task: Based on the given geographical description information and picture description information, provide relevant historical background information and historical poetry information.

[0182] Geographical description information: Beijing Tongzhou Grand Canal.

[0183] Image description: There is a wide river in the picture, with green trees on the river bank and a temple on the opposite bank.

[0184] Output requirements: Provide relevant historical background information and historical poetry information, within 300 words.

[0185] Output format: {'answer': historical background information: ''; historical poetry information: ''}".

[0186] Step S207: Generate second task description information according to a preset poem format corresponding to the target poem to be generated.

[0187] Step S208: Constructing second prompt information for the poetry generation task based on the image description information, geographical description information, historical description information and the second task description information.

[0188] Step S209: input the second prompt information into the preset poetry generation model so that the preset poetry generation model can generate the target poem corresponding to the target image.

[0189] Among them, the preset poetry generation model can be an optimized model obtained after fine-tuning and training a large language model such as the Qwen2 large model.

[0190] Specifically, fine-tuning training can be carried out by constructing a large number of data sets (including geographical description training information, image description training information, historical description training information as preset input, poetry text as preset output, and labeled poetry as label data corresponding to the poetry text), and the effect of poetry text generation after model training can be evaluated by means of automated evaluation, manual evaluation and other means.

[0191] Among them, the second prompt information prompt_poetry is constructed by combining geographical description information, historical description information, picture description information and second task description information as the input of the preset poetry generation model, so that the model can output a target poem that is closely related to the input second prompt information and creative.

[0192] It should be noted that the second prompt information is used to enable the preset poetry generation model to clearly identify the corresponding poetry generation task, so that it can analyze the relevance of the image description information, geographical description information, and historical description information contained in the second prompt information, and output appropriate poetry.

[0193] For example, the second prompt information may be prompt_poetry="

[0194] Task: Combine geographic description information and historical description information to generate a poem for the given image description information.

[0195] Geographical description information: Beijing Tongzhou Grand Canal.

[0196] Image description: There is a wide river in the picture, with green trees on the river bank and a temple on the opposite bank.

[0197] Historical description information: The Grand Canal from Beijing to Hangzhou reaches Zhuo County in the north and has undergone a long period of historical changes and development. Related poems include: "Two poems about the Bian River (one)" - Ten thousand dragon boats are among the green silk, and they carry everything to Yangzhou and never return. It must be God's will to open the Bian River, and there are no mountains for more than a thousand miles. "Occasional Poems on the Dike" - The canal transfers grain to the capital, and horses ride on the dike in the spring breeze. Jiuligang is close to Yuhuangba, and I have no long-term strategy but only fear.

[0198] Output requirements: Give the poem and title, within 30 words.

[0199] Output format: {'title': '', 'text': ''}".

[0200] It should also be noted that the preset poetry generation model may generate multiple corresponding target poems. Therefore, when the poetry generation model outputs multiple target poems, the generated multiple target poems can be displayed to the user so that the user can select the target poem.

[0201] Furthermore, after the user selects the target poem, the target poem can be marked on the target image with a suitable font, size, and position, thereby generating an image with the poem. Finally, the generated poem image is stored so that it is easy for users to share and use.

[0202] It can be understood that the above-mentioned various method embodiments mentioned in this application can be combined with each other to form a combined embodiment without violating the principle logic. Due to space limitations, this application will not go into details. Those skilled in the art can understand that in the above-mentioned method of the specific implementation method, the specific execution order of each step should be determined by its function and possible internal logic.

[0203] In addition, the present application also provides a poetry generation device, an electronic device, and a computer-readable storage medium, all of which can be used to implement any poetry generation method provided by the present application. The corresponding technical solutions and descriptions are referred to in the corresponding records of the method part and will not be repeated here.

[0204] Figure 3 A block diagram of a poetry generation device 30 provided in an embodiment of the present application.

[0205] Reference Figure 3 The embodiment of the present application provides a poetry generation device 30, and the poetry generation device 30 includes:

[0206] An image description generating module 31 is used to extract visual information from a target image for generating poetry, and obtain image description information corresponding to the target image, wherein the image description information is used to characterize the visual information contained in the target image;

[0207] A geographic description generating module 32 is used to extract the geographic location of the target image to obtain geographic description information corresponding to the target image, wherein the geographic description information is used to characterize the geographic entity corresponding to the target image;

[0208] A historical description generation module 33, used to input the image description information and the geographic description information into a preset historical description generation model to obtain historical description information corresponding to the target image, wherein the historical description information is used to characterize the historical and cultural information corresponding to the target image;

[0209] The poetry generation module 34 is used to input the image description information, the geographical description information and the historical description information into a preset poetry generation model to generate a target poem corresponding to the target image.

[0210] In an optional implementation, the geographic description generating module 32 extracts the geographic location of the target image to obtain geographic description information corresponding to the target image, including:

[0211] Extracting geographic location data corresponding to the target image from image metadata corresponding to the target image;

[0212] According to the geographic location data and the preset geographic entity type, searching a preset geographic database for first geographic description information matching the geographic location data, wherein the first geographic description information corresponds to the preset geographic entity type;

[0213] The geographic description information corresponding to the target image is obtained according to the first geographic description information corresponding to the preset geographic entity type.

[0214] In an optional implementation, the historical description generation module 33 inputs the image description information and the geographic description information into a preset historical description generation model to obtain the historical description information corresponding to the target image, including:

[0215] Determine a preset history description type corresponding to the history description information to be generated, and generate first task description information according to the preset history description type;

[0216] Constructing first prompt information for a historical description information generation task according to the image description information, the geographic description information and the first task description information;

[0217] The first prompt information is input into the preset history description generation model to obtain an output result of the preset history description generation model, and the output result is used as the history description information corresponding to the target image.

[0218] In an optional implementation, the preset historical description generation model is trained in the following manner:

[0219] Constructing first prompt training information for the historical description information generation task according to the first image description training information, the first geographic description training information, and the first task description training information corresponding to the preset historical description type;

[0220] Inputting the first prompt training information into a history description generation model to be trained, and obtaining first history description information corresponding to the preset history description type output by the history description generation model to be trained;

[0221] respectively calculating the semantic similarity and the topic similarity between the first historical description information and the first historical description tag corresponding to the first historical description information, to obtain a loss value corresponding to the first historical description information;

[0222] The model parameters of the history description generation model to be trained are adjusted according to the loss value corresponding to the first history description information to obtain the preset history description generation model.

[0223] In an optional implementation, the poem generation module 34 inputs the image description information, the geographic description information and the historical description information into a preset poem generation model to generate a target poem corresponding to the target image, including:

[0224] Generate second task description information according to a preset poem format corresponding to the target poem to be generated;

[0225] Constructing second prompt information for a poetry generation task according to the image description information, the geographic description information, the historical description information, and the second task description information;

[0226] The second prompt information is input into the preset poetry generation model so that the preset poetry generation model generates the target poem corresponding to the target image.

[0227] In an optional implementation, the preset poetry generation model is trained in the following way:

[0228] Constructing second prompt training information for the poetry generation task according to the second image description training information, the second geographic description training information, the historical description training information, and the second task description training information corresponding to the preset poetry format;

[0229] Inputting the second prompt training information into the poetry generation model to be trained to obtain poetry samples output by the poetry generation model to be trained;

[0230] Calculate similarity between the poem sample and the label poem corresponding to the poem sample, the second image description training information, the second geographic description training information, and the historical description training information to obtain a similarity calculation result;

[0231] The model parameters of the poetry generation model to be trained are adjusted according to the similarity calculation result to obtain the preset poetry generation model.

[0232] In an optional implementation, the similarity calculation is performed on the poem sample and the label poem corresponding to the poem sample, the second image description training information, the second geographic description training information, and the historical description training information to obtain a similarity calculation result, including:

[0233] Calculate the poetry similarity between the poetry sample and the label poetry corresponding to the poetry sample to obtain a first loss value corresponding to the poetry sample;

[0234] Calculate the word similarity between the word segment corresponding to the poem sample and the first keyword corresponding to the second image description training information to obtain a second loss value corresponding to the poem sample;

[0235] Calculate the word similarity between the segmented word corresponding to the poem sample and the second keyword corresponding to the second geographic description training information to obtain a third loss value corresponding to the poem sample;

[0236] Calculate the word similarity between the segmented word corresponding to the poem sample and the third keyword corresponding to the historical description training information to obtain a fourth loss value corresponding to the poem sample;

[0237] The similarity calculation result is obtained according to the first loss value, the second loss value, the third loss value and the fourth loss value.

[0238] Each module in the above-mentioned poetry generation device can be implemented in whole or in part by software, hardware and a combination thereof. Each of the above-mentioned modules can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each of the above modules.

[0239] Figure 4 A block diagram of an electronic device provided in an embodiment of the present application.

[0240] Reference Figure 4 An embodiment of the present application provides an electronic device, which includes: at least one processor 401; at least one memory 402, and one or more I / O interfaces 403, connected between the processor 401 and the memory 402; wherein the memory 402 stores one or more computer programs that can be executed by at least one processor 401, and the one or more computer programs are executed by at least one processor 401, so that at least one processor 401 can execute the above-mentioned poetry generation method.

[0241] Each module in the above electronic device can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0242] The present application also provides a computer-readable storage medium on which a computer program is stored, wherein the computer program implements the above-mentioned poetry generation method when executed by a processor / processing core. The computer-readable storage medium may be a volatile or non-volatile computer-readable storage medium.

[0243] An embodiment of the present application also provides a computer program product, including a computer-readable code, or a non-volatile computer-readable storage medium carrying a computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes the above-mentioned poetry generation method.

[0244] It will be appreciated by those skilled in the art that all or some of the steps, systems, and functional modules / units in the methods disclosed above may be implemented as software, firmware, hardware, and appropriate combinations thereof. In a hardware implementation, the division between the functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed by several physical components in cooperation. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or may be implemented as hardware, or may be implemented as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable storage medium, which may include a computer storage medium (or a non-transitory medium) and a communication medium (or a temporary medium).

[0245] As is known to those of ordinary skill in the art, the term computer storage media includes volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable program instructions, data structures, program modules or other data). Computer storage media include, but are not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), static random access memory (SRAM), flash memory or other memory technology, portable compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and can be accessed by a computer. In addition, it is known to those of ordinary skill in the art that communication media typically contain computer-readable program instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transmission mechanism, and may include any information delivery medium.

[0246] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0247] The computer program instructions for performing the operation of the present application can be assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions can be executed completely on a user's computer, partially on a user's computer, executed as an independent software package, partially on a user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (for example, using an Internet service provider to connect through the Internet). In some embodiments, by using the state information of a computer-readable program instruction to personalize an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA) or a programmable logic array (PLA), the electronic circuit can execute a computer-readable program instruction, thereby realizing various aspects of the present application.

[0248] The computer program product described herein may be implemented in hardware, software, or a combination thereof. In one optional embodiment, the computer program product is embodied as a computer storage medium, and in another optional embodiment, the computer program product is embodied as a software product, such as a software development kit (SDK), etc.

[0249] Various aspects of the present application are described herein with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present application. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0250] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0251] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0252] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of the module, program segment or instruction includes one or more executable instructions for realizing the logical function of the specification. In some alternative implementations, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can also be executed in the opposite order sometimes, depending on the function involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be realized by a dedicated hardware-based system that performs the function or action of the specification, or can be realized by a combination of special-purpose hardware and computer instructions.

[0253] Example embodiments have been disclosed herein, and although specific terms are employed, they are used and should be interpreted only in a general illustrative sense and not for limiting purposes. In some instances, it will be apparent to those skilled in the art that, unless otherwise expressly noted, features, characteristics, and / or elements described in conjunction with a particular embodiment may be used alone or in combination with features, characteristics, and / or elements described in conjunction with other embodiments. Therefore, those skilled in the art will appreciate that various changes in form and detail may be made without departing from the scope of the present application as set forth in the appended claims.

Claims

1. A method for generating poetry, characterized in that: The method comprises: Extracting visual information from a target image for generating poetry, and obtaining image description information corresponding to the target image, wherein the image description information is used to characterize the visual information contained in the target image; Extracting the geographic location of the target image to obtain geographic description information corresponding to the target image, wherein the geographic description information is used to characterize the geographic entity corresponding to the target image; Inputting the image description information and the geographic description information into a preset historical description generation model to obtain historical description information corresponding to the target image, wherein the historical description information is used to characterize the historical and cultural information corresponding to the target image; The image description information, the geographic description information and the historical description information are input into a preset poetry generation model to generate a target poem corresponding to the target image.

2. The method according to claim 1, characterized in that The step of extracting the geographic location of the target image to obtain geographic description information corresponding to the target image includes: Extracting geographic location data corresponding to the target image from image metadata corresponding to the target image; According to the geographic location data and the preset geographic entity type, searching a preset geographic database for first geographic description information matching the geographic location data, wherein the first geographic description information corresponds to the preset geographic entity type; The geographic description information corresponding to the target image is obtained according to the first geographic description information corresponding to the preset geographic entity type.

3. The method according to claim 1, characterized in that The step of inputting the image description information and the geographic description information into a preset historical description generation model to obtain the historical description information corresponding to the target image includes: Determine a preset history description type corresponding to the history description information to be generated, and generate first task description information according to the preset history description type; Constructing first prompt information for a historical description information generation task according to the image description information, the geographic description information and the first task description information; The first prompt information is input into the preset history description generation model to obtain an output result of the preset history description generation model, and the output result is used as the history description information corresponding to the target image.

4. The method according to claim 3, characterized in that The preset historical description generation model is trained in the following way: Constructing first prompt training information for the historical description information generation task according to the first image description training information, the first geographic description training information, and the first task description training information corresponding to the preset historical description type; Inputting the first prompt training information into a history description generation model to be trained, and obtaining first history description information corresponding to the preset history description type output by the history description generation model to be trained; respectively calculating the semantic similarity and the topic similarity between the first historical description information and the first historical description tag corresponding to the first historical description information, to obtain a loss value corresponding to the first historical description information; The model parameters of the history description generation model to be trained are adjusted according to the loss value corresponding to the first history description information to obtain the preset history description generation model.

5. The method according to any one of claims 1 to 4, characterized in that: The step of inputting the image description information, the geographic description information, and the historical description information into a preset poetry generation model to generate a target poem corresponding to the target image includes: Generate second task description information according to a preset poem format corresponding to the target poem to be generated; Constructing second prompt information for a poetry generation task according to the image description information, the geographic description information, the historical description information, and the second task description information; The second prompt information is input into the preset poetry generation model so that the preset poetry generation model generates the target poem corresponding to the target image.

6. The method according to claim 5, characterized in that The preset poetry generation model is trained in the following way: Constructing second prompt training information for the poetry generation task according to the second image description training information, the second geographic description training information, the historical description training information, and the second task description training information corresponding to the preset poetry format; Inputting the second prompt training information into the poetry generation model to be trained to obtain poetry samples output by the poetry generation model to be trained; Calculate similarity between the poem sample and the label poem corresponding to the poem sample, the second image description training information, the second geographic description training information, and the historical description training information to obtain a similarity calculation result; The model parameters of the poetry generation model to be trained are adjusted according to the similarity calculation result to obtain the preset poetry generation model.

7. The method according to claim 6, characterized in that The similarity calculation of the poem sample with the label poem corresponding to the poem sample, the second image description training information, the second geographic description training information and the historical description training information to obtain a similarity calculation result includes: Calculate the poetry similarity between the poetry sample and the label poetry corresponding to the poetry sample to obtain a first loss value corresponding to the poetry sample; Calculate the word similarity between the word segment corresponding to the poem sample and the first keyword corresponding to the second image description training information to obtain a second loss value corresponding to the poem sample; Calculate the word similarity between the segmented word corresponding to the poem sample and the second keyword corresponding to the second geographic description training information to obtain a third loss value corresponding to the poem sample; Calculate the word similarity between the segmented word corresponding to the poem sample and the third keyword corresponding to the historical description training information to obtain a fourth loss value corresponding to the poem sample; The similarity calculation result is obtained according to the first loss value, the second loss value, the third loss value and the fourth loss value.

8. A poetry generation device, characterized in that: The device comprises: An image description generation module, used to extract visual information from a target image for generating poetry, and obtain image description information corresponding to the target image, wherein the image description information is used to characterize the visual information contained in the target image; A geographic description generation module, used to extract the geographic location of the target image to obtain geographic description information corresponding to the target image, wherein the geographic description information is used to characterize the geographic entity corresponding to the target image; A historical description generation module, used for inputting the image description information and the geographic description information into a preset historical description generation model to obtain the historical description information corresponding to the target image, wherein the historical description information is used to characterize the historical and cultural information corresponding to the target image; The poetry generation module is used to input the image description information, the geographical description information and the historical description information into a preset poetry generation model to generate a target poem corresponding to the target image.

9. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores one or more computer programs that can be executed by the at least one processor, and one or more of the computer programs are executed by the at least one processor so that the at least one processor can execute the poetry generation method as described in any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When executed by a processor, the computer program implements the poetry generation method as described in any one of claims 1-7.

Images