Image text matching method, storage medium and terminal device

Abstract

The present application discloses a method for captioning an image, a storage medium, and a terminal device. The method comprises obtaining a plurality of descriptive sentences corresponding to an image to be captioned, and first emotional information corresponding to each of the descriptive sentences; obtaining second emotional information corresponding to the image to be captioned; and determining the caption information corresponding to the image to be captioned based on the plurality of descriptive sentences, the first emotional information corresponding to each of the plurality of descriptive sentences, and the second emotional information. In this way, the first emotional information corresponding to each of the descriptive sentences is matched with the second emotional information, which can improve the correlation between the emotional information expressed by the descriptive sentence and the second emotional information corresponding to the image to be captioned, thereby improving the correlation between the caption information and the target object in the image to be captioned, so that it can be determined that the caption information matches the target object, thereby improving the accuracy of the caption information.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image text matching method, a storage medium and a terminal device. Background Art

[0002] With the widespread adoption of mobile devices, users are increasingly taking photos and sharing them on social networks or other smart devices. They often add captions to their images to enhance the viewer's experience. However, existing methods for captioning images often suffer from a significant discrepancy between the text and the image content, failing to accurately convey the image's meaning. Summary of the Invention

[0003] The technical problem to be solved by this application is to provide an image text matching method, storage medium and terminal device in response to the shortcomings of the existing technology.

[0004] In order to solve the above technical problems, a first aspect of an embodiment of the present application provides an image text matching method, the method comprising:

[0005] Obtaining a plurality of descriptive sentences corresponding to the image to be captioned, and first emotion information corresponding to each descriptive sentence;

[0006] Obtaining the second emotional information corresponding to the image to be captioned;

[0007] Based on a plurality of descriptive sentences, the first emotional information corresponding to each of the plurality of descriptive sentences, and the second emotional information, the caption information corresponding to the image to be captioned is determined.

[0008] The method for captioning an image, wherein determining the caption information corresponding to the image to be captioned based on the plurality of descriptive sentences, the first emotion information corresponding to each of the plurality of descriptive sentences, and the second emotion information specifically includes:

[0009] For each description sentence, determining a matching degree between the first emotion information configured for the description sentence and the second emotion information;

[0010] Based on the matching degrees corresponding to the respective first emotion information, the text information corresponding to the image to be texted is determined.

[0011] The method for matching text with an image, wherein determining the text information corresponding to the image to be matched with a text based on the matching degree corresponding to each piece of first emotional information specifically includes:

[0012] The matching degrees corresponding to the respective first emotion information are respectively used as the matching degrees corresponding to the respective description sentences corresponding to the respective first emotion information;

[0013] A target description statement is selected from a plurality of description statements based on a matching degree, wherein the target description statement is one of the plurality of description statements, and a matching degree corresponding to the target description statement is greater than a matching degree corresponding to any description statement in the plurality of description statements except the target description statement.

[0014] The method for captioning an image, wherein the step of obtaining a plurality of descriptive sentences corresponding to the image to be captioned and the first emotional information corresponding to each of the descriptive sentences specifically includes:

[0015] Inputting the image to be captioned into a trained sentence prediction model;

[0016] The sentence prediction model is used to determine a plurality of descriptive sentences corresponding to the image to be captioned and the first emotion information corresponding to each descriptive sentence.

[0017] The image text matching method, wherein the sentence prediction model is configured with a plurality of default emotions and a plurality of preset description sentences, and the determining of the plurality of description sentences corresponding to the image to be matched with the text and the first emotion information corresponding to each description sentence by the sentence prediction model specifically includes:

[0018] Determining, by the sentence prediction model, a plurality of image groups corresponding to the image to be captioned and evaluation scores corresponding to each of the plurality of image groups, wherein each image group includes the image to be captioned, a default emotion from among a plurality of default emotions, and a preset description sentence from among a plurality of preset description sentences;

[0019] Determining, based on each image group and the evaluation score corresponding to each image group, a set of image groups corresponding to the image to be captioned;

[0020] For each target image group in the image group set, the preset description sentence included in the target image group is used as the description sentence corresponding to the image to be captioned, and the first emotion information included in the target image group is used as the first emotion information corresponding to the description sentence.

[0021] The image text matching method, wherein, for any two image groups among the plurality of image groups, the default emotions and / or preset description sentences in the two image groups are different.

[0022] The image text matching method, wherein the process of determining the evaluation score specifically includes:

[0023] For each image group, the sentence prediction model is used to determine the image features corresponding to the images to be captioned in the image group, the emotional features corresponding to the default emotions in the image group, and the sentence features corresponding to the preset description sentences in the image group; the sentence prediction model, the image features, the emotional features, and the sentence features are used to determine the evaluation score corresponding to the image group.

[0024] The image text matching method, wherein the sentence prediction model includes a feature extraction module, a linear module, and a word feature extraction module, wherein the image features corresponding to the image to be matched with text are determined by the sentence prediction model, the emotional features corresponding to the default emotion in the image group, and the sentence features corresponding to the preset description sentence in the image group specifically include:

[0025] Inputting the image to be accompanied by text in the image group into the feature extraction module, and determining the image features corresponding to the image to be accompanied by text by the feature extraction module;

[0026] Inputting the default emotion in the image group into the linear module, and determining the emotional features corresponding to the default emotion through the linear module;

[0027] The preset description sentence in the image group is input into the word feature extraction module, and the word feature extraction module determines the sentence feature corresponding to the preset description sentence.

[0028] The image captioning method, wherein the sentence prediction model further includes a first fusion module and a second fusion module, and determining the evaluation score corresponding to the image group using the sentence prediction model, the image features, the emotional features, and the sentence features specifically includes:

[0029] Inputting the image feature and the emotion feature into the first fusion module, and outputting the fusion feature through the first fusion module;

[0030] The fusion feature and the sentence feature are input into the second fusion module, and the evaluation score corresponding to the image group is output through the second fusion module.

[0031] The image captioning method, wherein the feature extraction module includes a residual unit and a first feedforward network unit, the residual unit is cascaded with the first feedforward network unit, the input item of the residual unit is the image to be captioned, and the output item of the first feedforward network unit is the image feature.

[0032] The image caption method, wherein the word feature extraction module includes a natural language unit and a second feedforward network unit, the natural language unit and the second feedforward network unit are cascaded, the input item of the natural language unit is a preset description sentence in the image group, and the output item of the second feedforward network unit is a sentence feature.

[0033] The method for matching text with an image, wherein the step of obtaining the second emotional information corresponding to the image to be matched with text is specifically as follows:

[0034] The image to be accompanied by text is input into a trained image sentiment analysis model, and the second sentiment information corresponding to the image to be accompanied by text is determined by the image sentiment analysis model.

[0035] The image text matching method, wherein the image sentiment analysis model includes a feature attention module and a visual attention module, and inputting the image to be matched with text into a trained image sentiment analysis model, and determining the second sentiment information corresponding to the image to be matched with text by the image sentiment analysis model specifically includes:

[0036] Inputting the image to be accompanied by text into the feature extraction module, and determining a first feature map of the image to be accompanied by text by the feature attention module;

[0037] Inputting the image to be accompanied by text into the visual attention module, and determining a visual attention map of the image to be accompanied by text by the visual attention module;

[0038] Based on the feature map and the visual attention map, the second emotion information corresponding to the image to be accompanied by text is determined.

[0039] The image text matching method, wherein, after determining the text matching information corresponding to the image to be matched based on the plurality of descriptive sentences, the first emotional information corresponding to each of the plurality of descriptive sentences, and the second emotional information, the method further comprises:

[0040] The text information is added to the image to be matched with a text, so as to obtain a text image corresponding to the image to be matched with a text.

[0041] A second aspect of an embodiment of the present application provides an image-based text matching device, the image-based text matching device comprising:

[0042] A first acquisition module is used to acquire a plurality of descriptive sentences corresponding to the image to be captioned, and first emotion information corresponding to each descriptive sentence;

[0043] A second acquisition module is used to obtain second emotional information corresponding to the image to be captioned;

[0044] The determination module is used to determine the text information corresponding to the image to be texted based on a plurality of descriptive sentences, the first emotional information corresponding to each of the plurality of descriptive sentences, and the second emotional information.

[0045] A third aspect of an embodiment of the present application provides a computer-readable storage medium, which stores one or more programs. The one or more programs can be executed by one or more processors to implement the steps in any of the image captioning methods described above.

[0046] A fourth aspect of an embodiment of the present application provides a terminal device, comprising: a processor, a memory, and a communication bus; the memory stores a computer-readable program executable by the processor;

[0047] The communication bus realizes the connection and communication between the processor and the memory;

[0048] When the processor executes the computer-readable program, the steps in any of the above-described image captioning methods are implemented.

[0049] Beneficial effects: Compared with the prior art, the present application provides an image captioning method, storage medium, and terminal device, the method comprising obtaining a plurality of descriptive statements corresponding to the image to be captioned, and first emotional information corresponding to each of the descriptive statements; obtaining second emotional information corresponding to the image to be captioned; and determining the caption information corresponding to the image to be captioned based on the plurality of descriptive statements, the first emotional information corresponding to each of the plurality of descriptive statements, and the second emotional information. In this way, the first emotional information corresponding to each of the descriptive statements is matched with the second emotional information, which can improve the correlation between the emotional information expressed by the descriptive statement and the second emotional information corresponding to the image to be captioned, thereby improving the correlation between the caption information and the target object in the image to be captioned, thereby making it possible to determine that the caption information matches the target object, thereby improving the accuracy of the caption information. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without inventive work.

[0051] Figure 1 A flowchart of the image captioning method provided in this application.

[0052] Figure 2 This is an example flow chart of the process for obtaining the evaluation score of the image caption method provided in this application.

[0053] Figure 3 This is an example flow chart of the process of obtaining the second emotional information of the image caption method provided in this application.

[0054] Figure 4 This is an example flow chart of the process of obtaining the second emotional information of the image caption method provided in this application.

[0055] Figure 5 This is a flowchart of the process of obtaining the feature map of the image caption method provided in this application.

[0056] Figure 6 This is a schematic diagram of the image to be captioned in the image captioning method provided in this application.

[0057] Figure 7 for Figure 6 The diagram shows a schematic diagram of the image to be captioned corresponding to the captioned image.

[0058] Figure 8 This is a flowchart of another embodiment of the image captioning method provided in this application.

[0059] Figure 9 This is a schematic diagram of the structural principle of the image captioning device provided in this application.

[0060] Figure 10 This is a schematic diagram of the structure of the terminal device provided in this application. DETAILED DESCRIPTION

[0061] This application provides an image captioning method, storage medium, and terminal device. To make the purpose, technical solution, and effects of this application more clear and explicit, the application is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0062] It will be understood by those skilled in the art that, unless expressly stated otherwise, the singular forms "a", "an", "said" and "the" used herein may also include the plural forms. It should be further understood that the term "comprising" used in the specification of the present application refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be intermediate elements. In addition, "connected" or "coupled" as used herein may include wireless connections or wireless couplings. The term "and / or" used herein includes all or any units and all combinations of one or more associated listed items.

[0063] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by those skilled in the art to which this application belongs. It should also be understood that terms such as those defined in common dictionaries should be understood to have meanings consistent with their meanings in the context of the prior art and will not be interpreted in an idealized or overly formal sense unless specifically defined as herein.

[0064] After research, the inventors found that with the popularization of mobile smart terminals, users will take pictures anytime and anywhere and share them to social networks or other smart terminals, and will add beautiful texts to the pictures during the picture sharing process to enhance the viewer's sense of immersion. However, the existing image text matching methods generally have a large deviation between the configured text and the image content, and are unable to accurately express the image content. For example, the text matching method based on image information keywords selects text matching information from a preset corpus. Due to the inaccuracy of the existing image information recognition itself, the correlation between the image information keywords and the image will be poor, thereby affecting the accuracy of the text matching information. For another example, the text matching information is generated based on image recognition / natural language models. Since the text matching information is determined based on the recognition results of the image object information, and the deviation of the recognition results will affect the accuracy of the text matching information, the text matching information cannot accurately express the image content.

[0065] In order to solve the above problem, in an embodiment of the present application, after obtaining the image to be accompanied by text, several descriptive sentences corresponding to the image to be accompanied by text, and the first emotional information corresponding to each descriptive sentence are obtained; the second emotional information corresponding to the image to be accompanied by text is obtained; based on the several descriptive sentences, the first emotional information corresponding to each of the several descriptive sentences, and the second emotional information, the accompanying text information corresponding to the image to be accompanied by text is determined. In this way, the first emotional information corresponding to each descriptive sentence is matched with the second emotional information, which can improve the correlation between the emotional information expressed by the descriptive sentence and the second emotional information corresponding to the image to be accompanied by text, thereby improving the relevance of the accompanying text information and the target object in the image to be accompanied by text, so that it can be determined that the accompanying text information matches the target object, thereby improving the accuracy of the accompanying text information.

[0066] The application content will be further explained below through description of embodiments in conjunction with the accompanying drawings.

[0067] This embodiment provides a method for attaching text to an image. The method can be performed by an image attachment device. The device can be implemented by software and applied to smart terminals such as smart phones, tablet computers, or personal digital assistants. Figure 1 , a method for captioning an image provided by this embodiment specifically includes:

[0068] S10. Obtain several descriptive sentences corresponding to the image to be captioned, and first emotion information corresponding to each descriptive sentence.

[0069] Specifically, the images to be accompanied by text can all be obtained by photographing through an imaging system (such as a camera, a video camera, an under-screen camera, etc.), or they can all be images photographed by other external devices (such as a digital camera, etc.) and stored in the terminal device, or they can all be images sent to the terminal device through the cloud. In this embodiment, the images to be accompanied by text are images photographed by an imaging system, wherein the imaging system can be configured by the terminal device itself or by other devices. For example, the images to be accompanied by text are all landscape images obtained by photographing a landscape scene with a mobile phone configured with an imaging system. For another example, the images to be accompanied by text are images of the same kitten photographed by a digital camera and sent to the terminal device with the kitten.

[0070] In an implementation of this embodiment, before obtaining the target confidence of the target object in the image to be captioned, the method further includes:

[0071] When an image selection operation is detected, a target image corresponding to the image selection operation is obtained, and the obtained target image is used as the image to be matched with text.

[0072] Specifically, the image selection operation can be formed by triggering a control button configured on the terminal device itself that runs the image configuration method provided by this embodiment, or it can be formed by calling an image through a third application (for example, the operation of selecting to upload a photo when using an application such as WeChat or Weibo, or the operation of selecting to send a photo when chatting through a chat tool such as WeChat or QQ, or the operation of selecting an image through photo editing software, etc.), or it can be formed based on a control instruction sent by an external device, etc. The target image is the image to be selected for the image selection operation, for example, an image to be sent to a friend, etc., and after determining the target image, the target image is used as the image to be matched with the text. In this way, after the image is selected according to the image selection operation, the image can be automatically configured, which brings convenience to the user.

[0073] Each of the several descriptive sentences is text information used to describe the target object in the image to be captioned, and the first emotional information corresponds to the descriptive sentence and is used to reflect the emotional state expressed by the descriptive sentence. The first emotional information can be one of joy, charm, elegance, sweetness and anxiety. For example, the first emotional information is charming, etc.

[0074] Each description statement in the plurality of description statements and the first emotion information corresponding to each description statement can be regarded as a description statement group, and the plurality of description statements can form a plurality of description statement groups. For any two description statement groups in the plurality of description statement groups, at least one target item in the description statements in the first description statement group of the two description statement groups and the first emotion information in the first description statement group is different from the reference item in the second description statement group of the two description statement groups, wherein when the target item is a description statement, the reference item is the description statement, and when the target item is the first emotion information, the reference item is the first emotion information.

[0075] For example: several description statements include description statement A, description statement B, and description statement C. Description statement A corresponds to the first emotional information a, description statement B corresponds to the first emotional information b, and description statement C corresponds to the first emotional information layer c. Then description statement A and the first emotional information a constitute description statement group 1, description statement B and the first emotional information b constitute description statement group 2, and description statement C and the first emotional information c constitute description statement group 3. Description statement B in description statement group 2 is different from description statement A in description statement group 1, and the first emotional information b in description statement group 2 is the same as the first emotional information a in description statement group 1; description statement B in description statement group 2 is different from description statement C in description statement group 3, and the first emotional information b in description statement group 2 is different from the first emotional information c in description statement group 3; description statement A in description statement group 1 is the same as description statement C in description statement group 3, and the first emotional information a in description statement group 1 is the same as the first emotional information c in description statement group 3.

[0076] In one implementation of this embodiment, the plurality of descriptive sentences corresponding to the image to be captioned and the first emotional information corresponding to each descriptive sentence can be obtained through a trained sentence prediction model, wherein the sentence prediction model is configured with a plurality of default emotions and a plurality of preset descriptive sentences. Accordingly, the step of obtaining the plurality of descriptive sentences corresponding to the image to be captioned and the first emotional information corresponding to each descriptive sentence specifically includes:

[0077] Inputting the image to be captioned into a trained sentence prediction model;

[0078] The sentence prediction model is used to determine a plurality of descriptive sentences corresponding to the image to be captioned and the first emotion information corresponding to each descriptive sentence.

[0079] Specifically, the sentence prediction model can be pre-trained and used to obtain several descriptive sentences corresponding to the image to be accompanied by text, and the first emotional information corresponding to each descriptive sentence. It can be understood that the input item of the sentence prediction model is the information to be accompanied by text, and the input items are several descriptive sentences and the first emotional information corresponding to each descriptive data. Several default emotions and several preset descriptive sentences are pre-configured for the sentence prediction model and are used as preset descriptive sentences and default emotions of the sentence prediction model. In other words, for the image to be accompanied by text input into the sentence prediction model, each of the several descriptive sentences corresponding to the image to be accompanied by text output by the sentence prediction model is included in several preset descriptive sentences, and the first emotional information corresponding to each descriptive sentence is included in several default emotions.

[0080] Each of the plurality of default emotions is distinct from another, and each of the plurality of preset descriptive statements is distinct from another. Furthermore, for each default emotion, at least one of the plurality of preset descriptive statements contains an emotional information that matches the default emotion. For example, the plurality of default emotions includes elegance, and a description statement A exists among the plurality of preset descriptive statements, reflecting the emotional information of elegance. In one implementation of this embodiment, the plurality of default emotions may include joy, charm, elegance, sweetness, and anxiety.

[0081] The several default emotions and several preset descriptive sentences are configured in the sentence prediction model as configuration parameters of the sentence prediction model. The sentence prediction model is preset with several configuration parameters, wherein the several configuration parameters may include learning rate parameters, reverse learning measurement parameters, default emotion parameters, preset description sentence parameters, etc. After determining the model structure of the sentence prediction model, it is necessary to configure the sentence prediction model with its corresponding several configuration parameters, and use the sentence prediction model configured with the configuration parameters as the sentence prediction model. Among them, the preset description sentence parameters are used to configure the preset description sentences corresponding to the sentence prediction model, so that the sentence prediction model can use the preset description sentences to annotate the image to be accompanied by the text, so as to obtain several descriptive sentences corresponding to the image to be accompanied by the text. The default emotion parameters are used to configure the default emotion corresponding to the sentence prediction model, so that the sentence prediction model can use the default emotion to annotate the emotional information corresponding to each descriptive sentence corresponding to the image to be accompanied by the text.

[0082] Based on this, the process of configuring the sentence prediction model with several default emotions and several preset descriptive sentences can be specifically as: using the several default emotions as default emotion parameters, using several default descriptive words as preset descriptive sentence parameters, and configuring the default emotion parameters and preset descriptive sentence parameters in the sentence prediction model, so that the default emotions configured by the sentence prediction model are several default emotions, and the configured preset descriptive sentences are several preset descriptive sentences. In this way, when the sentence prediction model determines the several descriptive sentences corresponding to the image to be matched with the text and the first emotional information corresponding to each descriptive sentence, the several preset descriptive sentences and the several default emotions are used as the several descriptive sentences corresponding to the image to be matched with the text and the first emotional information corresponding to each descriptive sentence.

[0083] In one implementation of this embodiment, the sentence prediction model is used to determine the plurality of description sentences corresponding to the image to be accompanied by text and the first emotion information corresponding to each description sentence.

[0084] Determining, by the sentence prediction model, a plurality of image groups corresponding to the image to be accompanied by text and an evaluation score corresponding to each of the plurality of image groups;

[0085] Determining, based on each image group and the evaluation score corresponding to each image group, a set of image groups corresponding to the image to be captioned;

[0086] For each target image group in the image group set, the preset description sentence included in the target image group is used as the description sentence corresponding to the image to be captioned, and the first emotion information included in the target image group is used as the first emotion information corresponding to the description sentence.

[0087] Specifically, the plurality of image groups are determined based on images to be accompanied by text, a plurality of default emotions, and a plurality of preset descriptive sentences. Each image group includes an image to be accompanied by text, a default emotion, and a preset descriptive sentence, wherein the default emotion is one of the plurality of default emotions, and the preset descriptive sentence is one of the plurality of preset descriptive sentences. For any two image groups in the plurality of image groups, the default emotions and / or preset descriptive sentences in the two image groups are different. In other words, the default emotions and preset descriptive sentences in the two image groups are not the same at the same time. For example, the plurality of image groups are terminal image group A and image group B, image group A includes default emotion 1a and preset descriptive sentence 1b, and image group B includes default emotion 2a and preset descriptive sentence 2b. Then, the default emotion 1a is different from the default emotion 2a, or the preset descriptive sentence 1b is different from the preset descriptive sentence 2b, or the default emotion 1a is different from the default emotion 2a, and the preset descriptive sentence 1b is different from the preset descriptive sentence 2b.

[0088] The number of image groups is determined based on the number of default emotions and the number of preset descriptive statements, where the number of image groups is equal to the product of the number of default emotions and the number of preset descriptive statements. For example, if the number of default emotions is 2 and the number of preset descriptive statements is 3, then the number of image groups is 6. In other words, for each default emotion, the number of image groups that includes that default emotion is equal to the number of preset descriptive statements; and for each preset descriptive statement, the number of image groups that includes that preset descriptive statement is equal to the number of default emotions.

[0089] For example: several default emotions include default emotion A and default emotion B, and several preset description sentences include preset description sentence a, preset description sentence b and preset description sentence c. Then, according to the image to be accompanied by the text, several default emotions and several default description sentences, the image group is determined to include: [image to be accompanied by the text, default emotion A, preset description sentence a]; [image to be accompanied by the text, default emotion A, preset description sentence b]; [image to be accompanied by the text, default emotion A, preset description sentence c]; [image to be accompanied by the text, default emotion B, preset description sentence a]; [image to be accompanied by the text, default emotion B, preset description sentence b]; [image to be accompanied by the text, default emotion B, preset description sentence c].

[0090] The evaluation score is used to reflect the correlation between the description statement and the first emotional information corresponding to the description statement and the picture, wherein the higher the score, the higher the correlation between the description statement and the first emotional information corresponding to the description statement and the picture; conversely, the lower the score, the lower the correlation between the description statement and the first emotional information corresponding to the description statement and the picture. For example, the evaluation score corresponding to the description information group A composed of the description statement and the first emotional information corresponding to the description statement is 90, and the evaluation score corresponding to the description information group A composed of the description statement and the first emotional information corresponding to the description statement is 80, then the correlation between the description information group A and the image to be accompanied by the text is higher than the correlation between the description information group B and the image to be accompanied by the text.

[0091] In one implementation of this embodiment, Figure 2 As shown, the process of determining the evaluation score specifically includes:

[0092] For each image group, the sentence prediction model is used to determine the image features corresponding to the images to be captioned in the image group, the emotional features corresponding to the default emotions in the image group, and the sentence features corresponding to the preset description sentences in the image group; the sentence prediction model, the image features, the emotional features, and the sentence features are used to determine the evaluation score corresponding to the image group.

[0093] Specifically, the image feature is a feature determined based on the image to be accompanied by the text, and is used to form an image feature vector reflecting the image detail information of the image to be accompanied by the text. The emotional feature is determined based on the default emotion, and is used to form an emotional vector reflecting the default emotion. The sentence feature is determined based on a preset description sentence, and is used to form a word vector reflecting the preset description sentence. The vector dimension of the image feature vector, the vector dimension of the emotional vector, and the vector dimension of the word vector are equal. For example, the vector dimension of the image feature vector, the vector dimension of the emotional vector, and the vector dimension of the word vector are all 500.

[0094] In one implementation of this embodiment, the sentence prediction model includes a feature extraction module, a linear module, and a word feature extraction module. The sentence prediction model determines the image features corresponding to the image to be captioned, the emotional features corresponding to the default emotion in the image group, and the sentence features corresponding to the preset description sentence in the image group, specifically including:

[0095] Inputting the image to be accompanied by text in the image group into the feature extraction module, and determining the image features corresponding to the image to be accompanied by text by the feature extraction module;

[0096] Inputting the default emotion in the image group into the linear module, and determining the emotional features corresponding to the default emotion through the linear module;

[0097] The preset description sentence in the image group is input into the word feature extraction module, and the word feature extraction module determines the sentence feature corresponding to the preset description sentence.

[0098] Specifically, the feature extraction module includes a residual unit and a first feedforward network unit. The input of the residual unit is the image to be captioned, and the input of the first feedforward network unit is the output of the residual unit. The output of the first feedforward network unit is the image features. For example, the residual unit uses a trained Resnet52 network. The input of the residual unit is an image with a scale of 3*224*224, and the output is 1*1*2048. The input of the feedforward network unit is 1*1*2048, and the output is a 500-dimensional vector.

[0099] The input item of the linear module is a 1*215 hot vector, and the output item is a 500-dimensional vector. The word feature extraction module includes a natural language unit and a second feedforward network unit. The input item of the natural language unit is a preset description sentence, the input item of the second feedforward network unit is the output item of the natural language unit, and the output item of the second feedforward network unit is a sentence feature. For example, the natural language unit adopts a trained Transformer model. The Transformer model includes 4 parallel network layers and an attention layer. The 4 parallel network layers are all connected to the attention layer, and each of the 4 parallel network layers is configured with 300 hidden units. The output item of the Transformer model is a 300-dimensional vector, the input item of the second feedforward network unit is a 300-dimensional vector, and the output item is a 500-dimensional vector.

[0100] The image group set corresponding to the image to be captioned includes a preset number of target image groups, and each target image group in the image group set is included in a plurality of image groups, and the evaluation scores corresponding to each target image group meet a preset condition. It is understood that each target image group in the image group set is a target image group selected from the plurality of image groups whose evaluation scores meet the preset condition; the evaluation scores of each image group in the plurality of image groups, excluding the image group set, do not meet the preset condition. The preset condition can be an image group whose evaluation score reaches a preset score threshold, or a preset number of image groups selected from the largest to the smallest evaluation scores, etc.

[0101] In one implementation of this embodiment, the sentence prediction model further includes a first fusion module and a second fusion module, and determining the evaluation score corresponding to the image group based on the sentence prediction model, the image feature, the emotional feature, and the sentence feature specifically includes:

[0102] Inputting the image feature and the emotion feature into the first fusion module, and outputting the fusion feature through the first fusion module;

[0103] The fusion feature and the sentence feature are input into the second fusion module, and the evaluation score corresponding to the image group is output through the second fusion module.

[0104] Specifically, the feature extraction module and the linear module are both connected to the first fusion module, and the first fusion module and the word feature extraction module are both connected to the second fusion module. The input item of the feature extraction module is the image to be captioned, the input item of the linear module is the default emotion, and the input item of the word feature extraction module is the preset description sentence. The input item of the first fusion module is the image features output by the feature extraction module and the emotion features output by the linear module. The input item of the second fusion module is the output item of the first fusion module and the sentence features output by the word feature extraction module. The output item of the second fusion module is the evaluation score corresponding to the image group.

[0105] Based on this, the process of obtaining the evaluation score corresponding to the image group can be: input the image to be accompanied by text into the feature extraction module, and determine the image features through the feature extraction module; input the default emotion into the linear module, and output the emotion features through the linear module; input the preset description sentence into the word feature extraction module, and output the sentence features through the word feature extraction module; input the image features and emotion features into the first fusion module, and output the fusion features through the first fusion module; input the fusion features and sentence features into the second fusion module, and output the evaluation score corresponding to the image group through the second fusion module.

[0106] In a specific implementation of this embodiment, the sentence prediction model includes an activation function layer and a log-likelihood function layer, and the activation function layer is configured with a softmax layer. After obtaining the evaluation scores corresponding to each image group, each evaluation score is input into the softmax layer, and each evaluation score is normalized to between 0 and 1 through the softmax layer. Then, each normalized evaluation score is input into the log-likelihood function layer, and the log-likelihood function layer is used to determine the image group set corresponding to the images to be accompanied by text. After obtaining the image group set, for each image group in the image group set, the preset description sentence included in the image group is used as the description sentence corresponding to the image to be accompanied by text, and the default emotion included in the image group is used as the first emotion information corresponding to the description sentence, so as to obtain several description sentences corresponding to the images to be accompanied by text and the first emotion information corresponding to each description sentence.

[0107] S20: Obtain the second emotional information corresponding to the image to be captioned.

[0108] Specifically, the second emotional information is used to describe the emotional state of the target object in the image to be captioned, wherein the second emotional information can be one of joy, charm, elegance, sweetness, and anxiety. For example, the second emotional information is elegance. In one implementation of this embodiment, the second emotional information can be determined by an image emotion analysis model. Accordingly, the second emotional information corresponding to the image to be captioned is specifically obtained as follows:

[0109] The image to be accompanied by text is input into a trained image sentiment analysis model, and the second sentiment information corresponding to the image to be accompanied by text is determined by the image sentiment analysis model.

[0110] Specifically, the image sentiment analysis model can be a pre-trained network model, and the image sentiment analysis model is used to obtain the second sentiment information of the image to be accompanied by the text. The input item of the image sentiment analysis model is the image to be accompanied by the text, and the output item of the image sentiment analysis model is the second sentiment information. In one implementation of this embodiment, the image sentiment analysis model includes a feature attention module and a visual attention module. The input item of the feature attention module is the image to be accompanied by the text, and the input item of the visual attention module is the image to be accompanied by the text. Correspondingly, Figure 3 and Figure 4 As shown, the step of inputting the image to be accompanied by text into a trained image sentiment analysis model and determining the second sentiment information corresponding to the image to be accompanied by text by using the image sentiment analysis model specifically includes:

[0111] Inputting the image to be accompanied by text into the feature attention module, and determining a feature map of the image to be accompanied by text by the feature attention module;

[0112] Inputting the image to be accompanied by text into the visual attention module, and determining a visual attention map of the image to be accompanied by text by the visual attention module;

[0113] Based on the feature map and the visual attention map, second emotion information corresponding to the image to be captioned is determined.

[0114] Specifically, the feature map is used to reflect the local features of the image to be accompanied by text, and the feature map is output by the feature attention module. The visual attention map is used to reflect the salient features of the image to be accompanied by text, and the visual attention map is output by the visual attention module. Figure 5 As shown, the feature attention module includes a feature extraction unit, a plurality of convolution units and a fusion unit. The input item of the feature extraction unit is the image to be matched with the text, the output item of the feature extraction unit is a plurality of first feature maps, the input item of one convolution unit in the plurality of convolution units is the image to be matched with the text, and the remaining convolution units in the plurality of convolution units except the convolution unit corresponding to the image to be matched with the text are combined with the plurality of first feature maps. Figure 1 One by one, the input item of each first feature map in the plurality of first feature maps is the input item of its corresponding convolution unit; the output items of the plurality of convolution units are the input items of the fusion unit, and the output items of the fusion unit are the feature maps of the image to be matched with text.

[0115] S30. Determine the text information corresponding to the image to be texted based on a plurality of descriptive statements, the first emotional information corresponding to each of the plurality of descriptive statements, and the second emotional information, wherein the text information is included in the plurality of descriptive statements.

[0116] Specifically, the text information is text information configured in the image to be texted, and is used to reflect the image information in the image to be texted. For example, the image to be texted is an image including a kitten, and the text information can be a cute kitten, which is used to reflect the kitten in the image to be texted. After the first emotional information and the second emotional information corresponding to each descriptive sentence are determined, the first emotional information and the second emotional information can be matched, and the text information corresponding to the image to be texted is determined based on the matching result. Accordingly, in an implementation method of this embodiment, the text information corresponding to the image to be texted based on a number of descriptive sentences, the first emotional information corresponding to each of the description sentences in the number of description sentences, and the second emotional information specifically includes:

[0117] For each description sentence, determining a matching degree between the first emotion information configured for the description sentence and the second emotion information;

[0118] Based on the matching degrees corresponding to the respective first emotion information, the text information corresponding to the image to be texted is determined.

[0119] Specifically, the matching degree is used to measure the degree of similarity between the emotional state reflected by the first emotional information and the emotional state reflected by the second emotional information. For example, if the first emotional information A corresponding to the description sentence a is charming, the first emotional information B corresponding to the description sentence b is anxious, and the second emotional information is joyful, then the degree of similarity between the first emotional information A and the second emotional information is higher than the degree of similarity between the first emotional information B and the second emotional information. After obtaining the matching degree corresponding to each piece of first emotional information, the description sentence corresponding to the first emotional information with the highest matching degree can be selected as the text information corresponding to the image to be accompanied by the text; alternatively, the first emotional information whose matching degree reaches a preset matching threshold can be selected, and a description sentence corresponding to the first emotional information can be randomly selected from the selected first emotional information as the text information corresponding to the image to be accompanied by the text.

[0120] In one implementation of this embodiment, determining the text information corresponding to the image to be texted based on the matching degree corresponding to each piece of first emotional information specifically includes:

[0121] The matching degrees corresponding to the respective first emotion information are respectively used as the matching degrees corresponding to the respective description sentences corresponding to the respective first emotion information;

[0122] A target description sentence is selected from several description sentences based on the matching degree.

[0123] Specifically, after obtaining the matching degree corresponding to each descriptive statement, the descriptive statement with the greatest matching degree is selected, and the selected descriptive statement is used as the target descriptive statement. It is understandable that the target descriptive statement is one of the multiple descriptive statements, and the matching degree corresponding to the target descriptive statement is greater than the matching degree corresponding to any of the multiple descriptive statements other than the target descriptive statement. Of course, when there are multiple descriptive statements with the greatest matching degree, the multiple descriptive statements with the greatest matching degree can be used as the target descriptive statement, and one of the multiple target descriptive statements can be randomly selected, or the multiple target descriptive statements can be output to the user for selection, etc.

[0124] In one implementation of this embodiment, after acquiring the text information, the text information can be added to the image to be texted. Accordingly, after determining the text information corresponding to the image to be texted based on the target confidence, the method further includes:

[0125] The text information is added to the image to be matched with a text, so as to obtain a text image corresponding to the image to be matched with a text.

[0126] Specifically, the adding of the text information to the image to be accompanied by text refers to adding the text information to an image area in the image to be accompanied by text, so that the text information is carried in the text image corresponding to the image to be accompanied by text. When adding the text information to the image to be accompanied by text, the image area of ​​the target object corresponding to the text information in the image to be accompanied by text can be obtained, and the text information is added outside the image area, and the distance between the image area where the text information is located and the image area meets a preset condition, for example, the distance is less than a preset distance threshold, etc.; or, the text information is added to a specified image area of ​​the image to be accompanied by text, for example, the upper left corner area, etc. For example, Figure 6 After adding the text information to the image to be matched, the following is obtained: Figure 7 The accompanying image.

[0127] In summary, the present application provides a method for captioning an image, the method comprising obtaining a plurality of descriptive statements corresponding to an image to be captioned, and first emotional information corresponding to each of the descriptive statements; obtaining second emotional information corresponding to the image to be captioned; and determining the caption information corresponding to the image to be captioned based on the plurality of descriptive statements, the first emotional information corresponding to each of the plurality of descriptive statements, and the second emotional information. In this way, the first emotional information corresponding to each of the descriptive statements is matched with the second emotional information, which can improve the correlation between the emotional information expressed by the descriptive statement and the second emotional information corresponding to the image to be captioned, thereby improving the correlation between the caption information and the target object in the image to be captioned, thereby making it possible to determine that the caption information matches the target object, thereby improving the accuracy of the caption information.

[0128] In one embodiment, in order to further improve the accuracy of the text information, before determining the text information corresponding to the image to be texted, the method of determining the text information corresponding to the image to be texted can be used. Figure 8 As shown, before obtaining a plurality of descriptive sentences corresponding to the image to be captioned and the first emotion information corresponding to each descriptive sentence, the method may further include:

[0129] A10, obtaining the target confidence of the target object in the image to be captioned;

[0130] A20. Based on the target confidence, determine a method for acquiring the text information corresponding to the image to be accompanied by a text, and use the acquisition method to determine the text information corresponding to the image to be accompanied by a text.

[0131] Specifically, the target object is an object carried by the image to be accompanied by a text, and the object can be an item, person, or animal in the image to be accompanied by a text, or it can be the shooting scene of the image to be accompanied by a text. For example, if the image to be accompanied by a text is an image carrying a kitten, then the target object can be the kitten in the image. The target confidence is used to reflect the credibility of the target object, wherein the larger the value of the target confidence is, the higher the credibility of the target object is; conversely, the smaller the value of the target confidence is, the lower the credibility of the target object is. In one implementation of this embodiment, the defect confidence ranges from 0 to 1, and when the target confidence is 1, the credibility of the target object is higher than when the target confidence is 0.1.

[0132] In one implementation of this embodiment, the image to be accompanied by text may carry one object or multiple objects (wherein, multiple objects include the case of two objects). When the image to be accompanied by text carries one object, the object can be directly used as the target object to obtain the target confidence of the target object. When the image to be accompanied by text carries multiple objects, it is necessary to obtain all objects in the image to be accompanied by text and then select one object from all the objects as the target object to obtain the target confidence of the target object of the image to be accompanied by text.

[0133] Based on this, in one implementation of this embodiment, obtaining the target confidence of the target object in the image to be captioned specifically includes:

[0134] Obtain several candidate objects in the image to be captioned, and the candidate confidence corresponding to each candidate object;

[0135] Based on the candidate confidences corresponding to the candidate objects, the target object corresponding to the image to be matched with text is determined to obtain the target confidence of the target object.

[0136] Specifically, the several candidate objects are all objects carried by the image to be accompanied by text, and the several candidate objects include all objects in the image to be accompanied by text. The candidate confidence corresponding to each candidate object in the several candidate objects is used to reflect the credibility of each candidate object. For example, the several candidate objects include candidate object A and candidate object B, candidate object A corresponds to candidate confidence a, and candidate object B corresponds to candidate confidence b. Then candidate confidence a is used to reflect the credibility of candidate object A, and candidate confidence b is used to reflect the credibility of candidate object B. When candidate confidence a is greater than candidate confidence b, it means that the credibility of candidate object A is higher than that of candidate object B.

[0137] The candidate objects and the candidate confidences corresponding to the candidate objects can be determined by traditional methods, such as machine learning methods, or by deep learning methods, such as convolutional neural network models. In one implementation of this embodiment, the candidate objects and the candidate confidences corresponding to the candidate objects are determined by deep learning methods. Accordingly, obtaining a number of candidate objects in the image to be accompanied by text, and the candidate confidences corresponding to each candidate object can be specifically as follows: inputting the image to be accompanied by text into a trained image recognition model, and outputting each candidate object in the image to be accompanied by text and the candidate confidences corresponding to each candidate object through the image recognition model. The image recognition model is a pre-trained neural network model, the input of which is the target image, and the output is each candidate object in the target image, and the candidate confidences corresponding to each candidate object. For example, the image recognition model is a convolutional neural network model.

[0138] The target object is included in a plurality of candidate objects, and the target confidence of the target object is greater than the candidate confidence corresponding to each candidate object other than the target object. It is understandable that the target object is the candidate object with the largest candidate confidence among the plurality of candidate objects. For example, the plurality of candidate objects include candidate object A, candidate object B, and candidate object C. The candidate confidence corresponding to candidate object A is 0.8, the candidate confidence corresponding to candidate object B is 0.3, and the candidate confidence corresponding to candidate object C is 0.9. Then candidate object C is the target object, and the candidate confidence corresponding to candidate object C is the target confidence, that is, the target confidence is 0.9.

[0139] Furthermore, in practical applications, when at least two candidate confidences are obtained among several candidate objects, the candidate confidence of all the candidate confidences with the highest confidence can be obtained, and the candidate confidence with the highest confidence can be selected as the target object from among all the obtained candidate confidences. For example, a candidate confidence with the highest confidence can be randomly selected as the target object, or the candidate confidence with the highest confidence that occupies the largest image area can be selected as the target object, or the candidate confidence with the highest confidence that is closest to the image center can be selected as the target object, etc.

[0140] For example, the candidate objects corresponding to the image to be captioned include candidate A, candidate B, candidate C, and candidate D. The candidate confidence corresponding to candidate A is 0.2, the candidate confidence corresponding to candidate B is 0.6, the candidate confidence corresponding to candidate C is 0.9, and the candidate confidence corresponding to candidate D is 0.9. Then, candidate C and candidate D are the two candidates with the highest candidate confidence. In this case, one candidate object is randomly selected from candidate C and candidate D as the target object, for example, candidate C is selected as the target object.

[0141] Furthermore, in step A20, since the text information is used to reflect the target object in the image to be texted, and the target confidence is used to reflect the degree of credibility of the target, the size of the target confidence will affect the accuracy of the text information. Therefore, when determining the acquisition method based on the target confidence, different acquisition methods can be determined based on the target confidence. For example, when the target confidence is greater than or equal to the first preset threshold, the credibility of the target object can be relatively good. At this time, a method of generating text information based on image recognition / natural language model with high dependence on the target object can be adopted. When the target confidence is less than the first preset threshold and greater than the second preset threshold, it means that the credibility of the target object is average. A text method of selecting text information based on image information keywords in a preset corpus with low dependence on the target object can be adopted. When the target confidence is less than or equal to the second preset threshold, it means that the credibility of the target object is relatively poor. At this time, the target object cannot be relied upon, so a text method of randomly selecting text information can be adopted.

[0142] Based on this, when determining a method for acquiring the text information corresponding to the image to be captioned based on the target confidence, the target confidence can be compared with a first preset threshold and a second preset threshold, and the acquisition method can be divided into three acquisition methods based on the comparison results of the target confidence with the first preset threshold and the second preset threshold. The first preset threshold and the second preset threshold can both be pre-set and serve as a basis for measuring whether the target object corresponding to the target confidence is trustworthy.

[0143] The first of the three acquisition methods is: when the target confidence is greater than or equal to the first preset threshold, the acquisition method of the text information in the image text method described in the above embodiment is adopted. Among them, the target confidence is greater than or equal to the first preset threshold, which means that the target object corresponding to the target confidence is credible. Conversely, when the target confidence is less than the first preset threshold, it means that the target object corresponding to the target confidence is untrustworthy. For example, if the first preset threshold is 0.8 and the target confidence is 0.9, the target object corresponding to the target confidence is credible. For another example, if the first preset threshold is 0.8 and the target confidence is 0.6, the target object corresponding to the target confidence is untrustworthy. In addition, the acquisition process of the first acquisition method has been described in the above embodiment, so it will not be repeated here. For details, please refer to the description of the above steps S10-S30.

[0144] In one implementation of this embodiment, the second acquisition method among the three acquisition methods includes:

[0145] When the confidence level is less than a first preset threshold and greater than a second preset threshold, obtaining image identification information corresponding to the image to be accompanied by text; wherein the first preset threshold is greater than the second preset threshold;

[0146] An image identification category corresponding to the image identification information is determined, and text information corresponding to the image to be texted is selected from a first preset corpus based on the image identification category.

[0147] Specifically, the second preset threshold can be pre-set and used to measure the credibility of the target object corresponding to the target confidence, wherein, when the target confidence is less than the first preset threshold and greater than the second preset threshold, it means that the credibility of the target object corresponding to the target confidence conforms to the method of determining the accompanying text information using the first preset corpus. Conversely, when the target confidence is less than the second preset threshold, it means that the credibility of the target object corresponding to the target confidence does not conform to the method of determining the accompanying text information using the first preset corpus. Of course, when the target confidence is greater than or equal to the first preset threshold, the credibility of the target object corresponding to the target confidence also does not conform to the method of determining the accompanying text information using the first preset corpus. For example, if the first preset threshold is 0.8 and the second preset threshold is 0.5, if the target confidence is 0.6, then the credibility of the target object corresponding to the target confidence conforms to the method of determining the accompanying text information using the first preset corpus. If the target confidence is 0.2, then the credibility of the target object corresponding to the target confidence does not conform to the method of determining the accompanying text information using the first preset corpus.

[0148] The image identification information is used to reflect the object information of the target object carried by the image to be captioned. The target object can be an object, person, animal, or scene in the image to be captioned. In this embodiment, the image identification information can be a keyword used to reflect the object information of the target object carried by the image to be captioned. For example, if the target object in the image to be captioned is a cat, the keyword can be "cat"; if the target object in the image to be captioned is a bicycle, the keyword can be "bicycle", etc.

[0149] The image identification category is included in a pre-established image identification category library. The image identification information corresponds to the image identification category containing the image identification information. For example, if the image identification information is sky, the image identification category corresponding to the image identification information may be a natural scenery category. Furthermore, the image identification category library includes a plurality of image identification categories, each of which is different from the other. For example, the image identification category library includes an architectural category, a natural scenery category, a person category, an animal category, and a plant category.

[0150] The image identification category includes several category levels, each of which is different from the other, and the category levels can be sorted from high to low to form a category level sequence. For example, the image identification category includes three category levels: plant, flower, and pink rose. The category level of the plant category is higher than the category level of the flower category, and the category level of the flower category is higher than the category level of the pink rose category. Therefore, the category identification of the image identification category is the plant category. If the image identification information includes the plant category, the image identification category corresponding to the image identification information is the plant category.

[0151] In one implementation of this embodiment, the hierarchical relationship among the multiple category levels included in the image identification category can be determined based on the number of occurrences of the category identifier corresponding to each category level, where category levels with more occurrences are ranked higher than category levels with fewer occurrences. For example, in the plant category, which includes the plant category, the flower category, and the pink rose category, the number of occurrences of the category identifier "plant" in the plant category is greater than the number of occurrences of the category identifier "flower" in the flower category, and the number of occurrences of the category identifier "flower" in the flower category is greater than the number of occurrences of the category identifier "pink rose" in the pink rose category.

[0152] For example, the pre-established image identification category library may include architecture category, natural scenery category, person category, animal category, plant category, person category, commodity category, vehicle category, non-natural image category, natural image category, public figure category, logo category, and other categories, among which:

[0153] Building categories include: building category, building night scene category and urban night scene category;

[0154] The natural scenery category includes: natural scenery category and sky category;

[0155] Character categories include: character category, character close-up category, and beauty category;

[0156] Animal categories include: Animal category, Mammal category, and Giraffe category;

[0157] Plant categories include: plant category, flower category, and pink rose category;

[0158] The categories of character activities include: character activity category, ceremony activity category and wedding category;

[0159] Product categories include: commodity category, food category, and rice porridge category;

[0160] The transportation categories include: transportation category, bicycle category, and folding bicycle;

[0161] The non-natural image categories include: non-natural image category, art painting category, and oil painting category;

[0162] The natural image categories include: natural image category, cosmic astronomy category, and meteor shower category;

[0163] Public Figure categories include: Public Figure Category, Public Figure Category, and Jackie Chan Category;

[0164] LOGO categories include: LOGO category and TCL category.

[0165] Of course, it is worth noting that for two image identification categories in the pre-established image identification category library, the two image identification categories may have the same category identification at some category levels, when at least one category identification at a category level is different in the two image categories. For example, in the image identification category library, there are image identification categories A and B with the category identification of the building category, where the category identification of image identification category A is city streets at one category level, and the category identification of image identification category B is urban night scenes at one category level; for another example, in the image identification category library, there are image identification categories C and D with the category identification of the human activity category, where the category identification of image identification category C is catering and entertainment activities at one category level, and the category identification of image identification category D is ceremonial activities at one category level.

[0166] In one implementation of this embodiment, the first preset corpus may be pre-set and include a plurality of first preset corpora, each of which corresponds to a category level within an image identification category in the image identification category library, and the plurality of first preset corpora may correspond to the same category level. It is understood that for any category level within any image identification category in the image identification category library, one or more first preset corpora can be obtained in the first preset corpus based on the category level.

[0167] Based on this, the step of determining the image identification category corresponding to the image identification information, and selecting the text information corresponding to the image to be texted from the first preset corpus based on the image identification category specifically includes:

[0168] Determining an image identification category corresponding to the image identification information, and determining an object category level corresponding to the image identification information;

[0169] Selecting a target corpus corresponding to the target category level from a first preset corpus;

[0170] When the target category level is the minimum category level among several category levels, the selected target corpus is used as the caption information corresponding to the image to be captioned.

[0171] Specifically, the target category level corresponds to the same level category identifier as the image identification information. For example, if the image identification information is a bicycle, then the target category level is the bicycle category in the vehicle image category. The target corpus is included in the first preset corpus, and the target corpus corresponds to the target category level. The target corpus can be one or multiple (wherein, multiple includes two cases). When there is one target corpus, the target corpus can be directly obtained. When there are multiple target corpora, multiple target corpora can be obtained, or one target corpus can be randomly selected from the multiple target corpora, or the most frequently used target corpus can be selected based on the frequency of use of the target corpus, or the most recently used target corpus can be selected based on the time when the target corpus was used. For example, if the target category level is "rice porridge", the target corpus corresponding to "rice porridge" includes: ["Drinking porridge can warm the stomach and drive away mental fatigue"; "Drinking porridge can maintain health, combining clearing and nourishing with a strong taste", "Drinking porridge can reduce internal heat, clearing away heat and detoxifying like a good medicine"], or if the target category level is "bicycle", the target corpus corresponding to "bicycle" includes: ["Taking a car is too fast, walking is too slow, only by cycling can you not miss the beautiful scenery"; "Riding on the road, the scenery is in your heart"; "Riding a bicycle is a good exercise, and it does not pollute the air"].

[0172] Furthermore, in order to improve the matching of the target corpus and the image to be captioned, after obtaining the target corpus, the category level of the category level corresponding to the target corpus in the image identification category can be determined, and the obtained target corpus can be modified based on the category level of the category level corresponding to the target corpus in the image identification category. In this embodiment, none of the candidate category levels in the candidate category levels is included in the target category level, and the target category level of the specification is the minimum level in the image identification category. At this time, the target corpus corresponding to the target category level is directly used as the caption information corresponding to the image to be captioned. For example, the target category level is the rice porridge category in the commodity category, and the commodity category includes commodity category, food category and rice porridge category, among which the rice porridge category is included in the food category and the commodity category, so the target corpus corresponding to the rice porridge category - "Drinking porridge can warm the stomach and drive away mental fatigue" can be directly used as the caption information corresponding to the image to be captioned.

[0173] In one implementation of this embodiment, when the target category level is not the minimum category level in the image identification category, the target corpus can be corrected using the category label of the minimum category level in the image identification category, and the corrected target corpus is used as the caption information corresponding to the image to be captioned. Accordingly, the method further includes:

[0174] When the target category level is the minimum category level among a plurality of category levels, obtaining the minimum category level among the plurality of category levels, and determining a reference category label corresponding to the minimum category level;

[0175] Obtaining a target category label selected from the target corpus, wherein the target category label corresponds to the target category level;

[0176] The target category label in the target corpus is replaced with the reference category label to obtain the caption information corresponding to the image to be captioned.

[0177] Specifically, the control category level is included in several candidate category levels, and each candidate category level in the several candidate category levels is not included in the control category level. Of course, the candidate category levels do not include the control category level itself. In other words, in this embodiment, the category level itself is not included in the category level itself. The target category label refers to a keyword used to represent the target category, and the control category label refers to a keyword used to represent the control category label. For example, the target category level is "LOGO", the target category label corresponding to the target category level is "LOGO", the control category level is "TCL", and the control category label corresponding to the control category level is "TCL".

[0178] Furthermore, after obtaining the reference category label, the text information corresponding to the target category label in the target corpus is obtained, and the reference category label is used to replace the selected text information to obtain the caption information corresponding to the image to be captioned. For example, the target category label corresponding to the target category level is "LOGO", the reference category label corresponding to the reference category level is "TCL", and the corpus corresponding to the target category level is ["LOGO is a brand that symbolizes technology and fashion", "LOGO is a household name", "LOGO brand has a very good product image"]. Then, by replacing "LOGO" in the target corpus with the reference category label "TCL", the caption information obtained is ["TCL is a brand that symbolizes technology and fashion", "TCL is a household name", "TCL brand has a very good product image"].

[0179] In addition, it is worth noting that when determining the image identification category corresponding to the image identification information, the third of the three acquisition methods is used to determine the text information corresponding to the image to be accompanied by the text. For the specific description of the third of the three acquisition methods, please refer to the following description.

[0180] In one implementation of this embodiment, the third of the three acquisition methods includes:

[0181] When the confidence level is less than or equal to a second preset threshold, a corpus is selected from the second preset corpus, and the selected corpus is used as the caption information corresponding to the image to be captioned.

[0182] Specifically, the second preset threshold may be pre-set and used to measure the credibility of the target object corresponding to the target confidence level. When the target confidence level is less than or equal to the second preset threshold, the credibility of the target object corresponding to the target confidence level is considered untrustworthy. At this point, the text information corresponding to the image to be captioned cannot be determined based on the target object of the image to be captioned. Instead, a corpus is selected from the second preset corpus and used as the text information corresponding to the image to be captioned. For example, if the second preset threshold is 0.5 and the target confidence level is 0.2, a corpus is selected from the second preset corpus and used as the text information corresponding to the image to be captioned.

[0183] The second preset corpus may be preconfigured and include a plurality of second preset corpora, each of which is different from the other. When selecting a corpus from the second preset corpus, a corpus may be randomly selected, or a corpus may be selected based on a selection frequency (e.g., selecting the corpus with the highest frequency), or a corpus may be selected based on a time when the corpus was added to the second preset corpus (e.g., randomly selecting a corpus whose addition time is within a preset range).

[0184] Based on the above-mentioned image text matching method, this embodiment provides an image text matching device, such as Figure 9 As shown, the image-based text matching device includes:

[0185] The first acquisition module 100 is used to acquire a plurality of descriptive sentences corresponding to the image to be captioned, and first emotion information corresponding to each descriptive sentence;

[0186] The second acquisition module 200 is used to obtain the second emotional information corresponding to the image to be captioned;

[0187] The determination module 300 is used to determine the caption information corresponding to the image to be captioned based on a plurality of descriptive sentences, the first emotion information corresponding to each of the plurality of descriptive sentences, and the second emotion information.

[0188] In addition, it is worth mentioning that the working process of the image-texting device provided in this embodiment is the same as the working process of the above-mentioned image-texting method. For details, please refer to the working process of the image-texting method, which will not be repeated here.

[0189] Based on the above-mentioned image caption method, this embodiment provides a computer-readable storage medium, which stores one or more programs. The one or more programs can be executed by one or more processors to implement the steps in the image caption method as described in the above-mentioned embodiment.

[0190] Based on the above image text matching method, this application also provides a terminal device, such as Figure 10 As shown, it includes at least one processor 20; a display screen 21; and a memory 22. It may also include a communications interface 23 and a bus 24. The processor 20, display screen 21, memory 22, and communications interface 23 can communicate with each other via bus 24. The display screen 21 is configured to display a preset user guidance interface in the initial setup mode. The communications interface 23 can transmit information. The processor 20 can call the logic instructions in the memory 22 to execute the method in the above embodiment.

[0191] In addition, the logic instructions in the memory 22 can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product.

[0192] The memory 22, as a computer-readable storage medium, can be configured to store software programs or computer-executable programs, such as program instructions or modules corresponding to the methods in the embodiments of the present disclosure. The processor 20 executes the software programs, instructions, or modules stored in the memory 22 to perform functional applications and data processing, thereby implementing the methods in the above embodiments.

[0193] The memory 22 may include a program storage area and a data storage area. The program storage area may store an operating system and at least one application required for a function; the data storage area may store data created based on the use of the terminal device. In addition, the memory 22 may include high-speed random access memory and non-volatile memory. For example, various media that can store program code, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, may also be transient storage media.

[0194] In addition, the specific process of loading and executing the multiple instructions in the storage medium and the processor in the terminal device has been described in detail in the above method and will not be described here one by one.

[0195] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for adding text to an image, characterized in that: The method comprises: Obtaining a plurality of descriptive sentences corresponding to the image to be captioned, and first emotion information corresponding to each descriptive sentence; Obtaining the second emotional information corresponding to the image to be captioned; Determining the caption information corresponding to the image to be captioned based on the plurality of descriptive sentences, the first emotion information corresponding to each of the plurality of descriptive sentences, and the second emotion information; The step of obtaining a plurality of description sentences corresponding to the image to be captioned, and the first emotion information corresponding to each description sentence specifically includes: Inputting the image to be captioned into a trained sentence prediction model; Determining, by the sentence prediction model, a plurality of descriptive sentences corresponding to the image to be captioned and first emotional information corresponding to each of the descriptive sentences; The sentence prediction model is configured with a plurality of default emotions and a plurality of preset description sentences, and the determining of the plurality of description sentences corresponding to the image to be captioned and the first emotion information corresponding to each description sentence by the sentence prediction model specifically includes: Determining, by the sentence prediction model, a plurality of image groups corresponding to the image to be captioned and evaluation scores corresponding to each of the plurality of image groups, wherein each image group includes the image to be captioned, a default emotion from among a plurality of default emotions, and a preset description sentence from among a plurality of preset description sentences; Determining, based on each image group and the evaluation score corresponding to each image group, a set of image groups corresponding to the image to be captioned; For each target image group in the image group set, the preset description sentence included in the target image group is used as the description sentence corresponding to the image to be captioned, and the default emotion included in the target image group is used as the first emotion information corresponding to the description sentence.

2. The image text matching method according to claim 1, characterized in that: The determining of the text information corresponding to the image to be texted based on the plurality of descriptive sentences, the first emotion information corresponding to each of the plurality of descriptive sentences, and the second emotion information specifically includes: For each description sentence, determining a matching degree between the first emotion information configured for the description sentence and the second emotion information; Based on the matching degrees corresponding to the respective first emotion information, the text information corresponding to the image to be texted is determined.

3. The image text matching method according to claim 2, characterized in that: The determining of the text information corresponding to the image to be texted based on the matching degree corresponding to each piece of first emotional information specifically includes: The matching degrees corresponding to the respective first emotion information are respectively used as the matching degrees corresponding to the respective description sentences corresponding to the respective first emotion information; A target description statement is selected from a plurality of description statements based on a matching degree, wherein the target description statement is one of the plurality of description statements, and a matching degree corresponding to the target description statement is greater than a matching degree corresponding to any description statement in the plurality of description statements except the target description statement.

4. The image text matching method according to claim 1, characterized in that: For any two image groups among the plurality of image groups, the default emotions and / or preset description sentences in the two image groups are different.

5. The image text matching method according to claim 1, characterized in that: The process of determining the evaluation score specifically includes: For each image group, the sentence prediction model is used to determine the image features corresponding to the images to be captioned in the image group, the emotional features corresponding to the default emotions in the image group, and the sentence features corresponding to the preset description sentences in the image group; the sentence prediction model, the image features, the emotional features, and the sentence features are used to determine the evaluation score corresponding to the image group.

6. The image text matching method according to claim 5, characterized in that: The sentence prediction model includes a feature extraction module, a linear module, and a word feature extraction module. The sentence prediction model determines the image features corresponding to the image to be matched with text, the emotional features corresponding to the default emotion in the image group, and the sentence features corresponding to the preset description sentence in the image group, specifically including: Inputting the image to be accompanied by text in the image group into the feature extraction module, and determining the image features corresponding to the image to be accompanied by text by the feature extraction module; Inputting the default emotion in the image group into the linear module, and determining the emotional features corresponding to the default emotion through the linear module; The preset description sentence in the image group is input into the word feature extraction module, and the word feature extraction module determines the sentence feature corresponding to the preset description sentence.

7. The image text matching method according to claim 6, characterized in that: The sentence prediction model further includes a first fusion module and a second fusion module. Determining the evaluation score corresponding to the image group by using the sentence prediction model, the image feature, the emotional feature, and the sentence feature specifically includes: Inputting the image feature and the emotion feature into the first fusion module, and outputting the fusion feature through the first fusion module; The fusion feature and the sentence feature are input into the second fusion module, and the evaluation score corresponding to the image group is output through the second fusion module.

8. The image text matching method according to claim 6, characterized in that: The feature extraction module includes a residual unit and a first feedforward network unit. The residual unit is cascaded with the first feedforward network unit. The input item of the residual unit is the image to be captioned, and the output item of the first feedforward network unit is the image feature.

9. The image text matching method according to claim 6, characterized in that: The word feature extraction module includes a natural language unit and a second feedforward network unit, the natural language unit and the second feedforward network unit are cascaded, the input item of the natural language unit is a preset description sentence in the image group, and the output item of the second feedforward network unit is a sentence feature.

10. The image text matching method according to claim 1, characterized in that: The method of obtaining the second emotional information corresponding to the image to be matched with text is specifically as follows: The image to be accompanied by text is input into a trained image sentiment analysis model, and the second sentiment information corresponding to the image to be accompanied by text is determined by the image sentiment analysis model.

11. The image text matching method according to claim 10, characterized in that: The image sentiment analysis model includes a feature attention module and a visual attention module. Inputting the image to be accompanied by text into the trained image sentiment analysis model and determining the second sentiment information corresponding to the image to be accompanied by text by using the image sentiment analysis model specifically includes: Inputting the image to be accompanied by text into the feature attention module, and determining a feature map of the image to be accompanied by text by the feature attention module; Inputting the image to be accompanied by text into the visual attention module, and determining a visual attention map of the image to be accompanied by text by the visual attention module; Based on the feature map and the visual attention map, the second emotion information corresponding to the image to be accompanied by text is determined.

12. The image text matching method according to any one of claims 1 to 11, characterized in that: After determining the caption information corresponding to the image to be captioned based on the plurality of descriptive sentences, the first emotion information corresponding to each of the plurality of descriptive sentences, and the second emotion information, the method further includes: The text information is added to the image to be matched with a text, so as to obtain a text image corresponding to the image to be matched with a text.

13. An image-based text matching device, the image-based text matching device being applied to the image-based text matching method according to any one of claims 1 to 12, characterized in that: The image-based text matching device includes: A first acquisition module is used to acquire a plurality of descriptive sentences corresponding to the image to be captioned, and first emotion information corresponding to each descriptive sentence; A second acquisition module is used to obtain second emotional information corresponding to the image to be captioned; The determination module is used to determine the text information corresponding to the image to be texted based on a plurality of descriptive sentences, the first emotional information corresponding to each of the plurality of descriptive sentences, and the second emotional information.

14. A computer-readable storage medium, characterized in that The computer-readable storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps in the image text captioning method according to any one of claims 1 to 12.

15. A terminal device, characterized in that: include: processor, memory, and communication bus; The memory stores a computer-readable program executable by the processor; The communication bus realizes the connection and communication between the processor and the memory; When the processor executes the computer-readable program, the steps of the image captioning method according to any one of claims 1 to 12 are implemented.

Images